Gesture recognition iteration method and device and head-mounted electronic equipment
Through the truth value annotation model and the Finetune model, the gesture recognition system is iteratively trained, and the gesture images collected by multiple cameras are used to generate true values, which solves the problem of fixed actions of the gesture recognition system, realizes personalized settings and iterative functions, and improves the user experience and recognition effects.
Patent Information
- Application Number
- CN202310528365.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-09
- Publication Date
- 2024-11-12
AI Technical Summary
The existing gesture recognition system lacks iterative functions and personalized settings, resulting in fixed gesture recognition interaction actions, unable to optimize the recognition effect, and poor user experience.
The truth value annotation model and Finetune model are used to collect user gesture images through multiple cameras, generate gesture 2D truth and 3D truth, and use these truth values to train the gesture recognition system for Finetune, realizing iterative functions and personalized settings.
The gesture recognition system has iterative functions and personalized settings, which improves the user experience, enhances the broadness and authenticity of the data set, and improves the generalization ability of gesture recognition.
Smart Images

Figure CN118942148A_ABST
Abstract
Description
[Technical field]
[0001] The embodiments of the present application relate to the field of communication technology, and in particular to a gesture recognition iteration method, device and head-mounted electronic device. [Background technology]
[0002] Currently, gesture interaction, handle interaction, and eye tracking can be used for interaction in head-mounted electronic devices. Among them, gesture interaction can convey a large amount of semantics, emotions, and information that conforms to human daily living habits, and is the research focus of human-computer interaction technology.
[0003] In the related art, deep neural networks are used for gesture recognition. Since the gesture recognition system has no iteration function and personalized setting function, the interactive actions of gesture recognition are basically fixed. In other words, the gesture recognition system can only recognize fixed actions such as confirmation, pinching, zooming in, and zooming out. At the same time, users cannot optimize gestures with poor recognition effects. [Summary of the invention]
[0004] The embodiments of the present application provide a gesture recognition iteration method, device and head-mounted electronic device, so that the gesture recognition system has an iteration function and personalized settings, so that the user can personalize the gesture recognition system according to their own needs, thereby improving the user experience.
[0005] In a first aspect, the present application provides a gesture recognition iteration method, which is applied to a head-mounted electronic device, wherein the head-mounted electronic device is equipped with a gesture recognition system and a gesture recognition optimization system, wherein the gesture recognition optimization system includes: a true value labeling model, a data acquisition module, and a Finetune model;
[0006] Determine a target tag, collect a first gesture image corresponding to the target tag, the first gesture image is a gesture image of the user at different angles at the current moment, and the first gesture image is a gesture image of the user at different angles at the current moment;
[0007] Inputting the first gesture image collected by the data collection module into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value, and using the gesture 2D truth value to generate a hand prediction frame, the target image includes the first gesture image;
[0008] The Finetune model is used to perform Finetune training on the gesture recognition system based on the gesture truth value of the target image. The gesture truth value includes the gesture 2D truth value, gesture 3D truth value and hand prediction box.
[0009] Through this application, the gesture recognition system can be trained using the real-time collected gesture truth values, so that the gesture recognition system has iteration functions and personalized setting functions, thereby improving the user experience.
[0010] In some embodiments, the first gesture image is captured by multiple cameras located on the head-mounted electronic device.
[0011] In some embodiments, the gesture recognition system includes: a 2D point detection unit, a 3D point detection unit, a hand detection unit, and a gesture classification unit;
[0012] 2D point detection unit: used to determine the gesture 2D points during gesture recognition;
[0013] 3D point detection unit: used to determine the gesture 3D points during gesture recognition;
[0014] Hand detection unit: used to generate hand prediction boxes during the recognition process;
[0015] A gesture classification unit: configured to output a gesture recognition result according to the gesture 2D point of the second gesture image, the gesture 3D point of the second gesture image, and the hand prediction frame of the second image;
[0016] The steps of using the Finetune model to perform Finetune training on the gesture recognition system according to the true value of the gesture of the target image include:
[0017] Fine-tune the 2D point detection unit, 3D point detection unit, hand detection unit and gesture classification unit using the target image gesture image.
[0018] In some embodiments, in the step of inputting the target image into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value,
[0019] The true value annotation model has a detection unit and a multi-camera algorithm fusion unit:
[0020] The step of inputting the first gesture image collected by the data collection module into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value comprises:
[0021] Input the target image into the detection unit, and output the hand 2D key points of the target image and the corresponding 2D key point confidences;
[0022] The hand 2D key points of the target image and the corresponding 2D key point confidences are input into the multi-camera algorithm fusion unit to output the gesture 2D true value and gesture 3D true value.
[0023] In some embodiments, the detection unit includes: a hand detection subunit and a large model 2D detection subunit;
[0024] Hand detection subunit: used to detect the hand prediction box of the target image;
[0025] Large model 2D detection subunit: used to output the hand 2D key points and the corresponding 2D key point confidences based on the hand prediction box of the target image.
[0026] In some embodiments, the method further includes: performing Finetune fine-tuning on the hand detection subunit and the large model 2D detection subunit using a Finetine module according to the true value of the gesture of the target image.
[0027] In some embodiments, the multi-camera fusion algorithm unit includes: a random triangulation subunit, a weighted density subunit, a Kalman filter subunit, and a multi-camera reprojection subunit;
[0028] The steps of inputting the hand 2D key points of the target image and the corresponding 2D key point confidences into the multi-camera algorithm fusion unit to output the gesture 2D true value and the gesture 3D true value include:
[0029] The 2D key point confidences corresponding to n groups of hand 2D key points are input to the random triangulation subunit, which traverses and combines the n groups of hand 2D key points in pairs, outputs k groups of hand 3D key points and corresponding 3D key point confidences, and outputs a group of hand 3D key points and corresponding 3D key point confidences for every two groups of 2D key points;
[0030] Input k groups of hand 3D key points and their corresponding key point confidences into the weighted density clustering subunit for weighted fusion to output the optimal 3D key points;
[0031] The optimal 3D key point is input into the Kalman filter subunit for filtering and outputting the 3D true value of the gesture;
[0032] The 3D truth value of the gesture is input into the multi-eye reprojection subunit, and the 3D truth value is transformed and projected at each angle to generate the 2D truth value of the gesture.
[0033] In some embodiments, the random triangulation subunit calculates the depth z-axis coordinate of the hand 3D key point using the following formula:
[0034] d=u L -u R ;
[0035] f b —Camera focal length;
[0036] u L 、u R —The imaging plane coordinates of the 2D key points of the gesture on the left and right cameras;
[0037] d—parallax;
[0038] z—the depth z-axis coordinate of the 3D key point of the hand;
[0039] b—Baseline distance of the camera of the first gesture image corresponding to the two sets of hand 2D key points.
[0040] In some embodiments, the step of inputting k groups of hand 3D key points and corresponding key point confidences into the weighted density clustering subunit for weighted fusion to output the optimal 3D key points includes:
[0041] Construct a data set D using k sets of hand 3D key points;
[0042] Select any data object point P from the data set D;
[0043] Set the maximum radius of the neighborhood and the minimum number of points in the neighborhood, and perform density clustering based on the maximum radius of the neighborhood, the minimum number of points in the neighborhood, the data object point P, and the data set D to obtain several clusters;
[0044] Select the group m with the most hand 3D key points from several clusters, and perform confidence-weighted averaging on all hand 3D key points in the m group to output the optimal 3D key points.
[0045] In some implementations, in the step of selecting a group m with the most hand 3D key points from a plurality of clusters, and performing confidence-weighted averaging on all hand 3D key points in the m group to output the optimal 3D key points, the optimal 3D key points are calculated using the following formula:
[0046]
[0047] —The coordinates of the jth hand 3D keypoint of the i-th group;
[0048] —Confidence of the jth hand 3D keypoint in the i-th group;
[0049] —Optimal 3D key points;
[0050] n—The number of 3D key points of the hand in the m group.
[0051] In some embodiments, the 3D true value of the gesture is input into the multi-eye reprojection subunit, and in the step of performing coordinate transformation and projection on the 3D true value at each angle to generate the 2D true value of the gesture,
[0052] The following formula is used to calculate the 2D true value of the gesture:
[0053]
[0054] X W —The horizontal coordinate of the 3D true value of the gesture in the rectangular coordinate system of space;
[0055] Y W —The 3D true value of the gesture is in the ordinate of the spatial rectangular coordinate system;
[0056] Z W —The vertical coordinate of the 3D true value of the gesture in the rectangular coordinate system of space;
[0057] R, T—rotation and translation matrix;
[0058] K—camera intrinsic parameter matrix;
[0059] M—camera extrinsic matrix;
[0060] a x 、a y , μ0, v0—camera intrinsic parameters;
[0061] u—2D true value horizontal coordinate of gesture;
[0062] v—2D true value ordinate of gesture.
[0063] In some embodiments, the target image also includes a cloud image, and the method further includes uploading the collected first gesture image to the cloud, and the target tag is any tag existing in the gesture recognition system.
[0064] In some embodiments, the target tag is a user-defined tag or any tag existing in the gesture recognition system;
[0065] If the target tag is a custom tag, the method further includes: defining the target tag.
[0066] In a second aspect, the present application provides a gesture recognition iteration device, which is applied to a head-mounted electronic device, wherein the head-mounted electronic device is equipped with a gesture recognition system and a gesture recognition optimization system, wherein the gesture recognition optimization system includes: a true value labeling model, a data acquisition module, and a Finetune model, including:
[0067] A collection module: used to respond to the optimization instruction, determine the target tag, and collect the first gesture image corresponding to the target tag, where the first gesture image is the gesture image of the user at different angles at the current moment;
[0068] The calculation module is used to input the first gesture image collected by the data collection module into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value, and generate a hand prediction frame using the gesture 2D truth value, and the target image includes the first gesture image;
[0069] Training module: used to perform Finetune training on the gesture recognition system using the Finetune model based on the target label and the gesture truth value of the target image. The gesture truth value includes the gesture 2D truth value, gesture 3D truth value and hand prediction box.
[0070] In a third aspect, the present application provides a head-mounted electronic device, comprising:
[0071] at least one processor; and at least one memory in communication with the processor, wherein:
[0072] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method of the first aspect.
[0073] In a fourth aspect, the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions enable a computer to execute any method of the first aspect.
[0074] It should be understood that the second to fourth aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.
Brief Description of the Drawings
[0075] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0076] Figure 1 A schematic diagram of the structure of a head-mounted electronic device provided in an embodiment of the present application;
[0077] Figure 2 A schematic diagram of an iterative method for structural gesture recognition of a head-mounted electronic device provided in an embodiment of the present application;
[0078] Figure 3 A schematic diagram of a scenario provided for an embodiment of the present application;
[0079] Figure 4 A schematic diagram of a UI interface display in a scenario provided by an embodiment of the present application;
[0080] Figure 5 A schematic diagram of multi-camera image acquisition provided in an embodiment of the present application;
[0081] Figure 6 A schematic diagram of the structure of a detection unit provided in an embodiment of the present application;
[0082] Figure 7(a) to Figure 7(c) An exemplary detection effect diagram of a detection unit provided in the present application;
[0083] Figure 8A structural diagram of a multi-camera fusion algorithm unit provided in an embodiment of the present application;
[0084] Fig. 9 A schematic diagram of gesture recognition shown in an exemplary embodiment of a scenario provided in an embodiment of the present application;
[0085] Fig.10 A schematic diagram of gesture recognition shown in another exemplary embodiment of a scenario provided in an embodiment of the present application;
[0086] Fig.11 A schematic diagram of gesture recognition shown in an exemplary embodiment of another scenario provided in an embodiment of the present application;
[0087] Fig.12 A schematic diagram of gesture recognition in different scenarios provided by an embodiment of the present application;
[0088] Fig.13 A flowchart of a gesture recognition iterative method provided in an embodiment of the present application;
[0089] Fig.14 A structural diagram of a gesture recognition iteration device provided in an embodiment of the present application. [Specific implementation method]
[0090] In order to better understand the technical solution of the embodiments of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0091] Figure 1 A schematic structural diagram of a head mounted electronic device 100 is shown.
[0092] The head-mounted electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management unit 140, a power management unit 141, a battery 142, an antenna 1, an antenna 2, a mobile communication unit 150, a wireless communication unit 160, an audio unit 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor unit 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor unit 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0093] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the head-mounted electronic device 100. In other embodiments of the present application, the head-mounted electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0094] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0095] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.
[0096] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0097] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0098] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the head-mounted electronic device 100.
[0099] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio unit 170 via the I2S bus to achieve communication between the processor 110 and the audio unit 170. In some embodiments, the audio unit 170 can transmit an audio signal to the wireless communication unit 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.
[0100] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio unit 170 and the wireless communication unit 160 can be coupled via a PCM bus interface. In some embodiments, the audio unit 170 can also transmit audio signals to the wireless communication unit 160 via the PCM interface to implement the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
[0101] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication unit 160. For example, the processor 110 communicates with the Bluetooth unit in the wireless communication unit 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio unit 170 can transmit an audio signal to the wireless communication unit 160 through the UART interface to implement the function of playing music through a Bluetooth headset.
[0102] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to realize the shooting function of the head-mounted electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the head-mounted electronic device 100.
[0103] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication unit 160, the audio unit 170, the sensor unit 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0104] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the head-mounted electronic device 100, and can also be used to transmit data between the head-mounted electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.
[0105] It is understandable that the interface connection relationship between the units illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the head-mounted electronic device 100. In other embodiments of the present application, the head-mounted electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0106] The charging management unit 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management unit 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management unit 140 may receive wireless charging input through a wireless charging coil of the head-mounted electronic device 100. While the charging management unit 140 is charging the battery 142, it may also power the electronic device through the power management unit 141.
[0107] The wireless communication function of the head mounted electronic device 100 may be implemented through antenna 1, antenna 2, mobile communication unit 150, wireless communication unit 160, a modem processor, and a baseband processor.
[0108] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the head-mounted electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0109] The mobile communication unit 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the head-mounted electronic device 100. The mobile communication unit 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication unit 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication unit 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional units of the mobile communication unit 150 can be set in the processor 110. In some embodiments, at least some of the functional units of the mobile communication unit 150 can be set in the same device as at least some of the units of the processor 110.
[0110] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication unit 150 or other functional units.
[0111] The head mounted electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.
[0112] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the head-mounted electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0113] The head-mounted electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0114] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.
[0115] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the head-mounted electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0116] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the head mounted electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0117] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the head-mounted electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music and videos can be stored in the external memory card.
[0118] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the head-mounted electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the head-mounted electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.
[0119] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the head-mounted electronic device 100, which is different from the position of the display screen 194.
[0120] The key 190 includes a power button, a volume button, etc. The key 190 may be a mechanical key or a touch key. The head mounted electronic device 100 may receive key input and generate key signal input related to user settings and function control of the head mounted electronic device 100.
[0121] The following is a brief introduction to the gesture recognition system in the embodiment of the present application:
[0122] The gesture recognition system in the embodiment of the present application includes: a 2D point detection unit, a 3D point detection unit, a hand detection unit and a gesture classification unit;
[0123] 2D point detection unit: used to determine the gesture 2D points during gesture recognition;
[0124] 3D point detection unit: used to determine the gesture 3D points during gesture recognition;
[0125] Hand detection unit: used to generate hand prediction boxes during the recognition process;
[0126] A gesture classification unit: configured to output a gesture recognition result according to the gesture 2D point of the second gesture image, the gesture 3D point of the second gesture image, and the hand prediction frame of the second image;
[0127] The gesture recognition system in the embodiment of the present application has been pre-trained with a gesture database, that is, the gesture recognition system in the embodiment of the present application is already able to recognize gestures in the gesture database. A common method used by those skilled in the art is to extract features from the gesture database to calculate gesture 2D points, gesture 3D points, and hand prediction frames;
[0128] For example, those skilled in the art may calculate the 2D points of the gesture by extracting local features and global features, and calculate the 3D points of the gesture by using a knowledge fusion algorithm;
[0129] There are various ways to calculate gesture 2D points, gesture 3D points and hand prediction frames in the art. In this regard, in this application, the specific implementation method of calculating gesture 2D points, gesture 3D points and hand prediction frames is not limited;
[0130] In combination with the above, it can be seen that the gesture recognition system is trained using a gesture database, and the gesture database has no iteration function, and the user cannot personalize the gesture recognition system, which leads to the teaching of this application:
[0131] Please refer to Figure 2, a gesture recognition optimization system is mounted on a head-mounted electronic device, the gesture recognition optimization system includes: a truth value labeling model, a data acquisition module and a fine-tuning (Finetune) model. When a user inputs an optimization instruction, the head-mounted electronic device responds to the optimization instruction and determines a target label. The data acquisition module is used to collect a first gesture image, where the first gesture image is a gesture image of the user at different angles at the current moment; the first gesture image collected by the data acquisition module is input into the truth value labeling model to output a gesture 2D truth value and a gesture 3D truth value, and a hand prediction frame is generated using the gesture 2D truth value, and the target image includes the first gesture image; the gesture recognition system is Finetune trained using the Finetune model based on the gesture truth value of the target image, the gesture truth value includes the gesture 2D truth value, the gesture 3D truth value and the hand prediction frame, and when the user makes the gesture corresponding to the first gesture image again, the gesture recognition system can generate a target label, thereby achieving personalized setting and iteration of the gesture recognition system;
[0132] The following is a scenario where Figure 3 As shown, user 200 is using virtual reality (VR) glasses 400. When user 200 needs to perform personalized optimization on a certain gesture, user 200 first selects the gesture model optimization function, thereby triggering the optimization instruction. At this time, the local personalized function is selected, and the VR glasses determine the target tag, such as Figure 4 As shown, the user interface (UI) of the VR glasses 400 guides the user to make gestures, and the data acquisition model collects the first gesture image, where the first gesture image is the gesture image of the user at different angles at the current moment. The VR glasses input the target image into the truth value annotation model to output the gesture two-dimensional (2decimal, 2D) truth value and the gesture three-dimensional (3decimal, 3D) truth value, and use the gesture 2D truth value to generate a hand prediction frame. The VR glasses train the gesture recognition system according to the gesture truth value of the target image, and the gesture truth value includes the gesture 2D truth value, the gesture 3D truth value and the hand prediction frame;
[0133] It is worth noting that, first, the target tag is the tag to be optimized. The target tag can be any tag existing in the gesture recognition system, that is, the default tag. For example, assuming that the user thinks that the recognition effect of the zoom-in gesture is not good, the zoom-out gesture tag is the tag to be optimized. Of course, the user 200 can set the target tag by himself. For example, assuming that the user 200 needs to add a search gesture tag, the search gesture tag is the tag to be optimized. In this embodiment, the definition of the target tag needs to be determined. In other words, the user 200 needs to input the definition of the target tag;
[0134] Correspondingly, assuming that the target tag is any tag in the gesture recognition system, the gesture guided by the UI interface of the VR glasses 400 to the user may be a gesture corresponding to the target tag, or a gesture customized by the user. That is to say, assuming that the target tag selected by the user is zoom out, the UI interface of the VR glasses 400 may guide the user to make a gesture corresponding to the zoom out tag, or a gesture redefined by the user. That is to say, the user may customize both the target tag and the gesture to perform local personalized settings on the gesture recognition system.
[0135] Finally, the target image is an image used to optimize gesture recognition. When the user selects the local personalized function, the target image only includes the first gesture image.
[0136] It should be noted that the truth value annotation model generally refers to a model that can output the 2D truth value of a gesture and the 3D truth value of a gesture. The method that can be implemented by a person skilled in the art is to use a multi-camera to collect a first gesture image, use semi-supervised knowledge distillation and multimodal fusion to calculate the 2D truth value of the gesture and the 3D truth value of the gesture, and use the 2D truth value of the gesture to fit and generate a hand prediction frame;
[0137] That is to say, in the related art, there are related records that can output the 2D true value of gestures and the 3D true value of gestures, and the description of the true value annotation model described below is not a limitation of this application;
[0138] Based on this, the gesture recognition system has an iterative function, and users can customize gesture labels and gestures, which improves the user experience;
[0139] Based on the above, please refer to Figure 5 , the method that can be implemented by the personnel in this field is that the data acquisition module uses a multi-eye camera to collect the first gesture image, but because the multi-eye camera is fixed indoors, it cannot cover all scenes, hand shapes and movements actually used by the user. For example, for outdoor scenes, the multi-eye camera cannot collect images, resulting in insufficient extensiveness and practical authenticity of the data set. In order to solve this problem, in the embodiment of the present application, the first gesture image is collected by multiple cameras located in the head-mounted electronic device;
[0140] Furthermore, the head-mounted electronic device simultaneously turns on multiple cameras to capture the first gesture image;
[0141] In this scenario embodiment, when the user first activates the gesture model optimization function, the head-mounted electronic device turns on multiple cameras, and when the user makes a gesture, multiple cameras of the VR glasses start to collect the first gesture image;
[0142] Understandably, the images collected by VR glasses can cover all scenarios, which improves the breadth and authenticity of the data set;
[0143] In combination with the above, the gesture recognition system includes a 2D point detection unit, a 3D point detection unit, a hand detection unit and a gesture classification unit. Here, the gesture recognition system is trained by Finetune fine-tuning. Further, the parameters of the 2D point detection unit, the 3D point detection unit, the hand detection unit and the gesture classification unit are fine-tuned by using the Finetune model, so that when the user makes a gesture corresponding to the first gesture image, a target label is generated, that is, at this time, the gesture recognition system has an iteration function;
[0144] The following is an explanation of the true value annotation model in the embodiment of the present application:
[0145] In this application, the true value annotation model is equipped with a detection unit and a multi-camera algorithm fusion unit:
[0146] Detection unit: used to output the 2D key points of the hand of the target image and the corresponding 2D key point confidence;
[0147] Please note that, please refer to Figure 6 ,The detection unit includes: a hand detection subunit and a large model 2D detection subunit;
[0148] Hand detection subunit: used to detect the hand prediction box of the target image;
[0149] Large model 2D detection subunit: used to output the hand 2D key points and the corresponding 2D key point confidences based on the hand prediction box of the target image.
[0150] Please refer to Figure 7(a) to Figure 7(c) Assuming that the VR glasses have n cameras, the gesture detection unit on the VR glasses inputs n gesture images collected at the same time from different angles, that is, the number of the first gesture images is n, and after normalization and other preprocessing, it is input to the hand detection subunit. The hand detection subunit obtains the hand prediction boxes (bounding boxes, abbreviated as bbox) at n different angles. The specific prediction box is shown in Figure 7(a). Here, the hand prediction box is represented as in Represents the x,y position coordinates of the upper left corner and lower right corner of the hand respectively;
[0151] The input of the large model 2D detection subunit is the hand bbox, that is, when the hand detection subunit generates the box shown in Figure 7(a), the image input to the large model 2D detection subunit is shown in Figure 7(b);
[0152] In the application, the large model 2D detection subunit requires high precision and low speed, so the accuracy needs to be improved as much as possible in this application. The method adopted in this application is to output the hand 2D key points and the corresponding 2D key point confidence based on the heat map method;
[0153] Furthermore, a Sigma heat map is predicted for each key point in the hand bbox, the score at each position is predicted, and multiple different Sigma heat maps are fused and superimposed to supervise the neural network.
[0154] It should be noted that the above method of using a heat map to output the hand 2D key points and the corresponding 2D key point confidences is only one achievable method in this application, and this application does not limit the method of outputting the hand 2D key points and the corresponding 2D key point confidences;
[0155] For example, as shown in FIG7( c ), the large model 2D detection subunit outputs 21 hand 2D key points P 2d =(x i, y i ) and the corresponding 2D key point confidence (Not all 21 hand 2D key points are shown in the figure);
[0156] Multi-camera algorithm fusion unit: used to output the 2D true value of the gesture and the 3D true value of the gesture based on the 2D key points of the hand in the target image and the corresponding 2D key point confidence;
[0157] In combination with the above, one achievable method in the art is to use a knowledge distillation algorithm to fuse images captured by multiple cameras to generate a 3D true value of a gesture, and project the 3D true value of the gesture at various angles to obtain a 2D true value of the gesture. In order to further improve the computing power and the accuracy of the multi-camera algorithm fusion unit, the embodiment of the present application uses a clustering weighted method to remove redundant data, fuse the optimal 3D points, i.e., the 3D true value of the gesture, and project the optimal 3D points at each angle to generate a 2D true value of the gesture.
[0158] For details, please refer to Figure 8 ,The multi-camera fusion algorithm unit includes: random triangulation subunit, weighted density clustering subunit, Kalman filter subunit and multi-camera reprojection subunit;
[0159] The following is an explanation of each unit:
[0160] Random triangulation subunit: used to traverse and combine n groups of hand 2D key points in pairs, output k groups of hand 3D key points and corresponding 3D key point confidences, and output a group of hand 3D key points and corresponding 3D key point confidences for every two groups of 2D key points;
[0161] Understandably, each first gesture image outputs a set of 2D key points of the hand;
[0162] It should be noted that the random triangulation subunit input is n sets of 2D hand key points P at different angles.2d , n groups of hand 2D key points P 2d Traverse the random combination in pairs and output k groups of hand 3D key points and the corresponding 3D key point confidences. Combining probability theory, we know that
[0163] The random triangulation subunit uses the following formula to calculate the depth z-axis coordinate of the hand 3D key point:
[0164] d=u L -u R ;
[0165] f b —Camera focal length;
[0166] u L 、u R —The imaging plane coordinates of the 2D key points of the gesture on the left and right cameras;
[0167] d—parallax;
[0168] z—the depth z-axis coordinate of the 3D key point of the hand;
[0169] b—Baseline distance of the camera of the first gesture image corresponding to the two sets of hand 2D key points.
[0170] b—the baseline distance between the two groups of cameras that capture the first gesture image corresponding to the 2D key points of the hand;
[0171] After obtaining the z-axis coordinate of the depth of the 3D key point of the hand, a person skilled in the art can obtain the horizontal coordinate and the vertical coordinate of the 3D key point of the hand, and the method of the horizontal coordinate and the vertical coordinate of the 3D key point of the hand is not limited;
[0172] At the same time, the 3D key point confidence is composed of the corresponding two 2D point confidences and Multiply them together to get;
[0173] For example, in combination with the above, the output of the large model 2D detection subunit is 21 hand 2D key points P 2d =(x i, y i ) and the corresponding 2D key point confidence
[0174] Here, the random triangulation subunit obtains n groups of 21 hand 2D key points P 2d , and n groups of 21 hand 2D key points P 2d Traverse the combination of two by two, for each group of 21 hand 2D key points P 2d Obtain the 3D key points of the hand and the corresponding 3D key point confidence;
[0175] Weighted density clustering subunit: used to output the optimal 3D key points based on k groups of hand 3D key points and their corresponding key point confidences;
[0176] Specifically, firstly, k groups of hand 3D key points are used to construct a data set D, and any data object point P is selected from the data set D. With point P as the center, the maximum radius of the neighborhood and the minimum number of points in the neighborhood are set, and density clustering is performed according to the maximum radius of the neighborhood, the minimum number of points in the neighborhood, the data object point P and the data set D to obtain several clusters. Finally, the group m with the most hand 3D key points is selected from the several clusters, and the confidence-weighted average of all hand 3D key points in the m group is performed to output the optimal 3D key point.
[0177] Furthermore, in the step of selecting a group m with the most hand 3D key points from a plurality of clusters and performing confidence-weighted averaging on all hand 3D key points in the m group to output the optimal 3D key point, the optimal 3D key point is calculated using the following formula:
[0178]
[0179] —The coordinates of the jth hand 3D keypoint of the i-th group;
[0180] —Confidence of the jth hand 3D keypoint in the i-th group;
[0181] —Optimal 3D key points;
[0182] n—The number of 3D key points of the hand in the m group.
[0183] In this way, abnormal samples are eliminated;
[0184] Kalman filter subunit: used to filter the optimal 3D key points and output the 3D true value of the gesture;
[0185] The input of the Kalman filter subunit is the posterior estimate of the previous moment (moment a-1) Specifically, the dynamic transfer equation of the dynamic variables in the Kalman filter can be established to estimate the state at the current moment (time a), and the prior estimate value at time a can be output. This outputs the stable optimal 3D key point, i.e. the 3D true value of the gesture;
[0186] Multi-eye reprojection subunit: used to transform and project the 3D true value of the gesture at each angle to generate the 2D true value of the gesture;
[0187] The following formula is used to calculate the 2D true value of the gesture:
[0188]
[0189] X W —The horizontal coordinate of the 3D true value of the gesture in the rectangular coordinate system of space;
[0190] Y W —The 3D true value of the gesture is in the ordinate of the spatial rectangular coordinate system;
[0191] Z W —The vertical coordinate of the 3D true value of the gesture in the rectangular coordinate system of space;
[0192] R, T—rotation and translation matrix;
[0193] K—camera intrinsic parameter matrix;
[0194] M—camera extrinsic matrix;
[0195] a x 、a y , μ0, v0—camera intrinsic parameters;
[0196] u—2D true value horizontal coordinate of gesture;
[0197] v—2D true value vertical coordinate of gesture.
[0198] v—2D true value ordinate of gesture;
[0199] Thus, the gesture 2D true value and the gesture 3D true value are generated. The gesture 2D true value is used to generate a hand prediction frame that fits the hand by traversing the upper, lower, left, and right extreme values;
[0200] In summary, in the step of inputting the hand 2D key points of the target image and the corresponding 2D key point confidences into the multi-camera algorithm fusion unit to output the gesture 2D true value and the gesture 3D true value:
[0201] First, the 2D key point confidences corresponding to n groups of hand 2D key points are input into the random triangulation subunit, which traverses and combines n groups of hand 2D key points in pairs, outputs k groups of hand 3D key points and corresponding 3D key point confidences, and outputs a group of hand 3D key points and corresponding 3D key point confidences for every two groups of 2D key points;
[0202] Secondly, k groups of hand 3D key points and their corresponding key point confidences are input into the weighted density clustering subunit for weighted fusion to output the optimal 3D key points;
[0203] Again, the optimal 3D key point is input into the Kalman filter subunit for filtering and outputting the 3D true value of the gesture;
[0204] Finally, the 3D truth value of the gesture is input into the multi-eye reprojection subunit, and the 3D truth value is transformed and projected at each angle to generate the 2D truth value of the gesture.
[0205] At the same time, during the application, the aforementioned detection unit can also be fine-tuned by Finetune, that is, the parameters of the hand detection subunit and the large model 2D detection subunit can be fine-tuned by Finetune to improve the accuracy of the hand detection unit and the large model 2D detection subunit;
[0206] Based on all the above examples, please refer to Fig. 9 When the user triggers the optimization instruction, multiple cameras on the VR glasses are turned on. If the user selects local personalization, assuming that the user only wants to optimize a gesture with poor recognition effect, the user selects the default label, then the UI interface of the VR glasses guides the user to make a gesture, which can be a default gesture, that is, a gesture corresponding to the default label, or any gesture of the user. At this time, multiple cameras of the VR glasses are turned on. After the user makes a gesture according to the guidance of the UI interface, multiple cameras of the VR glasses capture the first gesture image and save the first gesture captured image, that is, locally save personalized data, input the pre-processed first gesture image into the truth value annotation model to output the gesture 2D truth value, gesture 3D truth value and hand prediction frame, and use the personalized Finetune model to train the detection unit of the truth value annotation model and the gesture recognition system;
[0207] In some embodiments, in combination Fig.10 , when the user triggers the optimization instruction, multiple cameras on the VR glasses can be turned on at the same time. If the user selects local personalization and hopes to build a local personalized gesture library, that is, the user needs to define the custom label, and the UI interface of the VR glasses will guide the user to make gestures. Since it is a custom label, there is no gesture. At this time, the UI interface of the head-mounted electronic device will guide the user to make any gesture. Combined with the above, it can be seen that the camera of the head-mounted electronic device is in the on state. When the user makes a gesture according to the guidance of the UI interface, the multiple cameras on the VR glasses capture the first gesture image and save the first gesture capture image, that is, save the personalized data locally, and input the pre-processed first gesture image into the truth value annotation model to output the gesture 2D truth value. Gesture 3D truth value and hand prediction box, and use the personalized Finetune model to train the detection unit of the truth value annotation model and the gesture recognition system;
[0208] In view of Fig. 9 as well as Fig.10 The saved local data (first gesture image) is used to train the gesture recognition system. Fig. 9 as well as Fig.10The optimized gesture recognition system can be called a personalized gesture recognition system;
[0209] On the basis of the above embodiments, it should be noted that the head-mounted electronic device includes but is not limited to the VR glasses in this scenario, and also includes AR glasses, etc. The solution for the user to optimize the gesture recognition system using augmented reality (AR) glasses is consistent with the above;
[0210] This application has the following advantages:
[0211] 1) Using the real-time collected user's real-time gesture images to train the gesture recognition system, so that the gesture recognition system has iterative and custom functions;
[0212] 2) Using multiple cameras of head-mounted electronic devices to collect user gesture images in real time, the extensiveness and authenticity of the data set are improved, and the generalization of the gesture recognition system is improved;
[0213] 3) The optimal 3D points are fused by weighted clustering and the true value of gestures is calculated, which improves the computing power of the model;
[0214] The following is another scenario of an embodiment of the present application: Figure 3 The user 200 is shown using VR glasses 400. When the user 200 needs to perform personalized optimization on a certain gesture, the user 200 first selects the gesture model optimization function, then selects the cloud function, the user determines the target tag, the VR glasses 400 turns on multiple cameras, and uses the UI interface to guide the user to make gestures;
[0215] It is worth noting that in this context, the target label is just any label that exists in the gesture recognition system, that is, the default label;
[0216] Consistent with the aforementioned scenario, the gesture guided by the UI interface of the VR glasses 400 to be performed by the user may be a gesture corresponding to the target tag, i.e., a default gesture, or a user-defined gesture;
[0217] When the user makes a gesture, a first gesture image is collected. Here, the first gesture image is the gesture image of the user at different angles at the current moment.
[0218] VR glasses input the target image into the truth value annotation model to output the 2D truth value of the gesture and the 3D truth value of the gesture, and use the 2D truth value of the gesture to generate the hand prediction frame.
[0219] It is worth noting that the target image here includes not only the first gesture image but also the gesture image stored in the cloud, that is, in the current situation, the VR glasses upload the first gesture image to the cloud to generate the target image;
[0220] At the same time, the method of calculating the true value of the gesture based on the target image is consistent with the above, which will not be repeated here. Then the VR glasses use the gesture 2D true value, gesture 3D true value, and hand prediction frame to Finetune the gesture recognition system, the large model 2D detection subunit, and the hand detection subunit, that is, adjust the parameters of the 2D point detection unit, 3D point detection unit, hand detection unit, gesture classification unit, large model 2D detection subunit, and hand detection subunit in the gesture recognition system.
[0221] Please refer to Fig.11 When the user triggers the optimization command, multiple cameras on the VR glasses can be turned on at the same time. If the user is universal in the cloud, the user selects the default label at this time, and the UI interface of the VR glasses guides the user to make a gesture. Here it can be a default gesture, that is, the gesture corresponding to the default label, or any gesture of the user. When the user makes a gesture according to the guidance of the UI interface, the output is about to model the first gesture image, and the VR glasses upload the first gesture image to the cloud. The cloud aggregates the data of all users, and the data here includes the first gesture image and gesture images of other users. All the data in the cloud are processed and the processed data is input into the truth value annotation model to output the 2D truth value of the gesture. The 3D truth value of the gesture and the hand prediction box are trained using the universal Finetune model to train the detection unit of the truth value annotation model and the gesture recognition system.
[0222] Please refer to Fig.12 When the user triggers the optimization command, the multiple cameras of the VR glasses are turned on. The UI interface of the head-mounted electronic device first asks whether to trigger local personalized optimization or cloud-based universal optimization. If the user chooses local personalized optimization, the user makes a custom tag and makes any gesture. Fig.10 The gesture recognition system is optimized in this way. If the user optimizes the default gesture, Fig. 9 The gesture recognition system is optimized in the manner shown. If the user chooses cloud-based universalization, Fig.11 The gesture recognition system is optimized in the manner shown;
[0223] It is understandable that the personalized Finetune model and the generalized Finetune model here can be one Finetune model, that is, when the user selects local personalization, the Finetune model is called a localized Finetune model, and when the user selects cloud generalization, Finetune is called a cloud generalized model. The difference between the localized Finetune model and the generalized Finetune model is only the difference in the usage scenarios.
[0224] Correspondingly, the personalized gesture recognition system and the universal gesture recognition system here can be the same gesture recognition system, that is, when the user selects local personalization, the trained gesture recognition system is called a personalized gesture recognition system, and when the user selects cloud universalization, the trained gesture recognition system is called a universal gesture recognition system;
[0225] Fig.13 A flowchart of a gesture recognition iteration method provided in an embodiment of the present application is applied to a head-mounted electronic device, wherein the head-mounted electronic device may be equipped with a gesture recognition system and a gesture recognition optimization system, and the gesture recognition optimization system may include: a true value labeling model, a data acquisition module, and a Finetune model;
[0226] In response to the optimization instruction, a target tag is determined, and a first gesture image corresponding to the target tag is collected, wherein the first gesture image may be a gesture image of the user at different angles at the current moment;
[0227] Inputting the first gesture image collected by the data collection module into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value, and generating a hand prediction frame using the gesture 2D truth value, wherein the target image includes the first gesture image;
[0228] The Finetune model is used to perform Finetune training on the gesture recognition system based on the target label and the gesture truth value of the target image, where the gesture truth value includes the gesture 2D truth value, the gesture 3D truth value and the hand prediction box.
[0229] In some embodiments, the first gesture image is captured by multiple cameras located on the head-mounted electronic device.
[0230] In some embodiments, the gesture recognition system includes: a 2D point detection unit, a 3D point detection unit, a hand detection unit, and a gesture classification unit, wherein:
[0231] 2D point detection unit: used to determine the gesture 2D points during gesture recognition;
[0232] 3D point detection unit: used to determine the gesture 3D points during gesture recognition;
[0233] Hand detection unit: used to generate hand prediction boxes during the recognition process;
[0234] A gesture classification unit: configured to output a gesture recognition result according to the gesture 2D point of the second gesture image, the gesture 3D point of the second gesture image, and the hand prediction frame of the second image;
[0235] The steps of using the Finetune model to perform Finetune training on the gesture recognition system according to the true value of the gesture of the target image include:
[0236] Fine-tune the 2D point detection unit, 3D point detection unit, hand detection unit and gesture classification unit using the target image gesture image.
[0237] In some embodiments, in the step of inputting the target image into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value,
[0238] The true value annotation model has a detection unit and a multi-camera algorithm fusion unit:
[0239] The step of inputting the first gesture image collected by the data collection module into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value comprises:
[0240] Input the target image into the detection unit, and output the hand 2D key points of the target image and the corresponding 2D key point confidences;
[0241] The hand 2D key points of the target image and the corresponding 2D key point confidences are input into the multi-camera algorithm fusion unit to output the gesture 2D true value and gesture 3D true value.
[0242] In some embodiments, the detection unit includes: a hand detection subunit and a large model 2D detection subunit;
[0243] Hand detection subunit: used to detect the hand prediction box of the target image;
[0244] Large model 2D detection subunit: used to output the hand 2D key points and the corresponding 2D key point confidences based on the hand prediction box of the target image.
[0245] In some embodiments, the method further includes: performing Finetune fine-tuning on the hand detection subunit and the large model 2D detection subunit using a Finetine module according to the true value of the gesture of the target image.
[0246] In some embodiments, the multi-camera fusion algorithm unit includes: a random triangulation subunit, a weighted density clustering subunit, a Kalman filter subunit, and a multi-camera reprojection subunit;
[0247] The steps of inputting the hand 2D key points of the target image and the corresponding 2D key point confidences into the multi-camera algorithm fusion unit to output the gesture 2D true value and the gesture 3D true value include:
[0248] The 2D key point confidences corresponding to n groups of hand 2D key points are input to the random triangulation subunit, which traverses and combines the n groups of hand 2D key points in pairs, outputs k groups of hand 3D key points and corresponding 3D key point confidences, and outputs a group of hand 3D key points and corresponding 3D key point confidences for every two groups of 2D key points;
[0249] Input k groups of hand 3D key points and their corresponding key point confidences into the weighted density clustering subunit for weighted fusion to output the optimal 3D key points;
[0250] The optimal 3D key point is input into the Kalman filter subunit for filtering and outputting the 3D true value of the gesture;
[0251] The 3D truth value of the gesture is input into the multi-eye reprojection subunit, and the 3D truth value is transformed and projected at each angle to generate the 2D truth value of the gesture.
[0252] In some embodiments, the random triangulation subunit calculates the depth z-axis coordinate of the hand 3D key point using the following formula:
[0253] d=u L -u R ;
[0254] f b —Camera focal length;
[0255] u L 、u R —The imaging plane coordinates of the 2D key points of the gesture on the left and right cameras;
[0256] d—parallax;
[0257] z—the depth z-axis coordinate of the 3D key point of the hand;
[0258] b—Baseline distance of the camera of the first gesture image corresponding to the two sets of hand 2D key points.
[0259] b—the baseline distance between the two groups of cameras that capture the first gesture image corresponding to the 2D key points of the hand;
[0260] In some embodiments, the step of inputting k groups of hand 3D key points and corresponding key point confidences into the weighted density clustering subunit for weighted fusion to output the optimal 3D key points includes:
[0261] Construct a data set D using k sets of hand 3D key points;
[0262] Select any data object point P from the data set D;
[0263] Set the maximum radius of the neighborhood and the minimum number of points in the neighborhood, and perform density clustering based on the maximum radius of the neighborhood, the minimum number of points in the neighborhood, the data object point P, and the data set D to obtain several clusters;
[0264] Select the group m with the most hand 3D key points from several clusters, and perform confidence-weighted averaging on all hand 3D key points in the m group to output the optimal 3D key points.
[0265] In some implementations, in the step of selecting a group m with the most hand 3D key points from a plurality of clusters, and performing confidence-weighted averaging on all hand 3D key points in the m group to output the optimal 3D key points, the optimal 3D key points are calculated using the following formula:
[0266]
[0267] —The coordinates of the jth hand 3D keypoint of the i-th group;
[0268] —Confidence of the jth hand 3D keypoint in the i-th group;
[0269] —Optimal 3D key points;
[0270] n—Number of 3D key points of the hand in group n.
[0271] In some embodiments, the 3D true value of the gesture is input into the multi-eye reprojection subunit, and in the step of performing coordinate transformation and projection on the 3D true value at each angle to generate the 2D true value of the gesture,
[0272] The following formula is used to calculate the 2D true value of the gesture:
[0273]
[0274] X W —The horizontal coordinate of the 3D true value of the gesture in the rectangular coordinate system of space;
[0275] Y W —The 3D true value of the gesture is in the ordinate of the spatial rectangular coordinate system;
[0276] Z W —The vertical coordinate of the 3D true value of the gesture in the rectangular coordinate system of space;
[0277] R, T—rotation and translation matrix;
[0278] K—camera intrinsic parameter matrix;
[0279] M—camera extrinsic matrix;
[0280] a x 、ay , μ0, v0—camera intrinsic parameters;
[0281] u—2D true value horizontal coordinate of gesture;
[0282] v—2D true value ordinate of gesture.
[0283] In some implementations, the target image also includes a cloud image, and the method further includes uploading the collected first gesture image to the cloud, and the target tag is any tag existing in the gesture recognition system.
[0284] In some embodiments, the target tag is a user-defined tag or any tag existing in the gesture recognition system;
[0285] If the target tag is a custom tag, the method further includes: defining the target tag.
[0286] Second, Fig.14 A structural diagram of a gesture recognition iteration device provided in an embodiment of the present application, which is applied to a head-mounted electronic device, wherein the head-mounted electronic device is equipped with a gesture recognition system and a gesture recognition optimization system, wherein the gesture recognition optimization system includes: a true value labeling model, a data acquisition module, a gesture recognition system, and a Finetune model;
[0287] A collection module: used to respond to the optimization instruction, determine the target tag, and collect the first gesture image corresponding to the target tag, where the first gesture image is the gesture image of the user at different angles at the current moment;
[0288] The calculation module is used to input the first gesture image collected by the data collection module into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value, and generate a hand prediction frame using the gesture 2D truth value, and the target image includes the first gesture image;
[0289] Training module: used to perform Finetune training on the gesture recognition system using the Finetune model based on the target label and the gesture truth value of the target image. The gesture truth value includes the gesture 2D truth value, gesture 3D truth value and hand prediction box.
[0290] The embodiment of the present application provides a head-mounted electronic device, which can be a terminal device or a circuit device built into the terminal device. The head-mounted electronic device can be used to perform the functions / steps in the above method embodiment.
[0291] An embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the functions / steps in the above method embodiment.
[0292] The present application actually has three kinds of relationships, for example, A and / or B can represent the situation where A exists alone, A and B exist at the same time, or B exists alone. A and B can be singular or plural. The character " / " generally indicates that the objects before and after are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0293] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0294] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0295] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0296] The above are only specific implementations of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A gesture recognition iterative method, characterized in that: Applied to a head-mounted electronic device, the head-mounted electronic device is equipped with a gesture recognition system and a gesture recognition optimization system, the gesture recognition optimization system includes: a true value labeling model, a data acquisition module and a Finetune model; In response to the optimization instruction, a target tag is determined, and a first gesture image corresponding to the target tag is collected, where the first gesture image is a gesture image of the user at different angles at the current moment; Inputting the first gesture image collected by the data collection module into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value, and using the gesture 2D truth value to generate a hand prediction frame, the target image includes the first gesture image; Finetune training is performed on a gesture recognition system using a Finetune model according to the target label and the gesture truth value of the target image, wherein the gesture truth value includes a gesture 2D truth value, a gesture 3D truth value, and a hand prediction frame.
2. The method according to claim 1, characterized in that The first gesture image is collected by multiple cameras located on the head-mounted electronic device.
3. The method according to claim 1, characterized in that: The gesture recognition system comprises: a 2D point detection unit, a 3D point detection unit, a hand detection unit and a gesture classification unit; The 2D point detection unit is used to determine the gesture 2D point during the gesture recognition process; The 3D point detection unit is used to determine the gesture 3D point during the gesture recognition process; The hand detection unit is used to generate a hand prediction frame during the recognition process; The gesture classification unit is used to output a gesture recognition result according to the gesture 2D point of the second gesture image, the gesture 3D point of the second gesture image and the hand prediction frame of the second image; The step of performing Finetune training on the gesture recognition system using the Finetune model according to the true value of the gesture of the target image comprises: Fine-tune the 2D point detection unit, the 3D point detection unit, the hand detection unit and the gesture classification unit using the target image gesture image.
4. The method according to claim 3, characterized in that In the step of inputting the target image into the truth value annotation model to output the 2D truth value of the gesture and the 3D truth value of the gesture, The true value annotation model is provided with a detection unit and a multi-camera algorithm fusion unit: The step of inputting the first gesture image collected by the data collection module into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value comprises: Input the target image to the detection unit, and output the hand 2D key points of the target image and the corresponding 2D key point confidences; The hand 2D key points of the target image and the corresponding 2D key point confidences are input into the multi-camera algorithm fusion unit to output the gesture 2D true value and the gesture 3D true value.
5. The method according to claim 3, characterized in that: The detection unit includes: a hand detection subunit and a large model 2D detection subunit; The hand detection subunit is used to detect the hand prediction frame of the target image; The large model 2D detection subunit is used to output the hand 2D key points and the corresponding 2D key point confidences according to the hand prediction frame of the target image.
6. The method according to claim 5, characterized in that The method further includes: performing Finetune training on the hand detection subunit and the large model 2D detection subunit using the Finetine module according to the true value of the gesture of the target image.
7. The method according to claim 4, characterized in that The multi-camera fusion algorithm unit includes: a random triangulation subunit, a weighted density clustering subunit, a Kalman filter subunit and a multi-camera reprojection subunit; The step of inputting the hand 2D key points of the target image and the corresponding 2D key point confidences into the multi-camera algorithm fusion unit to output the gesture 2D true value and the gesture 3D true value comprises: Inputting the 2D key point confidences corresponding to n groups of hand 2D key points into the random triangulation subunit, the random triangulation subunit traverses and combines the n groups of hand 2D key points in pairs, outputs k groups of hand 3D key points and corresponding 3D key point confidences, and outputs a group of hand 3D key points and corresponding 3D key point confidences for every two groups of 2D key points; Inputting k groups of 3D key points of the hand and the corresponding key point confidences into the weighted density clustering subunit for weighted fusion to output the optimal 3D key points; Inputting the optimal 3D key point into the Kalman filter subunit for filtering and outputting the gesture 3D true value; The gesture 3D true value is input into the multi-eye re-projection sub-unit, and the 3D true value is coordinate transformed and projected at each angle to generate the gesture 2D true value.
8. The method according to claim 5, characterized in that The random triangulation subunit calculates the depth z-axis coordinate of the hand 3D key point using the following formula: d=u L -u R ; f b —Camera focal length; u L 、u R —The imaging plane coordinates of the 2D key points of the gesture on the left and right cameras; d—parallax; z—the depth z-axis coordinate of the 3D key point of the hand; b—Baseline distance of the camera of the first gesture image corresponding to the two sets of hand 2D key points.
9. The method according to claim 6, characterized in that The step of inputting k groups of 3D key points of the hand and the corresponding key point confidences into the weighted density clustering subunit for weighted fusion and outputting the optimal 3D key points comprises: Construct a data set D using k sets of hand 3D key points; Select any data object point P from the data set D; Setting a maximum neighborhood radius and a minimum number of neighborhood points, and performing density clustering according to the maximum neighborhood radius, the minimum number of neighborhood points, the data object point P, and the data set D to obtain a plurality of clusters; A group m with the most hand 3D key points is selected from a plurality of clusters, and confidence-weighted averaging is performed on all hand 3D key points in the m group to output the optimal 3D key point.
10. The method according to claim 9, characterized in that In the step of selecting a group m with the most hand 3D key points from a plurality of clusters, and performing confidence-weighted averaging on all hand 3D key points in the m groups to output the optimal 3D key points, the optimal 3D key points are calculated using the following formula: —The coordinates of the jth hand 3D keypoint of the i-th group; —Confidence of the jth hand 3D keypoint in the i-th group; —Optimal 3D key points; n—The number of 3D key points of the hand in the m group.
11. The method according to claim 1, characterized in that: In the step of inputting the gesture 3D true value into the multi-eye reprojection subunit, performing coordinate transformation and projection on the 3D true value at each angle to generate the gesture 2D true value, The following formula is used to calculate the 2D true value of the gesture: X W —The horizontal coordinate of the 3D true value of the gesture in the rectangular coordinate system of space; Y W —The 3D true value of the gesture is in the ordinate of the spatial rectangular coordinate system; Z W —The vertical coordinate of the 3D true value of the gesture in the rectangular coordinate system of space; R, T—rotation and translation matrix; K—camera intrinsic parameter matrix; M—camera extrinsic matrix; a x 、a y , μ0, v0—camera intrinsic parameters; u—2D true value horizontal coordinate of gesture; v—2D true value vertical coordinate of gesture.
12. The method according to any one of claims 1 to 11, characterized in that: The target image also includes a cloud image. The method also includes uploading the collected first gesture image to the cloud. The target tag is any tag existing in the gesture recognition system.
13. The method according to any one of claims 1 to 11, characterized in that The target tag is a user-defined tag or any tag existing in the gesture recognition system; If the target tag is a custom tag, the method further includes: defining the target tag.
14. A gesture recognition iteration device, characterized in that: Applied to a head-mounted electronic device, the head-mounted electronic device is equipped with a gesture recognition system and a gesture recognition optimization system, the gesture recognition optimization system includes: a true value labeling model, a data acquisition module, a gesture recognition system and a Finetune model; A collection module: used to determine a target tag in response to an optimization instruction, and collect a first gesture image corresponding to the target tag, where the first gesture image is a gesture image of the user at different angles at the current moment; A calculation module: used for inputting the first gesture image collected by the data collection module into the truth value annotation model to output the gesture 2D truth value and the gesture 3D truth value, and generating a hand prediction frame using the gesture 2D truth value, wherein the target image includes the first gesture image; Training module: used to perform Finetune training on the gesture recognition system using the Finetune model according to the target label and the gesture truth value of the target image, wherein the gesture truth value includes the gesture 2D truth value, the gesture 3D truth value and the hand prediction frame.
15. A head-mounted electronic device, characterized in that: include: at least one processor; and at least one memory in communication with the processor, wherein: The memory stores program instructions executable by the processor, and the processor can execute the method according to any one of claims 1 to 13 by calling the program instructions.
16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the method according to any one of claims 1 to 13.
Citation Information
Patent Citations
Three-dimensional voxel gesture attitude estimation method based on multi-task information complementation
CN113095262A
SMPL-X model parameter training method and device, equipment and storage medium
CN116051698A
Pre-Training With Alignments For Recurrent Neural Network Transducer Based End-To-End Speech Recognition
US20210312905A1
Hand gesture detection method and apparatus, and computer storage medium
US20220277595A1
