Hand tracking method and device

Abstract

The invention provides a hand tracking method and device, and the method comprises the steps: obtaining a to-be-recognized image obtained by photographing the hand of a user through a specific camera, and obtaining the hand motion data, collected by an intelligent ring, of the user, and enabling the frame rate of the to-be-recognized image to be smaller than a specified frame rate; determining a key point position of a hand based on the to-be-recognized image, and intercepting a hand image from the to-be-recognized image based on the key point position of the hand; and determining a final hand posture estimation result by using the to-be-recognized image, the hand image, the key point position of the hand and the hand motion data. According to the scheme, the demand of hand tracking on camera data is reduced through the hand motion data provided by the intelligent ring, so that the hardware power consumption and the computing power demand are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D scene interaction, and in particular to a hand tracking method and device. Background Art

[0002] With the development of science and technology, the application of 3D (three-dimensional) scene interaction is becoming more and more widespread. During the process of users interacting with 3D scenes, it is necessary to track the user's hands to achieve 3D scene interaction.

[0003] The existing method of tracking hands is to collect high-density image data through a high-frame rate (such as 60FPS, FPS is the frame rate) camera, and then process the high-density image data based on pure computer vision methods to achieve hand tracking. However, the above-mentioned method of tracking hands requires high-performance cameras and high computing power to be realized, and high-performance cameras and high computing power require high power consumption, which leads to the problems of high hardware power consumption and high computing power requirements. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a hand tracking method and device to solve the problems of high hardware power consumption and high computing power requirements in existing hand tracking methods.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a hand tracking method, the method comprising:

[0007] Acquire an image to be identified obtained by photographing a user's hand with a specific camera, and acquire hand motion data of the user collected by a smart ring, wherein the smart ring is worn on any finger of the user, and a frame rate of the image to be identified is less than a specified frame rate;

[0008] Determine the key point positions of the hand based on the image to be recognized, and extract the hand image from the image to be recognized based on the key point positions of the hand;

[0009] The final hand posture estimation result is determined by using the image to be recognized, the hand image, the key point positions of the hand and the hand motion data.

[0010] Preferably, after extracting the hand image from the image to be recognized, the method further includes:

[0011] Based on the hand motion data and the hand image, a final gesture recognition result of the user is obtained by identification.

[0012] Preferably, using the image to be identified, the hand image, the key point positions of the hand and the hand motion data, determining the final hand posture estimation result includes:

[0013] Determining an initial hand posture estimation result based on the image to be recognized, the hand image and the key point positions of the hand;

[0014] The final hand posture estimation result is determined using the initial hand posture estimation result, the hand motion data, the hand image and the key point positions of the hand, and the frame rate of the final hand posture estimation result is higher than the frame rate of the initial hand posture estimation result.

[0015] Preferably, the final gesture recognition result includes: a gesture event detection result and a gesture state detection result;

[0016] Based on the hand motion data and the hand image, identifying and obtaining a final gesture recognition result of the user includes:

[0017] Performing gesture detection using the hand motion data to identify and obtain a gesture event detection result;

[0018] The hand image and the hand motion data are input into a preset gesture state detection model for processing to obtain a gesture state detection result, wherein the gesture state detection model is trained based on the first sample data.

[0019] Preferably, determining an initial hand posture estimation result based on the image to be recognized, the hand image and the key point positions of the hand includes:

[0020] The image to be identified, the hand image and the key point positions of the hand are input into a preset first posture estimation model for processing to obtain an initial hand posture estimation result of the hand, wherein the first posture estimation model is trained based on the second sample data.

[0021] Preferably, using the initial hand posture estimation result, the hand motion data, the hand image and the key point positions of the hand to determine the final hand posture estimation result includes:

[0022] The initial hand posture estimation result, the hand motion data, the hand image and the key point positions of the hand are input into a preset second posture estimation model for processing to obtain a final hand posture estimation result of the hand, and the second posture estimation model is trained based on third sample data.

[0023] A second aspect of an embodiment of the present invention discloses a hand tracking device, the device comprising:

[0024] an acquisition unit, configured to acquire an image to be recognized obtained by photographing a user's hand by a specific camera, and acquire hand motion data of the user collected by a smart ring, wherein the smart ring is worn on any finger of the user, and a frame rate of the image to be recognized is less than a specified frame rate;

[0025] A processing unit, configured to determine the key point positions of the hand based on the image to be identified, and to extract a hand image from the image to be identified based on the key point positions of the hand;

[0026] A determination unit is used to determine a final hand posture estimation result by using the image to be identified, the hand image, the key point positions of the hand and the hand motion data.

[0027] Preferably, the device further comprises:

[0028] The recognition unit is used to obtain a final gesture recognition result of the user based on the hand motion data and the hand image.

[0029] Preferably, the determining unit includes:

[0030] A first determination module, configured to determine an initial hand posture estimation result based on the image to be recognized, the hand image and the key point positions of the hand;

[0031] The second determination module is used to determine the final hand posture estimation result using the initial hand posture estimation result, the hand motion data, the hand image and the key point positions of the hand, and the frame rate of the final hand posture estimation result is higher than the frame rate of the initial hand posture estimation result.

[0032] Preferably, the final gesture recognition result includes: a gesture event detection result and a gesture state detection result; the recognition unit includes:

[0033] A recognition module, used to perform gesture detection using the hand motion data to identify and obtain a gesture event detection result;

[0034] A processing module is used to input the hand image and the hand movement data into a preset gesture state detection model for processing to obtain a gesture state detection result, wherein the gesture state detection model is trained based on the first sample data.

[0035] Based on the above-mentioned embodiment of the present invention, a hand tracking method and device are provided. The method is as follows: obtaining an image to be identified by photographing the user's hand with a specific camera, and obtaining the user's hand motion data collected by a smart ring, wherein the frame rate of the image to be identified is less than a specified frame rate; determining the key point position of the hand based on the image to be identified, and extracting the hand image from the image to be identified based on the key point position of the hand; using the image to be identified, the hand image, the key point position of the hand, and the hand motion data, determining the final hand posture estimation result. In this solution, the hand motion data collected by the smart ring and the image with a frame rate less than the specified frame rate are used to determine the final hand posture estimation result to achieve hand tracking; the hand motion data provided by the smart ring is used to reduce the demand for camera data for hand tracking, thereby reducing hardware power consumption and reducing computing power requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 A flow chart of a hand tracking method provided by an embodiment of the present invention;

[0038] Figure 2 An example diagram of a data flow of a hand tracking method provided by an embodiment of the present invention;

[0039] Figure 3 A structural block diagram of a hand tracking device provided by an embodiment of the present invention;

[0040] Figure 4 Another structural block diagram of a hand tracking device provided by an embodiment of the present invention;

[0041] Figure 5 Another structural block diagram of a hand tracking device provided by an embodiment of the present invention;

[0042] Figure 6 Another structural block diagram of a hand tracking device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0045] When users interact with 3D scenes, they need to track their hands to achieve 3D scene interaction. The existing method of tracking hands is to collect high-density image data through a high-frame rate camera (such as a binocular camera or a monocular camera with a lidar), and then process the high-density image data based on a pure computer vision method to achieve hand tracking. However, the method of tracking hands requires high-performance cameras and high computing power to achieve, and the hardware cost of high-performance cameras is relatively high. In addition, high-performance cameras and high computing power require high power consumption, resulting in high hardware cost, high hardware power consumption, and high computing power requirements.

[0046] To solve the above problems, this solution proposes a hand tracking method and device, which utilizes the hand motion data collected by a smart ring and images with a frame rate less than a specified frame rate to determine the final hand posture estimation result to achieve hand tracking; the hand motion data provided by the smart ring is used to reduce the demand for camera data for hand tracking, thereby reducing hardware power consumption and reducing computing power requirements.

[0047] It should be noted that the FPS mentioned in the following content of this solution is frame rate (also known as frames per second), IMU is inertial sensor, and RGB is red, green and blue.

[0048] Overall, the core content of this solution is: based on the hand motion data (IMU data, including acceleration and angular velocity data in 6 axes, 200FPS) collected by the smart ring worn by the user (such as worn on the index finger), combined with a specific camera of any viewing angle (such as a monocular RGB camera, a binocular camera or a depth camera), the three-dimensional spatial position of the hand can be accurately tracked and gestures detected, thereby supporting 3D scene interaction, editing and other task requirements. This solution has a low requirement for the frame rate of the image collected by the specific camera. The frame rate of the image collected by the specific camera is less than 5FPS (such as the frame rate of the image is 1FPS). Only using images with a low frame rate (such as about 1FPS) can estimate the position and posture of the hand in space, and combine the hand motion data provided by the smart ring to estimate the posture and position of the hand in real time, which can reduce the power consumption of the camera and reduce the computing power required to process the image.

[0049] In addition, this solution also realizes accurate gesture event detection (such as detecting gestures such as pinching, making a fist, snapping fingers, etc.) and gesture state detection (such as detecting whether the two fingers are pinched, whether the hand is making a fist, etc.) based on hand motion data and images. Combined with the final hand posture estimation result, 3D scene interaction (such as selection, dragging, moving objects, etc.) can be realized.

[0050] After applying this solution, one of the application scenarios of this solution is: in the scenario of naked-eye 3D screen interaction, by using the difference between the 2D flat display screen and the optical image projected to both eyes, the user can directly use both hands to interact with the objects in the 3D scene through the smart ring worn on the index finger, thereby realizing naked-eye 3D screen interaction. This solution uses a smart ring to alleviate the problems of high hardware power consumption and high computing power requirements in the existing hand tracking methods, and can reduce the power consumption of the camera in the mobile scene to improve the battery life. For example: the existing AR / VR glasses rely on high frame rate cameras to capture hand movements. In this solution, the smart ring, a lightweight wearable device, is used to replace the controller, thereby reducing the power consumption of the camera.

[0051] The above content is a general description of the present solution from an overall level. The present solution will be described in detail below through various embodiments.

[0052] See also Figure 1 , shows a flow chart of a hand tracking method provided by an embodiment of the present invention, the hand tracking method comprising the following steps:

[0053] Step S101: obtaining an image to be identified by photographing a user's hand with a specific camera, and obtaining the user's hand movement data collected by a smart ring.

[0054] It should be noted that the smart ring is worn on any finger of the user, for example, the smart ring (one type of smart ring) is worn on the index finger of the user; the specific camera can be a monocular RGB camera, a binocular camera or a depth camera and other devices that can capture images. The frame rate of the image to be recognized obtained by capturing the user's hand with the specific camera is less than the specified frame rate, that is, the image to be recognized is low frame rate data, for example, the frame rate of the image to be recognized is less than 5FPS.

[0055] In the specific implementation of step S101, a specific camera of any viewing angle is used to shoot the user's hand in real time to obtain an image to be recognized (such as an RGB image), and a smart ring is used to collect the user's hand movement data in real time.

[0056] It should be noted that the smart ring is equipped with at least an IMU sensor. The hand motion data collected by the smart ring is the IMU data (or IMU signal), and the hand motion data at least includes angular velocity and acceleration.

[0057] Step S102: determining the key point positions of the hand based on the image to be recognized, and extracting the hand image from the image to be recognized based on the key point positions of the hand.

[0058] In the specific implementation of step S102, the position of the key points of the hand in the image to be identified is detected to obtain the position of the key points of the hand. The position of the key points of the hand is: the position of the key points of the hand in the image to be identified.

[0059] Specifically, a hand key point detection algorithm (such as Mediapipe) is used to identify the positions of the user's hand key points in the image to be identified, thereby obtaining the positions of the hand key points.

[0060] It should be noted that the frame rate of the key point positions of the hand detected in the above manner is lower than the specified frame rate, that is, the key point positions of the hand are low frame rate data.

[0061] A hand image is cut out from the image to be identified through the key point positions of the hand. Specifically, after the key point positions of the hand in the image to be identified are identified, an image of a fixed size is cut out from the key point positions of the image to be identified as the hand image.

[0062] In addition, the horizontal position and vertical position of the user's hand can also be determined through the key point positions of the hand.

[0063] Step S103: Determine the final hand posture estimation result using the image to be identified, the hand image, the key point positions of the hand and the hand motion data.

[0064] In the process of implementing step S103, the initial hand posture estimation result is determined based on the image to be identified, the hand image and the key point positions of the hand. The initial hand posture estimation result includes the hand position and rotation; the frame rate of the initial hand posture estimation result is less than the specified frame rate, that is, the initial hand posture estimation result is a posture estimation result with a low frame rate.

[0065] It should be noted that when the specific camera is a depth camera, data other than the hand image is needed in the process of estimating the initial hand estimation result, so it is necessary to use the image to be identified, the hand image and the key point positions of the hand to determine the initial hand posture estimation result.

[0066] In some embodiments, the specific implementation method for determining the initial hand posture estimation result is: inputting the image to be identified, the hand image and the key point positions of the hand into a preset first posture estimation model for processing to obtain the initial hand posture estimation result of the hand, and the first posture estimation model is trained based on the second sample data.

[0067] It should be noted that the second sample data at least includes a sample image, a sample hand image (cropped from the sample image), the key point position of the sample hand (detected from the sample image), and a real label; the neural network model is trained by the second sample data to obtain a first posture estimation model, and the first posture estimation model is used to output an initial hand posture estimation result.

[0068] It should be further noted that, in practical applications, the method for determining the initial hand posture estimation result can be selected according to the type of specific camera used; when the specific camera is a monocular RGB camera, the initial hand posture estimation result can be determined by the first posture estimation model mentioned above;

[0069] When the specific camera is a binocular camera, the disparity data collected by the binocular camera may be used to determine the initial hand posture estimation result;

[0070] When the specific camera is a depth camera, the initial hand posture estimation result can be determined using the original depth map acquired by the depth camera and the key point positions of the hand.

[0071] After determining the initial hand posture estimation result, the final hand posture estimation result is determined using the initial hand posture estimation result, hand motion data, hand image and key point positions of the hand. The frame rate of the final hand posture estimation result is higher than the frame rate of the initial hand posture estimation result, that is, the final hand posture estimation result is high frame rate data.

[0072] In some embodiments, the specific implementation method for determining the final hand posture estimation result is: inputting the initial hand posture estimation result, hand motion data, hand image and key point positions of the hand into a preset second posture estimation model for processing to obtain the final hand posture estimation result of the hand, and the second posture estimation model is trained based on the third sample data.

[0073] It should be noted that the third sample data at least includes the sample initial hand posture estimation result, sample hand motion data, sample hand image, key point position of the sample hand, and true label; the pre-built neural network model is trained by the third sample data to obtain the second posture estimation model, and the second posture estimation model is used to output the final hand posture estimation result.

[0074] It should be further explained that, in the process of determining the final hand posture estimation result, the key point position of the hand with a low frame rate and the hand image with a low frame rate are input as features into the second posture estimation model, and the initial hand posture estimation result with a low frame rate and the hand motion data with a high frame rate (such as 200FPS) are input into the second posture estimation model; the second posture estimation model fuses the aforementioned input initial hand posture estimation results, hand motion data, hand image and key point position of the hand to obtain the final hand posture estimation result with a high frame rate, and hand tracking can be achieved through the final hand posture estimation result.

[0075] The above content is about how to estimate the final hand posture estimation result of the hand.

[0076] In some embodiments, the present solution can also implement gesture event detection and gesture status checking; in a specific implementation, based on hand motion data and hand images, the user's final gesture recognition result is identified, and the final gesture recognition result includes: gesture event detection result and gesture status detection result.

[0077] The following explains how to identify and obtain gesture event detection results and gesture state detection results respectively.

[0078] The implementation method of identifying gesture event detection results is as follows: gesture detection is performed using hand motion data to identify gesture event detection results. The gesture event detection results include user gesture events (such as pinching, flicking, snapping, and clenching fists, etc.) identified based on hand motion data (IMU data).

[0079] Specifically, a gesture classification model is pre-constructed, which can use the IMU data collected by the smart ring to identify gesture events. The F1 score of the gesture classification model is not lower than the performance index threshold (such as the F1 score is not lower than 98%), where the F1 score is an indicator for evaluating the performance of the gesture classification model.

[0080] The hand motion data is processed using a gesture classification model and a short time window (within 500 milliseconds) algorithm to identify and obtain gesture event detection results. In the specific implementation, the hand motion data within the short time window is input into the gesture classification model to obtain the gesture event detection results output by the gesture classification model.

[0081] That is to say, the input of the gesture classification model is hand motion data of a certain length, and the gesture event detection result output by the gesture classification model contains the gesture events corresponding to the hand motion data of this segment, wherein "hand motion data of a certain length" means that the length of the hand motion data is as close to the shortest time window as possible (within 500 milliseconds), which can minimize the response delay of gesture event recognition.

[0082] It should be noted that gesture events refer to events that occur within a short time window (such as 1 second), such as pinching, flicking, snapping, and clenching fists. The characteristics of the hand motion data (IMU data) corresponding to these gesture events are relatively obvious, so the hand motion data can be directly used to detect gesture event detection results.

[0083] The above is a description of the implementation method of identifying and obtaining gesture event detection results.

[0084] The implementation method of identifying and obtaining the gesture state detection result is as follows: the hand image and the hand motion data are input into a preset gesture state detection model for processing to obtain the gesture state detection result, and the gesture state detection model is trained based on the first sample data.

[0085] It should be noted that the gesture state refers to the state within a longer time range, such as whether the two fingers are pinched together or whether the hand is clenched into a fist. Since the characteristics of the hand motion data (IMU data) are not obvious when the hand is still and maintains a unified gesture state, it is necessary to fuse the hand image as a stronger feature when detecting the gesture state for classification and detection of the gesture state.

[0086] Therefore, the hand image and hand motion data are input into a preset gesture state detection model for processing to obtain a gesture state detection result, which includes gesture state categories (such as whether two fingers are pinched, whether the hand is clenched into a fist, etc.).

[0087] The first sample data at least includes sample IMU data, sample hand images, and real labels; a neural network model is trained by the first sample data to obtain a gesture state detection model, and the gesture state detection model is used to output a gesture state detection result.

[0088] It is worth noting that the length of the hand motion data input to the gesture state detection model is within a short time window.

[0089] For example: a hand image and hand motion data within a short time window are input into a gesture state detection model to obtain a gesture state detection result output by the gesture state detection model.

[0090] The above is a description of the implementation method of identifying and obtaining gesture state detection results.

[0091] It can be seen from the above content that the user's final gesture recognition result includes: the gesture event detection result and gesture state detection result detected by the above method; the gesture events and gesture states detected provide interaction primitives for spatial operations.

[0092] In an embodiment of the present invention, the hand motion data collected by the smart ring and images with a frame rate less than a specified frame rate are used to determine the final hand posture estimation result to achieve hand tracking. The hand motion data provided by the smart ring is used to reduce the demand for camera data for hand tracking, thereby reducing hardware power consumption and reducing computing power requirements.

[0093] To better explain the above embodiments of the present invention Figure 1 The following content is Figure 2 A data flow example diagram of a hand tracking method is shown, which illustrates the data flow of hand tracking.

[0094] like Figure 2 As can be seen from the shown content, the hand tracking method includes three parts: "hand key point detection", "initial hand posture estimation" and "final hand posture estimation".

[0095] For the "hand key point detection" part, the input data of the "hand key point detection" part is: the low-frame rate image to be identified obtained by taking the user's hand by a specific camera; the output result of the "hand key point detection" part is: the low-frame rate hand key point position and the low-frame rate hand image.

[0096] For the "Initial Hand Pose Estimation" part, the input data of the "Initial Hand Pose Estimation" part is: the image to be recognized at a low frame rate, the key point position of the hand at a low frame rate, and the hand image at a low frame rate; the output result of the "Initial Hand Pose Estimation" part is: the initial hand pose estimation result at a low frame rate.

[0097] For the "final hand pose estimation" part, the input data of the "final hand pose estimation" part is: the key point position of the hand at a low frame rate and the hand image at a low frame rate as features, as well as the high frame rate hand motion data (such as 200FPS hand motion data) and the initial hand pose estimation result at a low frame rate, where the high frame rate hand motion data is collected by the smart ring; the output result of the "final hand pose estimation" part is: the final hand pose estimation result at a high frame rate.

[0098] The above is the description of the data flow of hand tracking. Figure 2 The implementation principles of each part can be found in the above embodiments of the present invention. Figure 1 The contents in will not be repeated here.

[0099] Corresponding to the hand tracking method provided by the above embodiment of the present invention, see Figure 3 , an embodiment of the present invention also provides a structural block diagram of a hand tracking device, the hand tracking device includes: an acquisition unit 301, a processing unit 302 and a determination unit 303;

[0100] The acquisition unit 301 is used to acquire the image to be identified obtained by photographing the user's hand with a specific camera, and acquire the user's hand motion data collected by the smart ring. The smart ring is worn on any finger of the user, and the frame rate of the image to be identified is less than the specified frame rate.

[0101] The processing unit 302 is used to determine the key point positions of the hand based on the image to be identified, and to extract the hand image from the image to be identified based on the key point positions of the hand.

[0102] The determination unit 303 is used to determine the final hand posture estimation result by using the image to be identified, the hand image, the key point positions of the hand and the hand movement data.

[0103] In an embodiment of the present invention, the hand motion data collected by the smart ring and images with a frame rate less than a specified frame rate are used to determine the final hand posture estimation result to achieve hand tracking. The hand motion data provided by the smart ring is used to reduce the demand for camera data for hand tracking, thereby reducing hardware power consumption and reducing computing power requirements.

[0104] Preferably, combined Figure 3 , see Figure 4 , shows another structural block diagram of a hand tracking device provided by an embodiment of the present invention, the hand tracking device also includes:

[0105] The recognition unit 304 is used to obtain a final gesture recognition result of the user based on the hand motion data and the hand image.

[0106] Preferably, combined Figure 3 , see Figure 5 , shows another structural block diagram of a hand tracking device provided by an embodiment of the present invention, wherein the determination unit 303 includes:

[0107] The first determination module 3031 is used to determine the initial hand posture estimation result based on the image to be identified, the hand image and the key point positions of the hand.

[0108] In the specific implementation, the first determination module 3031 is specifically used to: input the image to be identified, the hand image and the key point position of the hand into a preset first posture estimation model for processing to obtain an initial hand posture estimation result of the hand, and the first posture estimation model is trained based on the second sample data.

[0109] The second determination module 3032 is used to determine the final hand posture estimation result using the initial hand posture estimation result, hand motion data, hand image and key point positions of the hand, and the frame rate of the final hand posture estimation result is higher than the frame rate of the initial hand posture estimation result.

[0110] In the specific implementation, the second determination module 3032 is specifically used to: input the initial hand posture estimation result, hand motion data, hand image and the key point position of the hand into a preset second posture estimation model for processing to obtain the final hand posture estimation result of the hand, and the second posture estimation model is trained based on the third sample data.

[0111] Preferably, combined Figure 4 , see Figure 6 , shows another structural block diagram of a hand tracking device provided by an embodiment of the present invention, wherein the recognition unit 304 includes:

[0112] The recognition module 3041 is used to perform gesture detection using the hand motion data to identify and obtain gesture event detection results.

[0113] The processing module 3042 is used to input the hand image and the hand motion data into a preset gesture state detection model for processing to obtain a gesture state detection result. The gesture state detection model is trained based on the first sample data.

[0114] In summary, the embodiments of the present invention provide a hand tracking method and device, which utilizes hand motion data collected by a smart ring and images with a frame rate less than a specified frame rate to determine the final hand posture estimation result to achieve hand tracking; the hand motion data provided by the smart ring is used to reduce the demand for camera data for hand tracking, thereby reducing hardware power consumption and reducing computing power requirements.

[0115] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0116] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0117] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A hand tracking method, characterized in that: The method comprises: Acquire an image to be identified obtained by photographing a user's hand with a specific camera, and acquire hand motion data of the user collected by a smart ring, wherein the smart ring is worn on any finger of the user, and a frame rate of the image to be identified is less than a specified frame rate; Determine the key point positions of the hand based on the image to be recognized, and extract the hand image from the image to be recognized based on the key point positions of the hand; The final hand posture estimation result is determined by using the image to be recognized, the hand image, the key point positions of the hand and the hand motion data.

2. The method according to claim 1, characterized in that After extracting the hand image from the image to be recognized, the method further includes: Based on the hand motion data and the hand image, a final gesture recognition result of the user is obtained by identification.

3. The method according to claim 1, characterized in that Determining a final hand posture estimation result by using the image to be recognized, the hand image, the key point positions of the hand, and the hand motion data, including: Determining an initial hand posture estimation result based on the image to be recognized, the hand image and the key point positions of the hand; The final hand posture estimation result is determined using the initial hand posture estimation result, the hand motion data, the hand image and the key point positions of the hand, and the frame rate of the final hand posture estimation result is higher than the frame rate of the initial hand posture estimation result.

4. The method according to claim 2, characterized in that: The final gesture recognition results include: gesture event detection results and gesture state detection results; Based on the hand motion data and the hand image, identifying and obtaining a final gesture recognition result of the user includes: Performing gesture detection using the hand motion data to identify and obtain a gesture event detection result; The hand image and the hand motion data are input into a preset gesture state detection model for processing to obtain a gesture state detection result, wherein the gesture state detection model is trained based on the first sample data.

5. The method according to claim 3, characterized in that: Determining an initial hand posture estimation result based on the image to be recognized, the hand image, and the key point positions of the hand, including: The image to be identified, the hand image and the key point positions of the hand are input into a preset first posture estimation model for processing to obtain an initial hand posture estimation result of the hand, wherein the first posture estimation model is trained based on the second sample data.

6. The method according to claim 3, characterized in that: Determining a final hand posture estimation result by using the initial hand posture estimation result, the hand motion data, the hand image, and the key point positions of the hand, comprising: The initial hand posture estimation result, the hand motion data, the hand image and the key point positions of the hand are input into a preset second posture estimation model for processing to obtain a final hand posture estimation result of the hand, and the second posture estimation model is trained based on third sample data.

7. A hand tracking device, characterized in that: The device comprises: an acquisition unit, configured to acquire an image to be recognized obtained by photographing a user's hand by a specific camera, and acquire hand motion data of the user collected by a smart ring, wherein the smart ring is worn on any finger of the user, and a frame rate of the image to be recognized is less than a specified frame rate; A processing unit, configured to determine the key point positions of the hand based on the image to be identified, and to extract a hand image from the image to be identified based on the key point positions of the hand; A determination unit is used to determine a final hand posture estimation result by using the image to be identified, the hand image, the key point positions of the hand and the hand motion data.

8. The device according to claim 7, characterized in that The device also includes: The recognition unit is used to obtain a final gesture recognition result of the user based on the hand motion data and the hand image.

9. The device according to claim 7, characterized in that The determining unit comprises: A first determination module, configured to determine an initial hand posture estimation result based on the image to be identified, the hand image and the key point positions of the hand; The second determination module is used to determine the final hand posture estimation result using the initial hand posture estimation result, the hand motion data, the hand image and the key point positions of the hand, and the frame rate of the final hand posture estimation result is higher than the frame rate of the initial hand posture estimation result.

10. The device according to claim 8, characterized in that The final gesture recognition result includes: gesture event detection result and gesture state detection result; the recognition unit includes: A recognition module, used to perform gesture detection using the hand motion data to identify and obtain a gesture event detection result; A processing module is used to input the hand image and the hand movement data into a preset gesture state detection model for processing to obtain a gesture state detection result, wherein the gesture state detection model is trained based on the first sample data.

Images