Gesture tracking method and apparatus, device, readable storage medium, and program product

Abstract

The present application provides a gesture tracking method and apparatus, a device, a readable storage medium, and a program product. The method comprises: acquiring a current hand image corresponding to a current user; performing a search to determine whether hand skeleton information corresponding to the current user is stored; and, if it is found that the hand skeleton information corresponding to the current user is stored, determining a hand pose on the basis of the current hand image and the corresponding hand skeleton information. The method of the present application can solve the problem in the existing technology of significant computational resource waste.

Description

Hand gesture tracking method, device, equipment, readable storage medium and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 29, 2024, with application number 202410224082.1 and application name “Hand gesture tracking method, device, equipment, readable storage medium and program product”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of virtual reality, and in particular to a gesture tracking method, apparatus, device, readable storage medium, and program product. Background Art

[0003] Virtual reality is a combination of multiple technologies, including real-time three-dimensional computer graphics technology, wide-angle (wide field of view) stereoscopic display technology, tracking technology for the observer's head, eyes and hands, as well as tactile / force feedback, stereo sound, network transmission, voice input and output technology, etc.

[0004] Taking hand tracking technology as an example, the current execution of hand size estimation is initiated when the hand appears in multiple images. When gesture tracking is initiated, the hand size is calculated in real time, resulting in a large waste of computing resources. Summary of the Invention

[0005] The present application provides a gesture tracking method, apparatus, device, readable storage medium, and program product to solve the problem of large waste of computing resources caused by the prior art.

[0006] In a first aspect, an embodiment of the present application provides a gesture tracking method, the method comprising:

[0007] Collect the current frame hand image corresponding to the current user;

[0008] Check whether hand skeleton information corresponding to the current user is stored;

[0009] If hand skeleton information corresponding to the current user is found, the hand posture is determined based on the current frame hand image and the corresponding hand skeleton information.

[0010] In one possible design, searching whether hand skeleton information corresponding to the current user is stored includes:

[0011] Determine the ID of the current user;

[0012] According to the ID of the current user, searching the storage unit for hand skeleton information that matches the ID of the current user;

[0013] The storage unit is configured in the head-mounted device or the electronic device, or the storage unit is communicatively connected to the head-mounted device or the electronic device to support access by the head-mounted device or the electronic device.

[0014] In one possible design, determining the ID of the current user includes:

[0015] The ID of the current user is determined through iris recognition or face recognition.

[0016] In one possible design, determining the ID of the current user includes:

[0017] For the hand picture of the current frame, if the hand of the current user is detected in all the historical frame pictures collected within a preset time period before the current moment, then read the ID of the user corresponding to the historical frame picture and use the ID of the user corresponding to the historical frame picture as the ID of the current user; or

[0018] For the hand image of the current frame, if the current moment has not reached the moment for performing the next identity recognition operation, the ID of the user corresponding to the last ID recognition operation is read, and the ID of the user corresponding to the last identity recognition operation is used as the ID of the current user.

[0019] In one possible design, the method further includes:

[0020] If it is determined that the hand skeleton information corresponding to the current user is not stored, prompting the current user to place the hand within the common viewing area of ​​the multi-camera field of view or the field of view angle of the depth camera;

[0021] If a hand is detected within the common viewing area of ​​the multi-camera field of view or within the field of view angle of the depth camera, the multi-camera or the depth camera is used to identify the hand of the current user to obtain hand skeleton information; the hand skeleton information includes the size of the hand;

[0022] The hand skeleton information of the current user is bound to the identified ID of the current user and stored.

[0023] In one possible design, determining the hand posture according to the current frame hand image and the corresponding hand skeleton information includes:

[0024] Based on the hand image in the current frame, the detection frame of the hand area is obtained through the target detection model;

[0025] According to the detection frame of the hand area, key point information of the hand is obtained through a key point detection model;

[0026] The hand posture is determined based on the key point information of the hand and the corresponding hand skeleton information.

[0027] In a possible design, before determining the hand posture based on the current frame hand image and the corresponding hand skeleton information, the method further includes:

[0028] Check whether the ID of the current user has changed;

[0029] If it is determined that the ID has been changed, then according to the changed ID, search whether the hand skeleton information corresponding to the changed ID is stored;

[0030] If it is determined that the ID has not changed, the operation of determining the hand posture is performed.

[0031] In one possible design, the method further includes:

[0032] Determine a reprojection error function based on the camera's projection matrix and the key point information of the hand, where the key point information of the hand includes the position of the 3D point, the first position of the 2D point under each camera obtained by model inference, and the second position of the 2D point obtained by projecting the 3D point;

[0033] Determine a relative distance error function based on the position of the 3D point, the hand skeleton information obtained by table lookup, the first relative distance from any skeleton point to the wrist point obtained by model inference, and the second relative distance from any skeleton point to the wrist point obtained by 3D point calculation;

[0034] Determine the finger motion smoothing error function based on the hand posture at the previous moment and the hand posture at the current moment;

[0035] According to the reprojection error function, the relative distance error function and the finger motion smoothing error, the objective function of the hand posture model is determined by using preset weights;

[0036] According to the first position, the hand skeleton information, the first relative distance, and the hand posture at the previous moment, the minimum value of the objective function is solved by minimizing the distance between the first position and the second position, minimizing the difference between the first relative distance and the second relative distance, and minimizing the difference between the hand posture at the previous moment and the hand posture at the current moment to determine the hand posture; wherein the hand skeleton information is obtained by table lookup based on a mapping relationship with the user's ID.

[0037] In a second aspect, an embodiment of the present application provides a gesture tracking device, which is applied to a head-mounted device or an electronic device, wherein the head-mounted device or the electronic device is configured to support storage of hand skeleton information corresponding to a new user; the device comprises:

[0038] The acquisition module is used to acquire the hand image of the current frame corresponding to the current user;

[0039] A search module is used to search whether hand skeleton information corresponding to the current user is stored;

[0040] The hand posture determination module is used to determine the hand posture according to the current frame hand image and the corresponding hand skeleton information when the hand skeleton information corresponding to the current user is found.

[0041] In a third aspect, an embodiment of the present application provides a head-mounted device, the head-mounted device including a camera, a storage unit, and a processing unit;

[0042] Wherein, the camera is used to collect pictures corresponding to the user;

[0043] The storage unit is used to store user IDs, user hand skeleton information, and a mapping relationship between user IDs and corresponding user hand skeleton information;

[0044] The processing unit is used to execute the method described in any one of the first aspects.

[0045] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory;

[0046] The memory stores computer-executable instructions;

[0047] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method as described in any one of the first aspects.

[0048] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in any one of the first aspects is implemented.

[0049] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method as described in any one of the first aspects.

[0050] The gesture tracking method, apparatus, device, readable storage medium and program product provided in this embodiment first collect the current frame hand picture corresponding to the current user; then search whether the hand skeleton information corresponding to the current user is stored; if the hand skeleton information corresponding to the current user is found to be stored, the hand posture is determined based on the current frame hand picture and the corresponding hand skeleton information. Therefore, based on each frame of the collected picture, the present application only needs to identify the hand skeleton information once for each user, and execute the subsequent gesture tracking process to perform gesture estimation. It is not necessary to identify and calibrate the hand skeleton information again, thereby saving computational complexity and improving the efficiency of the algorithm. At the same time, since the hand skeleton information is obtained in advance, it is not forced to appear in the binocular field of view overlap area for hand estimation, and monocular gesture tracking can be run for a long time, thereby making the algorithm enablement range larger and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] FIG1 is a schematic diagram of a scene of a gesture tracking method provided by an embodiment of the present application;

[0053] FIG2 is a schematic diagram of a flow chart of a gesture tracking method provided in an embodiment of the present application;

[0054] FIG3 is a flow chart of a gesture tracking method according to another embodiment of the present application;

[0055] FIG4 is a flowchart of a gesture tracking method provided in another embodiment of the present application;

[0056] FIG5 is a flowchart of a gesture tracking method provided in yet another embodiment of the present application;

[0057] FIG6 is a schematic diagram of the structure of a gesture tracking device provided in an embodiment of the present application;

[0058] FIG7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way are interchangeable where appropriate, so that the embodiments of the application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0061] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0062] Currently, using hand tracking as an example, the applicant has discovered that in monocular images, there is inherent ambiguity between the size of a user's hand and its distance from the camera (for example, if the hand is far from the camera, the hand size in the captured image will be small; conversely, if the hand is close to the camera, the hand size in the captured image will be large). Therefore, hand size estimation is currently initiated when the hand appears in multi-view images. However, existing methods calculate the hand size in real time when gesture tracking is initiated, resulting in a significant waste of computing resources.

[0063] Therefore, in response to the above problems, the technical concept of this application is to store the hand skeleton information of the identified new user. When the user uses the head-mounted device to identify it next time, the corresponding hand skeleton information can be directly obtained from the storage without the need for re-identification. Combined with each frame of the collected image, hand estimation is performed to achieve gesture tracking, which saves computing power resources and improves gesture recognition efficiency.

[0064] In practical applications, the gesture tracking method can be applied to fields such as virtual reality (VR), which is not specifically limited here. For example, referring to FIG1 , FIG1 is a scene diagram of the gesture tracking method provided by an embodiment of the present application, wherein the scene includes a gesture tracking device, which can be a head-mounted device, including a camera (for example, a monocular camera, a multi-camera camera, etc., the monocular camera 101 is taken as an example below and will not be repeated), a storage unit 102, and a processing unit 103. Among them, the monocular camera 101 is used to collect pictures corresponding to the user; the storage unit 102 is used to store the user ID, the user's hand skeleton information, and the mapping relationship between the user ID and the hand skeleton information of the corresponding user; the processing unit 103 is used to perform gesture tracking operations based on the pictures collected by the monocular camera and the user-related information stored in the storage unit. That is, according to the pictures collected by the monocular camera, whether the hand skeleton information corresponding to the current user is stored in the storage unit can be searched. If the hand skeleton information corresponding to the current user is found to be stored, the hand posture is determined according to the hand picture of the current frame and the corresponding hand skeleton information. Among them, the pictures captured by the monocular camera can also be stored in the storage unit for subsequent processing, such as: inspection or detection of tracking effects, etc. The application scenarios of the subsequent processing are not specifically limited here.

[0065] As shown in Figure 2, Figure 2 is a flow chart of the gesture tracking method provided by an embodiment of the present application. The user turns on the head-mounted device or turns on the gesture tracking function (here refers to the start of the gesture tracking algorithm, i.e., hand tracking starts), and starts the gesture tracking operation:

[0066] Step 1: After the algorithm starts, for each frame of the picture, first read the user ID;

[0067] Step 2: Then, based on the ID, determine whether the hand skeleton information corresponding to the user already exists in the device storage (here, the storage unit 102). If so, execute step 4; otherwise, execute step 3.

[0068] Step 3: If the user's hand skeleton information is not available, the user's hand skeleton needs to be initially identified to obtain the hand skeleton information, which can include the hand size and bone size. The hand skeleton information is then bound to the user's ID (or user ID) (e.g., iris ID), and then step 4 is executed;

[0069] Step 4: Check whether the user ID has changed. If so, proceed to step 5. If not, proceed to step 6.

[0070] Step 5: Change the hand skeleton information according to the user ID and execute step 6;

[0071] Step 6: Perform hand detection, key point detection and hand posture estimation.

[0072] The above steps can store the hand skeleton information of the newly identified user. The next time the user uses the headset to identify the user, the corresponding hand skeleton information can be directly retrieved from the storage without the need for re-identification. This can solve the problem of wasting a lot of computing resources by calculating the hand size in real time when gesture tracking is activated. It can also handle the situation where the hand does not appear in the overlap area of ​​the binocular field of view. If the hand does not appear in the overlap area, the hand size cannot be calculated.

[0073] Therefore, in this application, based on each frame of the collected image, it is only necessary to identify the hand skeleton information once for each user, and execute the subsequent gesture tracking process to perform gesture estimation. There is no need to identify and calibrate the hand skeleton information again, thereby saving computational complexity and improving the efficiency of the algorithm. At the same time, since the hand skeleton information is available in advance, it is not forced to appear in the binocular field of view overlap area for hand estimation, and monocular gesture tracking can be run for a long time, thereby expanding the algorithm's enabled range and improving the user experience.

[0074] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0075] FIG3 is a flow chart of a gesture tracking method according to another embodiment of the present application. The gesture tracking method may include:

[0076] S301: Collect a hand image of the current frame corresponding to the current user.

[0077] In this embodiment, the execution entity may be a gesture tracking device, which is installed in a gesture tracking device, which may be an electronic device or a head-mounted device. The head-mounted device or electronic device is configured to support storage of hand skeletal information corresponding to a new user. Optionally, the gesture tracking device may also be configured in a server, which may be communicatively connected to a data acquisition device and a storage unit, respectively. The data acquisition device may include a monocular camera or a multi-camera camera.

[0078] For example, taking a head-mounted device as an example, the head-mounted device can pre-identify the hand skeleton information corresponding to a new user, and store the user's ID, the user's corresponding hand skeleton information, and the mapping relationship between the user's ID and the corresponding hand skeleton information. Taking the monocular camera in the head-mounted device as an example, when the user's hand is placed within the field of view of the monocular camera, the monocular camera can capture the current frame hand image corresponding to the current user. The current frame hand image here is a picture of the current user's hand (which can be a complete hand or a partial hand, which is not specifically limited here).

[0079] In one possible design, the method further includes:

[0080] If it is determined that the hand skeleton information corresponding to the current user is not stored, prompting the current user to place the hand within the common viewing area of ​​the multi-camera field of view or the field of view angle of the depth camera;

[0081] If a hand is detected within the common viewing area of ​​the multi-camera field of view or within the field of view angle of the depth camera, the multi-camera or the depth camera is used to identify the hand of the current user to obtain hand skeleton information; the hand skeleton information includes the size of the hand;

[0082] The hand skeleton information of the current user is bound to the identified ID of the current user and stored.

[0083] In this embodiment, if it is the first time for the user to use it, the hand skeleton information of the user can be identified by triggering the hand skeleton initialization process, as shown in Figure 4, which shows the hand skeleton information registration process for a new user. The application scenario of this registration process includes a camera, a computing unit, and a storage space (which can be referred to as a storage unit here); wherein, the camera collects the user's hand image (here refers to a photo, i.e., image), and sends the collected photo to the computing unit, which identifies the user's hand skeleton information (i.e., hand skeleton or Hand skeleton), and binds the user's ID (here refers to the iris ID, i.e., lris ID) and the corresponding hand skeleton information, and stores it in the storage unit. Subsequent users can quickly retrieve the user's hand skeleton information after the ID (e.g., iris ID, face ID (i.e., facial ID or face ID)) is identified.

[0084] Specifically, in the hand skeleton initialization process, the user is required to place the hand (here refers to the hand) in a specified position (the overlap of the binocular field of view or within the FOV of the depth camera), and then the user's hand is modeled through the binocular camera or the depth camera to obtain the user's hand skeleton information, which includes the size of the hand, the size of the hand bones, etc. The user's ID is then generated or determined through iris recognition, and the user's ID and the corresponding hand skeleton information are bound, and the user's ID and the user's hand skeleton information and the result of the binding of the two are stored in a storage unit. For subsequent gesture tracking, if the user's hand skeleton information has been stored, the user's hand skeleton information can be directly obtained through the user's ID for hand estimation. There is no need to re-identify and calculate the user's hand skeleton information, which saves computing resources and time, thereby improving the response speed of the device.

[0085] Among them, the hand skeleton information of the current user is bound to the identified ID of the current user (combined with the registration process shown in Figure 4), and a mapping relationship between the user's ID and the hand skeleton information of the user can be formed. For example, a mapping table is generated based on the user's ID and the corresponding hand skeleton information. The mapping table in the storage unit (see Table 1, using Key to represent the user's ID (here the user's ID can refer to the user's iris ID, such as lris_ID_1, lris_ID_2, ... in Table 1) and Value to represent the user's hand skeleton information (or the identifier of the hand skeleton information, and the corresponding hand skeleton information can be obtained from the storage through the identifier of the hand skeleton information) (such as Hand_skeleton_1, Hand_skeleton_2, ... in Table 1)) can be used to find out whether a certain user's hand skeleton information exists. The binding method and binding result of the two are not specifically limited here.

[0086] Table 1

[0087] S302: Check whether hand skeleton information corresponding to the current user is stored.

[0088] S303: If hand skeleton information corresponding to the current user is found, determine the hand posture according to the current frame hand image and the corresponding hand skeleton information.

[0089] In this embodiment, a storage unit is queried to determine whether hand skeleton information matching the current user is stored therein. For example, Table 1 is searched to determine whether hand skeleton information corresponding to the current user exists in Table 1. If hand skeleton information corresponding to the current user is found, the hand pose of the current user can be determined by combining the hand image of the current frame with the hand skeleton information, and gesture tracking can be implemented based on the hand pose determined for each frame.

[0090] Specifically, FIG5 shows a tracking process. First, camera 1 captures the hand image of the current user (including the current frame hand image corresponding to the current user, which contains a complete hand or a partial hand). The camera sends the captured image to the computing unit. Camera 2 captures the iris image of the current user's eye and sends the iris image to the computing unit. The computing unit searches the storage space for hand skeleton information (i.e., hand skeleton) corresponding to the current user's ID (e.g., lris ID). If a hand skeleton corresponding to the lris ID is found, hand detection can be performed by identifying the current frame hand image. Based on the detection frame (e.g., rectangular frame) of the detected hand area, a key point detection operation is performed to obtain the key point information of the hand. Based on the key point information of the hand and the corresponding hand skeleton information, the hand posture is determined, thereby achieving gesture tracking.

[0091] The gesture tracking method provided in the embodiment of the present application first collects the current frame hand picture corresponding to the current user; then searches whether the hand skeleton information corresponding to the current user is stored; if the hand skeleton information corresponding to the current user is found to be stored, the hand posture is determined based on the current frame hand picture and the corresponding hand skeleton information. Therefore, based on each frame of the collected picture, the present application only needs to identify the hand skeleton information once for each user, and execute the subsequent gesture tracking process to perform gesture estimation. It is not necessary to identify and calibrate the hand skeleton information again, thereby saving computational complexity and improving the efficiency of the algorithm. At the same time, since the hand skeleton information is available in advance, it is not forced to appear in the binocular field of view overlap area for hand estimation, and monocular gesture tracking can be run for a long time, thereby expanding the algorithm enablement range and improving the user experience.

[0092] In one possible design, searching whether hand skeleton information corresponding to the current user is stored includes:

[0093] Determine the ID of the current user;

[0094] According to the ID of the current user, searching the storage unit for hand skeleton information that matches the ID of the current user;

[0095] The storage unit is configured in the head-mounted device or the electronic device, or the storage unit is communicatively connected to the head-mounted device or the electronic device to support access by the head-mounted device or the electronic device.

[0096] In this embodiment, taking the head-mounted device as an example, the process of searching whether the hand skeleton information of the current user is stored is as follows: first, the ID of the user whose gesture is to be tracked (here, the current user, the user wearing the head-mounted device) is determined, and then the hand skeleton information corresponding to the ID is searched from the storage unit. If so, the hand skeleton information bound to the ID is obtained. Since the size of a person's hand will not change significantly after adulthood, if the user's hand skeleton information is stored, the hand skeleton information can be used directly, which solves the problem of a large waste of computing resources when calculating the hand size in real time when gesture tracking is started in the prior art. At the same time, since the hand skeleton information is obtained in advance, it is not forced to appear in the binocular field of view overlap area for hand estimation, and monocular gesture tracking can be run for a long time, which can make the algorithm (here, the gesture tracking algorithm or method) enable range larger, thereby improving the user experience.

[0097] In one possible design, before searching for hand skeleton information, it is necessary to determine the ID of the current user. Determining the ID of the current user can be achieved in at least two ways:

[0098] Method 1: Determine the ID of the current user through iris recognition or face recognition.

[0099] In this embodiment, for each captured frame, the user's ID is first read to ensure that the corresponding hand skeleton information obtained is accurate, thereby improving the accuracy of gesture tracking. Identifying or reading the user's ID can use iris recognition technology or face recognition technology to identify the user's iris ID or face ID.

[0100] Taking iris recognition as an example, the user ID can be obtained through iris recognition. Iris recognition technology uses iris texture information to identify individuals, and is a form of biometrics within artificial intelligence. The typical steps include eye detection, iris detection, normalization, iris feature extraction, and encoding.

[0101] It should be noted that the iris ID used in this embodiment is only one method of ID authentication, and it can also be replaced by face ID, etc. There is no specific limitation on the method of identifying or determining the user's ID and the specific form of the ID.

[0102] Method 2: For the hand picture of the current frame, if the hand of the current user is detected in all the historical frames collected within a preset time period before the current moment, then read the ID of the user corresponding to the historical frame and use the ID of the user corresponding to the historical frame as the ID of the current user; or

[0103] For the hand image of the current frame, if the current moment has not reached the moment for performing the next identity recognition operation, the ID of the user corresponding to the last ID recognition operation is read, and the ID of the user corresponding to the last identity recognition operation is used as the ID of the current user.

[0104] In this embodiment, to conserve resources and considering that users rarely switch between user accounts over time, the need for real-time ID recognition can be determined based on certain conditions. For example, if the user's hand remains in the field of view for a predetermined period of time, it indicates that the user has not switched, and the ID corresponding to the previous image can be used. Alternatively, if the recognition interval has not been reached, the ID from the last recognition can be used.

[0105] In one possible design, determining the hand posture according to the current frame hand image and the corresponding hand skeleton information includes:

[0106] Based on the hand image in the current frame, the detection frame of the hand area is obtained through the target detection model;

[0107] According to the detection frame of the hand area, key point information of the hand is obtained through a key point detection model;

[0108] The hand posture is determined based on the key point information of the hand and the corresponding hand skeleton information.

[0109] In this embodiment, gesture estimation is achieved by using a target detection model + a key point detection model in combination with a nonlinear optimization method: the image obtained by the monocular / binocular camera is input into the target detection model to obtain the bounding box (here refers to the detection box, such as a rectangular box) of the hand area; wherein, the camera is not limited to using an RGB camera, a monochrom camera, or a depth camera. The image of the hand area is sent to the key point detection network (here refers to the key point detection model) to obtain the key point information of the hand. The key point information of the hand here includes at least the key points of the hand (for example, 21 joints), and the connecting lines between two adjacent key points, etc., and the 3D position of the 21 joints of the hand in the world coordinate system is restored using a nonlinear optimization method.

[0110] In a possible design, before determining the hand posture based on the current frame hand image and the corresponding hand skeleton information, the method further includes:

[0111] Check whether the ID of the current user has changed;

[0112] If it is determined that the ID has been changed, then according to the changed ID, search whether the hand skeleton information corresponding to the changed ID is stored;

[0113] If it is determined that the ID has not changed, the operation of determining the hand posture is performed.

[0114] In this embodiment, as shown in FIG2 , in order to further ensure that the hand skeleton information used is accurate, before determining the hand posture based on the current frame hand image and the corresponding hand skeleton information, the user's ID can be once again identified in real time and compared with the determined user's ID to check whether it has changed. Taking iris recognition as an example, if ID recognition through iris will run at a low frequency in the background, then if it is found that the user ID has changed, the hand skeleton information needs to be switched accordingly, that is, switched to the hand skeleton information corresponding to the currently identified user ID.

[0115] In one possible design, the method further includes:

[0116] Determine a reprojection error function based on the camera's projection matrix and the key point information of the hand, where the key point information of the hand includes the position of the 3D point, the first position of the 2D point under each camera obtained by model inference, and the second position of the 2D point obtained by projecting the 3D point;

[0117] Determine a relative distance error function based on the position of the 3D point, the hand skeleton information obtained by table lookup, a first relative distance from any skeleton point to the wrist point obtained by model inference, and a second relative distance from any skeleton point to the wrist point obtained by 3D point conversion;

[0118] Determine the finger motion smoothing error function based on the hand posture at the previous moment and the hand posture at the current moment;

[0119] According to the reprojection error function, the relative distance error function and the finger motion smoothing error, the objective function of the hand posture model is determined by using preset weights;

[0120] According to the first position, the hand skeleton information, the first relative distance, and the hand posture at the previous moment, the minimum value of the objective function is solved by minimizing the distance between the first position and the second position, minimizing the difference between the first relative distance and the second relative distance, and minimizing the difference between the hand posture at the previous moment and the hand posture at the current moment to determine the hand posture; wherein the hand skeleton information is obtained by table lookup based on a mapping relationship with the user's ID.

[0121] Among them, the first position, the hand skeleton information, the first relative distance, and the hand posture at the previous moment are known quantities, and the position of the 3D point, the second position, the second relative distance, and the hand posture at the current moment are all quantities to be solved.

[0122] In this embodiment, a nonlinear optimization method is used to solve the parameters of the 3D hand model (including the positions of the 3D points of the hand) to further determine the hand pose. The objective function of this nonlinear optimization method includes reprojection error, relative distance error, and motion smoothing error. The camera here can be a monocular camera, a multi-camera camera, or a depth camera, without specific limitation.

[0123] The reprojection error is the distance between the 2D keypoints projected from the 3D hand model (here, 2D points projected from 3D points) and the 2D keypoints derived from the deep learning model (here, 2D points from each camera inferred by the model). If multiple cameras are present, the 3D hand model is projected onto each camera to obtain 2D keypoints for each camera.

[0124] The relative distance error is the difference between the relative distance calculated using the 3D hand model (the second relative distance from any skeletal point to the wrist point calculated using 3D point calculations) and the relative distance obtained by the deep learning model (here, the first relative distance from any skeletal point to the wrist point inferred by the model). The required hand size information (here, hand skeletal information) is obtained through a table lookup based on a mapping relationship with the user's ID. If multiple cameras are used, a relative distance error function is calculated for each of these cameras.

[0125] The motion smoothing error is calculated based on the hand pose at the previous moment and the hand pose at the current moment. Specifically, the nonlinear optimization method is used to solve the problem: the hand model can be represented by θ (i.e., hand pose), which contains 26 degrees of freedom. We solve this problem through the optimization method. The objective function (here θ) consists of three parts: the reprojection error E 2D , relative distance error E dist , and finger motion smoothing error E temporal . The formula for θ is as follows:

[0126] Among them, w1 and w2 are weights.

[0127] E 2D The formula is as follows:

[0128] Among them, π j is the projection matrix of the jth camera, p i is the i-th 3D joint (here refers to the position of the 3D point), which is a function of θ. is the i-th 2D joint (here refers to the position of the 2D point) under the j-th camera.

[0129] E dist The formula is as follows:

[0130] Among them, dist j It's about p i (θ) is a function of p i (θ) is the distance to camera j; id is the user's ID number (this ID refers to iris ID or face ID, etc.), and φ(id) is the hand scale (i.e., hand size) as a function of id. This can be achieved using a lookup table, eliminating the need for nonlinear optimization in the original method. This saves computing power and allows for faster acquisition of the user's hand skeleton information, thereby improving gesture tracking efficiency. is the distance value of the i-th 2D joint under the j-th camera, obtained by deep learning model inference (here refers to the first relative distance from any bone point to the wrist point obtained by model inference).

[0131] E temporal The formula is as follows:

[0132] Among them, θ t-1 is the hand pose at time t-1.

[0133] In this application, after the user wears the head-mounted display device (here refers to the head-mounted device) for the first time, the hand skeleton information can be obtained. When the hand tracking (here refers to hand tracking or gesture tracking) function is used subsequently, if the user does not change, there is no need to calculate and calibrate the hand size, hand skeleton size and other information in real time, thereby saving calculations and improving the efficiency of the algorithm. At the same time, since the user's hand skeleton and size information is already available in advance, it is not necessary to force the hand to appear in the binocular field of view overlap area, which solves the problem that the existing technical solutions cannot handle the situation where the hand does not appear in the binocular field of view overlap area. In addition, since the size of the hand is known, the gesture tracking method provided by this application can run monocular hand tracking for a long time, which can expand the algorithm enablement range and improve the user experience.

[0134] In order to implement the gesture tracking method, this embodiment provides a gesture tracking device, see Figure 6, which is a structural schematic diagram of the gesture tracking device provided in the embodiment of the present application; the gesture tracking device includes: an acquisition module 601, a search module 602 and a hand posture determination module 603.

[0135] The acquisition module 601 is used to acquire the hand image of the current frame corresponding to the current user;

[0136] Search module 602, used to search whether hand skeleton information corresponding to the current user is stored;

[0137] The hand posture determination module 603 is used to determine the hand posture according to the current frame hand image and the corresponding hand skeleton information when the hand skeleton information corresponding to the current user is found.

[0138] In this embodiment, the acquisition module 601, the search module 602 and the hand posture determination module 603 are used to collect the current frame hand picture corresponding to the current user; then a search is performed to determine whether the hand skeleton information corresponding to the current user is stored; if the hand skeleton information corresponding to the current user is found to be stored, the hand posture is determined based on the current frame hand picture and the corresponding hand skeleton information. Therefore, based on each frame of the collected picture, the present application only needs to identify the hand skeleton information once for each user, and execute the subsequent gesture tracking process to perform gesture estimation. It is not necessary to identify and calibrate the hand skeleton information again, thereby saving computational complexity and improving the efficiency of the algorithm. At the same time, since the hand skeleton information is obtained in advance, it is not forced to appear in the binocular field of view overlap area for hand estimation, and monocular gesture tracking can be run for a long time, thereby making the algorithm enablement range larger and improving the user experience.

[0139] The gesture tracking device provided in this embodiment can be used to implement the technical solution of the above-mentioned gesture tracking method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0140] In one possible design, the search module includes: an ID determination unit and a search unit;

[0141] An ID determination unit, configured to determine the ID of the current user;

[0142] A search unit, configured to search the storage unit for hand skeleton information matching the ID of the current user according to the ID of the current user;

[0143] The storage unit is configured in the head-mounted device or the electronic device, or the storage unit is communicatively connected to the head-mounted device or the electronic device to support access by the head-mounted device or the electronic device.

[0144] In one possible design, the ID determination unit is specifically configured to:

[0145] The ID of the current user is determined through iris recognition or face recognition.

[0146] In one possible design, the ID determination unit is specifically configured to:

[0147] For the hand picture of the current frame, if the hand of the current user is detected in all the historical frame pictures collected within a preset time period before the current moment, then read the ID of the user corresponding to the historical frame picture and use the ID of the user corresponding to the historical frame picture as the ID of the current user; or

[0148] For the hand image of the current frame, if the current moment has not reached the moment for performing the next identity recognition operation, the ID of the user corresponding to the last ID recognition operation is read, and the ID of the user corresponding to the last identity recognition operation is used as the ID of the current user.

[0149] In one possible design, the apparatus further includes: a processing module; the processing module is configured to:

[0150] If it is determined that the hand skeleton information corresponding to the current user is not stored, prompting the current user to place the hand within the common viewing area of ​​the multi-camera field of view or the field of view angle of the depth camera;

[0151] If a hand is detected within the common viewing area of ​​the multi-camera field of view or within the field of view angle of the depth camera, the multi-camera or the depth camera is used to identify the hand of the current user to obtain hand skeleton information; the hand skeleton information includes the size of the hand;

[0152] The hand skeleton information of the current user is bound to the identified ID of the current user and stored.

[0153] In one possible design, the hand posture determination module is specifically configured to:

[0154] Based on the hand image in the current frame, the detection frame of the hand area is obtained through the target detection model;

[0155] According to the detection frame of the hand area, key point information of the hand is obtained through a key point detection model;

[0156] The hand posture is determined based on the key point information of the hand and the corresponding hand skeleton information.

[0157] In one possible design, the processing module is further configured to:

[0158] Before determining the hand posture according to the current frame hand image and the corresponding hand skeleton information, detecting whether the current user ID has been changed;

[0159] When it is determined that the ID has been changed, searching whether hand skeleton information corresponding to the changed ID is stored according to the changed ID;

[0160] When it is determined that the ID has not changed, an operation of determining the hand pose is performed.

[0161] In one possible design, the hand posture determination module is further configured to:

[0162] Determine a reprojection error function based on the camera's projection matrix and the key point information of the hand, where the key point information of the hand includes the position of the 3D point, the first position of the 2D point under each camera obtained by model inference, and the second position of the 2D point obtained by projecting the 3D point;

[0163] Determine a relative distance error function based on the position of the 3D point, the hand skeleton information obtained by table lookup, the first relative distance from any skeleton point to the wrist point obtained by model inference, and the second relative distance from any skeleton point to the wrist point obtained by 3D point calculation;

[0164] Determine the finger motion smoothing error function based on the hand posture at the previous moment and the hand posture at the current moment;

[0165] According to the reprojection error function, the relative distance error function and the finger motion smoothing error, the objective function of the hand posture model is determined by using preset weights;

[0166] According to the first position, the hand skeleton information, the first relative distance, and the hand posture at the previous moment, the minimum value of the objective function is solved by minimizing the distance between the first position and the second position, minimizing the difference between the first relative distance and the second relative distance, and minimizing the difference between the hand posture at the previous moment and the hand posture at the current moment to determine the hand posture; wherein the hand skeleton information is obtained by looking up a table based on a mapping relationship with the user's ID. In order to implement the above-mentioned gesture tracking method, this embodiment provides a head-mounted device, as shown in Figure 1, the head-mounted device includes a camera (for example, a monocular camera 101 or a multi-camera camera), a storage unit 102, and a processing unit 103;

[0167] Wherein, the camera is used to collect pictures corresponding to the user;

[0168] The storage unit is used to store user IDs, user hand skeleton information, and a mapping relationship between user IDs and corresponding user hand skeleton information;

[0169] The processing unit is used to execute the method described in any one of the first aspects.

[0170] The head-mounted device provided in this embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effects are similar, and will not be repeated here in this embodiment.

[0171] To implement the method of the above embodiment, this embodiment provides an electronic device. Figure 7 is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. As shown in Figure 7, the electronic device of this embodiment includes: a processor 701 and a memory 702; wherein the memory 702 is used to store computer-executable instructions; the processor 701 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0172] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0173] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0174] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms. In addition, the functional modules in the various embodiments of the present application can be integrated into a processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0175] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the methods of each embodiment of the present application. It should be understood that the above-mentioned processor can be a central processing unit (English: Central Processing Unit, referred to as: CPU), or other general-purpose processors, digital signal processors (English: Digital Signal Processor, referred to as: DSP), application-specific integrated circuits (English: Application Specific Integrated Circuit, referred to as: ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0176] The memory may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus. The above-mentioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0177] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an application-specific integrated circuit (ASIC). Of course, the processor and storage medium can also exist as discrete components in an electronic device or a main control device.

[0178] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A gesture tracking method, characterized in that: Applied to a head-mounted device or electronic device, the head-mounted device or electronic device is used to support storage of hand skeleton information corresponding to a new user; the method includes: Collect the current frame hand image corresponding to the current user; Check whether hand skeleton information corresponding to the current user is stored; If hand skeleton information corresponding to the current user is found, the hand posture is determined based on the current frame hand image and the corresponding hand skeleton information.

2. The method according to claim 1, characterized in that The searching for whether hand skeleton information corresponding to the current user is stored includes: Determine the ID of the current user; According to the ID of the current user, searching the storage unit for hand skeleton information that matches the ID of the current user; The storage unit is configured in the head-mounted device or the electronic device, or the storage unit is communicatively connected to the head-mounted device or the electronic device to support access by the head-mounted device or the electronic device.

3. The method according to claim 2, characterized in that Determining the ID of the current user includes: The ID of the current user is determined through iris recognition or face recognition.

4. The method according to claim 2, characterized in that Determining the ID of the current user includes: For the hand picture of the current frame, if the hand of the current user is detected in all the historical frame pictures collected within a preset time period before the current moment, then read the ID of the user corresponding to the historical frame picture and use the ID of the user corresponding to the historical frame picture as the ID of the current user; or For the hand image of the current frame, if the current moment has not reached the moment for performing the next identity recognition operation, the ID of the user corresponding to the last ID recognition operation is read, and the ID of the user corresponding to the last identity recognition operation is used as the ID of the current user.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: If it is determined that the hand skeleton information corresponding to the current user is not stored, prompting the current user to place the hand within the common viewing area of ​​the multi-camera field of view or the field of view angle of the depth camera; If a hand is detected within the common viewing area of ​​the multi-camera field of view or within the field of view angle of the depth camera, the multi-camera or the depth camera is used to identify the hand of the current user to obtain hand skeleton information; the hand skeleton information includes the size of the hand; The hand skeleton information of the current user is bound to the identified ID of the current user and stored.

6. The method according to any one of claims 1 to 4, characterized in that The determining of the hand posture according to the current frame hand image and the corresponding hand skeleton information includes: Based on the hand image in the current frame, the detection frame of the hand area is obtained through the target detection model; According to the detection frame of the hand area, key point information of the hand is obtained through a key point detection model; The hand posture is determined based on the key point information of the hand and the corresponding hand skeleton information.

7. The method according to any one of claims 1 to 4, characterized in that Before determining the hand posture according to the current frame hand image and the corresponding hand skeleton information, the method further includes: Check whether the ID of the current user has changed; If it is determined that the ID has been changed, then according to the changed ID, search whether the hand skeleton information corresponding to the changed ID is stored; If it is determined that the ID has not changed, the operation of determining the hand posture is performed.

8. The method according to claim 1, characterized in that The method further comprises: Determine a reprojection error function based on the camera's projection matrix and the key point information of the hand, where the key point information of the hand includes the position of the 3D point, the first position of the 2D point under each camera obtained by model inference, and the second position of the 2D point obtained by projecting the 3D point; Determine a relative distance error function based on the position of the 3D point, the hand skeleton information obtained by table lookup, the first relative distance from any skeleton point to the wrist point obtained by model inference, and the second relative distance from any skeleton point to the wrist point obtained by 3D point calculation; Determine the finger motion smoothing error function based on the hand posture at the previous moment and the hand posture at the current moment; According to the reprojection error function, the relative distance error function and the finger motion smoothing error, the objective function of the hand posture model is determined by using preset weights; According to the first position, the hand skeleton information, the first relative distance, and the hand posture at the previous moment, the minimum value of the objective function is solved by minimizing the distance between the first position and the second position, minimizing the difference between the first relative distance and the second relative distance, and minimizing the difference between the hand posture at the previous moment and the hand posture at the current moment to determine the hand posture; wherein the hand skeleton information is obtained by table lookup based on a mapping relationship with the user's ID.

9. A gesture tracking device, characterized in that: Applicable to a head-mounted device or electronic device, the head-mounted device or electronic device is used to support storage of hand skeleton information corresponding to a new user; the device includes: The acquisition module is used to acquire the hand image of the current frame corresponding to the current user; A search module is used to search whether hand skeleton information corresponding to the current user is stored; The hand posture determination module is used to determine the hand posture according to the current frame hand image and the corresponding hand skeleton information when the hand skeleton information corresponding to the current user is found.

10. A head-mounted device, characterized in that: The head-mounted device includes a camera, a storage unit, and a processing unit; Wherein, the camera is used to collect pictures corresponding to the user; The storage unit is used to store the user ID, the user's hand skeleton information, and the mapping relationship between the user ID and the corresponding user's hand skeleton information; The processing unit is used to execute the method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images