Hand 3D joint detection method and system based on 2D heat map

Abstract

The application discloses a hand 3D joint detection method and system based on a 2D heat map, and belongs to the technical field of computer vision. The method comprises the following steps: a convolution network model is used to extract features of a hand picture to obtain a feature map; regression processing is performed on feature points in the feature map to obtain a depth value z of a hand joint in a wrist coordinate system and a depth value z0 of a wrist joint in a camera coordinate system; regression processing is performed on the feature points in the feature map to obtain a 2D heat map of the hand joint, and 2D coordinates of the hand joint are determined; and the 3D position of the hand joint is determined according to the depth value z' of the hand joint in the wrist coordinate system, the depth value z0 of the wrist joint in the camera coordinate system and the 2D coordinates. The application obtains the depth value and the 2D heat map of the hand joint respectively, and then obtains the position of the hand 3D joint, so that the data amount of processing is reduced, and the operation speed and accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and system for detecting 3D joints of a human hand based on 2D heatmaps. Background Technology

[0002] In 3D hand joint detection, existing methods often involve analyzing hand images to construct a 3D heatmap. This heatmap then determines the positions of each joint in 3D space, thus completing the 3D hand joint detection. However, obtaining this heatmap often requires a neural network model to analyze the hand's 3D pixels, or voxels. For images, analyzing pixels results in a massive amount of data; analyzing voxels with a neural network model further increases the processing volume, ultimately slowing down the hand recognition process. Furthermore, this method of directly constructing a 3D heatmap using voxels suffers from low accuracy due to the complexity of 3D analysis. Summary of the Invention

[0003] In view of the problems of large data volume, complex detection process and low accuracy in the existing 3D hand joint detection process, this application proposes a 3D hand detection method and system based on 2D heat map.

[0004] One technical solution of this application provides a method for detecting 3D joints of a hand based on a 2D heatmap, comprising: extracting features from a hand image using a pre-trained convolutional network model to obtain a feature map corresponding to the hand image; and using the convolutional network model to perform regression processing on the depth values ​​of the feature points in the feature map in the wrist coordinate system and the depth values ​​in the camera coordinate system to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system, wherein the wrist coordinate system is a system with the wrist position as the origin, and its x, y A 3D coordinate system with the z-axis parallel to the corresponding axes of the camera coordinate system is used. A convolutional network model is used to regress the 2D heatmap of the feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each hand joint in the pixel coordinate system. Based on the 2D heatmap of the hand joint in the pixel coordinate system, the 2D coordinates of each hand joint are determined. Based on the depth value z of the hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system, as well as the 2D coordinates, the 3D position of each hand joint in the camera coordinate system is determined.

[0005] Optionally, a pre-trained convolutional network model can be used to extract features from the hand image to obtain the corresponding feature map. This includes: reducing the number of model channels in each semantic layer of the Backbone layer of the convolutional network model, and using the model channels to extract features from the hand image to obtain the feature map.

[0006] Optionally, a convolutional network model is used to regress the depth values ​​of feature points in the feature map in the wrist coordinate system and the depth values ​​in the camera coordinate system to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system. This includes: in the wrist coordinate system, analyzing the distance of each hand joint relative to the xoy plane in the wrist coordinate system using a convolutional network model, and using this distance as the depth value z of the corresponding hand joint; and analyzing the distance of the wrist joint relative to the xoy plane in the camera coordinate system using a convolutional network model, and using this distance as the depth value z0 of the corresponding wrist joint.

[0007] Optionally, a convolutional network model is used to perform regression processing on the 2D heatmap of feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each hand joint in the pixel coordinate system. This includes: pre-constructing a pixel coordinate system on the hand image; identifying each hand joint in the feature map in the pixel coordinate system through a convolutional network model, and representing the identified hand joints separately through the heatmap to obtain the 2D heatmap.

[0008] Optionally, regression processing is performed on the 2D heatmap of feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each hand joint in the pixel coordinate system. This includes: performing classification regression processing on the feature map to obtain the initial heatmap corresponding to each hand joint; and upsampling the initial heatmap multiple times to increase the size of the heatmap and obtain the 2D heatmap.

[0009] One technical solution of this application provides a 3D hand joint detection system based on 2D heatmaps, comprising: a module for extracting features from a hand image using a pre-trained convolutional network model to obtain a feature map corresponding to the hand image; and a module for using a convolutional network model to perform regression processing on the depth values ​​of feature points in the feature map in the wrist coordinate system and the depth values ​​in the camera coordinate system to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system, wherein the wrist coordinate system is a system with the wrist position as the origin, and its x, y, z coordinates are... The system includes a 3D coordinate system with axes parallel to the corresponding axes of the camera coordinate system; a module for using a convolutional network model to perform regression processing on the 2D heatmap of feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each hand joint in the pixel coordinate system; a module for determining the 2D coordinates of each hand joint based on the 2D heatmap of the hand joint in the pixel coordinate system; and a module for determining the 3D position of each hand joint in the camera coordinate system based on the depth value z of the hand joint in the wrist coordinate system, the depth value z0 of the wrist joint in the camera coordinate system, and the 2D coordinates.

[0010] In one technical solution of this application, a computer-readable storage medium is provided, which stores computer instructions that are operated to execute the 3D joint detection method for a human hand based on a 2D heat map in Solution 1.

[0011] In one technical solution of this application, a computer device is provided, which includes a processor and a memory, wherein the memory stores computer instructions, and the processor operates the computer instructions to execute the method for detecting 3D joints of a human hand based on a 2D heat map in Solution 1.

[0012] The beneficial effects of this application are as follows: This application decomposes the 3D detection process of the human hand to obtain the depth value z of the hand joint in the wrist coordinate system, the depth value z0 of the wrist joint in the camera coordinate system, and the 2D heat map in the pixel coordinate system. By decoupling the depth value z from the plane xy, the 3D problem is transformed into a 2D heat map prediction and a set of numerical z regression problems, thereby reducing the amount of data to be processed and improving the speed and accuracy of the operation. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating one implementation of the 3D joint detection method for human hand based on 2D heatmaps in this application.

[0015] Figure 2 This is a schematic diagram of one embodiment of the 3D joint detection system for human hands based on 2D heat maps, as described in this application.

[0016] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a product or device comprising a series of steps or units is not necessarily limited to those units explicitly listed, but may include other units not explicitly listed or inherent to such products or devices.

[0019] In 3D hand detection, existing methods often involve analyzing hand images to construct a 3D heatmap of the hand's joints. This heatmap allows for the determination of the positions of each joint in 3D space, thus completing the 3D joint detection. However, obtaining this heatmap often requires a neural network model to analyze the hand's 3D pixels, or voxels. For images, analyzing pixels results in a massive amount of data; analyzing voxels with a neural network model further increases the processing load, ultimately slowing down the hand detection process. Furthermore, this method of directly constructing a 3D heatmap using voxels suffers from low accuracy due to the complexity of 3D analysis.

[0020] To address the aforementioned issues, and considering that hand deformation is primarily two-dimensional, this application separates the existing method of directly constructing a 3D heatmap of the hand using voxels to detect 3D joints. The 3D joint detection is decomposed into detecting the 2D heatmap in pixel coordinates and detecting the z-coordinate value in wrist coordinates. The results of these two methods are then combined to ultimately complete the detection of the 3D joints of the hand.

[0021] Therefore, this application proposes a method and system for detecting 3D joints of a hand based on a 2D heatmap. The method includes: extracting features from a hand image using a pre-trained convolutional network model to obtain a feature map corresponding to the hand image; and using the convolutional network model to regress the depth values ​​of the feature points in the feature map in the wrist coordinate system and in the camera coordinate system to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system, wherein the wrist coordinate system is based on the wrist position as the origin, and its x, y... A 3D coordinate system with the z-axis parallel to the corresponding axes of the camera coordinate system is used. A convolutional network model is used to regress the 2D heatmap of the feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each hand joint in the pixel coordinate system. Based on the 2D heatmap of the hand joint in the pixel coordinate system, the 2D coordinates of each hand joint are determined. Based on the depth value z of the hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system, as well as the 2D coordinates, the 3D position of each hand joint in the camera coordinate system is determined.

[0022] This application decomposes the 3D joint detection process of the human hand, and obtains the depth value z of the human hand in the wrist coordinate system, the depth value z0 of the wrist joint in the camera coordinate system, and the 2D heat map in the pixel coordinate system. By decoupling z from the plane xy, the 3D problem is transformed into a 2D problem, thereby reducing the amount of data to be processed and improving the speed and accuracy of the operation.

[0023] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0024] Figure 1 The diagram shows a flowchart of one embodiment of the 3D joint detection method for human hand based on 2D heat map according to this application.

[0025] exist Figure 1 In the embodiment shown, the hand 3D joint detection method based on 2D heat map of this application includes process S101, which uses a pre-trained convolutional network model to extract features from the hand image to obtain the feature map corresponding to the hand image.

[0026] In this embodiment, when performing 3D detection of a hand using a captured image of a hand, a pre-trained convolutional network model is first used to extract features from the hand image to obtain the feature map corresponding to the hand image.

[0027] Optionally, a pre-trained convolutional network model can be used to extract features from the hand image to obtain the corresponding feature map. This includes: reducing the number of model channels in each semantic layer of the Backbone layer of the convolutional network model, and using the model channels to extract features from the hand image to obtain the feature map.

[0028] In this optional embodiment, the detection of hands using hand images does not require excessively precise image information. Furthermore, the more model channels analyzed in an image, the greater the corresponding data processing volume, increasing the model's data processing burden. These model channels correspond to specific image features, such as texture and color. Therefore, when detecting hands, only the model channels corresponding to features useful for hand detection need to be retained, accurately detecting hands while reducing data processing volume. Thus, unlike existing technologies that use standard convolutional network models for in-vehicle human detection, this application reduces the number of model channels in each semantic layer of the Backbone layer of the convolutional network model, utilizing a lower number of model channels to complete feature extraction from in-vehicle images.

[0029] Specifically, in the implementation of the seat-based in-vehicle human detection method of this application, the number of model channels in each semantic layer C3-C5 of the Backbone base layer can be halved. For example, the number of channels in a 32-convolutional layer is halved to 16. By reducing the number of model channels, the data processing volume of the convolutional network model is greatly reduced, and the data processing speed is improved.

[0030] exist Figure 1 In the embodiment shown, the hand 3D joint detection method based on 2D heat map of this application includes process S102, which uses a convolutional network model to perform regression processing on the depth values ​​of feature points in the feature map in the wrist coordinate system and the depth values ​​in the camera coordinate system to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system. The wrist coordinate system is a 3D coordinate system with the wrist position as the origin and its x, y, and z axes are parallel to the corresponding axes of the camera coordinate system.

[0031] In this embodiment, after obtaining the feature map corresponding to the hand image, a convolutional network model is used to perform regression processing on the depth values ​​of the feature points in the feature map in the wrist coordinate system and the depth values ​​of the wrist feature points in the camera coordinate system, thereby obtaining the depth values ​​z of each hand joint in the wrist coordinate system and z0 of the wrist joint in the camera coordinate system. The wrist coordinate system is a 3D coordinate system with the wrist position as the origin, and the directions of its x, y, and z axes are parallel to the corresponding x, y, and z axes of the camera coordinate system. The camera coordinate system can be constructed using existing standard procedures; the specific coordinate system construction process is not described in detail in this application.

[0032] Optionally, a convolutional network model is used to perform regression processing on the depth values ​​of feature points in the feature map in the wrist coordinate system and the depth values ​​of wrist joints in the camera coordinate system to obtain the depth values ​​z and z0 of each hand joint in the wrist coordinate system. This includes: in the wrist coordinate system, analyzing the distance of each hand joint relative to the xoy plane in the wrist coordinate system using a convolutional network model, and using this distance as the corresponding depth value z of the hand joint in the wrist coordinate system; and analyzing the distance of the wrist joint relative to the xoy plane in the camera coordinate system using a convolutional network model, and using this distance as the corresponding depth value z0 of the wrist joint in the camera coordinate system.

[0033] In this optional embodiment, a wrist coordinate system is first constructed, with the wrist as the origin, according to the corresponding standards. A convolutional neural network model is used to regress the feature map corresponding to the hand image to obtain the distance of each hand joint from the xoy plane of the wrist coordinate system. This distance is then used as the depth value z of the hand joint in the wrist coordinate system. The convolutional network model is then used to analyze the wrist joints to obtain the distance of each wrist joint from the xoy plane of the camera coordinate system, and this distance is used as the depth value z0 of the wrist joint in the camera coordinate system.

[0034] exist Figure 1 In the embodiment shown, the hand 3D joint detection method based on 2D heat map of this application includes process S103, which uses a convolutional network model to perform regression processing on the 2D heat map of feature points in the feature map in the pixel coordinate system to obtain the 2D heat map of each hand joint in the pixel coordinate system.

[0035] In this optional embodiment, for the detection of non-rigid objects such as hands, this application employs heatmaps to describe the joints of the hand. In the above process S102, after obtaining the depth values ​​z of each hand joint in the wrist coordinate system and z0 of the wrist joint in the camera coordinate system, only the planar xy information of the hand joints is needed to determine their positions in three-dimensional space. Therefore, this application utilizes a convolutional network model to perform 2D heatmap regression processing on the feature map corresponding to the hand image in the pixel coordinate system, obtaining 2D heatmaps corresponding to each hand joint. The planar heatmap contains the xy information of the hand joints. Combined with the depth values ​​z and z0 obtained in the above process, the hand joints can finally be determined in a 3D environment, completing the 3D hand joint detection.

[0036] Optionally, a convolutional network model is used to perform regression processing on the 2D heatmap of feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each hand joint in the pixel coordinate system. This includes: pre-constructing a pixel coordinate system on the hand image; identifying each hand joint in the feature map in the pixel coordinate system through a convolutional network model, and representing the identified hand joints separately through the heatmap to obtain the 2D heatmap.

[0037] In this optional embodiment, a pixel coordinate system is pre-constructed. Under the pixel coordinate system, a convolutional network model is used to identify each hand joint in the feature map, and finally a 2D heat map corresponding to each hand joint is obtained.

[0038] Optionally, regression processing is performed on the 2D heatmap of the feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each hand joint in the pixel coordinate system, including: performing regression processing on the feature map to obtain the initial heatmap corresponding to each hand joint; upsampling the initial heatmap multiple times to increase the size of the heatmap and obtain the 2D heatmap.

[0039] In this optional embodiment, an initial heatmap corresponding to the hand joints is obtained by performing planar heatmap regression on the feature map. Considering that 2D heatmaps have high resolution requirements, the heatmap displayed on the feature map obtained after feature extraction cannot clearly show the specific location information of the heatmap. Therefore, it is necessary to perform multiple upsampling operations on the obtained initial heatmap to increase its size and obtain a clear 2D heatmap.

[0040] Specifically, three upsampling operations can be performed, with each upsampling involving a doubling of the image size, resulting in a final planar heatmap that is one-quarter the size of the hand image. It should be noted that the number of upsampling operations and the upsampling factor can be reasonably set according to different accuracy requirements and other conditions; this application does not impose specific limitations.

[0041] exist Figure 1 In the embodiment shown, the 3D detection method for human hand based on 2D heat map of this application includes process S104, which determines the 2D coordinates of each human hand joint according to the 2D heat map of human hand joints in pixel coordinate system.

[0042] In this embodiment, after obtaining the planar heat map information of the human hand joints, the planar coordinates xy of the human hand joints can be obtained through the 2D heat map of the human hand joints, thereby determining the 2D coordinates of the human hand joints.

[0043] exist Figure 1 In the embodiment shown, the 3D detection method for human hand based on 2D heat map of this application includes process S105, which determines the 3D position of each human hand joint in the camera coordinate system according to the depth value z of the human hand joint in the wrist coordinate system, the depth value z0 of the wrist joint in the camera coordinate system, and the 2D coordinates.

[0044] In this embodiment, after the above steps, the depth values ​​of the hand joints in the wrist coordinate system and the depth values ​​of the wrist joints in the camera coordinate system, as well as the 2D coordinates corresponding to the 2D heat map in the pixel coordinate system, are obtained respectively. Therefore, by converting the two accordingly, the position information of each hand joint can be determined under the same 3D coordinate system, thereby completing the 3D joint detection of the hand.

[0045] The following example illustrates the 3D hand detection method based on 2D heatmaps proposed in this application. First, during 3D hand detection, the transformation to coordinates involves determining the information of each hand joint. For convenience and to avoid errors, this method focuses on detecting a single hand. When detecting 3D joints of the hand, because the hand is a non-rigid body, direct 3D joint regression results in significant errors. Conventional methods require using a 3D heatmap to predict 3D joint coordinates. However, due to the large number of parameters in a 3D heatmap, and the fact that the error in z-coordinates is often greater than that in x-y-coordinates, decoupling is necessary. Therefore, this application proposes decoupling the planar x-y coordinate information from the z-coordinate information, separating them to prevent errors in the z-coordinate value from affecting the accuracy of the planar x-y coordinates. Furthermore, the x-y regression in this application is performed in a pixel coordinate system. Regressing the 2D heatmap of hand joints in the pixel coordinate system avoids the problem of non-standard wrist coordinates, thus improving the accuracy of hand detection.

[0046] Specifically, for a single hand, there are 21 joints. Therefore, when performing depth value regression in the wrist coordinate system, 21 depth values ​​z are obtained; simultaneously, the depth value z0 of the wrist joint in the camera coordinate system is obtained; similarly, 2D heatmaps of the 21 hand joints are also obtained. The positions of the 21 hand joints in the planar heatmap can be represented by these 21 images. Finally, by combining these three data, the individual hand joints in space are obtained, thus completing the 3D joint detection of the human hand.

[0047] Specifically, by decoupling the 3D detection problem of human hand joints into a 2D heatmap regression problem, in specific tasks where depth information is not required and only 2D joint information of human hand joints is needed, it is possible to choose to perform only 2D heatmap regression processing, which provides greater flexibility for more specific tasks.

[0048] Specifically, the 2D coordinates of the hand joints obtained through the 2D heatmap are pixel coordinates, and based on the pixel coordinates P... pixel = (x, y), using the camera intrinsic parameter K to inversely project onto the normalized coordinate system, we get P′ = (x′, y′) = K -1 *P pixel At this point, the dimension changes from pixels to the physical unit m or mm. Then, combining the z-coordinate value of the depth of the hand joint obtained in the wrist coordinate system and the z0 value of the depth of the wrist joint in the camera coordinate system, through P... cam=((z+z0)*x′,(z+z0)*y′,(z+z0)), which gives the coordinates of the 3D joints of the hand in the camera coordinate system, thus determining the position of the 3D joints of the hand. When defining the wrist coordinate system, the directions of the x, y, and z axes are consistent with the directions of the three axes of the pixel coordinate system, with the wrist joint as the origin, and the units are mm or m.

[0049] This application's 2D heatmap-based 3D hand joint detection method decomposes the 3D hand joint detection process, transforming the complex problem of 3D heatmap regression processing of stereo pixels into a simple 2D heatmap regression problem. By decoupling the depth values ​​of hand joints in the wrist coordinate system from the depth values ​​of wrist joints in the camera coordinate system and the xy-plane, the 3D problem is transformed into a 2D heatmap prediction and a set of numerical z-regression problems. This reduces the amount of data processed, allows for separate regression processing of depth information with high error rates without affecting the accuracy of the xy-plane, and improves both computational speed and accuracy.

[0050] Figure 2 An embodiment of the 3D hand detection system based on 2D heatmaps of this application is shown.

[0051] exist Figure 2 In the illustrated embodiment, the 2D heatmap-based 3D hand detection system of this application includes a module 201 for extracting features from a hand image using a pre-trained convolutional network model to obtain a feature map corresponding to the hand image; and a module 202 for using a convolutional network model to perform regression processing on the depth values ​​of feature points in the feature map in the wrist coordinate system and the depth values ​​in the camera coordinate system to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system. The wrist coordinate system is defined with the wrist position as the origin, and its x, y, and z axes are respectively aligned with the camera... The system includes a 3D coordinate system with corresponding axes parallel to each other; a module 203 for using a convolutional network model to perform regression processing on the 2D heatmap of feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each hand joint in the pixel coordinate system; a module 204 for determining the 2D coordinates of each hand joint based on the 2D heatmap of the hand joint in the pixel coordinate system; and a module 205 for determining the 3D position of each hand joint in the camera coordinate system based on the depth value z of the hand joint in the wrist coordinate system, the depth value z0 of the wrist joint in the camera coordinate system, and the 2D coordinates.

[0052] Optionally, in module 201, the number of model channels in each semantic layer of the Backbone layer in the convolutional network model is reduced, and the model channels are used to extract features from the hand image to obtain a feature map.

[0053] Optionally, in module 202, in the wrist coordinate system, the distance between each hand joint and the xoy plane in the wrist coordinate system is analyzed by a convolutional network model, and the distance is used as the depth value z of the corresponding hand joint; the distance between the wrist joint and the xoy plane in the camera coordinate system is analyzed by a convolutional network model, and the distance is used as the depth value z0 of the corresponding wrist joint.

[0054] Optionally, in module 203, a pixel coordinate system is pre-constructed on the hand image; under the pre-established pixel coordinate system, each hand joint in the feature map is identified through a convolutional network model, and the identified hand joints are represented by heatmaps to obtain 2D heatmaps.

[0055] Optionally, in module 203, classification and regression processing is performed on the feature map to obtain the initial heat map corresponding to each hand joint point; the initial heat map is upsampled multiple times to increase the size of the heat map and obtain a 2D heat map.

[0056] This application's 2D heatmap-based 3D hand joint detection method decomposes the 3D hand joint detection process, transforming the complex problem of 3D heatmap regression processing of stereo pixels into a simple 2D heatmap regression problem. By decoupling the depth values ​​of hand joints in the wrist coordinate system and the depth values ​​of wrist joints in the camera coordinate system from the xy-plane, the 3D problem is transformed into a 2D problem, thereby reducing the amount of data processed. Depth information, which has a high error rate, is regressed separately without affecting the accuracy of hand joints in the xy-plane, thus improving both computational speed and accuracy.

[0057] In one specific embodiment of this application, a computer-readable storage medium stores computer instructions, wherein the computer instructions are operated to perform the 2D heatmap-based 3D joint detection method for a human hand described in any embodiment. The storage medium may be located directly in hardware, in a software module executed by a processor, or in a combination of both.

[0058] Software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in this art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium.

[0059] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, but alternatively, it can be any conventional processor, controller, microcontroller, or state machine. The processor can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration. Alternatively, the storage medium can be integrated with the processor. The processor and storage medium can reside in an ASIC. The ASIC can reside in the user terminal. Alternatively, the processor and storage medium can reside as discrete components in the user terminal.

[0060] In one specific embodiment of this application, a computer device includes a processor and a memory, the memory storing computer instructions, wherein the processor operates the computer instructions to execute the 2D heatmap-based 3D joint detection method for human hand described in any embodiment.

[0061] In the embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0062] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0063] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. A method for detecting 3D joints of a human hand based on 2D heatmaps, characterized in that, include: A pre-trained convolutional network model is used to extract features from a human hand image to obtain a feature map corresponding to the human hand image. Using the convolutional network model, depth value regression processing is performed on the depth values ​​of feature points in the feature map in the wrist coordinate system and the depth values ​​in the camera coordinate system to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system. The wrist coordinate system is a 3D coordinate system with the wrist position as the origin and its x, y, and z axes parallel to the corresponding axes of the camera coordinate system. Using the convolutional network model, heatmap regression processing is performed on the 2D heatmap of the feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each of the hand joints in the pixel coordinate system. The pixel coordinate system is a planar coordinate system pre-constructed on the hand image, with the upper left corner of the image as the origin, the horizontal rightward direction as the positive x-axis, and the vertical downward direction as the positive y-axis. Based on the 2D heatmap of the hand joints in the pixel coordinate system, the 2D coordinates of each hand joint are determined; and Based on the depth value z of the hand joint in the wrist coordinate system, the depth value z0 of the wrist joint in the camera coordinate system, and the 2D coordinates, the 3D position of each hand joint in the camera coordinate system is determined.

2. The method for detecting 3D joints of a human hand based on 2D heatmaps according to claim 1, characterized in that, The step of extracting features from a hand image using a pre-trained convolutional network model to obtain a feature map corresponding to the hand image includes: The number of model channels in each semantic layer of the Backbone layer in the convolutional network model is reduced, and the model channels are used to extract features from the hand image to obtain the feature map.

3. The method for detecting 3D joints of a human hand based on 2D heatmaps according to claim 1, characterized in that, The process of using the convolutional network model to perform depth regression processing on the depth values ​​of feature points in the feature map in the wrist coordinate system and the depth values ​​in the camera coordinate system, to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system, includes: In the wrist coordinate system, the distance between each of the human hand joints and the xoy plane in the wrist coordinate system is analyzed by the convolutional network model, and the distance is used as the depth value z of the corresponding human hand joint. The distance between the wrist joint and the xoy plane in the camera coordinate system is analyzed using the convolutional network model, and this distance is used as the depth value z0 of the corresponding wrist joint.

4. The method for detecting 3D joints of a human hand based on 2D heatmaps according to claim 1, characterized in that, The step of using the convolutional network model to perform heatmap regression processing on the 2D heatmap of the feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each of the human hand joints in the pixel coordinate system includes: The pixel coordinate system is pre-constructed on the image of the human hand; In the pixel coordinate system, the convolutional network model identifies each of the human hand joints in the feature map, and the identified human hand joints are represented by heatmaps to obtain the 2D heatmaps.

5. The method for detecting 3D joints of a human hand based on a 2D heatmap according to claim 1 or 4, characterized in that, The step of performing heatmap regression processing on the 2D heatmap of the feature points in the feature map in the pixel coordinate system to obtain the 2D heatmap of each of the human hand joints in the pixel coordinate system includes: The feature map is subjected to classification and regression processing to obtain the initial heat map corresponding to each of the human hand joints; The initial heatmap is upsampled multiple times to increase its size, thus obtaining the 2D heatmap.

6. A 3D joint detection system for a human hand based on 2D heatmaps, characterized in that, include: A module for extracting features from a human hand image using a pre-trained convolutional network model to obtain a feature map corresponding to the human hand image; The module is used to perform depth value regression processing on the depth values ​​of feature points in the feature map in the wrist coordinate system and the depth values ​​in the camera coordinate system using the convolutional network model, so as to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system. The wrist coordinate system is a 3D coordinate system with the wrist position as the origin and its x, y, and z axes are parallel to the corresponding axes of the camera coordinate system. The module is used to perform heatmap regression processing on the 2D heatmap of the feature points in the feature map in the pixel coordinate system using the convolutional network model, so as to obtain the 2D heatmap of each of the hand joints in the pixel coordinate system. The pixel coordinate system is a planar coordinate system pre-constructed on the hand image, with the upper left corner of the image as the origin, the horizontal rightward direction as the positive x-axis, and the vertical downward direction as the positive y-axis. A module for determining the 2D coordinates of each of the hand joints based on a 2D heatmap of the hand joints in the pixel coordinate system; and A module for determining the 3D position of each hand joint in the camera coordinate system based on the depth value z of the hand joint in the wrist coordinate system, the depth value z0 of the wrist joint in the camera coordinate system, and the 2D coordinates.

7. The 3D joint detection system for a human hand based on a 2D heat map according to claim 6, characterized in that, In the module that uses the convolutional network model to perform depth value regression processing on the depth values ​​of feature points in the feature map in the wrist coordinate system and the depth values ​​in the camera coordinate system, to obtain the depth value z of each hand joint in the wrist coordinate system and the depth value z0 of the wrist joint in the camera coordinate system, In the wrist coordinate system, the distance between each hand joint and the xoy plane in the wrist coordinate system is analyzed using the convolutional network model, and this distance is used as the depth value z of the corresponding hand joint. The distance between the wrist joint and the xoy plane in the camera coordinate system is analyzed using the convolutional network model, and this distance is used as the depth value z0 of the corresponding wrist joint.

8. The 3D joint detection system for a human hand based on a 2D heat map according to claim 6, characterized in that, In the module that utilizes the convolutional network model to perform heatmap regression processing on the 2D heatmap of the feature points in the feature map in the pixel coordinate system, and obtains the 2D heatmap of each of the human hand joints in the pixel coordinate system, A pixel coordinate system is pre-constructed on the image of the human hand, and under the pixel coordinate system, the convolutional network model is used to identify each of the human hand joints in the feature map, and the identified human hand joints are represented by heat maps to obtain the 2D heat map.

9. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions that are operated to perform the 2D heatmap-based 3D joint detection method for human hands as described in any one of claims 1-5.

10. A computer device comprising a processor and a memory, the memory storing computer instructions, wherein: The processor operates computer instructions to execute the 2D heatmap-based 3D joint detection method for human hands as described in any one of claims 1-5.

Images