3D Gesture Recognition Method and System Based on a Single RGB Image

Abstract

The present invention discloses a three-dimensional gesture recognition method and system based on a single RGB image. First, a hand segmentation network is used to generate a hand mask that only contains the hand, and the original RGB image is cropped using the hand mask to obtain a hand image that only contains the hand. Then, a residual network is used to extract the features of the cropped hand image, reducing the computational amount and preventing overfitting. Next, a two-dimensional key point detection network integrates a convolutional neural network into a prediction framework that can learn and serialize rich spatial information models, continuously refining the key point positions of the hand. Finally, three-dimensional key points are reconstructed, generating three-dimensional key points from the two-dimensional key points and outputting the most likely gesture. The present invention fully explores the three-dimensional gesture recognition method based on a single RGB image, so as to achieve the most direct human-computer interaction method through gestures in a virtual scene, enabling the machine to understand human expressions.

Description

Technical Field

[0001] The present invention belongs to the field of gesture recognition. More specifically, it is a three-dimensional gesture recognition method and system based on a single RGB image. Background Art

[0002] For gesture estimation based on wearable devices, generally an integrated sensor is placed in the wearable device for gesture estimation, which is completely unaffected by hand occlusion, can quickly obtain data information, and has relatively high accuracy. Commonly used wearable devices include motion sensors and data gloves. However, due to their high price, complex operation, and the need to wear sensor devices each time, it is very inconvenient, which results in their not being widely used in daily life.

[0003] Early three-dimensional gesture estimation was based on depth images. For example, a gesture model composed of multiple key points was predefined and fitted to the input depth image. First, a large number of candidate gesture models were created, then a gesture model that best matched the current depth image was selected from the large number of created gesture models. After that, an objective function was designed to compare the similarity between the depth map and the gesture model. By minimizing this objective function, the best hand model was found. The computational cost was high in the optimization problem of the objective function, and the accuracy of gesture estimation highly depended on the design of the similarity function, which was disadvantageous for gesture estimation. Generally, good results were achieved for static gesture estimation, but the effect was not so ideal when the image was rotated, distorted, or resized; for another example, when using multiple views for three-dimensional gesture estimation, the two-dimensional key point coordinates were directly estimated from the depth map through a two-dimensional heat map, and then the three-dimensional key points were estimated by means of multiple views to generate geometrically consistent key points of the hand. However, the multi-view camera system was costly and was generally used in laboratories. Currently, most works on three-dimensional gesture estimation from a single RGB image have limitations and require additional depth maps or multi-view images. Summary of the Invention

[0004] Object of the Invention: Aiming at the deficiencies of the prior art, the object of the present invention is to provide a three-dimensional gesture recognition method and system based on a single RGB image, which uses only a single RGB image as input, predicts two-dimensional key points through a hand segmentation network and a two-dimensional key point detection network in sequence, and then calculates the absolute depth of the wrist key point to reconstruct the three-dimensional key points, having the advantages of low cost, small computational amount, and high accuracy.

[0005] Technical Solution: To achieve the above object of the invention, the present invention adopts the following technical solution:

[0006] A three-dimensional gesture recognition method based on a single RGB image includes the following steps:

[0007] (1) Input a single RGB image into the hand segmentation network to generate a hand mask containing only the hand, and then use the hand mask to crop the original RGB image to obtain a hand image containing only the hand;

[0008] (2) Use the residual network ResNet-50 to extract the features of the cropped hand image;

[0009] (3) using a two-dimensional key point detection network to refine the key point positions of the hand; the two-dimensional key point detection network adopts a serialized structure, which is divided into multiple stages to continuously narrow the key point detection range, and uses a confidence map to represent the key point results of the hand. The latter stage directly operates on the confidence map of the previous stage and outputs increasingly refined key point positions of the hand; the input of the latter stage is the output of the previous stage and the features of the hand image extracted by the residual network;

[0010] (4) Reconstructing 3D key points based on 2D key points to obtain 3D gestures; during the reconstruction process, the absolute depth value of the wrist key point is first calculated, and then the relative coordinates with the wrist key point are used to locate the 3D key point; the absolute depth value of the wrist key point is α x , α y , A real and A img are the focal length multiplied by the pixel distance factor for the x and y axes, and the area of the hand in real space and image space, respectively.

[0011] As a priority, in the two-dimensional keypoint detection network, at each position z = (u, v) in the image, all confidence scores of the p-th keypoint at stage t are It is expressed as:

[0012]

[0013] in, w and h are the width and height of the image, respectively. t∈{1,...,T} represents T stages. p(x,y) represents the coordinates of the pth key point. is the confidence score when the pth key point is determined at image position z in the tth stage;

[0014] The confidence set of all key points is recorded as In the stage t>1, the classifier predicts the confidence of the key point position based on two inputs: one is the image features, and the other is the contextual content information output by the classifier in the previous stage; the confidence estimate of each key point calculated in each stage becomes more and more refined, and the image features used in subsequent stages are different from those used in the first stage.

[0015] As a priority, let P be an arbitrary point P(X, Y, Z) in three-dimensional space, and p be the projection p(x, y) of the spatial point P on the image plane; the reconstruction of three-dimensional key points from two-dimensional key points is expressed as:

[0016]

[0017] where f is the focal length, Z root is the absolute depth of the wrist key point, and Z r is the depth of the p-th key point relative to the wrist key point.

[0018] As a priority, the hand segmentation network performs multi-layer convolution and pooling operations on the input image. The number of convolution kernel channels in the convolution operation after each pooling becomes twice the original to compensate for the features lost in pooling; the output image only contains the image of the hand, and finally, upsampling is performed to generate a hand mask.

[0019] As a priority, the output feature map size of the ResNet-50 is 46×46×512. The number of input channels of the two-dimensional key point detection network in the first stage is 512, and it outputs an image of size 46×46×21 through a series of convolution operations; a series of convolution operations refer to 7 convolutional layers, including 5 convolutional layers with a convolution kernel size of 7×7, a stride of 1, a padding of 3, and a channel number of 128, 1 convolutional layer with a convolution kernel size of 1×1, a stride of 1, a padding of 0, and a channel number of 128, and 1 convolutional layer with a convolution kernel size of 1×1, a stride of 1, a padding of 0, and a channel number of 21; before the second stage, the output 46×46×21 of the first stage and the feature map 46×46×512 of the ResNet-50 are concatenated, and the output channel number is 533 to compensate for the lost feature information; the input in the second stage is 46×46×533, and the process of the first stage is repeated, and so on, to output the position of the gesture key point that is becoming more and more refined.

[0020] A three-dimensional gesture recognition system based on a single RGB image includes:

[0021] A hand region detection module for inputting a single RGB image into the hand segmentation network to generate a hand mask that only contains the hand, and cropping the original RGB image using the hand mask to obtain a hand image that only contains the hand;

[0022] A hand feature extraction module for extracting the features of the cropped hand image using the residual network ResNet-50;

[0023] The two-dimensional key point detection module is used to refine the positions of the key points of the hand by using a two-dimensional key point detection network; the two-dimensional key point detection network adopts a serialized structure, divides into multiple stages to continuously narrow the key point detection range, represents the key point results of the hand in the form of a confidence map, and the subsequent stage directly operates on the confidence map of the previous stage to output the positions of the key points of the hand that are increasingly refined; the input of the subsequent stage is the output of the previous stage and the features of the hand image extracted by the residual network;

[0024] And the three-dimensional key point detection module is used to reconstruct three-dimensional key points based on the two-dimensional key points to obtain three-dimensional gestures; during the reconstruction process, first obtain the absolute depth value of the wrist key point, and then use the relative coordinates with the wrist key point for three-dimensional key point positioning; where the absolute depth value of the wrist key point α x ,α y ,A real and A img are the focal length multiplied by the pixel distance factors of the x and y axes, and the areas of the hand in the real space and the image space respectively.

[0025] A computer system includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the three-dimensional gesture recognition method based on a single RGB image are implemented.

[0026] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the three-dimensional gesture recognition method based on a single RGB image are implemented.

[0027] Beneficial effects: The three-dimensional gesture recognition method based on a single RGB image proposed by the present invention overcomes two major drawbacks of previous gesture recognition methods: First, most gesture recognitions only detect two-dimensional key points. Gesture estimation is more challenging than human body estimation because hand joints are more flexible, with agile movements and self-occlusion effects. Second, the detection of three-dimensional key points for gesture recognition often relies on depth maps and cannot estimate the three-dimensional key points of the hand from a single RGB image. To address these two drawbacks, the present invention performs three-dimensional gesture recognition on RGB images. The overall structure adopts a cascaded multi-task structure. The hand mask generated by the hand segmentation network is used to crop the area near the hand. The cropped hand image is then used by the residual network for feature extraction to reduce the computational amount and improve accuracy. The two-dimensional key point detection is generated by the two-dimensional key point detection network. Then, the three-dimensional key points are reconstructed from the two-dimensional key points. Perspective projection can be used to reconstruct the three-dimensional joint points. However, directly obtaining the three-dimensional key points from the RGB image will cause depth ambiguity because there is no depth map to provide depth information. To solve this problem, the absolute depth value of the wrist key point is calculated. The depth of each key point of the hand is represented by the sum of the depth of the wrist key point and the relative depth of the wrist key point. The most likely three-dimensional gesture is output. The entire network's three-dimensional gesture recognition method based on a single RGB image improves the generalization ability of gesture estimation. Experiments have proven the effectiveness of the method of the present invention, and its recognition performance is good on different data sets. Description of the Drawings

[0028] Figure 1 It is a schematic structural diagram of an embodiment of the present invention, mainly including a hand segmentation network, a residual network, a two-dimensional key point detection network, and a three-dimensional key point reconstruction.

[0029] Figure 2 It is a schematic diagram of the projection of three-dimensional key points in the image coordinate system. Detailed Embodiment

[0030] For a clearer understanding of the technical features, objectives, and effects of the present invention, the specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0031] As Figure 1 shown, a three-dimensional gesture recognition method based on a single RGB image disclosed in an embodiment of the present invention uses the RGB image as the input for gesture recognition and performs three-dimensional gesture recognition without a depth map providing depth information. Considering that the device requirements for obtaining a depth map are high, generally a depth camera is needed. To generalize the ability of gesture recognition, the present invention does not use a depth map but uses an RGB image. The specific steps are as follows:

[0032] 1. Input a single RGB image into the hand segmentation network to generate a hand mask that only contains the hand, and cut out a hand image that only contains the hand.

[0033] In this step, the hand segmentation network detects the hand region. The hand region detection problem can be transformed into a score map of the palm center position. Calculate the score map for each position in the input image to find the most likely palm center position; perform non-maximum suppression to complete hand edge detection; perform a binarization operation on the detected hand region image to generate a hand mask; crop the original image according to the hand mask. The detection of the target hand is beneficial for subsequent gesture recognition.

[0034] Specifically, the input of the gesture segmentation network is an image of 368×368×3. After multiple convolutional and pooling operations, its size becomes 46×46×1. The output is an image containing only the hand. Then, through upsampling, an enlarged hand image containing only the hand is generated, with the image size of 368×368×1. The hand segmentation network consists of 14 convolutional layers, 3 pooling layers, and 1 upsampling layer. First, it goes through two convolutional layers with a convolutional kernel size of 3×3 and a stride of 1. Then, it passes through a max-pooling layer with a convolutional kernel size of 4×4 and a stride of 2, and the image is reduced to half of its original size. After each pooling, the number of channels of the convolutional kernel in the subsequent convolutional operation becomes twice that of the original to compensate for the features lost during pooling. The output image contains only the hand. Finally, upsampling is performed to generate a hand mask; the original image is cropped according to the hand mask to segment the target hand in the image and reduce the difficulty of subsequent gesture recognition.

[0035] Second, the residual network extracts features from the hand image containing only the hand obtained by cropping the hand mask generated by the hand segmentation network and outputs a feature map.

[0036] The residual network extracts features from the cropped hand image, which can prevent the phenomenon of overfitting caused by too many network layers. In the present invention, the residual network ResNet-50 is adopted for feature extraction. ResNet-50 was pre-trained on the ImageNet dataset. Using ResNet-50 as the basic framework, the size of the output feature map is 46×46×512, reducing the computational amount.

[0037] Third, the 2D key point detection network refines the key point positions of the hand.

[0038] The two-dimensional key point detection network continuously refines the key point positions due to its serial structure, and we apply it to the key point detection of the hand. The two-dimensional key point detection network estimates the key point coordinates of the hand and represents the hand key point results in the form of a confidence map. The input of the two-dimensional key point detection network is a hand image that only contains the hand, and the image size is 46×46×512. The gesture key point detection of the two-dimensional key point detection network adopts a serial structure, which means it is divided into multiple stages to continuously narrow the key point detection range; it uses T stages to refine the hand key point results, and the subsequent stage directly operates on the confidence map of the previous stage to output the increasingly refined hand key point positions.

[0039] In the first stage, the number of input channels is 512, and through a series of convolution operations, the output image size is 46×46×21; a series of convolution operations refer to 7 convolutional layers, including 5 convolutional layers with a convolutional kernel size of 7×7, a stride of 1, a padding of 3, and a channel number of 128, 1 convolutional layer with a convolutional kernel size of 1×1, a stride of 1, a padding of 0, and a channel number of 128, and 1 convolutional layer with a convolutional kernel size of 1×1, a stride of 1, a padding of 0, and a channel number of 21, to control the output dimension.

[0040] Before the second stage, the output of the first stage 46×46×21 and the feature map 46×46×512 passing through ResNet-50 are concatenated, and the output channel number is 533 to make up for the lost feature information.

[0041] In the second stage, the input is 46×46×533, and the process of the first stage is repeated, and so on, to output the increasingly refined gesture key point positions.

[0042] Specifically, for 21 key points, the coordinates of the p-th key point position of the gesture output can be expressed as:

[0043] p(x,y)=z (1)

[0044] where p(x,y) represents the coordinates of the p-th key point position, and z is the position (u,v) of the point in the image.

[0045] The key point confidence output by the classifier g1 (the classifier finally connected to the convolutional layer in the first stage) can be expressed as:

[0046]

[0047] where is the confidence when the classifier g1 determines the p-th key point at the image position z in the first stage, refers to the feature extracted at the image position z, and d is the dimension of the extracted feature vector.

[0048] At each position z of the image, all the confidence scores of the key point p at stage t are It can be expressed as:

[0049]

[0050] where z = (u, v), w and h are the width and height of the image respectively, and t ∈ {1,..., T} represents T stages.

[0051] Denote the set of confidence scores of all key points as At stages where t > 1, the classifier predicts the confidence scores of the positions of the image key points based on two inputs, one is the image feature, and the other is the context content information output by the classifier in the previous stage.

[0052]

[0053] where is the image feature, and ψ t＞1 (·) is the mapping from the confidence score b t-1 to the context feature. The estimation of the confidence score for each key point becomes more and more refined at each stage, and the image feature x' used in subsequent stages is different from the image feature x used in the first stage.

[0054] IV. Reconstructing 3D Key Points Generate 3D key points using 2D key points. If the image provides sufficient information about the position of the root node (wrist key point) of the hand in the image space, the most likely gesture can be output.

[0055] Perspective projection can be used to reconstruct 3D joint points, but directly obtaining 3D key points from an RGB image will cause depth ambiguity because there is no depth map to provide depth information. To solve this problem, we first obtain the absolute depth value of the wrist key point, and then use the relative coordinates relative to the wrist key point in perspective projection for 3D key point localization.

[0056] The absolute depth value of the wrist key point is related to the area of the hand in the real space and the image space. For the x-axis and y-axis centered on the wrist key point, the length of an object on the x-axis and y-axis in the image space of the hand and the length of an object in the real space. On the x-axis, according to the definition of tanh, the absolute depth value of the wrist key point is the focal length multiplied by the ratio of the length of the hand on the sensor on the x-axis to the length of the hand in the real space. On the y-axis, according to the definition of the tanh function, the absolute depth value of the wrist key point is the focal length multiplied by the ratio of the length of the hand on the sensor on the y-axis to the length of the hand in the real space. Where the length of the hand in the image space is equal to the per-pixel distance factor multiplied by the length of the hand on the sensor.

[0057] For the x-axis and y-axis centered on the wrist key point, the definition of tan can be expressed as:

[0058]

[0059] where Z root , f, l x,sensor and l x,real are respectively the distance from the camera to the wrist key point, the focal length, the length of an object on the hand in the image space on the x-axis, and the length of an object in the real space on the x-axis.

[0060] The distance Z from the camera to the wrist key point root can be expressed as:

[0061]

[0062] where p x is the distance factor per pixel on the x-axis, l x,real , l x,img , l y,real , l y,img are respectively the lengths of an object in the real space and the image space, and fp x represents the focal length multiplied by the pixel distance factor on the x-axis, and is uniformly represented by α x .

[0063] The above equation also holds within the y-axis. The distance Z from the camera to the wrist key point root can be expressed as:

[0064]

[0065] Finally, on the x and y axes, by multiplying the two representations of Z root in the above formula and taking the square root, the absolute depth Z root of the root key point of the hand can be expressed as:

[0066]

[0067] where, α x , α y , A real and A img are respectively the pixel distance factors (pixel) of the focal length multiplied by the x and y axes, and the areas of the hand in the real space and the image space.

[0068] Perspective projection is the conversion between the three-dimensional key point coordinates in the world coordinate system and the two-dimensional key point coordinates in the image coordinate system; the three-dimensional key points in the world coordinate system are converted into three-dimensional key points in the camera coordinate system through rotation and translation; the world coordinate system means that the camera and the object to be photographed can be placed at any position in the space environment, and a coordinate system is established in the space environment to represent the positions of the camera and the object to be photographed. The camera coordinate system is a three-dimensional rectangular coordinate system established with the focus center of the camera as the origin and the optical axis as the Z-axis. Multiplying the three-dimensional key point coordinates in the world coordinate system by the external camera parameter matrix can obtain the camera coordinate system. The external camera parameter matrix refers to the matrix obtained by multiplying the matrices that rotate and translate the three directions of the XYZ axes in the world coordinate system around three different directions to obtain the camera coordinate system.

[0069] Figure 2 It represents the two-dimensional key point coordinates obtained by projecting the three-dimensional key points in the camera coordinate system onto the image coordinate system. Multiplying the three-dimensional key point coordinates in the camera coordinate system by the internal camera parameter matrix gives the two-dimensional key point coordinates in the image coordinate system.

[0070] The internal camera parameter matrix refers to the internal parameter matrix of the camera

[0071] The depth of each key point of the hand is represented by the sum of the depth of the wrist key point and the depth relative to the wrist key point. The depth of each key point of the hand relative to the wrist key point is generated by performing a soft-argmax operation on the output three-dimensional Gaussian heatmap of the constructed upsampler. Among them, soft-argmax can extract the coordinates of the depth relative to the wrist key point from the three-dimensional Gaussian heatmap. The constructed upsampler is generated by 3 transposed convolutional layers and 1 convolutional layer. Among them, the transposed convolutional layers all have batch normalization layers and ReLU activation functions, and the convolutional layer is a 1×1 convolutional layer. The input of the upsampler is the feature map generated by the residual network. For details, please refer to the literature [X.Sun,B.Xiao,F.Wei,S.Liang,and Y.Wei.Integralhuman pose regression.In ECCV,2018].

[0072] Mapping two-dimensional key points to three-dimensional key points, P is an arbitrary point P(X,Y,Z) in three-dimensional space, and p is the projection p(x,y) of the spatial point P on the image plane. According to the definition of perspective projection, the reconstruction of three-dimensional key points from two-dimensional key points can be expressed as:

[0073]

[0074] where f is the effective focal length; Z root is the absolute depth of the wrist key point; Z r is the depth of the p-th key point relative to the wrist key point.

[0075] Table 1 shows the probabilities of correct key-point detection of the present invention (abbreviated as MCPM in English) on different datasets (Stereo Hand Pose Tracking Benchmark, Dexter Object, EgoDexter). Here, only the situation of 10 epochs is recorded, and the probabilities of correct key-point detection are shown in percentage form, and the correct key-point probabilities are the averages. Table 2 shows the probabilities of correct key-point detection of MCPM on the Stereo Hand Pose Tracking Benchmark dataset under different thresholds and different epochs. The Stereo Hand Pose Tracking Benchmark dataset is a dataset about gestures provided and published by Zhang et al., and the data types are RGB data and depth data, where the RGB data is 640×480 pixels and the depth data is 640×480 pixels. The dataset contains 12 independent sequences, a total of 18,000 images, and provides two-dimensional and three-dimensional annotations of 21 key points. It consists of 6 different backgrounds, each background has two sequences, namely counting and random poses, and each sequence has 1500 images. The Dexter Object dataset was established by Sridhar et al. and is a dataset taken from the third perspective. The data types include RGB data and depth data, where the RGB data is 640×480 pixels and the depth data is 640×480 pixels. The characteristic of the data is that different actors interact with objects under different cluttered backgrounds. It consists of 6 sequences, grasping a small cuboid, grasping a large cuboid, pinching a small cuboid, occluding a small cuboid, rigidly moving a small cuboid, and rotationally moving a small cuboid. The dataset provides two-dimensional and three-dimensional annotations of the left hand. The EgoDexter dataset was established by Mueller et al. and is a large-scale dataset centered on the self. The data types are RGB data and depth data, where the RGB data is 640×480 pixels and the depth data is 640×480 pixels. This dataset is divided into 4 situations: hands interacting with objects on the table; hands interacting with utensils in the kitchen; hands interacting with some other cluttered objects. The dataset provides two-dimensional and three-dimensional annotations of gestures. We used the above three datasets in the experiment. In the experiment, the resolution of the dataset was reduced to 368×368×3, and then the weights of the entire network were initialized. Secondly, the standard Softmax cross-entropy loss function was used for iterative training, the number of iterations was 80, and the learning rate was set to 0.01. It decays by 0.0005 every 10 rounds.

[0076] Table 1: Probability of correct key-point detection (PCK) of the MCPM method on different datasets

[0077]

[0078] Table 2: Probability of Correct Detection of Key Points (σPCK) of MCPM under Different Thresholds and Different Epochs on the Stereo Hand Pose Tracking Benchmark Dataset

[0079]

[0080] Based on the same inventive concept, a three-dimensional gesture recognition system disclosed in an embodiment of the present invention based on a single RGB image includes: a hand region detection module, configured to input a single RGB image into a hand segmentation network to generate a hand mask containing only the hand, and crop the original RGB image using the hand mask to obtain a hand image containing only the hand; a hand feature extraction module, configured to extract features of the cropped hand image using the Residual Network ResNet-50; a two-dimensional key point detection module, configured to refine the positions of the key points of the hand using a two-dimensional key point detection network; and a three-dimensional key point detection module, configured to reconstruct three-dimensional key points based on the two-dimensional key points to obtain a three-dimensional gesture. During the reconstruction process, the absolute depth value of the wrist key point is first obtained, and then the three-dimensional key points are located using the relative coordinates with the wrist key point.

[0081] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described modules can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein. The division of the modules is only a logical function division, and there may be other division methods in actual implementation. For example, multiple modules can be combined or integrated into another system.

[0082] Based on the same inventive concept, a computer system disclosed in an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the steps of the three-dimensional gesture recognition method based on a single RGB image are implemented.

[0083] Based on the same inventive concept, a computer-readable storage medium disclosed in an embodiment of the present invention stores a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional gesture recognition method based on a single RGB image are implemented.

[0084] Those skilled in the art can understand that the technical solution of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer system (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present invention. The storage medium includes: various media that can store computer programs, such as USB flash drives, mobile hard disks, read-only memory ROM, random access memory RAM, magnetic disks, or optical discs.

Claims

1. A three-dimensional gesture recognition method based on a single RGB image, characterized in that, It includes the following steps: (1) Input a single RGB image into the hand segmentation network to generate a hand mask that only contains the hand, and use the hand mask to crop the original RGB image to obtain a hand image that only contains the hand; (2) Use the Residual Network ResNet-50 to extract the features of the cropped hand image; (3) Refine the key point positions of the hand using a two-dimensional key point detection network; the two-dimensional key point detection network adopts a serialized structure, divides into multiple stages to continuously narrow the key point detection range, represents the key point results of the hand in the form of confidence maps, and the subsequent stage directly operates on the confidence map of the previous stage to output the key point positions of the hand that are increasingly refined; the input of the subsequent stage is the output of the previous stage and the features of the hand image extracted by the residual network; at each position z=(u, v) of the image, all confidence scores of the p-th key point with coordinates p(x, y) at stage t are Expressed as: Among them, w and h are the width and height of the image respectively, is the confidence score when determining the p-th key point at the image position z in the t-th stage; Denote the confidence set of all key points as (4) Reconstruct three-dimensional key points based on two-dimensional key points to obtain three-dimensional gestures; during the reconstruction process, first obtain the absolute depth value of the wrist key point, and then use the relative coordinates with the wrist key point for three-dimensional key point positioning. The depth of each key point of the hand is represented by the sum of the depth of the wrist key point and the depth relative to the wrist key point, and the most likely three-dimensional gesture is output; where the absolute depth value of the wrist key point α x , α y , A real and A img are the focal length multiplied by the pixel distance factors of the x and y axes, and the areas of the hand in the real space and the image space, respectively.

2. The three-dimensional gesture recognition method based on a single RGB image according to claim 1, wherein In the two-dimensional key point detection network, at the stage where t > 1, the classifier predicts the confidence of the key point position based on two inputs, one is the image feature, and the other is the context content information output by the classifier in the previous stage; the estimation of the confidence for each key point becomes more and more refined in each stage, and the image features used in the subsequent stages are different from the image features used in the first stage.

3. The three-dimensional gesture recognition method based on a single RGB image according to claim 1, wherein Let P be any point P(X, Y, Z) in the three-dimensional space, and p be the projection p(x, y) of the space point P on the image plane; the reconstruction of the two-dimensional key point into the three-dimensional key point is expressed as: where f is the focus, and Z root is the absolute depth of the wrist key point, and Z r is the depth of the p-th key point relative to the wrist key point.

4. The 3D gesture recognition method based on a single RGB image according to claim 1, wherein The hand segmentation network performs multiple convolutional and pooling operations on the input image, and the number of convolutional kernel channels of the convolutional operation after each pooling becomes twice the original to make up for the features lost by pooling; The output image only contains the image of the hand, and finally upsampling is performed to generate the hand mask.

5. The 3D gesture recognition method based on a single RGB image according to claim 1, characterized in that The output feature map of the ResNet-50 has a size of 46×46×512, the number of input channels of the two-dimensional key point detection network in the first stage is 512, and through a series of convolutional operations, the output image size is 46×46×21; a series of convolutional operations refers to 7 convolutional layers, including 5 convolutional layers with a convolutional kernel size of 7×7, a stride of 1, a padding of 3, and a channel number of 128, 1 convolutional layer with a convolutional kernel size of 1×1, a stride of 1, a padding of 0, and a channel number of 128, and 1 convolutional layer with a convolutional kernel size of 1×1, a stride of 1, a padding of 0, and a channel number of 21; Before the second stage, the output 46×46×21 of the first stage and the feature map 46×46×512 passing through the ResNet-50 are concatenated, and the output channel number is 533 to make up for the lost feature information; the input in the second stage is 46×46×533, and the process of the first stage is repeated, and so on, to output the gesture key point positions that are more and more refined.

6. A three-dimensional gesture recognition system based on a single RGB image, characterized in that, It includes: A hand region detection module for inputting a single RGB image into the hand segmentation network to generate a hand mask that only contains the hand, and using the hand mask to crop the original RGB image to obtain a hand image that only contains the hand; A hand feature extraction module for using the Residual Network ResNet-50 to extract the features of the cropped hand image; The two-dimensional key point detection module is used to refine the key point positions of the hand by using a two-dimensional key point detection network; the two-dimensional key point detection network adopts a serialized structure, which is divided into multiple stages to continuously narrow the key point detection range, and represents the key point results of the hand in the form of a confidence map. The subsequent stage directly operates on the confidence map of the previous stage to output the key point positions of the hand that are increasingly refined; the input of the subsequent stage is the output of the previous stage and the features of the hand image extracted by the residual network; at each position z=(u,v) of the image, all confidence scores of the p-th key point with coordinates p(x,y) at stage t are Expressed as: Among them, w and h are the width and height of the image respectively, is the confidence score when determining the p-th key point at the image position z in the t-th stage; Denote the set of confidence levels of all key points as And a three-dimensional key point detection module for reconstructing the three-dimensional key points based on the two-dimensional key points to obtain a three-dimensional gesture; during the reconstruction process, first obtain the absolute depth value of the wrist key point, and then use the relative coordinates with the wrist key point for three-dimensional key point positioning. For the depth of each key point of the hand, it is represented by the sum of the depth of the wrist key point and the depth relative to the wrist key point, and the most likely three-dimensional gesture is output. The absolute depth value of the wrist key point α x , α y , A real and A img are the focal length multiplied by the pixel distance factors of the x and y axes, and the areas of the hand in the real space and the image space, respectively.

7. The 3D gesture recognition system based on a single RGB image according to claim 6, characterized in that, In the two-dimensional key point detection network, at the stage where t > 1, the classifier predicts the confidence of the key point position based on two inputs. One is the image feature, and the other is the context content information output by the classifier in the previous stage. The estimation of the confidence for each key point becomes more refined in each stage, and the image features used in subsequent stages are different from those used in the first stage.

8. The three-dimensional gesture recognition system based on a single RGB image according to claim 6, wherein Let P be an arbitrary point P(X, Y, Z) in three-dimensional space, and p be the projection p(x, y) of the spatial point P on the image plane. The reconstruction of the two-dimensional key point into the three-dimensional key point is expressed as: Among them, f is the focus, Z root is the absolute depth of the wrist key point, Z r is the depth of the p-th key point relative to the wrist key point.

9. A computer system, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the steps of the three-dimensional gesture recognition method based on a single RGB image according to any one of claims 1-5.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the three-dimensional gesture recognition method based on a single RGB image according to any one of claims 1-5.

Images