Two-dimensional hand pose estimation method and system based on multi-scale feature fusion network

Abstract

The application discloses a two-dimensional hand gesture estimation method and system based on a multi-scale feature fusion network. The method comprises the following steps: collecting original hand single-color images to form a data set and performing pretreatment; performing data expansion processing on the training set; constructing a multi-scale feature fusion network based on a two-dimensional high-resolution heat map for two-dimensional hand gesture estimation; training the multi-scale feature fusion network; and feeding a hand single-color image to be estimated into the multi-scale feature fusion network after training to obtain a predicted two-dimensional hand gesture of a hand joint. The method provides more accurate and rich relative position information between hand joint points in the adjacent area by using a two-dimensional high-resolution heat map, and fuses hand gesture features of different scales, so that the detection accuracy of the two-dimensional hand joint points can be effectively improved, and the estimation effect and performance level of the two-dimensional hand gesture are improved.

Description

Technical Field

[0001] The present invention relates to the field of image detection technology, and in particular to a two-dimensional hand posture estimation method and system based on a multi-scale feature fusion network. Background Art

[0002] Hands play a crucial role in our daily lives. They are one of the most frequently used parts of the body, an important way for humans to communicate with the real world non-verbally, and a key role in human-computer interaction. Therefore, hand pose estimation technology is a key step in achieving effective interaction and communication between robots and humans. At the two-dimensional level, single color images suffer from occlusion between the hand and other objects, as well as self-occlusion. Many joints are invisible, which is the most common challenge currently faced by 2D hand pose estimation research. However, most existing deep learning-based 2D hand pose estimation methods for single color images use single-scale feature maps or low-resolution heat maps to extract joint position information, which limits the accuracy of joint detection. Summary of the Invention

[0003] In response to the above problems, the present invention provides a two-dimensional hand posture estimation method and system based on a multi-scale feature fusion network. By utilizing a multi-scale feature fusion network based on a two-dimensional high-resolution heat map, it can focus on both global and local information, and thus extract more feature information of hand joints from the image, thereby improving the detection accuracy of joints.

[0004] A first aspect of the present invention provides a two-dimensional hand posture estimation method based on a multi-scale feature fusion network, the method comprising the following steps:

[0005] Collect original single color images of the hand to form a data set, and preprocess the real three-dimensional coordinate annotations of the hand joints in the data set;

[0006] The real three-dimensional coordinate annotations of the hand joints in the pre-processed data set are divided into a training set, a validation set and a test set according to a set ratio, and the training set is subjected to data expansion processing;

[0007] A multi-scale feature fusion network based on a two-dimensional high-resolution heat map is constructed, the multi-scale feature fusion network comprising a feature extractor, a first multi-scale feature fusion module and a second multi-scale feature fusion module, the feature extractor being configured to perform feature extraction on an input hand single-color image to obtain a feature signal and input the feature signal to the first multi-scale feature fusion module, the first multi-scale feature fusion module being configured to perform different scale feature sampling and fusion processing on the feature signal to obtain a first high-resolution heat map and input the first high-resolution heat map to the second multi-scale feature fusion module, the second multi-scale feature fusion module being configured to perform different scale feature sampling and fusion processing on the first high-resolution heat map to obtain a second high-resolution heat map, and the second high-resolution heat map being subjected to convolution processing to obtain a final two-dimensional hand posture.

[0008] The multi-scale feature fusion network is trained, and the training set after data augmentation processing is sent into the multi-scale feature fusion network in batches, in the training process, the first multi-scale feature fusion module performs loss calculation on the obtained first high-resolution heat map and an artificially generated hand joint real heat map, the second multi-scale feature fusion module performs loss calculation on the obtained second high-resolution heat map and the artificially generated hand joint real heat map, and the multi-scale feature fusion network parameters are optimized according to the double loss calculation results.

[0009] The hand single-color image to be estimated is sent into the multi-scale feature fusion network after training to obtain a two-dimensional hand posture of a hand joint predicted by the multi-scale feature fusion network.

[0010] A further technical solution of the present application is that the real three-dimensional coordinate labels of hand joints in the data set are preprocessed, specifically including:

[0011] The real three-dimensional coordinate labels of hand joints in the camera coordinate system are converted to the image coordinate system according to the pinhole imaging principle;

[0012] The camera intrinsic parameter matrix in the data set is obtained;

[0013] The real three-dimensional coordinate labels of hand joints in the data set are converted to real two-dimensional coordinates according to the conversion function relationship from the image coordinate system to the pixel coordinate system and the camera intrinsic parameter matrix.

[0014] A further technical solution of the present application is that the training set is subjected to data augmentation processing, specifically including:

[0015] The training set is taken as a green screen as a picture background to obtain a hand training sample;

[0016] Three additional sample versions are provided for the images in the hand training sample by replacing the picture background, changing the picture brightness and changing the picture contrast, obtaining the data expanded training sample.

[0017] A further technical solution of the present application is that the feature extractor includes a convolution layer, a batch normalization layer and two consecutive residual modules, the feature extractor processes the input original hand single color image into a size of 256x256, then reduces the resolution to 128x128 after filtering by a convolution kernel of 7x7x64, and then uses two consecutive residual modules to amplify the channel number to 256 to obtain a high channel number feature signal of 128x128x256, and inputs the high channel number feature signal into the first multi-scale feature fusion module.

[0018] A further technical solution of the present application is that the first multi-scale feature fusion module divides the input high channel number feature signal of 128x128x256 into two paths for semantic information extraction before downsampling, one path extracts the top layer features of the high channel number feature signal through a residual module, and the other path obtains features F k ,,k∈(1,2,3,4), after the feature map reaches the lowest resolution of 8x8, the features are extracted through four residual modules, and then four nearest neighbor upsampling operations are used to fuse the top layer features extracted through the residual module, and finally the first high resolution heat map of 128x128x256 is output.

[0019] A further technical solution of the present application is that the second multi-scale feature fusion module repeatedly performs different scale feature sampling and fusion processing on the input first high resolution heat map of 128x128x256 to obtain the second high resolution heat map of 128x128x256, and then reduces the channel number to 21 through two layers of 1x1 convolution to generate the final two-dimensional hand pose.

[0020] The second aspect of the present application is a two-dimensional hand pose estimation system based on a multi-scale feature fusion network, comprising:

[0021] A data preprocessing unit is configured to collect original hand single color images to form a data set, and to preprocess the real three-dimensional coordinate labels of hand joints in the data set;

[0022] A data expansion unit is configured to divide the real three-dimensional coordinate labels of hand joints in the preprocessed data set into a training set, a validation set and a test set according to a set proportion, and to perform data expansion processing on the training set;

[0023] a network construction unit, configured to construct a multi-scale feature fusion network based on a two-dimensional high-resolution heat map, the multi-scale feature fusion network comprising a feature extractor, a first multi-scale feature fusion module and a second multi-scale feature fusion module, the feature extractor being configured to perform feature extraction on an input hand single-color image to obtain a feature signal and input the feature signal into the first multi-scale feature fusion module, the first multi-scale feature fusion module being configured to perform different scale feature sampling and fusion processing on the feature signal to obtain a first high-resolution heat map and input the first high-resolution heat map into the second multi-scale feature fusion module, the second multi-scale feature fusion module being configured to perform different scale feature sampling and fusion processing on the first high-resolution heat map to obtain a second high-resolution heat map, and perform convolution processing on the second high-resolution heat map to obtain a final two-dimensional hand posture;

[0024] a network training unit, configured to train the multi-scale feature fusion network, and send the training set after data augmentation processing into the multi-scale feature fusion network in batches, in the training process, the first multi-scale feature fusion module is configured to perform loss calculation on the obtained first high-resolution heat map and an artificially generated hand joint real heat map, the second multi-scale feature fusion module is configured to perform loss calculation on the obtained second high-resolution heat map and the artificially generated hand joint real heat map, and the multi-scale feature fusion network parameters are optimized according to the double loss calculation results in a reverse propagation manner;

[0025] a two-dimensional hand posture estimation result generation unit, configured to send a hand single-color image to be estimated into the multi-scale feature fusion network after training, and obtain a two-dimensional hand posture of a hand joint predicted by the multi-scale feature fusion network.

[0026] Further technical solutions of the present application are as follows: the data preprocessing unit is configured to preprocess the real three-dimensional coordinate labels of hand joints in the data set, specifically including:

[0027] the real three-dimensional coordinate labels of hand joints in the camera coordinate system are converted to the image coordinate system according to the pinhole imaging principle;

[0028] an intracamera parameter matrix in the data set is obtained;

[0029] the real three-dimensional coordinate labels of hand joints in the data set are converted into real two-dimensional coordinates according to a conversion function relationship from the image coordinate system to the pixel coordinate system and the intracamera parameter matrix.

[0030] Further technical solutions of the present application are as follows: the data augmentation unit is configured to perform data augmentation processing on the training set, specifically including:

[0031] hand training samples are obtained by taking the green screen as a picture background of the training set;

[0032] Three additional sample versions are provided for the images in the hand training sample by replacing the picture background, changing the picture brightness and changing the picture contrast, obtaining the data expanded training sample.

[0033] A further technical solution of the present application is that the feature extractor includes a convolution layer, a batch normalization layer and two consecutive residual modules, the feature extractor processes the input original hand single color image into 256*256 size, then filters it through a 7*7*64 convolution kernel to reduce the resolution to 128*128, then uses two consecutive residual modules to amplify the channel number to 256 to obtain a high channel number feature signal of 128*128*256, and inputs the first multi-scale feature fusion module.

[0034] The two-dimensional hand posture estimation method and system based on the multi-scale feature fusion network provided by the present application effectively improves the detection ability of the neural network for the hand joint points by combining the integration of high-resolution heat maps and different scale context information in the network. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is the two-dimensional hand posture estimation method flowchart based on the multi-scale feature fusion network in the embodiment one of the present application;

[0036] Figure 2 is the multi-scale feature fusion network structure based on the two-dimensional high-resolution heat map in the embodiment one of the present application;

[0037] Figure 3 is the multi-scale feature fusion network implementation flowchart based on the two-dimensional high-resolution heat map in the embodiment one of the present application;

[0038] Figure 4 is the two-dimensional hand posture estimation system structure based on the multi-scale feature fusion network in the embodiment two of the present application. DETAILED DESCRIPTION

[0039] The present application will be further described in detail below in combination with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, in order to facilitate the description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0040] Before any embodiments are described in further detail, it should be noted that some embodiments are described as processes or methods, although other embodiments can also be implemented as appropriate software code or instructions. It should also be noted that some of the embodiments described herein can be implemented as a system, method, or computer program product. The embodiments can be implemented using a programmed computer or processor. The computer or processor can be programmed with software code or instructions to implement some or all of the steps of the embodiments described herein.

[0041] In addition, the terms "first", "second", and the like, can be used herein to describe various directions, actions, steps or elements, etc. but such directions, actions, steps or elements are not limited by such terms. Such terms are only used to distinguish one direction, action, step or element from another direction, action, step or element. For example, a first multi-scale feature fusion module can be called a second multi-scale feature fusion module without departing from the scope of the present application, and similarly, a second multi-scale feature fusion module can be called a first multi-scale feature fusion module. Both the first multi-scale feature fusion module and the second multi-scale feature fusion module are speed difference values, but they are not the same speed difference value. The terms "first", "second", and the like, cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0042] Embodiments of the present application are directed to a two-dimensional hand pose estimation method and system based on a multi-scale feature fusion network, which provides the following embodiments:

[0043] Based on the first embodiment of the present application

[0044] The present embodiment is used to illustrate a two-dimensional hand pose estimation method based on a multi-scale feature fusion network. The embodiment is implemented under the mainstream Pytorch deep learning framework, the Pytorch version is 1.6.0, the program development language is the mainstream Python3 in the current deep learning field, the operating system uses 64-bit Ubuntu18.04, the CPU of the development environment uses 64-bit Intel Xeon E5-2620, the GPU uses TeslaV100 with 32G video memory, and the memory size is 128G, which meets the training requirements of the multi-scale feature fusion network of the present embodiment. Referring to Figure 1 The present embodiment discloses a two-dimensional hand pose estimation method based on a multi-scale feature fusion network, comprising the following steps:

[0045] S110, collect original hand single-color images to form a data set, and pre-process the real three-dimensional coordinate labels of the hand joints in the data set;

[0046] Preferably, the pre-processing of the real three-dimensional coordinate labels of the hand joints in the data set specifically includes: converting the real three-dimensional coordinate labels of the hand joints in the camera coordinate system to the image coordinate system according to the pinhole imaging principle; obtaining the camera intrinsic parameter matrix in the data set; and converting the real three-dimensional coordinate labels of the hand joints in the data set to real two-dimensional coordinates according to the conversion function relationship from the image coordinate system to the pixel coordinate system and the camera intrinsic parameter matrix.

[0047] In the implementation process, the original hand single-color images are collected, and the real three-dimensional coordinate labels of the hand joints provided in the data set are pre-processed. The collected hand single-color images are derived from the FreiHAND data set released in 2019, which contains a large number of hand and object partial occlusion, hand and object full occlusion, hand partial self-occlusion, hand full self-occlusion, mixed occlusion and other rich and comprehensive occlusion posture samples, covering most cases from easy to difficult in the hand occlusion scene. The FreiHAND data set is a real hand data set, which is more conducive to improving the generalization ability of the model. Most of the previous methods use synthetic virtual hand data sets, which makes the performance very poor when they are applied to real scenes. The data set contains 130,240 training samples and 3,960 test samples. Each training sample includes a color image with a resolution of 224x224, a three-dimensional coordinate label of 21 hand joints with a camera intrinsic parameter matrix, and a hand segmentation mask. The pre-processing process of the real three-dimensional coordinate labels of the hand joints provided in the data set includes: first, converting the real three-dimensional coordinate labels of the hand joints in the camera coordinate system to the image coordinate system according to the pinhole imaging principle; then obtaining the camera intrinsic parameter matrix in the FreiHAND data set; finally, converting the real three-dimensional coordinate labels of the hand joints in the data set to real two-dimensional coordinates according to the conversion function relationship from the image coordinate system to the pixel coordinate system and the camera intrinsic parameter matrix, thereby completing the pre-processing task of the coordinate data.

[0048] S120, dividing the real three-dimensional coordinate labels of the hand joints in the pre-processed data set into a training set, a validation set and a test set according to a set proportion, and performing data augmentation processing on the training set;

[0049] Preferably, the training set is subjected to data augmentation processing, specifically including: taking the training set as a picture background with a green screen to obtain hand training samples, and providing three additional sample versions for the images in the hand training samples by replacing the picture background, changing the picture brightness, and changing the picture contrast to provide three additional sample versions for the images in the hand training samples, to obtain the data augmented training samples.

[0050] In the implementation process, the pre-processed original single color image and the real three-dimensional coordinate label of the hand joint provided in the data set are divided into a training set, a validation set and a test set according to a set proportion, and 32560 real hand training samples with a green screen as a background in the training set are processed by replacing the picture background, changing the picture brightness and contrast, etc. After the post-processing strategy, each of the training samples with a green screen as a background is provided with three additional sample versions, so that a total of 130240 training samples are obtained through the data expansion processing.

[0051] S130, a multi-scale feature fusion network based on a two-dimensional high-resolution heat map is constructed, the multi-scale feature fusion network includes a feature extractor, a first multi-scale feature fusion module and a second multi-scale feature fusion module, the feature extractor extracts features from the input hand single color image to obtain a feature signal and inputs the feature signal into the first multi-scale feature fusion module, the first multi-scale feature fusion module performs different scale feature sampling and fusion processing on the feature signal to obtain a first high-resolution heat map and inputs the first high-resolution heat map into the second multi-scale feature fusion module, the second multi-scale feature fusion module performs different scale feature sampling and fusion processing on the first high-resolution heat map to obtain a second high-resolution heat map, and the second high-resolution heat map is subjected to convolution processing to obtain a final two-dimensional hand pose.

[0052] Preferably, the feature extractor includes a convolution layer, a batch normalization layer and two consecutive residual modules, the feature extractor processes the input original hand single color image to a size of 256x256, then reduces the resolution to 128x128 after filtering by a 7x7x64 convolution kernel, and then uses two consecutive residual modules to amplify the channel number to 256 to obtain a high-channel number feature signal of 128x128x256, and inputs the high-channel number feature signal into the first multi-scale feature fusion module.

[0053] Preferably, the first multi-scale feature fusion module divides the input high-channel number feature signal of 128x128x256 into two paths for semantic information extraction before downsampling, one of which extracts the top features of the high-channel number feature signal through a residual module; the other path obtains features F kk∈(1,2,3,4), after the feature map reaches the lowest resolution of 8x8, the feature is extracted through four residual modules, and then four nearest neighbor upsampling is used to fuse with the top layer feature extracted by the residual module, and finally output the first high resolution heat map of 128x128x256.

[0054] Preferably, the second multi-scale feature fusion module repeatedly performs different scale feature sampling and fusion processing on the input first high resolution heat map of 128x128x256 to obtain the second high resolution heat map of 128x128x256, and then reduces the channel number to 21 through two 1x1 convolution to generate the final two-dimensional hand pose.

[0055] In the specific implementation process, a multi-scale feature fusion network based on a two-dimensional high resolution heat map is constructed, and the network structure is as shown in Figure 2 As shown in the figure. First, in the feature extractor, the input structure without the maximum pooling layer is designed to prevent further loss of feature information caused by compression of the heat map, so that the resolution of the feature map input to the multi-scale feature fusion module and the hand joint heat map output by the hand joint heat map are all 128x128. The feature map size after four-level downsampling is doubled, which is 64x64, 32x32, 16x16, and 8x8 respectively. In addition, the feature map size after upsampling corresponding to each level of downsampling layer is also doubled, so that a high-quality high-resolution heat map can be generated.

[0056] Secondly, for the overall structure design of the network, the feature extractor resizes the original color image to a size of 256x256, and then filters it through a 7x7x64 convolution kernel to reduce the resolution to 128x128 to reduce the calculation cost. Then, two consecutive residual modules are designed to expand the channel number to 256 and input into the first multi-scale feature fusion module. The high channel number feature will be divided into two paths before each downsampling to extract higher level semantic information. One path is obtained through four maximum pooling operations and residual modules to obtain features F k k∈(1,2,3,4). After the feature map reaches the lowest resolution of 8x8, the network starts to extract features through four residual modules, and then four nearest neighbor upsampling is used to realize the fusion operation with the top layer feature extracted by the residual network. The last feature fusion layer of the second multi-scale feature fusion module outputs a high resolution heat map of 128x128x256, and then reduces the channel number to 21 through two 1x1 convolution. The network further predicts the final two-dimensional hand pose.

[0057] It should be noted that the output structure of the network is designed, and due to the characteristics of the hand pose estimation task, in the last layer 1x1 convolution, the original 16 channels used for human pose estimation are designed to 21 channels to meet the actual task requirements of hand pose estimation.

[0058] S140, training the multi-scale feature fusion network, the training set after data augmentation processing is sent into the multi-scale feature fusion network in batches, the multi-scale feature fusion network further includes an intermediate supervision module, in the training process, the first multi-scale feature fusion module inputs the obtained first high-resolution heat map into the intermediate supervision module and the artificial generated hand joint real heat map for loss calculation, the second multi-scale feature fusion module obtains the second high-resolution heat map and the artificial generated hand joint real heat map for loss calculation, and the multi-scale feature fusion network parameters are optimized according to the double loss calculation results and back propagation;

[0059] Further, the first multi-scale feature fusion module inputs the obtained first high-resolution heat map into the intermediate supervision module and the artificial generated hand joint real heat map for loss calculation, the second multi-scale feature fusion module obtains the second high-resolution heat map and the artificial generated hand joint real heat map for loss calculation, the total loss is obtained by adding the two parts of loss, and the multi-scale feature fusion network parameter is updated through RMSprop (Root Mean Square Prop) and the total loss back propagation, wherein the batch size is 4, the training period is 200 epochs, and the learning rate is 0.001.

[0060] In the specific implementation process, the multi-scale feature fusion network based on the two-dimensional high-resolution heat map is trained, the training set after data division is sent into the multi-scale feature fusion network in batches, and the heat map of each hand joint i is predicted through the cascade of the encoder-decoder Where HxW is the resolution of each heat map. Each hand joint corresponds to a heat map, and each pixel value on the heat map represents the confidence of the two-dimensional pixel position being a hand joint. The position with the highest confidence is the two-dimensional coordinate of the predicted hand joint. That is, the heat map predicted by the network is obtained, and the loss is calculated with the artificial generated hand joint real heat map. Different from the traditional detection network which only supervises the loss function designed at the end of the network, the heat map H mThe two-dimensional label is converted from the real three-dimensional coordinate label of the hand joint provided in the FreiHAND dataset through a coordinate conversion function relationship of graphics.

[0061] S150, inputting the hand single color image to be estimated into the multi-scale feature fusion network after training, and obtaining the two-dimensional hand posture of the hand joint predicted by the multi-scale feature fusion network.

[0062] In the specific implementation process, the hand single color image to be estimated is input into the network after training, and the network process is shown in Figure 3 After receiving the single color image data, the network outputs 21 hand joint heat maps s with a resolution of 128*128 through 1 convolution, 4 maximum pooling, 4 upsampling and 4 fusion operations, wherein each heat map represents two-dimensional coordinate information of a hand joint. The heat map is mapped to a feature space with different resolutions through maximum pooling, and different scale feature information is added and fused, so as to fully and comprehensively extract the mutual dependence information of the hand joint from the local to the whole, and finally obtain 21 high-resolution heat maps predicted by the network, thereby improving the detection effect of the joint.

[0063] Based on the embodiment 2 of the present application

[0064] The two-dimensional hand posture estimation system 400 based on the multi-scale feature fusion network provided in the embodiment 2 of the present application can execute the two-dimensional hand posture estimation method based on the multi-scale feature fusion network provided in the embodiment 1 of the present application, has the corresponding function modules and beneficial effects of the execution method, and the device can be realized by software and / or hardware (integrated circuit), and generally can be integrated in a server or a terminal device. Figure 4 is a structural schematic diagram of the two-dimensional hand posture estimation system 400 based on the multi-scale feature fusion network in the embodiment 2 of the present application. Referring to Figure 4 , the two-dimensional hand posture estimation system 400 based on the multi-scale feature fusion network of the embodiment of the present application can specifically include:

[0065] The data preprocessing unit 410 is used for collecting original hand single color images to form a data set, and pre-processing the real three-dimensional coordinate label of the hand joint in the data set;

[0066] The data expansion unit 420 is configured to divide the hand joint real three-dimensional coordinate labels in the preprocessed data set into a training set, a verification set and a test set according to a set proportion, and perform data expansion processing on the training set;

[0067] The network construction unit 430 is configured to construct a multi-scale feature fusion network based on a two-dimensional high-resolution heat map, the multi-scale feature fusion network comprising a feature extractor, a first multi-scale feature fusion module and a second multi-scale feature fusion module, the feature extractor being configured to perform feature extraction on an input hand single-color image to obtain a feature signal and input the feature signal into the first multi-scale feature fusion module, the first multi-scale feature fusion module being configured to perform different scale feature sampling and fusion processing on the feature signal to obtain a first high-resolution heat map and input the first high-resolution heat map into the second multi-scale feature fusion module, the second multi-scale feature fusion module being configured to perform different scale feature sampling and fusion processing on the first high-resolution heat map to obtain a second high-resolution heat map, and perform convolution processing on the second high-resolution heat map to obtain a final two-dimensional hand pose.

[0068] The network training unit 440 is configured to train the multi-scale feature fusion network, and input the training set after data expansion processing into the multi-scale feature fusion network in batches, in the training process, the first multi-scale feature fusion module performs loss calculation on the obtained first high-resolution heat map and an artificially generated hand joint real heat map, the second multi-scale feature fusion module performs loss calculation on the obtained second high-resolution heat map and the artificially generated hand joint real heat map, and the multi-scale feature fusion network parameters are optimized according to the double loss calculation results.

[0069] The two-dimensional hand pose estimation result generation unit 450 is configured to input a hand single-color image to be estimated into the trained multi-scale feature fusion network, and obtain a two-dimensional hand pose of a hand joint predicted by the multi-scale feature fusion network.

[0070] Preferably, the data preprocessing unit 410 is configured to preprocess the hand joint real three-dimensional coordinate labels in the data set, specifically including:

[0071] According to the pinhole imaging principle, the hand joint real three-dimensional coordinate labels in the camera coordinate system are converted into the image coordinate system;

[0072] The camera intrinsic parameter matrix in the data set is obtained;

[0073] According to the conversion function relationship from the image coordinate system to the pixel coordinate system and the camera intrinsic parameter matrix, the hand joint real three-dimensional coordinate labels in the data set are converted into real two-dimensional coordinates.

[0074] Preferably, the data augmentation unit 420 performs data augmentation processing on the training set, specifically including:

[0075] The training set is taken as a green screen as a picture background to obtain a hand training sample;

[0076] The image in the hand training sample is replaced with a picture background, the picture brightness is changed, and the picture contrast is changed to provide three additional sample versions for the image in the hand training sample, obtaining a data-augmented training sample.

[0077] Preferably, the feature extractor includes a convolutional layer, a batch normalization layer, and two consecutive residual modules. The feature extractor processes the input original hand single-color image into a size of 256x256, and then reduces the resolution to 128x128 after filtering with a 7x7x64 convolutional kernel. Then, the two consecutive residual modules are used to amplify the channel number to 256 to obtain a high-channel feature signal of 128x128x256, and input into the first multi-scale feature fusion module.

[0078] In addition to the above-mentioned units, the two-dimensional hand pose estimation system 400 based on the multi-scale feature fusion network of embodiment 2 can also include other components, however, since these components are irrelevant to the content of the present disclosure, their illustrations and descriptions are omitted here.

[0079] The specific working process of the two-dimensional hand pose estimation system 400 based on the multi-scale feature fusion network is described above with reference to the description of the two-dimensional hand pose estimation method based on the multi-scale feature fusion network of embodiment 1, and will not be repeated here.

[0080] In summary, through the embodiments, it can be seen that the two-dimensional hand pose estimation method and system based on the multi-scale feature fusion network provided by the present application effectively improves the detection ability of the neural network for hand joint points by combining high-resolution heat maps and the integration of different scale context information in the network. Its beneficial effects are: the use of two-dimensional high-resolution heat maps provides more accurate and rich relative position information between hand joint points in the adjacent area, and the fusion of hand pose features of different scales can effectively improve the detection accuracy of two-dimensional hand joint points and improve the estimation effect and performance level of the current two-dimensional hand pose.

[0081] Note that the above merely describes preferred embodiments of the present application and the principles of the technology applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, modifications and substitutions can be made without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A two-dimensional hand gesture estimation method based on a multi-scale feature fusion network, characterized in that, The method comprises the following steps: Collecting original hand single-color image groups to form a data set, and pre-processing the real three-dimensional coordinate labels of the hand joint points in the data set; Dividing the real three-dimensional coordinate labels of the hand joint points in the pre-processed data set into a training set, a validation set and a test set according to a set proportion, and performing data augmentation processing on the training set; Constructing a multi-scale feature fusion network based on a two-dimensional high-resolution heat map, wherein the multi-scale feature fusion network comprises a feature extractor, a first multi-scale feature fusion module and a second multi-scale feature fusion module, the feature extractor extracts features from an input hand single-color image to obtain feature signals and inputs the feature signals into the first multi-scale feature fusion module, the first multi-scale feature fusion module performs different scale feature sampling and fusion processing on the feature signals to obtain a first high-resolution heat map and inputs the first high-resolution heat map into the second multi-scale feature fusion module, the second multi-scale feature fusion module performs different scale feature sampling and fusion processing on the first high-resolution heat map to obtain a second high-resolution heat map, and the second high-resolution heat map is subjected to convolution processing to obtain a final two-dimensional hand posture; Training the multi-scale feature fusion network, feeding the data augmentation processed training set into the multi-scale feature fusion network in batches, in the training process, the first multi-scale feature fusion module performs loss calculation on the obtained first high-resolution heat map and an artificially generated real heat map of the hand joint points, the second multi-scale feature fusion module performs loss calculation on the obtained second high-resolution heat map and the artificially generated real heat map of the hand joint points, and the multi-scale feature fusion network parameters are optimized according to the double loss calculation results and back propagation; Feeding a hand single-color image to be estimated into the trained multi-scale feature fusion network to obtain a two-dimensional hand posture of the hand joint points predicted by the multi-scale feature fusion network; The first multi-scale feature fusion module divides the input high-channel-number feature signal of 128x128x256 into two paths for semantic information extraction before downsampling, one path extracts the top layer feature of the high-channel-number feature signal through a residual module, and the other path obtains features F of different scale spaces through four maximum pooling operations and a residual module k After the feature map reaches the lowest resolution of 8x8, the features are extracted through four residual modules, and then four nearest neighbor upsampling is used to fuse the top layer features extracted through the residual modules, and finally the first high-resolution heat map of 128x128x256 is output. The second multi-scale feature fusion module repeatedly performs the different scale feature sampling and fusion processing of the first multi-scale feature fusion module on the input first high-resolution heat map of 128x128x256 to obtain the second high-resolution heat map of 128x128x256, reduces the channel number of the second high-resolution heat map to 21 through two 1x1 convolution, and generates a final two-dimensional hand posture.

2. The two-dimensional hand pose estimation method based on the multi-scale feature fusion network according to claim 1, characterized in that, The pre-processing of the real three-dimensional coordinate labels of the hand joint points in the data set comprises: Converting the real three-dimensional coordinate labels of the hand joint points in the camera coordinate system to the image coordinate system according to the pinhole imaging principle; Obtaining the camera intrinsic parameter matrix in the data set; Converting the real three-dimensional coordinate labels of the hand joint points in the data set to real two-dimensional coordinates according to the conversion function relationship from the image coordinate system to the pixel coordinate system and the camera intrinsic parameter matrix.

3. The two-dimensional hand pose estimation method based on the multi-scale feature fusion network according to claim 1, characterized in that, The data augmentation processing of the training set comprises: Obtaining hand training samples by taking the green screen as the picture background of the training set; Three additional sample versions are provided for the images in the hand training sample by replacing the picture background, changing the picture brightness and changing the picture contrast, obtaining the data expanded training sample.

4. The two-dimensional hand pose estimation method based on the multi-scale feature fusion network according to claim 1, characterized in that, The feature extractor includes a convolution layer, a batch normalization layer and two consecutive residual modules, the feature extractor processes the input original hand single color image into 256*256 size, and then filters through a 7*7*64 convolution kernel to reduce the resolution to 128*128, and then uses two consecutive residual modules to amplify the channel number to 256 to obtain a high channel number feature signal of 128*128*256, and inputs the first multi-scale feature fusion module.

5. A two-dimensional hand pose estimation system based on a multi-scale feature fusion network, characterized in that, Comprise: A data preprocessing unit is configured to collect original hand single color images to form a data set, and preprocess the real three-dimensional coordinate labels of hand joints in the data set; A data expansion unit is configured to divide the real three-dimensional coordinate labels of hand joints in the preprocessed data set into a training set, a validation set and a test set according to a set proportion, and perform data expansion processing on the training set; A network construction unit is configured to construct a multi-scale feature fusion network based on a two-dimensional high-resolution heat map, the multi-scale feature fusion network comprising a feature extractor, a first multi-scale feature fusion module and a second multi-scale feature fusion module, the feature extractor being configured to extract features from an input hand single color image to obtain a feature signal and input the feature signal into the first multi-scale feature fusion module, the first multi-scale feature fusion module being configured to sample and fuse different scale features of the feature signal to obtain a first high-resolution heat map and input the first high-resolution heat map into the second multi-scale feature fusion module, the second multi-scale feature fusion module being configured to sample and fuse different scale features of the first high-resolution heat map to obtain a second high-resolution heat map, and perform convolution processing on the second high-resolution heat map to obtain a final two-dimensional hand pose; A network training unit is configured to train the multi-scale feature fusion network, and send the training set after data expansion processing into the multi-scale feature fusion network in batches, in the training process, the first multi-scale feature fusion module performs loss calculation on the obtained first high-resolution heat map and the artificially generated real heat map of hand joints, the second multi-scale feature fusion module performs loss calculation on the obtained second high-resolution heat map and the artificially generated real heat map of hand joints, and the multi-scale feature fusion network parameters are optimized according to the double loss calculation results; A two-dimensional hand pose estimation result generation unit is configured to input a hand single color image to be estimated into the trained multi-scale feature fusion network to obtain a two-dimensional hand pose of hand joints predicted by the multi-scale feature fusion network. The first multi-scale feature fusion module divides the input high-channel-number feature signal of 128x128x256 into two paths for semantic information extraction before downsampling, one path extracts the top layer feature of the high-channel-number feature signal through a residual module, and the other path obtains features F of different scale spaces through four maximum pooling operations and a residual module k After the feature map reaches the lowest resolution of 8x8, the features are extracted through four residual modules, and then four nearest neighbor upsampling is used to fuse the top layer features extracted through the residual module, and finally the first high-resolution heat map of 128x128x256 is output. The second multi-scale feature fusion module repeatedly performs different scale feature sampling and fusion processing on the input first high-resolution heat map of 128x128x256 to obtain the second high-resolution heat map of 128x128x256, reduces the channel number to 21 through two layers of 1x1 convolution on the second high-resolution heat map, and generates a final two-dimensional hand pose.

6. The two-dimensional hand pose estimation system based on a multi-scale feature fusion network of claim 5, wherein, The data preprocessing unit pre-processes the real three-dimensional coordinate labels of the hand joint points in the data set, and specifically includes: According to the pinhole imaging principle, the real three-dimensional coordinate labels of the hand joint points in the camera coordinate system are converted to the image coordinate system; Obtain the camera intrinsic parameter matrix in the data set; According to the conversion function relationship from the image coordinate system to the pixel coordinate system and the camera intrinsic parameter matrix, the real three-dimensional coordinate labels of the hand joint points in the data set are converted to real two-dimensional coordinates.

7. The two-dimensional hand pose estimation system based on a multi-scale feature fusion network of claim 5, wherein, The data augmentation unit performs data augmentation processing on the training set, specifically including: The training set is taken as a green screen picture background to obtain a hand training sample; The image in the hand training sample is replaced with a picture background, the picture brightness is changed, and the picture contrast is changed to provide three additional sample versions for the image in the hand training sample, and obtain the data augmented training sample.

8. The two-dimensional hand pose estimation system based on a multi-scale feature fusion network of claim 5, wherein, The feature extractor includes a convolution layer, a batch normalization layer and two consecutive residual modules. The feature extractor processes the input original hand single color image into a size of 256x256, and then reduces the resolution to 128x128 after filtering by a 7x7x64 convolution kernel. Then, the two consecutive residual modules are used to enlarge the channel number to 256 to obtain a high channel number feature signal of 128x128x256, and input the first multi-scale feature fusion module.

Images