Hand image key point position acquisition method and device and computer equipment

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a hand image key point position acquisition method, device and computer equipment. The method comprises the following steps: inputting a target hand image into a semantic segmentation neural network to obtain a semantic segmentation image; positioning a first and a second rectangular frame position of left and right hand images in the target hand image; judging whether the left and right hand images overlap; if yes, setting pixel values of the target hand image except the left and right hand images to 0; respectively cutting images in the first and the second rectangular frame to obtain a first and a second screenshot; preprocessing the first and the second screenshot to generate a first and a second image; inputting the first and the second image into a pre-trained key point regression neural network to generate a first and a second heat map set; obtaining a key point position of the left hand image based on the first heat map set, and obtaining a key point position of the right hand image based on the second heat map set. The method, device and computer equipment can improve the accuracy of hand information recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular, the present application relates to a hand image key point position acquisition method, device and computer equipment. BACKGROUND

[0002] The main technical problems in the field of hand information recognition at present are that the object tracking model cannot simultaneously identify two objects with very high overlap, for example, when clapping hands, the hands have more than 80% overlapping area, at this time the object tracking network can usually only identify the hand that is not blocked in a specific angle, while the blocked hand cannot be identified. In addition, the neural network has great difficulty in identifying and distinguishing highly similar objects, and the left and right hands of a person are almost completely the same in terms of skin color, texture, geometric shape and other characteristics. Therefore, it is very difficult to distinguish left and right hands through ordinary neural networks when they are highly overlapped. At present, there is an urgent need for a new technical solution to solve the technical problem that hand information recognition cannot be accurately completed under the condition of overlapping hands, so that the hand information recognition technology can meet the needs of more application scenarios. SUMMARY

[0003] The main purpose of the present application is to provide a hand image key point position acquisition method, device and computer equipment, which aims to solve the technical problem of low accuracy of hand information recognition in the prior art under the condition of overlapping hands.

[0004] In order to achieve the above-mentioned purpose of the application, the present application provides a hand image key point position acquisition method, which comprises:

[0005] inputting a target hand image into a pre-trained semantic segmentation neural network to obtain a semantic segmentation graph corresponding to the target hand image, wherein the target hand image includes a left hand image and a right hand image;

[0006] locating the first rectangular frame position and the second rectangular frame position of the left hand image and the right hand image in the hand image according to the semantic segmentation graph;

[0007] determining whether the left hand image and the right hand image overlap;

[0008] if the left hand image and the right hand image overlap, setting the pixel value of the image part of the target hand image except the left hand image to 0 based on the semantic segmentation graph to obtain a first optimized target hand image, and setting the pixel value corresponding to the image part of the target hand image except the right hand image to 0 to obtain a second optimized target hand image;

[0009] obtaining a first screenshot by cropping the image within the first rectangular frame in the first optimization target hand image, and obtaining a second screenshot by cropping the image within the second rectangular frame in the second optimization target hand image;

[0010] preprocessing the first screenshot and the second screenshot to generate a first image and a second image;

[0011] inputting the first image and the second image into a pre-trained key point regression neural network to generate a first heat map set corresponding to the first image and a second heat map set corresponding to the second image, wherein each of the first heat map set and the second heat map set includes at least one heat map, and each of the heat maps is a Gaussian heat map with sub-pixel information;

[0012] obtaining the key point position of the left hand image based on the first heat map set, and obtaining the key point position of the right hand image based on the second heat map set.

[0013] In some embodiments, the step of obtaining the semantic segmentation map corresponding to the hand image based on the pre-trained semantic segmentation neural network includes:

[0014] inputting a target hand image with a first size into a pre-trained semantic segmentation neural network to obtain a first feature map with a second size, wherein the first feature map includes a first channel, a second channel and a third channel, and the first channel, the second channel and the third channel respectively represent the categories of a background image, a left hand image and a right hand image in the hand image, and the first size is greater than the second size;

[0015] adjusting the size of the first feature map to generate a second feature map with the first size;

[0016] obtaining the semantic segmentation map corresponding to the hand image based on the second feature map.

[0017] In some embodiments, the first heat map set includes a first position heat map and a first depth heat map, and the step of obtaining the key point position of the left hand image based on the first heat map set includes:

[0018] obtaining a maximum value position of the first position heat map;

[0019] extracting a first matrix from the first position heat map with the maximum value position of the first position heat map as the center;

[0020] normalizing the first matrix to obtain a first normalization result;

[0021] obtaining the position coordinates of the key point of the left hand image based on the first normalization result;

[0022] obtaining a maximum value position of the first depth heat map;

[0023] extracting a second matrix from the first depth heat map with the maximum value position of the first depth heat map as the center;

[0024] normalizing the second matrix to obtain a second normalization result;

[0025] obtaining the depth coordinates of the key points of the left hand image based on the second normalization result;

[0026] obtaining the key point positions of the left hand image based on the position coordinates and the depth coordinates of the left hand image.

[0027] In some embodiments, the second heat map set includes a second position heat map and a second depth heat map, and the step of obtaining the key point positions of the right hand image based on the second heat map set includes:

[0028] obtaining a maximum value position of the second position heat map;

[0029] extracting a third matrix from the second position heat map with the maximum value position of the second position heat map as the center;

[0030] normalizing the third matrix to obtain a third normalization result;

[0031] obtaining the position coordinates of the key points of the right hand image based on the third normalization result;

[0032] obtaining a maximum value position of the second depth heat map;

[0033] extracting a fourth matrix from the second depth heat map with the maximum value position of the second depth heat map as the center;

[0034] normalizing the fourth matrix to obtain a fourth normalization result;

[0035] obtaining the depth coordinates of the key points of the right hand image based on the fourth normalization result;

[0036] obtaining the key point positions of the right hand image based on the position coordinates and the depth coordinates of the key points of the right hand image.

[0037] In some embodiments, the step of preprocessing the first screenshot and the second screenshot to generate a first image and a second image includes:

[0038] transforming the sizes of the first screenshot and the second screenshot into a preset size respectively to generate a first image and a second image.

[0039] In an embodiment, a hand image key point position acquisition method comprises the following steps:

[0040] If the left hand image and the right hand image do not overlap, third and fourth rectangular frame positions of the left hand image and the right hand image in the hand image are located according to the semantic segmentation map;

[0041] Based on the semantic segmentation map, pixel values corresponding to image parts other than the left hand image and the right hand image in the target hand image are set to 0, and a third optimized target hand image is obtained;

[0042] Based on the third optimized target hand image, images within the third and fourth rectangular frames are respectively intercepted, and a third and a fourth cut image are obtained;

[0043] The third and fourth cut images are preprocessed to generate a third and a fourth image;

[0044] The third and fourth images are input into a pre-trained key point regression neural network to generate a third and a fourth heat map set corresponding to the third and fourth images, wherein the third and fourth heat map sets each include at least one heat map, and each heat map is a Gaussian heat map with sub-pixel information;

[0045] Based on the third heat map set, key point positions of the left hand image are obtained, and based on the fourth heat map set, key point positions of the right hand image are obtained.

[0046] In some embodiments, a training data set for training the semantic segmentation neural network and the key point regression neural network is obtained based on a double-hand overlapping image.

[0047] The present application also provides a hand image key point position acquisition device, comprising:

[0048] A semantic segmentation map acquisition module is configured to acquire a semantic segmentation map corresponding to a hand image based on a pre-trained semantic segmentation neural network, wherein the hand image includes a left hand image and a right hand image;

[0049] A positioning module is configured to locate first and second rectangular frame positions of the left hand image and the right hand image in the hand image according to the semantic segmentation map;

[0050] A judgment module is configured to judge whether the left hand image and the right hand image overlap according to the first and second rectangular frame positions;

[0051] A first execution module is configured to, when the left hand image and the right hand image overlap,

[0052] The semantic segmentation graph sets the pixel value of the image part of the target hand image except the left hand image to 0 to obtain a first optimized target hand image, and sets the pixel value corresponding to the image part of the target hand image except the right hand image to 0 to obtain a second optimized target hand image;

[0053] A screenshot module is configured to capture an image in the first rectangular frame in the first optimized target hand image to obtain a first screenshot, and capture an image in the second rectangular frame in the second optimized target hand image to obtain a second screenshot;

[0054] A preprocessing module is configured to pre-process the first screenshot and the second screenshot to generate a first image and a second image;

[0055] A heat map set acquisition module is configured to input the first image and the second image into a pre-trained key point regression neural network to generate a first heat map set corresponding to the first image and a second heat map set corresponding to the second image, wherein each of the first heat map set and the second heat map set includes at least one heat map, and each of the heat maps is a Gaussian heat map with sub-pixel information;

[0056] A key point position acquisition module is configured to acquire the key point position of the left hand image based on the first heat map set, and acquire the key point position of the right hand image based on the second heat map set.

[0057] The application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the hand image key point position acquisition method provided in any of the above embodiments when executing the computer program.

[0058] The application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program implements the steps of the hand image key point position acquisition method provided in any of the above embodiments when executed by a processor.

[0059] The hand image key point position acquisition method, device and computer device provided by the application can combine a semantic segmentation neural network and a key point regression neural network, remove the pixel information of a non-target hand in a target hand image by using the semantic information provided by the semantic segmentation neural network, reduce the noise generated by a highly similar object on neural network reasoning, and improve the accuracy of hand information recognition in the case of overlapping hands. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1A position diagram of hand image key points of an embodiment of the present application;

[0061] Figure 2 A flow diagram of a hand image key point position acquisition method of an embodiment of the present application;

[0062] Figure 3 A flow diagram of step S10 of the hand image key point position acquisition method of an embodiment of the present application;

[0063] Figure 4 A flow diagram of step S80 of the hand image key point position acquisition method of an embodiment of the present application;

[0064] Figure 5 A flow diagram of a hand image key point position acquisition method of another embodiment of the present application;

[0065] Figure 6 A structure diagram of a hand image key point position acquisition device of an embodiment of the present application;

[0066] Figure 7 A structure diagram of a computer device of an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be given to the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0068] In the prior art, the scheme of reconstructing a human hand model or performing gesture recognition by using a hand image is generally realized based on 21 key point positions of left / right palms. Please refer to Figure 1 The 21 key points include a key point 0 at the root of a palm and 4 key points 1-20 corresponding to each finger. However, since the existing object tracking model is difficult to identify two objects with very high overlap at the same time, for example, when two hands are clapped, the hand has more than 80% overlapping area, at this time, the object tracking model can usually only identify the image of a hand that is not blocked in a specific view, while the image of the other hand that is completely (or partially) blocked cannot be accurately identified. In addition, the existing ordinary neural network has great difficulty in identifying and distinguishing highly similar objects, but the skin color, texture, geometric shape and other characteristics of the left and right hands of a person are almost completely the same, so when they are highly overlapped, it is very difficult to distinguish the left / right hand images by using an ordinary neural network.

[0069] Based on this, please refer to Figure 2The embodiment of the present application provides a hand image key point position acquisition method, which comprises steps S10-S80, and the detailed description of each step of the method is as follows.

[0070] In one embodiment, the hand image key point position acquisition method comprises the following steps.

[0071] S10, input a target hand image into a pre-trained semantic segmentation neural network to obtain a semantic segmentation graph corresponding to the target hand image, wherein the target hand image comprises a left hand image and a right hand image.

[0072] S20, locate first and second rectangular frame positions of the left hand image and the right hand image in the hand image according to the semantic segmentation graph.

[0073] S30, determine whether the left hand image and the right hand image overlap according to the first and second rectangular frame positions.

[0074] S40, if the left hand image and the right hand image overlap, set pixel values of an image part except the left hand image in the target hand image to 0 based on the semantic segmentation graph to obtain a first optimized target hand image, and set pixel values corresponding to an image part except the right hand image in the target hand image to 0 to obtain a second optimized target hand image.

[0075] S50, cut the image in the first rectangular frame in the first optimized target hand image to obtain a first cut image, and cut the image in the second rectangular frame in the second optimized target hand image to obtain a second cut image.

[0076] S60, pre-process the first cut image and the second cut image to generate a first image and a second image.

[0077] S70, input the first image and the second image into a pre-trained key point regression neural network to generate a first heat map set corresponding to the first image and a second heat map set corresponding to the second image, wherein the first heat map set and the second heat map set each comprise at least one heat map, and each heat map is a Gaussian heat map with sub-pixel information.

[0078] S80, obtain key point positions of the left hand image based on the first heat map set, and obtain key point positions of the right hand image based on the second heat map set.

[0079] The target hand image described above can be obtained from an image acquisition device or an electronic device with image data storage function, which can be a smart phone, a tablet computer, a desktop computer, a laptop computer, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a PC device, a personal digital assistant (PDA), a routing device, a virtual reality device (VR / AR glasses), etc. The target hand image includes a left hand image and a right hand image, wherein the left hand image can be partially (or entirely) overlapped or completely separated. The target hand image is input into a pre-trained semantic segmentation neural network to obtain a semantic segmentation image corresponding to the target hand image. In this embodiment, the training sample set of the semantic segmentation neural network is constructed based on a large number of hand images with overlapped hands, so that the present application has good recognition effect for the input left and right hand images with overlapped target hand images, and the obtained hand image key point position is more accurate. It should be noted that the semantic segmentation neural network used in the present application can be an existing semantic segmentation model with good semantic segmentation effect for hand images, or an improved semantic segmentation model based on an existing semantic segmentation model, which is not limited herein.

[0080] As described in step S20 above, after obtaining the semantic segmentation image corresponding to the target hand image, the first rectangular frame position and the second rectangular frame position of the left hand image and the right hand image in the target hand image are located according to the semantic segmentation image. Generally, the target object in the target hand image can be found in the detection picture by an object tracking algorithm such as YOLO, Faster RCNN, etc., and displayed by a rectangular frame, so as to facilitate subsequent processing (such as classification) of the target object. In this embodiment, the target object described above is the left and right hand in the target hand image, and the detection picture is the target hand image.

[0081] As described in step S30 above, after locating the first rectangular frame position and the second rectangular frame position, whether the left hand image and the right hand image in the target hand image are overlapped can be determined according to the first rectangular frame position and the second rectangular frame position. Specifically, whether the left hand image and the right hand image in the target hand image are overlapped can be determined by judging whether there is an overlapping part between the image area in the first rectangular frame and the image area in the second rectangular frame. Whether the image areas are overlapped can be determined by the following formula:

[0082]

[0083] wherein, I oU is Intersection over Union, A is the image area in the first rectangular frame, and B is the image area in the second rectangular frame.

[0084] When I o When U is 0, it means that there is no overlap between the left hand image and the right hand image in the target hand image. o When U is not 0, it means that there is overlap between the left hand image and the right hand image in the target hand image.

[0085] As described in step S40 above, when the left and right hand images in the target hand image overlap, the pixel values of the image part other than the left hand image in one target hand image (there are multiple backups of the target hand image) are set to 0 based on the semantic segmentation map obtained in step S10 to obtain a first optimized target hand image; and the pixel values corresponding to the image part other than the right hand image in another target hand image are set to 0 to obtain a second optimized target hand image. By setting the pixel values of the image part other than the left hand image or the right hand image to 0, the pixel information of the non-target hand (left hand or right hand) in the target hand image can be removed, thereby reducing the noise generated by the highly similar object in the subsequent neural network inference, and improving the accuracy of hand information recognition (in this embodiment, the hand information is the key point position of the hand) in the case of left and right hand overlap.

[0086] As described in step S50 above, based on the position information (which can be expressed as pixel coordinates in the target hand image) of the first rectangular frame and the second rectangular frame, the image within the first rectangular frame in the first optimized target hand image is intercepted, and the image within the second rectangular frame in the second optimized target hand image is intercepted to obtain a first screenshot and a second screenshot, wherein the first screenshot and the second screenshot respectively contain the left hand image and the right hand image with the pixel information of the non-target hand removed.

[0087] As described in steps S60-S80 above, before inputting the first screenshot and the second screenshot into the pre-trained key point regression neural network, the first screenshot and the second screenshot are usually preprocessed for better regression effect or to save computing resources, etc. For example, in some embodiments, the size of the first screenshot and the second screenshot is transformed to a preset size (such as 128*128) to adapt to the pre-trained key point regression neural network, so as to obtain more accurate prediction results.

[0088] The key point regression neural network in the present application adopts a sub-pixel heat map based neural network model. The heat map regression model has the advantage of strong generalization ability, and can obtain better prediction results in prediction even if the training sample distribution is uneven during model training. Similarly, the heat map regression model adopted in the present application can be an existing heat map regression model with high precision (such as a heat map regression model combined with a heat map and an offset), or a key point heat map regression model improved based on an existing heat map regression model, which is not limited herein.

[0089] After inputting the first image and the second image obtained by preprocessing the first screenshot and the second screenshot into the pre-trained key point regression neural network, a first heat map set corresponding to the first image and a second heat map set corresponding to the second image are generated, wherein the first heat map set and the second heat map set each include at least one heat map, and each heat map is a Gaussian heat map with sub-pixel information; the key point positions of the left hand image in the target hand image are obtained based on the first heat map set, and the key point positions of the right hand image in the target hand image are obtained based on the second heat map set. The key point positions, i.e. the pixel coordinates of the key points in the target hand image, can be obtained through the above steps S60-S80. By repeating the above steps S60-S80, 21 key point positions of the left and right hand images in the target hand image, i.e. the pixel coordinates of the 21 key points of the left and right hand images in the target hand image, can be obtained, and the key point pixel coordinates can be defined as (x i , y i , z i ). The heat map output by the pre-trained key point regression neural network includes the above key point pixel coordinates.

[0090] In some embodiments, the semantic segmentation neural network is improved based on a STDC (Short-Term Dense Concatenate) model. Please refer to Figure 3 , the step of inputting the target hand image into the pre-trained semantic segmentation neural network to obtain a semantic segmentation graph corresponding to the target hand image, includes:

[0091] S101, inputting a target hand image with a first size into a pre-trained semantic segmentation neural network to obtain a first feature map with a second size, wherein the first feature map includes a first channel, a second channel and a third channel, and the first channel, the second channel and the third channel respectively represent the categories of a background image, a left hand image and a right hand image in the hand image, wherein the first size is greater than the second size;

[0092] S102, adjust the size of the first feature map to generate a second feature map with the first size;

[0093] S103, obtain a semantic segmentation map corresponding to the target hand image based on the second feature map.

[0094] As described in steps S101-S103, in the embodiment, a target hand image (which can be in the form of a grayscale picture) with a size of H*W (where H is the height and W is the width, i.e. the first size) is input into a pre-trained semantic segmentation neural network, which outputs a first feature map with a size of (H / n)*(W / n) (i.e. the second size), where n is a positive integer greater than or equal to 1, and preferably n can take the value 4; in addition, the first feature map is a three-channel feature map, where the first channel, the second channel and the third channel respectively represent the categories of the background image, the left hand image and the right hand image in the input target hand image. In the embodiment, the input image with the first size is output as an output image with the second size in the semantic segmentation neural network, in order to reduce the amount of calculation to adapt to some hardware systems with lower computing power.

[0095] In order to obtain a final semantic segmentation map with the same size as the original image, the size of the first feature map is adjusted, i.e. the size of the first feature map is converted from the second size back to the first size to obtain a second feature map. Specifically, the second size can be enlarged by n times by means of bilinear interpolation to obtain the second feature map.

[0096] After obtaining the second feature map, softmax normalization is performed on the channel direction of the second feature map, and the argmax function value is maximized to find the category (i.e. left hand, right hand or background) of each pixel in the second feature map. The final semantic segmentation map can be obtained by the following formula:

[0097] final out =argmax dim＝1 (softmax dim＝1 (seg out ))

[0098] where final out is the final semantic segmentation map output, and its size can be represented as H*W; seg out is the second feature map, and its size can be represented as 3*(H / n)*(W / n), 3 is the number of channels, and n is a positive integer greater than or equal to 1; final out only contains three values, which can be 0, 1 and 2, 0 representing background, 1 representing left hand and 2 representing right hand; dim is the dimension.

[0099] In the embodiment, the training process of the semantic segmentation neural network (exemplarily, n is 4) is as follows:

[0100] In one training, a gray image with left and right hand images with a size of H*W is taken as an input image, and a corresponding ground truth segmentation image is taken as a training target. The ground truth segmentation image marks the pixel region of the background image part as 0, marks the pixel region of the left hand image as 1, and marks the pixel region of the right hand image as 2. After one forward propagation, the semantic segmentation neural network outputs a first feature map seg out . The width and height of the first feature map seg out are enlarged by 4 times through bilinear interpolation to become a second feature map, and then cross-entropy loss calculation is performed on the second feature map and the ground truth segmentation image, as shown in the following formula:

[0101]

[0102] wherein, CEloss is a cross-entropy loss function; H is the height of the input image, and W is the width of the input image; is the probability that the (j, k) pixel on the ground truth segmentation image is i, i=0 represents that the pixel belongs to the background image part, i=1 represents that the pixel belongs to the left hand image part, and i=2 represents that the pixel belongs to the right hand image part.

[0103] Since the above ground truth segmentation image is manually annotated, y are known, and y j,k is the value of the element with the coordinate (i, j, k) on the second feature map. After the loss value is found, the network parameters are updated in a stochastic gradient descent manner. It should be noted that, since the cross-entropy loss value is calculated through the first feature map seg out in the training, the parameter final out is not used, and therefore, softmax normalization and argmax function value maximization are not required in the training process. In order to solve the problem that the image recognition is difficult due to the overlapping of the left and right hands, the training data set of the semantic segmentation neural network provided in the embodiment includes a large number of overlapping hand images and corresponding ground truth segmentation images. After training, the semantic segmentation neural network provided in the embodiment can accurately segment the left hand image and the right hand image in the target hand image regardless of whether the left hand image and the right hand image in the target hand image overlap.

[0104] In some embodiments, referring to Figure 4 , the first heat map set includes a first position heat map and a first depth heat map, and the step of obtaining the key point positions of the left hand image based on the first heat map set includes:

[0105] S801, acquire a maximum position of the first position heat map;

[0106] S802, extract a first matrix from the first position heat map with the maximum position of the first position heat map as the center;

[0107] S803, normalize the first matrix to obtain a first normalization result;

[0108] S804, obtain a position coordinate of a key point of the left hand image based on the first normalization result;

[0109] S805, acquire a maximum position of the first depth heat map;

[0110] S806, extract a second matrix from the first depth heat map with the maximum position of the first depth heat map as the center;

[0111] S807, normalize the second matrix to obtain a second normalization result;

[0112] S808, obtain a depth coordinate of the key point of the left hand image based on the second normalization result;

[0113] S809, obtain a key point position of the left hand image based on the position coordinate and the depth coordinate of the left hand image.

[0114] As described in steps S801-S809, the pixel coordinates of the left / right hand key point in the embodiment of the application can be defined as (x, y, z). In actual working conditions, if the three-dimensional heat map is directly output, the calculation amount will be very large, therefore, the pre-trained key point regression neural network in the embodiment splits the three-dimensional heat map into a two-dimensional heat map H(x, y) and a one-dimensional heat map D(z), so as to effectively reduce the data amount and improve the efficiency of obtaining the key point position of the target hand image. The two-dimensional heat map H(x, y) is the position heat map, and the one-dimensional heat map D(z) is the depth heat map.

[0115] In the embodiment, the maximum position of the first position heat map can be obtained by the following formula:

[0116]

[0117] wherein H1 is the first position heat map, is the maximum position of the first position heat map.

[0118] The maximum position of the first position heat map H1 is The first matrix h1 is extracted from the first position heat map, and specifically, the first matrix h1 can be an n*n matrix, where n is a positive integer, and the value of n can be set according to actual design requirements, which is not limited herein.

[0119] The first matrix h1 is subjected to softmax normalization to obtain a first normalized result (i.e., the position coordinate expectation value of the key point), which can be obtained by the following formula:

[0120] (x1′,y1′)=∑softmax(τh1 i,j )*(i,j)

[0121] wherein,

[0122] (i,j) is an element h1 i,j In the coordinate position of the first position heat map, the first matrix h1 is an n*n matrix, and n is a positive integer. is the maximum value position of the first position heat map, τ is the softmax temperature, and is a preset value; (x1′,y1′) is the first normalized result, i.e., the position coordinate expectation value of the key point.

[0123] In some embodiments, in order to reduce the amount of calculation, the length and width of the output heat map can be changed to 1 / p of the original image (in this embodiment, it is the first image) of the input heat map regression model, where p is a positive integer (for example, the length and width of the heat map are changed to 1 / 4 of the original image). In this case, after the position coordinate expectation value of the key point is calculated by using the position heat map, the position coordinate expectation value of the key point needs to be converted to obtain the position coordinate of the key point in the input original image according to the ratio relationship between the length and width of the position heat map and the length and width of the input original image. Specifically, in this embodiment, the conversion formula is as follows:

[0124]

[0125] wherein, (x1,y1) is the position coordinate of the key point in the first image, (x1′,y1′) is the position coordinate expectation value of the key point of the first position heat map, s x is the ratio of the width of the first position heat map to the width of the first image, s y is the ratio of the length of the first position heat map to the length of the first image.

[0126] Similarly, in this embodiment, the maximum value position of the first depth heat map can be obtained by the following formula:

[0127]

[0128] D1 is a first position heat map, is a maximum position of the first depth heat map.

[0129] The maximum position of the first depth heat map D1 is taken as the center, a second matrix d1 is extracted from the first depth heat map, and specifically, the second matrix d1 can be a 1*n matrix, where n is a positive integer, and the value of n is the same as the value of n of the first matrix h1.

[0130] The second matrix d1 is subjected to softmax normalization to obtain a second normalized result (i.e., a depth coordinate expected value of the key point), and the second normalized result can be obtained by the following formula:

[0131] (z1') = ∑ softmax(τd1 i )*(i)

[0132] wherein, (i) is an element d1 i in the second matrix d1, and d2 is a 1*n matrix in the coordinate position in the first depth heat map, and n is a positive integer; is a maximum position of the first depth heat map, τ is a softmax temperature, and is a preset value; (z1') is a second normalized result, i.e., a depth coordinate expected value of the key point. In this embodiment, the depth coordinate expected value (z1') of the key point is set as the depth coordinate (1) of the key point of the first image.

[0133] The position coordinate (x1, y1) and the depth coordinate (z1 of the key point of the first image are combined, i.e., the position (x1, y1, z1 of one key point of the first image (i.e., the pixel coordinate of the key point in the first image) is obtained. It should be noted that if the first image and the first screenshot (the first screenshot corresponds to the left hand image in the target hand image) have a size conversion relationship, then the position (x1, y1, z1 of one key point in the left hand image in the target hand image can be obtained after the position (x1, y1, z1 of the key point of the first image is subjected to corresponding size conversion; and if the first image and the first screenshot do not have a size conversion relationship, then the position (x1, y1, z1 of one key point in the left hand image in the target hand image is the position (x1, y1, z1 of the key point of the first image.

[0134] In the embodiment, after each input first image is processed by the key point regression neural network, M sets of heat maps can be obtained, each set of heat maps including one position heat map and one depth heat map. In the prior art, the scheme of reconstructing a human hand model or performing gesture recognition by using a hand image is generally based on 21 key points of left / right palms, and therefore, in the embodiment, the value of M is preferably 21. In other embodiments, the value of M can be selected according to actual design requirements, for example, when a higher / lower recognition accuracy is required, the value of M can be appropriately increased / decreased to meet the actual design requirements, and therefore, the value of M is not limited.

[0135] The sub-pixel information of the Gaussian heat map corresponding to the target hand image can be successfully extracted by the steps S801-S809 in the embodiment, and the key point regression neural network provided in the embodiment acquires the final heat map by splitting the three-dimensional heat map into a two-dimensional heat map H(x, y) and a one-dimensional heat map D(z), which has a more simplified neural network structure, a faster inference speed, and a more accurate key point position extraction effect compared with the commonly used heat map regression model in the prior art.

[0136] In some embodiments, the second heat map set includes a second position heat map and a second depth heat map, and the step of obtaining the key point position of the right hand image based on the second heat map set includes:

[0137] S8010, obtaining a maximum value position of the second position heat map;

[0138] S8011, extracting a third matrix from the second position heat map with the maximum value position of the second position heat map as the center;

[0139] S8012, normalizing the third matrix to obtain a third normalized result;

[0140] S8013, obtaining the position coordinates of the key points of the right hand image based on the third normalized result;

[0141] S8014, obtaining a maximum value position of the second depth heat map;

[0142] S8015, extracting a fourth matrix from the first depth heat map with the maximum value position of the first depth heat map as the center;

[0143] S8016, normalizing the fourth matrix to obtain a fourth normalized result;

[0144] S8017, obtaining the depth coordinates of the key points of the right hand image based on the fourth normalized result;

[0145] S8018, based on the position coordinates and depth coordinates of the key points of the right-hand image, obtain the key point positions of the right-hand image.

[0146] It should be noted that in this embodiment, the method of obtaining the key point positions of the right-hand image in the target hand image is similar to the method of obtaining the key point positions of the left-hand image in the target hand image provided in the embodiment shown in Figure 4 the embodiment shown in Figure 4 the related principle description part.

[0147] The key point regression neural network in this embodiment can be obtained by the following method:

[0148] Firstly, the key point regression neural network has an input, i.e. the first / second image generated by the target hand image to be trained, when training. After processing by the neural network, the neural network outputs two heat maps corresponding to an input image, i.e. the position heat map and the depth heat map. During training, the labels corresponding to the input at this time are provided, i.e. the true value position heat map and the true value depth heat map. At this time, the mean square error loss (MES loss) is calculated between the neural network output heat map and the corresponding label heat map, and then the gradient back propagation is used for parameter updating of the neural network.

[0149] Secondly, the key point regression neural network has two inputs, i.e. the first / second image generated by the target hand image to be trained and the historical key point positions corresponding thereto, when training. After processing by the neural network, the neural network outputs two heat maps corresponding to an input image, i.e. the position heat map and the depth heat map. During training, the labels corresponding to the input at this time are also provided, i.e. the true value position heat map and the true value depth heat map. At this time, the mean square error loss (MES loss) is calculated between the neural network output heat map and the corresponding label heat map, and then the gradient back propagation is used for parameter updating of the neural network.

[0150] In one embodiment, please refer to Figure 5 The hand image key point position obtaining method further comprises:

[0151] S90, if the left image and the right-hand image do not overlap, based on the semantic segmentation map, the pixel value corresponding to the image part of the target hand image except the left-hand image and the right-hand image is set to 0, and a third optimized target hand image is obtained;

[0152] S100, based on the third optimized target hand image, the image in the first rectangular frame and the image in the second rectangular frame are respectively intercepted, and a third screenshot and a fourth screenshot are obtained;

[0153] S110, pre-process the third screenshot and the fourth screenshot to generate a third image and a fourth image;

[0154] S120, input the third image and the fourth image into a pre-trained key point regression neural network to generate a third heat map set corresponding to the third image and a fourth heat map set corresponding to the fourth image, wherein each of the third heat map set and the fourth heat map set includes at least one heat map, and each of the heat maps is a Gaussian heat map with sub-pixel information.

[0155] S130, obtain the key point position of the left hand image based on the third heat map set, and obtain the key point position of the right hand image based on the fourth heat map set.

[0156] As described in step S90, in the embodiment, when the left and right hand images in the target hand image do not overlap, the pixel values corresponding to the image parts other than the left and right hand images in the target hand image are set to 0 based on the semantic segmentation map obtained in step S10, so as to remove the pixel information of the background image (i.e. the image parts other than the left and right hand images) in the target hand image, thereby reducing the noise generated by the background image information for subsequent neural network inference, and improving the accuracy of hand information recognition (in the embodiment, the hand information is the key point position of the hand).

[0157] As described in step S100, based on the position information (which can be expressed as pixel coordinates in the target hand image) of the first rectangular frame and the second rectangular frame, the image in the first rectangular frame is intercepted in a third optimized target hand image (the third optimized target hand image has multiple backups), and the image in the second rectangular frame is intercepted in another third optimized target hand image to obtain a third screenshot and a fourth screenshot, wherein the third screenshot and the fourth screenshot respectively include the left hand image and the right hand image with the pixel information of the background image removed.

[0158] It should be noted that the working process and principle in steps S110-S130 in the embodiment are similar to those of steps S60-S80 in the embodiment shown in Figure 2 and will not be described here in detail. Please refer to the principle description part of the embodiment shown in Figure 2 .

[0159] The hand image key point position acquisition method provided in the application combines the semantic segmentation neural network and the key point regression neural network, and in the case of overlapping hands, the semantic information provided by the semantic segmentation neural network can remove the pixel information of the non-target hand in the target hand image, thereby reducing the noise generated by the highly similar objects for neural network inference, and improving the accuracy of hand information recognition in the case of overlapping hands.

[0160] The embodiment of the present application also provides a hand image key point position acquisition device, which refers to Figure 6 The device comprises:

[0161] a semantic segmentation map acquisition module 601, configured to acquire a semantic segmentation map corresponding to a hand image based on a pre-trained semantic segmentation neural network, wherein the hand image comprises a left hand image and a right hand image;

[0162] a positioning module 602, configured to position a first rectangular frame position and a second rectangular frame position of the left hand image and the right hand image in the hand image according to the semantic segmentation map;

[0163] a judgment module 603, configured to judge whether the left hand image and the right hand image overlap according to the first rectangular frame position and the second rectangular frame position;

[0164] a first execution module 604, configured to, when the left hand image and the right hand image overlap, set pixel values of image parts other than the left hand image in the target hand image to 0 based on the semantic segmentation map to obtain a first optimized target hand image, and set pixel values corresponding to image parts other than the right hand image in the target hand image to 0 to obtain a second optimized target hand image;

[0165] a screenshot module 605, configured to take a screenshot of an image in the first rectangular frame in the first optimized target hand image to obtain a first screenshot, and take a screenshot of an image in the second rectangular frame in the second optimized target hand image to obtain a second screenshot;

[0166] a preprocessing module 606, configured to pre-process the first screenshot and the second screenshot to generate a first image and a second image;

[0167] a heat map set acquisition module 607, configured to input the first image and the second image into a pre-trained key point regression neural network to generate a first heat map set corresponding to the first image and a second heat map set corresponding to the second image, wherein the first heat map set and the second heat map set each comprise at least one heat map, and each heat map is a Gaussian heat map with sub-pixel information;

[0168] a key point position acquisition module 608, configured to acquire key point positions of the left hand image based on the first heat map set, and acquire key point positions of the right hand image based on the second heat map set.

[0169] In this embodiment, the target hand image includes a left hand image and a right hand image, where the left hand image can be partially (or entirely) overlapped or completely separated. The semantic segmentation map acquisition module 601 inputs the target hand image into a pre-trained semantic segmentation neural network to obtain a semantic segmentation map corresponding to the target hand image. In this embodiment, the training sample set of the semantic segmentation neural network is obtained based on a large number of hand images with overlapped hands, so that the present application has good recognition effect for the input left and right hand images with overlapped target hand images, and the obtained hand image key point position is more accurate. It should be noted that the semantic segmentation neural network used in the present application can be an existing semantic segmentation model with good semantic segmentation effect for hand images, or an improved semantic segmentation model based on an existing semantic segmentation model, which is not limited here.

[0170] In this embodiment, after obtaining the semantic segmentation map corresponding to the target hand image, the positioning module 602 positions the first rectangular frame position and the second rectangular frame position of the left hand image and the right hand image in the target hand image according to the semantic segmentation map. Generally, the object tracking algorithm such as YOLO, Faster RCNN, etc. can be used to find the target object in the picture to be detected, and a rectangular frame is used to display it, so as to facilitate subsequent processing (such as classification) of the target object. In this embodiment, the above-mentioned target object is the left and right hands in the target hand image, and the picture to be detected is the target hand image.

[0171] In this embodiment, when the first rectangular frame position and the second rectangular frame position are positioned, the judgment module 603 judges whether the left hand image and the right hand image in the target hand image are overlapped according to the first rectangular frame position and the second rectangular frame position. Specifically, by judging whether there is an overlapping part between the image area in the first rectangular frame and the image area in the second rectangular frame, it can be judged whether the left hand image and the right hand image in the target hand image are overlapped. Whether the image areas are overlapped can be judged by the following formula:

[0172]

[0173] Where I o U is the intersection over union, A is the image area of the first rectangular frame, and B is the image area of the second rectangular frame.

[0174] When I o U is 0, it means that the left hand image and the right hand image in the target hand image are not overlapped, and when I o U is not 0, it means that the left hand image and the right hand image in the target hand image are overlapped.

[0175] In this embodiment, when the left and right hand images in the target hand image overlap, the first execution module 604 sets the pixel values of the image part other than the left hand image in one target hand image (there are multiple backups of the target hand image) to 0 based on the above-mentioned semantic segmentation map, to obtain a first optimized target hand image; and sets the pixel values corresponding to the image part other than the right hand image in another target hand image to 0, to obtain a second optimized target hand image. By setting the pixel values of the image part other than the left hand image or the right hand image to 0, the pixel information of the non-target hand (left hand or right hand) in the target hand image can be removed, thereby reducing the noise generated by the highly similar object for subsequent neural network reasoning, so that in the case of left and right hand overlap, the accuracy of hand information recognition (in this embodiment, the hand information is the key point position of the hand) is improved.

[0176] Subsequently, the screenshot module 605 obtains a first screenshot and a second screenshot by capturing the image in the first rectangular frame in one target hand image with a rectangular frame and capturing the image in the second rectangular frame in another target hand image with a rectangular frame based on the position information (which can be expressed as pixel coordinates in the target hand image) of the first rectangular frame and the second rectangular frame and the target hand images (two) with the rectangular frame, wherein the first screenshot and the second screenshot respectively contain the left hand image and the right hand image with the pixel information of the non-target hand removed.

[0177] Before inputting the above-mentioned first screenshot and second screenshot into the pre-trained key point regression neural network, in order to obtain better regression effect or save computing resources, etc., the first screenshot and the second screenshot are preprocessed by the preprocessing module 606, for example, in some embodiments, the size of the first screenshot and the second screenshot is transformed to a preset size (such as 128*128) to adapt to the pre-trained key point regression neural network, so as to obtain more accurate prediction results.

[0178] The key point regression neural network in this application adopts a sub-pixel heat map based neural network model, because the heat map regression model has the advantage of strong generalization ability, and even if the distribution of training samples is uneven during model training, better prediction results can be obtained during prediction. Similarly, the heat map regression model used in this application can be an existing heat map regression model with high accuracy (such as a heat map regression model combined with heatmap and offset), or a key point heat map regression model improved based on the existing heat map regression model, which is not limited here.

[0179] The first image and the second image obtained after preprocessing the first screenshot and the second screenshot are input into the pre-trained key point regression neural network, and a first heat map set corresponding to the first image and a second heat map set corresponding to the second image are generated by the heat map set acquisition module 607, wherein the first heat map set and the second heat map set each include at least one heat map, and each heat map is a Gaussian heat map with sub-pixel information; the key point position acquisition module 608 obtains the key point positions of the left hand image in the target hand image based on the first heat map set, and obtains the key point positions of the right hand image in the target hand image based on the second heat map set.

[0180] It can be understood that each component of the hand image key point position acquisition apparatus proposed in the present application can realize the function of any one of the hand image key point position acquisition methods provided in any one of the above embodiments, and the specific structure will not be described again.

[0181] Please refer to Figure 7 In the embodiments of the present application, a computer device is also provided, and the internal structure of the computer device can be as shown in Figure 7 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a storage medium and an internal memory. The storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the storage medium. The database of the computer device is used to store related data of a hand image key point position acquisition method. The network interface of the computer device is used to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement the hand image key point position acquisition method provided in any one of the above embodiments.

[0182] In the embodiments of the present application, a computer readable storage medium is also provided, which can be non-volatile or volatile, and has a computer program stored thereon. The computer program is executed by the processor to implement the steps of the hand image key point position acquisition method provided in any one of the above embodiments.

[0183] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, storage, databases, or other media in this application and in examples provided herein refers to both non-transitory and / or transitory storage media. Non-transitory storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Transitory storage media can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), extended SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0184] The hand image key point position acquisition method and device and the computer device provided in the application can remove pixel information of a non-target hand in a target hand image by using semantic information provided by the semantic segmentation neural network, reduce noise generated by highly similar objects in neural network reasoning, and improve the accuracy of hand information recognition in the case of overlapping hands.

[0185] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, device, article or method including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, device, article or method. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0186] The above description is only the preferred embodiment of the application, and does not limit the patent scope of the application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the application.

Claims

1. A method for obtaining hand image key point positions, characterized in that, The method comprises the following steps: inputting a target hand image into a pre-trained semantic segmentation neural network to obtain a semantic segmentation graph corresponding to the target hand image, wherein the target hand image comprises a left hand image and a right hand image; locating first and second rectangular frame positions of the left hand image and the right hand image in the target hand image according to the semantic segmentation graph; determining whether the left hand image and the right hand image overlap according to the first and second rectangular frame positions; if the left hand image and the right hand image overlap, setting pixel values of an image part other than the left hand image in the target hand image to 0 based on the semantic segmentation graph to obtain a first optimized target hand image, and setting pixel values corresponding to an image part other than the right hand image in the target hand image to 0 to obtain a second optimized target hand image; cutting out an image in the first rectangular frame in the first optimized target hand image to obtain a first cutout image, and cutting out an image in the second rectangular frame in the second optimized target hand image to obtain a second cutout image; preprocessing the first and second cutout images to generate a first image and a second image; inputting the first and second images into a pre-trained key point regression neural network to generate a first heat map set corresponding to the first image and a second heat map set corresponding to the second image, wherein each of the first and second heat map sets comprises at least one heat map, and each heat map is a Gaussian heat map with sub-pixel information; obtaining key point positions of the left hand image based on the first heat map set, and obtaining key point positions of the right hand image based on the second heat map set.

2. The method of claim 1, wherein, The step of inputting a target hand image into a pre-trained semantic segmentation neural network to obtain a semantic segmentation graph corresponding to the target hand image comprises: inputting a target hand image with a first size into a pre-trained semantic segmentation neural network to obtain a first feature map with a second size, wherein the first feature map comprises a first channel, a second channel and a third channel, and the first channel, the second channel and the third channel represent the categories of a background image, a left hand image and a right hand image in the hand image respectively, and the first size is greater than the second size; adjusting the size of the first feature map to generate a second feature map with the first size; obtaining a semantic segmentation graph corresponding to the target hand image based on the second feature map.

3. The method of claim 1, wherein, The first heat map set comprises a first position heat map and a first depth heat map, and the step of obtaining key point positions of the left hand image based on the first heat map set comprises: obtaining a maximum value position of the first position heat map; extracting a first matrix from the first position heat map with the maximum value position of the first position heat map as the center; normalizing the first matrix to obtain a first normalized result; obtaining position coordinates of key points of the left hand image based on the first normalized result; obtaining a maximum value position of the first depth heat map; extracting a second matrix from the first depth heat map with the maximum value position of the first depth heat map as the center; normalizing the second matrix to obtain a second normalization result; obtaining the depth coordinates of the key points of the left hand image based on the second normalization result; obtaining the key point positions of the left hand image based on the position coordinates and the depth coordinates of the left hand image.

4. The method of claim 1, wherein, The second heat map set includes a second position heat map and a second depth heat map, and the step of obtaining the key point positions of the right hand image based on the second heat map set includes: obtaining the maximum value position of the second position heat map; extracting a third matrix from the second position heat map with the maximum value position of the second position heat map as the center; normalizing the third matrix to obtain a third normalization result; obtaining the position coordinates of the key points of the right hand image based on the third normalization result; obtaining the maximum value position of the second depth heat map; extracting a fourth matrix from the second depth heat map with the maximum value position of the second depth heat map as the center; normalizing the fourth matrix to obtain a fourth normalization result; obtaining the depth coordinates of the key points of the right hand image based on the fourth normalization result; obtaining the key point positions of the right hand image based on the position coordinates and the depth coordinates of the key points of the right hand image.

5. The method of claim 1, wherein, The step of preprocessing the first and second screenshots to generate the first and second images includes: transforming the sizes of the first and second screenshots into preset sizes respectively to generate the first and second images.

6. The method of claim 1, wherein, Further comprising: if the left hand image and the right hand image do not overlap, setting the pixel values corresponding to the image parts other than the left hand image and the right hand image in the target hand image to 0 based on the semantic segmentation map to obtain a third optimized target hand image; based on the third optimized target hand image, respectively cutting the images in the first and second rectangular frames to obtain third and fourth screenshots; preprocessing the third and fourth screenshots to generate third and fourth images; inputting the third and fourth images into a pre-trained key point regression neural network to generate a third heat map set corresponding to the third image and a fourth heat map set corresponding to the fourth image, wherein the third and fourth heat map sets each include at least one heat map, and each heat map is a Gaussian heat map with sub-pixel information; obtaining the key point positions of the left hand image based on the third heat map set and the key point positions of the right hand image based on the fourth heat map set.

7. The method of claim 1, wherein, The training data set for training the semantic segmentation neural network and the key point regression neural network is obtained based on a hand image with overlapping hands.

8. A hand image key point position acquisition apparatus, characterized by comprising: comprising: a semantic segmentation map acquisition module configured to acquire a semantic segmentation map corresponding to a target hand image based on a pre-trained semantic segmentation neural network, wherein the target hand image includes a left hand image and a right hand image; a positioning module configured to position first and second rectangular frame positions of the left and right hand images in the target hand image according to the semantic segmentation map; a judging module configured to judge whether the left and right hand images overlap according to the first and second rectangular frame positions; a first execution module configured to, when the left and right hand images overlap, set pixel values of image portions of the target hand image except the left hand image to 0 based on the semantic segmentation map to obtain a first optimized target hand image, and set pixel values corresponding to image portions of the target hand image except the right hand image to 0 to obtain a second optimized target hand image; a screenshot module configured to take a screenshot of the first rectangular frame in the first optimized target hand image to obtain a first screenshot, and take a screenshot of the second rectangular frame in the second optimized target hand image to obtain a second screenshot; a preprocessing module configured to pre-process the first and second screenshots to generate a first image and a second image; a heat map set obtaining module configured to input the first and second images into a pre-trained key point regression neural network to generate a first heat map set corresponding to the first image and a second heat map set corresponding to the second image, wherein the first and second heat map sets each include at least one heat map, and each heat map is a Gaussian heat map with sub-pixel information; a key point position obtaining module configured to obtain key point positions of the left hand image based on the first heat map set, and obtain key point positions of the right hand image based on the second heat map set. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8. The processor executes the computer program to implement the steps of the hand image key point position obtaining method in any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the hand image key point position obtaining method in any one of claims 1-7.

Images