Hand key point detection method, device, equipment and medium

By combining high semantic and low semantic features for hand key point detection, the problem of limited offset map improvement in the prior art is solved, and the accuracy of hand key point detection is significantly improved.

CN112966592BInactive Publication Date: 2024-11-22BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202110236032.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-03
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When using neural networks, the existing hand key point detection methods only rely on roughly estimated feature maps for correction, resulting in limited improvement in offset maps and unable to output more accurate prediction results.

Method used

By inputting the image to be detected into the first sub-model to obtain rough estimates and high-semantic hierarchical features, and inputting the second sub-model to combine the low-semantic hierarchical features to predict the offset vector to improve the accuracy of the final prediction results of hand key point detection. sex.

Benefits of technology

By combining high-semantic and low-semantic features, the refined network can predict based on richer semantic information, significantly improving the output accuracy of the hand key point detection model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112966592B_ABST
    Figure CN112966592B_ABST
Patent Text Reader

Abstract

The present disclosure provides a method, device, equipment and medium for detecting key points of a hand, which relates to the field of artificial intelligence technology, specifically the field of computer vision and deep learning technology. The method for detecting key points of a hand includes: inputting an image to be detected into a first sub-model to obtain a first prediction result and a first feature map output by the first sub-model, wherein the first prediction result represents a preliminary predicted position of each key point of the hand; based on the first prediction result, obtaining a first key point feature from the first feature map; based on the first prediction result, obtaining a second key point feature from the image to be detected; inputting the first key point feature and the second key point feature into a second sub-model to obtain a second prediction result output by the second sub-model, wherein the second prediction result represents a correction to the preliminary predicted position of each key point; and combining the first prediction result and the second prediction result to obtain a detection result of the key points of a hand.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically the field of computer vision and deep learning technology, and in particular to a hand key point detection method, device, equipment and medium. Background Art

[0002] Artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] With the progress of society and the rapid development of science and technology, short videos, live broadcasts, online education and other industries are constantly emerging. In various interactive scenarios, there are more and more functional requirements for interaction based on hand key point information. By realizing the detection of hand key points in videos or images, the interactivity between users and electronic devices can be improved, and the user experience can be improved.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a hand key point detection method, device, equipment and medium.

[0006] According to one aspect of the present disclosure, a hand key point detection method is provided, including: inputting an image to be detected into a first sub-model to obtain a first prediction result and a first feature map output by the first sub-model, wherein the image to be detected includes a hand, and the first prediction result represents a preliminary predicted position of each key point of at least one key point of the hand; based on the first prediction result, obtaining a first key point feature from the first feature map, the first key point feature including features corresponding to each key point respectively; based on the first prediction result, obtaining a second key point feature from the image to be detected, the second key point feature including features corresponding to each key point respectively; inputting the first key point feature and the second key point feature into a second sub-model to obtain a second prediction result output by the second sub-model, wherein the second prediction result represents a correction to the preliminary predicted position of each key point; and combining the first prediction result and the second prediction result to obtain a hand key point detection result.

[0007] According to another aspect of the present disclosure, a training method for a hand key point detection model is also provided, the hand key point detection model includes a first sub-model and a second sub-model, and the training method includes: obtaining a sample image and marking a sample true result, the sample image includes a hand, and the sample true result represents the true position of each key point of at least one key point of the hand; using the sample image and the sample true result to train the first sub-model; inputting the sample image into the trained first sub-model to obtain a first sample prediction result and a first sample feature map output by the first sub-model, the first sample prediction result represents the preliminary predicted position of each key point; based on the first sample prediction result, obtaining a first sample key point feature from the first sample feature map, the first sample key point feature includes features corresponding to each key point respectively; based on the first sample prediction result, obtaining a second sample key point feature from the sample image, the second sample key point feature includes features corresponding to each key point respectively; inputting the first sample key point feature and the second sample key point feature into the second sub-model to obtain a second sample prediction result output by the second sub-model, wherein the second sample prediction result represents a correction to the preliminary predicted position of each key point; and adjusting the parameters of the second sub-model based on the sample true result, the first sample prediction result, and the second sample prediction result.

[0008] According to another aspect of the present disclosure, a hand key point detection device is also provided, including: a first prediction module, configured to input an image to be detected into a first sub-model to obtain a first prediction result and a first feature map output by the first sub-model, wherein the image to be detected includes a hand, and the first prediction result represents a preliminary predicted position of each key point of at least one key point of the hand; a first feature acquisition module, configured to acquire a first key point feature from the first feature map based on the first prediction result, the first key point feature including features corresponding to each key point respectively; a second feature acquisition module, configured to acquire a second key point feature from the image to be detected based on the first prediction result, the second key point feature including features corresponding to each key point respectively; a second prediction module, configured to input the first key point feature and the second key point feature into a second sub-model to obtain a second prediction result output by the second sub-model, wherein the second prediction result represents a correction to the preliminary predicted position of each key point; and a third prediction module, configured to combine the first prediction result and the second prediction result to obtain a hand key point detection result.

[0009] According to another aspect of the present disclosure, a training device for a hand key point detection model is also provided. The hand key point detection model includes a first sub-model and a second sub-model. The training device includes: an acquisition module, configured to acquire a sample image and annotate a sample true result, the sample image includes a hand, and the sample true result represents the true position of each key point of at least one key point of the hand; a training module, configured to train the first sub-model using the sample image and the sample true result; a first prediction module, configured to input the sample image into the trained first sub-model to obtain a first sample prediction result and a first sample feature map output by the first sub-model, the first sample prediction result represents the preliminary predicted position of each key point; a first feature acquisition module, configured to obtain a first sample prediction result and a first sample feature map output by the first sub-model, the first sample prediction result represents the preliminary predicted position of each key point based on the first sample prediction result; The first sample key point feature is obtained from the first sample feature map, and the first sample key point feature includes features corresponding to each key point respectively; the second feature acquisition module is configured to obtain the second sample key point feature from the sample image based on the first sample prediction result, and the second sample key point feature includes features corresponding to each key point respectively; the second prediction module is configured to input the first sample key point feature and the second sample key point feature into the second sub-model to obtain the second sample prediction result output by the second sub-model, wherein the second sample prediction result represents the correction to the preliminary prediction position of each key point; and the parameter adjustment module is configured to adjust the parameters of the second sub-model based on the sample true result, the first sample prediction result and the second sample prediction result.

[0010] According to another aspect of the present disclosure, a computer device is provided, including: a memory, a processor, and a computer program stored in the memory, wherein the processor is configured to execute the computer program to implement the steps of the above-mentioned hand key point detection method and training method.

[0011] According to another aspect of the present disclosure, a computer-readable storage medium storing a program is also provided. The program includes instructions. When the instructions are executed by a processor of an electronic device, the electronic device executes the above-mentioned hand key point detection method and training method.

[0012] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein when the computer program is executed by a processor, the steps of the above-mentioned hand key point detection method and training method are implemented.

[0013] According to one or more embodiments of the present disclosure, the image to be detected is input into the first sub-model to obtain a rough estimate of the position of the hand key points in the image to be detected output by the first sub-model, i.e., the first prediction result, and the high semantic level features output by the first sub-model, i.e., the first feature map; based on the first prediction result, the high semantic level features at the hand key point positions in the first feature map and the low semantic level features near the hand key point positions in the image to be detected are obtained; these features are then input into the second sub-model, i.e., the refined model, to obtain an offset vector for correcting the rough estimate result; and finally, the rough estimate and the offset vector are combined to obtain the final result of the hand key point detection. The present disclosure uses the combined features of the high semantic level features and the low semantic level features as the input of the second sub-model, i.e., the refined network, so that the refined network can predict the offset vector based on richer semantic information, thereby being able to output a more accurate offset vector, and thus being able to improve the accuracy of the final prediction result output by the hand key point detection model.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0016] Figure 1 A flow chart of a hand key point detection method according to an exemplary embodiment of the present disclosure is shown;

[0017] Figure 2 A schematic diagram of a hand key point detection model according to an exemplary embodiment of the present disclosure is shown;

[0018] Figure 3 A schematic diagram showing an offset vector according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 4 A flow chart of a hand key point detection method according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 5-Figure 6 A flowchart of a method for training a hand key point detection model according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 7 A structural block diagram of a hand key point detection device according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 8 A structural block diagram showing a training device for a hand key point detection model according to an exemplary embodiment of the present disclosure; and

[0023] Fig. 9 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0025] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0026] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0027] In the related art, the existing method first obtains the feature map of the image to be detected through a neural network when detecting the key points of the hand, then generates a preliminary prediction result of the key points of the hand based on the feature map, and then generates an offset map based on the feature map to correct the preliminary prediction result to obtain the final prediction result. However, since this method only uses the feature map used to generate a rough estimate of the key points of the hand as the basis for further correction of the preliminary prediction result, the offset map obtained by this method can only bring very limited improvement to the prediction result.

[0028] In order to solve the above problems, the present disclosure inputs the image to be detected into the first sub-model to obtain a rough estimate of the position of the hand key points in the image to be detected output by the first sub-model, i.e., the first prediction result, and obtains the high semantic level features output by the first sub-model, i.e., the first feature map; based on the first prediction result, obtain the high semantic level features at the position of the hand key points in the first feature map and the low semantic level features near the position of the hand key points in the image to be detected; then input these features into the second sub-model, i.e., the refined model, to obtain an offset vector for correcting the rough estimate result; and finally combine the rough estimate and the offset vector to obtain the final result of the hand key point detection. The present disclosure uses the combined features of the high semantic level features and the low semantic level features as the input of the second sub-model, i.e., the refined network, so that the refined network can predict the offset vector based on richer semantic information, thereby being able to output a more accurate offset vector, and further being able to improve the accuracy of the final prediction result output by the hand key point detection model.

[0029] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0030] According to one aspect of the present disclosure, a method for detecting key points of a hand is provided. Figure 1 As shown, the detection method may include: step S101, inputting the image to be detected into the first sub-model to obtain a first prediction result and a first feature map output by the first sub-model, wherein the image to be detected includes a hand, and the first prediction result represents the preliminary predicted position of each key point of at least one key point of the hand; step S102, based on the first prediction result, obtaining a first key point feature from the first feature map, wherein the first key point feature includes features corresponding to each key point respectively; step S103, based on the first prediction result, obtaining a second key point feature from the image to be detected, wherein the second key point feature includes features corresponding to each key point respectively; step S104, inputting the first key point feature and the second key point feature into the second sub-model to obtain a second prediction result output by the second sub-model, wherein the second prediction result represents a correction to the preliminary predicted position of each key point; and step S105, combining the first prediction result and the second prediction result to obtain a hand key point detection result. Therefore, by using the combined features of high semantic level features and low semantic level features as the input of the second sub-model, namely the refined network, the refined network can predict the offset vector based on richer semantic information, thereby being able to output a more accurate offset vector, thereby improving the accuracy of the final prediction result output by the hand key point detection model.

[0031] According to some embodiments, the image to be detected may be, for example, a single image or a frame in a continuous frame of a video, which is not limited here. The image to be detected may include only one hand or multiple hands, which is not limited here. According to different preset rules, different numbers of hand key points may be set. Exemplarily, the number of hand key points is set to 21 key points from the palm to the fingertips.

[0032] According to some embodiments, the first sub-model may be a convolutional neural network. Figure 2 As shown, the first submodel 202 may include, for example, at least one convolution layer 2021 (and a pooling layer), so as to output at least one feature map based on the input image to be detected 201. The first submodel may also include at least one deconvolution layer 2022, so as to upsample the feature map, and then post-process the upsampled feature map 2023 to make a preliminary prediction of the key point position, so as to obtain a first prediction result 203. Exemplarily, the first feature map is the feature map output by the deconvolution layer closest to the output end.

[0033] According to some embodiments, the first prediction result 203 may be, for example, a Gaussian heat map, and each heat map peak may represent a preliminary predicted position of a key point of a hand in the image to be detected, that is, a rough estimate of the position of the key point of the hand in the image to be detected. By using the Gaussian heat map, the key points of the hand in the image to be detected may be presented in a more intuitive form.

[0034] According to some embodiments, the correction of the initial predicted position of each key point represented by the second prediction result may be, for example, an offset vector. Figure 3 As shown, when the size of the first prediction result is small, or when the confidence of the first prediction result is low, the preliminary prediction position of the key point in the first prediction result corresponds to an area 302 including a center point 301 in the image to be detected, and the real position of the key point should be located at 303. These positions are vectorized based on the origin 300 of the image to be detected, and a vector 304 of the preliminary prediction position, a vector 306 of the real position, and an offset vector 305 representing the correction of the preliminary prediction position can be obtained. Exemplarily, each offset vector can be a vector including an x-direction offset value and a y-direction offset value.

[0035] According to some embodiments, based on the first prediction result 203, a first key point feature 204 is obtained from the first feature map, and the first key point feature 204 includes features corresponding to each key point. The features corresponding to each key point included in the first key point feature may include pixel values ​​of the area corresponding to the preliminary predicted position of each key point in the first feature map. When the size of the first feature map is less than or equal to the size of the first prediction result, the preliminary predicted position of each key point in the first prediction result can correspond to a pixel on the first feature map; and when the size of the first feature map is greater than or equal to the size of the first prediction result, the preliminary predicted position of each key point in the first prediction result can correspond to an area on the first feature map; and then the value of this pixel or all pixels in this area can be used as the first key point feature. Since the first feature map is obtained after at least one round of convolution operation (and deconvolution operation), each pixel in the first feature map is a feature obtained based on an area in the image to be detected, so that it can be used as feature information of a high semantic level. Thus, the first key point features including the features of the area corresponding to the preliminary predicted position of each key point on the first feature map are input into the second sub-model, so that the second sub-model can predict the offset vector based on the high semantic level features corresponding to the preliminary predicted position of the key point.

[0036] According to some embodiments, based on the first prediction result 203, a second key point feature 205 is obtained from the image to be detected 201, and the second key point feature 205 includes features corresponding to each key point. The features corresponding to each key point included in the second key point feature may include pixel values ​​of the area corresponding to the preliminary predicted position of each key point in the image to be detected. Since the first prediction result is a rough estimate of the position of the key points of the hand, the size of the Gaussian heat map output by the first prediction result, i.e., the first sub-model, is less than or equal to the image to be detected. When the first prediction result is smaller than the image to be detected, the preliminary predicted position of each key point in the first prediction result can correspond to an area on the image to be detected; when the first prediction result is comparable to the size of the image to be detected, the pixel corresponding to the preliminary predicted position of each key point in the image to be detected or the pixel and its surrounding area can be selected; and then the values ​​of all pixels in this area can be used as the second key point feature. In contrast to the feature map, each pixel in the image to be detected is a semantic feature of the lowest level. Therefore, the second key point features containing the features of the area corresponding to the preliminary predicted position of each key point on the image to be detected are input into the second sub-model, so that the second sub-model can predict the offset vector based on the low semantic level features corresponding to the preliminary predicted position of the key point.

[0037] According to some embodiments, the size of the first prediction result may be the same as the first feature map. When the size of the first prediction result is the same as the first feature map, each pixel in the first prediction result corresponds one-to-one to each pixel in the first feature map, and therefore, each heat map peak in the first prediction result can correspond to a pixel in the first feature map (i.e., a high semantic level feature vector). The size of the first prediction result may be smaller than the image to be detected. When the size of the first prediction result is smaller than the image to be detected, each pixel in the first prediction result can correspond to an area in the image to be detected, and therefore, each heat map peak in the first prediction result can correspond to multiple pixels in the image to be detected (i.e., multiple low semantic level feature vectors).

[0038] According to some embodiments, the second sub-model 206 may be, for example, a fully connected neural network, a one-dimensional convolutional neural network, or a two-dimensional convolutional neural network. Step S104 may be, for example: arranging the first key point features in a specific order (e.g., the order of key point numbers) to obtain the first key point features in a matrix form, wherein the length of the matrix is ​​the number of key points (e.g., 21), and the width of the matrix is ​​the number of channels of the first feature map; arranging the second key point features in the same order to obtain the second key point features in a matrix form, wherein the length of the matrix is ​​the number of key points, and the width of the matrix is ​​the number of channels of the image to be detected multiplied by the number of pixels in the image to be detected corresponding to each heat map peak; splicing the matrixed first key point features and the second key point features according to the key point correspondence relationship to obtain a spliced ​​feature (the splicing process is not shown in the figure); and inputting the spliced ​​feature into the second sub-model to obtain the second prediction result output by the second sub-model.

[0039] According to some embodiments, the first key point feature may also include structural information of the hand based on the preliminary predicted position of each key point. In addition to including high semantic level features corresponding to the preliminary predicted position of each key point, the first key point feature may also include the relative position relationship between these preliminary predicted positions, so that the structural information of the hand can be retained, and the second sub-model can predict the offset vector of the key point based on the structural relationship and obtain more accurate hand key point detection results.

[0040] According to some embodiments, the second sub-model may be a graph convolutional neural network. A graph convolutional neural network is a powerful neural network architecture for graph data, which can extract features from graph data and then perform corresponding tasks based on these features. Step S104 may, for example, include: constructing a second sub-model input based on the first key point feature and the second key point feature. The second sub-model input is a graph including at least one node and at least one edge, at least one node corresponds to at least one key point of the hand, each of the at least one node stores the features of the key point corresponding to the node included in the first key point feature and the second key point feature, and at least one edge represents the structural information of the hand. Thus, through the above method, graph data that can more fully reflect the hand structure information is constructed as the input of the second sub-model, and a graph convolutional neural network is used to predict the key point offset based on the structural relationship between the preliminary predicted positions in a more optimal way, so as to further improve the accuracy of the model. Exemplarily, the feature vector included in each node in the second sub-model input is obtained by splicing the low semantic level features and the high semantic level features corresponding to the preliminary predicted position of the key point corresponding to the node.

[0041] According to some embodiments, when the image to be detected includes multiple hands, the image to be detected can be input into the first sub-model to obtain a first prediction result including preliminary prediction results for each key point of each hand, but it is necessary to distinguish the preliminary prediction results for the key points of different hands in the first prediction result, and obtain the first key point feature and the second key point feature for each hand based on the first prediction result, and then splice the first key point feature and the second key point feature corresponding to each hand and input them into the second sub-model to obtain the offset vector of the key point of each hand.

[0042] According to some embodiments, Figure 4 As shown, the hand key point detection method may also include: step S401, inputting the image to be detected into the first sub-model to obtain a first prediction result, a first feature map and a second feature map different from the first feature map output by the first sub-model; step S404, based on the first prediction result, obtaining a third key point feature from the second feature map; and step S405, inputting the first key point feature, the second key point feature and the third key point feature into the second sub-model to obtain a second prediction result output by the second sub-model. Figure 4 Steps S402-S403, S406 and Figure 1Thus, by using the mid-semantic level features between the high-semantic level features and the low-semantic level features, the semantic richness of the input features of the second sub-model can be further enriched, thereby improving the accuracy of the offset vector output by the second sub-model, and further improving the accuracy of the final output result.

[0043] According to some embodiments, step S105, combining the first prediction result and the second prediction result to obtain the hand key point detection result, for example, may include translating the initial prediction position of each key point in the first prediction result with the offset vector direction of the corresponding second prediction result to obtain the final prediction result. Figure 2 As shown, in the output layer 207, a vector sum operation is performed on the position vector of the heat map peak of the Gaussian heat map 203 output by the first sub-model 202 and the corresponding offset vector in the second prediction result output by the second sub-model 206 to obtain a Gaussian heat map after the heat map peak position is updated, that is, the hand key point detection result.

[0044] According to another aspect of the present disclosure, a method for training a hand key point detection model is also provided, and the hand key point detection model may include a first sub-model and a second sub-model. Figure 5As shown, the training method may include: step S501, obtaining a sample image and marking a sample true result, wherein the sample image includes a hand, and the sample true result represents the true position of each key point of at least one key point of the hand; step S502, training a first sub-model using the sample image and the sample true result; step S503, inputting the sample image into the trained first sub-model to obtain a first sample prediction result and a first sample feature map output by the first sub-model, wherein the first sample prediction result represents the preliminary predicted position of each key point; step S504, obtaining a first sample key point feature from the first sample feature map based on the first sample prediction result, wherein In the step S505, based on the first sample prediction result, the second sample key point feature is obtained from the sample image, wherein the second sample key point feature includes the feature corresponding to each key point; step S506, the first sample key point feature and the second sample key point feature are input into the second sub-model to obtain the second sample prediction result output by the second sub-model, wherein the second sample prediction result represents the correction of the preliminary prediction position of each key point; and step S507, based on the sample true result, the first sample prediction result and the second sample prediction result, the parameters of the second sub-model are adjusted. Thus, by using the combined features combining high semantic level features and low semantic level features to train the second sub-model, i.e., the refined model, the trained refined model can predict the offset vector based on richer semantic information, so that the refined model can output a more accurate offset vector, and thus a more accurate hand key point detection model can be obtained.

[0045] According to some embodiments, the sample image may be, for example, a single image or a frame in a continuous frame of a video, which is not limited here. The sample image may include only one hand or multiple hands, which is not limited here. According to different preset rules, different numbers of hand key points may be set. Exemplarily, the number of hand key points is set to 21 key points from the palm to the fingertips. In addition, the sample image may be preprocessed, such as different scale scaling, rotation angles, color space disturbance enhancement, etc., to increase the number of samples and enhance the generalization of the trained model. The true result of the sample may be, for example, a Gaussian heat map, and each heat map peak can represent the position of a key point of a hand in the image to be detected.

[0046] According to some embodiments, the first sub-model may be a pre-trained neural network model. Exemplarily, the first sub-model is a pre-trained convolutional neural network model. Since the pre-trained model already has the preliminary ability to recognize or detect targets, the use of the pre-trained model can significantly reduce the training cost of the first sub-model. By further training the first sub-model using sample images and sample true results, the first sub-model can be enabled to have the ability to preliminarily predict the positions of the key points of the hand. Exemplarily, the sample true results are Gaussian heat maps after downsampling, so that the first sub-model can more efficiently calculate the preliminary prediction results of the key points of the hand, so that the model has better performance.

[0047] It can be understood that the relationship between the sample image, the first sample feature map and the first sample prediction result is similar to that of the aforementioned image to be detected, the first feature map and the first prediction result, and will not be elaborated here.

[0048] According to some embodiments, the first sample key point feature may also include structural information of the hand based on the preliminary predicted position of each key point. In addition to including high semantic level features corresponding to the preliminary predicted position of each key point, the first sample key point feature may also include the relative position relationship between these preliminary predicted positions, so that the structural information of the hand can be retained, and the second sub-model can predict the offset vector of the key point based on the structural relationship after training, and obtain more accurate hand key point detection results.

[0049] According to some embodiments, the second sub-model may be a graph convolutional neural network. Step S506 may include: constructing a second sub-model input based on the first sample key point feature and the second sample key point feature, the second sub-model input being a graph including at least one node and at least one edge, at least one node corresponding to at least one key point of the hand, each of the at least one node storing the features of the key point corresponding to the node included in the first sample key point feature and the second sample key point feature, and at least one edge representing the structural information of the hand. Thus, through the above method, graph data that can more fully reflect the hand structure information is constructed as the input of the second sub-model, and the graph convolutional neural network is used to predict the key point offset based on the structural relationship between the preliminary predicted positions in a more optimal way, so as to further improve the accuracy of the trained model. Exemplarily, the feature vector included in each node in the second sub-model input is obtained by splicing the low semantic level features and the high semantic level features corresponding to the preliminary predicted position of the key point corresponding to the node.

[0050] According to some embodiments, Figure 6As shown, the training method may also include: step S603, inputting the sample image into the trained first sub-model to obtain a first sample prediction result, a first sample feature map and a second sample feature map different from the first sample feature map output by the first sub-model; step S606, based on the first sample prediction result, obtaining a third sample key point feature from the second sample feature map; and step S607, inputting the first sample key point feature, the second sample key point feature and the third sample key point feature into the second sub-model to obtain a second sample prediction result output by the second sub-model. Figure 6 Steps S601-S602, S604-S605, and S608 in Figure 5 Thus, by using the mid-semantic level features between the high-semantic level features and the low-semantic level features, the semantic richness of the input features of the second sub-model can be further enriched, thereby improving the accuracy of the offset vector output by the trained second sub-model, and further improving the accuracy of the final output result.

[0051] According to another aspect of the present disclosure, a hand key point detection device is also provided. Figure 7 As shown, the hand key point detection device 700 may include: a first prediction module 701, configured to input the image to be detected into the first sub-model to obtain a first prediction result and a first feature map output by the first sub-model, wherein the image to be detected includes a hand, and the first prediction result represents the preliminary predicted position of each key point of at least one key point of the hand; a first feature acquisition module 702, configured to obtain a first key point feature from the first feature map based on the first prediction result, the first key point feature includes features corresponding to each key point respectively; a second feature acquisition module 703, configured to obtain a second key point feature from the image to be detected based on the first prediction result, the second key point feature includes features corresponding to each key point respectively; a second prediction module 704, configured to input the first key point feature and the second key point feature into the second sub-model to obtain a second prediction result output by the second sub-model, wherein the second prediction result represents a correction to the preliminary predicted position of each key point; and a third prediction module 705, configured to combine the first prediction result and the second prediction result to obtain a hand key point detection result.

[0052] The operations of modules 701 to 705 of the hand key point detection device 700 are similar to the operations of steps S101 to S105 described above, and are not described in detail here.

[0053] According to another aspect of the present disclosure, a training device for a hand key point detection model is also provided, wherein the hand key point detection model includes a first sub-model and a second sub-model. Figure 8 As shown, the training device 800 of the hand key point detection model may include: an acquisition module 801, configured to acquire a sample image and annotate a sample true result, the sample image includes a hand, and the sample true result represents the true position of each key point of at least one key point of the hand; a training module 802, configured to train a first sub-model using the sample image and the sample true result; a first prediction module 803, configured to input the sample image into the trained first sub-model to obtain a first sample prediction result and a first sample feature map output by the first sub-model, the first sample prediction result represents the preliminary predicted position of each key point; a first feature acquisition module 804, configured to acquire a first feature from the first sample feature map based on the first sample prediction result. Sample key point features, the first sample key point features include features corresponding to each key point; the second feature acquisition module 805 is configured to acquire the second sample key point features from the sample image based on the first sample prediction results, the second sample key point features include features corresponding to each key point; the second prediction module 806 is configured to input the first sample key point features and the second sample key point features into the second sub-model to obtain the second sample prediction results output by the second sub-model, wherein the second sample prediction results represent the correction of the preliminary predicted position of each key point; and the parameter adjustment module 807 is configured to adjust the parameters of the second sub-model based on the sample true results, the first sample prediction results and the second sample prediction results.

[0054] The operations of modules 801 to 807 of the training device 800 for the hand key point detection model are similar to the operations of steps S501 to S507 described above, and are not described in detail here.

[0055] According to an embodiment of the present disclosure, an electronic device, a readable storage medium and a computer program product are also provided.

[0056] refer to Fig. 9The structural block diagram of the electronic device 900 that can be used as the server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0057] like Figure 7 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0058] Multiple components in the device 900 are connected to the I / O interface 905, including: an input unit 906, an output unit 907, a storage unit 908, and a communication unit 909. The input unit 906 can be any type of device that can input information to the device 900. The input unit 906 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 907 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 908 can include but is not limited to a disk, an optical disk. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a 1302.11 device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0059] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the method of hand key point detection. For example, in some embodiments, the method of hand key point detection may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the method of hand key point detection described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the method of hand key point detection in any other appropriate manner (e.g., by means of firmware).

[0060] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0061] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0062] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0063] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0064] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0065] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0066] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0067] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for detecting key points of hands, include: Inputting the image to be detected into the first sub-model to obtain a first prediction result and a first feature map output by the first sub-model, wherein the image to be detected includes a hand, and the first prediction result represents a preliminary predicted position of each key point of at least one key point of the hand; Based on the first prediction result, obtaining a first key point feature from the first feature map, where the first key point feature includes features corresponding to each of the key points; Based on the first prediction result, obtaining a second key point feature from the image to be detected, where the second key point feature includes features corresponding to each of the key points; Inputting the first key point feature and the second key point feature into a second sub-model to obtain a second prediction result output by the second sub-model, wherein the second prediction result represents a correction to the preliminary predicted position of each key point; and The first prediction result and the second prediction result are combined to obtain a hand key point detection result.

2. The hand key point detection method according to claim 1, in, The first key point feature also includes structural information of the hand based on the preliminary predicted position of each key point.

3. The hand key point detection method as claimed in claim 2, in, The second sub-model is a graph convolutional neural network. The step of inputting the first key point feature and the second key point feature into the second sub-model comprises: Based on the first key point feature and the second key point feature, a second sub-model input is constructed, wherein the second sub-model input is a graph including at least one node and at least one edge, wherein the at least one node corresponds one-to-one to at least one key point of the hand, and each of the at least one node stores the features of the key point corresponding to the node respectively included in the first key point feature and the second key point feature, and the at least one edge represents the structural information of the hand.

4. The hand key point detection method according to claim 1, in, The features respectively corresponding to each of the key points included in the second key point features include pixel values ​​of an area corresponding to a preliminary predicted position of each of the key points in the image to be detected.

5. The hand key point detection method according to claim 1, in, The features included in the first key point features and corresponding to each of the key points include pixel values ​​of an area in the first feature map corresponding to a preliminary predicted position of each of the key points.

6. The hand key point detection method as claimed in claim 1, further comprising: include: Obtaining a second feature map output by the first sub-model that is different from the first feature map; Based on the first prediction result, obtaining a third key point feature from the second feature map; as well as The first key point feature, the second key point feature and the third key point feature are input into a second sub-model to obtain a second prediction result output by the second sub-model.

7. The hand key point detection method according to claim 1, in, The first prediction result is a Gaussian heat map, which includes a plurality of heat map peaks representing a preliminary predicted position of each key point of the at least one key point of the hand.

8. The hand key point detection method according to claim 7, in, The size of the first prediction result is the same as that of the first feature map, and the size of the first prediction result is smaller than that of the image to be detected.

9. The hand key point detection method according to claim 1, in, The first sub-model is a convolutional neural network.

10. A training method for a hand key point detection model, the hand key point detection model comprising a first sub-model and a second sub-model, the training method include: Acquire a sample image and annotate a sample true result, wherein the sample image includes a hand, and the sample true result represents a true position of each key point of at least one key point of the hand; Training the first sub-model using the sample image and the sample true result; Inputting the sample image into a trained first sub-model to obtain a first sample prediction result and a first sample feature map output by the first sub-model, wherein the first sample prediction result represents a preliminary predicted position of each key point; Based on the first sample prediction result, obtaining first sample key point features from the first sample feature map, where the first sample key point features include features corresponding to each of the key points; Based on the first sample prediction result, obtaining second sample key point features from the sample image, where the second sample key point features include features corresponding to each of the key points; Inputting the first sample key point feature and the second sample key point feature into the second sub-model to obtain a second sample prediction result output by the second sub-model, wherein the second sample prediction result represents a correction to the preliminary predicted position of each key point; as well as Adjust the parameters of the second sub-model based on the true sample result, the first sample prediction result and the second sample prediction result.

11. The training method according to claim 10, in, The first sample key point feature also includes structural information of the hand based on the preliminary predicted position of each key point.

12. The training method according to claim 11, in, The second sub-model is a graph convolutional neural network. The step of inputting the first sample key point feature and the second sample key point feature into the second sub-model includes: Based on the first sample key point feature and the second sample key point feature, a second sub-model input is constructed, wherein the second sub-model input is a graph including at least one node and at least one edge, wherein the at least one node corresponds one-to-one to at least one key point of the hand, and each of the at least one node stores the features of the key point corresponding to the node respectively included in the first sample key point feature and the second sample key point feature, and the at least one edge represents the structural information of the hand.

13. The training method according to claim 10, further comprising: include: Obtaining a second sample feature map output by the first sub-model that is different from the first sample feature map; Based on the first sample prediction result, obtaining a third sample key point feature from the second sample feature map; as well as The first sample key point feature, the second sample key point feature and the third sample key point feature are input into a second sub-model to obtain a second sample prediction result output by the second sub-model.

14. The training method according to claim 10, in, The first sub-model is a pre-trained neural network model.

15. A hand key point detection device, include: A first prediction module is configured to input an image to be detected into a first sub-model to obtain a first prediction result and a first feature map output by the first sub-model, wherein the image to be detected includes a hand, and the first prediction result represents a preliminary predicted position of each key point of at least one key point of the hand; A first feature acquisition module is configured to acquire a first key point feature from the first feature map based on the first prediction result, where the first key point feature includes features corresponding to each of the key points; A second feature acquisition module is configured to acquire second key point features from the image to be detected based on the first prediction result, where the second key point features include features corresponding to each of the key points; a second prediction module configured to input the first key point feature and the second key point feature into a second sub-model to obtain a second prediction result output by the second sub-model, wherein the second prediction result represents a correction to the preliminary predicted position of each key point; and The third prediction module is configured to combine the first prediction result and the second prediction result to obtain a hand key point detection result.

16. A training device for a hand key point detection model, the hand key point detection model comprising a first sub-model and a second sub-model, the training device include: An acquisition module is configured to acquire a sample image and annotate a sample true result, wherein the sample image includes a hand, and the sample true result represents a true position of each key point of at least one key point of the hand; A training module, configured to train the first sub-model using the sample image and the sample true result; A first prediction module is configured to input the sample image into a trained first sub-model to obtain a first sample prediction result and a first sample feature map output by the first sub-model, wherein the first sample prediction result represents a preliminary predicted position of each key point; A first feature acquisition module is configured to acquire first sample key point features from the first sample feature map based on the first sample prediction result, where the first sample key point features include features corresponding to each of the key points; A second feature acquisition module is configured to acquire second sample key point features from the sample image based on the first sample prediction result, where the second sample key point features include features corresponding to each of the key points; A second prediction module is configured to input the first sample key point feature and the second sample key point feature into the second sub-model to obtain a second sample prediction result output by the second sub-model, wherein the second sample prediction result represents a correction to the preliminary predicted position of each key point; as well as A parameter adjustment module is configured to adjust the parameters of the second sub-model based on the actual result of the sample, the first sample prediction result and the second sample prediction result.

17. A computer device, include: a memory, a processor and a computer program stored on the memory, The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1 to 14.

18. A computer-readable storage medium storing a program, the program comprising instructions, which, when executed by a processor of an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 14.

19. A computer program product comprising a computer program, in, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

Citation Information

Patent Citations

  • Table key point detection model training method, apparatus and device, and storage medium

    CN111507265A

  • Method and apparatus for recognizing body movement

    EP3605394A1

  • Image processing method and device

    WO2020233333A1