Image key point detection method and device, equipment and storage medium

Abstract

The invention provides an image key point detection method, device and equipment and a computer readable storage medium. According to the method, for each pixel in a feature map extracted from a target image, hand detection and key point positioning are carried out on a corresponding image area in an original target image based on a multi-dimensional feature vector of the pixel, classification of the image area and prediction of different key points about the left hand and the right hand in the image area are synchronously carried out, and the target image is obtained. And based on a classification result, selecting a key point prediction result as a key point set under the pixel, thereby realizing detection of all hand key points in the original target image. Through the method, a single key point prediction model can be trained, the algorithm development period and cost are reduced, iterative optimization is facilitated, key point positioning difficulty is reduced by distinguishing key point positioning problems of the left hand and the right hand, and positioning precision is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and more specifically, to an image key point detection method, apparatus, device and storage medium. Background Art

[0002] Hand key point localization is an important technology in the field of computer vision, which is to extract the position of hand key points from images or videos. This technology is widely used in many fields such as gesture recognition, human-computer interaction, and biometric recognition.

[0003] Common hand key point localization can be divided into two-dimensional hand key point localization and three-dimensional hand key point localization. Among them, two-dimensional hand key point localization is to extract the coordinates of the hand key points in the two-dimensional coordinate system (i.e., x, y coordinates) from the image. Due to the development of Internet technology and the popularization of image acquisition equipment, the two-dimensional hand key point algorithm can collect massive and rich image data for training models. However, since the two-dimensional image is only a projection of the three-dimensional hand structure on the imaging plane, the collected image is prone to hand occlusion problems and cannot reflect the complete three-dimensional structure of the hand. Three-dimensional hand key point localization adds the prediction of the depth z of each key point on the basis of extracting x, y coordinates, which can better reflect the spatial structure of the hand. However, since there is no depth information in general images, data collection needs to be carried out in the laboratory, resulting in the inability to include rich environmental changes and it is difficult to collect samples close to the actual environment, affecting the final generalization ability of the model.

[0004] Therefore, an efficient and accurate image key point detection method is needed. Summary of the invention

[0005] In order to solve the above problems, the present invention simultaneously performs hand key point detection and positioning by utilizing encoded consistent two-dimensional hand key point samples and three-dimensional hand key point samples to train a single key point prediction model including hand detection and hand key point positioning sub-networks.

[0006] Embodiments of the present disclosure provide an image key point detection method, apparatus, device, and computer-readable storage medium.

[0007] An embodiment of the present disclosure provides an image key point detection method, comprising: acquiring a target image; extracting a feature map of the target image, each pixel in the feature map having a multidimensional feature vector associated with a corresponding image region in the target image; and for each pixel in the feature map: determining a classification of the image region corresponding to the pixel based on the multidimensional feature vector of the pixel, the classification comprising left hand and right hand; predicting a first key point set in the image region when the classification of the image region corresponding to the pixel is a left hand, and a second key point set in the image region when the classification of the image region is a right hand, based on the multidimensional feature vector of the pixel; and selecting one from the first key point set and the second key point set as the key point set corresponding to the pixel based on the determined classification of the image region corresponding to the pixel.

[0008] An embodiment of the present disclosure provides an image key point detection device, comprising: an image acquisition module, configured to acquire a target image; a feature extraction module, configured to extract a feature map of the target image, each pixel in the feature map having a multidimensional feature vector associated with a corresponding image region in the target image; and a key point prediction module, configured to, for each pixel in the feature map: determine, based on the multidimensional feature vector of the pixel, a classification of the image region corresponding to the pixel, the classification comprising left hand and right hand; predict, based on the multidimensional feature vector of the pixel, a first key point set in the image region when the classification of the image region corresponding to the pixel is left hand, and a second key point set in the image region when the classification of the image region is right hand; and select one from the first key point set and the second key point set as the key point set corresponding to the pixel based on the determined classification of the image region corresponding to the pixel.

[0009] An embodiment of the present disclosure provides an image key point detection device, comprising: one or more processors; and one or more memories, wherein a computer executable program is stored in the one or more memories, and when the computer executable program is executed by the processor, the image key point detection method as described above is performed.

[0010] An embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, and when the instructions are executed by a processor, the method for detecting key points in an image as described above is implemented.

[0011] The embodiment of the present disclosure provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image key point detection method according to the embodiment of the present disclosure.

[0012] Compared with the traditional hand key point positioning method, the method provided by the embodiment of the present disclosure can simultaneously perform hand detection and hand key point positioning through multi-task learning training of a single key point prediction model, which reduces the algorithm development cycle and cost, is conducive to iterative optimization, and simultaneously utilizes two-dimensional and three-dimensional labeled data, which can not only provide rich scene changes, but also accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved.

[0013] The method provided by the embodiment of the present disclosure performs hand detection and key point positioning on the corresponding image area in the original target image based on the multi-dimensional feature vector of each pixel in the feature map extracted from the target image, wherein the classification of the image area and the prediction of different key points about the left and right hands in the image area are performed synchronously, and the key point prediction result is selected as the key point set under the pixel based on the classification result, thereby realizing the detection of all hand key points in the original target image. The method of the embodiment of the present disclosure can train a single key point prediction model that performs hand detection and hand key point positioning at the same time, which reduces the algorithm development cycle and cost, is conducive to iterative optimization, and by utilizing two-dimensional and three-dimensional labeled data, it can provide rich scene changes and accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure, and a person of ordinary skill in the art can obtain other drawings based on these drawings without creative work.

[0015] Figure 1 is a schematic diagram showing a scenario of processing an input target image according to an embodiment of the present disclosure;

[0016] Figure 2 is a flowchart showing a conventional hand key point detection method;

[0017] Figure 3A is a flow chart showing a method for detecting key points in an image according to an embodiment of the present disclosure;

[0018] Figure 3B is a schematic diagram showing a flow chart of an image key point detection method according to an embodiment of the present disclosure;

[0019] Figure 4 is a schematic diagram showing the flow of a single-step positioning method according to an embodiment of the present disclosure and a traditional two-step positioning method;

[0020] Figure 5 is a schematic diagram showing key point detection results for the left hand and the right hand according to an embodiment of the present disclosure;

[0021] Figure 6 is a schematic diagram showing a flow chart of training a key point prediction model according to an embodiment of the present disclosure;

[0022] Figure 7 is a schematic diagram showing a final key point prediction of a hand object and a mask vector corresponding to a two-dimensional key point sample and a three-dimensional key point sample according to an embodiment of the present disclosure;

[0023] Figure 8 is a schematic diagram showing an image key point detection device according to an embodiment of the present disclosure;

[0024] Fig. 9 A schematic diagram showing an image key point detection device according to an embodiment of the present disclosure; and

[0025] Fig.10 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0027] In this specification and the accompanying drawings, substantially the same or similar steps and elements are represented by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance or ranking.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0029] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0030] The image key point detection method disclosed in the present invention may be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. For example, for the image key point detection method based on artificial intelligence, it can analyze each pixel in the feature map extracted from the target image in a manner similar to that of humans identifying human hands in the image through the naked eye, so as to identify the hand object from the corresponding image area in the target image and perform key point detection on it, thereby determining the position of the hand key point in the target image.

[0031] The image key point detection method disclosed in the present invention can be based on deep learning. Deep learning is an algorithm in machine learning that is based on characterizing and learning data. Observations (e.g., an image) can be represented in a variety of ways, such as a vector of intensity values ​​for each pixel, or more abstractly represented as a series of edges, regions of specific shapes, etc. Using certain specific representation methods makes it easier to learn tasks from examples (e.g., face recognition or hand recognition). The benefit of deep learning is to replace manual feature acquisition with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction. Optionally, the image key point detection method disclosed in the present invention can be based on a deep fully convolutional neural network (FCNN). FCNN can classify images at the pixel level (that is, classify each pixel point), and it can accept input images of any size, and use a deconvolution layer to upsample the feature map of the last convolution layer to restore it to the same size as the input image, so that a prediction can be generated for each pixel, while retaining the spatial information in the original input image, and finally classifying the pixels on the upsampled feature map. In the image key point detection method disclosed in the present invention, FCNN can be used to extract image features of the target image to output a feature map containing several pixels, each of which corresponds to an image area (receptive field) in the original target image. By analyzing each pixel, hand key point detection of the original target image can be achieved.

[0032] The image key point detection method disclosed in the present invention can be based on computer vision (CV) technology. Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, three-dimensional technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0033] For example, the image key point detection method disclosed in the present invention can be used for hand key point detection. Hand key point detection aims to extract the position of hand key points from images or videos, which is an important technology in the field of computer vision. This technology is widely used in many fields such as gesture recognition, human-computer interaction, and biometric recognition. Common hand key point positioning methods can be divided into image processing-based methods and deep learning-based methods. Among them, the image processing-based method uses the domain knowledge of professionals combined with the basic methods of digital image processing to extract hand key points from images. This type of method does not require manual labeling of samples, which can shorten the development cycle to a certain extent, but requires more sophisticated parameter adjustment and cannot adapt to complex environmental changes. The machine learning-based method is the current mainstream key point positioning method. This method can automatically learn effective information from massive labeled data to locate hand key points, and has stronger robustness, but the heavy work of sample labeling in the early stage requires a lot of manpower and material investment.

[0034] In summary, the solutions provided by the embodiments of the present disclosure involve technologies such as artificial intelligence, deep learning, computer vision, and hand key point detection. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0035] Figure 1 is a schematic diagram of a scene showing processing of an input target image according to an embodiment of the present disclosure.

[0036] like Figure 1As shown, the user can send the target image (or video frame) to be processed to the server through the user terminal, so that the server can perform key point detection processing on the target image. Subsequently, the server can return the result of the key point detection (for example, the key point set of the target image) to the user terminal through the network, for example, displayed on the image interface of the user terminal. Optionally, the user terminal can specifically include a smart phone, a tablet computer, a laptop portable computer, a desktop computer, a vehicle-mounted terminal, a wearable device, etc., but is not limited to this. For example, the user terminal can also be an image / video acquisition device, etc., so that the collected image / video is directly sent to the processor end for processing. The user terminal can also be a client that installs a browser or various applications (including system applications and third-party applications). Optionally, the network can be an Internet of Things based on the Internet and / or telecommunications network, which can be a wired network or a wireless network. For example, it can be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a cellular data communication network, etc., which can realize an electronic network that can realize an information exchange function. The user terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application is not limited here. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Of course, in addition to the above example scenario where the server performs image processing and returns the processing results to the user terminal, the image processing method of the present disclosure can also be directly executed by the user terminal without being implemented through the network and server.

[0037] The ability to perceive the shape and movement of the hand is a prerequisite for improving the user experience in various technical fields and platforms. For example, it can form the basis of sign language understanding and gesture control, and can also overlay digital content and information on the physical world in augmented reality. Robust real-time hand perception is undoubtedly a challenging computer vision task because hands are often occluded (e.g., occlusions from finger / palm bending, and occlusions such as handshakes) and lack high-contrast patterns.

[0038] As mentioned above, common hand key point positioning can be divided into two-dimensional hand key point positioning and three-dimensional hand key point positioning according to the dimension of the labeled data. Two-dimensional hand key point positioning is to extract the x and y coordinates of the hand key points from the image. Due to the development of Internet technology and the popularization of image acquisition equipment, the two-dimensional hand key point algorithm can collect a large amount of image data with rich changes for training models. However, since the two-dimensional image is only a projection of the three-dimensional hand structure on the imaging plane, the collected image is prone to hand occlusion problems and cannot reflect the complete three-dimensional structure of the hand. Three-dimensional hand key point positioning adds the prediction of the depth z of each key point on the basis of extracting x and y coordinates, which can better reflect the spatial structure of the hand. However, since there is no depth information in general images, data collection needs to be carried out in the laboratory, which cannot contain rich environmental changes and it is difficult to collect samples close to the actual environment, affecting the final generalization ability of the model. In addition, the commonly used hand key point positioning methods can also be divided into image processing-based methods and deep learning-based methods according to the processing method. The image processing-based method uses the domain knowledge of professionals combined with the basic methods of digital image processing to extract hand key points from the image. This type of method does not require manual labeling of samples, which can shorten the development cycle to a certain extent, but requires more sophisticated parameter adjustments and cannot adapt to complex environmental changes. The machine learning-based method is the current mainstream key point positioning method. This method can automatically learn effective information from massive labeled data to locate the key points of the hand, and has stronger robustness. However, the heavy work of sample labeling in the early stage requires a lot of manpower and material resources.

[0039] Figure 2 FIG. 1 is a flowchart showing a conventional method for detecting key points of a hand. Figure 2 As shown, the hand key point detection method can mainly include two parts: a hand detection module and a hand key point positioning module. Among them, in the hand detection module, the collected image can be preprocessed first, including normalizing the image to a standard size, and then the image can be divided into different regions, and each region can be independently judged, that is, whether the region contains a hand target. Further, for the area containing the hand target, the circumscribed rectangle containing the hand target (i.e., the hand rectangle) can be predicted to obtain a more accurate hand position. Next, in the key point positioning module, the input hand area (i.e., the image area within the hand rectangle) can be first normalized to scale it to a standard size. Then, the image can be input into the key point positioning model to obtain the position of the hand key points relative to the input image, and finally the hand key points can be transformed to the original image scale.

[0040] In the hand key point detection method described above, first, after completing the hand detection, the hand key point detection module needs to be called once for each hand area. Its positioning speed decreases sharply with the increase in the number of hand targets in the image. In scenes with a large number of hand targets, it is easy to generate a large time overhead. In addition, such a method requires training two independent modules at the same time, which increases the cycle and cost of algorithm development and is not conducive to iterative optimization. Secondly, this method does not distinguish between the key point detection of the left and right hands. However, although the left and right hands are similar, they actually have a mirror relationship and have different texture features under the same key point form. The current method increases the difficulty of the problem and limits the improvement of model accuracy. Thirdly, this method cannot use three-dimensional and two-dimensional labeled data at the same time, resulting in the inability to accurately describe the structural information of the hand target when only two-dimensional input is used, and the inability to provide rich scene changes when only three-dimensional data is used.

[0041] Based on this, the present disclosure provides an image key point detection method, which simultaneously performs hand key point detection and positioning by utilizing encoded consistent two-dimensional hand key point samples and three-dimensional hand key point samples to train a single key point prediction model including hand detection and hand key point positioning sub-networks.

[0042] Compared with the traditional hand key point positioning method, the method provided by the embodiment of the present disclosure can simultaneously perform hand detection and hand key point positioning through multi-task learning training of a single key point prediction model, which reduces the algorithm development cycle and cost, is conducive to iterative optimization, and simultaneously utilizes two-dimensional and three-dimensional labeled data, which can not only provide rich scene changes, but also accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved.

[0043] The method provided by the embodiment of the present disclosure performs hand detection and key point positioning on the corresponding image area in the original target image based on the multi-dimensional feature vector of each pixel in the feature map extracted from the target image, wherein the classification of the image area and the prediction of different key points about the left and right hands in the image area are performed synchronously, and the key point prediction result is selected as the key point set under the pixel based on the classification result, thereby realizing the detection of all hand key points in the original target image. The method of the embodiment of the present disclosure can train a single key point prediction model that performs hand detection and hand key point positioning at the same time, which reduces the algorithm development cycle and cost, is conducive to iterative optimization, and by utilizing two-dimensional and three-dimensional labeled data, it can provide rich scene changes and accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved.

[0044] Figure 3A is a flow chart showing an image key point detection method 300 according to an embodiment of the present disclosure. Figure 3B Detailed description is given of the flowchart of a method for detecting key points in an image according to an embodiment of the present disclosure.

[0045] like Figure 3B As shown, the image key point detection method disclosed in the present invention can mainly include two stages: feature extraction and detection and positioning. In the feature extraction stage, feature maps are extracted from the target image to obtain image information about each image area in the target image. In the detection and positioning stage, the extracted feature maps can be processed, including using the feature maps to simultaneously perform target detection and key point positioning. Figure 3A The image key point detection method disclosed in the present invention is described in detail.

[0046] In step 301, a target image may be acquired. Optionally, the target image may be a single image or a video frame from a video, and the target image may or may not include a target object to be detected (for example, a hand target in the present disclosure). For example, the target image may be an image of the current head-mounted display perspective such as captured by a head-mounted display device, or an image frame such as captured by a video capture device, and the present disclosure does not limit this.

[0047] In step 302, a feature map of the target image may be extracted, and each pixel in the feature map may have a multi-dimensional feature vector associated with a corresponding image region in the target image.

[0048] Optionally, before feature extraction is performed on the acquired target image, a preprocessing operation may be performed on it, for example, including but not limited to operations such as scaling, cropping and splicing on the target image, so that it meets the image requirements of the input feature extraction network.

[0049] According to an embodiment of the present disclosure, extracting the feature map of the target image may include: using a pre-trained convolutional neural network to extract the feature map of the target image, the multidimensional feature vector of each pixel in the feature map may indicate image information in a corresponding image area in the target image, and the corresponding image area is related to the receptive field area of ​​the convolutional neural network.

[0050] like Figure 3B As shown, the feature map of the target image can be extracted by a deep model including a pre-trained convolutional neural network. Optionally, the input target image can be first normalized to a standard size and then fed into the convolutional neural network to perform feature transformation on the target image, so as to transform the original image into a high-dimensional space, so as to obtain the high-dimensional image features of the target image in the high-dimensional space.

[0051] Optionally, a convolutional neural network (e.g., a deep fully convolutional neural network) can be used to extract image features. For example, for an image with an input dimension of W×H, a convolutional neural network can output a feature map with a dimension of w×h×c, that is, an image with a length of w and a width of h, where each pixel in the image contains a c-dimensional feature vector that is used to describe the image information in the corresponding receptive field in the original target image. Among them, c is the dimension of the features extracted by the neural network, and the size of c is the dimension of projecting the input image into the feature space. That is to say, the larger c is, the more complex the model is and the stronger the descriptive ability is, but at the same time there is a greater risk of overfitting, and more training samples are required to obtain ideal results; on the contrary, when c is small, the model is simpler and it is difficult to describe the complete sample changes. Therefore, c is a hyperparameter involved in building a network model, and its value is usually adjusted during model training.

[0052] As an example, the deep feature extraction process is the calculation process of the deep convolutional neural network. The common deep convolutional neural network structure can be stacked by basic network units. For example, a general network unit can include the following parts: a linear transformation layer, such as a "convolutional network layer" or a "fully connected network layer"; a regularization layer, such as a "batch regularization layer" or an "instance regularization layer"; and a nonlinear layer, such as a sigmoid layer or a tanh layer. In the feature extraction stage, the input target image can be calculated by multiple network units to finally obtain abstract image features.

[0053] The above-mentioned deep model can be pre-trained. During the training process of the deep model, the parameters of the deep model can be adjusted so that the model can obtain the image features most relevant to the positioning target from the target image for subsequent detection and positioning.

[0054] It should be understood that the image key point detection method disclosed in the present invention does not rely on the network structure of a specific feature extraction model. It can select a suitable basic network according to the requirements for detection accuracy, calculation speed, etc. in different application scenarios. For example, common basic networks may include but are not limited to: ResNet, DenseNet, etc. Optionally, when the image key point detection method disclosed in the present invention is implemented in an edge computing device (such as virtual reality (VR) glasses, mobile phones and other mobile devices), due to the limitations of computing power and power consumption, it is necessary to adopt a model that is as simple as possible. At this time, it is necessary to reduce the complexity of the model and sacrifice some accuracy to ensure the speed of the model. In addition, the image key point detection method disclosed in the present invention is implemented on a personal computer (PC) or a high-end computing platform (such as a desktop computer, workstation and other equipment). Since the platform itself has powerful computing power, the complexity of the model can be appropriately increased to make more accurate predictions. The image key point detection method disclosed in the present invention does not limit the specific network structure of the deep model for feature extraction.

[0055] After obtaining the feature map of the target image in the high-dimensional space through the above feature extraction, the extracted feature map can be used to complete the detection and positioning operations through multi-task learning in the detection and positioning stage.

[0056] In step 303, for each pixel in the feature map, operations shown in steps 3031-3033 may be performed.

[0057] Figure 4 is a schematic diagram showing the flow of a single-step positioning method according to an embodiment of the present disclosure and a traditional two-step positioning method, wherein: Figure 4 (a) in FIG. 4 shows the two-step positioning method used in the traditional hand key point detection method. Figure 4 (b) shows the single-step positioning method used in the image key point detection method disclosed in the present invention.

[0058] As mentioned above and with reference to Figure 4 In (a), the traditional hand key point detection method requires a cascade detection and positioning model to complete the final hand key point detection work, wherein the target image can output multiple hand regions contained in the target image (for example, hand image 1, ..., hand image n, etc.) through the hand region detection module, and discard the image region without hand targets. Since the output result of the hand region detection needs to be scaled and normalized (i.e., Figure 4 The image in (a) must be normalized before it can be sent to the hand key point localization module, so it is difficult to adopt an end-to-end training framework, which requires independent training of two models, increasing the difficulty of development. Moreover, when there are multiple hand targets in the target image, the hand key point localization model needs to be called independently for each hand target, which increases the overall time consumption of hand key point detection.

[0059] In the single-step key point positioning method disclosed in the present invention, Figure 4 As shown in (b), after obtaining a feature map from the target image through common feature extraction, the feature map can be input into two independent modules for independent prediction: the hand region detection module and the hand key point positioning module. The hand region detection module can be used to predict the category of the original image region corresponding to the current pixel in the feature map (including predicting whether the current pixel belongs to the detection target, that is, whether there is a detection target in the original image region corresponding to the pixel), while the hand key point positioning module can be used to predict the key points in the current image region. The above two modules can use multi-task learning to share the image features extracted by the feature extraction network to complete detection and positioning respectively.

[0060] The following will refer to Figure 3BThe processing of the detection and positioning stage of the present disclosure (ie, the processing in the above step 303) is described in detail.

[0061] In step 3031, a classification of the image region corresponding to the pixel may be determined based on the multi-dimensional feature vector of the pixel, and the classification may include left hand and right hand.

[0062] Alternatively, if Figure 3B As shown, for each pixel in the feature map output by the depth model, target detection can be performed based on the multi-dimensional feature vector of the pixel. For example, in an embodiment of the present disclosure, for a hand target, the classification of the image area in the original target image corresponding to the pixel can be determined based on the multi-dimensional feature vector of the pixel, that is, whether there is a hand target, and the specific classification when it is determined that there is a hand target, such as left hand and right hand.

[0063] Figure 5 is a schematic diagram showing the key point detection results of the left hand and the right hand according to an embodiment of the present disclosure. Figure 5 As shown in the figure, the left hand and the right hand have similar structures, but they are actually mirror images, that is, the texture information presented by the left and right hands is not consistent for the same key point distribution. However, in the existing key point localization methods, the same model is usually used for key point localization for the left and right hands, ignoring the mirror image relationship between the two hands. This requires predicting the same key point distribution for completely different image textures, which increases the complexity of the model in dealing with key point detection problems.

[0064] Therefore, in view of the above problems, the image key point detection method disclosed in the present invention can distinguish the left and right hands in the detection and positioning stage, including in the detection part (for example, in Figure 4 In the hand region detection module shown in (b) of FIG. 1 ), the image region is classified with respect to the left and right hands, and in the positioning part (for example, in the Figure 4 In the hand key point localization module shown in (b) in FIG, the key points of the left and right hands are predicted separately.

[0065] According to an embodiment of the present disclosure, determining the classification of the image area corresponding to the pixel may include: detecting whether there is a target object in the image area, the target object may include the left hand and the right hand of a human body; in the case of detecting the presence of the left hand of a human body in the image area, determining that the image area is classified as the left hand; and in the case of detecting the presence of the right hand of a human body in the image area, determining that the image area is classified as the right hand.

[0066] Optionally, in Figure 4In the hand region detection module shown in (b) of FIG. 1 , the above-mentioned determination of the classification of the image region corresponding to the pixel can be performed, which can perform target detection and classification based on the multi-dimensional feature vector of the pixel, and output the prediction result that the image region corresponding to the pixel belongs to each classification. For example, the prediction result can be output in the form of probability, that is, the probability that the image region belongs to each classification.

[0067] In addition to the above-mentioned classification of the image regions of the left hand and the right hand, there may also be a region that does not include the hand target, in which case the region may be considered as part of the background region. Therefore, according to an embodiment of the present disclosure, the classification may also include the background.

[0068] Correspondingly, the determination of the classification of the image area corresponding to the pixel may also include: in the case where it is detected that the left hand and the right hand of the human body do not exist in the image area, determining that the image area is classified as background. Optionally, when it is determined that the image area corresponding to a certain pixel belongs to the background, since the detection target of the present disclosure (i.e., the hand target) does not exist therein, subsequent processing may no longer be performed on the image area. In other words, in the case where it is determined that the image area corresponding to the pixel is classified as background, the subsequent positioning processing of the pixel can be skipped, and the detection and positioning of the next pixel can be directly entered.

[0069] In step 3032, based on the multidimensional feature vector of the pixel, a first set of key points in the image area when the image area corresponding to the pixel is classified as a left hand, and a second set of key points in the image area when the image area is classified as a right hand can be predicted.

[0070] Optionally, as described above, in the image key point detection method disclosed in the present invention, a separate detection method can be used for the left and right hands, that is, in the example Figure 4 The hand key point localization module shown in (b) simultaneously uses two processing branches to predict two groups of key points, corresponding to the key points when the image area is classified as the left hand and the key points when the image area is classified as the right hand.

[0071] According to an embodiment of the present disclosure, the classification of the image region corresponding to the pixel can be performed by a pre-trained detection sub-network, and the prediction of the first key point set and the second key point set can be performed by a pre-trained key point localization sub-network. Specifically, the detection sub-network and the key point localization sub-network can correspond to the above-mentioned Figure 4 The hand area detection module and the hand key point localization module shown in (b) in the figure can share the feature map of the target image output by the deep model in the detection and localization stage and perform corresponding operations respectively.

[0072] According to an embodiment of the present disclosure, in the key point localization subnetwork, two different localization models may be used to respectively predict the key point set in the image region corresponding to the pixel when the image region is classified as the left hand and the right hand. As described above, different localization branches may be set for the left hand and the right hand in the key point localization subnetwork, and two different parameterized models may be used to predict the feature points based on the same feature input to obtain the key point sets in the two cases.

[0073] In step 3033, based on the determined classification of the image region corresponding to the pixel, one of the first key point set and the second key point set may be selected as the key point set corresponding to the pixel.

[0074] Alternatively, if Figure 3B As shown, when it is determined that there is a detection target in the image area, and the key point prediction when the image area is classified as a left hand and the key point prediction when the image area is classified as a right hand are obtained respectively through two processing branches, a selection can be made from the two key point predictions based on the classification of the image area, that is, a set of key point predictions corresponding to the classification are selected as the final key points.

[0075] As an example, the above steps can be expressed as:

[0076]

[0077] Among them, kpt represents the prediction result of the final key point, where kpt L kpt is the key point prediction result when the image area is classified as left hand. R is the key point prediction result when the image area is classified as right hand, P L , P R , P BG denote the probability of the image region output by the detection subnetwork being classified as the left hand, right hand, and background, respectively. That is, when the image region output by the detection subnetwork is classified as the left hand (P L ≥P R ≥P BG ), the final key point prediction result adopts the key point prediction result when the image area is classified as left hand, and the image area classified as right hand (P BG ≤P L <P R ), the final key point prediction result adopts the key point prediction result when the image area is classified as right hand.

[0078] In addition, the classification of the above image area is background (ie, PBG >P L And P BG >P R ), the image key point detection method disclosed in the present invention may further include: for each pixel in the feature map, when the image area corresponding to the pixel is classified as background, not performing the prediction and selection on the pixel. That is, when the detection subnetwork determines that there is no detection target in the image area corresponding to the pixel, no subsequent processing is performed on the image area, but directly transfers to the detection and positioning processing of the next pixel.

[0079] According to an embodiment of the present disclosure, the detection subnetwork and the key point localization subnetwork may constitute a key point prediction model, which takes a feature map of the target image as input and a set of key points in the target image as output, and each key point in the key point set may have depth information.

[0080] The above-mentioned image key point detection method disclosed in the present invention can enable the detection subnetwork and the key point positioning subnetwork to perform multi-task learning and learn more robust image features. More specifically, more advanced hand structure information can be added to the detection subnetwork, and richer input image changes can be added to the key point positioning subnetwork. In addition, complete end-to-end training can be achieved to achieve better key point detection results through a single key point prediction model. In addition, the image key point detection method disclosed in the present invention is not affected by the number of hand targets present in the target image, and the detection and positioning of all hand targets in the target image can be completed through a single forward calculation of the neural network.

[0081] The following will refer to Figure 6 and Figure 7 The training process of the key point prediction model including the detection subnetwork and the key point localization subnetwork is introduced in detail.

[0082] Figure 6 is a schematic diagram showing a flow chart of training a key point prediction model according to an embodiment of the present disclosure.

[0083] According to an embodiment of the present disclosure, the pre-trained detection sub-network and the pre-trained key point localization sub-network may be obtained by training the key point prediction model. Figure 6 As shown, in the model training process of the present invention, unified training of the detection subnetwork and the key point positioning subnetwork can be achieved by performing end-to-end training on the key point prediction model including the detection subnetwork and the key point positioning subnetwork.

[0084] According to an embodiment of the present disclosure, the key point prediction model may be trained based on a training image set including two-dimensional key point samples and three-dimensional key point samples, and the three-dimensional key point samples may have depth information. Specifically, each training image in the training image set may include one or more hand objects, each of the one or more hand objects may have a two-dimensional key point sample or a three-dimensional key point sample for training, the two-dimensional key point sample may indicate the two-dimensional coordinates of the key point associated with the hand object in the training image, the three-dimensional key point sample may indicate the three-dimensional coordinates of the key point associated with the hand object in the training image, and the three-dimensional coordinates may include the two-dimensional coordinates and depth information.

[0085] Alternatively, if Figure 6 As shown, the training data for key point prediction model training may include a training image set and key point labeled samples of each training image in the training image set. The training images in the training image set may be two-dimensional images or video frames that are the same as the target images, and the key point labeled samples of each training image may be two-dimensional key point samples or three-dimensional key point samples.

[0086] Specifically, the two-dimensional key point sample can mark the key points of the hand in the two-dimensional image. Due to the development of Internet technology and the popularization of image acquisition equipment, the acquisition of hand images in a two-dimensional natural environment has become very convenient. However, since the two-dimensional image is only a projection of the hand on the image plane, there is inevitably a problem of key point occlusion. For the occluded key points, the marker needs to infer the position of the key points based on experience, and for the key points that cannot be determined, the confidence needs to be set to 0. This puts higher requirements on the marking work, and it is difficult to maintain a stable marking quality due to manual marking. In order to ensure quality, different markers are usually required to repeatedly mark the same target, which greatly increases the marking cost. The three-dimensional key point sample can better overcome the self-occlusion problem of the hand image, but it requires a complex image acquisition system and cannot be calibrated directly from the two-dimensional image. For example, common image acquisition systems may include but are not limited to: an acquisition system based on binocular / multi-camera cameras, which selects angles without self-occlusion from images acquired from different angles for marking; sensor-based marking, which requires wearing sensors on the hands to reduce the impact of occlusion. These acquisition systems can usually only perform image acquisition under stable laboratory conditions and cannot simulate the complex environmental changes in the actual production environment, which affects the generalization ability of the training model. Table 1 below gives a comparison of the labeling of the above two key point samples.

[0087] Table 1 Comparison of samples with different markers

[0088] Sample category quantity Marking Difficulty Environmental Information Structural information 2D keypoint samples many high Rich 2D 3D key point samples few Low single 3D

[0089] For the above two types of key point samples, the image key point detection method disclosed in the present invention can unify key point samples of different marking types into the same training framework, thereby simultaneously training the key point prediction model by utilizing key point samples of different marking types, so that the key point prediction model can learn the rich environmental information of two-dimensional images and the rich structural information of three-dimensional images, thereby achieving the purpose of improving the key point positioning accuracy.

[0090] According to an embodiment of the present disclosure, in the training of the key point prediction model, a training label with a unified format can be set for each hand object in each training image in the training image set, and the format can be a vector with a predetermined dimension, and the predetermined dimension can include the classification dimension, the key point dimension when the classification of the image area corresponding to the hand object is predicted to be the left hand, and the key point dimension when the classification of the image area corresponding to the hand object is predicted to be the right hand, wherein the key point dimension can be related to the number of key points of the hand object and the dimension of the three-dimensional coordinates of each key point.

[0091] Optionally, the sample input and key point detection results of the training image can be in units of hand objects. Specifically, a set of key point samples (which can be two-dimensional key point samples or three-dimensional key point samples) can be collected for each hand object in the training image, and the output of the key point prediction model can also be a set of key point location predictions for each hand object in the training image.

[0092] As described above, each hand object in the training image may be provided with a training label having a unified format, wherein the unified format may include a vector having a fixed length and a length partition setting, wherein the length partition setting of the vector may include a length interval for representing the aforementioned classification prediction, a length interval for representing a key point set when the classification prediction of the image area corresponding to the hand object is the left hand, and a length interval for representing a key point set when the classification prediction of the image area corresponding to the hand object is the right hand. For example, in order to encode the labeled samples under different labeling rules into a unified label format, in an embodiment of the present disclosure, the training label format of the hand object may be uniformly set to have a fixed length N cls +3N p +3N p A vector, wherein the length division setting of the vector includes three length intervals, and the three length intervals correspond to N cls 、3N p and 3N p Among them, N clsIt can represent the dimension of the subdivision category. For example, here, different labeled samples can include three subdivision categories: left hand, right hand, and background. Therefore, N cls =3, where each dimension can indicate the probability that the hand object belongs to each category. p It can represent the number of hand key points. Here, as an example, according to the commonly used hand key point marking rules, each hand can contain 21 hand key points, so N p =21,3N p N p The three-dimensional coordinate dimension of the hand key points (i.e., key point dimension). p +3N p That is, the key point positioning results when the classification of the image area is predicted as the left hand and the right hand respectively. The final key point prediction result of the key point prediction model can have 3N p Of course, the above-mentioned setting method of the training label is only used as an example and not as a limitation in the present disclosure, and the image key point detection method of the present disclosure can also adopt various other setting methods.

[0093] According to an embodiment of the present disclosure, in the training of the key point prediction model, a mask vector with the same dimension as the key point dimension can also be set for each hand object in each training image in the training image set, so as to uniformly process the two-dimensional key point samples and the three-dimensional key point samples.

[0094] Optionally, in order to make the two-dimensional key point samples and the three-dimensional key point samples have a unified representation, a mask data structure, i.e., a mask vector, with the same dimension as the final key point output (i.e., the same dimension as the above-mentioned key point dimension) can be set for each hand object.

[0095] Figure 7 is a schematic diagram showing the final key point prediction of the hand object and the mask vectors corresponding to the two-dimensional key point samples and the three-dimensional key point samples according to an embodiment of the present disclosure. Figure 7 As shown, the final key point prediction of the hand object and the mask vectors corresponding to the two-dimensional key point samples and the three-dimensional key point samples (ie, 2D mask vectors and 3D mask vectors) have the same format, where the three-dimensional coordinate dimension of each key point is used as the unit.

[0096] Among them, according to an embodiment of the present disclosure, when two-dimensional key point samples are used for training the hand object, the value of the dimension used to indicate depth information in the three-dimensional coordinate dimension of each key point in the mask vector of the hand object can be set to 0, and the values ​​of the remaining two-dimensional coordinate dimensions can all be set to 1; and when three-dimensional key point samples are used for training the hand object, the value of the three-dimensional coordinate dimension of each key point in the mask vector of the hand object can all be set to 1.

[0097] Alternatively, if Figure 7 As shown, the final key point prediction result of each hand label can correspond to a hand mask vector of the same dimension. Specifically, when the input is a three-dimensional key point sample, the masks of all dimensions in the mask vector are 1, and when the input is a two-dimensional key point sample, the masks corresponding to the depth bits in the mask vector are all set to 0.

[0098] The specific encoding method of the training labels of the hand objects under the above two marking rules will be introduced below.

[0099] For 3D key point samples, the original data usually contains the image and the world coordinates of the hand in the image in the real 3D space. Therefore, the world 3D coordinates can be converted to the camera 3D coordinates using the following equation:

[0100]

[0101] Among them, X w , Y w , Z w Represents the coordinates of the key points of the hand in the world coordinate system, and They are the rotation matrix and translation matrix obtained through camera calibration, X c , Y c , Z c is the coordinates of the hand key points in the transformed camera coordinate system.

[0102] Next, the camera coordinates can be further converted to image coordinates. Optionally, different transformation methods can be used for the x, y and z coordinates. For the x and y coordinates, similar triangles can be used to establish the relationship between the camera coordinates and the image coordinates, and for the depth information z, the absolute depth can be converted to a relative depth before transformation.

[0103] For example, the specific transformation formula can be expressed as:

[0104]

[0105] Where f is the focal length of the camera, x and y are X c , Yc For the depth information z, we can first calculate the absolute depth Z of each hand key point. c Transform to the relative depth Z relative to the root node (for example, the root node here can be selected as the key point at the wrist) c -Z0, where Z0 is the depth value of the root node. Then, the same transformation coefficient can be used Transform the relative depth to a space consistent with the x and y scales.

[0106] For 2D keypoint samples, the original data may contain the image and the x, y coordinates of the hand keypoints manually marked on the image, but the depth coordinate information is missing. Therefore, z can be set to zero and the corresponding bit of the depth information in the mask vector can be set to 0.

[0107] As described above, the above training labels and mask vectors related to the marking rules of the key point samples can be set for each hand object in the training image, so that the key point samples under different marking rules (for example, two-dimensional key point samples and three-dimensional key point samples) are unified into the same training framework to perform consistent training operations on the key point samples under different marking rules. By using a unified training framework to train the key point prediction model for key point samples under different marking rules, the key point prediction model can learn information of various dimensions from, for example, two-dimensional key point samples and three-dimensional key point samples, and predict information of the remaining dimensions with only prior information of some dimensions. For example, the trained key point detection model can obtain the three-dimensional key point positioning information from the input two-dimensional image.

[0108] Therefore, next, according to an embodiment of the present disclosure, in the training of the key point prediction model, the key point prediction loss function of the key point prediction model for the hand object can be determined based on the difference between the prediction results of the key point set of each hand object by the key point prediction model and the training label of the hand object, and the mask vector, and the key point prediction model can be adjusted based on the optimization of the loss function.

[0109] Optionally, the above-mentioned mask vector can be used as a weighting coefficient for the difference between the prediction result of the key point prediction model for each hand object's key point set and the training label of the hand object, so that the calculation of the loss function can adopt a consistent calculation method for key point samples under different labeling rules, thereby reducing the calculation complexity.

[0110] As an example, the least squares method can be used to calculate the key point prediction loss, where the least squares loss function after introducing the mask can be defined as:

[0111]

[0112] Among them, r i，x 、r i，y 、r i，z is the prediction result of the key point prediction model for the i-th key point of the hand object, g i，x , g i，y , g i，z is the manual calibration result of the ith key point (i.e., the coordinate value corresponding to the ith key point in the training label of the hand object), M i，x 、M i，y 、M i，z is the mask value corresponding to each dimension of the i-th key point in the mask vector. For example, for a two-dimensional key point sample, M i，z = 0, and for 3D key point samples, M i，z =1.

[0113] In addition, similar to the description of the reference step 303, since it is not known in advance whether the classification of the current image area is the left hand or the right hand in the actual prediction process, in order to be consistent with the actual situation, in the training and testing process of the key point prediction model, the classification result of the detection sub-network is also used to determine the prediction result of the final key point. In this case, even if the detection sub-network cannot accurately give the classification of the current image area in the early stage of the training of the key point prediction model, as the training process proceeds, the purpose of selecting different key point prediction results for different classifications can still be achieved.

[0114] As described above, the image key point detection method disclosed in the present invention performs hand detection and key point positioning on the corresponding image area in the original target image based on the multi-dimensional feature vector of the pixel for each pixel in the feature map extracted from the target image, wherein the classification of the image area and the prediction of different key points about the left and right hands in the image area are performed synchronously, and the key point prediction result is selected as the key point set under the pixel based on the classification result, thereby realizing the detection of all hand key points in the original target image. The method of the embodiment of the present invention can train a single key point prediction model that simultaneously performs hand detection and hand key point positioning, which reduces the algorithm development cycle and cost, is conducive to iterative optimization, and by utilizing two-dimensional and three-dimensional labeled data, it can provide rich scene changes and accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved.

[0115] Figure 8 is a schematic diagram showing an image key point detection device 800 according to an embodiment of the present disclosure.

[0116] According to an embodiment of the present disclosure, the image key point detection device 800 may include an image acquisition module 801 , a feature extraction module 802 , and a key point prediction module 803 .

[0117] The image acquisition module 801 may be configured to acquire a target image. Optionally, the image acquisition module 801 may perform the operations described above with reference to step 301 .

[0118] Optionally, the target image may be a single image or a video frame from a video, and the target image may or may not include a target object to be detected (for example, a hand target in the present disclosure). For example, the target image may be an image of the current head display perspective captured by a head display device, or an image frame captured by a video capture device.

[0119] The feature extraction module 802 may be configured to extract a feature map of the target image, each pixel in the feature map having a multi-dimensional feature vector associated with a corresponding image region in the target image. Optionally, the feature extraction module 802 may perform the operations described above with reference to step 302 .

[0120] Optionally, the input target image can be first normalized to a standard size, and then sent to a convolutional neural network to perform feature transformation on the target image, so as to transform the original image into a high-dimensional space, so as to obtain the image features of the target image in the high-dimensional space. For example, a deep fully convolutional neural network can be used to extract image features. For an image with an input dimension of W×H, a convolutional neural network can output a feature map of dimension w×h×c, i.e., an image with a length of w and a width of h, where each pixel in the image contains a c-dimensional feature vector for describing the image information in the corresponding receptive field in the original target image.

[0121] The key point prediction module 803 may be configured to: for each pixel in the feature map: determine the classification of the image region corresponding to the pixel based on the multidimensional feature vector of the pixel, the classification including left hand and right hand; predict the first key point set in the image region when the classification of the image region corresponding to the pixel is left hand, and the second key point set in the image region when the classification of the image region is right hand based on the multidimensional feature vector of the pixel; and select one from the first key point set and the second key point set as the key point set corresponding to the pixel based on the determined classification of the image region corresponding to the pixel. Optionally, the key point prediction module 803 may perform the operations described above with reference to step 303.

[0122] In the image key point detection device disclosed in the present invention, after obtaining a feature map of the target image through the feature extraction module 802, the feature map can be input into the key point prediction module 803 for detection and positioning processing, wherein the key point prediction module 803 can include two independent modules for independent prediction: a hand area detection module and a hand key point positioning module. The hand area detection module can be used to predict the category of the original image area corresponding to the current pixel in the feature map (including predicting whether the current pixel belongs to the detection target, that is, whether there is a detection target in the original image area corresponding to the pixel), and the hand key point positioning module can be used to predict the key points in the current image area. The above two modules can share the image features extracted by the feature extraction network in a multi-task learning manner to complete detection and positioning respectively.

[0123] For example, for each pixel in the feature map output by the depth model, target detection can be performed based on the multidimensional feature vector of the pixel. In the embodiment of the present disclosure, for a hand target, the classification of the image area in the original target image corresponding to the pixel can be determined based on the multidimensional feature vector of the pixel, that is, whether there is a hand target, and the specific classification when it is determined that there is a hand target, such as left hand and right hand.

[0124] In the image key point detection device disclosed in the present invention, the left and right hands can be processed differently in the detection and positioning stage, including classifying the image area with respect to the left and right hands in the hand area detection module in the key point prediction module 803, and separately predicting the key points of the left and right hands in the hand key point positioning module in the key point prediction module 803.

[0125] Optionally, in the hand region detection module in the key point prediction module 803, the classification of the image region corresponding to the pixel can be determined, which can perform target detection and classification based on the multi-dimensional feature vector of the pixel, and output the prediction result that the image region corresponding to the pixel belongs to each classification. For example, the prediction result can be output in the form of probability, that is, the probability that the image region belongs to each classification.

[0126] In addition to the above-mentioned classification of the image regions of the left hand and the right hand, there may also be a region that does not include the hand target, in which case the region may be considered as part of the background region. Therefore, according to an embodiment of the present disclosure, the classification may also include the background.

[0127] Optionally, when it is determined that the image area corresponding to a certain pixel belongs to the background, since the detection target of the present disclosure (i.e., the hand target) does not exist therein, subsequent processing may not be performed on the image area. That is, when it is determined that the image area corresponding to the pixel is classified as the background, the key point prediction module 803 may skip the subsequent positioning processing of the pixel and directly enter the detection and positioning of the next pixel.

[0128] Optionally, as described above, in the key point prediction module 803, a separate detection method can be used for the left and right hands, that is, two processing branches are used simultaneously in the hand key point positioning module to predict two groups of key points, which correspond to the key points when the image area is classified as the left hand and the key points when the image area is classified as the right hand. As described above, different positioning branches can be set for the left hand and the right hand in the key point positioning subnetwork, and two different parameterized models are used to predict the feature points based on the same feature input to obtain the key point sets in two cases.

[0129] Optionally, when it is determined that there is a detection target in the image area, and the key point prediction when the image area is classified as a left hand and the key point prediction when the image area is classified as a right hand are obtained respectively through two processing branches, a selection can be made from the two key point predictions based on the classification of the image area, that is, a set of key point predictions corresponding to the classification are selected as the final key points.

[0130] Optionally, for the case where the image area is classified as the background, the image key point detection method disclosed in the present invention may further include: for each pixel in the feature map, when the image area corresponding to the pixel is classified as the background, not performing the prediction and selection on the pixel. That is, when the detection subnetwork determines that there is no detection target in the image area corresponding to the pixel, the key point prediction module 803 does not perform any subsequent processing on the image area, but directly transfers to the detection and positioning processing of the next pixel.

[0131] According to yet another aspect of the present disclosure, an image key point detection device is also provided. Fig. 9 A schematic diagram of an image key point detection device 2000 according to an embodiment of the present disclosure is shown.

[0132] like Fig. 9 As shown, the image key point detection device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores a computer readable code, and when the computer readable code is run by the one or more processors 2010, the image key point detection method as described above may be executed.

[0133] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., and may be an X86 architecture or an ARM architecture.

[0134] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0135] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Fig.10 The architecture of the computing device 3000 shown in FIG. Fig.10 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the image key point detection method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Fig.10 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig.10 One or more components of a computing device are shown.

[0136] According to another aspect of the present disclosure, a computer-readable storage medium is also provided. Computer-readable instructions are stored on the computer storage medium. When the computer-readable instructions are executed by a processor, the image key point detection method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiment of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronously linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0137] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the image key point detection method according to the embodiments of the present disclosure.

[0138] Embodiments of the present disclosure provide an image key point detection method, apparatus, device, and computer-readable storage medium.

[0139] Compared with the traditional hand key point positioning method, the method provided by the embodiment of the present disclosure can simultaneously perform hand detection and hand key point positioning through multi-task learning training of a single key point prediction model, which reduces the algorithm development cycle and cost, is conducive to iterative optimization, and simultaneously utilizes two-dimensional and three-dimensional labeled data, which can not only provide rich scene changes, but also accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved.

[0140] The method provided by the embodiment of the present disclosure performs hand detection and key point positioning on the corresponding image area in the original target image based on the multi-dimensional feature vector of each pixel in the feature map extracted from the target image, wherein the classification of the image area and the prediction of different key points about the left and right hands in the image area are performed synchronously, and the key point prediction result is selected as the key point set under the pixel based on the classification result, thereby realizing the detection of all hand key points in the original target image. The method of the embodiment of the present disclosure can train a single key point prediction model that performs hand detection and hand key point positioning at the same time, which reduces the algorithm development cycle and cost, is conducive to iterative optimization, and by utilizing two-dimensional and three-dimensional labeled data, it can provide rich scene changes and accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved.

[0141] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains at least one executable instruction for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0142] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0143] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for detecting key points of an image, comprising: Get the target image; Extracting a feature map of the target image, each pixel in the feature map having a multi-dimensional feature vector associated with a corresponding image region in the target image; as well as For each pixel in the feature map: Based on the multidimensional feature vector of the pixel, determining a classification of the image region corresponding to the pixel, the classification comprising left hand and right hand; Based on the multidimensional feature vector of the pixel, predicting a first key point set in the image region when the image region corresponding to the pixel is classified as a left hand, and a second key point set in the image region when the image region is classified as a right hand; as well as Based on the determined classification of the image region corresponding to the pixel, one of the first key point set and the second key point set is selected as the key point set corresponding to the pixel.

2. The method of claim 1, wherein: Extracting the feature map of the target image includes: A pre-trained convolutional neural network is used to extract a feature map of the target image, wherein a multidimensional feature vector of each pixel in the feature map indicates image information in a corresponding image area in the target image, and the corresponding image area is related to a receptive field area of ​​the convolutional neural network.

3. The method of claim 1, wherein: The determining of the classification of the image region corresponding to the pixel is performed by a pre-trained detection subnetwork, and the determining of the classification of the image region corresponding to the pixel comprises: Detecting whether there is a target object in the image area, the target object includes a left hand and a right hand of a human body; In the case where a left hand of a human body is detected in the image region, determining that the image region is classified as a left hand; and In the case where the right hand of the human body is detected to exist in the image region, the classification of the image region is determined to be the right hand.

4. The method of claim 3, wherein: Said classification also includes background; The determining of the classification of the image region corresponding to the pixel further comprises: In the case where it is detected that the left hand and the right hand of the human body do not exist in the image region, determining that the image region is classified as background; The method further comprises: For each pixel in the feature map, if the image region corresponding to the pixel is classified as background, the prediction and selection are not performed on the pixel.

5. The method of claim 3, wherein: The prediction of the first set of key points and the second set of key points is performed by a pre-trained key point localization subnetwork, wherein in the key point localization subnetwork, two different localization models are used to respectively predict the key point sets in the image region corresponding to the pixel when the image region is classified as the left hand and the right hand.

6. The method of claim 5, wherein: The detection subnetwork and the key point localization subnetwork constitute a key point prediction model, which takes the feature map of the target image as input and takes a set of key points in the target image as output, wherein each key point in the key point set has depth information.

7. The method of claim 6, wherein: The pre-trained detection sub-network and the pre-trained key point localization sub-network are obtained by training the key point prediction model; The key point prediction model is trained based on a training image set including two-dimensional key point samples and three-dimensional key point samples, and the three-dimensional key point samples have depth information.

8. The method of claim 7, wherein: Each training image in the training image set includes one or more hand objects, and each of the one or more hand objects has a two-dimensional key point sample or a three-dimensional key point sample for training; The two-dimensional key point samples indicate the two-dimensional coordinates of the key points associated with the hand object in the training image, and the three-dimensional key point samples indicate the three-dimensional coordinates of the key points associated with the hand object in the training image, and the three-dimensional coordinates include the two-dimensional coordinates and depth information.

9. The method of claim 8, wherein: In the training of the key point prediction model, a training label with a unified format is set for each hand object in each training image in the training image set, and the format is a vector with a predetermined dimension, and the predetermined dimension includes the classification dimension, the key point dimension when the classification of the image area corresponding to the hand object is predicted to be the left hand, and the key point dimension when the classification of the image area corresponding to the hand object is predicted to be the right hand, wherein the key point dimension is related to the number of key points of the hand object and the dimension of the three-dimensional coordinates of each key point.

10. The method of claim 9, wherein: In the training of the key point prediction model, a mask vector having the same dimension as the key point dimension is also set for each hand object in each training image in the training image set, so as to uniformly process the two-dimensional key point samples and the three-dimensional key point samples; Wherein, when a two-dimensional key point sample is used for training the hand object, the value of the dimension indicating the depth information in the three-dimensional coordinate dimension of each key point in the mask vector of the hand object is set to 0, and the values ​​of the remaining two-dimensional coordinate dimensions are all set to 1; and When the three-dimensional key point samples are used for training the hand object, the value of the three-dimensional coordinate dimension of each key point in the mask vector of the hand object is set to 1.

11. The method of claim 10, wherein: During the training of the key point prediction model, the key point prediction loss function of the key point prediction model for each hand object is determined based on the difference between the prediction result of the key point prediction model for the key point set of each hand object and the training label of the hand object, and the mask vector, and the key point prediction model is adjusted based on the optimization of the loss function.

12. An image key point detection device, comprising: An image acquisition module is configured to acquire a target image; a feature extraction module configured to extract a feature map of the target image, each pixel in the feature map having a multi-dimensional feature vector associated with a corresponding image region in the target image; as well as The key point prediction module is configured to, for each pixel in the feature map: Based on the multidimensional feature vector of the pixel, determining a classification of the image region corresponding to the pixel, the classification comprising left hand and right hand; Based on the multidimensional feature vector of the pixel, predicting a first key point set in the image region when the image region corresponding to the pixel is classified as a left hand, and a second key point set in the image region when the image region is classified as a right hand; as well as Based on the determined classification of the image region corresponding to the pixel, one of the first key point set and the second key point set is selected as the key point set corresponding to the pixel.

13. An image key point detection device, comprising: one or more processors; as well as One or more memories, wherein a computer executable program is stored, and when the computer executable program is executed by the processor, the method of any one of claims 1-11 is performed.

14. A computer program product stored on a computer-readable storage medium and comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any one of claims 1 to 11.

15. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 11 when executed by a processor.

Images