Image key point detection method and apparatus, and device and storage medium

Abstract

Provided in the present disclosure are an image key point detection method and apparatus, and a device and a computer-readable storage medium. In the method, for each pixel in a feature map extracted from a target image, hand detection and key point positioning are performed on a corresponding image area in the original target image on the basis of a multi-dimensional feature vector of the pixel, wherein the classification of the image area and the prediction of different key points regarding left and right hands in the image area are synchronously executed, and on the basis of a classification result, a key point prediction result is selected as a key point set under the pixel, thereby realizing the detection of all hand key points in the original target image. By means of the method, a single key point prediction model can be trained, thereby reducing the algorithm development cycle and costs, and facilitating iterative optimization; moreover, the issues of key point positioning regarding left and right hands are distinguished, thereby reducing the difficulty of key point positioning, and improving the positioning precision.

Description

Image key point detection method, device, equipment and storage medium

[0001] This application claims priority to Chinese Patent Application No. 202310756059.2 filed on June 25, 2023. The contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The present disclosure relates to the field of computer vision, and more specifically, to a method, apparatus, device, and storage medium for detecting key points in an image. Background Art

[0003] Hand keypoint localization is an important technology in the field of computer vision. It extracts the positions of hand keypoints from images or videos. This technology is widely used in many fields such as gesture recognition, human-computer interaction, and biometric recognition.

[0004] Common hand keypoint localization methods can be categorized as two-dimensional (2D) and three-dimensional (3D) hand keypoint localization. 2D hand keypoint localization involves extracting the coordinates (i.e., x and y coordinates) of hand keypoints from an image in a two-dimensional coordinate system. Thanks to the development of internet technology and the widespread availability of image acquisition devices, 2D hand keypoint algorithms can collect massive amounts of richly varied image data for model training. However, since a 2D image is merely a projection of the 3D hand structure onto the imaging plane, the captured image is prone to hand occlusion and fails to reflect the complete 3D structure of the hand. 3D hand keypoint localization, in addition to extracting x and y coordinates, adds a prediction of the depth z of each keypoint, which can better reflect the spatial structure of the hand. However, since general images lack depth information, data collection must be conducted in a laboratory, which fails to capture rich environmental variations and makes it difficult to collect samples that closely resemble the actual environment, impacting the model's ultimate generalization ability.

[0005] Therefore, an efficient and accurate image keypoint detection method is needed.

[0006] Summary of the Invention

[0007] To solve the above problems, the present disclosure simultaneously performs hand key point detection and localization by utilizing encoded consistent two-dimensional hand key point samples and three-dimensional hand key point samples to train a single key point prediction model including hand detection and hand key point localization sub-networks.

[0008] Embodiments of the present disclosure provide a method, apparatus, device, and computer-readable storage medium for detecting image key points.

[0009] An embodiment of the present disclosure provides an image key point detection method, comprising: acquiring a target image; extracting a feature map of the target image, wherein each pixel in the feature map has a multidimensional feature vector associated with a corresponding image area in the target image; and for each pixel in the feature map: determining a classification of the image area corresponding to the pixel based on the multidimensional feature vector of the pixel, the classification including left hand and right hand; predicting a first key point set in the image area when the classification of the image area corresponding to the pixel is left hand, and a second key point set in the image area when the classification of the image area is right hand based on the multidimensional feature vector of the pixel; and selecting one from the first key point set and the second key point set as the key point set corresponding to the pixel based on the determined classification of the image area corresponding to the pixel.

[0010] An embodiment of the present disclosure provides an image key point detection device, comprising: an image acquisition module, configured to acquire a target image; a feature extraction module, configured to extract a feature map of the target image, each pixel in the feature map having a multidimensional feature vector associated with a corresponding image area in the target image; and a key point prediction module, configured to, for each pixel in the feature map: determine, based on the multidimensional feature vector of the pixel, a classification of the image area corresponding to the pixel, the classification including left hand and right hand; predict, based on the multidimensional feature vector of the pixel, a first key point set in the image area when the classification of the image area corresponding to the pixel is left hand, and a second key point set in the image area when the classification of the image area is right hand; and select one from the first key point set and the second key point set as the key point set corresponding to the pixel based on the determined classification of the image area corresponding to the pixel.

[0011] An embodiment of the present disclosure provides an image key point detection device, comprising: one or more processors; and one or more memories, wherein a computer executable program is stored in the one or more memories, and when the computer executable program is executed by the processor, the image key point detection method described above is performed.

[0012] An embodiment of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon. When the instructions are executed by a processor, the instructions are used to implement the above-mentioned image key point detection method.

[0013] Embodiments of the present disclosure provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image keypoint detection method according to an embodiment of the present disclosure.

[0014] Compared to traditional hand keypoint localization methods, the method provided by the embodiments of the present disclosure can simultaneously perform hand detection and hand keypoint localization using a single keypoint prediction model trained through multi-task learning. This reduces the algorithm development cycle and cost, facilitates iterative optimization, and utilizes both two-dimensional and three-dimensional labeled data, providing rich scene variations while accurately describing hand structure information. Furthermore, by distinguishing between left and right hand keypoint localization issues, the difficulty of keypoint localization is reduced and positioning accuracy is improved.

[0015] The method provided by the embodiment of the present disclosure performs hand detection and key point positioning on the corresponding image area in the original target image based on the multi-dimensional feature vector of each pixel in the feature map extracted from the target image, wherein the classification of the image area and the prediction of different key points about the left and right hands in the image area are performed synchronously, and the key point prediction result is selected as the key point set under the pixel based on the classification result, thereby realizing the detection of all hand key points in the original target image. The method of the embodiment of the present disclosure can train a single key point prediction model that performs hand detection and hand key point positioning at the same time, reducing the algorithm development cycle and cost, facilitating iterative optimization, and by utilizing two-dimensional and three-dimensional labeled data, it can provide rich scene changes and accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some exemplary embodiments of the present disclosure. For those skilled in the art, other drawings can be derived from these drawings without inventive effort.

[0017] FIG1 is a schematic diagram illustrating a scenario of processing an input target image according to an embodiment of the present disclosure;

[0018] FIG2 is a flow chart illustrating a conventional method for detecting key points of a hand;

[0019] FIG3A is a flowchart illustrating a method for detecting key points in an image according to an embodiment of the present disclosure;

[0020] FIG3B is a schematic diagram illustrating a flow chart of a method for detecting key points in an image according to an embodiment of the present disclosure;

[0021] FIG4 is a flow chart illustrating a single-step positioning method according to an embodiment of the present disclosure and a traditional two-step positioning method;

[0022] FIG5 is a schematic diagram showing key point detection results for the left hand and the right hand according to an embodiment of the present disclosure;

[0023] FIG6 is a schematic diagram illustrating a process of training a key point prediction model according to an embodiment of the present disclosure;

[0024] 7 is a schematic diagram illustrating final key point prediction of a hand object and mask vectors corresponding to two-dimensional key point samples and three-dimensional key point samples according to an embodiment of the present disclosure;

[0025] FIG8 is a schematic diagram illustrating an image key point detection device according to an embodiment of the present disclosure;

[0026] FIG9 shows a schematic diagram of an image key point detection device according to an embodiment of the present disclosure; and

[0027] FIG10 shows a schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present disclosure more apparent, the exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0029] In this specification and the accompanying drawings, substantially the same or similar steps and elements are denoted by the same or similar reference numerals, and repeated descriptions of these steps and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance or ranking.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0031] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0032] The image key point detection method disclosed in the present invention can be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. For example, for the image key point detection method based on artificial intelligence, it can analyze each pixel in the feature map extracted from the target image in a way similar to how humans recognize human hands in the image with the naked eye, so as to identify the hand object from the corresponding image area in the target image and perform key point detection on it, thereby determining the position of the hand key point in the target image.

[0033] The image key point detection method disclosed in the present invention can be based on deep learning. Deep learning is an algorithm in machine learning that is based on data representation learning. Observations (e.g., an image) can be represented in a variety of ways, such as a vector of intensity values ​​for each pixel, or more abstractly represented as a series of edges, regions of specific shapes, etc. Using certain specific representation methods makes it easier to learn tasks from examples (e.g., face recognition or hand recognition). The advantage of deep learning is that it replaces manual feature acquisition with efficient algorithms for unsupervised or semi-supervised feature learning and hierarchical feature extraction. Optionally, the image key point detection method disclosed in the present invention can be based on a deep fully convolutional neural network (FCNN). FCNN can classify images at the pixel level (that is, classify each pixel). It can accept input images of any size, and uses a deconvolution layer to upsample the feature map of the last convolution layer to restore it to the same size as the input image, so that a prediction can be generated for each pixel while retaining the spatial information in the original input image. Finally, the pixel classification is performed on the upsampled feature map. In the image key point detection method disclosed in the present invention, FCNN can be used to extract image features of the target image to output a feature map containing several pixels, each of which corresponds to an image area (receptive field) in the original target image. By analyzing each pixel, hand key point detection of the original target image can be achieved.

[0034] The image key point detection method disclosed in the present invention can be based on computer vision (CV) technology. Computer vision is a science that studies how to make machines "see". More specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, track and measure targets, and further performs graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, three-dimensional technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0035] For example, the image keypoint detection method disclosed herein can be used for hand keypoint detection. Hand keypoint detection aims to extract the locations of hand keypoints from images or videos and is an important technology in the field of computer vision. This technology has been widely used in many fields, including gesture recognition, human-computer interaction, and biometric recognition. Common hand keypoint localization methods can be divided into image processing-based methods and deep learning-based methods. Among them, image processing-based methods use the domain knowledge of professionals combined with basic digital image processing methods to extract hand keypoints from images. This type of method does not require manual sample labeling and can shorten the development cycle to a certain extent, but requires relatively fine parameter adjustment and cannot adapt to complex environmental changes. Machine learning-based methods are the current mainstream keypoint localization methods. This method can automatically learn effective information to locate hand keypoints from massive amounts of labeled data and has greater robustness. However, the initial heavy work of sample labeling requires a large amount of manpower and material investment.

[0036] In summary, the solutions provided by the embodiments of the present disclosure involve technologies such as artificial intelligence, deep learning, computer vision, and hand key point detection. The embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0037] FIG1 is a schematic diagram illustrating a scenario of processing an input target image according to an embodiment of the present disclosure.

[0038] As shown in Figure 1, the user can send the target image (or video frame) to be processed to the server through the user terminal, so that the server can perform key point detection processing on the target image. Subsequently, the server can return the results of the key point detection (for example, the key point set of the target image) to the user terminal through the network, for example, displaying it on the image interface of the user terminal. Optionally, the user terminal can specifically include a smart phone, a tablet computer, a laptop portable computer, a desktop computer, a vehicle-mounted terminal, a wearable device, etc., but is not limited to this. For example, the user terminal can also be an image / video acquisition device, etc., to send the collected image / video directly to the processor end for processing. The user terminal can also be a client that installs a browser or various applications (including system applications and third-party applications). Optionally, the network can be an Internet of Things based on the Internet and / or telecommunications network, which can be a wired network or a wireless network. For example, it can be an electronic network that can realize information exchange functions such as a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a cellular data communication network, etc. The user terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application is not limited here. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Of course, in addition to the above example scenario where the server performs image processing and returns the processing results to the user terminal, the image processing method of the present disclosure can also be executed directly by the user terminal without going through the network and server.

[0039] The ability to perceive the shape and movement of the hand is a prerequisite for improving the user experience in various technical fields and platforms. For example, it can form the basis of sign language understanding and gesture control, and can also overlay digital content and information on the physical world in augmented reality. Robust real-time hand perception is undoubtedly a challenging computer vision task because hands are often occluded (for example, due to folded fingers / palms and occlusions such as handshakes) and lack high-contrast patterns.

[0040] As mentioned above, common hand keypoint localization methods can be categorized into two-dimensional (2D) and three-dimensional (3D) methods based on the dimensionality of the labeled data. 2D hand keypoint localization involves extracting the x and y coordinates of hand keypoints from images. Thanks to the advancement of internet technology and the widespread availability of image acquisition devices, 2D hand keypoint algorithms can collect vast amounts of diverse image data for model training. However, since 2D images are merely projections of the 3D hand structure onto the imaging plane, the captured images are prone to hand occlusion and fail to reflect the complete 3D structure of the hand. 3D hand keypoint localization, in addition to extracting x and y coordinates, adds a prediction of the depth z of each keypoint, which can better reflect the spatial structure of the hand. However, since typical images lack depth information, data collection must be conducted in a laboratory, which fails to capture rich environmental variations and makes it difficult to collect samples that closely resemble real-world environments, impacting the model's ultimate generalization ability. Furthermore, commonly used hand keypoint localization methods can be categorized based on their processing approach into image processing-based methods and deep learning-based methods. Image processing-based methods leverage domain knowledge from professionals combined with basic digital image processing methods to extract hand keypoints from images. These methods don't require manual sample labeling, which can shorten development cycles to a certain extent. However, they require sophisticated parameter adjustments and are unable to adapt to complex environmental changes. Machine learning-based methods are currently the mainstream keypoint localization methods. These methods automatically learn effective information from massive amounts of labeled data to locate hand keypoints, offering greater robustness. However, the initial sample labeling process involves extensive human and material investment.

[0041] Figure 2 is a flowchart illustrating a conventional hand keypoint detection method. As shown in Figure 2, the hand keypoint detection method can primarily include two parts: a hand detection module and a hand keypoint localization module. In the hand detection module, the captured image can first be preprocessed, including normalizing the image to a standard size. The image can then be divided into different regions, and each region can be independently determined to determine whether it contains a hand target. Furthermore, for regions containing a hand target, the bounding rectangle (i.e., the hand rectangle) containing the hand target can be predicted to obtain a more accurate hand position. Next, in the keypoint localization module, the input hand region (i.e., the image region within the hand rectangle) can first be size-normalized to scale it to a standard size. The image can then be input into a keypoint localization model to obtain the positions of the hand keypoints relative to the input image. Finally, the hand keypoints can be transformed to the original image scale.

[0042] In the hand key point detection method described above, first, after completing hand detection, the hand key point detection module needs to be called once for each hand region. Its positioning speed decreases sharply as the number of hand targets in the image increases. In scenes with a large number of hand targets, it is easy to generate a large time overhead. In addition, such a method requires training two independent modules at the same time, which increases the cycle and cost of algorithm development and is not conducive to iterative optimization. Secondly, this method does not distinguish between the key point detection of the left and right hands. However, although the left and right hands are similar, they actually have a mirror relationship and have different texture features under the same key point morphology. The current method increases the difficulty of the problem and limits the improvement of model accuracy. Thirdly, this method cannot use three-dimensional and two-dimensional labeled data at the same time, resulting in the inability to accurately describe the structural information of the hand target when only two-dimensional input is used, and the inability to provide rich scene changes when only three-dimensional data is used.

[0043] Based on this, the present disclosure provides an image key point detection method, which simultaneously performs hand key point detection and positioning by utilizing encoded consistent two-dimensional hand key point samples and three-dimensional hand key point samples to train a single key point prediction model including hand detection and hand key point positioning sub-networks.

[0044] Compared to traditional hand keypoint localization methods, the method provided by the embodiments of the present disclosure can simultaneously perform hand detection and hand keypoint localization using a single keypoint prediction model trained through multi-task learning. This reduces the algorithm development cycle and cost, facilitates iterative optimization, and utilizes both two-dimensional and three-dimensional labeled data, providing rich scene variations while accurately describing hand structure information. Furthermore, by distinguishing between left and right hand keypoint localization issues, the difficulty of keypoint localization is reduced and positioning accuracy is improved.

[0045] The method provided by the embodiment of the present disclosure performs hand detection and key point positioning on the corresponding image area in the original target image based on the multi-dimensional feature vector of each pixel in the feature map extracted from the target image, wherein the classification of the image area and the prediction of different key points about the left and right hands in the image area are performed synchronously, and the key point prediction result is selected as the key point set under the pixel based on the classification result, thereby realizing the detection of all hand key points in the original target image. The method of the embodiment of the present disclosure can train a single key point prediction model that performs hand detection and hand key point positioning at the same time, reducing the algorithm development cycle and cost, facilitating iterative optimization, and by utilizing two-dimensional and three-dimensional labeled data, it can provide rich scene changes and accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved.

[0046] Fig. 3A is a flow chart illustrating a method 300 for detecting key points in an image according to an embodiment of the present disclosure. Fig. 3B is a schematic flow chart illustrating a method for detecting key points in an image according to an embodiment of the present disclosure.

[0047] As shown in FIG3B , the image key point detection method disclosed herein can mainly include two stages: feature extraction and detection and positioning. In the feature extraction stage, a feature map is extracted from the target image to obtain image information about each image area in the target image. In the detection and positioning stage, the extracted feature map can be processed, including using the feature map to simultaneously perform target detection and key point positioning. The image key point detection method disclosed herein will be described in detail below with reference to FIG3A .

[0048] In step 301, a target image may be acquired. Optionally, the target image may be a single image or a video frame from a video, and the target image may or may not include a target object to be detected (e.g., a hand target in the present disclosure). For example, the target image may be an image of the current head-mounted display perspective, such as captured by a head-mounted display device, or an image frame, such as captured by a video capture device, although this disclosure is not limited thereto.

[0049] In step 302 , a feature map of the target image may be extracted, where each pixel in the feature map may have a multi-dimensional feature vector associated with a corresponding image region in the target image.

[0050] Optionally, before feature extraction is performed on the acquired target image, a preprocessing operation may be performed on it, for example, including but not limited to operations such as scaling, cropping and splicing on the target image so that it meets the image requirements of the input feature extraction network.

[0051] According to an embodiment of the present disclosure, extracting the feature map of the target image may include: using a pre-trained convolutional neural network to extract the feature map of the target image, the multidimensional feature vector of each pixel in the feature map may indicate image information in a corresponding image area in the target image, and the corresponding image area is related to the receptive field area of ​​the convolutional neural network.

[0052] As shown in Figure 3B, a feature map of the target image can be extracted using a deep model that includes a pre-trained convolutional neural network. Optionally, the input target image can first be normalized to a standard size and then fed into the convolutional neural network to perform feature transformation on the target image, thereby transforming the original image into a high-dimensional space to obtain high-dimensional image features of the target image in the high-dimensional space.

[0053] Optionally, a convolutional neural network (e.g., a deep fully convolutional neural network) can be used to extract image features. For example, for an image with an input dimension of W×H, a convolutional neural network can output a feature map of dimension w×h×c, that is, an image with a length of w and a width of h, where each pixel in the image contains a c-dimensional feature vector that describes the image information within the corresponding receptive field in the original target image. Among them, c is the dimension of the features extracted by the neural network, and the size of c is the dimension of projecting the input image into the feature space. That is to say, the larger c is, the more complex the model is and the stronger its descriptive ability is, but there is also a greater risk of overfitting, and more training samples are required to obtain ideal results; on the contrary, when c is smaller, the model is simpler and it is difficult to describe the complete sample changes. Therefore, c is a hyperparameter involved in constructing a network model, and its value is usually adjusted during model training.

[0054] For example, the deep feature extraction process is the computational process of a deep convolutional neural network. A common deep convolutional neural network structure can be composed of a stack of basic network units. For example, a typical network unit may include the following components: a linear transformation layer, such as a "convolutional network layer" or a "fully connected network layer"; a regularization layer, such as a "batch normalization layer" or an "instance normalization layer"; and a nonlinear layer, such as a sigmoid layer or a tanh layer. During the feature extraction phase, the input target image is processed through multiple network units to ultimately obtain abstract image features.

[0055] The above-mentioned depth model can be pre-trained. During the training process of the depth model, the parameters of the depth model can be adjusted so that the model can obtain image features that are most relevant to the positioning target from the target image for subsequent detection and positioning.

[0056] It should be understood that the image key point detection method disclosed in the present invention does not rely on the network structure of a specific feature extraction model. It can select a suitable basic network based on the requirements for detection accuracy, computing speed, etc. in different application scenarios. For example, common basic networks may include but are not limited to: ResNet, DenseNet, etc. Optionally, when the image key point detection method disclosed in the present invention is implemented in an edge computing device (such as virtual reality (VR) glasses, mobile phones and other mobile devices), due to the limitations of computing power and power consumption, it is necessary to adopt a model that is as simple as possible. At this time, it is necessary to reduce the complexity of the model and sacrifice some accuracy to ensure the speed of the model. In addition, when the image key point detection method disclosed in the present invention is implemented on a personal computer (PC) or a high-end computing platform (such as a desktop computer, workstation and other equipment), since the platform itself has powerful computing power, the complexity of the model can be appropriately increased to make more accurate predictions. The image key point detection method disclosed in the present invention does not impose any restrictions on the specific network structure of the deep model for feature extraction.

[0057] After obtaining the feature map of the target image in high-dimensional space through the above feature extraction, the extracted feature map can be used to complete the detection and positioning operations through multi-task learning in the detection and positioning stage.

[0058] In step 303 , for each pixel in the feature map, operations shown in steps 3031 - 3033 may be performed.

[0059] Figure 4 is a flow chart showing a single-step positioning method according to an embodiment of the present disclosure and a traditional two-step positioning method, wherein (a) in Figure 4 shows the two-step positioning method used in the traditional hand key point detection method, and (b) in Figure 4 shows the single-step positioning method used in the image key point detection method of the present disclosure.

[0060] As described above and referring to (a) in FIG4 , the traditional hand key point detection method requires a cascade of two detection and positioning models to complete the final hand key point detection work, wherein the target image can output multiple hand regions contained in the target image (for example, hand image 1, ..., hand image n, etc.) through the hand region detection module, and the image region without hand targets is discarded. Since the results output by the hand region detection need to be scaled and normalized (i.e., the image normalization processing in (a) in FIG4 ) before being sent to the hand key point localization module, it is difficult to adopt an end-to-end training framework, resulting in the need to train two models independently, which increases the difficulty of development. Moreover, when there are multiple hand targets in the target image, the hand key point localization model needs to be called independently for each hand target, which increases the overall time consumption of hand key point detection.

[0061] In the single-step key point localization method disclosed in the present invention, as shown in (b) in FIG4 , after obtaining a feature map from the target image through common feature extraction, the feature map can be input into two independent modules for independent prediction: a hand area detection module and a hand key point localization module. The hand area detection module can be used to predict the category of the original image area corresponding to the current pixel in the feature map (including predicting whether the current pixel belongs to the detection target, that is, whether there is a detection target in the original image area corresponding to the pixel), and the hand key point localization module can be used to predict the key points in the current image area. The above two modules can use multi-task learning to share the image features extracted by the feature extraction network to complete detection and positioning respectively.

[0062] The processing of the detection and positioning stage of the present disclosure (ie, the processing in the above step 303 ) will be described in detail below with reference to FIG. 3B .

[0063] In step 3031, a classification of the image region corresponding to the pixel may be determined based on the multi-dimensional feature vector of the pixel, and the classification may include left hand and right hand.

[0064] Optionally, as shown in FIG3B , for each pixel in the feature map output by the depth model, target detection can be performed based on the multidimensional feature vector of the pixel. For example, in an embodiment of the present disclosure, for a hand target, the classification of the image region in the original target image corresponding to the pixel can be determined based on the multidimensional feature vector of the pixel, that is, whether a hand target exists, and the specific classification of the hand target in the case of determining the presence of the hand target, such as left hand or right hand.

[0065] FIG5 is a schematic diagram showing the key point detection results for the left and right hands according to an embodiment of the present disclosure. As shown in FIG5 , the left and right hands have similar structures, but they are actually mirror images. That is, for the same key point distribution, the texture information presented by the left and right hands is inconsistent. However, in existing key point localization methods, the same model is usually used to locate key points for the left and right hands, ignoring the mirror image relationship between the two hands. This requires predicting the same key point distribution for completely different image textures, which increases the complexity of the model in handling key point detection problems.

[0066] Therefore, in response to the above problems, the image key point detection method disclosed in the present invention can distinguish between the left and right hands in the detection and positioning stage, including classifying the image area with respect to the left and right hands in the detection part (for example, in the hand area detection module as shown in (b) in Figure 4), and separately predicting the key points of the left and right hands in the positioning part (for example, in the hand key point positioning module as shown in (b) in Figure 4).

[0067] According to an embodiment of the present disclosure, determining the classification of the image area corresponding to the pixel may include: detecting whether there is a target object in the image area, the target object may include the left hand and the right hand of a human body; in the case of detecting the presence of the left hand of a human body in the image area, determining the classification of the image area as the left hand; and in the case of detecting the presence of the right hand of a human body in the image area, determining the classification of the image area as the right hand.

[0068] Alternatively, the hand region detection module shown in FIG4(b) may perform the above-described determination of the classification of the image region corresponding to the pixel. This module may perform target detection and classification based on the multidimensional feature vector of the pixel and output a prediction result of whether the image region corresponding to the pixel belongs to each classification. For example, the prediction result may be output in the form of a probability, i.e., the probability that the image region belongs to each classification.

[0069] In addition to the above-mentioned classification of the left and right hand image regions, there may also be regions that do not include the hand target. In this case, the region can be considered as part of the background region. Therefore, according to an embodiment of the present disclosure, the classification may also include the background.

[0070] Correspondingly, the determination of the classification of the image area corresponding to the pixel may also include: in the case where it is detected that the left hand and the right hand of the human body do not exist in the image area, determining that the image area is classified as background. Optionally, when it is determined that the image area corresponding to a certain pixel belongs to the background, since the detection target of the present disclosure (i.e., the hand target) does not exist therein, subsequent processing may no longer be performed on the image area. That is, in the case where it is determined that the image area corresponding to the pixel is classified as background, the subsequent positioning processing of the pixel can be skipped and the detection and positioning of the next pixel can be directly entered.

[0071] In step 3032, based on the multidimensional feature vector of the pixel, a first set of key points in the image area when the image area corresponding to the pixel is classified as left-handed, and a second set of key points in the image area when the image area is classified as right-handed can be predicted.

[0072] Optionally, as described above, in the image key point detection method disclosed herein, separate detection methods can be used for the left and right hands, that is, two processing branches are simultaneously used in the hand key point positioning module as shown in (b) in Figure 4 to predict two groups of key points, respectively corresponding to the key points when the image area is classified as the left hand and the key points when the image area is classified as the right hand.

[0073] According to an embodiment of the present disclosure, the classification of the image region corresponding to the pixel can be performed by a pre-trained detection subnetwork, and the prediction of the first key point set and the second key point set can be performed by a pre-trained key point localization subnetwork. Specifically, the detection subnetwork and the key point localization subnetwork can correspond to the hand region detection module and the hand key point localization module shown in (b) of Figure 4, respectively. These two subnetworks can share the feature map of the target image output by the depth model during the detection and localization stage and perform corresponding operations respectively.

[0074] According to an embodiment of the present disclosure, in the keypoint localization subnetwork, two different localization models can be used to respectively predict the keypoint sets in the image region corresponding to the pixel when the image region is classified as the left hand and the right hand. As described above, in the keypoint localization subnetwork, different localization branches can be set for the left hand and the right hand, respectively, and two different parameterized models can be used to predict feature points based on the same feature input to obtain keypoint sets in both cases.

[0075] In step 3033, based on the determined classification of the image region corresponding to the pixel, one of the first key point set and the second key point set may be selected as the key point set corresponding to the pixel.

[0076] Optionally, as shown in Figure 3B, when it is determined that there is a detection target in the image area, and the key point prediction when the image area is classified as the left hand and the key point prediction when the image area is classified as the right hand are obtained respectively through two processing branches, a selection can be made from the two key point predictions based on the classification of the image area, that is, a set of key point predictions corresponding to the classification is selected as the final key points.

[0077] As an example, the above steps can be expressed as:

[0078] Among them, kpt represents the prediction result of the final key point, where kpt L kpt is the key point prediction result when the image area is classified as left hand. R is the key point prediction result when the image area is classified as right hand, P L 、P R 、P BG They represent the probability of the image region output by the detection sub-network being classified as the left hand, right hand, and background. That is, when the image region output by the detection sub-network is classified as the left hand (P L ≥P R ≥P BG ), the final key point prediction result adopts the key point prediction result when the image area is classified as left hand, and the image area output by the detection subnetwork is classified as right hand (P BG ≤P L <P R ), the final key point prediction result adopts the key point prediction result when the image area is classified as right hand.

[0079] In addition, the classification of the above image area is background (ie, P BG >P L And P BG >P R ), the image keypoint detection method of the present disclosure may further include: for each pixel in the feature map, if the image area corresponding to the pixel is classified as background, not performing the prediction and selection on the pixel. In other words, when the detection subnetwork determines that there is no detection target in the image area corresponding to the pixel, no subsequent processing is performed on the image area, and the detection and positioning processing of the next pixel is directly transferred to.

[0080] According to an embodiment of the present disclosure, the detection subnetwork and the key point localization subnetwork can constitute a key point prediction model, which takes the feature map of the target image as input and a set of key points in the target image as output, and each key point in the key point set can have depth information.

[0081] The above-mentioned image key point detection method disclosed in the present invention can enable the detection sub-network and the key point positioning sub-network to perform multi-task learning and learn more robust image features. More specifically, more advanced hand structure information can be added to the detection sub-network, and richer input image changes can be added to the key point positioning sub-network. In addition, complete end-to-end training can be achieved, and better key point detection results can be achieved through a single key point prediction model. In addition, the image key point detection method disclosed in the present invention is not affected by the number of hand targets present in the target image, and the detection and positioning of all hand targets in the target image can be completed through a single forward calculation of the neural network.

[0082] The training process of the key point prediction model including the detection subnetwork and the key point positioning subnetwork will be described in detail below with reference to FIG6 and FIG7.

[0083] FIG6 is a schematic diagram illustrating a process of training a key point prediction model according to an embodiment of the present disclosure.

[0084] According to an embodiment of the present disclosure, the pre-trained detection sub-network and the pre-trained keypoint localization sub-network can be obtained by training the keypoint prediction model. As shown in Figure 6, during the model training process of the present disclosure, the keypoint prediction model including the detection sub-network and the keypoint localization sub-network can be trained end-to-end to achieve unified training of the detection sub-network and the keypoint localization sub-network.

[0085] According to an embodiment of the present disclosure, the key point prediction model may be trained based on a training image set including two-dimensional key point samples and three-dimensional key point samples, and the three-dimensional key point samples may have depth information. Specifically, each training image in the training image set may include one or more hand objects, and each of the one or more hand objects may have a two-dimensional key point sample or a three-dimensional key point sample for training, the two-dimensional key point sample may indicate the two-dimensional coordinates of the key point associated with the hand object in the training image, the three-dimensional key point sample may indicate the three-dimensional coordinates of the key point associated with the hand object in the training image, and the three-dimensional coordinates may include the two-dimensional coordinates and depth information.

[0086] Optionally, as shown in FIG6 , the training data used for key point prediction model training may include a training image set and key point labeled samples for each training image in the training image set. The training images in the training image set may be two-dimensional images or video frames identical to the target image, and the key point labeled samples for each training image may be two-dimensional key point samples or three-dimensional key point samples.

[0087] Specifically, 2D keypoint samples can be used to label keypoints of the hand in a 2D image. Thanks to the development of internet technology and the widespread availability of image acquisition equipment, capturing hand images in 2D natural environments has become very convenient. However, since 2D images are merely projections of the hand onto the image plane, keypoint occlusion is inevitable. For occluded keypoints, the labeler must infer their location based on experience, and for keypoints that cannot be determined, the confidence level must be set to 0. This places higher demands on the labeling process, and manual labeling makes it difficult to maintain consistent labeling quality. To ensure quality, multiple labelers are often required to repeatedly label the same object, significantly increasing labeling costs. 3D keypoint samples can effectively overcome the self-occlusion problem in hand images, but they require complex image acquisition systems and cannot be directly calibrated from 2D images. For example, common image acquisition systems include, but are not limited to: binocular / multi-camera-based systems, which select angles without self-occlusion from images captured from different angles for labeling; and sensor-based labeling systems, which require sensors to be worn on the hand to mitigate the effects of occlusion. These acquisition systems typically only capture images under stable laboratory conditions and are unable to simulate the complex environmental changes found in actual production environments, which affects the generalization ability of the trained model. Table 1 below compares the labeling of the two keypoint samples mentioned above.

[0088] Table 1 Comparison of samples with different markers

[0089] For the above two types of key point samples, the image key point detection method disclosed in the present invention can unify key point samples of different marking types into the same training framework, thereby simultaneously training the key point prediction model by utilizing key point samples of different marking types, so that the key point prediction model can learn the rich environmental information of two-dimensional images and the rich structural information of three-dimensional images, thereby achieving the purpose of improving the key point positioning accuracy.

[0090] According to an embodiment of the present disclosure, in the training of the key point prediction model, a training label with a unified format can be set for each hand object in each training image in the training image set, and the format can be a vector with a predetermined dimension, and the predetermined dimension can include the dimension of the classification, the key point dimension when the classification of the image area corresponding to the hand object is predicted to be the left hand, and the key point dimension when the classification of the image area corresponding to the hand object is predicted to be the right hand, wherein the key point dimension can be related to the number of key points of the hand object and the dimension of the three-dimensional coordinates of each key point.

[0091] Optionally, the sample input of the training image and the key point detection results can be based on hand objects. Specifically, a set of key point samples (which can be two-dimensional key point samples or three-dimensional key point samples) can be collected for each hand object in the training image, and the output of the key point prediction model can also be a set of key point location predictions for each hand object in the training image.

[0092] As described above, each hand object in the training image can be provided with a training label having a unified format, wherein the unified format can include a vector having a fixed length and a length partition setting, and the length partition setting of the vector can include a length interval for representing the aforementioned classification prediction, a length interval for representing a key point set when the classification prediction of the image area corresponding to the hand object is the left hand, and a length interval for representing a key point set when the classification prediction of the image area corresponding to the hand object is the right hand. For example, in order to encode labeled samples under different labeling rules into a unified label format, in an embodiment of the present disclosure, the training label format of the hand object can be uniformly set to have a fixed length N cls +3N p +3N p A vector, wherein the length division setting of the vector includes three length intervals, which correspond to N cls 、3N p and 3N p Among them, N cls It can represent the dimension of the subdivision category. For example, here, different labeled samples can include three subdivision categories: left hand, right hand, and background. Therefore, N cls =3, where each dimension can indicate the probability that the hand object belongs to each category. p It can represent the number of hand key points. Here, as an example, according to the commonly used hand key point marking rules, each hand can contain 21 hand key points, so N p =21,3N p Indicates N pThe three-dimensional coordinate dimensions of the hand key points (i.e., key point dimensions). p +3N p That is, the key point positioning results when the classification of the image area is predicted as the left hand and the right hand respectively. The final key point prediction result of the key point prediction model can have 3N p Of course, the above-mentioned setting method of training labels is only used as an example and not as a limitation in this disclosure. The image key point detection method disclosed in this disclosure can also adopt various other setting methods.

[0093] According to an embodiment of the present disclosure, in the training of the key point prediction model, a mask vector with the same dimension as the key point dimension can also be set for each hand object in each training image in the training image set, so as to uniformly process the two-dimensional key point samples and the three-dimensional key point samples.

[0094] Optionally, in order to make the two-dimensional key point samples and the three-dimensional key point samples have a unified representation, a mask data structure, i.e., a mask vector, can be set for each hand object with the same dimension as the final key point output (that is, the same dimension as the above-mentioned key point dimension).

[0095] FIG7 is a schematic diagram illustrating the final keypoint prediction of a hand object and mask vectors corresponding to two-dimensional keypoint samples and three-dimensional keypoint samples according to an embodiment of the present disclosure. As shown in FIG7 , the final keypoint prediction of the hand object and the mask vectors corresponding to the two-dimensional keypoint samples and three-dimensional keypoint samples (i.e., 2D mask vectors and 3D mask vectors) have the same format, with the three-dimensional coordinate dimension of each keypoint as the unit.

[0096] Among them, according to an embodiment of the present disclosure, when two-dimensional key point samples are used for training the hand object, the value of the dimension used to indicate depth information in the three-dimensional coordinate dimension of each key point in the mask vector of the hand object can be set to 0, and the values ​​of the remaining two-dimensional coordinate dimensions can all be set to 1; and when three-dimensional key point samples are used for training the hand object, the value of the three-dimensional coordinate dimension of each key point in the mask vector of the hand object can all be set to 1.

[0097] Optionally, as shown in Figure 7, the final key point prediction result of each hand label can correspond to a hand mask vector of the same dimension. Specifically, when the input is a three-dimensional key point sample, the masks of all dimensions in the mask vector are 1, and when the input is a two-dimensional key point sample, the masks corresponding to the depth bits in the mask vector are all set to 0.

[0098] The following will introduce the specific encoding methods of the training labels of hand objects under the above two marking rules.

[0099] For 3D keypoint samples, the original data usually contains the image and the world coordinates of the hand in the image in the real 3D space. Therefore, the world 3D coordinates can be converted to camera 3D coordinates using the following equation:

[0100] Among them, X w 、Y w , Z w Represents the coordinates of the key points of the hand in the world coordinate system, and They are the rotation matrix and translation matrix obtained by camera calibration, X c 、Y c , Z c is the coordinate of the hand key points in the transformed camera coordinate system.

[0101] Next, the camera coordinates can be further converted to image coordinates. Optionally, different transformation methods can be used for the x, y, and z coordinates. For the x and y coordinates, similar triangles can be used to establish the relationship between the camera coordinates and the image coordinates. For the depth information z, the absolute depth can be converted to relative depth before the transformation.

[0102] For example, the specific transformation formula can be expressed as:

[0103] Among them, f is the focal length of the camera, x and y are X c 、Y c For the depth information z, we can first calculate the absolute depth Z of each hand key point. c Transform to the relative depth Z relative to the root node (for example, the root node here can be selected as the key point at the wrist) c -Z0, where Z0 is the depth value of the root node. Then, the same transformation coefficient can be used Transform the relative depth to a space consistent with the x and y scales.

[0104] For 2D keypoint samples, the original data may contain the image and the x, y coordinates of the hand keypoints manually marked on the image, but lacks depth coordinate information. Therefore, z can be set to zero and the corresponding bit of the depth information in the mask vector can be set to 0.

[0105] As described above, the above-mentioned training labels and mask vectors related to the marking rules of the key point samples can be set for each hand object in the training image, so that the key point samples under different marking rules (for example, two-dimensional key point samples and three-dimensional key point samples) are unified into the same training framework to perform consistent training operations on the key point samples under different marking rules. By using a unified training framework to train the key point prediction model for key point samples under different marking rules, the key point prediction model can learn information of various dimensions from, for example, two-dimensional key point samples and three-dimensional key point samples, and predict information of the remaining dimensions when only having prior information of some dimensions. For example, the trained key point detection model can obtain the three-dimensional key point positioning information from the input two-dimensional image.

[0106] Therefore, next, according to an embodiment of the present disclosure, in the training of the key point prediction model, the key point prediction loss function of the key point prediction model for the hand object can be determined based on the difference between the prediction results of the key point prediction model for each hand object and the training label of the hand object, and the mask vector, and the key point prediction model can be adjusted based on the optimization of the loss function.

[0107] Optionally, the above-mentioned mask vector can be used as a weighting coefficient for the difference between the prediction result of the key point prediction model for each hand object's key point set and the training label of the hand object, so that the calculation of the loss function can adopt a consistent calculation method for key point samples under different labeling rules, thereby reducing the computational complexity.

[0108] As an example, the least squares method can be used to calculate the key point prediction loss, where the least squares loss function after introducing the mask can be defined as:

[0109] Among them, r i,x 、r i,y 、r i,z is the prediction result of the key point prediction model for the i-th key point of the hand object, g i,x 、g i,y 、g i,z is the manual calibration result of the i-th key point (i.e., the coordinate value corresponding to the i-th key point in the training label of the hand object), M i,x 、M i,y 、M i,z is the mask value corresponding to each dimension of the i-th key point in the mask vector. For example, for a two-dimensional key point sample, M i,z = 0, and for 3D key point samples, M i,z =1.

[0110] Furthermore, similar to the description above with reference to step 303, since it is not possible to know in advance whether the current image region is classified as left-hand or right-hand during the actual prediction process, to maintain consistency with the actual situation, the classification results of the detection subnetwork are also used to determine the final key point prediction results during the training and testing of the key point prediction model. In this case, even if the detection subnetwork cannot accurately classify the current image region in the early stages of training the key point prediction model, as the training process progresses, the goal of selecting different key point prediction results for different classifications can still be achieved.

[0111] As described above, the image key point detection method disclosed herein performs hand detection and key point positioning on the corresponding image area in the original target image based on the multi-dimensional feature vector of each pixel in the feature map extracted from the target image, wherein the classification of the image area and the prediction of different key points about the left and right hands in the image area are performed synchronously, and the key point prediction result is selected as the key point set under the pixel based on the classification result, thereby realizing the detection of all hand key points in the original target image. The method of the embodiment of the present disclosure can train a single key point prediction model that performs hand detection and hand key point positioning at the same time, reducing the algorithm development cycle and cost, facilitating iterative optimization, and by utilizing two-dimensional and three-dimensional labeled data, it can provide rich scene changes and accurately describe the hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved.

[0112] FIG8 is a schematic diagram illustrating an image key point detection apparatus 800 according to an embodiment of the present disclosure.

[0113] According to an embodiment of the present disclosure, the image key point detection device 800 may include an image acquisition module 801 , a feature extraction module 802 , and a key point prediction module 803 .

[0114] The image acquisition module 801 may be configured to acquire a target image. Optionally, the image acquisition module 801 may perform the operations described above with reference to step 301 .

[0115] Optionally, the target image can be a single image or a video frame from a video, and the target image can include or exclude the target object to be detected (for example, a hand target in the present disclosure). For example, the target image can be an image of the current head-mounted display perspective, such as captured by a head-mounted display device, or an image frame, such as captured by a video capture device.

[0116] The feature extraction module 802 may be configured to extract a feature map of the target image, wherein each pixel in the feature map has a multi-dimensional feature vector associated with a corresponding image region in the target image. Optionally, the feature extraction module 802 may perform the operations described above with reference to step 302 .

[0117] Optionally, the input target image can first be normalized to a standard size and then fed into a convolutional neural network to perform feature transformation on the target image, thereby transforming the original image into a high-dimensional space to obtain the image features of the target image in the high-dimensional space. For example, a deep fully convolutional neural network can be used to extract image features. For an image with an input dimension of W×H, the convolutional neural network can output a feature map of dimensions w×h×c, that is, an image with a length of w and a width of h, where each pixel in the image contains a c-dimensional feature vector that describes the image information within the corresponding receptive field in the original target image.

[0118] The key point prediction module 803 can be configured to: for each pixel in the feature map, determine the classification of the image region corresponding to the pixel based on the multidimensional feature vector of the pixel, the classification including left hand and right hand; based on the multidimensional feature vector of the pixel, predict a first key point set in the image region when the classification of the image region corresponding to the pixel is left hand, and a second key point set in the image region when the classification of the image region is right hand; and based on the determined classification of the image region corresponding to the pixel, select one from the first key point set and the second key point set as the key point set corresponding to the pixel. Optionally, the key point prediction module 803 can perform the operations described above with reference to step 303.

[0119] In the image key point detection device disclosed in the present invention, after obtaining a feature map of the target image through the feature extraction module 802, the feature map can be input into the key point prediction module 803 for detection and positioning processing, wherein the key point prediction module 803 can include two independent modules for independent prediction: a hand area detection module and a hand key point positioning module. The hand area detection module can be used to predict the category of the original image area corresponding to the current pixel in the feature map (including predicting whether the current pixel belongs to the detection target, that is, whether there is a detection target in the original image area corresponding to the pixel), and the hand key point positioning module can be used to predict the key points in the current image area. The above two modules can use multi-task learning to share the image features extracted by the feature extraction network to complete detection and positioning respectively.

[0120] For example, for each pixel in the feature map output by the depth model, target detection can be performed based on the multidimensional feature vector of the pixel. In an embodiment of the present disclosure, for a hand target, the classification of the image region in the original target image corresponding to the pixel can be determined based on the multidimensional feature vector of the pixel, that is, whether a hand target exists, and the specific classification of the hand target in the case of determining the presence of the hand target, such as left hand or right hand.

[0121] In the image key point detection device disclosed in the present invention, the left and right hands can be treated differently in the detection and positioning stage, including classifying the image area with respect to the left and right hands in the hand area detection module in the key point prediction module 803, and separately predicting the key points of the left and right hands in the hand key point positioning module in the key point prediction module 803.

[0122] Optionally, the hand region detection module within the key point prediction module 803 may determine the classification of the image region corresponding to the pixel. This module may perform object detection and classification based on the multidimensional feature vector of the pixel and output a prediction result indicating whether the image region corresponding to the pixel belongs to each classification. For example, the prediction result may be output in the form of a probability, i.e., the probability that the image region belongs to each classification.

[0123] In addition to the above-mentioned classification of the left and right hand image regions, there may also be regions that do not include the hand target. In this case, the region can be considered as part of the background region. Therefore, according to an embodiment of the present disclosure, the classification may also include the background.

[0124] Optionally, when the image area corresponding to a pixel is determined to be background, since the detection target of the present disclosure (i.e., the hand target) is not present therein, subsequent processing may not be performed on the image area. In other words, when the image area corresponding to the pixel is determined to be background, the key point prediction module 803 may skip subsequent positioning processing of the pixel and directly proceed to detection and positioning of the next pixel.

[0125] Optionally, as described above, in the key point prediction module 803, separate detection can be used for the left and right hands. That is, two processing branches are simultaneously used in the hand key point localization module to predict two sets of key points, corresponding to the key points when the image region is classified as the left hand and the key points when the image region is classified as the right hand. As described above, different localization branches can be set for the left hand and the right hand in the key point localization subnetwork, and two different parameterized models can be used to predict feature points based on the same feature input to obtain key point sets for both cases.

[0126] Optionally, when it is determined that there is a detection target in the image area, and the key point prediction when the image area is classified as the left hand and the key point prediction when the image area is classified as the right hand are obtained respectively through two processing branches, a selection can be made from the two key point predictions based on the classification of the image area, that is, a set of key point predictions corresponding to the classification is selected as the final key points.

[0127] Optionally, for situations where an image region is classified as background, the image keypoint detection method of the present disclosure may further include: for each pixel in the feature map, if the image region corresponding to the pixel is classified as background, not performing the prediction and selection on the pixel. In other words, when the detection subnetwork determines that there is no detection target in the image region corresponding to the pixel, the keypoint prediction module 803 does not perform any subsequent processing on the image region, but directly proceeds to the detection and positioning processing of the next pixel.

[0128] According to another aspect of the present disclosure, an image key point detection device is further provided. FIG9 shows a schematic diagram of an image key point detection device 2000 according to an embodiment of the present disclosure.

[0129] As shown in FIG9 , the image key point detection device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 may store computer-readable code, which, when executed by the one or more processors 2010, may execute the image key point detection method described above.

[0130] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor, and may be an X86 architecture or an ARM architecture.

[0131] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0132] For example, the method or apparatus according to an embodiment of the present disclosure may also be implemented with the aid of the architecture of the computing device 3000 shown in FIG10 . As shown in FIG10 , the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, and the like. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the image key point detection method provided by the present disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, the architecture shown in FIG10 is merely exemplary. When implementing different devices, one or more components in the computing device shown in FIG10 may be omitted according to actual needs.

[0133] According to another aspect of the present disclosure, a computer-readable storage medium is also provided. The computer storage medium has computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor, the image key point detection method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0134] Embodiments of the present disclosure also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image keypoint detection method according to an embodiment of the present disclosure.

[0135] Embodiments of the present disclosure provide a method, apparatus, device, and computer-readable storage medium for detecting image key points.

[0136] Compared to traditional hand keypoint localization methods, the method provided by the embodiments of the present disclosure can simultaneously perform hand detection and hand keypoint localization using a single keypoint prediction model trained through multi-task learning. This reduces the algorithm development cycle and cost, facilitates iterative optimization, and utilizes both two-dimensional and three-dimensional labeled data, providing rich scene variations while accurately describing hand structure information. Furthermore, by distinguishing between left and right hand keypoint localization issues, the difficulty of keypoint localization is reduced and positioning accuracy is improved.

[0137] The method provided by the embodiment of the present disclosure performs hand detection and key point positioning on the corresponding image area in the original target image based on the multi-dimensional feature vector of each pixel in the feature map extracted from the target image, wherein the classification of the image area and the prediction of different key points about the left and right hands in the image area are performed synchronously, and the key point prediction result is selected as the key point set under the pixel based on the classification result, thereby realizing the detection of all hand key points in the original target image. The method of the embodiment of the present disclosure can train a single key point prediction model that performs hand detection and hand key point positioning at the same time, reducing the algorithm development cycle and cost, facilitating iterative optimization, and by utilizing two-dimensional and three-dimensional labeled data, it can provide rich scene changes and accurately describe hand structure information. In addition, by distinguishing the key point positioning problems of the left and right hands, the difficulty of key point positioning is reduced and the positioning accuracy is improved.

[0138] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of the code, and the module, program segment, or a part of the code contains at least one executable instruction for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0139] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0140] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will appreciate that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for detecting key points of an image, comprising: Get the target image; Extracting a feature map of the target image, each pixel in the feature map having a multi-dimensional feature vector associated with a corresponding image region in the target image; as well as For each pixel in the feature map: Based on the multidimensional feature vector of the pixel, determining a classification of the image region corresponding to the pixel, the classification comprising left hand and right hand; Based on the multidimensional feature vector of the pixel, predicting a first key point set in the image region when the image region corresponding to the pixel is classified as a left hand, and a second key point set in the image region when the image region is classified as a right hand; as well as Based on the determined classification of the image region corresponding to the pixel, one of the first key point set and the second key point set is selected as the key point set corresponding to the pixel.

2. The method of claim 1, wherein: Extracting the feature map of the target image includes: A pre-trained convolutional neural network is used to extract a feature map of the target image, wherein a multidimensional feature vector of each pixel in the feature map indicates image information in a corresponding image area in the target image, and the corresponding image area is related to a receptive field area of ​​the convolutional neural network.

3. The method of claim 1, wherein: The determining of the classification of the image region corresponding to the pixel is performed by a pre-trained detection subnetwork, and the determining of the classification of the image region corresponding to the pixel comprises: Detecting whether there is a target object in the image area, the target object includes a left hand and a right hand of a human body; In the case where a left hand of a human body is detected in the image region, determining that the image region is classified as a left hand; and In the case where the right hand of the human body is detected to exist in the image region, the classification of the image region is determined to be the right hand.

4. The method of claim 3, wherein: Said classification also includes background; The determining of the classification of the image region corresponding to the pixel further comprises: In the case where it is detected that the left hand and the right hand of the human body do not exist in the image region, determining that the image region is classified as background; The method further comprises: For each pixel in the feature map, if the image region corresponding to the pixel is classified as background, the prediction and selection are not performed on the pixel.

5. The method of claim 3, wherein: The prediction of the first set of key points and the second set of key points is performed by a pre-trained key point localization subnetwork, wherein in the key point localization subnetwork, two different localization models are used to respectively predict the key point sets in the image region corresponding to the pixel when the image region is classified as the left hand and the right hand.

6. The method of claim 5, wherein: The detection subnetwork and the key point localization subnetwork constitute a key point prediction model, which takes the feature map of the target image as input and takes a set of key points in the target image as output, wherein each key point in the key point set has depth information.

7. The method of claim 6, wherein: The pre-trained detection sub-network and the pre-trained key point localization sub-network are obtained by training the key point prediction model; The key point prediction model is trained based on a training image set including two-dimensional key point samples and three-dimensional key point samples, and the three-dimensional key point samples have depth information.

8. The method of claim 7, wherein: Each training image in the training image set includes one or more hand objects, and each of the one or more hand objects has a two-dimensional key point sample or a three-dimensional key point sample for training; The two-dimensional key point samples indicate the two-dimensional coordinates of the key points associated with the hand object in the training image, and the three-dimensional key point samples indicate the three-dimensional coordinates of the key points associated with the hand object in the training image, and the three-dimensional coordinates include the two-dimensional coordinates and depth information.

9. The method of claim 8, wherein: In the training of the key point prediction model, a training label with a unified format is set for each hand object in each training image in the training image set, and the format is a vector with a predetermined dimension, and the predetermined dimension includes the classification dimension, the key point dimension when the classification of the image area corresponding to the hand object is predicted to be the left hand, and the key point dimension when the classification of the image area corresponding to the hand object is predicted to be the right hand, wherein the key point dimension is related to the number of key points of the hand object and the dimension of the three-dimensional coordinates of each key point.

10. The method of claim 9, wherein: In the training of the key point prediction model, a mask vector having the same dimension as the key point dimension is also set for each hand object in each training image in the training image set, so as to uniformly process the two-dimensional key point samples and the three-dimensional key point samples; Wherein, when a two-dimensional key point sample is used for training the hand object, the value of the dimension indicating the depth information in the three-dimensional coordinate dimension of each key point in the mask vector of the hand object is set to 0, and the values ​​of the remaining two-dimensional coordinate dimensions are all set to 1; and When the three-dimensional key point samples are used for training the hand object, the value of the three-dimensional coordinate dimension of each key point in the mask vector of the hand object is set to 1.

11. The method of claim 10, wherein: During the training of the key point prediction model, the key point prediction loss function of the key point prediction model for each hand object is determined based on the difference between the prediction result of the key point prediction model for the key point set of each hand object and the training label of the hand object, and the mask vector, and the key point prediction model is adjusted based on the optimization of the loss function.

12. An image key point detection device, comprising: An image acquisition module is configured to acquire a target image; a feature extraction module configured to extract a feature map of the target image, each pixel in the feature map having a multi-dimensional feature vector associated with a corresponding image region in the target image; as well as The key point prediction module is configured to, for each pixel in the feature map: Based on the multidimensional feature vector of the pixel, determining a classification of the image region corresponding to the pixel, the classification comprising left hand and right hand; Based on the multidimensional feature vector of the pixel, predicting a first key point set in the image region when the image region corresponding to the pixel is classified as a left hand, and a second key point set in the image region when the image region is classified as a right hand; as well as Based on the determined classification of the image region corresponding to the pixel, one of the first key point set and the second key point set is selected as the key point set corresponding to the pixel.

13. An image key point detection device, comprising: one or more processors; as well as One or more memories, wherein a computer executable program is stored, and when the computer executable program is executed by the processor, the method of any one of claims 1-11 is performed.

14. A computer program product stored on a computer-readable storage medium and comprising computer instructions which, when executed by a processor, cause a computer device to perform the method of any one of claims 1 to 11.

15. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the instructions are used to implement the method according to any one of claims 1 to 11 when executed by a processor.

Images