Keypoint detection method and apparatus

Abstract

This disclosure provides a keypoint detection method and apparatus. The method includes: determining a target image; detecting keypoint coordinates in the target image using a first network in a keypoint detection model to obtain predicted keypoint coordinates of the target image; wherein the keypoint detection model further includes a second network, which generates a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During multiple training processes of the keypoint detection model, the keypoint heatmap output by the second network is used to assist in improving the keypoint prediction accuracy of the first network. Thus, in the keypoint detection model, while ensuring the efficiency of keypoint detection by utilizing keypoint coordinate detection, the accuracy of keypoint coordinate detection is improved by utilizing the keypoint heatmap prediction, achieving a balance between keypoint detection efficiency and keypoint detection accuracy.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a key point detection method and device. Background Technology

[0002] Keypoint detection is a task that uses neural networks to predict key points on a target object in a given image. For example, facial keypoint detection involves using a neural network to predict facial keypoints in a given face image. These keypoints are points that are defined in advance by humans, such as eyebrow points, corner of the eye points, and facial contour points.

[0003] Currently, keypoint detection methods based on heatmaps focus on global features and geometric information of the image to detect keypoints, achieving high detection accuracy. However, to maintain image resolution, these methods perform upsampling or multi-level feature fusion operations, resulting in deep network structures, high computational cost, and long inference times.

[0004] Therefore, how to balance the detection accuracy and efficiency of key points is an urgent problem to be solved. Summary of the Invention

[0005] This disclosure provides a key point detection method and device to overcome the problem of balancing key point detection accuracy and efficiency in images.

[0006] In a first aspect, embodiments of this disclosure provide a key point detection method, including:

[0007] Identify the target image;

[0008] The first network in the key point detection model is used to detect the key point coordinates of the target image and obtain the predicted key point coordinates of the target image.

[0009] The keypoint detection model also includes a second network, which is used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to help improve the keypoint prediction accuracy of the first network.

[0010] Secondly, embodiments of this disclosure provide a model training method, including:

[0011] Determine the training data, which includes training images and the corresponding sample labels for the training images;

[0012] The keypoint detection model is trained based on the training data;

[0013] The keypoint detection model includes a first network and a second network. The first network is used to detect keypoint coordinates in the image, and the second network is used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to help improve the keypoint prediction accuracy of the first network.

[0014] Thirdly, embodiments of this disclosure provide a key point detection device, including:

[0015] The determining unit is used to determine the target image;

[0016] The detection unit is used to detect the key point coordinates of the target image through the first network in the key point detection model, and obtain the predicted key point coordinates of the target image.

[0017] The keypoint detection model also includes a second network, which is used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to help improve the keypoint prediction accuracy of the first network.

[0018] Fourthly, embodiments of this disclosure provide a model training device, comprising:

[0019] The determining unit is used to determine the training data, which includes training images and the sample labels corresponding to the training images.

[0020] The training unit is used to train the keypoint detection model based on the training data.

[0021] The keypoint detection model includes a first network and a second network. The first network is used to detect keypoint coordinates in the image, and the second network is used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to help improve the keypoint prediction accuracy of the first network.

[0022] Fifthly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, causing the at least one processor to perform the key point detection method as described in the first aspect or various possible designs of the first aspect, or to perform the model training method as described in the second aspect or various possible designs of the second aspect.

[0023] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the key point detection method as described in the first aspect or various possible designs of the first aspect, or implements the model training method as described in the second aspect or various possible designs of the second aspect.

[0024] In a seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, the computer program product comprising computer execution instructions, which, when executed by a processor, implement the key point detection method as described in the first aspect or various possible designs of the first aspect, or implement the model training method as described in the second aspect or various possible designs of the second aspect.

[0025] The keypoint detection method and apparatus provided in this disclosure detect keypoint coordinates of a target image through a first network in a keypoint detection model, thereby obtaining predicted keypoint coordinates of the target image. The keypoint detection model also includes a second network, which generates a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to improve the keypoint detection accuracy of the first network.

[0026] Therefore, this embodiment of the disclosure considers that the network used for keypoint coordinate prediction has a smaller structure and higher keypoint detection efficiency, while the network used for keypoint heatmap prediction has a deeper structure and higher keypoint detection accuracy. Based on the complementarity of these two, a keypoint detection model is designed that includes a first network for keypoint coordinate detection and a second network for keypoint heatmap generation. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to improve the keypoint detection accuracy of the first network. Thus, in keypoint detection, the first network ensures both keypoint detection efficiency and improves keypoint detection accuracy, achieving a balance between the two. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 An example diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0029] Figure 2Schematic diagram of the key point detection model provided in the embodiments of this disclosure Figure 1 ;

[0030] Figure 3 Schematic flowchart of the key point detection method provided in the embodiments of this disclosure Figure 1 ;

[0031] Figure 4 A schematic flowchart illustrating the model training method provided in this embodiment of the disclosure;

[0032] Figure 5 A flowchart illustrating a single training process of the keypoint detection model provided in this embodiment of the disclosure. Figure 1 ;

[0033] Figure 6 A flowchart illustrating a single training process of the keypoint detection model provided in this embodiment of the disclosure. Figure 2 ;

[0034] Figure 7 Schematic diagram of the key point detection model provided in the embodiments of this disclosure Figure 2 ;

[0035] Figure 8 A flowchart illustrating a single training process of the keypoint detection model provided in this embodiment of the disclosure. Figure 3 ;

[0036] Figure 9 This is a structural block diagram of the key point detection device provided in the embodiments of this disclosure;

[0037] Figure 10 A structural block diagram of the model training device provided in the embodiments of this disclosure;

[0038] Figure 11 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0040] refer to Figure 1 , Figure 1 This is an example diagram illustrating an application scenario provided by an embodiment of this disclosure.

[0041] like Figure 1As shown, the application scenario is image keypoint detection, which includes an image processing device 101. In this scenario, the image processing device 101 detects keypoints on a given image using a neural network. These keypoints are points that are predefined by a person. The image processing device 101 can be a terminal or a server. Figure 1 Taking the image processing device 101 as an example as a server.

[0042] Optionally, the application scenario also includes an image acquisition device 102, which communicates with the image processing device 101, for example, via a network. The image acquisition device 102 can send the acquired images to the image processing device 101 for key point detection. The image acquisition device 102 is a terminal device with a camera, such as a camera, mobile phone, tablet computer, smart wearable device, or smart home appliance.

[0043] The application scenario can be real-time online image processing, where the image processing device 101 performs key point detection on images acquired in real-time by the image acquisition device 102. Alternatively, it can be offline image processing, where the image processing device 101 performs key point detection on images not acquired in real-time.

[0044] The image processing device 101 and the image acquisition device 102 can be the same device, for example, performing key point detection on an image taken by a user using a mobile phone. Alternatively, the image processing device 101 and the image acquisition device 102 can be different devices.

[0045] Optionally, this application scenario involves facial landmark detection in images. For example, with user authorization, facial landmark detection can be performed on a user's selfie image to identify the user or generate interesting image effects based on the identified facial landmarks. Facial landmarks include, for example, eyebrow points, corner of the eye points, and facial contour points.

[0046] In related technologies, facial landmark detection methods are mainly divided into two types:

[0047] (1) A regression-based method is used, which combines a neural network with coordinate regression to predict the coordinates of facial key points in an image. In this method, a fully connected layer is connected after the backbone network. The fully connected layer performs coordinate regression based on the abstract image features extracted by the backbone network and outputs the predicted coordinates of multiple key points. For example, for a task of detecting 10 key points, the fully connected layer will output 10×2 values, which represent the X and Y coordinates of the 10 key points.

[0048] (2) Heatmap-based method: This method uses a neural network to predict heatmaps corresponding to multiple facial key points in an image. The color intensity of each pixel in the heatmap represents the probability that the pixel is a key point, and one key point can be identified on each heatmap. For example, for a task to detect 10 key points, 10 heatmaps will be output, with one heatmap corresponding to the location of one key point.

[0049] The inventors discovered that regression-based methods, with their smaller neural network size and faster keypoint detection speed, primarily extract global features without considering geometric information, resulting in lower keypoint detection accuracy. Heatmap-based methods, on the other hand, address geometric information and achieve higher keypoint detection accuracy. However, to maintain image resolution, they typically involve upsampling or multi-level feature fusion operations, leading to deeper networks, higher computational cost, and longer inference time, further reducing keypoint detection efficiency. Therefore, these methods struggle to balance keypoint detection accuracy and efficiency. Furthermore, there is currently no method that combines keypoint coordinate regression with keypoint heatmap prediction.

[0050] The inventors discovered that the backbone network of the neural network in the regression-based method and the backbone network of the neural network in the heatmap-based method can be the same, both used to extract abstract image features. Therefore, key point coordinate prediction and key point heatmap prediction can be combined to solve the above problems by utilizing the complementarity of key point coordinate prediction and key point heatmap prediction, that is, to improve the accuracy and efficiency of key point detection, and to achieve a balance between key point detection accuracy and key point detection efficiency.

[0051] Based on the above ideas, this disclosure provides a keypoint detection method and device. In this embodiment, the keypoint detection model includes a first network and a second network. The first network is used for predicting the coordinates of keypoints in an image, and the second network is used to generate a keypoint heatmap based on the keypoint coordinates output by the first network and the image features extracted by the first network. During model training, the keypoint heatmap output by the second network is used to help improve the keypoint prediction accuracy of the first network. When the model is applied, the first network is used to detect the coordinates of keypoints in the image.

[0052] Therefore, based on the high keypoint detection efficiency of the keypoint coordinate regression network (i.e., the first network), the keypoint heatmap prediction network (i.e., the first network and the second network) which simultaneously considers the spatial geometric information and global abstract features of the image is used to improve the keypoint detection accuracy of the keypoint coordinate prediction network. That is, by utilizing the complementarity of keypoint coordinate prediction and keypoint heatmap prediction, the accuracy of keypoint coordinate prediction is improved. This saves network computation while ensuring keypoint detection accuracy, thus achieving a balance between keypoint detection efficiency and keypoint detection accuracy.

[0053] Optionally, the image detected in this embodiment is a face image, and the key points are facial key points.

[0054] Below, several embodiments of the key point detection method are provided. It should be noted that the execution subject of the method embodiments of this disclosure is an electronic device, which may be a terminal or a server.

[0055] refer to Figure 2 , Figure 2 Schematic diagram of the key point detection model provided in the embodiments of this disclosure Figure 1 .

[0056] like Figure 2 As shown, the keypoint detection model includes a first network and a second network. The first network includes a feature extraction layer and a coordinate regression layer, and the second network includes a feature extraction layer (…). Figure 2 (Not shown). The feature extraction layer in the first network can be considered as a shared feature extraction layer between the first and second networks. After the image is input into the first network, the feature extraction layer in the first network extracts a feature map. This feature map is then input into the coordinate regression layer to obtain the predicted keypoint coordinates output by the first network. Based on the feature map extracted by the feature extraction layer in the first network and the predicted keypoint coordinates output by the first network, the second network performs keypoint heatmap prediction to obtain the predicted keypoint heatmap output by the second network.

[0057] In this network, the coordinate regression layer generates predicted keypoint coordinates based on the image features output by the feature extraction layer, and the feature extraction layer in the second network is used to perform global abstract feature extraction based on the image features extracted by the feature extraction layer in the first network and the predicted keypoint coordinates output by the first network, thereby generating a predicted keypoint heatmap.

[0058] from Figure 2 As can be seen, the key point detection model includes a complete key point heatmap generation model, which corresponds to the key point detection method based on heatmap. Considering that the structure of this model is relatively deep and the key point detection efficiency is not high, the first network can be used for key point detection to ensure the key point detection efficiency. At the same time, during the training process, the second network is used to help improve the key point detection accuracy of the first network.

[0059] based on Figure 2 The key point detection model shown is for reference. Figure 3 , Figure 3 Schematic flowchart of the key point detection method provided in the embodiments of this disclosure Figure 1 .like Figure 3 As shown, the key point detection method includes:

[0060] S301. Determine the target image.

[0061] The target image is the image from which key point detection of the target object is to be performed.

[0062] Optionally, the target object is a face, the key points are predefined facial key points, such as eyebrow points, corner of the eye points, facial contour points, etc., and the target image is the image to be detected for facial key points.

[0063] In this embodiment, one or more target images are determined.

[0064] In one example, the target image is acquired by user input, or a target image is received from an image acquisition device, or an image displayed on the user terminal or a video frame in a video played on the user terminal is determined as the target image, so as to perform real-time online key point detection on the target image.

[0065] In another example, a target image is obtained from an image database or a video database to perform offline keypoint detection on the target image.

[0066] S302. Through the first network in the key point detection model, the key point coordinates of the target image are detected to obtain the predicted key point coordinates of the target image.

[0067] The keypoint detection model comprises a first network and a second network. The first network predicts the coordinates of keypoints in the image, while the second network predicts a keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. Therefore, it can be understood that the first and second networks share the same feature extraction layer (or the same backbone network). The first network predicts the coordinates of keypoints in the image based on the feature map extracted by the feature extraction layer, and the second network predicts the heatmap of keypoints in the image based on the feature map extracted by the feature extraction layer and the keypoint coordinates output by the first network.

[0068] The key point prediction model was obtained through multiple training sessions.

[0069] During the multiple training iterations of the keypoint prediction model, the keypoint heatmap output by the second network is used to assist in improving the keypoint prediction accuracy of the first network. Since the first network is used for keypoint coordinate prediction in an image, it focuses on the coordinates of keypoints on the image, i.e., the spatial geometric information of the image. The second network, however, generates a keypoint heatmap based on the keypoint coordinates output by the first network and the image features extracted by the first network. While the first network focuses on spatial geometric information, the second network focuses on the global abstract features of the image. Therefore, during training, the difference between the keypoint heatmap output by the second network and the actual keypoint heatmap can be used to assist the training of the first network, improving its keypoint detection accuracy without increasing its computational load.

[0070] In this embodiment, the target image can be input into the keypoint detection model, or the target image can be preprocessed such as denoising and cropping before being input into the keypoint detection model. In the keypoint detection model, the target image features can first be extracted through the feature extraction layer in the first network to obtain the feature map of the target image. Then, the coordinate regression layer in the first network processes the feature map of the target image to obtain the predicted keypoint coordinates of the target image, that is, the predicted coordinates of the keypoints on the target image.

[0071] In this embodiment, the keypoint detection model includes a first network for keypoint coordinate prediction and a second network for generating a keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During training, the second network is used to assist in improving the keypoint prediction accuracy of the first network. Therefore, in the keypoint detection process, utilizing the first network in the keypoint detection model to detect keypoint coordinates in the target image effectively improves the efficiency and accuracy of keypoint detection, achieving a balance between efficiency and accuracy in keypoint detection.

[0072] Since the training process of the key point detection model in this disclosure is quite critical, several embodiments of the training process of the key point detection model are provided below.

[0073] based on Figure 2 The key point detection model shown is for reference. Figure 4 , Figure 4 This is a schematic flowchart illustrating the model training method provided in an embodiment of this disclosure. Figure 4 As shown, the model training methods include:

[0074] S401. Determine the training data.

[0075] The training data includes training images and their corresponding sample labels. The training images serve as training samples, and the training images and their corresponding sample labels are used for supervised training of the keypoint detection model. The sample labels corresponding to the training images may include, for example, the actual coordinates of keypoints on the training images, or the location markers of keypoints on the training images.

[0076] In this embodiment, training data can be obtained from a database, from an authorized network or other authorized platform, or from user-input training data.

[0077] Optionally, the sample labels corresponding to the training images include the actual keypoint coordinates of the training images and the actual keypoint heatmap of the training images.

[0078] The actual keypoint coordinates of the training image are the true coordinates of the keypoints on the training image, and the actual keypoint heatmap of the training image is the true heatmap of the keypoints on the training image. These coordinates and heatmaps can be obtained through manual annotation or by processing the training image using a high-precision keypoint detection model.

[0079] S402. Based on the training data, train the key point detection model multiple times.

[0080] The structure of the key point detection model can be referred to the description in the foregoing embodiments, and will not be repeated here.

[0081] In this embodiment, training images from the training data can be input into a keypoint detection model. The keypoint detection model performs keypoint detection on the training images, obtaining the output data of the keypoint detection model. The output data includes the predicted keypoint coordinates of the training images output by a first network and the predicted keypoint heatmap of the training images output by a second network. Based on the output data of the keypoint detection model and the sample labels corresponding to the training images, the model parameters of the keypoint detection model are adjusted. When the sample labels corresponding to the training images include the actual keypoint coordinates and the actual keypoint heatmap of the training images, the model parameters of the keypoint detection model are adjusted based on the predicted keypoint coordinates of the training images output by the first network, the predicted keypoint heatmap of the training images output by the second network, the actual keypoint coordinates of the training images, and the actual keypoint heatmap of the training images, completing one training iteration. This training is performed multiple times until the number of training iterations exceeds a threshold, or until the difference between the output of the keypoint detection model and the sample labels of the training samples is less than a difference threshold, at which point the training of the keypoint detection model ends.

[0082] Among them, the predicted key point heatmap of the training image refers to the heatmap of multiple key points obtained by predicting (i.e. detecting) key points in the training image.

[0083] refer to Figure 5 , Figure 5 A flowchart illustrating a single training process of the keypoint detection model provided in this embodiment of the disclosure. Figure 1 .like Figure 5 As shown, a training process includes:

[0084] S501. Through the first network, key point coordinate detection is performed on the training image to obtain the predicted key point coordinates and feature map of the training image.

[0085] In this embodiment, the training image is input into the first network, and the feature extraction layer in the first network extracts features from the training image to obtain the feature map of the training image. Based on the feature map of the training image, the coordinates of the predicted key points of the training image are obtained in the coordinate regression layer in the first network.

[0086] S502. Based on the coordinates of the predicted key points in the training image, determine multiple predicted key points on the training image to obtain the initial predicted key point heatmap of the training image.

[0087] In this embodiment, after obtaining the predicted keypoint coordinates of the training image, multiple keypoints can be determined on the training image according to the predicted keypoint coordinates, resulting in heatmaps corresponding to multiple keypoints. Each keypoint corresponds to a heatmap, and the location of the corresponding keypoint is marked on each heatmap. For ease of distinction, the heatmap obtained based on the predicted keypoint coordinates output by the first network during training is referred to as the initial predicted keypoint heatmap of the training image.

[0088] For example, if the predicted keypoint coordinates include (1,1) and (2,2), then keypoints are determined at positions (1,1) and (2,2) in the training image respectively. An initial predicted keypoint heatmap is generated for the keypoint at position (1,1), and an initial predicted keypoint heatmap is also generated for the keypoint at position (2,2).

[0089] S503. Through the second network, feature extraction is performed on the initial key point heatmap and feature map of the training image to obtain the predicted key point heatmap of the training image.

[0090] In this embodiment, the initial predicted keypoint heatmap of the training image and the feature map of the training image output by the last feature extraction layer in the first network are input into the second network for feature extraction, resulting in the predicted keypoint heatmap output by the second network, i.e., the predicted keypoint heatmap of the training image. Thus, in this embodiment, in addition to the image features of the training image, global image information is also provided to the second network, facilitating the extraction of more global image features by the second network, improving the accuracy of the predicted keypoint heatmap generated by the second network, and consequently improving the keypoint detection accuracy of the first network.

[0091] S504. Adjust the parameters of the first network based on the differences between the actual keypoint coordinates and the predicted keypoint coordinates of the training image, as well as the differences between the actual keypoint heatmap and the predicted keypoint heatmap of the training image.

[0092] The loss function of the keypoint detection model includes a first loss function and a second loss function. The loss value of the first loss function reflects the difference between the actual keypoint coordinates and the predicted keypoint coordinates of the training image, while the loss value of the second loss function reflects the difference between the actual keypoint heatmap and the predicted keypoint heatmap of the training image.

[0093] In this embodiment, based on the first loss function, the actual keypoint coordinates of the training image, and the predicted keypoint coordinates of the training image, the difference between the actual keypoint coordinates and the predicted keypoint coordinates of the training image is determined, resulting in a first loss value (i.e., the function value of the first loss function). Based on the second loss function, the actual keypoint heatmap of the training image, and the predicted keypoint heatmap of the training image, the difference between the actual keypoint heatmap and the predicted keypoint heatmap of the training image is determined, resulting in a second loss value (i.e., the function value of the second loss function). Based on the first and second loss values, the parameters of the first network are adjusted using a model optimization algorithm.

[0094] The specific formula for the loss function and the model optimization algorithm are not limited.

[0095] Therefore, compared to training the first network solely based on the difference between the predicted keypoint coordinates and the actual keypoint coordinates in the training image, this embodiment considers that the first network does not focus on global image features. During training, a second network is used to extract global image features and generate a predicted keypoint heatmap of the training image. The difference between the predicted keypoint heatmap and the actual keypoint heatmap of the training image is used to assist the training of the first network, thus enabling the second network to assist the training of the first network. Therefore, the keypoint detection accuracy of the first network is effectively improved.

[0096] Optionally, the second network can be a pre-trained network.

[0097] Optionally, the first and second networks can be trained simultaneously during the training of the keypoint detection model.

[0098] Based on simultaneously training the first and second networks during the training process of the keypoint detection model, referencing Figure 6 , Figure 6 A flowchart illustrating a single training process of the keypoint detection model provided in this embodiment of the disclosure. Figure 2 .like Figure 6 As shown, a training process includes:

[0099] S601. Through the first network, key point coordinate detection is performed on the training image to obtain the predicted key point coordinates and feature map of the training image.

[0100] S602. Based on the coordinates of the predicted key points in the training image, determine multiple predicted key points on the training image to obtain the initial predicted key point heatmap of the training image.

[0101] S603. Through the second network, feature extraction is performed on the initial key point heatmap and feature map of the training image to obtain the predicted key point heatmap of the training image.

[0102] S604. Adjust the parameters of the first network based on the differences between the actual keypoint coordinates and the predicted keypoint coordinates of the training image, as well as the differences between the actual keypoint heatmap and the predicted keypoint heatmap of the training image.

[0103] The implementation principles and technical effects of S601 to S604 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0104] S605. Adjust the parameters of the second network based on the difference between the actual key point heatmap of the training image and the predicted key point heatmap of the training image.

[0105] In this embodiment, the difference between the actual keypoint heatmap and the predicted keypoint heatmap of the training image can be determined based on the second loss function, the actual keypoint heatmap of the training image, and the predicted keypoint heatmap of the training image, thus obtaining the second loss value. Based on the second loss value, the parameters of the second network are adjusted using a model optimization algorithm.

[0106] Thus, the key point detection model is trained once, that is, the first network and the second network are trained once. During the training process, the continuous optimization of the second network will improve the optimization effect of the first network, thereby continuously improving the key point detection accuracy of the first network.

[0107] refer to Figure 7 , Figure 7 Schematic diagram of the key point detection model provided in the embodiments of this disclosure Figure 2 .

[0108] like Figure 7 As shown, in Figure 2 Based on the model structure shown, the first network includes multiple downsampling layers and coordinate regression layers, and the second network includes multiple upsampling layers. That is, the feature extraction layers in the first network are multiple downsampling layers to extract more fine-grained local image features, and the feature extraction layers in the second network are multiple upsampling layers to extract more abstract global image features.

[0109] like Figure 7As shown, the last downsampling layer in the first network connects to the coordinate regression layer and the first upsampling layer in the second network. This allows the feature map extracted by the last downsampling layer to be input into the coordinate regression layer, obtaining the predicted keypoint coordinates of the image output by the coordinate regression layer. Simultaneously, the feature map extracted by the last downsampling layer is input into the first upsampling layer of the second network. Simultaneously, the initial predicted keypoint heatmap generated based on the predicted keypoint coordinates output by the coordinate regression layer is also used as input into multiple upsampling layers in the second network. In the second network, after upsampling processing by multiple upsampling layers, the final predicted keypoint heatmap of the image is obtained.

[0110] Therefore, combining Figure 7 The image processing steps of the key point prediction model are as follows:

[0111] In the keypoint detection model, the image is sampled through multiple downsampling layers of the first network, resulting in a feature map output by the last downsampling layer. This feature map is then input into a coordinate regression layer to obtain the predicted keypoint coordinates of the image, resulting in N*2 values, where N represents the number of keypoints, and the coordinates of each keypoint can be represented as (X, Y). Based on these N*2 values, an initial predicted keypoint heatmap of the image is obtained. When each upsampling layer in the second network upsamples the feature map output by the previous network layer, the initial predicted keypoint heatmap is input as feedback information to each upsampling layer. Finally, the predicted keypoint heatmap output by the second network is obtained.

[0112] Optionally, the coordinate regression layer is a fully connected layer (FC layer).

[0113] based on Figure 7 The key point detection model shown is for reference. Figure 8 , Figure 8 A flowchart illustrating a single training process of the keypoint detection model provided in this embodiment of the disclosure. Figure 3 .like Figure 8 As shown, the training process of a keypoint detection model includes:

[0114] S801. Through the first network, key point coordinate detection is performed on the training image to obtain the predicted key point coordinates and feature map of the training image.

[0115] In this embodiment, the training image is input into a first network. Multiple downsampling layers in the first network downsample the training image multiple times, resulting in a feature map output by the last downsampling layer in the first network, which is the feature map of the training image. This feature map is then input into a coordinate regression layer in the first network to obtain the predicted keypoint coordinates of the training image.

[0116] S802. Based on the coordinates of the predicted key points in the training image, determine multiple predicted key points on the training image to obtain the initial predicted key point heatmap of the training image.

[0117] The implementation principle and technical effects of S802 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0118] S803. The feature map obtained by fusing the initial predicted key point heatmap of the training image and the feature map of the training image output by the last downsampled layer in the first network is input into the first upsampling layer in the second network for upsampling processing.

[0119] S804. The feature map obtained by fusing the initial predicted key point heatmap of the training image and the feature map output by the first upsampling layer is input into the next upsampling layer in the second network for upsampling processing. After multiple upsampling layers, the predicted key point heatmap of the training image is obtained.

[0120] In this embodiment, during training, the input data of the upsampling layer of the second network includes a feature map resulting from feature fusion of the feature map output by the previous network layer and the initial predicted keypoint heatmap of the training image. This provides richer global image features for the upsampling process, improves the accuracy of the predicted keypoint heatmap, and thus enhances the training performance of the first network.

[0121] Optionally, during the feature fusion process of the feature map output by the previous network layer of the upsampling layer and the initial predicted key point heatmap of the training image, the initial predicted heatmap can be convolved to obtain the feature map corresponding to the initial predicted heatmap. This feature map can then be merged with the feature map output by the previous network layer, for example, by adding or weighting the pixel values, or by directly merging each pixel point after merging into two pixel values, one of which comes from the feature map of the initial predicted heatmap and the other from the feature map output by the previous network layer.

[0122] S805. Adjust the parameters of the first network based on the differences between the actual keypoint coordinates and the predicted keypoint coordinates of the training image, as well as the differences between the actual keypoint heatmap and the predicted keypoint heatmap of the training image.

[0123] The implementation principle and technical effects of S805 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0124] Optionally, a training process may also include: S806, adjusting the parameters of the second network based on the difference between the actual keypoint heatmap of the training image and the predicted keypoint heatmap of the training image.

[0125] The implementation principle and technical effects of S805 can be referred to in the aforementioned embodiments, and will not be repeated here.

[0126] Therefore, in this embodiment of the present disclosure, in the second network, the initial predicted key point heatmap generated based on the predicted key point coordinates output by the first network is fused with the feature map output by the previous network layer to provide more global image information for the upsampling layer in the second network, thereby improving the prediction accuracy of the key point heatmap of the second network and thus improving the key point detection accuracy of the first network.

[0127] In some embodiments, during the process of using a keypoint detection model to detect keypoints in the target image to be identified, in addition to obtaining the predicted keypoint coordinates of the target image through the first network, a second network can be used to further detect the predicted keypoint heatmap of the target image.

[0128] In this embodiment, because the training of the first network was improved during model training (adjusting the parameters of the first network based on the differences between the actual keypoint coordinates and the predicted keypoint coordinates of the training image, as well as the differences between the actual keypoint heatmap and the predicted keypoint heatmap of the training image), the keypoint detection accuracy of the first network was improved. Since the first network precedes the second network, the keypoint detection accuracy of the second network was also improved. Therefore, continuing to use the second network after the first network to generate the predicted keypoint heatmap of the target image, although with lower detection efficiency than using only the first network, provides a more accurate predicted keypoint heatmap of the target image, thus offering a way to improve the accuracy of the keypoint heatmap.

[0129] Optionally, in the method of continuing to detect and obtain the predicted keypoint heatmap of the target image through the second network, after obtaining the predicted keypoint coordinates of the target image, multiple predicted keypoints can be determined on the target image based on the predicted keypoint coordinates to obtain an initial predicted keypoint heatmap of the target image. Then, through the second network, the initial predicted keypoint heatmap of the target image and the feature map extracted from the keypoint coordinate detection of the target image are upsampled to obtain the final predicted keypoint heatmap of the target image. Thus, by using the initial predicted keypoint heatmap obtained based on the predicted keypoint coordinates as feedback information in the second network, the richness of image features in the second network is improved, thereby improving the accuracy of the predicted keypoint heatmap of the target image.

[0130] Furthermore, through the second network, the initial predicted keypoint heatmap of the target image and the feature map extracted from the keypoint coordinate detection of the target image are upsampled to obtain the predicted keypoint heatmap of the target image. This includes: fusing the initial predicted keypoint heatmap of the target image with the feature map of the target image output from the last downsampling layer in the first network, and inputting the resulting feature map into the first upsampling layer in the second network for upsampling; fusing the initial predicted keypoint heatmap of the target image with the feature map output from the first upsampling layer in the second network, and inputting the resulting feature map into the next upsampling layer for upsampling. After multiple upsampling layers, the predicted keypoint heatmap of the target image is obtained. The specific process can be referred to the above process of processing the training image by the second network, and will not be repeated here.

[0131] Based on any of the foregoing embodiments, in some embodiments, during model application, the second network is used to generate a corresponding keypoint heatmap (predicted keypoint heatmap of the target image) based on the target image, the keypoint coordinates output by the first network (predicted keypoint coordinates of the target image), and the feature map extracted by the first network; during model training, the second network is used to generate a corresponding keypoint heatmap (predicted keypoint heatmap of the training image) based on the training image, the keypoint coordinates output by the first network (predicted keypoint coordinates of the training image), and the feature map extracted by the first network. Thus, more global image information is introduced based on the original image during the keypoint detection process of the second network to improve the keypoint detection accuracy of the second network, thereby improving the keypoint detection accuracy of the first network.

[0132] Optionally, during model application, the target image, the initial predicted keypoint heatmap of the target image obtained based on the predicted keypoint coordinates output by the first network, and the feature map output by the last downsampling layer in the first network can be fused. The feature map obtained after feature fusion is then input into the first upsampling layer in the second network. The feature map obtained after feature fusion of the target image, the initial predicted keypoint heatmap of the target image, and the feature map output by the first upsampling layer is then input into the next upsampling layer in the second network for upsampling processing. After multiple upsampling layers, the predicted keypoint heatmap of the target image is obtained, thereby improving the keypoint detection accuracy of the second network.

[0133] Optionally, during model training, the training image, the initial predicted keypoint heatmap of the training image obtained based on the predicted keypoint coordinates output by the first network, and the feature map output by the last downsampling layer in the first network can be fused. The feature map obtained after feature fusion is then input into the first upsampling layer in the second network. The feature map obtained after feature fusion of the training image, the initial predicted keypoint heatmap of the training image, and the feature map output by the first upsampling layer is then input into the next upsampling layer in the second network for upsampling processing. After multiple upsampling layers, the predicted keypoint heatmap of the training image is obtained, thereby improving the keypoint detection accuracy of the second network.

[0134] Corresponding to the key point detection method in the above embodiments, Figure 9 This is a structural block diagram of a key point detection device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 9 The key point detection equipment includes: a determination unit 901 and a detection unit 902.

[0135] The determination unit 901 is used to determine the target image.

[0136] The detection unit 902 is used to detect the key point coordinates of the target image through the first network in the key point detection model, and obtain the predicted key point coordinates of the target image.

[0137] The keypoint detection model also includes a second network, which is used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to help improve the keypoint prediction accuracy of the first network.

[0138] In some embodiments, the training process of the keypoint detection model involves multiple training iterations. One training iteration of the keypoint detection model includes: using a first network to detect keypoint coordinates in a training image, obtaining predicted keypoint coordinates and a feature map of the training image; determining multiple predicted keypoints on the training image based on the predicted keypoint coordinates, obtaining an initial predicted keypoint heatmap of the training image; using a second network to extract features from the initial predicted keypoint heatmap and the feature map of the training image, obtaining a predicted keypoint heatmap of the training image; and adjusting the parameters of the first network based on the differences between the actual keypoint coordinates and the predicted keypoint coordinates of the training image, as well as the differences between the actual keypoint heatmap and the predicted keypoint heatmap of the training image.

[0139] In some embodiments, the first network includes multiple downsampling layers and coordinate regression layers, and the second network includes multiple upsampling layers. Based on the predicted keypoint coordinates of the training image, multiple predicted keypoints are determined on the training image to obtain an initial predicted keypoint heatmap of the training image. This includes: fusing the initial predicted keypoint heatmap of the training image with the feature map of the training image output by the last downsampling layer in the first network to obtain a feature map, and inputting it into the first upsampling layer in the second network for upsampling processing; fusing the initial predicted keypoint heatmap of the training image with the feature map output by the first upsampling layer to obtain a feature map, and inputting it into the next upsampling layer for upsampling processing. After multiple upsampling layers, the predicted keypoint heatmap of the training image is obtained.

[0140] In some embodiments, after extracting features from the initial predicted keypoint heatmap and the feature map of the training image through the second network to obtain the predicted keypoint heatmap of the training image, the method further includes: adjusting the parameters of the second network based on the difference between the actual keypoint heatmap of the training image and the predicted keypoint heatmap of the training image.

[0141] In some embodiments, the key point detection device further includes: a heatmap generation unit (not shown in the figure), configured to determine multiple predicted key points on the target image based on the predicted key point coordinates of the target image, thereby obtaining an initial predicted key point heatmap of the target image; and to perform feature extraction on the initial predicted key point heatmap of the target image and the feature map extracted from the key point coordinate detection of the target image through a second network, thereby obtaining a predicted key point heatmap of the target image.

[0142] The key point detection device provided in this embodiment can be used to execute the technical solution of the above-described key point detection method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0143] Corresponding to the model training method in the above embodiments, Figure 10 This is a structural block diagram of a model training device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 10 The model training device includes: a determination unit 1001 and a training unit 1002.

[0144] The determining unit 1001 is used to determine the training data, which includes training images and sample labels corresponding to the training images.

[0145] Training unit 1002 is used to train the key point detection model based on training data;

[0146] The keypoint detection model includes a first network and a second network. The first network is used to detect keypoint coordinates in the image, and the second network is used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to help improve the keypoint prediction accuracy of the first network.

[0147] In some embodiments, the sample labels include the actual keypoint coordinates of the training image and the actual keypoint heatmap of the training image. The training process of the keypoint detection model is multiple training iterations. In one training iteration of the keypoint detection model, the training unit 1002 is specifically used to: detect keypoint coordinates of the training image through a first network to obtain the predicted keypoint coordinates of the training image and the feature map of the training image; determine multiple predicted keypoints on the training image based on the predicted keypoint coordinates to obtain an initial predicted keypoint heatmap of the training image; extract features from the initial predicted keypoint heatmap and the feature map of the training image through a second network to obtain a predicted keypoint heatmap of the training image; and adjust the parameters of the first network based on the differences between the actual keypoint coordinates and the predicted keypoint coordinates, as well as the differences between the actual keypoint heatmap and the predicted keypoint heatmap.

[0148] In some embodiments, the first network includes a downsampling layer and a coordinate regression layer, and the second network includes multiple upsampling layers. During the process of extracting features from the initial predicted keypoint heatmap and the feature map of the training image through the second network to obtain the predicted keypoint heatmap of the training image, the training unit 1002 is specifically used to: input the feature map obtained by fusing the initial predicted keypoint heatmap and the feature map of the training image output by the last downsampling layer in the first network into the first upsampling layer in the second network for upsampling processing; input the feature map obtained by fusing the initial predicted keypoint heatmap and the feature map output by the first upsampling layer into the next upsampling layer for upsampling processing; and after multiple upsampling layers, obtain the predicted keypoint heatmap.

[0149] In some embodiments, the training unit 1002 is further configured to: adjust the parameters of the second network based on the difference between the actual key point heatmap and the predicted key point heatmap.

[0150] The model training device provided in this embodiment can be used to execute the technical solutions of the above-described model training method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.

[0151] refer to Figure 11The diagram illustrates a structural schematic of an electronic device 1100 suitable for implementing embodiments of the present disclosure. The electronic device 1100 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 11 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0152] like Figure 11 As shown, electronic device 1100 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 1101, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1102 or a program loaded from storage device 1108 into random access memory (RAM) 1103. RAM 1103 also stores various programs and data required for the operation of electronic device 1100. The processing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0153] Typically, the following devices can be connected to I / O interface 1105: input devices 1106 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 1107 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1108 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1109. Communication device 1109 allows electronic device 1100 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 11 An electronic device 1100 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0154] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1109, or installed from storage device 1108, or installed from ROM 1102. When the computer program is executed by processing device 1101, it performs the functions defined in the methods of embodiments of this disclosure.

[0155] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0156] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0157] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0158] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0160] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0161] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0162] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0163] In a first aspect, according to one or more embodiments of the present disclosure, a keypoint detection method is provided, comprising: determining a target image; detecting keypoint coordinates of the target image through a first network in a keypoint detection model to obtain predicted keypoint coordinates of the target image; wherein the keypoint detection model further comprises a second network, the second network being used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network, and during the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to assist in improving the keypoint prediction accuracy of the first network.

[0164] According to one or more embodiments of this disclosure, the training process of the keypoint detection model involves multiple training iterations. One training iteration of the keypoint detection model includes: using a first network to detect keypoint coordinates in a training image, obtaining predicted keypoint coordinates and a feature map of the training image; determining multiple predicted keypoints on the training image based on the predicted keypoint coordinates, obtaining an initial predicted keypoint heatmap of the training image; using a second network to extract features from the initial predicted keypoint heatmap and the feature map of the training image, obtaining a predicted keypoint heatmap of the training image; and adjusting the parameters of the first network based on the differences between the actual keypoint coordinates and the predicted keypoint coordinates of the training image, and the differences between the actual keypoint heatmap and the predicted keypoint heatmap of the training image.

[0165] According to one or more embodiments of this disclosure, the first network includes multiple downsampling layers and coordinate regression layers, and the second network includes multiple upsampling layers. The step of determining multiple predicted keypoints on the training image based on the predicted keypoint coordinates of the training image to obtain an initial predicted keypoint heatmap of the training image includes: fusing the initial predicted keypoint heatmap of the training image with a feature map obtained by fusing the feature map of the training image output by the last downsampling layer in the first network, and inputting this feature map into the first upsampling layer in the second network for upsampling processing; fusing the initial predicted keypoint heatmap of the training image with the feature map output by the first upsampling layer, and inputting this feature map into the next upsampling layer for upsampling processing; after multiple upsampling layers, the predicted keypoint heatmap of the training image is obtained.

[0166] According to one or more embodiments of this disclosure, after extracting features from the initial predicted keypoint heatmap and the feature map of the training image through the second network to obtain the predicted keypoint heatmap of the training image, the method further includes: adjusting the parameters of the second network based on the difference between the actual keypoint heatmap of the training image and the predicted keypoint heatmap of the training image.

[0167] According to one or more embodiments of this disclosure, after detecting keypoint coordinates of the target image through a first network in the keypoint detection model to obtain the predicted keypoint coordinates of the target image, the method further includes: determining multiple predicted keypoints on the target image based on the predicted keypoint coordinates of the target image to obtain an initial predicted keypoint heatmap of the target image; and performing feature extraction on the initial predicted keypoint heatmap of the target image and the feature map extracted from the keypoint coordinate detection of the target image through a second network to obtain the predicted keypoint heatmap of the target image.

[0168] Secondly, according to one or more embodiments of this disclosure, a model training method is provided, comprising: determining training data, the training data including training images and sample labels corresponding to the training images; training a keypoint detection model based on the training data; wherein the keypoint detection model includes a first network and a second network, the first network being used to detect keypoint coordinates in the image, and the second network being used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network, wherein during the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to assist in improving the keypoint prediction accuracy of the first network.

[0169] According to one or more embodiments of this disclosure, the sample label includes the actual keypoint coordinates of the training image and the actual keypoint heatmap of the training image. The training process of the keypoint detection model is multiple training iterations. One training iteration of the keypoint detection model includes: detecting keypoint coordinates of the training image using a first network to obtain predicted keypoint coordinates and a feature map of the training image; determining multiple predicted keypoints on the training image based on the predicted keypoint coordinates to obtain an initial predicted keypoint heatmap of the training image; extracting features from the initial predicted keypoint heatmap and the feature map of the training image using a second network to obtain a predicted keypoint heatmap of the training image; and adjusting the parameters of the first network based on the differences between the actual keypoint coordinates and the predicted keypoint coordinates, and the differences between the actual keypoint heatmap and the predicted keypoint heatmap.

[0170] According to one or more embodiments of this disclosure, the first network includes a downsampling layer and a coordinate regression layer, and the second network includes multiple upsampling layers. The step of extracting features from the initial predicted keypoint heatmap and the feature map of the training image through the second network to obtain a predicted keypoint heatmap of the training image includes: fusing the initial predicted keypoint heatmap and the feature map of the training image output by the last downsampling layer in the first network to obtain a feature map, and inputting this feature map into the first upsampling layer in the second network for upsampling processing; fusing the initial predicted keypoint heatmap and the feature map output by the first upsampling layer to obtain a feature map, and inputting this feature map into the next upsampling layer for upsampling processing; after multiple upsampling layers, the predicted keypoint heatmap is obtained.

[0171] According to one or more embodiments of this disclosure, after adjusting the parameters of the first network based on the difference between the actual key point coordinates and the predicted key point coordinates and the difference between the actual key point heatmap and the predicted key point heatmap, the method further includes: adjusting the parameters of the second network based on the difference between the actual key point heatmap and the predicted key point heatmap.

[0172] Thirdly, according to one or more embodiments of this disclosure, a keypoint detection device is provided, comprising: a determining unit for determining a target image; and a detection unit for detecting keypoint coordinates of the target image through a first network in a keypoint detection model to obtain predicted keypoint coordinates of the target image; wherein the keypoint detection model further comprises a second network, the second network being used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network, and during the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to assist in improving the keypoint prediction accuracy of the first network.

[0173] Fourthly, according to one or more embodiments of this disclosure, an image training device is provided, comprising: a determining unit, configured to determine training data, the training data including training images and sample labels corresponding to the training images; and a training unit, configured to train a keypoint detection model based on the training data; wherein the keypoint detection model includes a first network and a second network, the first network being configured to detect keypoint coordinates in an image, and the second network being configured to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network, wherein during the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to assist in improving the keypoint prediction accuracy of the first network.

[0174] Fifthly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory; the memory storing computer-executable instructions; the at least one processor executing the computer-executable instructions stored in the memory, such that the at least one processor performs the key point detection method as described in the first aspect or various possible designs of the first aspect, or performs the model training method as described in the second aspect or various possible designs of the second aspect.

[0175] In a sixth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, which, when executed by a processor, implement the key point detection method as described in the first aspect and various possible designs of the first aspect, or implement the model training method as described in the second aspect or various possible designs of the second aspect.

[0176] In a seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, the computer program product comprising computer execution instructions, which, when executed by a processor, implement the key point detection method as described in the first aspect and various possible designs of the first aspect, or implement the model training method as described in the second aspect or various possible designs of the second aspect.

[0177] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0178] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0179] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A key point detection method, comprising: Identify the target image; The target image is subjected to key point coordinate detection by the first network in the key point detection model to obtain the predicted key point coordinates of the target image; wherein, the first network includes a feature extraction layer and a coordinate regression layer, and the first network is used to detect key point coordinates of the target image. The key point detection model further includes a second network, which includes a feature extraction layer. The second network is used to generate a corresponding key point heatmap based on the key point coordinates output by the first network and the feature map extracted by the first network. During the training process of the key point detection model, the key point heatmap output by the second network is used to help improve the key point prediction accuracy of the first network.

2. The keypoint detection method according to claim 1, wherein the training process of the keypoint detection model involves multiple training iterations, wherein... One training process of the key point detection model includes: The first network is used to detect key point coordinates in the training image to obtain the predicted key point coordinates and feature map of the training image. Based on the coordinates of the predicted key points in the training image, multiple predicted key points are determined on the training image to obtain an initial predicted key point heatmap of the training image. Through the second network, feature extraction is performed on the initial predicted key point heatmap of the training image and the feature map of the training image to obtain the predicted key point heatmap of the training image. The parameters of the first network are adjusted based on the differences between the actual keypoint coordinates and the predicted keypoint coordinates of the training image, as well as the differences between the actual keypoint heatmap and the predicted keypoint heatmap of the training image.

3. The keypoint detection method according to claim 2, wherein the first network includes multiple downsampling layers and coordinate regression layers, the second network includes multiple upsampling layers, and the step of determining multiple predicted keypoints on the training image based on the predicted keypoint coordinates of the training image to obtain an initial predicted keypoint heatmap of the training image includes: The feature map obtained by fusing the initial predicted key point heatmap of the training image and the feature map of the training image output by the last downsampling layer in the first network is input into the first upsampling layer in the second network for upsampling processing. The feature map obtained by fusing the initial predicted key point heatmap of the training image and the feature map output by the first upsampling layer is input into the next upsampling layer for upsampling processing. After multiple upsampling layers, the predicted key point heatmap of the training image is obtained.

4. The keypoint detection method according to claim 2 or 3, after extracting features from the initial predicted keypoint heatmap of the training image and the feature map of the training image through the second network to obtain the predicted keypoint heatmap of the training image, it further includes: The parameters of the second network are adjusted based on the difference between the actual keypoint heatmap of the training image and the predicted keypoint heatmap of the training image.

5. The keypoint detection method according to any one of claims 1 to 3, wherein after detecting keypoint coordinates of the target image through the first network in the keypoint detection model to obtain the predicted keypoint coordinates of the target image, it further includes: Based on the predicted key point coordinates of the target image, multiple predicted key points are determined on the target image to obtain an initial predicted key point heatmap of the target image. The second network is used to extract features from the initial predicted key point heatmap of the target image and the feature map extracted from the key point coordinate detection of the target image to obtain the predicted key point heatmap of the target image.

6. A model training method, comprising: Determine the training data, which includes training images and sample labels corresponding to the training images; The keypoint detection model is trained based on the training data. The keypoint detection model includes a first network and a second network. The first network includes a feature extraction layer and a coordinate regression layer. The first network is used to detect keypoint coordinates in an image. The second network includes a feature extraction layer. The second network is used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to help improve the keypoint prediction accuracy of the first network.

7. The model training method according to claim 6, wherein the sample label includes the actual keypoint coordinates of the training image and the actual keypoint heatmap of the training image, and the training process of the keypoint detection model involves multiple training iterations, wherein... One training process of the key point detection model includes: The first network is used to detect key point coordinates in the training image to obtain the predicted key point coordinates and feature map of the training image. Based on the predicted key point coordinates, multiple predicted key points are determined on the training image to obtain an initial predicted key point heatmap of the training image. Through the second network, feature extraction is performed on the initial predicted key point heatmap and the feature map of the training image to obtain the predicted key point heatmap of the training image; The parameters of the first network are adjusted based on the differences between the actual key point coordinates and the predicted key point coordinates, as well as the differences between the actual key point heatmap and the predicted key point heatmap.

8. The model training method according to claim 7, wherein the first network includes a downsampling layer and a coordinate regression layer, the second network includes multiple upsampling layers, and the step of extracting features from the initial predicted keypoint heatmap and the feature map of the training image through the second network to obtain the predicted keypoint heatmap of the training image includes: The feature map obtained by fusing the initial predicted key point heatmap and the feature map of the training image output by the last downsampling layer in the first network is input into the first upsampling layer of the second network for upsampling processing. The feature map obtained by fusing the initial predicted key point heatmap and the feature map output by the first upsampling layer is input into the next upsampling layer for upsampling processing. After multiple upsampling layers, the predicted key point heatmap is obtained.

9. The model training method according to claim 7 or 8, after adjusting the parameters of the first network based on the difference between the actual keypoint coordinates and the predicted keypoint coordinates and the difference between the actual keypoint heatmap and the predicted keypoint heatmap, further includes: The parameters of the second network are adjusted based on the difference between the actual key point heatmap and the predicted key point heatmap.

10. A key point detection device, comprising: The determining unit is used to determine the target image; The detection unit is used to detect key point coordinates of the target image through the first network in the key point detection model to obtain the predicted key point coordinates of the target image; wherein, the first network includes a feature extraction layer and a coordinate regression layer, and the first network is used to detect key point coordinates of the target image; The key point detection model further includes a second network, which includes a feature extraction layer. The second network is used to generate a corresponding key point heatmap based on the key point coordinates output by the first network and the feature map extracted by the first network. During the training process of the key point detection model, the key point heatmap output by the second network is used to help improve the key point prediction accuracy of the first network.

11. A model training device, comprising: A determining unit is used to determine training data, wherein the training data includes training images and sample labels corresponding to the training images; The training unit is used to train the key point detection model based on the training data. The keypoint detection model includes a first network and a second network. The first network includes a feature extraction layer and a coordinate regression layer. The first network is used to detect keypoint coordinates in an image. The second network includes a feature extraction layer. The second network is used to generate a corresponding keypoint heatmap based on the keypoint coordinates output by the first network and the feature map extracted by the first network. During the training process of the keypoint detection model, the keypoint heatmap output by the second network is used to help improve the keypoint prediction accuracy of the first network.

12. An electronic device, comprising: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the key point detection method as described in any one of claims 1 to 5 or the model training method as described in any one of claims 6 to 9.

13. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the key point detection method as described in any one of claims 1 to 5 or the model training method as described in any one of claims 6 to 9.

14. A computer program product comprising computer-executable instructions, which, when executed by a processor, implement the keypoint detection method as described in any one of claims 1 to 5 or the model training method as described in any one of claims 6 to 9.

Images