An unsupervised target key point detection method based on contrastive learning

Abstract

The application provides an unsupervised target key point detection method based on contrast learning, and belongs to the field of computer vision, and comprises two modules of key point prediction based on image reconstruction and key point semantic constraint based on a contrast twin network. Specifically, sample data sets are obtained, each image in the sample data sets containing a target object to be detected; the sample data sets are subjected to data enhancement to obtain image pairs; the image pairs are input into a pre-trained key point prediction module based on image reconstruction to obtain preliminary key point detection results of the input images; then, a key point semantic constraint module based on a contrast twin network is used to add semantic constraints to the preliminary detected target key points, so that the accuracy and robustness of detection are improved. The semantic constraints can help the model to understand the relationship between different key points, enhance the distinguishing ability of the features, and thus more accurately locate the key points in a complex scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to an unsupervised target key point detection method based on contrastive learning. Background Technology

[0002] Keypoint detection is a fundamental task in computer vision, widely applied in various scenarios such as action recognition, 3D reconstruction, and facial recognition. Traditional keypoint detection methods largely rely on supervised learning using large amounts of manually labeled data. However, these methods suffer from high data acquisition costs and inconsistent labeling quality. Therefore, exploring an efficient unsupervised keypoint detection method is of great significance for reducing costs and improving the scalability and applicability of the system.

[0003] Current unsupervised keypoint detection methods have some limitations. First, these methods utilize reconstruction loss only once during training, causing the model to overemphasize image reconstruction rather than keypoint detection during optimization. This leads the image encoder to focus primarily on the overall features of the reconstruction, neglecting the specific features of keypoints and thus lacking semantic connections between them. Furthermore, while most existing methods employ convolutional neural networks to predict image pose-based keypoints, these methods often fail to effectively establish long-range connections between keypoints, which is crucial for accurately predicting their relative positions. Summary of the Invention

[0004] This invention proposes an unsupervised target keypoint detection method based on contrastive learning. This method effectively solves the challenges of traditional unsupervised keypoint detection by utilizing the self-attention mechanism and contrastive learning strategy in deep learning. Specifically, it automatically identifies and optimizes the detection of target keypoints from images without any manual annotation; furthermore, it establishes long-distance connections between keypoints and focuses on their semantic consistency. This not only reduces the dependence on labeled data but also improves the accuracy of keypoint detection and the model's adaptability and robustness to new scenes, opening up new possibilities for the development of keypoint detection technology.

[0005] An unsupervised target keypoint detection method based on contrastive learning, comprising the following steps:

[0006] S1. Obtain the sample dataset, which includes the images to be detected and all or a small number of annotation files, in which the annotation files contain the key point location information of the target in each image;

[0007] S2. Perform data augmentation on all images in the dataset to obtain image pairs containing augmented images, where the two images in the image pair have the same appearance but different poses;

[0008] S3. Construct a keypoint detection model based on contrastive learning, which consists of a keypoint prediction module based on image reconstruction and a keypoint semantic constraint module based on contrastive Siamese network;

[0009] S4. Use the prepared image pair containing the target to be detected as the model input. The input image pair contains the original image and the enhanced image. Use the constructed model to detect the target key points in the input image.

[0010] S5. Use the keypoint prediction module based on image reconstruction to make preliminary predictions of the target keypoints.

[0011] S6. Use the keypoint semantic constraint module based on contrastive Siamese network to perform semantic constraints on the initially predicted keypoints;

[0012] S7. Using the completed training sample dataset, train the contrastive learning keypoint detection model according to the steps above. During the training process, when the model reaches the preset prediction value after optimization, stop training and save the trained weights. Then, load the trained weights into the model to obtain the final detected target keypoint locations.

[0013] Advantages and positive effects of the present invention:

[0014] (1) This invention can accurately identify and locate key points of a target without the need for a large amount of manually labeled data. This method significantly improves the accuracy and reliability of key point detection through in-depth analysis of image features and enhancement of semantic constraints. This invention adopts an unsupervised learning method, which reduces the dependence on labeled data and makes the model more universal and flexible in application.

[0015] (2) Enhanced generalization ability of the model: By combining contrastive learning and self-attention mechanism, the method of the present invention can effectively handle pose changes and appearance variations of images, thereby enhancing the model's generalization ability in different environments and for different targets. This enables the model to adapt to a wider range of application scenarios, such as dynamic video surveillance and diverse face recognition.

[0016] (3) The model structure of this invention optimizes the computation process, especially in the process of feature extraction and semantic constraints using the global self-attention mechanism Transformer and the contrastive Siamese network, effectively reducing unnecessary computational overhead. This design enables the model to run efficiently even on devices with limited computing resources. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an unsupervised target key point detection method based on contrastive learning provided in an embodiment of the present invention.

[0018] Figure 2 This is a schematic diagram of key point semantic constraints based on a contrastive Siamese network, provided for an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of a comparative twin network structure provided in an embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of a target geometric feature encoder structure based on Transformer provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above objectives, this invention adopts the following technical solution.

[0022] This invention proposes an unsupervised target keypoint detection method based on contrastive learning, comprising the following steps:

[0023] S1. Obtain the sample dataset, which includes the images to be detected and all or a small number of annotation files, in which the annotation files contain the key point location information of the target in each image;

[0024] S2. Perform data augmentation on all images in the sample dataset to obtain image pairs containing augmented images, where the two images in the image pair have the same appearance but different poses;

[0025] S3. Construct a keypoint detection model based on contrastive learning, which consists of a keypoint prediction module based on image reconstruction and a keypoint semantic constraint module based on contrastive Siamese network;

[0026] S4. Use the prepared image pair containing the target to be detected as the model input. The input image pair contains the original image and the enhanced image. Use the constructed model to detect the target key points in the input image.

[0027] S5. First, the key point prediction module based on image reconstruction is used to make preliminary predictions of the target key points, as follows: Figure 1 As shown, specifically:

[0028] S51. In this embodiment, Unet networks with different structures are used as appearance feature encoders and geometric feature encoders for images, and SPADE networks are used as image decoders;

[0029] S52. The original image is used as input and passed through an image geometric feature encoder to output a heatmap of target key points representing the target pose features;

[0030] S53. The enhanced image is taken as input and processed by the image appearance feature encoder to output the appearance feature map of the target;

[0031] S54. The extracted appearance feature map and key point heatmap of the target are used as input and processed by an image decoder to generate a target image, wherein the target image represents the reconstruction of the original image;

[0032] S55. By calculating the similarity between the target image reconstructed by the image decoder and the original input image, and minimizing the mean square error between the two, a qualified geometric feature encoder is trained for the preliminary prediction of target key points.

[0033] S6. Then, the semantic constraint module for key points based on contrastive Siamese networks is used to perform semantic constraints on the initially predicted key points, such as... Figure 2 As shown, specifically:

[0034] S61. Input the image pair containing the original image and the enhanced image into the contrastive Siamese network to obtain the semantic feature vectors of the key points of the original image and the semantic feature vectors of the enhanced image, respectively. Then, calculate the negative cosine similarity between the two types of feature vectors to achieve semantic matching of the feature vectors, so as to optimize the similarity of the two output feature vectors.

[0035] S62. Compare the twin network structure as follows: Figure 3 As shown, the network consists of two twin branches with different structures. The first branch comprises a geometric feature encoder and two multilayer perceptrons, denoted as... The second branch consists of a geometric feature encoder and a multilayer perceptron, denoted as... .

[0036] S63. Let the input image pair be and Then the negative cosine similarity loss of the two types of semantic feature vectors It can be represented as:

[0037] ,

[0038] S64. The structure of the image geometric feature encoder is as follows: Figure 4As shown, the input image is first downsampled via convolution to generate multiple two-dimensional feature maps. Next, row and column pooling operations are performed on the two-dimensional feature maps to obtain one-dimensional feature vectors. Then, these feature vectors are concatenated into a single two-dimensional feature map, and a global self-attention map is calculated using a Transformer. Finally, the row and column attention map is element-wise multiplied with the two-dimensional feature map, and upsampled via convolution to obtain the final target geometric features.

[0039] S7. Using the completed training dataset, train the model according to the steps above. Stop training when the model reaches the prediction value after the number of optimizations, and save the trained weights. Load the trained weights into the model to obtain the final detected target key points.

[0040] The target object mentioned in step S1 can be any target such as a human face, human body, cat head, or airplane; the training dataset uses public datasets as the training and testing sets, including CelebA, Cat Heads, and BBCpose.

[0041] The image enhancement method in step S2 is either TPS transformation or affine transformation.

[0042] In step S51, Unet is selected as the appearance feature encoder for the image, or other deep neural networks are selected as the appearance feature encoder.

[0043] In step S52, the Unet convolutional neural network combined with the Transformer is selected as the geometric feature encoder of the image. The Unet convolutional neural network is used to extract the local features of the image, while the global self-attention mechanism is used to extract the overall features of the image.

[0044] In step S53, the SPADE network is selected as the image decoder, or other deep neural networks are selected as the image decoder.

[0045] The mean square error mentioned in step S54 is the difference between pixel-by-pixel values ​​between the original image and the target image.

[0046] The multilayer perceptron in step S61 consists of a fully connected layer, a batch normalization layer, and a nonlinear mapping layer.

[0047] Perform a stop gradient backpropagation operation on the second branch in step S61 to prevent the model from crashing.

[0048] In step S63, semantic constraints are added to the key points of the preliminary detection by minimizing the cosine similarity between the two types of semantic feature vectors.

[0049] In step S7, the model is iteratively optimized 150 times.

[0050] In summary, this invention effectively optimizes unsupervised target keypoint detection technology by using image pairs with identical appearances but different poses as input and constructing a contrastive learning model. It significantly improves the algorithm's generalization ability and optimizes the computational process.

[0051] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this specification may be replaced by other equivalent or similar alternative features unless otherwise specified.

Claims

1. An unsupervised target key point detection method based on contrastive learning, characterized in that, The method includes the following steps: S1. Obtain the sample dataset, which includes the images to be detected and a small number of annotation files, which contain the key point location information of the target in each image; S2. Perform data augmentation on all images in the dataset to obtain image pairs containing augmented images, where the two images in the image pair have the same appearance but different poses; S3. Construct a keypoint detection model based on contrastive learning, which consists of a keypoint prediction module based on image reconstruction and a keypoint semantic constraint module based on contrastive Siamese network; S4. Use the prepared image pair containing the target to be detected as the model input. The input image pair contains the original image and the enhanced image. Use the constructed model to detect the target key points in the input image. S5. Use the keypoint prediction module based on image reconstruction to make preliminary predictions of the target keypoints. S6. Use the keypoint semantic constraint module based on contrastive Siamese network to perform semantic constraints on the initially predicted keypoints; S7. Using the completed training sample dataset, train the contrastive learning keypoint detection model according to steps S1-S6 above. During the training process, when the model reaches the preset prediction value after optimization, stop training and save the trained weights. Then, load the trained weights into the model to obtain the final detected target keypoint location. S6 specifically refers to: S61. Construct a keypoint semantic constraint contrastive Siamese network. This network consists of two branches, where the first branch comprises a geometric feature encoder and two multilayer perceptrons, denoted as . The second branch consists of a geometric feature encoder and a multilayer perceptron, denoted as... ; Let the input image pair be and Then the negative cosine similarity loss of the two types of semantic feature vectors Represented as: ； S62. Input the image pairs into the keypoint semantic constraint comparison Siamese network to obtain two types of semantic feature vectors in the two branches respectively; S63. By calculating and minimizing the negative cosine similarity between the two types of feature vectors, semantic matching of feature vectors is achieved to optimize the similarity between the two output feature vectors.

2. The unsupervised target key point detection method based on contrastive learning as described in claim 1, characterized in that, S5 specifically refers to: S51. The original image is used as input and passed through an image geometric feature encoder to output a heatmap of target key points representing the target pose features; S52. The enhanced image is taken as input and passed through an image appearance feature encoder to output the appearance feature map of the target; S53. The extracted target appearance feature map and target key point heatmap are used as input and processed by an image decoder to generate a target image, where the target image represents the reconstruction of the original image; S54. Calculate the similarity between the target image reconstructed by the image decoder and the original input image, and minimize the mean square error between them; train the image geometric feature encoder for preliminary prediction of target key points.

3. The unsupervised target key point detection method based on contrastive learning as described in claim 2, characterized in that, The geometric feature encoder structure in steps 52 and 61 consists of a convolutional upsampling and downsampling Transformer. In addition, pooling in the row and column directions of the feature map is used to reduce the computational cost of the model.

4. The unsupervised target key point detection method based on contrastive learning as described in claim 1, characterized in that, The targets mentioned in step S1 are a human face, a human body, a cat's head, and an airplane; The training datasets used are publicly available datasets as both training and testing sets, including CelebA, Cat Heads, and BBCpose.

5. The unsupervised target key point detection method based on contrastive learning as described in claim 1, characterized in that, The image enhancement method in step S2 is either TPS transformation or affine transformation.

6. The unsupervised target key point detection method based on contrastive learning as described in claim 2, characterized in that, In step S51, Unet is selected as the appearance feature encoder for the image.

7. The unsupervised target key point detection method based on contrastive learning as described in claim 2, characterized in that, In step S52, the Unet convolutional neural network combined with the Transformer is selected as the geometric feature encoder of the image. The Unet convolutional neural network is used to extract the local features of the image, while the global self-attention mechanism is used to extract the overall features of the image.

8. The unsupervised target key point detection method based on contrastive learning as described in claim 2, characterized in that, In step S53, the SPADE network is selected as the image decoder.

9. The unsupervised target key point detection method based on contrastive learning as described in claim 2, characterized in that, The mean square error mentioned in step S54 is the difference between pixel-by-pixel values ​​between the original image and the target image.

10. The unsupervised target key point detection method based on contrastive learning as described in claim 3, characterized in that, The multilayer perceptron in step S61 consists of a fully connected layer, a batch normalization layer, and a nonlinear mapping layer.

11. The unsupervised target key point detection method based on contrastive learning as described in claim 3, characterized in that, Perform a stop gradient backpropagation operation on the second branch in step S61 to prevent the model from crashing.

12. The unsupervised target key point detection method based on contrastive learning as described in claim 3, characterized in that, In step S63, semantic constraints are added to the key points of the preliminary detection by minimizing the cosine similarity between the two types of semantic feature vectors.

13. The unsupervised target key point detection method based on contrastive learning as described in claim 3, characterized in that, In step S7, the model is iteratively optimized 150 times.

Images