Unsupervised target key point detection method based on comparative learning

Abstract

The invention provides an unsupervised target key point detection method based on comparative learning, which belongs to the field of computer vision and comprises a key point prediction module based on image reconstruction and a key point semantic constraint module based on a comparative twin network. The method specifically comprises the following steps: acquiring a sample data set, wherein each image contains a to-be-detected target object; performing data enhancement on the sample data set to obtain an image pair; inputting the image pair into a pre-trained key point prediction module based on image reconstruction to obtain a preliminary key point detection result of the input image; and then semantic constraints are added to the preliminarily detected target key points by using a key point semantic constraint module based on a comparative twin network, and the semantic constraints are used for improving the accuracy and robustness of detection. The semantic constraint can help the model to understand the relationship between different key points and enhance the distinguishing ability of the features, so that the key points can be positioned more accurately in a complex scene.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular relates to an unsupervised target key point detection method based on contrastive learning. Background Art

[0002] Target key point detection is a basic task in computer vision and is widely used in many scenarios such as action recognition, 3D reconstruction, and facial recognition. Traditional key point detection methods mostly rely on a large amount of manually annotated data for supervised learning. However, such methods have problems such as high data acquisition cost and uneven annotation quality. Therefore, exploring an efficient unsupervised key point detection method is of great significance to reduce costs and improve the scalability and applicability of the system.

[0003] There are several limitations to current unsupervised keypoint detection methods. First, this method only uses reconstruction loss once during training, which causes the model to focus too much on image reconstruction rather than keypoint detection as the optimization process progresses. This causes the image encoder to focus mainly on the overall features of the reconstruction and ignore the specific features of the keypoints, resulting in a lack of semantic connections between keypoints. In addition, although most existing methods use convolutional neural networks to predict keypoints based on image pose, these methods are usually unable to effectively establish long-distance connections between keypoints, which is critical for accurately predicting the relative positions of keypoints. Summary of the invention

[0004] The present invention proposes an unsupervised target key point detection method based on contrastive learning. This method uses the self-attention mechanism and contrastive learning strategy in deep learning to effectively solve the difficulties in traditional unsupervised key point detection, that is, automatically identifying and optimizing the detection of target key points from images without manual annotation; and establishing long-distance connections between key points and focusing on the semantic consistency between key points. This not only reduces the dependence on labeled data, but also improves the accuracy of key point detection and the adaptability and robustness of the model to new scenarios, opening up new possibilities for the development of key point detection technology.

[0005] An unsupervised target key point detection method based on contrastive learning, the method comprising the following steps:

[0006] S1. Obtain a sample data set, which includes the image to be detected and all or a small amount of annotation files, where the annotation files contain the key point location information of the target in each image;

[0007] S2. Perform data augmentation on all images in the dataset to obtain image pairs containing augmented images, where the two images in the image pairs have the same appearance but different poses;

[0008] S3. Construct a key point detection model based on contrastive learning, which consists of a key point prediction module based on image reconstruction and a key point semantic constraint module based on contrastive twin network;

[0009] S4. Using the prepared image pair containing the target to be detected as the model input, the input image pair includes the original image and the enhanced image, and using the constructed model to detect the target key points of the input image;

[0010] S5. Use the key point prediction module based on image reconstruction to make preliminary predictions of target key points.

[0011] S6. Use the key point semantic constraint module based on the contrastive twin network to semantically constrain the initially predicted key points;

[0012] S7. Use the constructed training sample data set to train the comparative learning key point detection model according to the above steps. During the training process, when the number of model optimizations reaches the preset prediction value, stop the training and save the trained weights; then, load the trained weights into the model to obtain the target key point positions for the final detection.

[0013] Advantages and positive effects of the present invention:

[0014] (1) The present invention can accurately identify and locate the key points of the target without the support of a large amount of manually annotated data. This method significantly improves the accuracy and reliability of key point detection through in-depth analysis of image features and strengthening of semantic constraints. The present invention adopts an unsupervised learning method, which reduces the dependence on annotated data, making the model more universal and flexible in application.

[0015] (2) Enhance the generalization ability of the model: By combining contrastive learning with the self-attention mechanism, the method of the present invention can effectively handle the posture changes and appearance variations of images, thereby enhancing the generalization ability of the model in different environments and for different targets. This enables the model to adapt to a wider range of application scenarios, such as dynamic video surveillance, diversified face recognition, etc.

[0016] (3) The model structure of the present invention optimizes the computational process, especially in the process of feature extraction and semantic constraint using the global self-attention mechanism Transformer and the contrastive twin network, effectively reducing unnecessary computational overhead. This design enables the model to run efficiently even on devices with limited computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of an unsupervised target key point detection method based on contrastive learning provided in an embodiment of the present invention.

[0018] Figure 2 A schematic diagram of key point semantic constraints based on a contrastive twin network provided in an embodiment of the present invention.

[0019] Figure 3 A schematic diagram of a comparative twin network structure provided for an embodiment of the present invention.

[0020] Figure 4 A schematic diagram of the structure of a Transformer-based target geometric feature encoder provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above-mentioned purpose, the present invention adopts the following technical scheme.

[0022] The present invention proposes an unsupervised target key point detection method based on contrastive learning, comprising the following steps:

[0023] S1. Obtain a sample data set, which includes the image to be detected and all or a small amount of annotation files, where the annotation files contain the key point location information of the target in each image;

[0024] S2. Perform data augmentation on all images in the sample dataset to obtain image pairs containing augmented images, where the two images in the image pair have the same appearance but different poses;

[0025] S3. Construct a key point detection model based on contrastive learning, which consists of a key point prediction module based on image reconstruction and a key point semantic constraint module based on contrastive twin network;

[0026] S4. Using the prepared image pair containing the target to be detected as the model input, the input image pair includes the original image and the enhanced image, and using the constructed model to detect the target key points of the input image;

[0027] S5. First, the key point prediction module based on image reconstruction is used to make a preliminary prediction of the target key points. The process is as follows: Figure 1 As shown, specifically:

[0028] S51. In this embodiment, Unet networks of different structures are used as appearance feature encoders and geometric feature encoders of the image, and SPADE networks are used as image decoders;

[0029] S52. The original image is used as input and the image geometric feature encoder outputs a target key point heat map representing the target posture feature;

[0030] S53. The enhanced image is taken as input and the appearance feature map of the target is outputted through the appearance feature encoder of the image;

[0031] S54. The extracted appearance feature map of the target and the key point heat map of the target are used as input and passed through an image decoder to generate a target image, wherein the target image represents a reconstruction of the original image;

[0032] S55. By calculating the similarity between the target image reconstructed by the image decoder and the input original image and minimizing the mean square error between the two, a qualified geometric feature encoder is trained for preliminary prediction of target key points;

[0033] S6. Then, the key point semantic constraint module based on the contrastive twin network is used to semantically constrain the key points of the initial prediction, such as Figure 2 As shown, specifically:

[0034] S61. An image pair including an original image and an enhanced image is input into the contrastive twin network to obtain the semantic feature vector of the key points of the original image and the semantic feature vector of the enhanced image respectively. Then, the negative cosine similarity between the two types of feature vectors is calculated to achieve semantic matching of the feature vectors to optimize the similarity of the two output feature vectors.

[0035] S62. Compare the twin network structures such as Figure 3 As shown in Figure 2, the network consists of two twin branches with different structures. The first branch consists of a geometric feature encoder and two multi-layer perceptrons, denoted as ; The second branch consists of a geometric feature encoder and a multi-layer perceptron, denoted as .

[0036] S63. Let the input image pair be and , then the negative cosine similarity loss of the two types of semantic feature vectors is It can be expressed as:

[0037] ,

[0038] S64. The structure of the image geometric feature encoder is as follows Figure 4As shown in the figure. First, the input image is downsampled by convolution to generate multiple two-dimensional feature maps. Then, the two-dimensional feature map is pooled in the row and column directions to obtain a one-dimensional feature vector. Then, these feature vectors are concatenated into a two-dimensional feature map, and the global self-attention map is calculated through Transformer. Finally, the row and column attention map is element-wise multiplied with the two-dimensional feature map, and the final target geometric features are obtained by upsampling through convolution.

[0039] S7. Use the constructed training data set to train the model according to the above steps. When the number of model optimizations reaches the predicted value, stop training and save the trained weights. Load the trained weights into the model to obtain the target key points for the final detection.

[0040] The target object described in step S1 is any target such as a human face, a human body, a cat head, or an airplane; the training data set uses a public data set as a training set and a test set, including CelebA, Cat Heads, and BBCpose.

[0041] The image enhancement method in step S2 is TPS transformation or affine transformation.

[0042] In step S51, Unet is selected as the appearance feature encoder of the image or other deep neural networks are selected as the appearance feature encoder.

[0043] In step S52, the Unet convolutional neural network combined with the Transformer is selected as the geometric feature encoder of the image. The Unet convolutional neural network is used to extract the local features of the image, and the global self-attention mechanism is used to extract the overall features of the image.

[0044] In step S53, a SPADE network is selected as an image decoder, or other deep neural networks are selected as image decoders.

[0045] The mean square error described in step S54 is to calculate the difference between the pixel values ​​of the original image and the target image.

[0046] The multilayer perceptron in step S61 consists of a fully connected layer, a batch normalization layer and a nonlinear mapping layer.

[0047] The gradient back propagation operation is stopped for the second branch in step S61 to prevent the model from crashing.

[0048] In step S63, semantic constraints are added to the key points detected initially by minimizing the cosine similarity of the two types of semantic feature vectors.

[0049] In step S7, the model is iteratively optimized 150 times.

[0050] In summary, the present invention effectively optimizes the target key point detection technology under unsupervised conditions by using image pairs with the same appearance but different postures as input and constructing a contrast learning model, significantly improving the generalization ability of the algorithm and optimizing the calculation process.

[0051] The above description is only a specific implementation of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes.

Claims

1. An unsupervised target key point detection method based on contrastive learning, characterized in that: The method comprises the following steps: S1. Obtain a sample data set, which includes the image to be detected and all or a small amount of annotation files, where the annotation files contain the key point location information of the target in each image; S2. Perform data augmentation on all images in the dataset to obtain image pairs containing augmented images, where the two images in the image pairs have the same appearance but different poses; S3. Construct a key point detection model based on contrastive learning, which consists of a key point prediction module based on image reconstruction and a key point semantic constraint module based on contrastive twin network; S4. Using the prepared image pair containing the target to be detected as the model input, the input image pair includes the original image and the enhanced image, and using the constructed model to detect the target key points of the input image; S5. Use the key point prediction module based on image reconstruction to make preliminary predictions of target key points. S6. Use the key point semantic constraint module based on the contrastive twin network to semantically constrain the initially predicted key points; S7. Use the constructed training sample data set to train the comparative learning key point detection model according to the above steps. During the training process, when the number of model optimizations reaches the preset prediction value, stop the training and save the trained weights; then, load the trained weights into the model to obtain the target key point positions for the final detection.

2. The unsupervised target key point detection method based on contrastive learning as claimed in claim 1, characterized in that: S5 is as follows: S51. The original image is used as input and the image geometric feature encoder outputs a target key point heat map representing the target posture feature; S52. The enhanced image is taken as input and the appearance feature map of the target is output through the image appearance feature encoder; S53. The extracted target appearance feature map and the target key point heat map are used as input and passed through an image decoder to generate a target image, wherein the target image represents a reconstruction of the original image; S54. Calculate the similarity between the target image reconstructed by the image decoder and the input original image, and minimize the mean square error between the two; the trained image geometric feature encoder is used for preliminary prediction of the target key points.

3. The unsupervised target key point detection method based on contrastive learning as claimed in claim 2, characterized in that: S6 is specifically: S61. Construct a key point semantic constraint contrast twin network, which consists of two branches. The first branch consists of a geometric feature encoder and two multi-layer perceptrons, denoted as , the second branch consists of a geometric feature encoder and a multi-layer perceptron, denoted as ; Let the input image pair be and , then the negative cosine similarity loss of the two types of semantic feature vectors is It is expressed as: ； S62. Input the image pair into the key point semantic constraint contrast twin network, thereby obtaining two types of semantic feature vectors in two branches respectively; S63. Semantic matching of feature vectors is achieved by calculating and minimizing the negative cosine similarity between two types of feature vectors to optimize the similarity of two output feature vectors.

4. The unsupervised target key point detection method based on contrastive learning as claimed in claim 3, characterized in that: The geometric feature encoder structure in step 52 and step 61 is composed of convolution, up-down sampling and superposition of Transformer. In addition, the feature map is pooled in the row and column directions to reduce the amount of model calculation.

5. The unsupervised target key point detection method based on contrastive learning as claimed in claim 1, characterized in that: The target object described in step S1 is any target such as a human face, a human body, a cat head, or an airplane; the training data set uses a public data set as a training set and a test set, including CelebA, Cat Heads, and BBCpose.

6. The unsupervised target key point detection method based on contrastive learning as claimed in claim 1, characterized in that: The image enhancement method in step S2 is TPS transformation or affine transformation.

7. The unsupervised target key point detection method based on contrastive learning as claimed in claim 2, characterized in that: In step S51, Unet is selected as the appearance feature encoder of the image or other deep neural networks are selected as the appearance feature encoder.

8. The unsupervised target key point detection method based on contrastive learning as claimed in claim 2, characterized in that: In step S52, the Unet convolutional neural network combined with the Transformer is selected as the geometric feature encoder of the image. The Unet convolutional neural network is used to extract the local features of the image, and the global self-attention mechanism is used to extract the overall features of the image.

9. The unsupervised target key point detection method based on contrastive learning as claimed in claim 2, characterized in that: In step S53, a SPADE network is selected as an image decoder, or other deep neural networks are selected as image decoders.

10. The unsupervised target key point detection method based on contrastive learning as claimed in claim 2, characterized in that: The mean square error described in step S54 is to calculate the difference between the pixel values ​​of the original image and the target image.

11. The unsupervised target key point detection method based on contrastive learning as claimed in claim 3, characterized in that: The multilayer perceptron in step S61 is composed of a fully connected layer, a batch normalization layer and a nonlinear mapping layer.

12. The unsupervised target key point detection method based on contrastive learning as claimed in claim 3, characterized in that: The gradient back propagation operation is stopped for the second branch in step S61 to prevent the model from crashing.

13. The unsupervised target key point detection method based on contrastive learning as claimed in claim 3, characterized in that: In step S63, semantic constraints are added to the key points detected initially by minimizing the cosine similarity of the two types of semantic feature vectors.

14. The unsupervised target key point detection method based on contrastive learning as claimed in claim 3, characterized in that: In step S7, the model is iteratively optimized 150 times.

Images