A vehicle key point detection method and system based on contrast learning
By using a contrastive learning approach, the keypoint detection network is optimized using the simCLR framework and a heatmap+softargmax regression model. This addresses the issues of strong data dependency and low detection accuracy in existing technologies, achieving stable and efficient detection across different data distributions.
Patent Information
- Application Number
- CN202211434100.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-16
- Publication Date
- 2025-12-09
- Estimated Expiration
- 2042-11-16
AI Technical Summary
Existing vehicle key point detection technologies suffer from high dependence on data volume, low detection accuracy, and poor transferability. In particular, heatmap-based methods suffer from severe key point jitter and high network computational cost.
We employ a contrastive learning approach, using the simCLR framework to adjust the pre-trained network model. We combine it with a heatmap+softargmax regression keypoint detection model, utilizing feature extraction and projection operations to construct a triplet loss function for optimization, thereby improving the stability and accuracy of keypoint detection.
It enhances the transferability of the key point detection network, improves the detection performance on different data distributions, reduces the dependence on data volume, and improves the detection accuracy and robustness of vehicle key points.
Smart Images
Figure CN116051860B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of visual detection, in particular to a vehicle key point detection method and system based on contrast learning. BACKGROUND
[0002] With the rapid development of deep learning, the intelligent transportation system has also developed rapidly, and the vehicle key point detection technology plays a fundamental and important role in the intelligent transportation system.
[0003] The existing key point detection technology is mainly divided into top-down and bottom-up two ways, usually the key point accuracy of top-down is much higher than that of bottom-up, top-down usually uses target detection technology to detect the target first, and then detects the key points through the key point detection network, while the key point detection network has the problems of large demand for training data and poor migration; and the mainstream method based on heatmap has the problems of serious key point jitter and large network calculation amount. SUMMARY
[0004] The embodiment of the present application provides a vehicle key point detection method and system based on contrast learning, to solve the problem of low detection accuracy of the traditional key point detection technology depending on the amount of data.
[0005] According to the first aspect of the embodiment of the present application, a vehicle key point detection method based on contrast learning is provided, comprising the following steps:
[0006] S10, obtaining a training sample image set;
[0007] S20, establishing a key point detection model based on heatmap+softargmax regression and training to obtain a pre-training network model;
[0008] S30, constructing a simCLR framework based on the pre-training network model, adjusting the pre-training network model through the simCLR framework to obtain a final network model;
[0009] S40, inputting a picture to be detected into the final network model to obtain vehicle key point information.
[0010] Preferably, the step S30 of constructing the simCLR framework based on the pre-training network model, adjusting the pre-training network model through the simCLR framework to obtain the final network model, comprises:
[0011] S301, performing twice random data augmentation on the input picture to obtain two pictures with different perspectives;
[0012] S302, inputting the two pictures with different perspectives into an encoder with a feature extraction function to perform feature extraction, and extracting two image features;
[0013] S303, performing a projection operation on the two image features to obtain a projection vector;
[0014] S304, calculating the similarity between the two projection vectors;
[0015] S305, constructing a triplet object;
[0016] S306, constructing a triplet loss function based on the triplet object, and iteratively optimizing the simCLR model through the triplet loss function to obtain a final network model.
[0017] Preferably, in the step S302, the encoder with the feature extraction function is a key point encoder.
[0018] Preferably, the key point encoder is trained in a supervised manner.
[0019] Preferably, the training process of the key point encoder comprises:
[0020] S001, detecting key points of a sample image by using a heatmap model to obtain a key point heat map of the sample image, and training a key point detection model to obtain network weights of a pre-training model; S002, normalizing the key point heat map of the sample image by using a normalization function to obtain a probability distribution value;
[0021] S003, solving a coordinate expectation of a position where a maximum value in the probability distribution value is located;
[0022] S004, constructing a mean square error loss function based on the coordinate expectation and a true value;
[0023] S005, optimizing and training network weights of the key point encoder based on the key point heat map of the sample image and the mean square error loss function to obtain a trained key point encoder model.
[0024] Preferably, in the step S001, an expression of the key point heat map of the sample image is: B K H W;
[0025] wherein B is a batch size, K is a key point number, H W is a size of a feature map.
[0026] According to a second aspect of the embodiments of the present application, a vehicle key point detection system based on contrast learning is provided, comprising:
[0027] an acquisition module configured to acquire a training sample image set;
[0028] a pre-training module configured to establish a key point detection model based on heatmap+softargmax regression and perform training to obtain a pre-trained network model;
[0029] a training module configured to construct a simCLR framework based on the pre-trained network model, adjust the pre-trained network model through the simCLR framework, and obtain a final network model;
[0030] a detection module configured to input a to-be-detected picture into the final network model to obtain vehicle key point information.
[0031] Preferably, the training module comprises:
[0032] a data enhancement module configured to perform twice random data enhancement on an input picture to obtain two pictures of different perspectives;
[0033] a feature extraction module configured to input the two pictures of different perspectives into an encoder with a feature extraction function to extract two image features;
[0034] a projection module configured to perform a projection operation on the two image features to obtain two projection vectors;
[0035] a calculation module configured to calculate the similarity between the two projection vectors;
[0036] an optimization module configured to construct a triplet object, construct a triplet loss function based on the triplet object, and iteratively optimize the simCLR model through the triplet loss function to obtain the final network model.
[0037] Preferably, the encoder with the feature extraction function in the feature extraction module is a key point encoder.
[0038] Preferably, the key point encoder is trained in a supervised manner, and the training process comprises:
[0039] a key point detection module configured to perform key point detection on a sample image by using a heatmap model to obtain a key point heat map of the sample image, and train the key point detection model to obtain network weights of the pre-trained model;
[0040] a normalization processing module configured to perform normalization processing on the key point heat map of the sample image by using a normalization function to obtain a probability distribution value;
[0041] a solving module configured to solve a coordinate expectation of a position where a maximum value in the probability distribution value is located;
[0042] The optimization training module is configured to construct a mean square error loss function based on a coordinate expectation and a true value, and to perform optimization training on network weights of the key point encoder based on a key point heat map of a sample image and the mean square error loss function, to obtain a trained key point encoder.
[0043] The embodiments of the present application have the following technical effects due to the above technical solutions:
[0044] The technical solutions provided by the present application can enhance the migration ability of the key point detection network in a contrast learning manner, so that the key point detection network can also obtain good results on data with different distributions, and improve the stability of key points and the robustness to weak computer graphics transformation.
[0045] In addition, the use of the heatmap model to constrain vehicle features can also improve the accuracy of vehicle key points and reduce the dependence of key point detection technology on data volume. DETAILED DESCRIPTION
[0046] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:
[0047] Figure 1 A flowchart of a vehicle key point detection method based on contrast learning provided by the embodiments of the present application;
[0048] Figure 2 A flowchart of a process of generating a pre-training network model by training a simCLR model in an unsupervised manner;
[0049] Figure 3 A flowchart of a process of generating a pre-training network model by training a simCLR model in an unsupervised manner in a specific embodiment;
[0050] Figure 4 A flowchart of a training process of a key point encoder;
[0051] Figure 5 A structural diagram of a vehicle key point detection system based on contrast learning provided by the embodiments of the present application;
[0052] Figure 6 A structural diagram of a training module;
[0053] Figure 7 A structural diagram of a key point encoder training process;
[0054] REFERENCE NUMERALS:
[0055] 10 is an acquisition module, 20 is a pre-training module, 30 is a training module, and 40 is a detection module;
[0056] 301 is a data enhancement module, 302 is a feature extraction module, 303 is a projection module, 304 is a calculation module, and 305 is an optimization module.
[0057] 001 is a key point detection module, 002 is a normalization processing module, 003 is a solving module, and 004 is an optimization training module. DETAILED DESCRIPTION
[0058] In order to make the technical solutions and advantages in the embodiments of the present application clearer, the exemplary embodiments of the present application are further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0059] Embodiment one
[0060] As shown in the figure, a vehicle key point detection method based on contrast learning in the embodiments of the present application comprises the following steps: Figure 1 S10, acquiring a training sample image set;
[0061] S20, establishing a key point detection model based on heatmap+softargmax regression and training to obtain a pre-trained network model;
[0062] S30, constructing a simCLR framework based on the pre-trained network model, adjusting the pre-trained network model through the simCLR framework, and obtaining a final network model;
[0063] S40, inputting a to-be-detected picture into the final network model to obtain vehicle key point information.
[0064] As shown in the figure,
[0065] , Figure 2 , Figure 3 In the embodiments, the step
[0066] The step S30 of constructing a simCLR framework based on the pre-trained network model, adjusting the pre-trained network model through the simCLR framework, and obtaining a final network model comprises:
[0067] S301, performing twice random data enhancement on the input picture to obtain two pictures with different perspectives;
[0068] S302, inputting the two pictures with different perspectives into an encoder with feature extraction function to extract two image features.
[0069] S303, projecting the two image features to obtain a projection vector;
[0070] S304, calculating the similarity between the two projection vectors;
[0071] S305, constructing a triplet object;
[0072] S306, constructing a triplet loss function based on the triplet object, and iteratively optimizing the simCLR model through the triplet loss function to obtain a final network model.
[0073] In this embodiment, the simCLR model is trained in an unsupervised manner, wherein the simCLR model includes two identical network modules, and for each input network minibatch;
[0074] First, the input picture in the mini batch is subjected to two random data augmentations (such as random cropping, filters, color filtering, and grayscale) to obtain two different views of the picture;
[0075] Then, the two pictures with different views are sent into two convolutional encoders to obtain image features;
[0076] After that, the image features are subjected to a nonlinear transformation for projection to obtain a projection vector;
[0077] Finally, the cosine similarity is used to measure the similarity of the projection.
[0078] Based on this objective, an optimization objective is constructed, specifically: for the same picture in the minibatch, the similarity of the two data augmentation projections is maximized, and the similarity of the projections between different pictures is minimized.
[0079] In the optimization objective, a triplet object is constructed, and a triplet loss function is constructed based on the triplet object; the goal of the triplet loss is to maximize the intra-class similarity and minimize the inter-class similarity.
[0080] First, the re-projection results of the same picture after two data augmentation transformations are compared in similarity, and second, one of the results is compared with a random picture in the minibatch in similarity.
[0081] Embodiment two
[0082] In order to better cope with the key point extraction task, in the embodiment, the traditional simCLR is adjusted, that is, an encoder with feature extraction function is used for feature extraction, and after feature extraction, the prediction results of the two branch features can be mapped back to the original image using the projection transformation matrix, so that the prediction of the same image under two different scales after enhancement is obtained.
[0083] As shown in Figure 4 Based on embodiment one, the vehicle key point detection method based on contrast learning in the embodiment of the application is shown in the step S302, and the encoder with feature extraction function is a key point encoder.
[0084] Specifically, the key point encoder is trained in a supervised manner.
[0085] In the embodiment, in step S20, a key point detection model based on heatmap+softargmax regression is trained, which is a single key point detection network; in step S30, a simCLR framework (including data enhancement, two shared parameter key point encoders, re-projection, and triple loss) is constructed based on the pre-trained model to fine-tune the pre-trained model of step S20, and the simCLR framework represents training with two twin networks, one of which is the key point detection model based on heatmap+softargmax regression in step S20.
[0086] In the embodiment, the training process of the key point encoder includes:
[0087] S001, using a heatmap model to detect key points of a sample image to obtain a key point heat map of the sample image; and training a key point detection model to obtain network weights of a pre-trained model;
[0088] S002, normalizing the key point heat map of the sample image by a normalization function to obtain a probability distribution value;
[0089] S003, solving the coordinate expectation of the position of the maximum value in the probability distribution value;
[0090] S004, constructing a mean square error loss function based on the coordinate expectation and the true value;
[0091] S005, optimizing and training the network weights of the key point encoder based on the key point heat map of the sample image and the mean square error loss function to obtain a trained key point encoder model.
[0092] Specifically, in the step S001, the expression of the key point heat map of the sample image is: B K H W; where B is the batch quantity, K is the number of key points, and H is the number of key points. W represents the size of the feature map.
[0093] In this embodiment, a heatmap-based keypoint encoder is used for keypoint detection, which uses a two-dimensional Gaussian distribution to represent the keypoint locations. Specifically, the output of the heatmap model is B. K H W, where: B is the batch quantity, K is the number of key points, and H is the number of key points. W represents the size of the feature map.
[0094] In this embodiment, a normalization process is added to the heatmap model. For example, the keypoint detection is transformed into a regression problem using the normalization function softargmax; softmax normalization is performed on the keypoint heatmap to obtain the probability of response at each location in the W and H dimensions, respectively; then, the expected coordinates of the location of the maximum value in the probability distribution are calculated; the expected coordinates of the location of the maximum value can be obtained by summing the index probabilities and then averaging them, where: Subsequently, the network weights of the keypoint encoder are optimized and trained based on the keypoint heatmap of the sample images and the mean squared error loss function.
[0095] The technical solution provided in this application enhances the transferability of the keypoint detection network through comparative learning, enabling it to achieve good results on data with different distributions, and improves the stability of keypoints, making it more robust to weak computer graphics transformations.
[0096] In addition, using heatmap models to constrain vehicle features can improve the accuracy of vehicle key points and reduce the dependence of key point detection technology on the amount of data.
[0097] A second aspect of this application also provides a vehicle key point detection system based on contrastive learning.
[0098] like Figure 5 As shown, a vehicle key point detection system based on contrastive learning includes:
[0099] Module 10 is used to acquire the training sample image set;
[0100] Pre-training module 20 is used to build and train a key point detection model based on heatmap+softargmax regression to obtain a pre-trained network model.
[0101] The training module 30 is configured to construct a simCLR framework based on the pre-trained network model, adjust the pre-trained network model through the simCLR framework, and obtain a final network model.
[0102] The detection module 40 is configured to input a to-be-detected picture into the final network model to obtain vehicle key point information.
[0103] As shown in Figure 6 the training module 30 comprises:
[0104] The data enhancement module 301 is configured to perform random data enhancement on an input picture twice to obtain two pictures with different perspectives.
[0105] The feature extraction module 302 is configured to input the two pictures with different perspectives into an encoder with a feature extraction function to extract two image features.
[0106] The projection module 303 is configured to perform a projection operation on the two image features to obtain two projection vectors.
[0107] The calculation module 304 is configured to calculate the similarity between the two projection vectors.
[0108] The optimization module 305 is configured to construct a triple object, construct a triple loss function based on the triple object, and iteratively optimize the simCLR model through the triple loss function to obtain the final network model.
[0109] In this embodiment, the encoder with a feature extraction function in the feature extraction module 302 is a key point encoder.
[0110] As shown in Figure 7 the key point encoder is trained in a supervised manner, and the training process comprises:
[0111] The key point detection module 001 is configured to perform key point detection on a sample image by using a heatmap model to obtain a key point heatmap of the sample image, and train a key point detection model to obtain network weights of a pre-trained model.
[0112] The normalization processing module 002 is configured to perform normalization processing on the key point heatmap of the sample image by using a normalization function to obtain a probability distribution value.
[0113] The solving module 003 is configured to solve a coordinate expectation of a position where a maximum value in the probability distribution value is located.
[0114] The optimization training module 004 is configured to construct a mean square error loss function based on the coordinate expectation and the true value, and to perform optimization training on the network weight of the key point encoder based on the key point heat map of the sample image and the mean square error loss function, so as to obtain the trained key point encoder.
[0115] The third aspect of the embodiment of the present application further provides a vehicle key point detection device based on contrast learning.
[0116] A vehicle key point detection based on contrast learning comprises:
[0117] A memory for storing a computer program;
[0118] A controller for implementing the vehicle key point detection method based on contrast learning as described above when the computer program is executed.
[0119] The memory can be a computer readable storage medium, which can include ROM, RAM, magnetic disk or optical disk, etc.
[0120] The device can be any device capable of realizing advertisement replacement, which can be various terminal devices, such as desktop computers, laptop computers, etc., and can be realized through software and / or hardware.
[0121] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.
[0122] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or one block or multiple blocks. Figure 1 The device that implements the functions specified in one flow or multiple flows and / or one block or multiple blocks.
[0123] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.
[0124] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.
[0125] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the preferred embodiments by those of skill in the art once they have the benefit of the present disclosure. Therefore, the appended claims are intended to encompass within their scope all possible variations and modifications of the preferred embodiments.
[0126] It is apparent that a person skilled in the art can make various changes and modifications to the application without departing from the spirit and scope thereof. Thus, if these modifications and changes fall within the scope of the claims and their equivalents, it is intended to include them in the application.
Claims
1. A vehicle keypoint detection method based on contrastive learning, characterized in that: The method comprises the following steps: S10, obtaining a training sample image set; S20, establishing a key point detection model based on heatmap+softargmax regression and training to obtain a pre-training network model; S30, constructing a simCLR framework based on the pre-training network model, adjusting the pre-training network model through the simCLR framework, and obtaining a final network model; S40, inputting a picture to be detected into the final network model to obtain vehicle key point information; The step S30 comprises: S301, performing twice random data augmentation on the input picture to obtain two pictures with different perspectives; S302, inputting the two pictures with different perspectives into an encoder with feature extraction function to extract two image features; S303, performing projection operation on the two image features to obtain projection vectors; S304, calculating the similarity between the two projection vectors; S305, constructing a triplet object; S306, constructing a triplet loss function based on the triplet object, and iteratively optimizing the simCLR model through the triplet loss function to obtain the final network model; The simCLR framework represents training with two twin networks, one of which is obtained by the key point detection model based on heatmap+softargmax regression in step S20; In the step S302, the encoder with feature extraction function is a key point encoder; The training process of the key point encoder comprises: S001, detecting key points of a sample image by using a heatmap model to obtain a key point heat map of the sample image; training the key point detection model to obtain network weights of the pre-training model; S002, performing normalization processing on the key point heat map of the sample image by using a normalization function to obtain a probability distribution value; S003, solving a coordinate expectation of a position where a maximum value in the probability distribution value is located; S004, constructing a mean square error loss function based on the coordinate expectation and a true value; S005, optimizing and training network weights of the key point encoder based on the key point heat map of the sample image and the mean square error loss function to obtain a trained key point encoder.
2. The vehicle keypoint detection method based on contrastive learning according to claim 1, wherein: The key point encoder is trained in a supervised manner.
3. The vehicle keypoint detection method based on contrastive learning according to claim 2, characterized in that: In the step S001, the expression of the key point heat map of the sample image is B K H W; wherein B is batch size, K is the number of keypoints, H W is the size of the feature map. 4.A vehicle keypoint detection system based on contrastive learning, characterized in that: The method comprises the following steps: An acquisition module (10) is configured to acquire a training sample image set; A pre-training module (20) is configured to establish a key point detection model based on heatmap+softargmax regression and train the model to obtain a pre-training network model; A training module (30) is configured to construct a simCLR framework based on the pre-training network model, adjust the pre-training network model through the simCLR framework, and obtain a final network model; A detection module (40) is configured to input a picture to be detected into the final network model to obtain vehicle key point information; The training module (30) comprises: The data enhancement module (301) is configured to perform twice random data enhancement on the input picture to obtain two pictures of different perspectives. The feature extraction module (302) is configured to input the two pictures of different perspectives into an encoder with feature extraction function to perform feature extraction, and extract two image features. The projection module (303) is configured to perform projection operation on the two image features to obtain two projection vectors. The calculation module (304) is configured to calculate the similarity between the two projection vectors. The optimization module (305) is configured to construct a triplet object, construct a triplet loss function based on the triplet object, and iteratively optimize the simCLR model through the triplet loss function to obtain a final network model. The simCLR framework represents training with two twin networks, one of which is obtained based on the key point detection model in the pre-training module (20) based on heatmap+softargmax regression. The encoder with feature extraction function in the feature extraction module (302) is a key point encoder. The key point encoder is trained in a supervised manner, and the training process includes: The key point detection module (001) is configured to perform key point detection on a sample image by using a heatmap model to obtain a key point heatmap of the sample image, and train a key point detection model to obtain network weights of a pre-training model. The normalization processing module (002) is configured to perform normalization processing on the key point heatmap of the sample image by using a normalization function to obtain a probability distribution value. The solving module (003) is configured to solve a coordinate expectation of a position where a maximum value of the probability distribution value is located. The optimization training module (004) is configured to construct a mean square error loss function based on the coordinate expectation and a true value, and perform optimization training on network weights of the key point encoder based on the key point heatmap of the sample image and the mean square error loss function to obtain a trained key point encoder.