Unsupervised aerial vehicle keypoint detection method based on isometric constraint and skeleton generation

Abstract

The application discloses an unsupervised aerial key point detection method based on equiformity constraint and skeleton generation. The method comprises the following steps: performing random affine transformation on an input image, extracting key points from the input image and the transformed image respectively by using a key point extraction network, and obtaining a first key point set and a second key point set; applying equiformity constraint on the first key point set and the second key point set, and calculating equiformity loss; drawing a skeleton graph by using the first key point set; applying a random mask to the input image to obtain an image with destroyed structure information; applying a weight to the image with destroyed structure information, and splicing the image with the skeleton graph in a channel dimension to obtain a spliced feature map; sending the spliced feature map to a decoder to obtain a reconstructed image; and applying a reconstruction constraint to the input image and the reconstructed image, and training the network by minimizing a total loss function. The application adopts an unsupervised learning mode, does not need a manually labeled data set, and greatly saves labeling cost.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer vision key point detection, and particularly relates to an unsupervised aerial vehicle key point detection method based on isometry constraint and skeleton generation. BACKGROUND

[0002] Key point detection is very important in computer vision, and it is the basis for image matching, target tracking, three-dimensional reconstruction, pose estimation and other tasks. For photoelectric tracking equipment, as the frame frequency and resolution are improved, the tracking task has higher requirements for the high-precision positioning capability of the flying target, and the key point detection technology is of great significance for the accurate measurement of the photoelectric system, which can obtain high semantic pose representation and perception, and is beneficial to subsequent pose estimation and trajectory analysis.

[0003] Traditional key point detection algorithms describe or model the selected local region, and detect the key points in the local region based on the calculated features or established model by using a classifier or a regressor. The traditional algorithms have the advantages of simple structure, low computational complexity and strong interpretability, but they need to manually design feature descriptors, are not strong in universality, and are not strong in robustness to illumination, pose change and motion blur. The key point detection method based on supervised learning automatically learns feature representation from a large amount of labeled data by means of a deep neural network, and can realize accurate key point positioning. However, its dependence on labeled data and high labeling cost limit its application in some fields. In contrast, the key point detection method based on unsupervised learning has a broader application prospect. In the process of network training, no real label of data is needed as supervision, which undoubtedly saves a lot of time and effort. They use the unsupervised learning method in deep learning to extract and represent the features of the target in the image, and after data integration processing, they expect to locate the stable and distinguishable points in the image. However, in practical application, it is very challenging because of the complexity of the scene and the flying target itself. The target may rotate, change in size, and be motion blurred during movement, and it may also be disturbed by factors such as illumination change, atmospheric turbulence and complex background, which will affect the apparent features of the target, resulting in detection point drift and semantic inconsistency. Therefore, it is still a problem to be solved in engineering application to develop a stable, robust and accurate unsupervised key point detection algorithm for actual scenes. SUMMARY

[0004] To solve the above technical problems, the application provides an unsupervised aerial vehicle key point detection method based on isometry constraint and skeleton generation. The isometry constraint is introduced into the key point detection network to solve the problem of semantic inconsistency of key points, and the skeleton is drawn by using the key points to extract the general structure information of the aircraft to solve the problem of key point drift.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An unsupervised aircraft key point detection method based on isovariability constraints and skeleton generation has the following steps:

[0007] Step 1: Process the input image Perform a random affine transformation to obtain the transformed image. ;

[0008] Step 2: Use a keypoint extraction network to process the input image. and the transformed image Extract key points to obtain the first and second key point sets B1={ } and B2={ },in This represents the number of keypoints. First, the Simple Baseline keypoint extraction network is used to extract keypoints from the input image. and the transformed image The first key point heat map Second key point heat map Heatmap of the first key point Use the Soft-argmax function to obtain the coordinates of each keypoint. Heatmap of the second key point Use the Softmax function to obtain the probability map of key points. For probability graphs Applying the inverse of the random affine transformation in step one yields the transformation probability diagram. , for the transformed probability graph The coordinates of each key point are obtained by taking the expected value in coordinate space. The formulas for calculating the coordinates of key points, probability maps, and transformation probability maps are as follows:

[0009] ,

[0010] in, These are the normalized pixel coordinates. It is the inverse transformation operation of random affine transformation;

[0011] Step 3: For the first and second key point sets B1={ } and B2={ Applying isovariability constraints results in isovariability loss. The calculation formula is as follows:

[0012] ,

[0013] in, is the number of samples in the dataset, is the key point set of the th sample, is the key point set of the th sample after random affine transformation;

[0014] Step four: draw the skeleton map using the key point set B1={ } In the first key point set, take two key points and Draw the differentiable edge map Assign a trainable weight greater than zero to each edge Get the weighted edge map Obtain the final skeleton map by taking the maximum value of each pixel in the weighted edge map :

[0015] ,

[0016] wherein is a hyperparameter that controls the thickness of the skeleton, is the normalized pixel coordinate Euclidean distance of the edge drawn between two key points and , and t is a control variable for calculating ;

[0017] Step five: divide the input image into several grids, and randomly mask 80% of the image data in the grid to obtain an image with destroyed structure information ;

[0018] Step six: apply a weight to the image with destroyed structure information , and concatenate it with the skeleton map in the channel dimension to obtain a concatenated feature map;

[0019] Step seven: send the concatenated feature map to the UNet decoder structure to obtain the reconstructed image ;

[0020] Step eight: apply the reconstruction constraint to the input image and the reconstructed image , and the reconstruction constraint is defined based on the perceptual loss, which is calculated by evaluating the activation values of the image at different levels in the convolutional neural network. The calculation formula of the reconstruction loss is as follows:

[0021] ,​

[0022] wherein, is the number of samples in the dataset, is the feature extractor, which is a pre-trained VGG convolutional neural network, represents the input image;

[0023] Step nine: calculate the total loss function, train the network by minimizing the total loss function, the total loss function , is the weight coefficient of balancing the reconstruction loss and the isometry loss.

[0024] The beneficial effects of the present application are:

[0025] (1) The present application does not need to manually label key point labels, saves manpower, material resources and financial resources, and can automatically discover structures and patterns in data, and has strong universality.

[0026] (2) The present application uses key points to draw skeletons, and simultaneously shares and updates the weight of each edge of the skeleton with each sample in the training set, thereby extracting general structural information of the aircraft and capturing complex structural dependency relationships between parts of the aircraft, and solving the problem of key point drift.

[0027] (3) The present application makes random affine transformation to the input image, then extracts the key points of the original image and the transformed image respectively and applies isometric constraints to them, thereby making the network learn various transformation relationships, which is more robust, and solving the problem of inconsistent semantics of key points. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the flow chart of the unsupervised aircraft key point detection method of the present application based on isometric constraint and skeleton generation;

[0029] Figure 2 is the flow chart of the isometric constraint;

[0030] Figure 3 is the skeleton drawing effect diagram;

[0031] Figure 4 is the key point detection effect diagram. DETAILED DESCRIPTION

[0032] The present application will be further described below in combination with the drawings and examples.

[0033] The application provides an unsupervised aircraft key point detection method based on isometry constraint and skeleton generation, solves the problems of the prior art, such as the need for a large amount of labeled data for training, and the occurrence of key point drift and semantic inconsistency in the face of rotation, scale change, motion blur and other situations or interference such as illumination change, atmospheric turbulence and complex background. The key point is essentially a feature, which is an abstract description of a fixed region or spatial physical relationship. It not only represents a position, but also represents the combination relationship of the context and the surrounding neighborhood, embodies the structural information, and the same type of aircraft should have the same topological structure, which can be represented by a similar skeleton structure. The key point is used to draw and generate skeleton information, which can not only learn the general structural features of the same type of aircraft, but also capture the complex structural dependency relationship between the parts of the aircraft, so that the network has stronger generalization and robustness and can cope with key point drift. When the aircraft posture changes, due to the existence of the robust topological structure of the skeleton, the detected key points can generally be distributed in each part of the aircraft, but the semantics of the key points may not be consistent at this time, for example, the head of a frame of aircraft may correspond to the tail of another frame of aircraft, because the network is difficult to handle such change relationship. The application solves the problem of semantic inconsistency by randomly performing affine transformation on the input image, extracting the key points of the original image and the transformed image respectively and applying isometry constraint to the key points.

[0034] As shown in the specific process of the application, the steps include the following: Figure 1

[0035] Step one: randomly performing affine transformation on the input image to obtain a transformed image ;

[0036] Affine transformation refers to a transformation of an image in two-dimensional or three-dimensional space by linear transformation (such as rotation, scaling, shearing, etc.) and translation, which maintains the parallel linear relationship and proportional relationship of the image. Affine transformation does not change the straightness of straight lines in the image and the proportional relationship of relative positions. Affine transformation can be represented by a 2x3 matrix, so that each point in the image is mapped to a new position. The matrix form of affine transformation is:

[0037] ,

[0038] determines the rotation, scaling and shearing operations, is the translation amount along the x and y directions. Since affine transformation is a combination of linear transformation (such as rotation, scaling, shearing) and translation, is used to control the rotation, scaling and shearing operations, ​​​​These six parameters are used to control translation operations. By changing these six parameters, different transformation effects can be achieved. That is, random affine transformation can simulate the random attitude changes of an aircraft.

[0039] Step 2: Use a keypoint extraction network to process the input image. and the transformed image Extract key points to obtain the first and second key point sets B1={ } and B2={ },in This refers to the number of key points. For example... Figure 2 The flowchart shown is an isovariability constraint. First, the Simple Baseline keypoint extraction network is used to extract the input images. and the transformed image The first key point heat map Second key point heat map Heatmap of the first key point Use the Soft-argmax function to obtain the coordinates of each keypoint. Heatmap of the second key point Use the Softmax function to obtain the probability map of key points. For probability graphs Applying the inverse of the random affine transformation in step one yields the transformation probability diagram. , for the transformed probability map The coordinates of each key point are obtained by taking the expected value in coordinate space. The formulas for calculating the coordinates of key points, probability maps, and transformation probability maps are as follows:

[0040] ,

[0041] in, These are the normalized pixel coordinates. It is the inverse transformation operation of random affine transformation;

[0042] Step 3: For the first and second key point sets B1={ } and B2={ Applying isovariability constraints results in isovariability loss. The calculation formula is as follows:

[0043] ,

[0044] in, It is the number of samples in the dataset. It is the first A set of key points for each sample It is the first The set of key points extracted after performing random affine transformation on a sample;

[0045] Step 4: Utilize the key point set B1={ Draw a skeleton diagram Select two key points from the first set of key points. and Draw a differentiable edge graph. Assign a trainable weight greater than zero to each edge. Obtain the weighted edge graph By taking the weighted edge graph The final skeleton map is obtained by finding the maximum value of each pixel. :

[0046] ,

[0047] in, It is a hyperparameter that controls the thickness of the skeleton. These are the normalized pixel coordinates. To two key points and The Euclidean distance of the drawn edge, t is the value used to calculate... The control variable. Figure 3 The image shows the effect of drawing the skeleton. For different rotations and scale changes, the network extracts the general skeletal structure information of the aircraft. The key points are stably distributed in various parts of the aircraft, and there is no key point drift.

[0048] Step 5: Input Image Divide the image into several grids, randomly mask 80% of the image data in each grid, and obtain an image whose structural information is destroyed. ;

[0049] Step Six: For images whose structural information has been destroyed Apply weights and skeleton diagram The feature maps are concatenated along the channel dimension to obtain the concatenated feature maps.

[0050] Step 7: Feed the stitched feature maps into the UNet decoder structure to obtain the reconstructed image. ;

[0051] Step 8: Process the input image and reconstructed images Reconstruction constraints are applied, based on the definition of perceptual loss, which is calculated by evaluating the activation values ​​of the image at different levels in the convolutional neural network. The formula for calculating the reconstruction loss is as follows:

[0052] ,

[0053] wherein, is the number of samples in the dataset, is the feature extractor, which is a pre-trained VGG convolutional neural network, represents the i-th input image;

[0054] Step nine: calculate the total loss function, train the network by minimizing the total loss function, the total loss function , is the weight coefficient for balancing the reconstruction loss and the isometry loss.

[0055] After steps 1-9 of the present application, a trained Simple Baseline key point extraction network can be obtained. By inputting an airplane picture into the network, stable and robust key point coordinates can be obtained. Figure 4 The effect diagram of the key point detection of the airplane is shown in the figure. In the face of different rotations and scale changes of the airplane, the semantics of the key points are consistent, which benefits from the isometry constraint. By randomly performing affine transformation on the input image, the key points of the original image and the transformed image are extracted respectively, and the isometry loss is calculated. The network can learn the various transformation relationships of the airplane posture, thereby being more robust.

[0056] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only for specific embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.​

Claims

1. An unsupervised aerial keypoint detection method based on isometric constraint and skeleton generation, characterized in that, The method comprises the following steps: Step one: to input image do random affine transformation, get transformed image ; Step 2: Use a keypoint extraction network to process the input image. and the transformed image Extract key points to obtain the first and second key point sets B1={ } and B2={ },in The number of keypoints; including: extracting keypoints from the input image using the Simple Baseline keypoint extraction network. and the transformed image The first key point heat map Second key point heat map Heatmap of the first key point Use the Soft-argmax function to obtain the coordinates of each keypoint. Heatmap of the second key point Use the Softmax function to obtain the probability map of key points. For probability graphs The inverse transform of the random affine transformation yields the transform probability diagram. , for the transformed probability graph The coordinates of each key point are obtained by taking the expected value in coordinate space. ; Step three: apply the equivariance constraint to the first and second keypoint sets B1={ } and B2={ }, and calculate the equivariance loss ; wherein the equivariance constraint is defined by norm, and the calculation formula of the equivariance loss is as follows: ， wherein, is the number of samples in the training data set, is the key point set of the th sample, is the key point set extracted from the th sample after random affine transformation. Step four: Utilizing the first set of key points Drawing a skeleton map ; Step five: applying a random mask to the input image Applying a random mask to the input image results in a structure information-destroyed image ; Step six: images with structural information destroyed Applying weights , and the skeleton map Concatenate in the channel dimension to obtain the concatenated feature map; Step seven: send the spliced feature map to the decoder to obtain the reconstructed image ; Step 8: Process the input image and reconstructed images Apply reconstruction constraints and calculate reconstruction loss. ; Step nine: calculating a total loss function, and training the network by minimizing the total loss function.

2. The unsupervised drone keypoint detection method based on isometric constraint and skeleton generation according to claim 1, characterized in that, In the step one, the random affine transformation refers to transforming the image by linear transformation in two-dimensional or three-dimensional space, so as to maintain the parallel linear relationship and the proportional relationship of the image.

3. The unsupervised drone keypoint detection method based on isometric constraint and skeleton generation according to claim 2, characterized in that, In step four, two key points are selected from the first set of key points. and Draw a differentiable edge graph. Assign a trainable weight greater than zero to each edge. Obtain the weighted edge graph By taking the weighted edge graph The final skeleton map is obtained by finding the maximum value of each pixel. : ， where, is a hyper-parameter controlling the backbone thickness, is the normalized pixel coordinate to two key points and the Euclidean distance of the drawn edge, t denotes the control variable for calculating .

4. The unsupervised drone keypoint detection method based on isometric constraint and skeleton generation according to claim 3, characterized in that, The step five includes dividing the input image into several grids, randomly shielding 80% of the image data in the grid to obtain an image with destroyed structure information .

5. The unsupervised drone keypoint detection method based on isometric constraint and skeleton generation according to claim 4, characterized in that, The step seven uses UNet decoder to reconstruct the original input image , and obtains a reconstructed image .

6. The unsupervised drone keypoint detection method based on isometric constraint and skeleton generation according to claim 5, characterized in that, In the step eight, the reconstruction constraint is defined based on a perceptual loss, which is calculated by evaluating the activation values of different layers of the image in the convolutional neural network, and the calculation formula of the reconstruction loss is as follows: ， in, It is the number of samples in the dataset. It is a feature extractor. Indicates the first One input image sample.

7. The unsupervised drone keypoint detection method based on isometric constraint and skeleton generation according to claim 6, characterized in that, the total loss function , is a weight coefficient balancing the reconstruction loss and the equivariance loss.

Images