An unsupervised unmanned aerial vehicle target detection method based on multi-granularity contrast

Abstract

The application discloses a kind of unsupervised unmanned aerial vehicle target detection methods based on multi-granularity contrast, mainly including the original unmanned aerial vehicle view image obtained to two random cutting and respectively different data enhancement are made to construct positive sample view pair;In positive sample view pair, according to the overlap rate of image block, establish multi-granularity area corresponding relationship, and save the index of each granularity in view;The image block sampled is input to a pair of initialization parameters same encoder to obtain image representation, wherein one side encoder uses normal gradient descent update, and the other side encoder then uses momentum to update parameter;Using the average pooling of different windows obtains all the representations of different granularities of encoder, then according to the index of each granularity, the representation is extracted respectively through mapping module and prediction module, using contrast loss function minimizes the difference between different granularity positive samples, while maximizing the difference between negative samples;The application can better capture image representation information, so as to improve the accuracy of unmanned aerial vehicle target detection.

Description

Technical Field

[0001] This invention belongs to the field of target detection and relates to an unsupervised UAV target detection method based on multi-granularity comparison. Background Technology

[0002] Object detection and recognition are important research topics in the field of computer vision. With the widespread application of artificial intelligence technology in computer vision, object detection, as one of the representative problems in computer vision, has received increasing attention, and object detection and recognition methods have been greatly developed, especially in various complex image processing fields. Advanced technologies such as computer vision, represented by object detection, have endowed UAVs with autonomous perception, analysis, and decision-making capabilities, enabling them to play an increasingly important role in real life. UAVs combined with intelligent object detection technology can autonomously locate targets of interest, fully leveraging their unique aerial perspective and high maneuverability to achieve flexible and efficient data collection capabilities. Traditional supervised training methods heavily rely on labeled datasets. Unsupervised learning methods, on the other hand, do not require expensive labeled data; that is, they do not require supervised data when learning image representations. Unsupervised methods can effectively solve the problems of insufficient labeled images and high labeling costs in UAV object detection.

[0003] Contrastive learning has achieved remarkable performance in unsupervised visual representation learning. The core idea of ​​contrastive learning is to distinguish different data-augmented views of the same image from other samples, thereby encouraging the model to capture appearance-invariant image representations from a series of data-augmented images. This contrastive approach performs well on single-object-centric datasets, capturing the global semantics of a given image, but it is also prone to missing local semantic information, thus degrading the performance of detail-sensitive downstream tasks such as object detection and instance segmentation.

[0004] To improve representation performance for object detection tasks, some methods have proposed region-level and pixel-level contrastive methods. These methods perform contrastive learning based on the correspondence between regions or pixels from different views and focus on local representations of the image rather than global representations, thus preserving more detail in the representation. However, these methods are all focused on a single semantic granularity, such as object-level semantics or region / pixel-level semantics, and therefore cannot achieve optimal performance on downstream tasks.

[0005] In UAV target detection and recognition tasks, the size of targets of the same or different classes varies greatly, and different objects require different semantic granularities. Unsupervised representations at a single granularity cannot effectively adapt to the object sizes in UAV target detection. Therefore, for unsupervised representation learning methods in UAV target detection, it is important to model different granularities of the image in the task. To this end, we propose a novel multi-granularity contrastive learning method to model the semantic consistency of positive sample views at multiple granularities. Specifically, we follow the characteristics of the Visual Self-Attention (ViT) model to construct a general multi-granularity correspondence between pairs of positive sample views from the image patch level to the image level. For each granularity, we perform contrastive learning based on the overlap rate between corresponding regions of the positive sample views. In this way, the trained model is guided to locate regions in another view given a context and capture the global and local representations of the image. Compared with other contrastive learning methods, our proposed method provides more accurate correspondences between positive sample views, thereby encouraging the model to capture multi-granular representations. Furthermore, existing ViT-based methods have high data requirements, and ViT models cannot generalize well without sufficient training data. This largely hinders the widespread application of the ViT architecture. To address this issue, our proposed multi-granularity comparison method introduces more diverse and refined training objectives, significantly improving data efficiency and the generalization ability of the trained ViT model. Summary of the Invention

[0006] The technical problem solved by this invention is to provide an unsupervised UAV target detection method based on multi-granularity comparison, which can improve the accuracy of UAV target detection, in order to address the shortcomings of the prior art.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] An unsupervised UAV target detection method based on multi-granularity comparison includes the following steps:

[0009] Step 1: Perform two random cropping, random horizontal and vertical flipping, and color changes on the acquired original drone view image to construct a pair of positive sample views with overlapping areas.

[0010] Step 2: Establish a multi-granularity region correspondence in the positive sample view pair and save the index of each granularity in the view.

[0011] Step 3: Place the positive sample pairs into the encoders with shared initial parameters to extract image representation information. One encoder is updated using gradient descent, while the other encoder is updated using momentum.

[0012] Step 4: Extract the representation obtained by the encoder based on the index of each granularity. Pass each granularity representation through the mapping module and the prediction module respectively. Use the contrastive loss function to minimize the difference between positive samples of each granularity, while maximizing the difference between negative samples.

[0013] Step 5: Fine-tune the trained model on the UAV target detection task and detect the results on the test dataset images.

[0014] Further, in step 1, two asymmetric data augmentations are performed on the original UAV view image to obtain two views, I1 and I2. Specifically, the process is as follows: First, a rectangular region for view I1 is cropped from the original image with random area and aspect ratios. The position and size data of the cropped region are then saved.

[0015] Box = [i, j, h, w]

[0016] Where Box represents the rectangle of the cropping area; i represents the x-coordinate of the top left corner of the cropping rectangle; j represents the y-coordinate of the top left corner of the cropping rectangle; h represents the height of the cropping rectangle; and w represents the width of the cropping rectangle.

[0017] Then, the rectangular area of ​​view I2 is cropped again. If the rectangular areas of the two views do not overlap, view I2 is cropped repeatedly until the two views overlap. After cropping the image twice, the rectangular boxes of the two views of the image are obtained, namely Box1 and Box2. Then, bilinear interpolation is used to restore the cropped image to the specified model input size. Then, the image is randomly flipped horizontally and vertically respectively.

[0018] Data augmentation of images is performed using torchvision's transform toolkit, which is a commonly used Python toolkit in the field of computer vision.

[0019] Furthermore, in order to enable the model to learn color-independent representation information, color transformation, grayscale transformation, Gaussian blur, and exposure operations are performed on the image.

[0020] Furthermore, in step 2, a multi-granularity region correspondence is established in view I1 and view I2. First, view I1 and view I2 are divided into different image blocks. and Where U represents the number of image blocks in each row, and V represents the number of image blocks in each column. and This can be represented as (v·w1, u·h1, w1, h1) and (v·w2, u·h2, w2, h2). Here, u and v represent the row and column indices of the image patch, respectively. 和 w1 represents respectively The length and width, h2 and w2 represent respectively The length and width. The overlapping area I of view I1 and view I2. o It can be represented as (x o y o w o h o ).

[0021] To obtain larger granular image patches, we will and Neighboring c×c regions are stitched together to form a larger image patch. The stitching formula is as follows:

[0022]

[0023]

[0024] Where concat(·) represents the image patch stitching function, and c∈{1,2,7,14} represents the coefficient of the stitching granularity. and The image patch sizes are respectively and

[0025] In view I1, and view I2 (or view I...) o The index range of overlapping image patches can be calculated using the following formula:

[0026]

[0027]

[0028] Where k and l represent respectively Row and column indices in view I1. Similarly, in view I... ２ ,and The index range of overlapping image patches can also be calculated using the following formula:

[0029]

[0030]

[0031] Where s and t represent respectively In view I ２ Row and column indexes in the data.

[0032] Positive sample image patch and The corresponding weighting coefficients can be obtained from their overlap rate:

[0033]

[0034] Where S(·) represents the area of ​​a given image patch. In general, the multi-scale corresponding weight coefficients between positive sample pairs can be written as...

[0035] Furthermore, in step 3, a visual Transformer is used to extract the image representation information. All image patches are subjected to a one-dimensional linear projection E(·) to obtain the corresponding token embedding vectors. To ensure that the image does not lose positional information, a learnable positional code E is added to all token embedding vectors. pos In addition to all existing tokens, a class token embedding is added to obtain global information about the image. Then, information exchange between tokens is achieved through the encoder layer of the Transformer. Each encoder layer in the transformer structure has the same structure. The adopted Transformer structure consists of a series of Transformer blocks with the same structure. Each Transformer block consists of a multi-head attention mechanism and a multi-layer perceptron. Layer Norm is used to normalize the input before each Transformer block, and residual connections are used after each Transformer block.

[0036] Input the token sequences of view I1 and view I2 into the basic encoder F respectively. Θ and momentum encoder F Ξ Obtaining image patch representation and The basic encoder F Θ and momentum encoder F Ξ The initialization parameters are the same for the basic encoder F. Θ Using normal gradient descent updates, the momentum encoder F... Ξ The parameters are updated using momentum. The momentum update formula is shown below:

[0037] Ξ=m·Ξ+(1-m)·Θ

[0038] Where Θ is the basic encoder F Θ The model parameters, Ξ is the momentum encoder F Ξ The model parameters are given by m, where m is the momentum update coefficient.

[0039] Furthermore, in step 4, since there are a large number of tokens, all token representations of views I1 and I2 are directly sent to the mapping module. and prediction module Feature mapping significantly increases computational and memory costs. Therefore, in the mapping module... and prediction module Previously, image blocks were sampled based on the image block index calculated in step 2. and To construct multi-granularity representations, we use flat pooling operations with a window size of c to aggregate image patch representations. The aggregated representation is as follows:

[0040]

[0041]

[0042] Then, the basic encoder F Θ The obtained characterization Sent to the mapping module and prediction module Perform feature mapping to obtain predicted features Momentum encoder F Ξ The obtained characterization Sent to the mapping module Obtain mapping features Combining the above multi-granularity representations, a contrastive loss function is used to minimize the difference between positive samples while maximizing the difference between negative samples. The formula for the contrastive loss function is shown below:

[0043]

[0044]

[0045] Where τ and N represent the temperature parameter and batch size, respectively, q (i) and z (i) Let q and z represent the representations of the i-th sample in the data batch, respectively.

[0046] To enable the model to learn multi-granular representations of images, the total loss consists of contrastive losses at four granularities, and the formula for the total loss is as follows:

[0047]

[0048] Where sg(·) represents the gradient truncation operation to prevent potential model collapse.

[0049] Further, in step 5, the Transformer model in the pre-trained base encoder is fine-tuned on an object detection dataset. The parameters of the pre-trained model are loaded into a standard Transformer model, and the mapping and prediction layer parameters of the pre-trained model are discarded. The Mask R-CNN model from the mmdetection library is used for object detection fine-tuning on the publicly available COCO and VisDrone datasets.

[0050] Beneficial effects

[0051] The model constructed in this invention is an unsupervised UAV target detection neural network based on multi-granularity contrast. It mainly includes constructing positive sample view pairs by randomly cropping the original UAV view image twice and performing different data augmentations on each. In the positive sample view pairs, a multi-granularity region correspondence is established based on the overlap rate of image patches, and the index of each granularity in the view is saved. The sampled image patches are input into a pair of encoders with identical initialization parameters to obtain image representations. One encoder uses normal gradient descent for updating, while the other encoder uses momentum to update its parameters. Average pooling with different windows is used to obtain all representations of different granularities passed through the encoders. After extracting the representations based on the index of each granularity, they are passed through a mapping module and a prediction module. A contrastive loss function is used to minimize the difference between positive samples of different granularities while maximizing the difference between negative samples. Experiments on the COCO and VisDrone datasets show that this invention can improve the accuracy of UAV target detection. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the overall method of the present invention.

[0053] Figure 2 This is a schematic diagram of the method of the present invention for sampling image blocks of different granularities. Detailed Implementation

[0054] like Figure 1 As shown, the unsupervised UAV target detection method based on multi-granularity comparison proposed in this invention mainly includes the following steps:

[0055] Step 1: Select the publicly available object detection datasets COCO and VisDrone for pre-training. Perform two asymmetric data augmentations on the training images in the datasets. First, crop the viewfinder from the original image with random area and aspect ratios. Figure 1 The rectangular area, then based on the view Figure 1 The area that overlaps with it is cropped out. Figure 2 A rectangular region is defined. The cropped area is between 0.2 and 1.0 times the size of the original image, and the aspect ratio of the cropped region is 3 / 4 to 4 / 3. The position and dimensions of the cropped region are saved.

[0056] Bilinear interpolation was then used to restore the cropped image to the specified model input size, where the model input size was 224×224 on all datasets. Next, the image was randomly flipped horizontally and vertically with a probability p = 0.5. Further, color transformation, grayscale conversion, and exposure adjustments were performed on the image, and the original data was normalized after adding Gaussian blur.

[0057] Step 2: Construct the correspondence between multi-granularity image block regions. In view I1 and view I... ２ A multi-granularity region correspondence is established. The base image patch size for the model input is 16×16, and views I1 and I2 are divided into different base image patches. and Where U = 14 represents the number of image blocks per row, and V = 14 represents the number of image blocks per column. We will and Neighboring 2×2, 7×7, and 14×14 regions are stitched together to form larger image blocks, resulting in image blocks of four granularity sizes: 1, 2, 7, and 14.

[0058] For each granularity of image patch, calculate the index of the image patch that overlaps with view I2 in view I1. The range of indexes for image patches overlapping with view I2 (or view I0) in view I1 can be calculated using the following formula:

[0059]

[0060]

[0061] Where k and l represent respectively The row and column indices in view I1. Then calculate. Image patch index that overlaps with view I2:

[0062]

[0063]

[0064] Where s and t represent respectively The row and column indices in view I2. Finally, the overlap rate of the overlapping image patches is calculated:

[0065]

[0066] Where S(·) represents the area of ​​a given image patch.

[0067] Step 3: Positive sample pairs are fed into the encoder with shared initial parameters to extract image representation information. All image patches undergo one-dimensional linear projection E(·), specifically using convolution operations to project all features onto a specified dimension to obtain the corresponding token embedding vectors. The convolution kernel and stride are both the size of the image patch, i.e., 16. A small version of the Transformer encoder is used as the backbone network of the model, with 3 input channels and 384 output channels for the convolution. A learnable positional encoding E is added to all token embedding vectors. pos The location encoding dimension is the same as the embedding dimension, which is 384. In addition to all the tokens, a class token embedding is added to obtain global information about the image.

[0068] Then, we use MHSA with h attention heads to realize the information exchange between tokens:

[0069]

[0070] Where 1≤l≤L, This represents the output of the (l-1)th layer Transformer, and Norm represents the batch normalization operation. Then, this method uses two layers of FFN to... For nonlinear transformations, ReLU is used as the activation function between linear layers, and the calculation formula is shown below:

[0071]

[0072] Here, FFN is defined as FFN(x) = W2(ReLU(W1x+b1)+b2). ​​The result is obtained from the last layer of the Transformer. As the final image representation, we used an MHSA with h=8 attention heads and a Transformer encoder with L=12 layers to obtain the image representation.

[0073] Step 4: Extract representations based on granularity indexes, construct mapping and prediction modules, and calculate contrastive loss. To extract multi-granularity representations, we use average pooling with the same window size as the granularity size to aggregate image patch representations, i.e., 1×1, 2×2, 7×7, and 14×14. Sampling is performed from all granularity representations based on the granularity index. Both the mapping and prediction modules use 3-layer linear layers, with ReLU as the activation function between linear layers and BatchNorm for feature normalization. The input dimension of the linear layers is 128, the hidden layer dimension is 2048, and the output dimension is 128. The contrastive loss function minimizes the difference between positive samples while maximizing the difference between negative samples. The formula for the contrastive loss function is shown below:

[0074]

[0075]

[0076] Where τ and N represent the temperature parameter and batch size, respectively, q (i) and z (i) Let q and z represent the representations of the i-th sample in the data batch, respectively. The total loss function is the sum of the contrastive losses at the four granularities:

[0077]

[0078] Where sg(·) represents the gradient truncation operation to prevent potential model collapse.

[0079] For the COCO and VisDrone datasets, we used a batch size of 256 and a learning rate of 1×10⁻⁶. -3 The AdamW optimizer with momentum of 0.996 and weight decay of 0.05 was used. The model was trained for 800 epochs, with the first 10 epochs used for learning rate warm-up. Furthermore, no gradient clipping was applied to the model on these datasets.

[0080] Step 5: Fine-tune the pre-trained Transformer model on the object detection dataset. Load the parameters of the pre-trained model into a standard Transformer model, discarding the mapping and prediction layer parameters of the pre-trained model. For the object detection task, use the Mask R-CNN model from the mmdetection library for fine-tuning. The batch size for fine-tuning is 8, and the learning rate is 6×10⁻⁶. -5 The weight decay was 0.03. The model was trained for a total of 12 epochs, with a learning rate warm-up of 1000 batches. The input sizes of the model on the COCO and VisDrone datasets were 1330×800 and 769×769, respectively.

Claims

1. An unsupervised UAV target detection method based on multi-granularity comparison, comprising the following steps: Step 1: Perform two random cropping, random horizontal and vertical flipping, and color change data enhancement on the acquired original drone view image to construct a pair of positive sample views with overlapping areas; Step 2, in the view and view Establish a multi-granularity region correspondence and save the index of each granularity in the view; first, the view... and view Divided into different image blocks and ,in This represents the number of image blocks in each row. Represents the number of image blocks in each column; and Represented as and ;in and These represent the row index and column index of the image patch, respectively; and Represent Length and width, and Represent Length and width; view and view Overlapping areas Represented as ; To obtain larger granular image patches, and Neighboring Regions are stitched together to form a larger image patch. The stitching formula is as follows: in This represents the image patch stitching function. A coefficient representing the particle size of the splicing; and The image patch sizes are respectively and ; In view In the middle, with view The index range of overlapping image patches is calculated using the following formula: in and They represent In view Row and column indexes in the view; similarly, in the view... ,and The index range of overlapping image patches is calculated using the following formula: in and They represent In view Row indexes and column indexes in the data; Positive sample image patch and The corresponding weighting coefficients are obtained from their overlap rate: in This represents the area of ​​a given image patch; Step 3: Place the positive sample pairs into the encoder with shared initial parameters to extract image representation information; One encoder uses gradient descent for updates, while the other encoder uses momentum for updates; Step 4: Extract the representation obtained by the encoder based on the index of each granularity. Pass each granularity representation through the mapping module and the prediction module respectively. Use the contrastive loss function to minimize the difference between positive samples at each granularity and maximize the difference between negative samples. Step 5: Fine-tune the trained model on the UAV target detection task and detect the results on the test dataset images.

2. The unsupervised UAV target detection method based on multi-granularity comparison according to claim 1, characterized in that, In step 1, the process of performing two asymmetric data augmentations on the original UAV view image is as follows: First, the view is cropped from the original image with random area and random width-to-height ratios. A rectangular area; save the position and size data of the cropping area: Then continue cropping the view. If the rectangular areas of two views do not overlap, then the views are repeatedly clipped. The process continues until the two views overlap; after cropping the image twice, the rectangular boxes of the two views, Box1 and Box2, are obtained; then, bilinear interpolation is used to restore the cropped image to the specified model input size, allowing the model to learn representations independent of image size and region; subsequently, the image is randomly flipped horizontally and vertically; color transformation, grayscale change, and exposure operations are performed on the image; finally, the original data is normalized.

3. The unsupervised UAV target detection method based on multi-granularity comparison according to claim 1, characterized in that, In step 3, a visual Transformer is used to extract the representational information of the image; all image patches are subjected to one-dimensional linear projection. This yields the corresponding token embedding vectors. To prevent the image from losing positional information, learnable positional codes are added to all token embedding vectors. In addition to all the tokens, a class token embedding is added to obtain global information of the image. Then, the information interaction between tokens is realized through the encoder layer of the Transformer. The Transformer structure adopted consists of a series of Transformer blocks with the same structure. Each Transformer block consists of a multi-head attention mechanism and a multi-layer perception mechanism. Layer Norm is used to normalize the input before each Transformer block, and residual connections are used after each Transformer block. The token sequences from View 1 and View 2 are input into two encoders with the same initialization parameters. One encoder updates its parameters using normal gradient descent, while the other updates its parameters using momentum. The momentum update formula is shown below: in Let be the parameters of the model after normal gradient descent update at step t. for( The model parameters of the momentum encoder under the following steps. This is the momentum update coefficient.

4. The unsupervised UAV target detection method based on multi-granularity comparison according to claim 1, characterized in that, In step 4, the representation obtained by the encoder is extracted according to the index of each granularity, and each granularity representation is processed by the mapping module and the prediction module respectively; in the mapping module and prediction module Previously, image patches were sampled based on the image patch index. and Use some window sizes The average pooling operation is used to aggregate image patch representations to construct a multi-granularity representation. The aggregated representation is as follows: Then, the basic encoder The obtained characterization Sent to the mapping module and prediction module Perform feature mapping to obtain predicted features Momentum encoder The obtained characterization Sent to the mapping module Obtain mapping features Combining the above multi-granularity representations, a contrastive loss function is used to minimize the difference between positive samples while maximizing the difference between negative samples. The formula for the contrastive loss function is shown below: in and These represent the temperature parameter and batch size, respectively. and These represent the first and second data items in the batch. Characterization of a sample and ; To enable the model to learn multi-granular representations of images, the total loss consists of contrastive losses at four granularities, and the formula for the total loss is as follows: in This indicates that gradient truncation prevents potential model collapse.

Images