Cross-view remote sensing image target detection method based on step-by-step positioning of background information

Abstract

The application provides a cross-view remote sensing image target detection method based on background information step-by-step positioning. First, two data sets for training are constructed by using existing DOTA data set and Google Earth respectively; then, a classification network based on a sliding window is built, and the constructed training data set is used for training to preliminarily locate a target area; next, a precise positioning model is built, and a new data set is constructed by using the preliminary positioning result to perform training, a rough detection result is generated by a first step MixFormer model, and a precise detection result is generated by a second step MixFormer model; finally, a final detection result is obtained by using the precise positioning model. The application realizes accurate positioning of a target through rough, fine and further fine processing, and can solve the problems of too large target shape change caused by large differences in unmanned aerial vehicle view angles and the problem that a target image in a satellite view angle cannot be recognized.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and specifically to a cross-view remote sensing image target detection method based on progressive localization using background information. Background Technology

[0002] Object detection is a computer vision technique that identifies all objects of interest in an image and determines their category and location. It is widely used in applications such as facial recognition payment, autonomous driving, and crop monitoring. With the development of deep learning, neural network-based object detection in remote sensing images has achieved significant results. In the field of remote sensing image object detection, it is usually necessary to determine the category of the detected targets, referring to a class of targets rather than a specific target. However, in actual military missions, detecting specific targets is crucial. This task can be summarized as: determining the location of a target in another image from an image of that target. Currently, tasks such as Re-ID and Person Search are similar in setup to this task, but these tasks are only used to detect specific people, and no similar tasks have been proposed for remote sensing image detection.

[0003] In actual military missions, aircraft cannot always remain directly above the target when photographing it (i.e., satellite view). Sometimes, due to differences in the aircraft's altitude and orientation, the size and shape of the target change (i.e., drone view). The difference in target shape between drone and satellite views is often significant. While tasks like Re-ID and Person Search have addressed some of these perspective differences, they only solve the problem of shape differences caused by parallel perspectives, not the problem of shape differences caused by large pitch angles. Since no such task has been proposed in the field of remote sensing image target detection, there is currently no dataset that simultaneously includes both satellite and drone perspectives. Furthermore, collecting remote sensing image data from different perspectives is extremely difficult and costly, and some locations are sensitive, making it difficult to obtain corresponding images. Therefore, we can only rely on a large amount of existing remote sensing data from the satellite perspective.

[0004] In summary, there are three main problems in achieving this task: 1) There is no remote sensing image dataset that simultaneously contains satellite and UAV perspectives; 2) There is no task for detecting specific targets in remote sensing images; 3) There is no method for detecting targets in UAV perspective images using satellite-view target images. These three problems are the focus of this invention. Summary of the Invention

[0005] To overcome the problem of significant target shape changes due to differences in UAV pitch perspectives, making target detection difficult using satellite images, this invention provides a cross-view remote sensing image target detection method based on progressive localization using background information. First, two training datasets are constructed using the existing DOTA dataset and Geogle Earth dataset. Then, a sliding window-based classification network is built and trained using the constructed training datasets to initially locate the target region. Next, a precise localization model is built and trained again using the preliminary localization results on a new dataset. The first step, using the MixFormer model, generates coarse detection results, and the second step, using the MixFormer model, generates precise detection results. Finally, the precise localization model is used to obtain the final detection result. This invention achieves accurate target localization through coarse, fine, and further fine processing, solving the problem of excessive target shape changes due to large differences in UAV viewpoints, making target images unrecognizable using satellite images.

[0006] A cross-view remote sensing image target detection method based on progressive localization using background information, characterized by the following steps: Step 1: Construct two training datasets: The first dataset is constructed by filtering existing DOTA datasets for images containing the ship category, and then performing an affine transformation on each image. Each set of images contains the original image and an affine-transformed image, serving as the satellite view and the UAV view, respectively. The second dataset consists of 225 sets of images collected from Geogle Earth, representing both satellite and UAV view images of ships, forming the training set and 542 sets of images, which together constitute the dataset. Each set of images includes satellite-view images and drone-view images; Step 2, train a model for initial target region localization: using the dataset obtained in Step 1 The sliding window-based classification network is trained to obtain the trained network; The sliding window-based classification network mainly includes a feature extraction backbone network ResNet50 and a classification network, wherein the ResNet50 network is pre-trained on ImageNet. The specific processing procedure of the sliding window-based classification network is as follows: For For each group of images in the dataset, a satellite-view image is used as the template image, denoted as... The image viewed from the drone's perspective is used as the search image, denoted as... The application uses a sliding window operation to search for images. Divided into 4 Image of size ,in, Indicates the sequence number of the segmented image. w The value range is [400, 512]; then, the template image in each group of images... and the segmented search image The image features are obtained by inputting them into the ResNet50 network. and Image features and The input is fed into a classification network to obtain the probability that each search image contains the template image. The probability value is then transformed to [0,1] using the Sigmoid function to obtain the final probability value that each segmented image contains the target. ; The classification network mainly consists of two global max pooling layers and a sigmoid function. The specific processing steps are as follows: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] Expand to and The same size, get and will and Multiply to get Then, and The inputs are fed into two global max pooling layers, and the outputs are concatenated together. The final probability value of each segmented image containing the target is obtained by passing the sigmoid function. The expression for the classification network processing procedure is as follows: (1) (2) in, For upsampling operation, This is the global max pooling layer, and [;] represents the join operation. This is an operation of the Sigmoid function; The loss function of a sliding window-based classification network is as follows: (3) in, This represents the binary cross-entropy loss. To determine whether the i-th segmented image contains the true class label of the target. It can be 0 or 1, where 1 indicates inclusion and 0 indicates exclusion; Step 3, Obtain the preliminary target area: [Database] The training set is input into the trained network obtained in step 2 to obtain the preliminary localization target region for each group of images. The specific processing procedure is as follows: For For each set of images in the training set, a satellite view image is used as the template image, and a drone view image is used as the search image. These are input into the network model trained in step 2 to obtain the probability that the four segmented images contain the template image. With the largest 3 The corresponding three drone view segmented images As a preliminary location of the target area; Step 4, train a model to accurately locate the target region: For The training set consists of three UAV-view segmented images of the target area obtained in step 3 and corresponding satellite-view images, forming a new dataset. The new dataset is used to train the precise localization model to obtain a trained model. The precise positioning model mainly includes a two-step MixFormer model and a scoring model. The specific processing steps of the first step, the MixFormer model, are as follows: using satellite imagery... Using the center point of the target detection bounding box as a reference, the satellite view image is cropped. Both the length and width are A square region was obtained, resulting in a satellite view image containing more background information. Then, satellite view images containing more background information are used. Image segmentation from the perspective of drones Inputting the data into the MixFormer model yields segmented images from the drone's perspective. The first positioning detection frame ;in, For cutting dimensions, and These are satellite view images. The length and height of the object detection box, and the coefficients. The value range of is [3, 7]. i =1,2,3; The specific processing steps of the MixFormer model in the second step are as follows: using the first localization detection box... Using the center point as a reference, crop the drone-view segmented image. Both the length and width are A square region is obtained, yielding a finely segmented UAV-view image. Meanwhile, satellite-view images Using the center point of the target detection bounding box as a reference, the satellite view image is cropped. Both the length and width are A square region is obtained to produce a satellite view image with less background information. Then, the finely segmented drone-view images are... and satellite view images with less background information The image is input into the MixFormer model to obtain a segmented image from the drone's perspective. The second positioning detection frame ;in, For cutting size one, and The first positioning detection frame Length and height, coefficient The value range of is [2,3]. For cutting size two, coefficient ; The scoring model calculates the similarity between the second localization detection box of each UAV-view segmented image and the target detection box of the satellite-view image as a score, and takes the second localization detection box with the highest score as the final detection box output. The loss function of the precise target region localization model is as follows: + (4) (5) (6) in, For the total network loss, For L1 loss function, For GIoU loss function, Images from the perspective of a drone The true detection frame, To detect the frame and The compression factor when placed in the smallest bounding box; Step 5, Data Detection: Using the dataset The test set in the image is used as input, and processed according to step 3. The three UAV view segmentation images and the corresponding satellite view images of the preliminary localized target area obtained from each set of images are used as a new set of images to form a new dataset. This dataset is then input into the trained precise localization model obtained in step 4, and the final UAV image detection box is output.

[0007] The beneficial effects of this invention are as follows: In the field of remote sensing image target detection, there are no tasks or methods for cross-view detection of specific targets. The target detection framework proposed in this invention, based on background information and progressive localization, can accurately locate targets through three consecutive steps: coarse, fine, and further fine. This solves the problem that target shape changes too much due to significant differences in UAV viewpoints, making target images from satellite perspectives unrecognizable. This invention is simple to implement, flexible in its methods, and has achieved significant detection results on Geogle Earth test data. Attached Figure Description

[0008] Figure 1 This is a flowchart of the cross-view remote sensing image target detection method based on background information for stepwise localization according to the present invention; Figure 2 This is a schematic diagram of a classification network based on a sliding window; Figure 3 These are the target detection results on Geogle Earth test data. Detailed Implementation

[0009] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0010] To address the problem of excessively variable target shape changes due to significant differences in UAV perspectives, making target images from satellite views unrecognizable, this invention provides an image target detection method that solves the cross-viewpoint problem. The overall flowchart is as follows: Figure 1 As shown, the specific implementation process is as follows: 1. Construct two training datasets The first dataset was constructed by filtering existing DOTA datasets for images containing the ship category, and then performing an affine transformation on each image. Each set of images contains the original image and the image after affine transformation, serving as the satellite view image and the UAV view image, respectively. Only the training set is available; the second is a dataset of satellite and drone-view images of ship types collected independently on Geogle Earth by this invention. ,in It consists of 225 training sets and 542 test sets, with each set including satellite-view images and drone-view images. Used for training step 2, The training set in [the dataset] is used to obtain the results of step 3 and to train step 4. The test set in the dataset is used to test the trained model.

[0011] 2. Train a model to initially locate the target region. This step mainly utilizes the training dataset. The sliding window-based classification network is trained to obtain a trained network.

[0012] To achieve better initial localization performance, this invention designs a classification network based on a sliding window, such as... Figure 2As shown, it mainly includes a feature extraction backbone network ResNet50 and a classification network. The ResNet50 network pre-trained on ImageNet is selected as the feature extraction backbone network, and the classification network mainly includes two global max pooling layers and a Sigmoid function.

[0013] The specific processing procedure of the sliding window-based classification network is as follows: For For each set of images in the dataset, the original image is used as the template image, i.e. The image after affine transformation is used as the search image, i.e. The application uses a sliding window operation to search for images. Divided into 4 Image of size ;in, Indicates the sequence number of the segmented image. H Indicates the height of the template image. W Indicates the width of the template image. C This represents the number of channels in an image. H′ Indicates the height of the search image. W′ Indicates the width of the search image. w The value range is [400, 512].

[0014] The template image in each group of images The search images of the 4 segments ( i The features are obtained by inputting the values ​​(1, 2, 3, 4) into a ResNet50 network. and ,Will Expand to and The same size and put Multiply to get . and The inputs are fed into two global max pooling layers, and the outputs are concatenated together. The result is then passed through a sigmoid function to obtain the final probability value that each segmented image contains the target. The expression for the classification network processing procedure is as follows: (7) (8) in, For upsampling operation, This is the global max pooling layer, and [;] represents the join operation. This refers to the Sigmoid function operation.

[0015] The loss function for the sliding window classification network is as follows: (9) in, This represents the binary cross-entropy loss. To segment the image Does it include the target's true category label, where It can be 0 or 1, where 1 indicates inclusion and 0 indicates exclusion; The first classification model obtained based on the sliding window model i The probability that a segmented image contains a template image.

[0016] 3. Obtain the initial location of the target area This step mainly utilizes the trained network obtained in step 2, and inputs it into the dataset obtained in step 1. The training set in the data is obtained The initial localization area of ​​the drone's perspective image in each set of images.

[0017] The specific processing procedure for obtaining the area of ​​the initial target location is as follows: For For each set of images, a satellite view image is used as the template image, and a drone view image is used as the search image. These are then input into the model trained in step 2 to obtain four segmented images. Probability of containing template image ( i =1,2,3,4). Find the three largest ones. The corresponding 3 segmented drone images This serves as the initial target area and as input for step 4.

[0018] 4. Train a model to accurately locate the target region. This step mainly involves taking the three UAV-view segmented images of the preliminary target area obtained in step 3 and the corresponding satellite-view images as a new set of images to form a new dataset. This new dataset is then used to train the precise localization model, resulting in a trained model. The precise target area localization model mainly consists of a two-step MixFormer model and a scoring model.

[0019] The specific processing steps of the first step, MixFormer model, are as follows: Given satellite view images... Target detection bounding box ,in, and For satellite view images Detection box Minimum coordinates Minimum coordinates and Set the coefficients for the length and height of the detection frame. The value range is [3,7]. and Multiplication and square root, and with coefficients Multiply, we get Cutting size The expression is as follows: (10) Satellite view images Points in the target detection box Based on the baseline, crop the satellite view image. Both the length and width are A square region was obtained, resulting in a satellite view image containing more background information. Satellite images containing a lot of background information and the Segmented images from a drone's perspective Inputting into the MixFormer model yields the result for the first... Segmented images from a drone's perspective The detection frame for the first localization ,in , and For detection box of Minimum coordinates Minimum coordinates and Detection boxes Length and height, i =1,2,3.

[0020] The second step, the specific processing of the MixFormer model, involves setting coefficients. Based on the output prediction box of the MixFormer model in the first step midpoint Based on the standard, cut the first Segmented images from a drone's perspective Both the length and width are A square region is obtained, resulting in a finely segmented satellite view image. ,coefficient The value range of is [2,3]. The calculation formula is as follows: (11) At the same time, set coefficients , If the value can be 2, it represents a satellite-view image. Points in the target detection bounding box Based on the baseline, crop the satellite view image. Both the length and width are A square region is obtained to produce a satellite view image with less background information. . The calculation formula is as follows: (12) Then, finely segmented satellite view images and satellite view images with less background information Inputting into the MixFormer model yields the result for the first... Finely segmented images from a drone's perspective The detection frame for the second positioning ,in , and For detection box of Minimum coordinates Minimum coordinates and Detection boxes Length and height.

[0021] The specific processing procedure of the scoring model is as follows: Since the MixFormer model obtained a total of 3 detection boxes in the first two steps... , i =1,2,3. In order to obtain a unique final output, this invention uses a scoring model to score the similarity between the target corresponding to each detection box and the target in the satellite view, and takes the prediction box corresponding to the highest score as the final result.

[0022] The loss function of the precise target region localization model is as follows: + (13) (14) (15) in, For L1 loss function, For GIoU loss function, For the total network loss, Images from the perspective of a drone The true detection frame, To detect the frame and The compression factor when placed within the smallest bounding box. Because... Since the training set data is relatively small, the original MixFormer model parameters are used as the initial values ​​for the parameters in the model for accurately locating the target region, which can prevent the model from overfitting.

[0023] 5. Data Detection This step mainly utilizes The test set of drone images and satellite images are input into the network trained in step 2, which outputs three segmented drone view images. These three segmented drone view images and the corresponding satellite view images are then used as a new set of images and input into the network trained in step 4. Finally, the detection box on the drone image is output, and the target contained in the detection box is consistent with the target in the satellite image.

[0024] For training the sliding window-based classification model, this embodiment selected 235 images containing ports and ships from the remote sensing data DOTA. Affine transformations were used to obtain 3964 data pairs as the training set. Each data pair consists of the original image and the transformed image, representing the satellite view and the UAV view, respectively. For training the precise positioning model, this embodiment collected 45 sets of data (225 pairs) related to ship categories from Geogle Earth as the training set, and collected 68 sets of data with smaller targets and 54 sets of data with larger targets (542 pairs) as the test set. These included scenarios such as navigation, berthing, shore repair, single target, and multiple target scenarios. Each data set consisted of one satellite view image taken at approximately 5000km and four or five UAV view images taken at approximately 4000km. Viewpoint difference was defined as a change in pitch angle of [0, 60] degrees, a change in rotation angle of [0, 360] degrees, and a scale difference of less than 30%.

[0025] Figure 3 The target detection results of the method of this invention on the Geogle Earth test set are presented. In Figure (a), the detection results of different small targets are shown, and in Figure (b), the detection results of different large targets are shown. The leftmost column is the template area of ​​the satellite view image, and the red box is the target; the others are the search areas of the UAV view image, and the red boxes are the detected targets.

[0026] When tested on the Geogle Earth test set with both small and large targets, this invention was considered a successful detection when the Intersection over Union (IoU) between the predicted and labeled bounding boxes was greater than 0.5. Accuracy was calculated by dividing the number of successfully detected targets by the total number of targets. Table 1 records the accuracy of this invention for the two target detection methods, showing that the method achieves good results, demonstrating its effectiveness and advancement. The impact of inputting different satellite viewpoint image sizes into the two-step MixFormer model, and the impact of cropping the UAV viewpoint image size in the second-step MixFormer model, were also compared, i.e., the coefficients... , , and The effects of this invention are considered, and the best results are obtained, as shown in Table 2. Due to the uniqueness and advancement of the task proposed in this invention, this invention is currently the only method to solve this task.

[0027] Table 1

[0028] Table 2

[0029] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-view remote sensing image target detection method based on progressive localization using background information, characterized in that... The steps are as follows: Step 1: Construct two training datasets: The first dataset is constructed by filtering existing DOTA datasets for images containing the ship category, and then performing an affine transformation on each image. Each set of images contains the original image and an affine-transformed image, serving as the satellite view and the UAV view, respectively. The second dataset consists of 225 sets of images collected from Geogle Earth, representing both satellite and UAV view images of ships, forming the training set and 542 sets of images, which together constitute the dataset. Each set of images includes satellite-view images and drone-view images; Step 2, train a model for initial target region localization: using the dataset obtained in Step 1 The sliding window-based classification network is trained to obtain the trained network; The sliding window-based classification network mainly includes a feature extraction backbone network ResNet50 and a classification network, wherein the ResNet50 network is pre-trained on ImageNet. The specific processing procedure of the sliding window-based classification network is as follows: For For each group of images in the dataset, a satellite-view image is used as the template image, denoted as... The image viewed from the drone's perspective is used as the search image, denoted as... The application uses a sliding window operation to search for images. Divided into 4 Image of size ,in, Indicates the sequence number of the segmented image. w The value range is [400, 512]; then, the template image in each group of images... and the segmented search image The image features are obtained by inputting them into the ResNet50 network. and Image features and The input is fed into a classification network to obtain the probability that each search image contains the template image. The probability value is then transformed to [0,1] using the Sigmoid function to obtain the final probability value that each segmented image contains the target. ; The classification network mainly consists of two global max pooling layers and a sigmoid function. The specific processing steps are as follows: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] Expand to and The same size, get and will and Multiply to get Then, and The inputs are fed into two global max pooling layers, and the outputs are concatenated together. The final probability value of each segmented image containing the target is obtained by passing the sigmoid function. The expression for the classification network processing procedure is as follows: (1) (2) in, For upsampling operation, This is the global max pooling layer, and [;] represents the join operation. This is an operation of the Sigmoid function; The loss function of a sliding window-based classification network is as follows: (3) in, This represents the binary cross-entropy loss. To determine whether the i-th segmented image contains the true class label of the target. It can be 0 or 1, where 1 indicates inclusion and 0 indicates exclusion; Step 3, Obtain the preliminary target area: [Database] The training set is input into the trained network obtained in step 2 to obtain the preliminary localization target region for each group of images. The specific processing procedure is as follows: For For each set of images in the training set, a satellite view image is used as the template image, and a drone view image is used as the search image. These are input into the network model trained in step 2 to obtain the probability that the four segmented images contain the template image. With the largest 3 The corresponding three drone view segmented images As a preliminary location of the target area; Step 4, train a model to accurately locate the target region: For The training set consists of three UAV-view segmented images of the target area obtained in step 3 and corresponding satellite-view images, forming a new dataset. The new dataset is used to train the precise localization model to obtain a trained model. The precise positioning model mainly includes a two-step MixFormer model and a scoring model. The specific processing steps of the first step, the MixFormer model, are as follows: using satellite imagery... Using the center point of the target detection bounding box as a reference, the satellite view image is cropped. Both the length and width are A square region was obtained, resulting in a satellite view image containing more background information. Then, satellite view images containing more background information are used. Image segmentation from the perspective of drones Inputting the data into the MixFormer model yields segmented images from the drone's perspective. The first positioning detection frame ;in, For cutting dimensions, and These are satellite view images. The length and height of the object detection box, and the coefficients. The value range of is [3, 7]. i =1,2,3; The specific processing steps of the MixFormer model in the second step are as follows: using the first localization detection box... Using the center point as a reference, crop the drone-view segmented image. Both the length and width are A square region is obtained, yielding a finely segmented UAV-view image. Meanwhile, satellite-view images Using the center point of the target detection bounding box as a reference, the satellite view image is cropped. Both the length and width are A square region is obtained to produce a satellite view image with less background information. Then, the finely segmented drone-view images are... and satellite view images with less background information The image is input into the MixFormer model to obtain a segmented image from the drone's perspective. The second positioning detection frame ;in, For cutting size one, and The first positioning detection frame Length and height, coefficient The value range of is [2,3]. For cutting size two, coefficient ; The scoring model calculates the similarity between the second localization detection box of each UAV-view segmented image and the target detection box of the satellite-view image as a score, and takes the second localization detection box with the highest score as the final detection box output. The loss function of the precise target region localization model is as follows: + (4) (5) (6) in, For the total network loss, For L1 loss function, For GIoU loss function, Images from the perspective of a drone The true detection frame, To detect the frame and The compression factor when placed in the smallest bounding box; Step 5, Data Detection: Using the dataset The test set in the image is used as input, and processed according to step 3. The three UAV view segmentation images and the corresponding satellite view images of the preliminary localized target area obtained from each set of images are used as a new set of images to form a new dataset. This dataset is then input into the trained precise localization model obtained in step 4, and the final UAV image detection box is output.

Images