A method for identifying and locating small targets for airborne wide-field-of-view reconnaissance payloads

Abstract

The present invention discloses a method for identifying and locating small targets for airborne wide-field reconnaissance payloads, including: preprocessing an initial wide-field reconnaissance image to generate a target image; inputting the target image into a wide-field reconnaissance image ground small target detection network model to perform target detection and obtain a prediction result; converting the prediction result into the initial wide-field reconnaissance image, performing non-maximum suppression on the prediction result, and obtaining the target detection result in the initial wide-field reconnaissance image; based on the target detection result, using the mapping relationship between pixel position and geographic location, converting the pixel position information in the target detection result into geographic location coordinates, and locating the target based on the geographic location coordinates. The present invention avoids repeated detection of targets in multiple slice images, and has a good positioning effect by adaptively preprocessing wide-field reconnaissance images with different image sizes and resolutions.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection for airborne reconnaissance payloads, and in particular to a method for identifying and locating small targets for airborne wide-field-of-view reconnaissance payloads. Background Art

[0002] In the application scenario of airborne reconnaissance payload, it is a very challenging task to detect wide-field targets, especially the detection and positioning of small ground targets. The images of airborne wide-field reconnaissance payloads are generally large, with large differences in image scale and resolution. There is a lot of background noise interference in the images. The small ground targets themselves are small in size and large in number. In addition, in different complex scenes such as cities, suburbs, and forests, the clarity, color, and spatial resolution of the targets are low, the contours and colors lack contrast, and the texture and geometric features are relatively blurred. The scale, rotation angle, and target density of the targets in different wide-field reconnaissance images are also different. How to quickly and accurately identify and locate small ground targets of concern in complex wide-field reconnaissance images is an urgent problem to be solved.

[0003] At present, the target detection method based on deep learning has become the mainstream of target detection in airborne wide-field reconnaissance images. Detection networks such as R-CNN, Fast R-CNN, Faster R-CNN, YOLO series, and SSD series have been applied in this field. At present, they are mainly used for targets such as buildings, vegetation, roads, water systems, bridges, and ships. These targets are relatively large in scale in wide-field reconnaissance images and are usually concentrated in a certain area. However, the recognition and detection performance of relatively sparsely distributed weak targets similar to the background and with blurred edges is poor. In recent years, some target detection methods based on improved Transformer structures have emerged, which greatly increase the complexity and computational cost of the model, and the deployment and application conditions are limited, especially for edge use.

[0004] In visual tasks, most small target detection methods are improved based on mainstream target detection networks, especially methods based on convolutional neural networks have great advantages in extracting underlying features. In wide-field reconnaissance image target detection, a variety of relatively effective methods have been proposed. The paper Remote Sensing Image Detection Based on Dense Connections and Feature Enhancement uses a densely connected network to improve YOLOv3, which improves the target detection effect on wide-field images; the paper SCRDet: Towards More Robust Detection for Small, Cluttered and Rotated Objects improves on the basis of RetinaNet, and adds spatial and channel attention to improve the detection accuracy of small targets; the paper Deformable DETR: Deformable Transformers for End-to-End Object Detection applies Transformer to the field of target detection, uses its self-attention mechanism to focus on global information, and better realizes visual modeling, but the detection effect of small targets in wide-field images needs to be improved.

[0005] Furthermore, target detection methods based on deep learning rely to a certain extent on the size and diversity of data sets. However, there are currently problems such as insufficient sample size of high-resolution wide-field reconnaissance image data sets, blocked data acquisition channels, and overfitting and reduced model generalization capabilities in training with small samples of new category data. As a result, the learned knowledge cannot be effectively generalized to new categories with a small number of samples. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a small target recognition and positioning method for an airborne wide-field-of-view reconnaissance payload to improve the effect of small target detection.

[0007] To achieve the above object, the present invention provides a method for identifying and locating small targets for an airborne wide-field-of-view reconnaissance payload, comprising:

[0008] Preprocessing the initial wide-field reconnaissance image to generate a target image;

[0009] Inputting the target image into a wide-field reconnaissance image ground small target detection network model for target detection to obtain a prediction result, wherein the wide-field reconnaissance image ground small target detection network model is constructed using an improved yolov5 network;

[0010] Converting the prediction result into the initial wide-field reconnaissance image, performing non-maximum suppression on the prediction result, and obtaining a target detection result in the initial wide-field reconnaissance image;

[0011] Based on the target detection result, the mapping relationship between the pixel position and the geographical location is utilized to convert the pixel position information in the target detection result into the geographical location coordinates, and the target is located based on the geographical location coordinates.

[0012] Preferably, the initial wide-field reconnaissance image is preprocessed, including:

[0013] Reading pixel information and geographic information in the initial wide-field reconnaissance image, calculating the real position through the pixel position according to the map projection relationship, and obtaining a mapping relationship between the pixel position and the geographic location;

[0014] Correcting the initial wide-field reconnaissance image, filtering out various interferences and noises in the image background, and adjusting terrain undulations, lens distortion, and camera tilt;

[0015] The resolution of the initial wide-field reconnaissance image is determined, and when the image resolution is not greater than a set value, adaptive scaling is performed; when the image resolution is greater than the set value, cropping is performed with a set pixel size and adaptive overlap to obtain a slice image.

[0016] Preferably, the wide-field reconnaissance image ground small target detection network model includes:

[0017] Data enhancement module: used to perform mixed data enhancement on the input image to obtain a diverse data set;

[0018] Feature extraction module: used to extract features from the diversity data set;

[0019] Feature fusion module: used to fuse the extracted features;

[0020] Coefficient adjustment module: used to generate semantic features of small sample data categories and feature coefficients of regions of interest, and adjust the meta-feature information that the network model focuses on;

[0021] Regression classification module: used to perform regression prediction on the extracted multi-scale fusion features.

[0022] Preferably, the feature extraction module is a CSwinTransformer module, which integrates global information, local information and contextual information through hierarchical feature representation, cross-window attention mechanism and local enhanced position encoding to perform feature extraction.

[0023] Preferably, the feature extraction module further includes:

[0024] SPD_conv unit: used to rearrange the spatial blocks of the feature map in the input diversity data set to enter the depth dimension feature extraction;

[0025] SPPFCSPC unit: used to improve the receptive field and feature expression ability of the model through multi-scale spatial pyramid pooling and convolution operations.

[0026] Preferably, the feature fusion module is a weighted bidirectional feature pyramid BiFPN structure, which is used to capture features at different scales in a data set, and add a second attention mechanism unit to process feature information to obtain fused features.

[0027] Preferably, the method for processing feature information by adding an attention mechanism unit is:

[0028] Attention = f(g(x), x)

[0029] In the formula, x represents the input feature, f(·) represents the processing of the input feature in combination with attention, g(·) represents the process of processing the input feature and generating attention, and Attention = f(g(x), x) represents the process of processing the input x based on attention g(x).

[0030] Preferably, the regression classification module is a decoupled detect module, which is used to introduce a branch network to process different feature maps separately, improve the IoU loss function to WIoU, and introduce a dynamic focusing mechanism to Instead of general IOU, the anchor frame is evaluated for quality and different gradient gains r = β / δα are assigned to anchor frames with different outliers. β-δ , where α and δ are the set hyper parameters, which make the anchor box gradient gain change dynamically with the bounding box.

[0031] Preferably, an attention mechanism unit can be added to the wide-field reconnaissance image ground small target detection network model to enhance the network's ability to handle objects of different sizes and complexities, and the position where the attention mechanism unit is added is adjusted according to the size and characteristics of the target of interest.

[0032] Preferably, the data set for training the wide-field reconnaissance image ground small target detection network model is a data set constructed by modeling and simulation, public data, satellite or aerial photography data, and modeling and simulation are used to generate target images of different angles, scales, categories, and scenes. The data set also includes easily confused background negative samples.

[0033] Compared with the prior art, the present invention has the following advantages and technical effects:

[0034] (1) The present invention sets a cropping method with a certain degree of overlap for sparsely distributed targets on a wide image to prevent missed detection of targets, and performs secondary maximum suppression when returning to the original image, thereby avoiding repeated detection of targets in multiple slice images, and adaptively preprocessing wide-field reconnaissance images with different image sizes and resolutions, thereby achieving good positioning effect;

[0035] (2) The present invention designs feature extraction, feature fusion and regression classification modules for small target detection, and comprehensively improves the small target detection effect by extracting global features from images, fusion of multi-scale features and other means, and improves the generalization ability and robustness of the model;

[0036] (3) Aiming at the problem of small sample target detection, the present invention not only enriches the data set construction method and data enhancement method, but also adds a coefficient adjustment module based on the attention mechanism. By inputting the small sample data into the coefficient adjustment module, a relatively large amount of base class learning experience can be utilized to help deal with the problem of small sample weak target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1 This is a flow chart of a method for identifying and locating small targets for an airborne wide-field-of-view reconnaissance payload according to an embodiment of the present invention;

[0039] Figure 2 It is a flow chart of detecting small ground target detection network model using wide field reconnaissance image of improved yolov5 according to an embodiment of the present invention;

[0040] Figure 3 Schematic diagram of the feature extraction module structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0041] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] The present invention proposes a method for identifying and locating small targets for airborne wide-field-of-view reconnaissance payloads, comprising:

[0044] Preprocessing the initial wide-field reconnaissance image to generate a target image;

[0045] The target image is input into a wide-field reconnaissance image ground small target detection network model for target detection to obtain a prediction result, and the wide-field reconnaissance image ground small target detection network model is constructed using an improved yolov5 network;

[0046] The prediction results are converted into the initial wide-field reconnaissance image, and non-maximum suppression is performed on the prediction results to obtain the target detection results in the initial wide-field reconnaissance image;

[0047] Based on the target detection results, the mapping relationship between pixel position and geographic location is used to convert the pixel position information in the target detection results into geographic location coordinates, and the target is located based on the geographic location coordinates.

[0048] Specifically, the initial wide-field reconnaissance image is preprocessed, including:

[0049] Read the pixel information and geographic information in the initial wide-field reconnaissance image, calculate the real position through the pixel position according to the map projection relationship, and obtain the mapping relationship between the pixel position and the geographic location;

[0050] Correct the initial wide-field reconnaissance image, filter out various interferences and noises in the image background, and adjust for terrain undulations, lens distortion, and camera tilt;

[0051] The resolution of the initial wide-field reconnaissance image is judged. When the image resolution is not greater than the set value, adaptive scaling is performed. When it is greater than the set value, it is cropped with the set pixel size and adaptive overlap to obtain a slice image.

[0052] The network model for detecting small ground targets in wide-field reconnaissance images includes:

[0053] Data enhancement module: used to perform mixed data enhancement on the input image to obtain a diverse data set;

[0054] Feature extraction module: used to extract features from the diversity data set;

[0055] Feature fusion module: used to fuse the extracted features;

[0056] Coefficient adjustment module: used to generate semantic features of small sample data categories and feature coefficients of regions of interest, and adjust the meta-feature information that the network model focuses on;

[0057] Regression classification module: used to perform regression prediction on the extracted multi-scale fusion features.

[0058] The feature extraction module is a CSwinTransformer module, which integrates global information, local information and context information through hierarchical feature representation, cross-window attention mechanism and local enhanced position encoding to extract features.

[0059] The feature extraction module also includes:

[0060] SPD_conv unit: used to rearrange the spatial blocks of the feature map in the input diversity data set to enter the depth dimension feature extraction;

[0061] SPPFCSPC unit: used to improve the receptive field and feature expression ability of the model through multi-scale spatial pyramid pooling and convolution operations.

[0062] The feature fusion module is a weighted bidirectional feature pyramid BiFPN structure, which is used to capture features at different scales in the data set and add a second attention mechanism unit to process the feature information to obtain the fused features.

[0063] The method of processing feature information by adding attention mechanism units is:

[0064] Attention = f(g(x), x)

[0065] In the formula, x represents the input feature, f(·) represents the processing of the input feature in combination with attention, g(·) represents the process of processing the input feature and generating attention, and Attention = f(g(x), x) represents the process of processing the input x based on attention g(x).

[0066] The regression classification module is a decoupled detect module. The decoupled detect module is used to introduce a branch network to process different feature maps separately, improve the IoU loss function to WIoU, and introduce a dynamic focusing mechanism to Instead of general IOU, the anchor frame is evaluated for quality and different gradient gains r = β / δα are assigned to anchor frames with different outliers. β-δ (where α and δ are set hyperparameters), which reduces the harmful gradients generated by low-quality anchor boxes and makes the anchor box gradient gain change dynamically with the bounding box, thereby improving the overall performance of the detector.

[0067] Furthermore, attention mechanism units can be added to the network model for detecting small ground targets in wide-field reconnaissance images to enhance the network's ability to handle objects of different sizes and complexities. The position where the attention mechanism units are added is adjusted according to the size and features of the target of interest.

[0068] The dataset for training the network model for detecting small ground targets in wide-field reconnaissance images is constructed using modeling and simulation, public data, and satellite or aerial photography data. The modeling and simulation generate target images of different angles, scales, categories, and scenes. The dataset also includes negative samples with easily confused backgrounds.

[0069] In order to more clearly express the technical solution of the present invention, the following specific embodiments are provided to introduce the solution:

[0070] A method for identifying and locating small targets for airborne wide-field-of-view reconnaissance payloads, such as Figure 1 , specifically including:

[0071] Step 1: Preprocess the wide-field reconnaissance image, extract image information and geographic location information, perform color, shadow, and perspective correction on the image, determine the image resolution, and perform adaptive scaling when the image is not larger than 1080P. If the image is larger than 1080P, it will be cropped into slices with the set pixel size and adaptive overlap;

[0072] Step 2: Input the image into the wide-field reconnaissance image ground small target detection network model for target detection, including confidence suppression and IOU suppression. According to the set confidence threshold, the detection frame with a higher value is retained, and the intersection-over-union ratio of two detection frames of the same category is further calculated. The detection frames of the same target are eliminated, thereby obtaining the target detection result in the initial wide-field reconnaissance image;

[0073] Step 3: Convert the detection results of the slice image to the original image, and perform non-maximum suppression on all results to obtain the target detection results in the wide-field reconnaissance image, and at the same time obtain the target pixel position information. Finally, convert the pixel position information in the detection result into geographic location coordinates.

[0074] The network model in step 2 is obtained by improving the ground small target detection method of wide field reconnaissance image of yolov5, as shown in Figure 2 , the method comprising:

[0075] Step 2.1: Dataset construction: For small sample targets, three methods are used to construct data sets: modeling and simulation, public data, and satellite or aerial photography data. Modeling and simulation generate target images of different angles, scales, categories, and scenes; public data sets obtain target images of similar categories and features; and aerial photography obtains real-life images. Adding negative samples with easily confused backgrounds to the data set helps the model learn how to distinguish between targets and backgrounds.

[0076] Step 2.2: Data enhancement: Input a certain batch_size of samples into the network, and first perform mixed data enhancement to further increase the diversity of the data set. Specifically, after each stage of the sample is processed by mosaic, it is mixed up or scaled according to the mixed data enhancement weight parameters λ1 and λ2. The scale operation selects typical 0.75 and 1.25 for scaling to obtain samples of different sizes, while avoiding a large amount of duplicate data and affecting the distribution ratio of different categories of data.

[0077] Step 2.3: Feature extraction: Use a backbone network based on the CSwinTransformer structure to improve yolov5. The specific implementation method is: use the CSwinTransformer module to replace the CSP module in yolov5; add the SPD_conv unit and the SPPFCSPC spatial pyramid pooling unit before the CSwinTransformer module, and add the attention mechanism module after the CSwinTransformer module.

[0078] Step 2.4: Feature fusion: The improved BiFPN bidirectional feature pyramid network is adopted. The bidirectional connection helps to better capture feature representations at different scales, and an attention mechanism module is added to the feature fusion layer to enhance the network’s ability to handle objects of different sizes and complexities.

[0079] The attention mechanism uses the following formula to represent the information processing process:

[0080] Attention=f(g(x),x)

[0081] Where f(·) represents the processing of input features in combination with attention, and g(·) represents the process of processing input features and generating attention.

[0082] Step 2.5: Coefficient adjustment: In particular, a coefficient adjustment network based on the attention mechanism is used to supplement the small sample data, generate the semantic features of the small sample data category and the feature coefficients of the region of interest, and adjust the meta-feature information that the network model focuses on. It should be noted that in this embodiment, the data enhancement of the sample mainly adopts the mixup method to help the model have a linear understanding of the sample.

[0083] Step 2.6: Regression classification: Use the decoupled detect module and introduce a branch network to process different feature maps separately; improve the IoU loss function to WIoU, and introduce a dynamic focus mechanism so that the anchor box gradient gain changes dynamically with the bounding box.

[0084] Furthermore, the attention mechanism module in step 2.3 and step 2.4 preferably uses the GAM global attention mechanism to improve the performance of the network model by reducing information loss and amplifying global interactions. The filling position can be adjusted according to the size and features of the target of interest to improve the model's ability to express image features and better capture key information in the image. A method for adding attention to the backbone network is as follows: Figure 3 .

[0085] In step 2.5, the coefficient adjustment network based on the attention mechanism preferably uses CBAM, and channel attention and spatial attention are introduced into the convolutional neural network to obtain the final attention-enhanced features to obtain more detail information of interest, and then generate the weight coefficients of the features of interest.

[0086] Specifically, in the network model for detecting small ground targets in wide-field reconnaissance images, the backbone network uses the CSwinTransformer module to effectively integrate global information, local information and contextual information through hierarchical feature representation, cross-window attention mechanism and local enhanced position encoding, thereby improving the multi-scale target detection performance; the SPD_conv unit is used to rearrange the spatial blocks of the input feature map into the depth dimension to improve the model's recognition performance for small targets and low-resolution images; the SPPFCSPC unit is used to improve the model's receptive field and feature expression capabilities through multi-scale spatial pyramid pooling and convolution operations; the feature fusion network uses an improved BiFPN network structure. This weighted bidirectional feature pyramid network helps to better capture feature representations at different sizes and improve the network's ability to handle objects of different sizes and complexities; the regression classification uses a decoupled detect module and optimizes the loss function to speed up the network convergence and improve detection accuracy.

[0087] Furthermore, this model can be deployed in a Docker container or embedded in the edge, and can be extended to the application of satellite-to-ground target recognition and positioning.

[0088] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for identifying and locating small targets for airborne wide-field-of-view reconnaissance payloads, characterized in that: include: Preprocessing the initial wide-field reconnaissance image to generate a target image; Inputting the target image into a wide-field reconnaissance image ground small target detection network model for target detection to obtain a prediction result, wherein the wide-field reconnaissance image ground small target detection network model is constructed using an improved yolov5 network; Converting the prediction result into the initial wide-field reconnaissance image, performing non-maximum suppression on the prediction result, and obtaining a target detection result in the initial wide-field reconnaissance image; Based on the target detection result, the pixel position information in the target detection result is converted into the geographical location coordinates by using the mapping relationship between the pixel position and the geographical location, and the target is located based on the geographical location coordinates; The wide-field reconnaissance image ground small target detection network model includes: Data enhancement module: used to perform mixed data enhancement on the input image to obtain a diverse data set; Feature extraction module: used to extract features from the diversity data set; Feature fusion module: used to fuse the extracted features; Coefficient adjustment module: used to generate semantic features of small sample data categories and feature coefficients of regions of interest, and adjust the meta-feature information that the network model focuses on; Regression classification module: used to perform regression prediction on the extracted multi-scale fusion features; The feature extraction module is a CSwinTransformer module, which integrates global information, local information and context information to extract features through hierarchical feature representation, cross-window attention mechanism and local enhanced position encoding; The feature extraction module also includes: SPD_conv unit: used to rearrange the spatial blocks of the feature map in the input diversity data set to enter the depth dimension feature extraction; SPPFCSPC unit: used to improve the receptive field and feature expression ability of the model through multi-scale spatial pyramid pooling and convolution operations; The feature fusion module is a weighted bidirectional feature pyramid BiFPN structure, which is used to capture features at different scales in the data set, and adds a second attention mechanism unit to process the feature information to obtain fused features; The method of processing feature information by adding attention mechanism units is: Attention = f(g(x), x) Where x represents the input feature, f(·) represents the processing of the input feature in combination with attention, g(·) represents the process of processing the input feature and generating attention, and Attention = f(g(x), x) represents the process of processing the input x based on attention g(x); The regression classification module is a decoupled detect module, which is used to introduce a branch network to process different feature maps separately, improve the IoU loss function to WIoU, and introduce a dynamic focusing mechanism to Instead of general IOU, the anchor frame is evaluated for quality and different gradient gains r = β / δα are assigned to anchor frames with different outliers. β-δ , where α and δ are the set hyper parameters, which make the anchor box gradient gain change dynamically with the bounding box.

2. The method for identifying and locating small targets for airborne wide-field-of-view reconnaissance payloads according to claim 1 is characterized in that: Preprocess the initial wide-field reconnaissance image, including: Reading pixel information and geographic information in the initial wide-field reconnaissance image, calculating the real position through the pixel position according to the map projection relationship, and obtaining a mapping relationship between the pixel position and the geographic location; Correcting the initial wide-field reconnaissance image, filtering out various interferences and noises in the image background, and adjusting terrain undulations, lens distortion, and camera tilt; The resolution of the initial wide-field reconnaissance image is determined, and when the image resolution is not greater than a set value, adaptive scaling is performed; when the image resolution is greater than the set value, cropping is performed with a set pixel size and adaptive overlap to obtain a slice image.

3. The method for identifying and locating small targets for airborne wide-field-of-view reconnaissance payloads according to claim 1 is characterized in that: The wide-field reconnaissance image ground small target detection network model can also add attention mechanism units to enhance the network's ability to handle objects of different sizes and complexities, and the position where the attention mechanism units are added is adjusted according to the size and features of the target of interest.

4. The method for identifying and locating small targets for airborne wide-field-of-view reconnaissance payloads according to claim 1, characterized in that: The data set for training the wide-field reconnaissance image ground small target detection network model is a data set constructed by modeling and simulation, public data, satellite or aerial photography data, and modeling and simulation are used to generate target images of different angles, scales, categories, and scenes. The data set also includes negative samples with easily confused backgrounds.

Images