Object detection method for large-scale aerial remote sensing images
Through the combination of the feature pyramid network and global detector of the global local coupling mechanism, the problem of large-size aerial remote sensing images missing context information due to image segmentation is solved, and the target detection accuracy and detection ability of super-large targets are significantly improved.
Patent Information
- Application Number
- CN202111313879.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-08
- Publication Date
- 2025-03-21
- Estimated Expiration
- 2041-11-08
AI Technical Summary
Large-size aerial remote sensing images have lost context information due to image segmentation, resulting in a decrease in target detection accuracy.
A feature pyramid network adopts a global local coupling mechanism, and obtains multi-scale feature maps through local feature extractors and global feature extractors, and performs feature fusion, and combines a global detector to detect super-large targets to solve the problem of missing context information.
It effectively improves the target detection accuracy of large-size aerial remote sensing images, alleviates the problem of missing context information caused by image segmentation, and improves the detection ability of super-large targets.
Smart Images
Figure CN113989645B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a target detection method for large-size aerial remote sensing images, and belongs to the technical field of target detection. Background Art
[0002] With the development of image sensors and aviation technology, the resolution of aerial remote sensing optical images has gradually increased, and the amount of data contained in the images has also increased significantly. Object detection is one of the common technical means of computer vision, which is used to infer the location and category of each target object in the image. In the application field of high-resolution aerial remote sensing images, object detection technology can play an important role in building recognition, natural disaster management, change detection, traffic planning, agricultural surveys, military and other aspects.
[0003] Compared with natural images in computer vision, aerial remote sensing images have larger resolution and size. Many aerial image datasets provide large-size images that cannot be directly read by GPUs. For example, the DOTA dataset contains large-size images with a resolution of 10000×10000. If the image is simply reduced to the size that can be accommodated in the memory, it will result in a significant decrease in accuracy. Therefore, the method commonly adopted by scholars is to divide all original images into sub-images of a fixed size that can be accommodated by the graphics card during data preprocessing, and there is a certain overlap between the sub-images. However, this preprocessing method often leads to two problems that are easily overlooked: one is that the global context information is segmented as the image is segmented; the other is that even if there is a certain overlap between adjacent sub-images, some very large targets may be incomplete on each sub-image and therefore cannot be fully detected. In aerial images, many targets are associated with large scenes, such as airplanes usually appear at airports, while ports only appear at the edge of large areas of water. In the fields of semantic segmentation and medical imaging, these global contexts from large-size images have been proven to be beneficial to the accuracy improvement of neural network models.
[0004] Nowadays, many excellent target detection models applied in the field of natural images have emerged, such as FasterR-CNN, CascadeR-CNN and YOLO. However, these methods only focus on the features of the object's own area, while ignoring the multi-scale and contextual information of the object. The human visual system can use context and more information to support our perceptual inference and judgment. In the field of computer vision, the role of contextual information is also very important. Many methods have shown that multi-scale features and contextual information are conducive to improving the accuracy of target detection. For example, the feature pyramid (FPNs) establishes a top-down pyramid structure, allowing the model to use multi-scale joint features. The Inside-outside network uses a recurrent neural network to capture multi-scale contextual information, significantly improving the detection accuracy, which shows the importance of multi-scale contextual information for target detection. AC-CNN can use attention mechanisms and recurrent neural networks to extract multi-scale contextual information. The MS-CNN model is used to perceive the contextual information of the object. The FPN structure can also use feature maps with a large degree of upsampling to extract surrounding contextual information. FA-SSD can use the attention mechanism to extract more significant contextual information, improving the performance of small target detection.
[0005] For aerial image target detection, context information plays an important role. For example, bridges and roads are very similar in texture information, but bridges often appear on water, so the context around the bridge provides us with a basis for discrimination. Many scholars have designed special context-aware models based on the characteristics of remote sensing images. The context-aware detection network (CAD-Net) uses GC-Net and PLC-Net to perceive context information at the global scene level and local object level respectively. CA-CNN extracts context information from multi-scale features through multiple context anchor boxes of different sizes. GLNet uses a long short-term memory (LSTM) network structure to extract global context information. Although these network structures for extracting context information have achieved good results, they are not specifically targeted at large-size images, so they cannot solve the problem of missing context information caused by image segmentation. Summary of the invention
[0006] Aiming at the problem that context information of large-scale aerial remote sensing images is lost due to image segmentation, the present invention provides a target detection method for large-scale aerial remote sensing images.
[0007] A target detection method for large-scale aerial remote sensing images of the present invention comprises:
[0008] The original remote sensing images form an original remote sensing image set, and each original remote sensing image is configured with an original image name;
[0009] Each original remote sensing image is divided into sub-images of preset sizes according to a predetermined overlap rate, and each sub-image is configured with a sub-image name, wherein the sub-image name includes the original image name, the sub-image sequence number, and the starting position information of the sub-image on the original remote sensing image;
[0010] According to the sub-image name of the current sub-image to be detected, the corresponding current original remote sensing image is retrieved from the original remote sensing image set;
[0011] A local feature extractor is used to extract features of the current sub-image to be detected to obtain a set of multi-scale sub-image feature maps;
[0012] After downsampling the current original remote sensing image, a global feature extractor is used to extract features to obtain a set of multi-scale original image feature maps;
[0013] A feature pyramid network with a global-local coupling mechanism is used to fuse the multi-scale sub-image feature map and the multi-scale original image feature map to obtain the fused feature map.
[0014] The fused feature map is passed through the first RPN to generate candidate boxes, and then through the first RolAlign for feature alignment, and then through the FC fully connected layer to change the feature dimension to obtain a one-dimensional fused feature map. The one-dimensional fused feature map is subjected to classification and regression operations respectively to obtain the fused target prediction result;
[0015] After a set of multi-scale original image feature maps are multi-scale transformed, candidate boxes are generated by the second RPN, which are first filtered and then feature aligned by the second RolAlign. The feature dimension is then changed by the FC fully connected layer to obtain a one-dimensional global feature map. The one-dimensional global feature map is classified and regressed respectively, and then NLS filtered to obtain the filtered global prediction result.
[0016] The fused target prediction result and the filtered global prediction result are fused again to obtain the detected target.
[0017] According to the object detection method of large-scale aerial remote sensing images of the present invention, a set of multi-scale sub-image feature maps output by the local feature extractor includes four sub-image feature maps of different scales;
[0018] A set of multi-scale original image feature maps output by the global feature extractor includes four original image feature maps of different scales;
[0019] The feature pyramid network of the global-local coupling mechanism adopts a top-down interactive fusion approach to transform four sub-image feature maps of different scales and four original image feature maps of different scales into five transformed multi-scale sub-image feature maps and five transformed multi-scale original image feature maps respectively; the five transformed multi-scale sub-image feature maps and the five transformed multi-scale original image feature maps are fused according to the corresponding scales to obtain a multi-scale fused feature map.
[0020] According to the target detection method of large-size aerial remote sensing images of the present invention, the method of fusing five transformed multi-scale sub-image feature maps and five transformed multi-scale original image feature maps according to corresponding scales includes first splicing the five transformed multi-scale sub-image feature maps and the five transformed multi-scale original image feature maps according to corresponding scales, and then performing dimensionality reduction processing.
[0021] According to the target detection method of large-size aerial remote sensing images of the present invention, the process of obtaining the fused feature map includes:
[0022] Performing intermediate level selection on the five transformed multi-scale sub-image feature maps to obtain the selected sub-image feature map;
[0023] Performing intermediate level selection on five transformed multi-scale original image feature maps to obtain a selected original image feature map;
[0024] The specified area in the selected original image feature map corresponding to the selected sub-image feature map is cropped, upsampled to the size of the original image feature map, and then feature fused with the selected sub-image feature map to obtain a fused feature map.
[0025] According to the target detection method for large-size aerial remote sensing images of the present invention, both the local feature extractor and the global feature extractor use ResNet-50 as the feature extractor.
[0026] According to the target detection method of large-scale aerial remote sensing images of the present invention, in the feature pyramid network of the global-local coupling mechanism, a mapping function is used to select the feature map level according to the size of the target candidate box; the mapping function is:
[0027]
[0028] Where k is the level number corresponding to the target candidate box, w is the length of the target candidate box, and h is the width of the target candidate box.
[0029] According to the object detection method of large-size aerial remote sensing images of the present invention, the local feature extractor and the global feature extractor perform feature extraction in the same manner;
[0030] The local feature extractor first uses a convolutional neural network to extract features at 5 depths in the order of gradually increasing feature dimensions for the current sub-image to be detected, and then performs 1*1 convolution calculation on the sub-image features extracted at the last four depths to obtain four sub-image feature maps of different scales.
[0031] According to the object detection method for large-scale aerial remote sensing images of the present invention, the feature pyramid network of the global-local coupling mechanism transforms four sub-image feature maps of different scales into five transformed multi-scale sub-image feature maps in the same way as transforming four original image feature maps of different scales into five transformed multi-scale original image feature maps;
[0032] The four sub-image feature maps of different scales are subjected to 3*3 convolution calculation and maximum pooling respectively to obtain five transformed multi-scale sub-image feature maps.
[0033] According to the target detection method for large-size aerial remote sensing images of the present invention, the predetermined overlap rate includes overlapping 200 pixels.
[0034] According to the target detection method of large-scale aerial remote sensing images of the present invention, the sub-image name format includes:
[0035] Original image name_sub-image sequence number_sub-image starting horizontal coordinate_sub-image starting vertical coordinate.jpg.
[0036] Beneficial effects of the invention: The invention can be used for target detection in aerial images. It alleviates the problem of target failure due to the lack of super-large targets through a fusion model, and can effectively improve the target detection accuracy of large-size aerial remote sensing images.
[0037] The present invention adopts a parallel local feature extractor and a global feature extractor to form a PGL parallel feature extraction module to obtain global image and sub-image information. Based on the feature pyramid FPN structure, a feature pyramid module GL-FPN with a global-local coupling mechanism is designed. By synchronously acquiring information in the sub-image and the original remote sensing image, the loss of contextual information caused by segmenting large-size images is significantly alleviated. In addition, the method of the present invention also designs an additional global detector for detecting extra-large targets on the thumbnail of the global image, further ensuring the reliability of target detection.
[0038] The present invention can form a module and be integrated into a commonly used target detection model, and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a flow chart of the target detection method of large-size aerial remote sensing images according to the present invention;
[0040] Figure 2 It is a schematic diagram of the data processing network of PGL and GL-FPN;
[0041] Figure 3 It is a schematic diagram of feature graph fusion of feature pyramid network with global-local coupling mechanism;
[0042] Figure 4is the true value label of Example 1 from the perspective of the sub-graph;
[0043] Figure 5 This is the baseline model shown in Example 1 from the perspective of a subgraph;
[0044] Figure 6 is a schematic diagram of the detection results of the present invention shown from the perspective of a sub-graph in Example 1;
[0045] Figure 7 is the true value label shown in Example 2 from the perspective of the sub-graph;
[0046] Figure 8 This is the baseline model shown in Example 2 from the perspective of a subgraph;
[0047] Fig. 9 is a schematic diagram of the detection results of the present invention shown from the perspective of a sub-graph in Example 2;
[0048] Fig.10 is the true value label shown in Example 3 from the perspective of the sub-graph;
[0049] Fig.11 This is the baseline model shown in Example 3 from the perspective of subgraphs;
[0050] Fig.12 is a schematic diagram of the detection results of the present invention shown from the perspective of a sub-graph in Example 3;
[0051] Fig.13 is the true value label shown in Example 4 from the perspective of the sub-graph;
[0052] Fig.14 This is the baseline model shown in Example 4 from the perspective of subgraphs;
[0053] Fig.15 is a schematic diagram of the detection results of the present invention shown from the perspective of a sub-graph in Example 4;
[0054] Fig.16 This is the baseline model shown in Example 5 from the original image perspective;
[0055] Fig.17 is a schematic diagram of the detection result of the present invention shown in Example 5 from the perspective of the original image;
[0056] Fig.18 This is the baseline model shown in Example 6 from the original image perspective;
[0057] Fig.19 is a schematic diagram of the detection result of the present invention shown in Example 6 from the perspective of the original image;
[0058] Fig. 20 This is the baseline model shown in Example 7 from the original image perspective;
[0059] Fig.21 is a schematic diagram of the detection result of the present invention shown in Example 7 from the perspective of the original image;
[0060] Fig. 22 This is the baseline model shown in Example 8 from the original image perspective;
[0061] Fig.23 is a schematic diagram of the detection result of the present invention shown in Example 8 from the perspective of the original image;
[0062] Fig.24 This is the baseline model shown in Example 9 from the original image perspective;
[0063] Fig.25 is a schematic diagram of the detection result of the present invention shown in Example 9 from the perspective of the original image;
[0064] Fig.26 This is the baseline model shown in Example 10 from the original image perspective;
[0065] Fig. 27 is a schematic diagram of the detection result of the present invention shown in Example 10 from the perspective of the original image;
[0066] Fig.28 This is the baseline model shown in Example 11 from the perspective of the original image;
[0067] Fig.29 is a schematic diagram of the detection result of the present invention shown in Example 11 from the perspective of the original image;
[0068] Fig.30 This is the baseline model shown in Example 12 from the perspective of the original image;
[0069] Fig.31 This is a schematic diagram of the detection results of the present invention shown in Example 12 from the perspective of the original image. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0071] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.
[0072] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.
[0073] Specific implementation method 1. Combination Figures 1 to 3 As shown, the present invention provides a method for detecting targets in large-scale aerial remote sensing images, which is characterized by comprising:
[0074] The original remote sensing images form an original remote sensing image set, and each original remote sensing image is configured with an original image name;
[0075] Each original remote sensing image is divided into sub-images of preset sizes according to a predetermined overlap rate, and each sub-image is configured with a sub-image name, wherein the sub-image name includes the original image name, the sub-image sequence number, and the starting position information of the sub-image on the original remote sensing image;
[0076] According to the sub-image name of the current sub-image to be detected, the corresponding current original remote sensing image is retrieved from the original remote sensing image set;
[0077] A local feature extractor is used to extract features of the current sub-image to be detected to obtain a set of multi-scale sub-image feature maps;
[0078] After downsampling the current original remote sensing image, a global feature extractor is used to extract features to obtain a set of multi-scale original image feature maps;
[0079] A feature pyramid network with a global-local coupling mechanism is used to fuse the multi-scale sub-image feature map and the multi-scale original image feature map to obtain the fused feature map.
[0080] The fused feature map is passed through the first RPN to generate candidate boxes, and then through the first RolAlign for feature alignment, and then through the FC fully connected layer to change the feature dimension to obtain a one-dimensional fused feature map. The one-dimensional fused feature map is subjected to classification and regression operations respectively to obtain the fused target prediction result;
[0081] After a set of multi-scale original image feature maps are multi-scale transformed, candidate boxes are generated by the second RPN, which are first filtered and then feature aligned by the second RolAlign. The feature dimension is then changed by the FC fully connected layer to obtain a one-dimensional global feature map. The one-dimensional global feature map is classified and regressed respectively, and then NLS filtered to obtain the filtered global prediction result.
[0082] The fused target prediction result and the filtered global prediction result are fused again to obtain the detected target.
[0083] In this embodiment, the local feature extractor and the global feature extractor constitute PGL (Parallel Global and Local Feature Extraction), which can extract the information of the sub-image and the corresponding global image respectively; then the two parts of the multi-scale features are deeply fused through the coupling of the feature pyramid network GL-FPN (global and local Feature Pyramid Networks) of the global-local coupling mechanism. In addition, the subsequent RPN, RolAlign, and FC fully connected layers form a detection branch with very small computational complexity, which can detect very large targets that are difficult to detect. The present invention is formed into an integral module and is easily integrated on most mainstream detectors.
[0084] In this embodiment, the original remote sensing images are all global images with a relatively large size, for example, a size of 10000*10000 pixels. In the deep learning model, such an image size cannot be directly input into the network, so it needs to be preprocessed and cut into small images according to a fixed size. The original remote sensing image can be cut into multiple sub-images according to a size of, for example, 1024*1024, and a certain overlap rate is guaranteed between different sub-images during cutting.
[0085] Further, combined with Figure 1 and Figure 2 As shown, a set of multi-scale sub-image feature maps output by the local feature extractor includes four sub-image feature maps of different scales;
[0086] A set of multi-scale original image feature maps output by the global feature extractor includes four original image feature maps of different scales;
[0087] The feature pyramid network of the global-local coupling mechanism adopts a top-down interactive fusion method to transform four sub-image feature maps of different scales and four original image feature maps of different scales into five transformed multi-scale sub-image feature maps and five transformed multi-scale original image feature maps, corresponding to Figure 2 P2-P6 on both sides of the middle; the five transformed multi-scale sub-image feature maps and the five transformed multi-scale original image feature maps are fused according to the corresponding scales to obtain a multi-scale fused feature map.
[0088] Furthermore, the method of fusing the five transformed multi-scale sub-image feature maps and the five transformed multi-scale original image feature maps according to corresponding scales includes first splicing the five transformed multi-scale sub-image feature maps and the five transformed multi-scale original image feature maps according to corresponding scales, and then performing dimensionality reduction processing.
[0089] Going a step further, combined Figure 3 As shown, the process of obtaining the fused feature map includes:
[0090] Performing intermediate level selection on the five transformed multi-scale sub-image feature maps to obtain the selected sub-image feature map;
[0091] Performing intermediate level selection on five transformed multi-scale original image feature maps to obtain a selected original image feature map;
[0092] The specified area in the selected original image feature map corresponding to the selected sub-image feature map is cropped, upsampled to the size of the original image feature map, and then feature fused with the selected sub-image feature map to obtain a fused feature map.
[0093] Figure 2 C1-C5 and M2-M5 correspond to the operation process of the local feature extractor and the global feature extractor. Figure 1 In the NLS filter, the confidence score of each detection result is transformed.
[0094] C1-C5 refers to the features of the image at different depths when it passes through the convolutional neural network. They are all feature maps, that is, feature vectors in the neural network. The feature dimension increases gradually from C1 to C5. Figure 2 As shown, it is calculated by 1*1 convolution, and P2-P6 are obtained by 3*3 convolution and pooling layer.
[0095] As an example, both the local feature extractor and the global feature extractor use ResNet-50 as the feature extractor.
[0096] Currently, the commonly used target detection methods divide the training data in the preprocessing step, and the corresponding original images will not appear in the model training process. Figure 1 As shown in the figure, in order to extract the global context information, we first establish an index structure of sub-images and the corresponding original images. According to the name of the sub-image, we can easily find the corresponding original image and zoom to the sub-image. Figure 1 The two images are of the same size and then used as the input of the convolutional model. The parallel backbone network formed by the local feature extractor and the global feature extractor has a structure similar to that of the twin network, but the method of the present invention does not include a weight sharing mechanism similar to that of the twin network. Instead, ResNet-50 is used as the feature extractor, which effectively solves the gradient vanishing problem and has a strong feature extraction capability. The parallel feature extractor will finally output two sets of features of the same size.
[0097] The classic feature pyramid network FPN is a top-down pyramid architecture that generates multi-scale features through mutual connections. It can effectively improve the model's target detection performance on aerial images. In order to more effectively provide multi-scale information and capture global context, this embodiment designs a global and local coupled pyramid model based on the pyramid network. At the same time, this embodiment notes that the coupling mechanism is sensitive to scale, and the effects of different scales vary greatly. Therefore, after extensive experiments, the rules are summarized and the optimal scale is selected. The feature pyramid FPN is a module with multiple inputs and multiple outputs. It uses a mapping function to assign feature levels to the candidate box according to its size. The mapping function is described as follows:
[0098]
[0099] Where k0 is the initial level, which is defined as 4 in the FPN structure of Faster R-CNN. w and h are the length and width of each candidate box. k corresponds to the level number assigned to each candidate box, combined with Figure 2 As shown, it is assigned to one of {P2, P3, P4, P5}. This allocation strategy is further improved in the ResNet network structure, resulting in five levels of feature maps, and the mapping function is described as follows:
[0100] Furthermore, in the feature pyramid network of the global-local coupling mechanism, a mapping function is used to select the feature map level according to the size of the target candidate box; the mapping function is:
[0101]
[0102] Where k is the level number corresponding to the target candidate box, w is the length of the target candidate box, and h is the width of the target candidate box.
[0103] Figure 2 As shown in Figure 1, the feature pyramid module of the global-local coupling mechanism fuses the features from the sub-image with the features from the original image and then outputs them to the RCNN part of the model. The features from the original image are fused with the features from the sub-image and are also directly output to the additional global detector part of the model. The two sets of outputs of the parallel feature extractor are named and They contain four feature maps of different scales. i Represents the number of channels, H i and W i Represent the height and width of each scale feature map respectively. After the FPN module is calculated, we get and At this time, each group contains five feature maps of different levels, each of which contains information of other scales. The number of channels between different levels is the same, but the size is different. Figure 3 The fusion process of the present invention is introduced, from the feature set from the global image Take out a specific level Then, according to the position of the sub-image on the global image at this time, The feature map of the corresponding position is cut out and expressed as In order to correspond In this embodiment, Upsample to same size. Then and Stitched together along the channel is represented as LG 2C×H×W Finally, LG is transformed into 2C×H×W Convert to LG C×H×W .
[0104] The main function of GL-FPN is to perform multi-scale transformation and fusion on the two sets of feature maps obtained in the previous part, so that the sub-map can contain information from the original global image. Figure 2 For a set of feature sub-graphs and a set of feature global graphs, a set of features are interactively fused in a top-down manner, and the four feature graphs in each group are transformed into five more representative feature graphs, thereby obtaining two sets of multi-scale feature graphs. Feature fusion is to fuse the corresponding scales of the two sets of feature graphs. When fusion, it is necessary to consider the position information of the sub-graphs in the global image, crop the corresponding positions, and then upsample to the same size. Then the two sets of feature graphs are fused, and the fusion method is to first splice and then reduce the dimension. The two sets of feature graphs that have undergone fusion are output according to the input size and size.
[0105] After being fused, the features from the global graph and the sub-graph are sent to different detection modules. RPN is used to generate candidate boxes, ROIAlign is used for feature alignment, and the FC fully connected layer is used to change the feature dimension. The classification layer and regression layer are then used to identify the type and accurately locate the target. Thus, the target detection results of the sub-graph and the target detection results of the global graph are obtained respectively.
[0106] Finally, the target detection results from the sub-graph are organically integrated with the target detection results from the global graph. The detection results from the global graph are screened using filters and the NLS confidence limit algorithm, and finally the optimal detection results are obtained.
[0107] In the present invention, features of all levels are not fused, because YOLOF and AugFPN propose that feature maps of different levels have different characteristics. YOLOF
[40] uses only one level of input and output and achieves almost the same performance as all input and output after expansion. Due to the difference in scale, different FPN feature levels have different focuses on deep semantic information and shallow detail information. This leads to the assumption that different levels in FPN respond differently to the coupling mechanism of the present invention. In order to further verify the impact of different levels of features on the results, a set of ablation experiments can be designed.
[0108] Going a step further, combined Figure 2 As shown, the local feature extractor performs feature extraction in the same way as the global feature extractor;
[0109] The local feature extractor first uses a convolutional neural network to extract features at 5 depths in the order of gradually increasing feature dimensions for the current sub-image to be detected, and then performs 1*1 convolution calculation on the sub-image features extracted at the last four depths to obtain four sub-image feature maps of different scales.
[0110] Going a step further, combined Figure 2 As shown, the feature pyramid network of the global-local coupling mechanism transforms four sub-image feature maps of different scales into five transformed multi-scale sub-image feature maps in the same way as it transforms four original image feature maps of different scales into five transformed multi-scale original image feature maps;
[0111] The four sub-image feature maps of different scales are subjected to 3*3 convolution calculation and maximum pooling respectively to obtain five transformed multi-scale sub-image feature maps.
[0112] Models based on convolutional neural networks usually use 1×1 convolution operations to reduce dimensions. The FPN model adds 3×3 convolutions during the feature fusion process to alleviate the overlapping effect caused by upsampling. In the model of the present invention, the global feature map is also formed by upsampling, and the correlation between adjacent pixels is strong. However, the upsampling ratio of the method of the present invention is related to the size of the original image, while the feature pyramid FPN is a standard double upsampling. On this basis, a set of ablation experiments are designed to explore how the size of the dimensionality reduction convolution kernel affects the coupling mechanism. The following cases are discussed respectively: {1×1, 3×3, 5×5, N×N}. The experimental results are shown in Table 8. The 3*3 convolution kernel can stably achieve the optimal effect, because the size of 3*3 can effectively alleviate the overlapping effect and avoid excessive interference. N refers to a group of grouped convolutions determined by the upsampling ratio, and the specific allocation standard is as follows:
[0113]
[0114] When cutting large-size aerial images, some objects with large width or height, such as airports and large ships, may be cut into several parts, resulting in reduced detection accuracy. Although the commonly used preprocessing methods all maintain a certain overlap rate, a too large overlap rate will lead to too many repeated detections, and a too small overlap rate has a limited effect. Therefore, this embodiment designs an additional detector to detect very large targets from the thumbnail of the original image. Through some control algorithms, it can effectively reduce the amount of calculation and avoid affecting the accuracy of the main detector.
[0115] Figure 1 The structure of the global detector is included in it, which is similar to the two-stage target detection model, including RPN (candidate box generation network), classifier and regressor. In order to reduce the amount of calculation and specialize in detecting very large targets, the global detector deletes those smaller candidate boxes through the threshold function, which is described as follows:
[0116]
[0117] Among them, height and width are the width and height of the candidate box. The threshold is 200 because the overlap rate of the conventional segmentation preprocessing step is 200. It is believed that the probability of objects exceeding this length being segmented is relatively high. When the flag is 0, it means that the candidate box will be discarded. However, if an object has the same confidence in the output of the main detector and the global detector, the detection result from the global detector should be abandoned because the main detector has a higher positioning accuracy. Therefore, when the predicted boxes of the two detectors have a high degree of overlap, the results from the global detector should be appropriately suppressed.
[0118] The present invention designs an algorithm named non-local suppression (NLS) based on the idea of soft-NMS, which can effectively suppress the confidence from the global detector according to the overlap rate.
[0119]
[0120]
[0121] In the NLS algorithm, g i and i They are the prediction boxes and their confidences from the global detector, L refers to all the prediction boxes from the main detector, NLS linearly reduces the confidence of repeated detection results, the more repeated areas, the greater the degree of suppression. Because the NLS algorithm does not require mutation suppression in extreme cases, there is no need to use a Gaussian penalty function. x is the horizontal coordinate of the image, and y is the vertical coordinate of the image.
[0122] Finally, during the training process, the loss function of this model can be defined as follows:
[0123]
[0124] and represent the classification and regression losses from the main detector, respectively, and Represents the classification and regression losses from the global detector. c and t l represents the prediction result from the main detector, t' c and t l ' represents the prediction result from the global detector. λ represents the balance hyperparameter between classification and regression loss and can be set to the same value as FasterR-CNN.
[0125] As an example, the predetermined overlap ratio includes overlapping 200 pixels.
[0126] As an example, the subgraph name format includes:
[0127] Original image name_sub-image sequence number_sub-image starting horizontal coordinate_sub-image starting vertical coordinate.jpg.
[0128] Experimental verification:
[0129] In order to evaluate the method proposed in the present invention, extensive experiments were conducted on the DOTA-v1.0 dataset, the DOTA-v1.5 dataset and the DOTA-v2.0 dataset. The experimental results show that the global local coupling detection mechanism (CGLNet) of the method of the present invention can effectively improve the performance of the model in the above three datasets. The results are shown in Tables 1 to 3, and all results are obtained from the official evaluation server. The mAP accuracy index of Faster R-CNN with CGLNet added is improved by 2.45% on DOTA-v1.0, 1.56% on DOTA-v1.5, and 1.67% on DOTA-v2. The Cascade R-CNN with CGLNet added is improved by 1.96% on DOTA-v1.0, 1.76% on DOTA-v1.5, and 1.07% on DOTA-v2.0. In addition, the integrated module using the method of the present invention has improved the recognition accuracy of most categories of images, which shows the importance of global context information for object detection. Among them, the more notable ones are that the detection accuracy of ground-track-field is improved by 9.15%, the detection accuracy of soccer-ball-field is improved by 16.20%, and the detection accuracy of container-crane is improved by 8.65%.
[0130] In addition, the performance is compared with two other types of target detection models, one is a benchmark detection model widely used in the field of natural images, and the other is a special detection model used in the field of aerial remote sensing images. The comparison results are shown in Tables 4 and 5. The first category includes SSD, YOLOv2, RetinaNet, YOLOv3, YOLOv4, R-FCN, Mask R-CNN, PANet, Faster R-CNN, Cascade R-CNN, Faster R-CNN H-OBB; the second category includes RICNN, ORConv, USB-BBR, OFIC, MS-VANs, GLNet, ICN, IAD R-CNN, CDD-Net, CAD-Net, SCANet. Compared with these existing methods, the method of the present invention can achieve better performance on the basis of the benchmark model, and the improvement of the present invention is not limited to specific categories, but can improve the detection accuracy of most categories. Categories with larger average target sizes have greater accuracy gains under the method of the present invention because the method of the present invention focuses more on large-scale global context information. In order to more clearly show the improvement in detection performance achieved by the method of the present invention, the detection results of the present invention are shown from the perspectives of sub-images and original images, as shown in FIG. Figures 4 to 31 shown.
[0131] In order to more rigorously explore the impact of each module and structure of the present invention on the detection results, a detailed ablation experiment is designed and carried out in this embodiment to analyze the impact of a single module and the impact of the structural hyperparameters of the coupling mechanism. In order to analyze the importance of each submodule in CGLNet, GL-FPN and the global detector are applied to the benchmark model to verify their effectiveness on three datasets. The benchmark model adopts Faster R-CNN of ResNet50-FPN. Table 6 shows the results of the ablation study of each component. Each module improves the performance of the model on the three datasets, and their combination also achieves better results, proving that there is no conflict between these submodules. In the detection results of these three datasets, GL-FPN greatly improves the performance on DOTA-v1.0, DOTA-v1.5 and DOTA-v2.0 by 2.04%, 1.18% and 1.47% respectively. The biggest improvement of DOTA-v1.0 is that it contains the least objects in the three datasets. If GL-FPN is removed and the global detector is added separately, the detection performance will be steadily improved by more than 0.2%. The performance improvement of the global head module remains stable on the three datasets, proving that the phenomenon of oversized objects being segmented is widespread in multiple datasets.
[0132] Since scale is sensitive to global context information, global context does not achieve better results for objects of all scales. To further explore this issue, additional ablation experiments can be performed, as shown in Table 7. The features of {P2, P3} have lower downsampling ratios and are rich in detail information. The context features that focus on global information will affect the feature expression, so these two levels are not emphasized when combining feature levels. However, different combinations lead to slight differences in performance, and only {P4} can achieve the best results on all three datasets. This is consistent with other views that the semantic information and details contained in the middle level of the FPN pyramid network are more balanced, so this level is more conducive to the fusion of global information. Using {P2} and {P3} can still produce better results than the baseline only on DOTA-v1.0, but the detection performance on DOTA-v1.5 will be reduced because the global features affect the expression of detail features. The accuracy change trends on the three datasets are similar, which also proves the stability of the method of the present invention. Specific embodiment:
[0134] mAP is used as the evaluation indicator.
[0135] In order to verify the performance of the present invention on large-scale aerial images, the DOTA
[17] dataset is used. This is a large-scale dataset containing aerial images for object detection. It contains a large number of large-size images, mainly from Google Earth, satellite and aerial images. The DOTA dataset has three versions: DOTA-v1.0, DOTA-v1.5 and DOTA-v2.0. The annotation box of HBB is adopted.
[0136] DOTA-v1.0 has a total of 2806 images, ranging in size from 800 to 4000 pixels, including 15 categories and 188282 objects. DOTA-v1.5 annotates many small objects that are difficult to detect, and in addition, adds a new category called "container crane". DOTA-v2.0 contains 402089 instances. DOTA-v2.0 is very different from previous versions, with a total of 18 categories ("airport" and "helipad" were added), 11286 images, and 1793658 objects. In the experiments, the results are all test set accuracy.
[0137] For convenience, MMDetection
[43] is used to implement and evaluate the present invention in order to test the performance of the module on the official benchmark. According to the settings of the official benchmark, the entire original image is converted into a 1024×1024 local image with a stride of 824. In the inference stage, the inference results from the local image are merged into the results of the global image by setting the NMS threshold to 0.3. All experiments are performed on the Ubuntu operating system, trained using a GPU graphics card (NVIDIA GeForce GTX1080Ti), and the total batch size is 1. The initial learning rate is set to 0.0025, and the learning rate adjustment schedule is set to the same as the "1x" schedule. The remaining hyperparameters are set to be consistent with those of the official benchmark, for example, the maximum number of candidate boxes and objects in each image is set to 2000.
[0138] Table 1 Performance comparison between DOTA-v1.0 and baseline models
[0139]
[0140] FR-H = Faster R-CNN model, CR-H = Cascade MaskR-CNN model, P = airplane, BD = baseball field, B = bridge, GTF = sports field runway, SV = small vehicle, LV = large vehicle, S = ship, TC = tennis court, BC = basketball court, ST = oil tank, SBF = football field, RA = roundabout, H = port, SP = swimming pool, HC = helicopter.
[0141] Table 2 Performance comparison between DOTA-v1.5 and baseline models
[0142]
[0143]
[0144] FR-H = Faster R-CNN model, CR-H = Cascade Mask R-CNN model, P = airplane, BD = baseball field, B = bridge, GTF = sports field track, SV = small vehicle, LV = large vehicle, S = ship, TC = tennis court, BC = basketball court, ST = oil tank, SBF = football field, RA = roundabout, H = port, SP = swimming pool, HC = helicopter, CC = crane boom.
[0145] Table 3 Performance comparison between DOTA-v2.0 and baseline models
[0146]
[0147] FR-H = Faster R-CNN model, CR-H = Cascade Mask R-CNN model, P = airplane, BD = baseball field, B = bridge, GTF = sports field runway, SV = small vehicle, LV = large vehicle, S = ship, TC = tennis court, BC = basketball court, ST = oil tank, SBF = football field, RA = roundabout, H = port, SP = swimming pool, HC = helicopter, CC = crane boom, A = airport, He = helipad.
[0148] Table 4 Performance comparison between DOTA-v1.0 and mainstream models
[0149]
[0150] FR-H = Faster R-CNN model, CR-H = CascadeMaskR-CNN model. The feature extraction structure used in the FR-H* model is different from that in FR-H.
[0151] Table 5 Performance comparison between DOTA-v1.5 and mainstream models
[0152]
[0153]
[0154] FR-H = Faster R-CNN model, CR-H = Cascade Mask R-CNN model.
[0155] Table 6 Ablation experiment of neutron module of the present invention
[0156]
[0157] Table 7 Ablation experiment of fusion level of GL-FPN
[0158]
[0159] Table 8 Ablation experiment of dimensionality reduction convolution kernel size of GL-FPN
[0160]
[0161] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.
Claims
1. A method for detecting targets in large-scale aerial remote sensing images, characterized in that include, The original remote sensing images form an original remote sensing image set, and each original remote sensing image is configured with an original image name; Each original remote sensing image is divided into sub-images of preset sizes according to a predetermined overlap rate, and each sub-image is configured with a sub-image name, wherein the sub-image name includes the original image name, the sub-image sequence number, and the starting position information of the sub-image on the original remote sensing image; According to the sub-image name of the current sub-image to be detected, the corresponding current original remote sensing image is retrieved from the original remote sensing image set; A local feature extractor is used to extract features of the current sub-image to be detected to obtain a set of multi-scale sub-image feature maps; After downsampling the current original remote sensing image, a global feature extractor is used to extract features to obtain a set of multi-scale original image feature maps; A feature pyramid network with a global-local coupling mechanism is used to fuse the multi-scale sub-image feature map and the multi-scale original image feature map to obtain the fused feature map. The fused feature map is passed through the first RPN to generate candidate boxes, and then through the first RolAlign for feature alignment, and then through the FC fully connected layer to change the feature dimension to obtain a one-dimensional fused feature map. The one-dimensional fused feature map is subjected to classification and regression operations respectively to obtain the fused target prediction result; After a set of multi-scale original image feature maps are multi-scale transformed, candidate boxes are generated by the second RPN, which are first filtered and then feature aligned by the second RolAlign. The feature dimension is then changed by the FC fully connected layer to obtain a one-dimensional global feature map. The one-dimensional global feature map is classified and regressed respectively, and then NLS filtered to obtain the filtered global prediction result. The fused target prediction result and the filtered global prediction result are fused again to obtain the detected target.
2. The target detection method for large-scale aerial remote sensing images according to claim 1 is characterized in that: A set of multi-scale sub-image feature maps output by the local feature extractor includes four sub-image feature maps of different scales; A set of multi-scale original image feature maps output by the global feature extractor includes four original image feature maps of different scales; The feature pyramid network of the global-local coupling mechanism adopts a top-down interactive fusion approach to transform four sub-image feature maps of different scales and four original image feature maps of different scales into five transformed multi-scale sub-image feature maps and five transformed multi-scale original image feature maps respectively; the five transformed multi-scale sub-image feature maps and the five transformed multi-scale original image feature maps are fused according to the corresponding scales to obtain a multi-scale fused feature map.
3. The target detection method for large-scale aerial remote sensing images according to claim 2 is characterized in that: The method for fusing five transformed multi-scale sub-image feature maps and five transformed multi-scale original image feature maps according to corresponding scales includes first splicing the five transformed multi-scale sub-image feature maps and the five transformed multi-scale original image feature maps according to corresponding scales, and then performing dimensionality reduction processing.
4. The target detection method for large-scale aerial remote sensing images according to claim 3 is characterized in that: The process of obtaining the fused feature map includes: Performing intermediate level selection on the five transformed multi-scale sub-image feature maps to obtain the selected sub-image feature map; Performing intermediate level selection on five transformed multi-scale original image feature maps to obtain a selected original image feature map; The specified area in the selected original image feature map corresponding to the selected sub-image feature map is cropped, upsampled to the size of the original image feature map, and then feature fused with the selected sub-image feature map to obtain a fused feature map.
5. The target detection method for large-scale aerial remote sensing images according to claim 4 is characterized in that: Both the local feature extractor and the global feature extractor use ResNet-50 as the feature extractor.
6. The target detection method for large-scale aerial remote sensing images according to claim 5, characterized in that: In the feature pyramid network of the global-local coupling mechanism, a mapping function is used to select the feature map level according to the size of the target candidate box; the mapping function is: Where k is the level number corresponding to the target candidate box, w is the length of the target candidate box, and h is the width of the target candidate box.
7. The target detection method for large-scale aerial remote sensing images according to claim 6, characterized in that: The local feature extractor performs feature extraction in the same way as the global feature extractor; The local feature extractor first uses a convolutional neural network to extract features at 5 depths in the order of gradually increasing feature dimensions for the current sub-image to be detected, and then performs 1*1 convolution calculation on the sub-image features extracted at the last four depths to obtain four sub-image feature maps of different scales.
8. The target detection method for large-scale aerial remote sensing images according to claim 7 is characterized in that: The feature pyramid network of the global-local coupling mechanism transforms four sub-image feature maps of different scales into five transformed multi-scale sub-image feature maps in the same way as it transforms four original image feature maps of different scales into five transformed multi-scale original image feature maps; The four sub-image feature maps of different scales are subjected to 3*3 convolution calculation and maximum pooling respectively to obtain five transformed multi-scale sub-image feature maps.
9. The target detection method of large-scale aerial remote sensing images according to claim 1, characterized in that: The predetermined overlap ratio includes overlapping by 200 pixels.
10. The target detection method of large-scale aerial remote sensing images according to claim 1, characterized in that: The subgraph name format includes: Original image name_sub-image sequence number_sub-image starting horizontal coordinate_sub-image starting vertical coordinate.jpg.