A small target detection system and method based on remote sensing images

Abstract

This invention discloses a small target detection system based on remote sensing images, comprising a feature extraction unit, an edge information fusion unit, a coarse processing unit, and a detection unit. The feature extraction unit acquires features at different scales in the remote sensing image, including low-level and high-level features. The edge information fusion unit fuses low-level features and outputs fused edge features. The coarse processing unit combines high-level features and fused edge features to perform a first classification and a first regression operation, outputting a pre-detection result. The detection unit performs a second classification and a second regression operation on the pre-detection result to generate a final detection result. A detection method based on the above detection system is also disclosed. This invention can detect small targets in remote sensing images through the detection system, effectively fuse edge features, and utilize multi-layer features in the feature pyramid to select anchor points to locate small targets, thereby effectively obtaining a final detection result related to the small targets.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and digital image processing technology, and in particular to a small target detection system and method based on remote sensing images. Background Technology

[0002] Object detection is a popular area of ​​computer vision and digital image processing, playing a crucial role in numerous fields such as robot navigation, intelligent video surveillance, and industrial inspection. With advancements in satellite launch technology and the continuous development of space sensors, remote sensing technology has made remarkable progress. Object detection also plays a vital role in processing large-scale optical remote sensing images, extending its influence to the field of space observation and finding wide applications in environmental monitoring, resource exploration, marine monitoring, precision agriculture, and urban planning. Due to its significant scientific research value and broad application prospects, this research direction has received considerable attention. Generally, the purpose of remote sensing object detection is to automatically identify and locate objects of interest, such as buildings and vehicles, in remote sensing images. With the rapid development of convolutional neural networks (CNNs), the performance of object detection in natural images has significantly improved in recent years, further driving the development of object detection in remote sensing images.

[0003] Ground-based cameras capture natural images from a horizontal or landscape perspective, while remote-sensing images are captured from an overhead view by satellites or high-altitude airborne cameras. Due to this unique imaging perspective, target detection in remote-sensing images presents a greater challenge. This is because remote-sensing images are characterized by complex background interference, dense and cluttered instances, and large-scale variations, making it difficult for remote-sensing target detection models to detect small targets. First, insufficient resolution becomes a problem because these targets may only occupy a limited number of pixels; second, complex backgrounds and occlusions can make small targets difficult to distinguish. Summary of the Invention

[0004] The main problem solved by this invention is to provide a small target detection system and method based on remote sensing images, which solves the problem that existing technologies have difficulty in detecting and distinguishing small targets in remote sensing images.

[0005] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a small target detection system based on remote sensing images, including a feature extraction unit, an edge information fusion unit, a coarse processing unit, and a detection unit;

[0006] The feature extraction unit is used to acquire features at different scales in the remote sensing image, including low-level features and high-level features.

[0007] The edge information fusion unit is used to fuse the low-level features and output edge fusion features;

[0008] The coarse processing unit is used to select anchor points by combining high-level features and edge fusion features, perform first classification and first regression operations on the anchor points, and output pre-detection results;

[0009] The detection unit is used to perform a second classification and a second regression operation on the pre-detection results to generate the final detection results.

[0010] In some embodiments, the edge information fusion unit includes five fixed-shape convolutional layers, two deformable convolutional layers, two connection modules, a weighted switcher, and a cascade module;

[0011] The first fixed-shape convolutional layer is used to receive the first low-level features, and the first convolutional features are obtained after convolution processing.

[0012] The second fixed-shape convolutional layer is used to receive the second low-level features, and the second convolutional features are obtained after convolution processing.

[0013] The first deformable convolutional layer is used to capture the first edge feature in the first convolutional feature and input it into the weighted switcher;

[0014] The second deformable convolutional layer is used to capture the second edge feature in the second convolutional feature, which is then input into the weighted switcher.

[0015] The weighted switcher is used to multiply the first edge feature and the second edge feature respectively and perform a convolution operation, and select the third edge feature and the fourth edge feature with discriminative output.

[0016] The first connection module is used to connect the first convolutional feature and the third edge feature residual to output the first connection feature;

[0017] The second connection module is used to connect the second convolutional feature and the fourth edge feature residual to output the second connection feature;

[0018] The third fixed-shape convolutional layer is used to receive the first connection feature, and the third convolutional feature is obtained after convolution processing.

[0019] The fourth fixed-shape convolutional layer is used to receive the second connection feature, and the fourth convolutional feature is obtained after convolution processing;

[0020] The cascading module is used to cascade the third convolutional feature and the fourth convolutional feature to output a cascaded feature.

[0021] The fifth fixed-shape convolutional layer is used to receive the cascaded features, and the edge fusion features are obtained after convolution processing.

[0022] In some embodiments, the coarse processing unit includes a progressive region proposal network, which includes an anchor selection module that performs a first-stage pre-regression operation, and an adaptive convolution module and an RPN network that perform a second-stage classification regression operation.

[0023] The anchor point selection module is used to select anchor points based on the edge blending features and the advanced features;

[0024] The adaptive convolution module is used to correct the anchor points, and inputs the corrected anchor points, the edge fusion features, and the high-level features into the RPN network;

[0025] The RPN network is used to perform a first classification and a first regression operation on the corrected anchor points, the edge fusion features, and the high-level features to obtain the pre-detection results.

[0026] In some embodiments, the feature extraction unit includes a CSPDarknet53 network for extracting four feature layers from the remote sensing image from low to high level, to obtain a first low-level feature, a second low-level feature, a first high-level feature, and a second high-level feature.

[0027] In some embodiments, the detection unit includes a normalization layer, a fully connected layer, and a classification and regression layer connected in sequence;

[0028] The normalization layer is used to receive the pre-detection results, normalize the pre-detection results, and obtain the normalized results.

[0029] The fully connected layer is used to map the normalization result to the mapping result;

[0030] The classification and regression layer is used to perform a second classification and a second regression based on the mapping result, using bounding boxes to show the target and indicate the target category, and obtain the final detection result.

[0031] In some embodiments, the anchor point selection module uses a template to dynamically select anchor points, where the anchor point is a bounding box with an IOU greater than the template, and the template is:

[0032]

[0033] Where w and h are the width and height of the target box, respectively, γ represents the scaling factor, c represents the minimum feature map whose height and width are both c, and IOU is the ratio of the area of ​​the intersection and the area of ​​the union of the bounding box and the target box.

[0034] In some embodiments, in the template, γ = 0.15, c = 12.

[0035] In some embodiments, the coarse processing unit employs a fusion loss function, which is:

[0036]

[0037] Among them, L cls This represents the cross-entropy loss corresponding to the pre-regression operation in the first stage. and α1 and α2 are the IOU losses corresponding to the first-stage pre-regression operation and the second-stage classification regression operation, respectively; α1 and α2 are the loss weights, and their sum is 1.

[0038] The present invention also provides a method for small target detection based on remote sensing images, comprising the following steps:

[0039] Acquire features at different scales in remote sensing images, including low-level features and high-level features;

[0040] By fusing the low-level features, the edge fusion feature is output.

[0041] Anchor points are selected by combining the advanced features and the edge fusion features, and the anchor points are subjected to a first classification and a first regression to output the pre-detection results;

[0042] The pre-detection results are then subjected to a second classification and a second regression to generate the final detection results.

[0043] In some embodiments, the step of fusing the low-level features and outputting edge-fused features includes:

[0044] Convolve the first low-level features and the second low-level features to obtain the first convolutional features and the second convolutional features;

[0045] The first convolutional feature and the second convolutional feature are subjected to deformable convolution to obtain the first edge feature and the second edge feature;

[0046] The first edge feature and the second edge feature are multiplied together and then convolved to select and output the third edge feature and the fourth edge feature.

[0047] The third and fourth edge features are cascaded and fused to obtain edge fusion features.

[0048] The beneficial effects of this invention are as follows: This invention discloses a small target detection system based on remote sensing images, including a feature extraction unit, an edge information fusion unit, a coarse processing unit, and a detection unit. The feature extraction unit is used to acquire features at different scales in the remote sensing image, including low-level features and high-level features. The edge information fusion unit is used to fuse low-level features and output edge-fused features. The coarse processing unit is used to combine high-level features and edge-fused features to perform a first classification and a first regression operation, outputting a pre-detection result. The detection unit is used to perform a second classification and a second regression operation on the pre-detection result to generate a final detection result. A detection method based on the above detection system is also disclosed. This invention can detect small targets in remote sensing images through the detection system, effectively fuse edge features, and utilize multi-layer feature selection anchor points in the feature pyramid to locate small targets in the remote sensing image, thereby effectively obtaining a final detection result related to the small target. Attached Figure Description

[0049] Figure 1 This is a schematic diagram illustrating the composition of an embodiment of a small target detection system based on remote sensing images according to the present invention;

[0050] Figure 2 This is a data flow diagram of an embodiment of a small target detection system based on remote sensing images according to the present invention;

[0051] Figure 3 This is a schematic diagram of the edge information fusion unit in an embodiment of a small target detection system based on remote sensing images according to the present invention;

[0052] Figure 4 This is a schematic diagram of the coarse processing unit in one embodiment of a small target detection system based on remote sensing images according to the present invention;

[0053] Figure 5 This is a flowchart illustrating an embodiment of a small target detection method based on remote sensing images according to the present invention;

[0054] Figure 6 This is a detailed flowchart of step S2 in an embodiment of a small target detection method based on remote sensing images according to the present invention;

[0055] Figure 7 This is a schematic diagram illustrating the effect of step S3 in an embodiment of the small target detection method based on remote sensing images of the present invention;

[0056] Figure 8 This is a schematic diagram of the detection result of an embodiment of a small target detection method based on remote sensing images according to the present invention. Detailed Implementation

[0057] To facilitate understanding of the present invention, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0058] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0059] In practice, thanks to its powerful feature extraction and representation capabilities, deep learning has rapidly become the cornerstone of object detection in recent years. Numerous object detection frameworks based on classic deep learning have emerged, broadly categorized into one-stage and two-stage methods. One-stage methods directly regress the final detection result from deep features and dense anchor points, achieving higher computational efficiency and lower memory consumption. However, this approach typically reduces accuracy and robustness, especially for complex scenes. Two-stage methods first generate a series of candidate regions that may contain the target object through a Region Proposal Network (RPN), then classify and regress these candidate regions. This method has high computational complexity but also high detection accuracy. However, when directly applied to remote sensing images, the results of these methods are unsatisfactory. This approach heavily relies on high-quality proposals but requires heuristic anchor point settings. However, the existing Cascade RPN abandons this traditional setup, placing an anchor point at each feature point and performing multi-stage refinement. Although Cascade RPN exhibits superior performance on objects of general scale, its inherent limitations prevent it from ensuring that every small target has enough potential anchor points to maintain a relatively close distance. Furthermore, Cascade RPN only marks qualified anchors at a single feature pyramid level as positive samples, discarding many anchors at other levels that could locate small targets.

[0060] Meanwhile, general-purpose detectors often struggle to handle small objects due to insufficient or low-quality positive samples. Current popular detectors utilize overlap-based or distance-based strategies to select positive priors for objects during training. However, small instances typically occupy extremely limited regions, resulting in very little overlap between densely packed anchors and the real target, far exceeding the positive IOU (Intersection-over-Union) threshold. In other words, existing positive sample criteria are too stringent when applied to small or micro-objects, limiting the number of samples available for optimization. Lowering the threshold for defining positive samples might increase the number of positive samples, but it inevitably introduces many low-quality samples, impacting the model's optimization process and the learning objective.

[0061] Based on this, the present invention provides a small target detection system and method based on remote sensing images, which can further solve the above problems.

[0062] like Figure 1 The diagram illustrates a small target detection system based on remote sensing images, comprising a feature extraction unit 1, an edge information fusion unit 2, a coarse processing unit 3, and a detection unit 4. The feature extraction unit 1 acquires features at different scales in the remote sensing image, including low-level and high-level features. The edge information fusion unit 2 fuses low-level features to output fused edge features. The coarse processing unit 3 combines high-level features and fused edge features to select anchor points, performs a first classification and a first regression operation on the anchor points, and outputs a pre-detection result. The detection unit 4 performs a second classification and a second regression operation on the pre-detection result to generate the final detection result.

[0063] Combination Figure 2 The system uses feature extraction unit 1 to extract low-level and high-level features corresponding to different scales in remote sensing image 10. The extracted dual-layer low-level features are input into edge information fusion unit 2, which performs edge information interaction on the input low-level features to obtain edge fusion features. Then, the high-level features and edge fusion features are convolved and concatenated respectively, and simultaneously input into coarse processing unit 3. Coarse processing unit 3 uses templates to dynamically select anchor points and performs first classification and first regression operations on the anchor points to obtain pre-detection results. Finally, detection unit 4 performs second classification and second regression operations on the pre-detection results to obtain the final detection result 20 related to small targets in remote sensing image 10.

[0064] In this system, the edge information fusion unit 2 interacts with the input low-level features to adaptively capture multi-directional edge features between low-level features of different scales. Simultaneously, the coarse processing unit 3 obtains pre-detection results related to the small target, and the detection unit 4 further obtains the final detection results 20 further related to the small target.

[0065] like Figure 2 As shown, the feature extraction unit 1 includes a CSPDarknet53 network, which is used to extract four feature layers from low to high level of the remote sensing image 10, to obtain the first low-level feature 11, the second low-level feature 12, the first high-level feature 13, and the second high-level feature 14.

[0066] Generally speaking, low-level features are high-resolution images, which are more conducive to the detection of small targets. Therefore, this system applies the edge information fusion unit 2 to the low-level feature map.

[0067] In this embodiment, the first low-level feature 11 and the second low-level feature 12 are input into the edge information fusion unit 2. This unit performs edge information interaction between the two input features, adaptively captures multi-directional edge features between low-level features of different scales, learns multi-scale non-local dependencies in the learning space, and enhances the network's feature representation ability for small targets.

[0068] like Figure 2 and Figure 3 As shown, the edge information fusion unit 2 includes five fixed-shape convolutional layers, two deformable convolutional layers, two connection modules, a weighted switcher, and a cascade module.

[0069] Specifically, the first fixed-shape convolutional layer 21 receives the first low-level feature 11 and, after convolution processing, obtains the first convolutional feature. The second fixed-shape convolutional layer 22 receives the second low-level feature 12 and, after convolution processing, obtains the second convolutional feature. The first deformable convolutional layer 23 captures the first edge feature in the first convolutional feature and inputs it to the weighted switcher 25. The second deformable convolutional layer 24 captures the second edge feature in the second convolutional feature and also inputs it to the weighted switcher 25. The weighted switcher 25 multiplies the first edge feature and the second edge feature correspondingly and selects to output the discriminative third edge feature and fourth edge feature. The first connection module 26 connects the residuals of the first convolutional feature and the third edge feature and outputs the first connection feature. The second connection module 27 connects the residuals of the second convolutional feature and the fourth edge feature and outputs the second connection feature. The third fixed-shape convolutional layer 28 receives the first connection feature and, after convolution processing, obtains the third convolutional feature. The fourth fixed-shape convolutional layer 29 receives the second connection feature and, after convolution processing, obtains the fourth convolutional feature. The cascade module 201 concatenates the third and fourth convolutional features to output cascaded features. The fifth fixed-shape convolutional layer 202 receives the cascaded features and performs convolution processing to obtain edge fusion features.

[0070] like Figure 3 As shown, after receiving the first low-level feature 11 and the second low-level feature 12, the edge information fusion unit 2 first performs 3*3 convolution operations on the first low-level feature 11 and the second low-level feature 12 through the first fixed-shape convolutional layer 21 and the second fixed-shape convolutional layer 22, respectively, adjusting them to have the same number of channels to obtain the first convolutional feature and the second convolutional feature. This prepares for obtaining edge features of different scales through a weighted switcher, facilitating subsequent feature concatenation.

[0071] Then, the first deformable convolutional layer 23 and the second deformable convolutional layer 24 receive the first convolutional feature and the second convolutional feature respectively, and use the DCN v3 operator to perform deformable convolution on the two to obtain the first edge feature and the second edge feature.

[0072] In this embodiment, the DCN v3 operator can be used to extract edge features using convolutional layers and seamlessly embed the edge features into the neural network.

[0073] Furthermore, the weighted selector 25 multiplies the first edge feature and the second edge feature respectively, and then performs a 3*3 convolution operation to select and output the third edge feature and the fourth edge feature that have discriminative power.

[0074] In this embodiment, the weighted selector 25 helps to suppress background noise and, through mutual weighting, focuses more attention on the region of interest, selecting discriminative edge features, namely the third edge feature and the fourth edge feature.

[0075] Then, the third edge feature is residually connected to the first convolutional feature and the fourth edge feature is residually connected to the second convolutional feature through the first connection module 26 and the second connection module 27, respectively, to obtain the first connection feature and the second connection feature. Next, the first connection feature and the second connection feature are convolved 3*3 by the third fixed-shape convolutional layer 28 and the fourth fixed-shape convolutional layer 29, respectively, to obtain the third convolutional feature and the fourth convolutional feature. The third convolutional feature and the fourth convolutional feature are then convolved through the convolution module 201 to obtain the convolutional feature. Finally, the convolutional feature is convolved 3*3 by the fifth fixed-shape convolutional layer 202 to obtain the edge fusion feature containing rich edge information.

[0076] In this embodiment, the first connection module 26, the second connection module 27 and the cascade module 201 are used to fuse edge features at two scales, which can prevent some valuable clues or information from being lost during the deformable convolution operation.

[0077] like Figure 2 and Figure 4 As shown, the coarse processing unit 3 includes a Progressive Region Proposal Network (PRPN) 30, which comprises an anchor selection module 31, an adaptive convolutional module 32, and an RPN network 33. The anchor selection module 31 selects anchor points based on edge fusion features and high-level features, performing the first-stage pre-regression operation. The adaptive convolutional module 32 and the RPN network 33 perform the second-stage classification and regression operation. The adaptive convolutional module 32 refines the anchor points, inputting the refined anchor points, edge fusion features, and high-level features into the RPN network 33. The RPN network 33 performs the first classification and first regression operations on the refined anchor points, edge fusion features, and high-level features to obtain the pre-detection result. Here, the anchor point is part of the feature (figure) and can also be called an anchor box.

[0078] Furthermore, in Figure 2 In the process, the second high-level feature 14 is convolved to obtain the fourth high-level feature 17; the first high-level feature 13 is convolved and concatenated with the feature map upsampled from the fourth high-level feature 17 to obtain the third high-level feature 16; the feature map upsampled from the third high-level feature 16 is concatenated with the edge fusion feature to obtain the third low-level feature 15. The third low-level feature 15, the third high-level feature 16, and the fourth high-level feature 17 can be simultaneously input into the coarse processing unit 3.

[0079] Furthermore, the anchor point selection module 31 selects a suitable anchor point from the obtained third low-level feature 15, third high-level feature 16 and fourth high-level feature 17.

[0080] In this method, anchor points are dynamically selected using a template. The anchor point is the bounding box with an IOU greater than that of the template. The template is as follows:

[0081]

[0082] Where w and h are the width and height of the target box, respectively, γ represents the scaling factor, c represents the minimum feature map with both height and width being c, and IOU is the ratio of the area of ​​the intersection to the area of ​​the union of the bounding box and the target box.

[0083] Furthermore, γ = 0.15, c = 12.

[0084] This approach employs a region-based anchor selection strategy, ensuring a relatively sufficient number of potential anchors for targets of various sizes. c=12 corresponds to the minimum target size of 12*12 defined in the Small Object Detection dAtasets (SODA) dataset. This dynamic anchor selection mechanism allows smaller targets to have lower thresholds, but the minimum threshold will not fall below 0.25. The aim is to prevent too many low-quality samples from affecting the optimization, providing sufficient samples for extremely small targets, and allowing for adjustments to different datasets.

[0085] Furthermore, using a scaling factor γ and selecting the maximum value (max) prevents the optimization from being overwhelmed by low-quality priors, i.e., preventing the acquisition of too many low-quality samples with an IOU less than 0.25, also known as negative samples. Negative samples are anchor points that do not closely match the size of the small target to be detected. Too many negative samples will affect the accuracy of the network model; therefore, using γ and max operations here ensures the accuracy of the system's results.

[0086] In this embodiment, the anchor selection module 31 retains three Feature Pyramid Network (FPN) levels, namely, retaining the anchors corresponding to the third low-level feature 15, the third high-level feature 16, and the fourth high-level feature 17 to perform the first-stage pre-regression operation. Using IOU as the criterion for mining potential anchors avoids the optimization inconsistency problem in the multi-stage regression of the Progressive Region Proposal Network 30. Simultaneously, it can mine enough potential anchors for extremely small targets, while anchors matching larger targets naturally have higher IOUs and can still receive appropriate attention. Therefore, the anchor selection module 31 can retain anchors for large targets as well as those for small and extremely small targets, corresponding to targets of different sizes and proportions.

[0087] Furthermore, the adaptive convolutional module 32 and the RPN network 33 perform the second-stage classification and regression operation.

[0088] The adaptive convolution module 32 is used to correct the anchor points, and the corrected anchor points, along with the third low-level feature 15, the third high-level feature 16, and the fourth high-level feature 17, are simultaneously input into the RPN network 33. The RPN network 33 then performs a coarse classification and regression on the corrected anchor points and these features.

[0089] In this process, the adaptive convolution module 32 performs adaptive convolution on multiple anchor points selected by the anchor point selection module 31 to obtain the offset of the anchor points relative to the small target, thus correcting the anchor points. Then, the corrected anchor points and features are input into the RPN network 33, which performs a coarse first classification and first regression.

[0090] In this embodiment, the adaptive convolution module 32 can perform adaptive convolution on the anchor point, thereby calibrating the anchor point to make it fit the actual small target better.

[0091] In this embodiment, after the first stage of pre-regression, the offset of the anchor point is captured by the adaptive convolution module 32 and input into the RPN network 33 along with the features. The features are aligned by adaptive convolution, and the RPN network 33 is used to perform the first classification and first regression operations on the corrected anchor point and features to obtain the pre-detection result.

[0092] The pre-detection result is an intermediate image, in which the small target is marked in remote sensing image 10 by anchor points that have a lot of overlap with the small target.

[0093] In this embodiment, the coarse processing unit 3 adopts a fusion loss function, which is:

[0094]

[0095] Among them, L cls This represents the cross-entropy loss corresponding to the pre-regression operation in the first stage. and α1 and α2 are the IOU losses corresponding to the first-stage pre-regression operation and the second-stage classification regression operation, respectively; α1 and α2 are the loss weights, and their sum is 1.

[0096] In this embodiment, α1 = 0.1 and α2 = 0.9.

[0097] In some other embodiments, α1 and α2 may also be other values.

[0098] like Figure 2As shown, the detection unit 4 includes a normalization layer 41, a fully connected layer 42, and a classification and regression layer 43 connected in sequence. The normalization layer 41 receives the pre-detection results and normalizes them to obtain a normalized result. The fully connected layer 42 maps the normalized result to the mapped result. The classification and regression layer 43 performs a second classification and a second regression based on the mapped result. Small targets are then displayed using bounding boxes through the second classification and second regression to obtain the final detection result 20.

[0099] In this embodiment, the normalization layer 41 normalizes the pre-detection results to obtain a normalized result, ensuring that input anchor points with different scales can all obtain output features of the same size. Further, the fully connected layer 42 connects multiple output features and maps them to the mapping result. The classification and regression layer 43 performs a second classification and a second regression based on the mapping result, using bounding boxes to indicate small targets in the remote sensing image 10, simultaneously indicating the category of the small targets, to obtain the final detection result 20. The final detection result 20 is the final image in the remote sensing image 10 that marks the small targets.

[0100] At the same time, the total loss function can be obtained as: L = L PRPN +L′ cls +L′ reg , where L' cls and L' reg These represent the cross-entropy loss and IOU loss for the second classification and second regression processes, respectively.

[0101] Combination Figure 2 and Figure 5 As shown, based on the same inventive concept, the present invention also provides a method for small target detection based on remote sensing images, comprising the following steps:

[0102] Step S1: Obtain features at different scales in the remote sensing image. Features at different scales include low-level features and high-level features.

[0103] Among them, combined Figure 2 The remote sensing image 10 can be represented as I∈R H×W×3 The low-level features include the first low-level feature 11 and the second low-level feature 12, and the high-level features include the first high-level feature 13 and the second high-level feature 14.

[0104] Furthermore, the first low-level feature 11 is... The second low-level feature 12 is in, and These represent the height and width of the corresponding feature maps, respectively, and the height of the first low-level feature 11 and the second low-level feature 12 is equal to the height of the input remote sensing image 10. The widths of the first low-level feature 11 and the second low-level feature 12 are equal to the width of the input remote sensing image 10. The first low-level feature 11 has 128 channels, and the second low-level feature 12 has 256 channels. Correspondingly, the height of the first high-level feature 13 is equal to the height of the input remote sensing image 10. The width is the width of the input remote sensing image 10. The number of channels is 512. The height of the second high-level feature 14 is the height of the input remote sensing image 10. The width is the width of the input remote sensing image 10. The number of channels is 1024.

[0105] Step S2: Fuse low-level features and output edge fusion features.

[0106] The low-level features include a first low-level feature 11 and a second low-level feature 12. The first low-level feature 11 and the second low-level feature 12 are exchanged through the edge information fusion unit 2 of the above system to output an edge fusion feature containing effective edge information.

[0107] like Figure 6 As shown, step S2 specifically includes the following sub-steps:

[0108] Step S21: Convolve the first low-level feature and the second low-level feature to obtain the first convolutional feature and the second convolutional feature.

[0109] Specifically, a 3x3 convolution operation is performed on the first low-level feature 11 and the second low-level feature 12 respectively to obtain the first convolutional feature and the second convolutional feature. The number of channels of both is adjusted to 256 to facilitate subsequent feature concatenation. This can be represented as:

[0110]

[0111] in, The first convolutional feature, E1 is the first low-level feature (11), and E2 is the second low-level feature (12). This represents a 3x3 convolution, where i = 1 or i = 2.

[0112] Step S22: Perform deformable convolution on the first convolution feature and the second convolution feature to obtain the first edge feature and the second edge feature.

[0113] Specifically, the DCN v3 operator is used to perform deformable convolution on the first and second convolutional features to obtain the first and second edge features. This can be represented as:

[0114]

[0115] in, The first convolutional feature, For the second convolution feature, The first edge feature, This is the second edge feature. This represents the convolution operation of the DCN v3 operator.

[0116] Step S23: Multiply the first edge feature and the second edge feature respectively, then perform convolution, and select the third edge feature and the fourth edge feature to output.

[0117] Specifically, the first and second edge features are first multiplied together, then a 3x3 convolution operation is performed, and the third and fourth edge features with discriminative output are selected. This can be represented as:

[0118]

[0119] in, The first edge feature, This is the second edge feature. This is the third edge feature. This is the fourth edge feature. F1(·) and F2(·) both represent 3*3 convolution, but F1(·) and F2(·) perform convolution on features of different scales, thus obtaining the third edge feature corresponding to the first edge feature and the fourth edge feature corresponding to the second edge feature.

[0120] Step S24: Cascade and fuse the third edge feature and the fourth edge feature to obtain the edge fusion feature.

[0121] Specifically, it includes the following steps:

[0122] First, the third edge feature is residually connected to the first convolutional feature, and the fourth edge feature is residually connected to the second convolutional feature to obtain the first connection feature and the second connection feature, respectively.

[0123] Then, a 3*3 convolution is performed on the first and second connection features to obtain the third and fourth convolution features.

[0124] The third and fourth convolutional features are then concatenated to obtain concatenated features.

[0125] Finally, a 3*3 convolution is performed on the cascaded features to obtain edge fusion features containing rich edge information.

[0126] The operation of step S24 can be represented as follows:

[0127]

[0128] Among them, F e For edge blending features, This is the third edge feature. This is the fourth edge feature. The first convolutional feature, For the second convolution feature, F(·) represents a 3*3 convolution.

[0129] Step S3: Combine advanced features and edge fusion features to select anchor points, perform first classification and first regression on the anchor points, and output the pre-detection results.

[0130] Specifically, the second high-level feature 14 is first convolved with a 3*3 convolution to obtain the fourth high-level feature 17; the first high-level feature 13 is convolved with a 3*3 convolution and then concatenated with the feature map upsampled from the fourth high-level feature 17 to obtain the third high-level feature 16; the feature map upsampled from the third high-level feature 16 is then concatenated with the edge fusion feature to obtain the third low-level feature 15.

[0131] like Figure 7 As shown, anchor points are selected in the first image 40 based on the third low-level feature 15, the third high-level feature 16, and the fourth high-level feature 17. Then, adaptive convolution is used in the second image 50 to obtain the offset of the anchor points relative to the small target, and the anchor points are corrected. Finally, based on these features and the corrected anchor points, a first classification and a first regression are performed to obtain the pre-detection result, namely, the third image 60 which identifies the small target by anchor points that have a significant overlap with the small target. Simultaneously, this process effectively captures the anchor points corresponding to the small target, enabling precise localization of the small target.

[0132] Step S4: Perform a second classification and a second regression on the pre-detection results to generate the final detection results.

[0133] Combination Figure 2 As shown, the pre-detection results are first normalized to ensure that all input anchor points of different scales have the same output feature size. Then, the output features are mapped onto the image for second classification and second regression. Bounding boxes are used to accurately label small targets in the remote sensing image 10, while also indicating the category of the small targets, resulting in the final detection result 20.

[0134] like Figure 8 As shown, this figure displays a final detection result obtained using this detection method. The figure shows small targets marked with bounding boxes and the categories to which the small targets belong.

[0135] In this embodiment, the feature extraction unit 1 extracts multi-layer features, and the edge information fusion unit 2 performs edge information interaction on the two-layer low-level features, which can effectively fuse the edge features. The coarse processing unit 3 and the detection unit 4 use the multi-layer features in the feature pyramid to select anchor points to locate small targets, thereby effectively obtaining the final detection result 20 related to the small targets in the remote sensing image 10.

[0136] Therefore, this invention discloses a small target detection system based on remote sensing images, including a feature extraction unit, an edge information fusion unit, a coarse processing unit, and a detection unit. The feature extraction unit is used to acquire features at different scales in the remote sensing image, including low-level features and high-level features. The edge information fusion unit is used to fuse low-level features and output edge-fused features. The coarse processing unit is used to combine high-level features and edge-fused features to perform a first classification and a first regression operation, outputting a pre-detection result. The detection unit is used to perform a second classification and a second regression operation on the pre-detection result to generate a final detection result. A detection method based on the above detection system is also disclosed. This invention can detect small targets in remote sensing images through the detection system, effectively fuse edge features, and use multi-layer features in the feature pyramid to select anchor points to locate small targets, thereby effectively obtaining a final detection result related to the small targets.

[0137] The above are merely embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A small target detection system based on remote sensing images, characterized in that, It includes a feature extraction unit, an edge information fusion unit, a coarse processing unit, and a detection unit; The feature extraction unit is used to acquire features at different scales in the remote sensing image. The features at different scales include low-level features and high-level features. The low-level features include a first low-level feature and a second low-level feature, and the high-level features include a first high-level feature and a second high-level feature. The edge information fusion unit is used to fuse the low-level features and output edge fusion features; wherein, the second high-level feature is convolved to obtain a fourth high-level feature; the first high-level feature is convolved and concatenated with the feature map upsampled from the fourth high-level feature to obtain a third high-level feature; the feature map upsampled from the third high-level feature is concatenated with the edge fusion features to obtain a third low-level feature; the third low-level feature, the third high-level feature, and the fourth high-level feature are input into the coarse processing unit; The edge information fusion unit includes five fixed-shape convolutional layers, two deformable convolutional layers, two connection modules, a weighted switcher, and a cascade module; and: The first fixed-shape convolutional layer is used to receive the first low-level feature, and the first convolutional feature is obtained after convolution processing. The second fixed-shape convolutional layer is used to receive the second low-level feature, and the second convolutional feature is obtained after convolution processing. The first deformable convolutional layer is used to capture the first edge feature in the first convolutional feature and input it into the weighted switcher; The second deformable convolutional layer is used to capture the second edge feature in the second convolutional feature, which is then input into the weighted switcher. The weighted switcher is used to multiply the first edge feature and the second edge feature respectively and perform a convolution operation, and select the third edge feature and the fourth edge feature with discriminative output. The first connection module is used to connect the first convolutional feature and the third edge feature residual to output the first connection feature; The second connection module is used to connect the second convolutional feature and the fourth edge feature residual to output the second connection feature; The third fixed-shape convolutional layer is used to receive the first connection feature, and the third convolutional feature is obtained after convolution processing. The fourth fixed-shape convolutional layer is used to receive the second connection feature, and the fourth convolutional feature is obtained after convolution processing; The cascading module is used to cascade the third convolutional feature and the fourth convolutional feature to output a cascaded feature. The fifth fixed-shape convolutional layer is used to receive the cascaded features, and the edge fusion features are obtained after convolution processing; The coarse processing unit is used to select anchor points by combining the high-level features and the edge fusion features, perform a first classification and a first regression operation on the anchor points, and output a pre-detection result; wherein, the coarse processing unit is used to dynamically select anchor points from the third low-level features, the third high-level features, and the fourth high-level features using a template, wherein the anchor point is a bounding box with an IOU greater than the template, and the template is: in, and These are the width and height of the target bounding box, respectively. Indicates the scaling factor. c The minimum feature map has a height and width of 1. c The IOU is the ratio of the area of ​​the intersection to the area of ​​the union of the bounding box and the target box; The detection unit is used to perform a second classification and a second regression operation on the pre-detection results to generate the final detection results.

2. The small target detection system according to claim 1, characterized in that, The coarse processing unit includes a progressive region proposal network, which includes an anchor selection module that performs a first-stage pre-regression operation, and an adaptive convolution module and an RPN network that perform a second-stage classification regression operation. The anchor point selection module is used to select anchor points based on the edge blending features and the advanced features; The adaptive convolution module is used to correct the anchor points, and inputs the corrected anchor points, the edge fusion features, and the high-level features into the RPN network; The RPN network is used to perform a first classification and a first regression operation on the corrected anchor points, the edge fusion features, and the high-level features to obtain the pre-detection results.

3. The small target detection system according to claim 2, characterized in that, The feature extraction unit includes a CSPDarknet53 network, used to extract four feature layers from the remote sensing image from low to high level, to obtain the first low-level feature, the second low-level feature, the first high-level feature, and the second high-level feature.

4. The small target detection system according to claim 3, characterized in that, The detection unit includes a normalization layer, a fully connected layer, and a classification and regression layer connected in sequence. The normalization layer is used to receive the pre-detection results, normalize the pre-detection results, and obtain the normalized results. The fully connected layer is used to map the normalization result to the mapping result; The classification and regression layer is used to perform a second classification and a second regression based on the mapping result, using bounding boxes to show the target and indicate the target category, and obtain the final detection result.

5. The small target detection system according to claim 4, characterized in that, In the template, .

6. The small target detection system according to claim 5, characterized in that, The coarse processing unit employs a fusion loss function, which is: ； in, This represents the cross-entropy loss corresponding to the pre-regression operation in the first stage; and These represent the IOU losses for the first-stage pre-regression operation and the second-stage classification regression operation, respectively. and The weights are the loss weights, and the sum of the two is 1.

7. A method for small target detection based on remote sensing images, characterized in that, Includes the following steps: Features at different scales are acquired in remote sensing images, including low-level features and high-level features; wherein, the low-level features include a first low-level feature and a second low-level feature, and the high-level features include a first high-level feature and a second high-level feature; The low-level features are fused to output edge fusion features; wherein, the first low-level features and the second low-level features are convolved to obtain a first convolutional feature and a second convolutional feature; the first convolutional feature and the second convolutional feature are deformably convolved to obtain a first edge feature and a second edge feature; the first edge feature and the second edge feature are multiplied correspondingly and then convolved to select and output a third edge feature and a fourth edge feature; the third edge feature and the fourth edge feature are cascaded and fused to obtain the edge fusion features; The second high-level feature is convolved to obtain the fourth high-level feature; the first high-level feature is convolved and concatenated with the feature map upsampled from the fourth high-level feature to obtain the third high-level feature; the feature map upsampled from the third high-level feature is concatenated with the edge fusion feature to obtain the third low-level feature; the third low-level feature, the third high-level feature, and the fourth high-level feature are combined. Anchor points are selected by combining the high-level features and the edge fusion features. A first classification and a first regression are performed on the anchor points, and a pre-detection result is output. Specifically, anchor points are dynamically selected from the third low-level features, the third high-level features, and the fourth high-level features using a template. The anchor point is a bounding box with an IOU greater than the template. The template is: in, and These are the width and height of the target bounding box, respectively. Indicates the scaling factor. c The minimum feature map has a height and width of 1. c The IOU is the ratio of the area of ​​the intersection to the area of ​​the union of the bounding box and the target box; The pre-detection results are then subjected to a second classification and a second regression to generate the final detection results.

Images