A method and system for small target detection in remote sensing images based on multi-scale features

Abstract

This invention discloses a method and system for small target detection in remote sensing images based on multi-scale features. The method involves: acquiring a remote sensing image to be detected; performing dehazing and brightness enhancement processing on the remote sensing image; inputting the brightness-enhanced image into a trained remote sensing image small target detection model to obtain the small target detection result; and the working principle of the trained remote sensing image small target detection model includes: extracting multi-scale features from the brightness-enhanced remote sensing image; performing feature fusion and feature enhancement on the extracted multi-scale features to obtain a feature map; fixing the size of regions of interest of different sizes in the feature map to obtain proposed candidate boxes for the target; applying non-maximum suppression to the extracted proposed candidate boxes to obtain the predicted bounding box coordinates of the target; drawing the minimum bounding rectangle of the target's region; and using the minimum bounding rectangle as the predicted bounding box of the target to obtain the target detection result in the remote sensing image.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method and system for detecting small targets in remote sensing images based on multi-scale features. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] Remote sensing imagery, due to its high viewing angle, contains rich location and feature information. Given such abundant image sources, generating effective social benefits is a challenge for researchers. Research in remote sensing image target detection has consistently been a hot topic, with applications in environmental monitoring, engineering construction, agricultural production, urban planning, and military reconnaissance. In environmental monitoring, remote sensing technology can be applied to forestry for monitoring, tracking, and disaster analysis. In engineering construction, remote sensing technology can be used for preliminary detection, greatly aiding engineering design. In agricultural production, remote sensing technology can be used in farmland and crop cultivation to predict crop yields and monitor crop diseases. In urban planning, remote sensing technology can be used to conveniently obtain urban hydrological information. Furthermore, the military value of remote sensing technology has been explored. By detecting and identifying military targets, precise monitoring of ground conditions in the monitored area can be achieved, effective reconnaissance of military facilities in the monitored area can be conducted, and information on equipment and troop mobilization in the area can be obtained.

[0004] Traditional remote sensing target detection methods are mainly based on feature extraction algorithms and machine learning, which require rich experience and complete prior knowledge. For example, algorithms such as Histogram of Gradients (HOG), Scale Invariant Feature Transform (SIFT), and Local Binary Pattern (LBP) are used to extract image features and train Support Vector Machines (SVMs) as classifiers for target detection. However, most of these methods are only effective in specific environments and have poor scalability and generalization ability.

[0005] With the continuous improvement of computing power, deep learning has been widely used in object detection. Compared with traditional object detection algorithms, deep learning models directly extract features from the original image by simulating the human brain's visual perception system and pass them layer by layer to obtain high-dimensional information of the image.

[0006] Currently, deep learning detection models can be broadly classified into two categories: The first category divides object detection into two stages: candidate box selection and object classification, with representative algorithms including R-CNN (Region-Convolutional Neural Networks) and Faster R-CNN; The second category is a one-stage model that treats classification and localization as regression tasks, with typical representative algorithms including YOLO (You Only Look Once) and Retina-Net.

[0007] Small target detection in remote sensing images is a key and challenging area of ​​research in computer vision, with significant applications in security, military, and manufacturing. Compared to conventional targets, small targets are characterized by weak features and limited information, making them extremely difficult to distinguish from similar backgrounds or adjacent targets. The challenges are further amplified when faced with complex environments such as low illumination and shadow occlusion.

[0008] The current difficulties in detecting small targets in remote sensing images are mainly reflected in four aspects:

[0009] (i) Weak target representation ability. According to the aforementioned definition, small targets account for a relatively small proportion of an image and contain very few effective features. This results in insufficient representation ability of small targets themselves, further affecting subsequent localization and recognition.

[0010] (ii) Target features are easily lost. Current target detectors often involve multiple downsampling operations when extracting image features, resulting in small targets having only a few or tens of pixels left for subsequent detection;

[0011] (iii) Targets are easily obscured by complex backgrounds. In large-size remote sensing images, small targets contain far fewer pixels than the background pixels of the remote sensing image, making them easily obscured and difficult for detectors to detect;

[0012] (iv) Target localization is difficult. Small targets are relatively small in size, making bounding box regression more difficult compared to large targets.

[0013] To address the problem of small object detection, scholars both domestically and internationally have proposed numerous improved methods. Yu et al. proposed a scale-matching strategy, cropping objects based on their size to reduce the scale difference between different objects and avoid losing small object information during image downsampling. Zoph et al. proposed an adaptive learning method to select the optimal data augmentation strategy, achieving a certain performance improvement in small object detection. Besides data augmentation, learning the object's contextual information can also improve small object detection performance. Chen et al. proposed a hierarchical contextual embedding framework, enhancing the feature representation of candidate regions by mining contextual cues. Hong et al. proposed a scale-selection pyramid network, avoiding inconsistent gradient calculations at different levels of features by constructing multi-scale contextual information. Furthermore, combining deep semantic information and shallow detail information through feature fusion can enhance the feature representation of small objects. Nayan et al. used upsampling and skip connections to extract multi-scale features at different network depths during training, significantly improving the detection accuracy and speed of small objects. Deng et al. proposed an extended feature pyramid network, using an additional high-resolution pyramid specifically for small object detection.

[0014] In summary, existing methods still face challenges in detecting small targets in remotely sensed images with complex backgrounds, lacking effective solutions. Most current methods focus primarily on the target's contextual or deep semantic information, neglecting the crucial role of fusing contextual and semantic information. While good detection results can be achieved through other methods such as data augmentation or feature fusion across different levels and scales, their robustness is weak, especially when small objects are in complex scenes or are too small to extract effective features and maintain appropriate detection accuracy. Summary of the Invention

[0015] To address the shortcomings of existing technologies, this invention provides a method and system for small target detection in remote sensing images based on multi-scale features; it focuses on the preservation and extraction of small target features and contextual information to solve the problem of spatial information loss in small target detection in remote sensing images.

[0016] In a first aspect, the present invention provides a method for detecting small targets in remote sensing images based on multi-scale features;

[0017] Methods for detecting small targets in remote sensing images based on multi-scale features include:

[0018] Acquire the remote sensing image to be detected;

[0019] The remote sensing image to be detected is subjected to dehazing and brightness enhancement processing;

[0020] The image with enhanced brightness is input into the trained remote sensing image small target detection model to obtain the small target detection results of the remote sensing image;

[0021] The working principle of the trained remote sensing image small target detection model includes: multi-scale feature extraction of the brightness-enhanced remote sensing image; feature fusion and feature enhancement of the extracted multi-scale features to obtain a feature map; fixing the size of regions of interest of different sizes in the feature map to obtain proposed candidate boxes for the target; obtaining the predicted bounding box coordinates of the target after non-maximum suppression of the extracted proposed candidate boxes; drawing the minimum bounding rectangle of the target's region; using the minimum bounding rectangle as the target's predicted bounding box to obtain the detection result of the target in the remote sensing image.

[0022] Secondly, the present invention provides a remote sensing image small target detection system based on multi-scale features;

[0023] A remote sensing image small target detection system based on multi-scale features includes:

[0024] The acquisition module is configured to acquire the remote sensing image to be detected;

[0025] The processing module is configured to perform dehazing and brightness enhancement processing on the remote sensing image to be detected.

[0026] The detection module is configured to input the brightness-enhanced image into the trained remote sensing image small target detection model to obtain the small target detection results of the remote sensing image;

[0027] The working principle of the trained remote sensing image small target detection model includes: multi-scale feature extraction of the brightness-enhanced remote sensing image; feature fusion and feature enhancement of the extracted multi-scale features to obtain a feature map; fixing the size of regions of interest of different sizes in the feature map to obtain proposed candidate boxes for the target; obtaining the predicted bounding box coordinates of the target after non-maximum suppression of the extracted proposed candidate boxes; drawing the minimum bounding rectangle of the target's region; using the minimum bounding rectangle as the target's predicted bounding box to obtain the detection result of the target in the remote sensing image.

[0028] Thirdly, the present invention also provides an electronic device, comprising:

[0029] Memory, used for non-transitory storage of computer-readable instructions; and

[0030] Processor, for executing the computer-readable instructions,

[0031] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0032] Fourthly, the present invention also provides a storage medium for non-transitory storage of computer-readable instructions, wherein, when the non-transitory computer-readable instructions are executed by a computer, the instructions for the method described in the first aspect are executed.

[0033] Fifthly, the present invention also provides a computer program product, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] This invention utilizes a remote sensing image small target detection model trained with an established image preprocessing and multi-scale feature extraction framework and a cascaded convolutional neural network to enhance the feature extraction capability of the small target detection model, thereby improving the representation capability of small targets while improving complex background environments.

[0036] This invention can enhance the semantic information of low-level feature maps while avoiding interference from inaccurate location information in high-level coarse-grained feature maps on accurate location information in low-level fine-grained feature maps. It can effectively alleviate the interference of complex background information in remote sensing images on target features and enhance the classification ability of detectors.

[0037] This invention can effectively suppress the expression of background information in remote sensing images and further enhance the detail information of small targets, thereby improving the model's performance in the task of detecting small targets in remote sensing images. Attached Figure Description

[0038] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0039] Figure 1 This is a flowchart of a remote sensing image small target detection method based on multi-scale features and cascaded structure;

[0040] Figure 2 This is a schematic diagram of the internal network structure of the improved feature pyramid network of the present invention;

[0041] Figures 3(a)-3(e) This is a diagram showing the test results of the present invention;

[0042] Figure 4 It is a Precision-Recall plot based on the comparison of different scales of IOU. Detailed Implementation

[0043] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0046] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0047] Example 1

[0048] This embodiment provides a method for detecting small targets in remote sensing images based on multi-scale features;

[0049] like Figure 1 As shown, a method for detecting small targets in remote sensing images based on multi-scale features includes:

[0050] S101: Acquire the remote sensing image to be detected;

[0051] S102: Perform dehazing and brightness enhancement processing on the remote sensing image to be detected;

[0052] S103: Input the brightness-enhanced image into the trained remote sensing image small target detection model to obtain the small target detection results of the remote sensing image;

[0053] The working principle of the trained remote sensing image small target detection model includes:

[0054] Multi-scale feature extraction is performed on the remote sensing image after brightness enhancement processing;

[0055] The extracted multi-scale features are fused and enhanced to obtain a feature map.

[0056] By fixing the size of regions of interest of different sizes in the feature map, we can obtain the proposed candidate boxes for the target.

[0057] After the extracted suggested candidate boxes are subjected to non-maximum suppression, the coordinates of the predicted bounding box of the target are obtained. The minimum bounding rectangle of the region where the target is located is drawn, and the minimum bounding rectangle is used as the predicted bounding box of the target to obtain the detection result of the target in the remote sensing image.

[0058] It should be understood that the term "small target" refers to a target with a small imaging size. There are generally two ways to define it: one is the absolute size, in the COCO dataset (which is a common target detection format in the field), targets smaller than 32×32 pixels are considered small targets; the other is the relative size, according to the definition of the International Society for Optics and Photonics, a small target is a target with an imaging area of ​​less than 80 pixels in a 256×256 pixel image, that is, if the size of the target is less than 0.12% of the original image, it can be considered a small target.

[0059] Furthermore, the defogging process is implemented using an atmospheric scattering model based on median filtering.

[0060] Furthermore, the defogging process specifically includes:

[0061] S101-1: Estimating atmospheric light intensity A based on the dark primary color prior theory;

[0062] S101-2: Based on atmospheric light intensity A, estimate dust and fog concentration V according to the median filtering principle;

[0063] S101-3: Achieve image dehazing based on dust and fog concentration V.

[0064] Furthermore, the defogging process specifically includes:

[0065] (1) Median filtering is used to smooth M(x,y), which preserves the boundary parts of the image while smoothing:

[0066] A(x,y)=median_sv(M(x,y))

[0067] Where sv is the size of the filtering window, and its value is the maximum size of the white target in the image.

[0068] (2) Some textures will still exist in the filtered image, and these textures may not be foggy and do not need to be dehazed. So, the local standard deviation of M(x,y) is subtracted from A(x,y) to determine that the remaining components are all fog components.

[0069] B(x,y)=A(x,y)-median_sv(|M(x,y)-A(x,y)|)

[0070] Median filtering is applied to |M(x,y)-A(x,y)| to estimate the local standard deviation of M.

[0071] (3) B(x,y) obeys the two constraints (1) and (2) to obtain:

[0072] V(x,y)=max(min(pB(x,y),M(x,y)),0)

[0073] The value p controls the dehazing capability, typically ranging from 0.9 to 0.95. In reality, air inevitably contains some impurity molecules. If the fog is completely removed, the image will look unrealistic and lose its sense of depth. Therefore, the constant p is introduced to selectively retain a portion of the fog covering distant objects.

[0074] It should be understood that median filtering is a non-linear smoothing technique that sets the gray value of each pixel to the median of the gray values ​​of all pixels within a neighborhood window of that pixel. In median filtering, noise components are difficult to select, so all noise can be removed with almost no impact on the original image. Therefore, in estimating dust and fog concentration, median filtering can provide a better estimate of the concentration. The purpose of establishing an atmospheric scattering model is to recover a clear image from a foggy image. The brightness of the recovered image is then adjusted by changing its mean and standard deviation.

[0075] For example, the atmospheric scattering model based on median filtering is as follows:

[0076] I(x,y)=J(x,y)t(x,y)+A[1-t(x,y)](0.1)

[0077] Where I(x,y) is the foggy image, J(x,y) is the restored image, A is the atmospheric light intensity, and t(x,y) is the medium propagation function, which is related to the depth of field and therefore cannot be calculated directly from the data, but needs to be estimated.

[0078] The first step is to estimate the atmospheric light intensity A using the dark color prior theory. Since the dark color image of a foggy image has higher pixel brightness values ​​in areas with denser fog, visually, the brightness of the dark color image corresponds to the estimate of fog concentration. In the dark color image, the brightness values ​​of each pixel are arranged in descending order to determine the position of the top 0.1% of pixels in the dark color image. These positions correspond to the original foggy image, and the maximum gray value in the foggy image is taken as the atmospheric light intensity A.

[0079] The second step is to estimate the dust concentration V based on the median filtering principle, using the following formula:

[0080] M(x,y)=min(I(x,y))(0.2)

[0081] Image I is the minimum value of each pixel in RGB, and M is a grayscale image.

[0082] The dust concentration V satisfies: 0 ≤ V(x,y) ≤ M(x,y);

[0083] (1) 0≤V(x,y) because all pixels are affected by atmospheric light.

[0084] (2) V(x,y)≤M(x,y) because the brightness value of each pixel in a foggy image contains two parts: atmospheric light and light reflected by the scene itself. Therefore, V(x,y) is less than or equal to the minimum value of the pixel in all channels.

[0085] Furthermore, the brightness enhancement process specifically includes:

[0086] The histogram of the dehazed image is manipulated to change its mean and variance, thereby enhancing brightness.

[0087] For example, adjusting the brightness of a dehazed image involves manipulating the image's histogram, changing the mean and standard deviation of the grayscale image's histogram:

[0088]

[0089] Where, x in For the input image, x out For the output image, m0 represents the expected mean of the image pixel distribution, s0 represents the expected standard deviation of the image pixel distribution, m is the mean of the original image, and s is the standard deviation of the original image.

[0090] Furthermore, such as Figure 2 As shown, the network structure of the trained remote sensing image small target detection model includes:

[0091] Res2Net network and feature pyramid network;

[0092] The Res2Net network includes: a first convolutional block conv_1, a second convolutional block conv_2, a third convolutional block conv_3, a fourth convolutional block conv_4, and a fifth convolutional block conv_5 connected sequentially from bottom to top; the output of the first convolutional block conv_1 outputs feature map c1, the output of the second convolutional block conv_2 outputs feature map c2, the output of the third convolutional block conv_3 outputs feature map c3, the output of the fourth convolutional block conv_4 outputs feature map c4, and the output of the fifth convolutional block conv_5 outputs feature map c5.

[0093] The feature pyramid network includes, from top to bottom, an upsampling unit y1, an adder J1, an upsampling unit y2, an adder J2, an upsampling unit y3, and an adder J3; the input of the upsampling unit y1 is connected to the output of the fifth convolutional block conv_5 through a 1*1 convolutional layer M1; the input of the adder J1 is connected to the output of the fourth convolutional block conv_4 through a 1*1 convolutional layer M2; the input of the adder J2 is connected to the output of the third convolutional block conv_3 through a 1*1 convolutional layer M3; and the input of the adder J3 is connected to the output of the second convolutional block conv_2 through a 1*1 convolutional layer M4.

[0094] The output of the 1*1 convolutional layer M1 is connected to the input of the 3*3 convolutional layer L1; the output of adder J1 is connected to the input of the 3*3 convolutional layer L2; the output of adder J2 is connected to the input of the 3*3 convolutional layer L3; and the output of adder J3 is connected to the input of the 3*3 convolutional layer L4.

[0095] The output of the 3x3 convolutional layer L1 is connected to the input of connector Q1;

[0096] The output of the 3x3 convolutional layer L2 is connected to the input of connector Q2;

[0097] The output of the 3x3 convolutional layer L3 is connected to the input of connector Q3;

[0098] The output of the 3x3 convolutional layer L4 is connected to the input of the 1x1 convolutional layer M8;

[0099] The output of the 1*1 convolutional layer M8 is connected to the input of the downsampling unit Z3, and the output of the 1*1 convolutional layer M8 is connected to the input of the 3*3 convolutional layer L8; the output of the downsampling unit Z3 is connected to the input of the connector Q3.

[0100] The output of connector Q3 is connected to the input of the 1*1 convolutional layer M7;

[0101] The output of the 1*1 convolutional layer M7 is connected to the input of the 3*3 convolutional layer L7; the output of the 1*1 convolutional layer M7 is connected to the input of the downsampling unit Z2; the output of the downsampling unit Z2 is connected to the input of the connector Q2.

[0102] The output of connector Q2 is connected to the input of the 1*1 convolutional layer M6;

[0103] The output of the 1*1 convolutional layer M6 is connected to the input of the 3*3 convolutional layer L6; the output of the 1*1 convolutional layer M6 is connected to the input of the downsampling unit Z1; the output of the downsampling unit Z1 is connected to the input of the connector Q1.

[0104] The output of connector Q1 is connected to the input of a 1*1 convolutional layer M5; the output of the 1*1 convolutional layer M5 is connected to the input of a 3*3 convolutional layer L5.

[0105] 3x3 convolutional layers L5, L6, L7, and L8 are used to output feature maps; the outputs of these three layers are all connected to the input of the Region Generation Network (RPN); and the outputs of these three layers are all connected to the input of ROI Pooling.

[0106] The output of the Region Generation Network (RPN) is connected to the Cascaded Detection Network, which outputs the detection results.

[0107] The output of convolutional layer L1 outputs feature map P5, the output of convolutional layer L2 outputs feature map P4, the output of convolutional layer L3 outputs feature map P3, and the output of convolutional layer L4 outputs feature map P2.

[0108] The output of convolutional layer L5 is feature map N5, the output of convolutional layer L6 is feature map N4, the output of convolutional layer L7 is feature map N3, and the output of convolutional layer L8 is feature map N2.

[0109] The output of the second convolutional block conv_2 is also connected to the input of the upsampling unit y1;

[0110] The output of the second convolutional block conv_2 is also connected to the input of connector Q1.

[0111] The function of connectors Q1, Q2 and Q3 is to realize the serial splicing of feature maps.

[0112] Further, the first convolutional block conv_1 includes a convolutional layer; the second convolutional block conv_2 includes a downsampling layer and three convolutional sub-blocks connected in sequence; the third convolutional block conv_3 includes a downsampling layer and four convolutional sub-blocks connected in sequence; the fourth convolutional block conv_4 includes a downsampling layer and 23 convolutional sub-blocks connected in sequence; and the fifth convolutional block conv_5 includes a downsampling layer and three convolutional sub-blocks connected in sequence. Each convolutional sub-block includes a 1*1 first convolutional layer, a 3*3 second convolutional layer, and a 1*1 third convolutional layer connected in sequence, with the input of the first convolutional layer and the output of the third convolutional layer connected by a residual connection.

[0113] For example, Res2Net: First, it adopts the standard ResNet101 network; the ResNet101 network includes five levels of feature extraction modules, but in order to maximize the preservation of information of shallow small objects while obtaining deep semantic information, the feature mapping in the ResNet residual block is divided into multiple channel groups, and a residual-like connection is designed between different channel groups, so that the network can improve the multi-scale expression ability at a finer level.

[0114] Each convolutional sub-block, after a 1×1 convolutional layer, divides the feature map into s subsets, each represented by x... i Let x be a subset of feature maps, where i ∈ {1, 2, 3, ..., s}. i They have the same spatial size, but the number of channels is [amount missing] of the original input feature map. Every x except x1 i Each has its corresponding 3×3 convolutional layer, respectively using K i Indicates that x i After convolutional layer K i The output is then set to y. i , the feature map subset x i After K i-1 Feature map subset x i-1 Input K after addition i Get the output y i , where y i The expression is defined as follows:

[0115]

[0116] Therefore, each 3×3 convolution kernel K in the residual block i It can receive information from all previous feature map subsets, i.e., {x j The feature information of the subset x of the feature map is such that j≤i}. j After a 3×3 convolution, a value greater than x can be obtained. j The perception field is larger, resulting in a larger output. Finally, to fuse information from different scales, all y... i They are connected in parallel and fused using 1×1 convolutions.

[0117] This grouping and merging strategy enables convolutional layers to process feature maps more effectively, allowing residual block output feature maps to contain different receptive fields. This facilitates the extraction of multi-scale features, enabling the network to capture local or global image features at a finer granular level.

[0118] Traditional feature pyramids suffer from inaccurate localization information in deep feature maps, highlighting the need to add bottom-up paths to propagate shallower, more semantically robust features and enhance the overall hierarchy. Therefore, following the feature pyramid definition of P2, P3, P4, and P5, we generate feature maps of the same size and add bottom-up paths, such as N2, N3, N4, and N5, to the normal feature pyramid. Path aggregation begins at P2, where P2 is N2, without any processing. Spatial downsampling is performed from P2 to P5, and the newly generated feature maps are finally represented by N2, N3, N4, and N5. High-resolution feature map N... i A downsampled image is generated using a 3×3 convolutional layer with a stride of 2. This downsampled image is then compared with a coarser feature map P. i+1 Lateral connections are performed. Each feature map element is summed, and the summed feature map is then passed through a 3×3 convolutional layer to generate a new feature map N. i+1 This downsampling structure is repeated up to N5, thus the modified feature pyramid completes the path aggregation function, which can better convey semantic information to the entire feature hierarchy.

[0119] It should be understood that the trained remote sensing image small target detection model can improve the semantic information representation of shallow feature maps, effectively reduce the interference of the surrounding environment in complex backgrounds, and thus improve the detection effect of related targets.

[0120] Furthermore, the cascaded detection network includes T cascaded regressors, each of which is implemented using the Region-CNN (Convolutional Neural Networks) detection network. Except for the first detection network which performs bounding box regression based on the candidate boxes generated by the RPN network, each subsequent detection network performs regression based on the bounding boxes predicted by the previous detection network.

[0121] Furthermore, the T cascaded regressors are defined as follows:

[0122]

[0123] Where T represents the total number of cascaded stages, and in a cascaded detection network, T = 3; x is the image input for the current stage; and b is the target bounding box corresponding to the image in the current stage. In cascaded bounding box regression, each regression model f i The sample distribution for the corresponding stage is b i Therefore, optimization is performed after each regression stage, and the regression effect will be continuously improved. ο represents cascading, and f1(x,b) represents ROI Pooling and the detection network Region-CNN.

[0124] Furthermore, the training process of the trained remote sensing image small target detection model includes:

[0125] Construct a training set; the training set consists of remote sensing images with known small target labels;

[0126] The training set is input into the remote sensing image small target detection model, and the model is trained using stochastic gradient descent, error backpropagation, and learning rate decay. Training is stopped when the total loss function of the model no longer decreases or the number of iterations reaches the maximum number of iterations, resulting in the trained remote sensing image small target detection model. The accuracy and recall of each trained model need to be evaluated on the validation set, and the best-performing model is tested on the test set to obtain its generalization ability.

[0127] Furthermore, the construction of the training set specifically includes:

[0128] (1) The dataset images are from the Gaofen-1 and Gaofen-2 satellites. The obtained remote sensing images are cropped by using a sliding window with a window size of 512×512 pixels and a step size of 512 pixels to obtain the cropped images.

[0129] (2) For each image, the small targets in the filtered images are manually labeled using the labelMe labeling tool. The labeling content includes the image name, target category label, and location information of the small targets in the image. After all images are labeled, they are converted into XML format suitable for training on the PyTorch platform and randomly divided into training, validation and test sets at a ratio of 9:1. Each set does not interfere with the others and there are no identical images to prevent data contamination. Finally, a general remote sensing image small target dataset is created.

[0130] Furthermore, the multi-scale feature extraction of the brightness-enhanced remote sensing image is achieved through a Res2Net network.

[0131] Furthermore, the extracted multi-scale features are fused and enhanced to obtain a feature map, which is then implemented through a feature pyramid network.

[0132] Furthermore, the proposed candidate boxes for the target are obtained by fixing the size of regions of interest of different sizes in the feature map through a Region Generation Network (RPN).

[0133] Furthermore, after the extracted suggested candidate boxes are subjected to non-maximum suppression, the predicted bounding box coordinates of the target are obtained. The minimum bounding rectangle of the region where the target is located is drawn, and the minimum bounding rectangle is used as the predicted bounding box of the target to obtain the detection result of the target in the remote sensing image. This is achieved through a cascaded detection network.

[0134] Feature fusion and feature enhancement are performed on the extracted multi-scale features. A small target detection model is designed for remote sensing images, focusing on the preservation and extraction of small target features and contextual information, which can solve the problem of spatial information loss in small target detection in remote sensing images.

[0135] In small object detection tasks, Res2Net and Feature Pyramid Network (FPN) can improve the semantic information representation of shallow feature maps, effectively reduce the interference of the surrounding environment in complex backgrounds, and thus improve the detection performance of relevant targets.

[0136] Figure 1 In the diagram, the image in the lower left corner is the input image. Image Dehazing is the image dehazing module. Res2Net is the backbone network used for basic feature extraction in object detection tasks. PAFPN is the Neck module used for efficient fusion and enhancement of features output by the backbone network. The Feature Map is the result obtained after processing by the previous three modules, which is the feature map (the result generated by neural network convolution, representing a feature in neural space). The ROI pooling layer uses pooling methods on ROIs of different sizes in the input Feature Map to obtain a fixed-size output feature map. The RPN layer extracts candidate regions of the target and generates candidate boxes in the image. The remaining parameters include IOU_thr = 0.5, 0.6, 0.7. Positive sample and negative sample are used to increase the IoU value of the proposal stage by stage, so that the proposals resampled in the previous stage can adapt to the next stage with a higher threshold.

[0137] Res2Net: After performing a 1×1 convolution, the feature map is divided into s equal subsets, denoted by xi, where i∈{1,2,…,s}. Each feature subset xi has the same spatial size as the input feature map, but with 1 / s of channels. Each xi, except x1, has a corresponding 3×3 convolution, denoted by Ki(). The output of Ki() is denoted by yi. The feature subset xi is added to the output of Ki-1(), and then input into Ki().

[0138] PAFPN: C2-C5 are feature maps of different sizes obtained from Res2Net, P2-P5 are feature maps obtained by top-down and same-layer connections, and P2-P5 are feature maps obtained by bottom-up and same-layer connections.

[0139] The following simulation will further illustrate the effects of the present invention:

[0140] 1. Simulation experimental conditions:

[0141] Simulation experimental conditions for this invention: Server GPU: GeForce RTX2080Ti, 12G video memory.

[0142] The operating platform for this invention is Ubuntu 18.04, Python 3.7, and PyTorch-gpu 1.19.0.

[0143] 2. Simulation content and analysis of experimental results:

[0144] This invention simulates the detection of small targets in remote sensing images using existing technologies (YOLO v3, Mask R-CNN, Faster R-CNN, Sparse R-CNN, SSD, Retinanet, and Grid R-CNN) and methods based on multi-scale features and cascaded structures. The simulation uses the UJN-Traffic dataset.

[0145] The input images used in the simulation experiments of this invention are the UJN-Traffic dataset.

[0146] The UJN-Traffic dataset was built by Jinan University in 2020 and has been continuously expanded since then. Its images are extracted from the Gaofen-1 and Gaofen-2 satellites, and the current dataset contains 100,000 samples of 4 types.

[0147] The simulation experiment compares the detection results of the cascaded convolutional neural network model with the results of the image preprocessing and multi-scale feature extraction framework.

[0148] To verify the effectiveness of this invention for remote sensing image detection, the detection results were evaluated using a common target detection evaluation index. The calculation results are plotted in Table 1.

[0149] Table 1. Comparison of simulation results of the present invention on the UJN-Traffic dataset

[0150]

[0151]

[0152] Table 1 shows the simulation results, which compare the accuracy of our proposed method with that of classic target detection algorithms. It can be seen that our proposed method exhibits better robustness. Figures 3(a)-3(e) This is a diagram showing the test results of the present invention; Figure 4 It is a Precision-Recall plot based on the comparison of different scales of IOU.

[0153] Figure 3(a) is the original remote sensing image; Figure 3(b) is a schematic diagram of the remote sensing image after processing using the method of the present invention; Figure 3(c) is a schematic diagram of the remote sensing image after processing using the Faster R-CNN algorithm; Figure 3(d) is a schematic diagram of the remote sensing image after processing using the SSD algorithm; Figure 3(e) is a schematic diagram of the remote sensing image after processing using the YOLO v3 algorithm.

[0154] This invention provides AP for small targets in the UJN-Traffic dataset. 50 The success rate is 88.2%. Compared with other types of classic algorithms, AP and AP2 are significantly higher. S and AR S All metrics are higher than other types of detection algorithms, although AP 50 It is one percentage point lower than the SSD algorithm, but far superior to the SSD algorithm in AP.

[0155] As shown in Table 1, the improved method trains a detection model that can extract more challenging samples and significantly enhances the ability to perceive the location of small objects compared to other mainstream traditional object detection methods. For the same object, the prediction accuracy provided by this invention is significantly higher than other algorithms, thus enabling more effective detection of small objects of different sizes and backgrounds. Because the feature fusion process not only improves feature extraction capabilities but also preserves rich semantic information at the underlying level to some extent, the model exhibits better transferability. Transfer experiments demonstrate that the detection model learned by this invention possesses certain transferability and generality.

[0156] In summary, the remote sensing image small target detection method based on multi-scale features and cascaded structure has good performance in detecting small targets in complex backgrounds in high-resolution optical remote sensing images.

[0157] Example 2

[0158] This embodiment provides a small target detection system for remote sensing images based on multi-scale features;

[0159] A remote sensing image small target detection system based on multi-scale features includes:

[0160] The acquisition module is configured to acquire the remote sensing image to be detected;

[0161] The processing module is configured to perform dehazing and brightness enhancement processing on the remote sensing image to be detected.

[0162] The detection module is configured to input the brightness-enhanced image into the trained remote sensing image small target detection model to obtain the small target detection results of the remote sensing image;

[0163] The working principle of the trained remote sensing image small target detection model includes: multi-scale feature extraction of the brightness-enhanced remote sensing image; feature fusion and feature enhancement of the extracted multi-scale features to obtain a feature map; fixing the size of regions of interest of different sizes in the feature map to obtain proposed candidate boxes for the target; obtaining the predicted bounding box coordinates of the target after non-maximum suppression of the extracted proposed candidate boxes; drawing the minimum bounding rectangle of the target's region; using the minimum bounding rectangle as the target's predicted bounding box to obtain the detection result of the target in the remote sensing image.

[0164] It should be noted that the acquisition module, processing module, and detection module described above correspond to steps S101 to S103 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0165] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0166] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0167] Example 3

[0168] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0169] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0170] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0171] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0172] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0173] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0174] Example 4

[0175] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A remote sensing image small target detection method based on multi-scale features, characterized in that, The method comprises the following steps: acquiring a remote sensing image to be detected; performing defogging processing and brightness enhancement processing on the remote sensing image to be detected; inputting the image after brightness enhancement processing into a trained remote sensing image small target detection model to obtain a small target detection result of the remote sensing image; wherein the working principle of the trained remote sensing image small target detection model comprises: performing multi-scale feature extraction on the remote sensing image after brightness enhancement processing; performing feature fusion and feature enhancement on the extracted multi-scale features to obtain a feature map; performing size fixing on different sizes of regions of interest in the feature map to obtain a target proposal box; obtaining a target prediction bounding box coordinate after non-maximum suppression on the extracted target proposal box, drawing a minimum bounding rectangle of the region where the target is located, taking the minimum bounding rectangle as the target prediction bounding box, and obtaining a detection result of the remote sensing image target; the network structure of the trained remote sensing image small target detection model comprises: a Res2Net network and a feature pyramid network; the feature pyramid network comprises: an upsampling unit y1, an adder J1, an upsampling unit y2, an adder J2, an upsampling unit y3 and an adder J3 connected in turn from top to bottom; the input end of the upsampling unit y1 is connected with the output end of the fifth convolutional block conv_5 through a 1*1 convolutional layer M1; the input end of the adder J1 is connected with the output end of the fourth convolutional block conv_4 through a 1*1 convolutional layer M2; the input end of the adder J2 is connected with the output end of the third convolutional block conv_3 through a 1*1 convolutional layer M3; the input end of the adder J3 is connected with the output end of the second convolutional block conv_2 through a 1*1 convolutional layer M4; the output end of the 1*1 convolutional layer M1 is connected with the input end of a 3*3 convolutional layer L1; the output end of the adder J1 is connected with the input end of a 3*3 convolutional layer L2; the output end of the adder J2 is connected with the input end of a 3*3 convolutional layer L3; the output end of the adder J3 is connected with the input end of a 3*3 convolutional layer L4; the output end of the 3*3 convolutional layer L1 is connected with the input end of a connector Q1; the output end of the 3*3 convolutional layer L2 is connected with the input end of a connector Q2; the output end of the 3*3 convolutional layer L3 is connected with the input end of a connector Q3; the output end of the 3*3 convolutional layer L4 is connected with the input end of a 1*1 convolutional layer M8; the output end of the 1*1 convolutional layer M8 is connected with the input end of a downsampling unit Z3 and the input end of a 3*3 convolutional layer L8; the output end of the downsampling unit Z3 is connected with the input end of the connector Q3.

2. The multi-scale feature based remote sensing image small target detection method according to claim 1, characterized in that, The defogging processing specifically comprises: estimating atmospheric light intensity based on dark channel prior theory ; estimating dust and fog concentration based on the atmospheric light intensity according to the median filter principle ; and realizing image defogging based on the dust and fog concentration . The brightness enhancement processing specifically comprises: operating on a histogram of the defogged image, changing the mean and variance of the histogram, so as to realize brightness enhancement processing.

3. The multi-scale feature based remote sensing image small target detection method of claim 1, wherein, the network structure of the trained remote sensing image small target detection model comprises: a Res2Net network and a feature pyramid network; The Res2Net network comprises, from bottom to top, a first convolutional block conv_1, a second convolutional block conv_2, a third convolutional block conv_3, a fourth convolutional block conv_4 and a fifth convolutional block conv_5; the output end of the first convolutional block conv_1 outputs a feature map c1, the output end of the second convolutional block conv_2 outputs a feature map c2, the output end of the third convolutional block conv_3 outputs a feature map c3, the output end of the fourth convolutional block conv_4 outputs a feature map c4, and the output end of the fifth convolutional block conv_5 outputs a feature map c5.

4. The multi-scale feature based remote sensing image small target detection method of claim 1, wherein, The output end of the connector Q3 is connected with the input end of the 1*1 convolutional layer M7; The output end of the 1*1 convolutional layer M7 is connected with the input end of the 3*3 convolutional layer L7; The output end of the 1*1 convolutional layer M7 is connected with the input end of the down-sampling unit Z2, and the output end of the down-sampling unit Z2 is connected with the input end of the connector Q2; The output end of the connector Q2 is connected with the input end of the 1*1 convolutional layer M6; The output end of the 1*1 convolutional layer M6 is connected with the input end of the 3*3 convolutional layer L6; The output end of the 1*1 convolutional layer M6 is connected with the input end of the down-sampling unit Z1, and the output end of the down-sampling unit Z1 is connected with the input end of the connector Q1; The output end of the connector Q1 is connected with the input end of the 1*1 convolutional layer M5, and the output end of the 1*1 convolutional layer M5 is connected with the input end of the 3*3 convolutional layer L5; The 3*3 convolutional layers L5, L6, L7 and L8 are used for outputting feature maps, the output ends of the 3*3 convolutional layers L5, L6, L7 and L8 are connected with the input end of the region generation network RPN, and the output ends of the 3*3 convolutional layers L5, L6, L7 and L8 are connected with the input end of the ROI Pooling.

5. The multi-scale feature based remote sensing image small target detection method of claim 4, wherein, The output end of the region generation network RPN is connected with the cascade structure detection network respectively, and the cascade structure detection network outputs a detection result; The output end of the convolutional layer L1 outputs a feature map P5, the output end of the convolutional layer L2 outputs a feature map P4, the output end of the convolutional layer L3 outputs a feature map P3, and the output end of the convolutional layer L4 outputs a feature map P2; The output end of the convolutional layer L5 outputs a feature map N5, the output end of the convolutional layer L6 outputs a feature map N4, the output end of the convolutional layer L7 outputs a feature map N3, and the output end of the convolutional layer L8 outputs a feature map N2; The output end of the second convolutional block conv_2 is also connected with the input end of the up-sampling unit y1; The output end of the second convolutional block conv_2 is also connected with the input end of the connector Q1; The cascade structure detection network comprises T cascaded regressors, each of which is realized by using a detection network Region-CNN; except that the first detection network performs boundary box regression based on the candidate frame generated by the RPN network, each of the subsequent detection networks performs regression based on the boundary frame predicted by the previous detection network.

6. The multi-scale feature based remote sensing image small target detection method of claim 3, wherein, The first convolutional block conv_1 includes one convolutional layer; the second convolutional block conv_2 includes a down-sampling layer and three convolutional sub-blocks connected in sequence; the third convolutional block conv_3 includes a down-sampling layer and four convolutional sub-blocks connected in sequence; the fourth convolutional block conv_4 includes a down-sampling layer and 23 convolutional sub-blocks connected in sequence; and the fifth convolutional block conv_5 includes a down-sampling layer and three convolutional sub-blocks connected in sequence; each convolutional sub-block includes a first 1*1 convolutional layer, a second 3*3 convolutional layer and a third 1*1 convolutional layer connected in sequence, and the input end of the first convolutional layer is in residual connection with the output end of the third convolutional layer. The training process of the trained remote sensing image small target detection model includes: constructing a training set; the training set is a remote sensing image with known small target labels; inputting the training set into the remote sensing image small target detection model, and training the model by using the random gradient descent, error back propagation and learning rate decay; when the total loss function of the model no longer decreases or the number of iterations reaches the maximum number of iterations, the training is stopped, and the trained remote sensing image small target detection model is obtained; the model obtained each time needs to be evaluated in terms of precision and recall rate on a validation set, and the best model is tested on a test set to obtain its generalization ability.

7. The multi-scale feature based remote sensing image small target detection system according to any one of claims 1-6, wherein, It includes: an acquisition module configured to acquire a remote sensing image to be detected; a processing module configured to perform defogging processing and brightness enhancement processing on the remote sensing image to be detected; a detection module configured to input the image after brightness enhancement processing into the trained remote sensing image small target detection model to obtain a small target detection result of the remote sensing image; wherein the working principle of the trained remote sensing image small target detection model includes: performing multi-scale feature extraction on the remote sensing image after brightness enhancement processing; performing feature fusion and feature enhancement on the extracted multi-scale features to obtain a feature map; performing size fixing on different size regions of interest in the feature map to obtain a target proposal box; obtaining a target prediction bounding box coordinate after non-maximum suppression on the extracted proposal box; drawing a minimum bounding rectangle of the target region, taking the minimum bounding rectangle as the target prediction bounding box, and obtaining the detection result of the remote sensing image target.

8. An electronic device, comprising: a memory for non-transiently storing computer readable instructions; and a processor for running the computer readable instructions, wherein the computer readable instructions, when executed by the processor, perform the method of any one of claims 1-6.

9. A storage medium characterized by, non-transiently storing computer readable instructions, wherein when the non-transient computer readable instructions are executed by a computer, the instructions perform the method of any one of claims 1-6.

Images