A remote sensing image small target detection method and system based on feature fusion
By using an end-to-end feature fusion network that combines handcrafted features and deep features, the problem of low accuracy in small target detection in remote sensing images is solved, and more efficient small target detection results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINESE PEOPLES LIBERATION ARMY UNIT 75220
- Filing Date
- 2023-10-17
- Publication Date
- 2026-06-02
Smart Images
Figure CN117292273B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, specifically to a method and system for detecting small targets in remote sensing images based on feature fusion. Background Technology
[0002] With the development of remote sensing satellite technology and the improvement of remote sensing data resolution, images provide increasingly richer information, and the application scope of remote sensing images in military and civilian fields is gradually expanding. Remote sensing image target detection is a crucial step in converting images from data into information, thereby enabling image interpretation and target surveillance. Target detection based on remote sensing images is of great significance for resource exploration, natural disaster assessment, and military target detection and identification.
[0003] Due to the long imaging distance, remote sensing images often exhibit small target sizes. There are two definitions of small targets: one is absolute scale, where targets smaller than 32×32 pixels are defined as small in the COCO dataset; the other is relative scale, where a target smaller than 0.12% of the original image is considered small. Because small targets occupy fewer pixels in the original image, they carry very limited information, lacking texture, shape, color, and other appearance information, making small target detection a persistent challenge in the field of object detection. Especially in recent years, due to the powerful feature extraction capabilities of Convolutional Neural Networks (CNNs), the accuracy of deep learning-based object detection methods has continuously improved. However, the accuracy of small target detection is typically only half that of medium and large targets. One major reason is the small scale of small targets; the convolution and pooling operations of CNNs reduce the target's resolution, causing the feature information of small targets to gradually weaken, making them difficult to detect.
[0004] To address this problem, research has primarily focused on multi-scale modeling. Early object detection methods in this area mainly employed two approaches: first, constructing image pyramids of different resolutions from the original image, and then using a fixed-resolution classifier to detect targets on each pyramid layer; second, using classifiers of different resolutions on the original image to detect targets within a smaller window of classifiers. The classic Haar feature + Adaboost classifier and HOG feature + SVM classifier DPM object detection frameworks both use image pyramids to handle multi-scale targets. Early CNN object detection frameworks also adopted this approach, extracting features from each layer of the image pyramid using CNNs. However, the image pyramid method has high computational and memory requirements. Furthermore, it doesn't fundamentally solve the problem that "a single network structure cannot adapt to objects of all sizes." To improve the detection accuracy of small objects, images are enlarged and more layers are added, which to some extent impairs the detection accuracy of large objects. In deep learning object detection networks, feature maps of different depths correspond to different receptive fields and represent varying levels of information abstraction. Shallow feature maps have small receptive fields and rich spatial information, while deep feature maps have large receptive fields and strong semantic information. Based on this, researchers have proposed a layer-by-layer prediction approach to address object detection at different scales. SSD introduces multi-reference and multi-resolution detection techniques to detect objects of different scales at different layers of the network. However, since regression and classification require both semantic and positional features of an object, and semantic features are extracted from deep feature maps, the layered prediction network structure struggles to simultaneously capture the semantic and positional features of small objects. To address this issue, Feature Pyramid Network (FPN) uses a top-down approach to upsample deep feature maps and fuse them with shallow feature maps. The fused feature map is then used for prediction. The shallow feature map, which incorporates deep features, can simultaneously capture the positional and semantic features of small objects, offering a certain advantage in detecting small objects. R-SSD also addresses the issue of duplicate bounding boxes in SSD object detection by improving the feature fusion method, while also improving the performance of small object detection to some extent. While fusing feature maps of different depths can improve small target detection performance to some extent, directly fusing information of different densities can cause semantic conflicts, limiting the expression of multi-scale features and making small targets easily obscured by conflicting information. Furthermore, researchers have explored combining attention mechanisms with encoding / decoding mechanisms to improve small target detection capabilities; however, these methods all upsample small-scale target feature maps and treat them as large-scale targets, ignoring the differences between small-scale and large-scale target feature maps. DetNet increases the size of deep feature maps and uses dilation techniques to increase the receptive field, improving small target detection accuracy to some extent. However, its use of upsampling low-resolution feature maps to create feature pyramids introduces artifacts into the features, which affects target detection performance.
[0005] In summary, feature fusion can effectively improve the performance of small object detection. However, the above feature fusion methods all fuse features at different depths, completely abandoning handcrafted features. Handcrafted features typically represent a specific image attribute, such as color and gradient, focusing primarily on low-level image representation. Deep learning features are learned from sample datasets to represent images, and the learned features usually represent the contours and shapes of objects, focusing primarily on high-level image representation. Low-level and high-level image features often have some complementarity; therefore, it is worth considering merging handcrafted features into deep feature embedding models. In fact, fusing handcrafted feature information into CNN networks for training can effectively improve model performance. However, existing methods do not use fused features in the feature embedding layer, and each module is trained independently. Therefore, they cannot learn complementary components between features, and the models do not mutually promote each other's learning.
[0006] As is well known, large-scale datasets and powerful deep features are key factors for the success of deep learning-based object detection. Currently, object detection networks are typically pre-trained on the ImageNet classification dataset and then optimized using object detection datasets. Objects in classification datasets are usually relatively large, while those in detection datasets are smaller. Furthermore, CNNs struggle to learn scale invariance, making it difficult for CNN-learned models to effectively generalize to small objects, which is one reason for the low performance in small object detection. Traditional handcrafted features (such as SIFT and SURF) are invariant to rotation, scale, and illumination. Therefore, fusing handcrafted features with deep features is an effective way to improve the performance of deep learning-based small object detection. However, there is currently no effective solution for end-to-end fusion of handcrafted features and deep features to achieve small object detection. Summary of the Invention
[0007] The purpose of this invention is to provide a method and system for small target detection in remote sensing images based on feature fusion. This scheme designs an end-to-end target detection network that fuses handcrafted features and deep features. In the feature extraction stage, handcrafted features and deep features are extracted from candidate regions respectively, then feature transformations are performed on each, and finally, the two features are embedded to achieve target detection. Furthermore, this scheme achieves integrated training of the feature transformation, feature embedding, and target detection modules, improving target detection performance.
[0008] This invention is implemented as follows:
[0009] A method for detecting small targets in remote sensing images based on feature fusion, the specific steps of which are as follows:
[0010] S1: Input image
[0011] The remote sensing images to be detected are input into a network architecture mainly composed of convolutional neural networks and feature fusion networks;
[0012] S2: Image Processing
[0013] After entering the network architecture, the image will enter the convolutional neural network and the region proposal network. The region proposal network will work with the convolutional neural network to perform pooling operations on the region of interest of the image, and at the same time output the depth features of the region of interest of the image to the feature fusion network. After entering the region proposal network, the image will obtain the region of interest of the image, and the handcrafted features of the region of interest of the image will be calculated and output to the feature fusion network.
[0014] S3: Feature Fusion
[0015] Deep features and handcrafted features are fed into a feature fusion network, which includes two feature transformation modules and one feature embedding module. The feature transformation modules perform feature transformation and dimensionality reduction through fully connected layers, while the feature embedding module embeds handcrafted features and deep features through fully connected layers, thereby achieving the purpose of fusing handcrafted features and deep features.
[0016] S4: Object Detection
[0017] By fusing handcrafted features and deep features to perform classification and regression tasks, the accuracy of small object detection is effectively improved.
[0018] A small target detection system for remote sensing images based on feature fusion includes a convolutional neural network, a region proposal network, a feature fusion network, and an input image module. The input image module is connected to both the convolutional neural network and the region proposal network, and the outputs of the convolutional neural network and the region proposal network are connected to the feature fusion network.
[0019] Preferably, the region proposal network, in conjunction with a convolutional neural network, performs region pooling on the remote sensing image. The region proposal network is used to obtain the region of interest in the image and differentiate the handcrafted features within the region of interest in the remote sensing image.
[0020] Preferably, the feature fusion network includes two feature transformation modules and one feature embedding module, and the feature fusion network integrates multiple transformers and fusion units.
[0021] Preferably, the transformer in the feature fusion network is defined as follows:
[0022] Suppose that the deep features and handcrafted features of a certain target represent x respectively. d and x h The transformation is defined as follows:
[0023] F(x d ,x h )=[x d T W d ,x h T W h ]
[0024] In the formula, the symbol [x,y] represents the concatenation of vectors x and y, introducing a transformation function. It is for the purpose of standardizing the input data. Limited to low-rank matrices, F(x) is defined as the converter of x.
[0025] Preferably, the fusion unit in the feature fusion network is defined as follows:
[0026] Suppose that the deep features and handcrafted features of a certain target represent x respectively. d and x h The transformation is defined as follows:
[0027] φ(x d ,x h )=F(x d ,x h ) T M
[0028] In the formula, It is the embedding matrix, and at the same time, by constraining M to be a low-rank matrix, it performs dimensionality reduction representation of high-dimensional concatenated vectors. In addition, in order to obtain low-dimensional feature embedding, the rank of M must be less than the rank of W, and φ(·) is defined as a merger of multiple transformed features.
[0029] Preferably, GoogLeNet, which has fewer network structure parameters, is selected as the CNN feature learning framework for the deep feature embedding aspect.
[0030] Preferably, the handcrafted feature is the HOG feature.
[0031] Preferably, the convolutional neural network uses the GoogLeNet network architecture.
[0032] Preferably, the transformer is an autoencoder, and the fusion unit is integrated into the feature embedding module.
[0033] Compared with the prior art, the beneficial effects of the present invention are:
[0034] This method designs a network that fuses deep features and handcrafted features to improve the feature discrimination ability of targets. Experiments were conducted on the DOTA dataset to verify the results, which show that compared with classic target detection algorithms, this method effectively improves the detection performance of small targets.
[0035] The experimental table is as follows:
[0036]
[0037] Attached Figure Description
[0038] Figure 1 This is a flowchart of the small target detection process of the present invention;
[0039] Figure 2 This is a structural block diagram of the design scheme of the present invention. Detailed implementation method:
[0040] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0041] The following description, in conjunction with the accompanying drawings and specific embodiments, provides further details:
[0042] Please see Figure 1 A method for detecting small targets in remote sensing images based on feature fusion is described, and the specific steps of the method are as follows:
[0043] S1: Input image
[0044] The remote sensing images to be detected are input into a network architecture mainly composed of convolutional neural networks and feature fusion networks;
[0045] S2: Image Processing
[0046] After entering the network architecture, the image will enter the convolutional neural network and the region proposal network. The region proposal network will work with the convolutional neural network to perform pooling operations on the region of interest of the image, and at the same time output the depth features of the region of interest of the image to the feature fusion network. After entering the region proposal network, the image will obtain the region of interest of the image, and the handcrafted features of the region of interest of the image will be calculated and output to the feature fusion network.
[0047] S3: Feature Fusion
[0048] Deep features and handcrafted features are fed into a feature fusion network, which includes two feature transformation modules and one feature embedding module. The feature transformation modules perform feature transformation and dimensionality reduction through fully connected layers, while the feature embedding module embeds handcrafted features and deep features through fully connected layers, thereby achieving the purpose of fusing handcrafted features and deep features.
[0049] S4: Object Detection
[0050] By fusing handcrafted features and deep features to perform classification and regression tasks, the accuracy of small object detection is effectively improved.
[0051] Please see Figure 2 A small target detection system for remote sensing images based on feature fusion is disclosed, comprising a convolutional neural network (CNN), a region proposal network (RPN), an input image module, and a feature fusion network. The input image module is connected to both the CNN and the RPN. The outputs of the CNN and the RPN are connected to the feature fusion network. The feature fusion network includes two feature transformation modules and one feature embedding module. The feature transformation modules perform feature transformation and dimensionality reduction through fully connected layers, while the feature embedding module embeds handcrafted and deep features through fully connected layers. This network achieves classification and regression tasks by fusing features, improving the accuracy of small target detection. The RPN works with the CNN to perform region-of-interest (ROI) pooling on the remote sensing image. The RPN is used to obtain the RPI of the image and differentiate the handcrafted features within the RPI. The feature fusion network integrates multiple transformers and fusion units.
[0052] Definition of a transformer in a feature fusion network:
[0053] Suppose that the deep features and handcrafted features of a certain target represent x respectively. d and x h The transformation is defined as follows:
[0054] F(x d ,x h )=[x d T W d ,x h T W h ]
[0055] In the formula, the symbol [x,y] represents the concatenation of vectors x and y, introducing a transformation function. It is for the purpose of standardizing the input data. Limited to low-rank matrices, F(x) is defined as the converter of x.
[0056] Definition of fusion unit in feature fusion network:
[0057] Suppose that the deep features and handcrafted features of a certain target represent x respectively. d and x h The transformation is defined as follows:
[0058] φ(x d ,x h )=F(x d ,x h ) T M
[0059] In the formula, It is the embedding matrix, and at the same time, by constraining M to be a low-rank matrix, it performs dimensionality reduction representation of high-dimensional concatenated vectors. In addition, in order to obtain low-dimensional feature embedding, the rank of M must be less than the rank of W, and φ(·) is defined as a merger of multiple transformed features.
[0060] Based on the above analysis, the entire model includes a depth feature representation x of an image. d and handmade feature representation x h With unknown parameter W d and W h Feature Transformer F(x) d ,x h ), feature fusion φ(x) with unknown parameter M d ,x h ) and multiple loss functions.
[0061] For deep feature embedding, GoogLeNet, with its fewer network structure parameters, was chosen as the CNN feature learning framework. Considering the heterogeneity between Histogram of Oriented Gradients (HOG) handcrafted features and CNN features, the handcrafted features in this scheme are HOG features, and HOG detectors have long been a crucial foundation for many object detectors and various computer vision applications. To balance feature invariance (including translation, scale, illumination, etc.) and non-linearity (distinguishing different object categories), the HOG descriptors are designed to be computed on a dense, uniformly spaced grid of cells, and overlapping local contrast normalization (on the "blocks") is used to improve accuracy.
[0062] Please see Figure 2 The convolutional neural network uses the GoogLeNet network architecture, with an autoencoder as the transformer and the fusion unit integrated into the feature embedding module.
[0063] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for detecting small targets in remote sensing images based on feature fusion, characterized in that, The specific steps of this method are as follows: S1: Input image The remote sensing images to be detected are input into a network architecture mainly composed of convolutional neural networks and feature fusion networks; S2: Image Processing After entering the network architecture, the image will enter the convolutional neural network and the region proposal network. The region proposal network will work with the convolutional neural network to perform pooling operations on the region of interest of the image, and at the same time output the depth features of the region of interest of the image to the feature fusion network. After entering the region proposal network, the image will obtain the region of interest of the image, and the handcrafted features of the region of interest of the image will be calculated and output to the feature fusion network. S3: Feature Fusion Deep features and handcrafted features are fed into a feature fusion network, which includes two feature transformation modules and one feature embedding module. The feature transformation modules perform feature transformation and dimensionality reduction through fully connected layers, while the feature embedding module embeds handcrafted features and deep features through fully connected layers, thereby achieving the purpose of fusing handcrafted features and deep features. S4: Object Detection By fusing handcrafted features and deep features to perform classification and regression tasks, the accuracy of small object detection is effectively improved.
2. A small target detection system for remote sensing images based on feature fusion, characterized in that, The system includes a convolutional neural network (CNN), a region proposal network (RPN), a feature fusion network, and an input image module. The input image module is connected to both the CNN and the RPN. The outputs of the CNN and the RPN are connected to the feature fusion network. The feature fusion network includes two feature transformation modules and one feature embedding module. The feature fusion network integrates multiple transformers and a fusion unit. The transformers in the feature fusion network are defined as follows: Suppose that the deep features and handcrafted features of a certain target represent... and The transformation is defined as follows: , In the formula, the symbol Representing vectors and Cascades, introducing transformation functions It is for the purpose of standardizing the input data. Limited to low-rank matrices, Defined as Converter; Definition of the fusion unit in the feature fusion network: Suppose that the deep features and handcrafted features of a certain target represent... and The transformation is defined as follows: , In the formula, It is an embedding matrix, and is subject to constraints. To reduce the dimensionality of high-dimensional concatenated vectors, a low-rank matrix is used. Furthermore, to obtain low-dimensional feature embeddings... The rank must be less than rank, Defined as a merger of multiple transformation features.
3. The remote sensing image small target detection system based on feature fusion according to claim 2, characterized in that, The region proposal network, in conjunction with a convolutional neural network, performs region pooling on the remote sensing image. The region proposal network is used to obtain the region of interest in the image and differentiate the handcrafted features in the region of interest of the remote sensing image.
4. The remote sensing image small target detection system based on feature fusion according to claim 2, characterized in that, For the deep feature embedding aspect, GoogLeNet, which has fewer network structure parameters, is selected as the CNN feature learning framework.
5. A small target detection system for remote sensing images based on feature fusion according to claim 2, characterized in that, The handcrafted features used are HOG features.
6. A small target detection system for remote sensing images based on feature fusion according to claim 2, characterized in that, The convolutional neural network uses the GoogLeNet network architecture.
7. A small target detection system for remote sensing images based on feature fusion according to claim 4, characterized in that, The transformer is an autoencoder, and the fusion unit is integrated into the feature embedding module.