A method for small target detection in remote sensing images guided by image pyramid

Through the image pyramid guidance network, the backbone network and image pyramid features are integrated, which solves the problem of low detection accuracy of small targets, achieves more efficient feature extraction and fusion, and improves the performance of small target detection.

CN113112450BActive Publication Date: 2024-11-05ZHEJIANG UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110246743.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-03-05
Publication Date
2024-11-05
Estimated Expiration
2041-03-05

AI Technical Summary

Technical Problem

The existing target detection method has poor effect on small target detection accuracy. The main reason is that small targets account for a small proportion in the image. After multi-layer convolution, only part of the feature information is included in the deep feature map, and the traditional method has parameter quantity. The problem of large-scale characteristic span.

Method used

The image pyramid guidance network is used to integrate the features of the backbone network with the corresponding hierarchical features of the image pyramid model, build a multi-level feature fusion path, enhance the combination of deep semantic information and shallow spatial information, and provide more small-target information.

Benefits of technology

It effectively improves the accuracy of small object detection, reduces the redundancy of feature extraction and classification, and improves detection performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113112450B_ABST
    Figure CN113112450B_ABST
Patent Text Reader

Abstract

A remote sensing image small target detection method guided by an image pyramid comprises: step 1: constructing a backbone network; step 2: constructing an image pyramid guided network, constructing an image pyramid guided network, and extracting image pyramid feature information; step 3: fusing backbone and image pyramid features, obtaining a backbone feature map R and an image pyramid feature map F based on steps 1 and 2, and fusing the image pyramid feature information into the backbone feature information as additional information; step 4: multi-level feature fusion; after obtaining the multi-scale feature O after the backbone feature and the image pyramid feature are fused in step 3, a top-down fusion path is constructed by using the side horizontal connection mode of the feature pyramid network to fuse the deep semantic information into the shallow rich spatial information. The present invention improves the accuracy of small target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of target detection, and in particular relates to a small target optimized target detection method. Background Art

[0002] Object detection is an important research direction in the field of computer vision. It combines the two major tasks of target positioning and target recognition and is widely used in intelligent monitoring, image retrieval, intelligent transportation, scene recognition and other fields. Traditional object detection algorithms mostly use manually designed features. The main steps are as follows: (1) Use a sliding window strategy to traverse the entire image and generate multiple candidate boxes; (2) Use different feature extraction methods to calculate the features in the candidate boxes and generate feature vectors; (3) Use machine learning methods such as support vector machines to classify the feature vectors and obtain the target detection results (Zhang Y, Li B, Lu H, et al. Sample-specific SVM learning for person re-identification [C] / / IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016.). Traditional algorithms are intuitive and simple, but generating candidate boxes will produce redundant windows, which seriously affects the speed and performance of subsequent feature extraction and classification.

[0003] At present, deep learning methods can adaptively learn image feature information by designing various network structures to achieve target recognition and positioning. Compared with traditional methods, they have greatly improved efficiency and accuracy and have been widely used in the field of computer vision and achieved great success. Object detection methods based on deep learning are mainly divided into single-stage models and two-stage models (Liu W, Anguelov D, Erhan D, et al. SSD: Single shot multi box detector [C] / / European Conference on Computer Vision, Amsterdam, NL, 2016: 21–37; Ren S, He K, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 39 (6): 1137-1149.). Among them, the single-stage model does not need to generate candidate boxes and directly outputs the location and category probability of the target; the two-stage model needs to first extract candidate boxes from the position where the target may exist on the feature map, and then classify the candidate boxes.

[0004] The above two object detection methods have achieved good performance in detection accuracy and efficiency for medium and large objects in scenes on general datasets, but their accuracy for small object detection is still low. The main reasons are:

[0005] (1) The target to be detected has only one or a few pixels in the entire image, and the amount of information that can be reflected by the corresponding pixel area is limited.

[0006] (2) In order to obtain deep semantic information, the network model needs to go through multiple layers of convolution and pooling operations to expand the receptive field. Originally, small objects account for a very small proportion of the image. After multiple convolutions, only part of the feature information is contained in the deep feature map, and it may even be filtered out during the convolution process.

[0007] The IPG-Net network introduces an image pyramid to provide more spatial information for the backbone network to solve the problem of low accuracy in small target detection (Liu Z, Gao G, Sun L, et al. IPG-Net: Image Pyramid Guidance Network for Small Object Detection [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020: 1026-1027.). However, its feature extraction module uses a 7*7 convolution kernel, resulting in a large number of parameters; a fixed number of residual blocks is used, resulting in a large span between the extracted features and the backbone network features.

[0008] Aiming at the difficulty of small target detection and the shortcomings of current methods, the present invention uses the currently popular MaskR-CNN network model as the basic framework for target extraction (He K, Gkioxari G, Dollar P, et al. Mask R-CNN [C] / / International Conference on Computer Vision (ICCV), Venice, IT, 2017: 2980-2988.), and uses the IPG-Net network structure further optimized for small targets as the backbone feature extraction network, and proposes an improved small target detection method. By constructing an image pyramid guided network to optimize its feature extraction and fusion modules, deep features and shallow features are combined with each other, providing more spatial information for the feature pyramid of the backbone network, and solving the problem of low small target detection accuracy. Summary of the invention

[0009] In view of the problem that in the feature extraction process of the Mask R-CNN algorithm, spatial information is lost after dozens or even more than a hundred layers of convolution operations, resulting in low small target detection accuracy, the present invention proposes an image pyramid guided network that fuses the features extracted by the backbone network with the features of the corresponding levels of the image pyramid model, so that each stage of the feature pyramid of the backbone network provides more small target information, and then fuses the features into a small target optimization method in the Mask R-CNN network framework.

[0010] According to the above principles, a remote sensing image small target detection method guided by an image pyramid of the present invention comprises the following steps:

[0011] Step 1: Build a backbone network;

[0012] The MaskR-CNN backbone network is constructed using a standard ResNet network. The ResNet network includes five levels of feature extraction modules, but in order to maximize the retention of shallow small object information while obtaining deep semantic information, the last level of ResNet is cancelled and only four levels are used. The first level uses a convolution layer with a stride of 2, and the resulting feature map is 1 / 2 of the original image resolution; the second level uses a maximum pooling layer with a stride of 2 to obtain a feature map that is 1 / 4 of the original image size; the third to fourth levels all use convolution layers with a stride of 2 to extract features, and the final output feature map is 1 / 16 of the original image. The feature map extracted by the ResNet backbone network is represented as R:

[0013] R={R1,R2,...,R n} (1)

[0014] Here, n represents the number of levels of the network.

[0015] Step 2: Build an image pyramid guided network;

[0016] This step constructs an image pyramid guidance network to extract image pyramid feature information, which specifically includes the following two sub-steps:

[0017] Step 2.1: Build an image pyramid;

[0018] Input image and construct its pyramid set I:

[0019] I={I1,I2,...,I n} (2)

[0020] Among them, n represents the number of levels of the image pyramid, which corresponds to the number of levels of the ResNet backbone network in step 1. The resolution of I1 in the image pyramid set is the same as the original input image, which is H×W, and then the length and width are gradually reduced to 1 / 2 of the original.

[0021] Step 2.2: Construct feature extraction module;

[0022] The feature extraction function module of the image pyramid guided network is to capture the details of small objects, mainly obtained from the shallow network. It mainly consists of the following two parts:

[0023] (1) First, three 3*3 convolution cascade operations are performed. This structure obtains the same receptive field as the 7*7 convolution operation while reducing the number of parameters, enhancing the nonlinear expression ability of the network and preventing overfitting. This increases the fine-grained feature extraction capability of the shallow convolution filter. Then a maximum pooling operation with a stride of 2 is performed.

[0024] (2) After the pooling operation, the image pyramid features extracted at each level need to be realized through the operation of residual blocks. The design of the number of residual blocks needs to take into account that as the number of levels in the Mask R-CNN backbone network increases, the number of convolutional layers also increases. The first level of the backbone network consists of 3 residual blocks (a total of 9 convolutional layers); the second level adds 4 residual blocks on the basis of the first level (that is, 12 convolutional layers are added, a total of 21 convolutional layers); the third level adds 23 residual blocks on the basis of the second level (that is, 69 convolutional layers are added, a total of 90 convolutional layers). A single residual block can provide richer spatial information, but as the level increases, the span of the image pyramid features and backbone features at the same level becomes larger and larger. In particular, the third level of ResNet101 has 23 residual blocks. Fusing two features with such a large span is not beneficial for optimizing the feature extraction results. Therefore, when extracting image pyramid features, the present invention adds one residual block at each level as the number of backbone network levels increases, that is, one residual block in the first stage, two residual blocks in the second stage, and three residual blocks in the third stage. This shortens the span between two features and retains spatial information and features of small objects.

[0025] Image pyramid feature map F extracted by the image pyramid guidance network:

[0026] F={F1,F2,...,F n} (3)

[0027] Among them, n represents the number of levels of the image pyramid, which is consistent with the number of levels of the backbone network.

[0028] Step 3: Fusion of backbone and image pyramid features;

[0029] Based on steps 1 and 2, we obtain the backbone feature map R and the image pyramid feature map F, and use the image pyramid feature information as additional information to fuse it into the backbone feature information. This fusion includes the following three sub-steps:

[0030] Step 3.1: Upsampling of pyramid feature maps;

[0031] The image pyramid feature map F i Perform 2 times upsampling to obtain the backbone feature map R i The result of consistent feature dimensions F i '.

[0032] Step 3.2: Backbone network integration;

[0033] The F obtained in step 3.1 i 'With the backbone feature map R i To perform the fusion:

[0034]

[0035] Among them, Conv 1*1 Represents a convolution function with a convolution kernel size of 1, used to change F' i The channel dimension is similar to R i The number of channels remains the same. Represents the element-by-element addition fusion operation, O' i Indicates the fusion result.

[0036] Step 3.3: Fusion feature map generation;

[0037] The O' obtained in step 3.2 i Perform feature extraction:

[0038] O i =Conv 3*3 (O' i ) (5)

[0039] Among them, O i Represents the result of feature extraction, Conv 3*3 The convolution function with a convolution kernel size of 3 is used to eliminate the aliasing effect of feature map distortion caused by discontinuous features after superposition due to upsampling, and generate the final fused feature map O:

[0040] O={O1,O2,...,O n} (6)

[0041] Step 4: Multi-level feature fusion;

[0042] After obtaining the multi-scale feature O after the fusion of the backbone feature and the image pyramid feature in step 3, a top-down fusion path is constructed using the side lateral connection method of the feature pyramid network to fuse the deep semantic information into the shallow rich spatial information. However, the feature map finally generated only contains the feature information of the current layer and deeper layers, but does not contain the shallower information, and cannot fully utilize the information of feature maps of all sizes. Therefore, based on the original connection from deep to shallow layers, a feature fusion network structure from shallow to deep is added. Specifically, it includes the following two sub-steps:

[0043] Step 4.1: Feature fusion from deep layer to shallow layer;

[0044] After obtaining the multi-scale feature O after the fusion of the backbone and image pyramid features in step 3, add a new feature map P n , whose size is similar to O n Consistent, the channel dimension is C; then P n Perform 2x upsampling 2↑, O n-1Use the convolution function Conv with a convolution kernel size of 1 1*1 Change its channel dimension, and finally add the two element by element Fusion generates a new feature map P n-1 :

[0045]

[0046] Repeat the above steps to generate a multi-scale feature map P:

[0047] P={P1,P2,...,P n} (8)

[0048] Step 4.2: Feature fusion from shallow to deep layers;

[0049] After generating multi-scale features P from deep to shallow layers, add a new feature map N1, whose size is consistent with P1 and channel dimension is C, and then i Through 3*3 convolution Conv with stride 2 3*3 Downsample to half of its original size, then connect it horizontally to P i+1 Element-wise addition The fusion generates a new feature map, which is then passed through the convolution function Conv with a convolution kernel size of 3. 3*3 Get N i+1 :

[0050]

[0051] Repeat the above steps to generate the final multi-level feature fusion map N:

[0052] N={N1,N2,...,N n} (10).

[0053] The advantage of the above scheme adopted by the present invention is that the present invention constructs an image pyramid to guide the network, optimizes its feature extraction and feature fusion modules, provides more small target information for each stage of the backbone network feature pyramid, and effectively solves the problem that the general target detection method is not effective in small target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Attached Figure 1 is a flow chart of the method of the present invention.

[0055] Attached Figure 2 It is the location of the study area in the embodiment of the present invention. Figure 2 (a) is the location of Zhejiang Province; Figure 2 (b) is a satellite image of the Yongkang Experimental Area.

[0056] Attached Figure 3It is a network structure optimized for small targets. Figure 3 (a) is the feature extraction module of the image pyramid guided network; Figure 3 (b) is the fusion of backbone and image pyramid features; Figure 3 (c) is the feature fusion from deep layer to shallow layer; Figure 3 (d) is the feature fusion from shallow to deep layers.

[0057] Attached Figure 4 is the recognition result of the current embodiment. Figure 4 (a) is the pine wood nematode disease sample selected by superimposing the original image; Figure 4 (b) the test results and evaluation thereof of the present invention; Figure 4 (c) is a local image and manually labeled samples; Figure 4 (d) is part of the test results and their evaluation of the present invention. DETAILED DESCRIPTION

[0058] In order to facilitate those skilled in the art to understand and implement the present invention, the following embodiments and appended Figure 1 The invention process of the present invention is further described in detail. It should be understood that the specific embodiments described herein are only used to explain the present invention, not to limit the present invention, and the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Therefore, based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0059] The study area of ​​this embodiment is part of Yongkang City, Hangzhou City, Zhejiang Province (such as Figure 2 As shown in Figure 2, the Yongkang test area selected the world view 3 satellite image, which was acquired on October 11, 2018. In the specific experiment, the test area image was cropped into an image block of 512×512 pixels. In order to reduce the splicing gap phenomenon when predicting the large image after cropping, the horizontal and vertical overlapping areas of 60 pixels were maintained during cropping. The dead plants of pine wood nematodes in the Yongkang test area are reddish brown. The canopy diameter of adult pine plants is generally about 1.5m, while the optimal spatial resolution of the worldview 3 multispectral image is only 1.2m. A single plant is a small target of one or several pixels on the remote sensing image. Therefore, when using the general target detection method to identify dead plants, there are problems such as low recognition accuracy.

[0060] The present invention discloses a remote sensing image small target detection method guided by an image pyramid, which aims to solve the problem of poor small target detection performance. The specific implementation of the present invention is described in conjunction with the accompanying drawings. The process is as follows:

[0061] Step 1: Build a backbone network;

[0062] This embodiment uses ResNet101 to construct the backbone network. The resolution of the original image is 512×512. The backbone network uses four levels of feature extraction modules, and the final output feature map is 1 / 16 of the size of the original image.

[0063] The final backbone network feature maps R1, R2, R3, and R4 have feature sizes of 256×256, 128×128, 64×64, and 32×32, and channel dimensions of 64, 256, 512, and 1024, respectively.

[0064] Step 2: Build an image pyramid guided network;

[0065] The input of the image pyramid guidance network consists of an image pyramid set. The initial I1 is consistent with the original image size, which is 512×512. Subsequently, the resolution of I2, I3, and I4 images is reduced by 2 times step by step.

[0066] The design of the feature extraction module of the image pyramid guided network is shown in step 2.2 (such as Figure 3 (a) As shown in the feature extraction method, the obtained image pyramid feature maps F2, F3, and F4 have feature sizes of 64×64, 32×32, and 16×16, and channel dimensions of 512, 1024, and 2048, respectively.

[0067] Step 3: Fusion of backbone and image pyramid features;

[0068] The image pyramid feature map F2 is upsampled by 2 times to obtain the result F'2 with the same feature dimension as the backbone feature map R2. The channel dimension of F'2 is changed by a convolution function with a convolution kernel of 1 to make it consistent with R2, and then element-by-element addition is performed for fusion. Finally, a convolution function with a convolution kernel of 3 is used to remove the aliasing effect (such as Figure 3 (b)), generate the final fused feature map O2.

[0069] Repeat the above steps to generate the final fused feature maps O2, O3, and O4, with feature sizes of 128×128, 64×64, and 32×32, and channel dimensions of 256, 512, and 1024, respectively.

[0070] Step 4: Multi-level feature fusion;

[0071] Add a new feature map P4, whose size is the same as O4, 32×32, and the channel dimension is 256. Then upsample P4 by 2 times, and change the channel dimension of O3 to 256 using the convolution function with a convolution kernel size of 1. Then add the two element by element and fuse them to generate a new feature map P3. Repeat the above steps (such as Figure 3(c)), generating multi-scale feature maps P1, P2, P3, and P4 from deep to shallow layers.

[0072] After generating multi-scale features from deep to shallow layers, add a new feature map N1, whose size is consistent with P1, 256×256, and the channel dimension is 256. Then N1 is downsampled to half of its original size through a 3*3 convolution with a stride of 2 to obtain X1. Then, through the side horizontal connection, X1 and P2 are added element by element to generate a new feature map, and then N2 is obtained through a convolution function with a convolution kernel size of 3. Repeat the above steps (such as Figure 3 (d)), the final multi-level feature fusion maps N1, N2, N3, N4 are generated, whose sizes are 256×256, 128×128, 64×64, 32×32 respectively, and the channel dimension is 256.

[0073] The above process mainly illustrates the process of optimizing the network structure for small target detection in the present invention. In the specific experiment, due to the small size of the dead plants themselves, even experienced people interpret them, and it is difficult to accurately judge in some cases. Therefore, the samples are divided into three categories according to the credibility: high credibility (High), that is, people can be sure that they are dead plants; medium credibility (Mid), which is more similar to dead plants; low credibility (Low), that is, some features support that it is a dead plant, but some support that it is not a dead plant. During the training process, 75% of the samples are randomly selected as the training set, and the remaining 25% are used as the test set to verify the reliability of the algorithm. The experimental results are as follows. Figure 4 As shown, it can be seen that the detection rate and accuracy rate in this area are both high, which illustrates the effectiveness of the method.

[0074] Compared with the results of manual visual interpretation, the accuracy of the proposed method reaches 0.85. Under the same experimental parameter settings, the accuracy of the proposed method is 2% higher than that of the original Mask R-CNN method, which illustrates the effectiveness of the small target optimization detection results of the present invention.

[0075] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also provides equivalent technical means that can be thought of by those skilled in the art based on the inventive concept.

Claims

1. A remote sensing image small target detection method guided by an image pyramid, comprising the following steps: Step 1: Build a backbone network; The Mask R-CNN backbone network is constructed using the standard ResNet network. The ResNet network includes five levels of feature extraction modules, but in order to maximize the retention of shallow small object information while obtaining deep semantic information, the last level of ResNet is cancelled and only four levels are used. The first level uses a convolution layer with a stride of 2, and the feature map obtained is 1 / 2 of the original image resolution. The second level uses a maximum pooling layer with a stride of 2 to obtain a feature map with a size of 1 / 4 of the original image. The third to fourth levels all use convolution layers with a stride of 2 to extract features, and the final output feature map is 1 / 16 of the original image. The feature map extracted by the ResNet backbone network is represented as R: R={R 1 ,R 2 ,...,R n } (1) in, n represents the number of levels of the network; Step 2: Build an image pyramid guided network; This step constructs an image pyramid guidance network to extract image pyramid feature information, which specifically includes the following two sub-steps: Step 2.1: Build an image pyramid; Input image and construct its pyramid set I: I={I 1 ,I 2 ,…,I n } (2) Where n represents the number of levels of the image pyramid, which corresponds to the number of levels of the ResNet backbone network in step 1; 1 The resolution is the same as the original input image, which is H×W, and then the length and width are gradually reduced to 1 / 2 of the original; Step 2.2: Construct feature extraction module; The feature extraction function module of the image pyramid guided network is to capture the details of small targets, mainly obtained from the shallow network; it mainly consists of the following two parts: (1) First, three 3*3 convolution cascade operations are performed. While obtaining the same receptive field as the 7*7 convolution operation, the number of parameters is reduced, the nonlinear expression ability of the network is enhanced, and overfitting is prevented. This increases the fine-grained feature extraction capability of the shallow convolution filter. Then, a maximum pooling operation with a stride of 2 is performed. (2) After the pooling operation, the image pyramid features extracted at each level need to be realized through the residual block operation. When extracting the image pyramid features, as the number of backbone network layers increases, one residual block is added to each level, that is, one residual block in the first stage, two residual blocks in the second stage, and three residual blocks in the third stage. Image pyramid feature map F extracted by the image pyramid guidance network: F={F 1 ,F 2 ,...,F n } (3) Among them, n represents the number of levels of the image pyramid, which is consistent with the number of levels of the backbone network; Step 3: Fusion of backbone and image pyramid features; Based on steps 1 and 2, we obtain the backbone feature map R and the image pyramid feature map F. We use the image pyramid feature information as additional information and fuse it into the backbone feature information. This fusion includes the following three sub-steps: Step 3.1: Upsampling of pyramid feature maps; The image pyramid feature map F i Perform 2 times upsampling to obtain the backbone feature map R i The result of consistent feature dimensions F i '; Step 3.2: Backbone network integration; The F obtained in step 3.1 i 'With the backbone feature map R i To perform the fusion: Among them, Conv 1*1 Represents a convolution function with a convolution kernel size of 1, used to change F i 'Channel dimension, so that it is consistent with R i The number of channels remains the same. Represents the element-by-element addition fusion operation, O' i represents the fusion result; Step 3.3: Fusion feature map generation; The O' obtained in step 3.2 i Re-extract features: THE i =Conv 3*3 (THE' i ) (5) Among them, O i Represents the result of feature extraction, Conv 3*3 The convolution function with a convolution kernel size of 3 is used to eliminate the aliasing effect of feature map distortion caused by discontinuous features after superposition due to upsampling, and generate the final fused feature map O: O={O 1 ,O 2 ,...,O n } (6) Step 4: Multi-level feature fusion; After obtaining the multi-scale feature O after the backbone feature and the image pyramid feature are fused in step 3, a top-down fusion path is constructed by using the side lateral connection method of the feature pyramid network to fuse the deep semantic information into the shallow rich spatial information; however, the feature map finally generated only contains the feature information of the current layer and deeper layers, but does not contain the information of the shallower layers, and cannot fully utilize the information of feature maps of all sizes; therefore, on the basis of the original connection from deep to shallow layers, a feature fusion network structure from shallow to deep is added; specifically, it includes the following two sub-steps: Step 4.1: Feature fusion from deep layer to shallow layer; After obtaining the multi-scale feature O after the fusion of the backbone and image pyramid features in step 3, add a new feature map P n , whose size is similar to O n Consistent, the channel dimension is C; then P n Perform 2x upsampling 2↑, O n-1 Use the convolution function Conv with a convolution kernel size of 1 1*1 Change its channel dimension, and finally add the two element by element Fusion generates a new feature map P n-1 : Repeat the above steps to generate a multi-scale feature map P: P={P 1 ,P 2 ,...,P n } (8) Step 4.2: Feature fusion from shallow to deep layers; After generating multi-scale features P from deep to shallow layers, add new feature maps N 1 , its size is similar to P 1 Consistent, the channel dimension is C, then N i Through 3*3 convolution Conv with stride 2 3*3 Downsample to half of its original size, then connect it horizontally to P i+1 Element-wise addition The fusion generates a new feature map, which is then passed through the convolution function Conv with a convolution kernel size of 3. 3*3 Get N i+1 : Repeat the above steps to generate the final multi-level feature fusion map N: N={N 1 ,N 2 ,...,N n } (10)。