A feature enhancement network optimization method and system based on self-decoding

Abstract

The present invention discloses a feature enhancement network optimization method and system based on self-decoding, comprising the following steps: S1, constructing a self-decoder module; S2, introducing the self-decoder module into a general object detection model based on a convolutional neural network, and introducing the constructed self-decoder module into the backbone network of the object detection model; S3, dividing the object detection data set; S4, performing prediction processing on the images in the data set; S5, using the training data to perform collaborative training with the object detection model and decoder; S6, removing the self-decoder; S7, feeding verification data into the detection model for detection; S8, obtaining detection results, mapping the detection results obtained in S7 to the corresponding positions of the original image, and annotating the predicted category information. The present invention has a strong embedding type, does not increase the prediction time, and is comparable to the prediction results of deeper backbone networks.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image building detection, and in particular to a feature enhancement network optimization method and system based on self-decoding. Background Art

[0002] With the rapid development of high-resolution satellites, it has become important to obtain useful target detection results from remote sensing images and thus understand satellite images, and it has become a research hotspot in the current remote sensing field. Among them, building detection is of great significance to urban planning, disaster detection, ecological environment management and military. Due to the powerful feature extraction capabilities of convolutional neural networks, target detection algorithms based on deep convolutional neural networks are currently popular in the field of building detection. However, compared with images of ordinary scenes, building detection in satellite images faces the following challenges: the scene complexity of building images is higher, and images of different phases and resolutions are very different. This leads to huge challenges in building image detection.

[0003] In 2014, Girshick et al. published the paper "Rich feature hierarchies for accurate object detection and semantic segmentation" (In Proceedings of the IEEE conference on computer vision and pattern recognition), successfully applying deep learning methods to object detection. This approach used a traditional selective search algorithm to crop images, then extracted features using a neural network, and finally performed classification using a support vector machine (SVM). However, this approach required feeding the cropped image into the network 2,000 times, significantly increasing network overhead and resulting in slower detection speeds.

[0004] In 2015, Girshick drew on the ideas of SPPNet and published the paper "Fast R-CNN" (In Proceedings of the IEEE international conference on computer vision). Fast R-CNN, an improvement on R-CNN, was proposed using a mapping approach. First, it takes an image and uses selective search to select suggested boxes. The original image is then fed into a convolutional neural network to obtain a feature map. For each suggested box, the corresponding position is found in the feature map (by proportionality). The feature boxes are then extracted (the depth remains unchanged). Each feature box is divided into a (7x7) grid. Each matrix is then stretched into a vector, which serves as the input to the subsequent fully connected layer. The fully connected layer outputs two vectors, which are used to calculate classification scores and bounding box regression. Finally, the non-maximum suppression method is used on the output score matrix to select a small number of boxes. This method not only improves detection speed but also accuracy. However, it does not solve the end-to-end problem, and detection speed remains relatively slow.

[0005] Shaoqing Ren, Kaiming He, and others published the paper "Faster R-CNN: Towards Real-time Object Detection with Region Proposal Networks" (Advances in Neural Information Processing Systems), further improving on Fast R-CNN. They introduced a Region Proposal Network (RPN) based on Fast R-CNN. The RPN generates proposal boxes based on the extracted features. The generated proposal boxes and feature maps are then processed by ROI pooling for cropping and then fed into the Fast R-CNN prediction head for prediction. This approach makes Fast R-CNN an end-to-end object detection network, significantly improving detection speed while maintaining accuracy.

[0006] However, compared with images of ordinary scenes, remote sensing images have higher scene complexity, differences in reality and resolution, which makes the general convolutional neural network-based detection model extract incomplete features in the feature extraction stage, resulting in unsatisfactory detection accuracy. Summary of the Invention

[0007] The purpose of the present invention is to address the shortcomings of the existing technology and propose a feature enhancement network optimization method based on self-decoding, which solves the problems of incomplete feature extraction and low detection accuracy caused by a series of reasons such as the high complexity of remote sensing image scenes.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A feature enhancement network optimization method and system based on self-decoding, comprising the following steps:

[0010] S1. Build a self-decoder module: Build a decoder consisting of four upsampling modules and one upsampling operation;

[0011] S2. Introducing the self-decoder module into a general convolutional neural network-based object detection model, and introducing the constructed self-decoder module into the backbone network of the object detection model;

[0012] S3, divide the object detection dataset;

[0013] S4, performing prediction processing on the images in the data set;

[0014] S5. Use the training data to perform collaborative training with the object detection model and decoder;

[0015] S6. Remove the self-decoder. During detection, do not use the self-decoder. Remove the self-decoder and only use the original detection model to predict the results.

[0016] S7, sending the verification data to the detection model for detection;

[0017] S8. Obtain the detection result, map the detection result obtained in S7 to the corresponding position of the original image and mark the predicted category information.

[0018] Preferably, the structure of the upsampling module in S1 is: upsampling → first residual block → second residual block → fusion layer; the structure of each residual block is: input layer → 1×1 convolution → 3×3 convolution → 1×1 convolution; the fusion layer is a fusion of the feature map generated after a 1×1 convolution and upsampling of the feature map input to each upsampling module and the feature map generated by the last residual block, wherein the residual block does not change the size of the feature map, the number of convolution kernels of each 1×1 convolution is 1 / 4 of the input channel, each convolution is followed by a bn layer and an activation function, and the last upsampling operation is followed by a sigmoid function.

[0019] Preferably, some annotated optical remote sensing images are selected from S3. The length and width of the images are not limited. Each batch will be preprocessed, the images in the batch will be scaled, and then padded to a uniform size. If the dataset category is more difficult to train, more images will be required. For a simple dataset, 1000+ images can be used. Then 67% of the images will form the training set, and 33% of the images will form the validation set and test set.

[0020] Preferably, in S4, the data in each mini-batch is horizontally flipped with a probability of 50%, and then the length and width ratio of the images in each mini-batch are fixed, the length is scaled to 400-600 and the size is an integer multiple of 32, and then the width is padded to 400-600 and the size is an integer multiple of 32.

[0021] Preferably, the collaborative training in S5 includes the following steps:

[0022] S51, input the preprocessed data into the backbone network (resnet50 or resnet101, etc.) for feature extraction to generate a feature map C5;

[0023] S52. Send the feature map C5 to the object detection head (YOLO detection head or Fast R-CNN detection head, etc.) to predict the category and bounding box, and then calculate the cross entropy loss and bounding box loss with the true label;

[0024] S53, the feature map C5 generated in the first step is sent to the self-decoder module for four upsampling modules, one upsampling operation and a sigmoid function, and then the feature map of the same size as the image after preprocessing is output, and then the mean square error loss function between the preprocessed image and the feature map is calculated;

[0025] S54: The loss functions of S52 and S53 are simultaneously transmitted back to update the entire network structure.

[0026] Preferably, S7 includes the following detection steps:

[0027] S71, input the preprocessed data into the backbone network for feature extraction to generate a feature map C5;

[0028] S72. Send the feature map C5 to the detection head of the object detection to predict the category and bounding box, and then generate the detection result through non-maximum suppression (NMS) processing.

[0029] Preferably, the construction of the self-decoding module and the embedding training of the self-decoding module described in S1, S2 and S5 design a corresponding self-decoder for the backbone network of the detection model, and use the constructed self-decoder for collaborative training with the object detection model to enhance the feature extraction capability in the backbone network and make the detection stronger.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] First, the structure has strong embedding properties and can be embedded in some mainstream object detection models, such as RetinaNet and Faster R-CNN.

[0032] Second, although it takes longer to train than the detection model without the self-decoding module, it does not increase the prediction time, and the prediction results are not inferior to those of deeper backbone networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the feature enhancement network optimization method based on self-decoding of the present invention;

[0034] Figure 2 This is a test result diagram of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0036] In the description of the present invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front end," "rear end," "both ends," "one end," "the other end," and the like, indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limiting the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0037] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "provided with," "connected," etc., should be understood in a broad sense. For example, "connected" may refer to a fixed connection, a detachable connection, or an integral connection; it may refer to a mechanical connection or an electrical connection; it may refer to a direct connection or an indirect connection through an intermediate medium; it may refer to internal communication between two components. Those skilled in the art will be able to understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0038] The present invention provides a technical solution of a feature enhancement network optimization method and system based on self-decoding

[0039] S1. Build a self-decoder module:

[0040] Construct a decoder consisting of four upsampling modules and one upsampling operation. The structure of the upsampling module is: upsampling → first residual block → second residual block → fusion layer; the structure of each residual block is: input layer → 1×1 convolution → 3×3 convolution → 1×1 convolution; the fusion layer is the feature map generated by performing a 1×1 convolution and upsampling on the feature map input to each upsampling module, and the feature map generated by the last residual block.

[0041] The residual block does not change the size of the feature map. The number of convolution kernels in each 1×1 convolution is 1 / 4 of the input channels. Each convolution is followed by a BN layer and an activation function, and the last upsampling operation is followed by a sigmoid function.

[0042] S2, introduces the self-decoder module into the general convolutional neural network based object detection model:

[0043] After introducing the constructed self-decoder module into the backbone network of the object detection model.

[0044] S3, divide the object detection dataset:

[0045] Select some annotated optical remote sensing images with no restrictions on image length and width (each batch will be preprocessed, the images in the batch will be scaled, and then padded to a uniform size). If the dataset is more difficult to train, more images will be required. Simple datasets can use 1000+ images. Then, 67% of the images will form the training set, and 33% of the images will form the validation set and test set;

[0046] S4, performs prediction processing on the images in the dataset:

[0047] First, the data in each mini-batch is horizontally flipped with a probability of 50%. Then, the length and width of the images in each mini-batch are fixed, and the length is scaled to an integer multiple of 32 between 400-600 pixels. The width is then padded to an integer multiple of 32 between 400-600 pixels.

[0048] S5, use the training data to jointly train the object detection model and decoder:

[0049] S51, input the preprocessed data into the backbone network (resnet50 or resnet101, etc.) for feature extraction to generate a feature map C5.

[0050] S52: The feature map C5 is sent to the object detection head (YOLO detection head or Fast R-CNN detection head, etc.) to predict the category and bounding box, and then the cross entropy loss and bounding box loss are calculated with the true label.

[0051] S53, the feature map C5 generated in the first step is sent to the self-decoder module for 4 upsampling modules, 1 upsampling operation and sigmoid function, and then the feature map of the same size as the image after preprocessing is output, and the mean square error loss function between the preprocessed image and the feature map is calculated.

[0052] S54, the loss functions of S52 and S53 are simultaneously transmitted back to update the entire network structure.

[0053] S6, remove the decoder:

[0054] During detection, the self-decoder is not used. Instead, the self-decoder is removed and only the original detection model is used to predict the results.

[0055] S7, send the verification data to the detection model for detection:

[0056] S71, input the preprocessed data into the backbone network for feature extraction to generate a feature map C5.

[0057] S72, the feature map C5 is sent to the detection head of the object detection to predict the category and bounding box, and then processed by non-maximum suppression (NMS) to generate the detection result.

[0058] S8, get the test results:

[0059] Map the detection results obtained in S7 to the corresponding positions of the original image and mark the predicted category information.

[0060] The effects of the present invention are further described below in conjunction with simulation:

[0061] 1. Simulation experiment conditions:

[0062] The simulation experiment conditions of the present invention are as follows: server GPU: GeForce RTX2080 Ti, video memory 12G.

[0063] The software platform for the simulation experiment of the present invention is: Ubuntu 18.04 system, Python 3.6, and pytorch-gpu 1.19.0.

[0064] 2. Analysis of simulation content and experimental results:

[0065] This paper simulates two existing technologies (Faster R-CNN detection method and retinaNet method) and their introduction of self-decoding modules for remote sensing image detection to obtain detection results. The simulation uses the following datasets: SSDD, DIOR, and 5M-Building.

[0066] The two existing technologies used in the simulation experiments are:

[0067] The Faster R-CNN detection method is an object detection method proposed by Shaoqing Ren, Kaiming He and others in "Ren, S., He, K., Girshick, R., Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 91-99.", referred to as the Faster R-CNN detection method.

[0068] The retinaNet detection method is an object detection method proposed by Lin et al. in "Lin, TY, Goyal, P., Girshick, R., He, K., Dollar, P. (2017). Focal loss′ for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988).".

[0069] The input images used in the simulation experiments of the present invention are the publicly available SSDD dataset, DIOR dataset, and 5M-Building dataset.

[0070] The SSDD dataset was released by Wuhan University in 2017. Its images are extracted from RadarSat-2, TerraSAR-X, and Sentinel-1. It contains 1,160 SAR ship images with an image size of 500 × 500 × 3. We use 1,000 ship images as the training set and 160 images as the validation set.

[0071] The DIOR dataset was released by Northwestern Polytechnical University in 2019. Its images are extracted from Google Earth. The images are 800 × 800 × 3 pixels in size and include 20 categories and 190,288 objects. We use 5,862 remote sensing images as the training set and 5,862 images as the validation set for testing and evaluation.

[0072] The 5M-Building dataset was released by the University of Jinan in 2019. Its images come from the Gaofen-2 satellite and contain nearly 30,000 panchromatic, multispectral, and fused images. We use 6,999 images as the training set and 3,557 images as the validation set.

[0073] Simulation experiment 1 is a comparison chart of the detection results of introducing the self-decoding module into Faster R-CNN.

[0074] Simulation Experiment 2 is the experimental result of using the existing technology retinaNet method under the above simulation conditions.

[0075] Simulation Experiment 3 is the experimental result of using the existing technology Faster R-CNN method under the above simulation conditions.

[0076] In order to verify the effectiveness of the present invention for remote sensing image detection, the detection results were evaluated using the Coco evaluation index. The calculation results are plotted in Table 1.

[0077] Table 1. Comparison of simulation results of the present invention on the SSDD dataset

[0078]

[0079]

[0080] Table 2. Comparison of simulation results of the present invention on the SSDD dataset

[0081] Model Backbone AP50 AP75 AP50:95 APs APm AP1 ARs ARm ARl Faster Resnet50 has +FPN 62.0 36.1 36.1 2.2 19.5 46.8 4.1 25.2 57.7 Faster improvements Resnet50 has +FPN 65.5 40.4 39.6 2.8 21.7 51.3 4.7 28.3 60.7 retinaNet Resnet50 has +FPN 55.4 35.1 33.6 0.4 15.6 45.9 2.2 21.8 57.1 retinaNet improvements Resnet50 has +FPN 60.4 42.2 39.4 0.7 19.5 53.4 2.9 25.5 62.5

[0082] Table 3. Comparison of simulation results of the present invention on the SSDD dataset

[0083]

[0084]

[0085] Combining Tables 1 and 2, it's clear that the detection algorithm trained with the decoder significantly improves both precision and recall. On most evaluation metrics, the decoder-introduced RetinaNet also slightly outperforms the unintroduced Faster R-CNN.

[0086] Combined with Table 3, the Faster R-CNN method with the self-decoder achieves a maximum of nearly 90% at AP50 and is much higher than the original detection method at AP75.

[0087] The following combination Figure 2 The effects of this simulation experiment are further explained.

[0088] Figure 2 a and b are the test images and the real location labels, b and e are the detection results of Faster R-CNN (backbone network resnext50), and c and f are the detection results of Faster R-CNN+self-decoding module (backbone network resnext50). Figure 2 In the figure, we can see that the prediction results for image b have some missed detections and false positives in the middle and bottom regions. Compared to image b, image c has fewer missed detections and false positives. In images e and f, most areas are well detected, but there are some false positives in the upper middle region of image e. This demonstrates that the present invention outperforms the original detection model without the present invention in terms of accuracy, missed detections, and false positives.

[0089] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A feature enhancement network optimization method based on self-decoding, characterized in that: The following steps are involved: S1. Construct a decoder module: Construct a decoder consisting of four upsampling modules and one upsampling operation. The structure of the upsampling module is as follows: upsampling → first residual block → second residual block → fusion layer. The structure of each residual block is as follows: input layer → 1×1 convolution → 3×3 convolution → 1×1 convolution. The fusion layer is to perform a 1×1 convolution and upsampling on the feature map input to each upsampling module, and fuse the feature map generated by the last residual block with the feature map generated by the last residual block. The residual block does not change the size of the feature map. The number of convolution kernels in each 1×1 convolution is 1 / 4 of the input channel. Each convolution is followed by a BN layer and an activation function, and the last upsampling operation is followed by a sigmoid function. S2. Introducing the self-decoder module into the object detection model based on the convolutional neural network, and introducing the constructed self-decoder module into the backbone network of the object detection model; S3, divide the object detection dataset; S4, performing prediction processing on the images in the data set; S5. Use the training data to perform collaborative training with the object detection model and decoder; Collaborative training includes the following steps: S51, inputting the preprocessed data into the backbone network for feature extraction to generate a feature map C5, wherein the backbone network is resnet50 or resnet101; S52, sending the feature map C5 to the detection head of the object detection to predict the category and bounding box, and then calculating the cross entropy loss and bounding box loss with the true label, wherein the detection head is a YOLO detection head or a Fast R-CNN detection head; S53, the feature map C5 generated in the first step is sent to the self-decoder module for four upsampling modules, one upsampling operation and a sigmoid function, and then the feature map of the same size as the image after preprocessing is output, and then the mean square error loss function between the preprocessed image and the feature map is calculated; S54, the loss functions of S52 and S53 are simultaneously transmitted back to update the entire network structure; S6. Remove the self-decoder. During detection, do not use the self-decoder. Remove the self-decoder and only use the original detection model to predict the results. S7, sending the verification data to the detection model for detection; S71, input the preprocessed data into the backbone network for feature extraction to generate a feature map C5; S72, sending the feature map C5 to the detection head of the object detection to predict the category and bounding box, and then generating the detection result through non-maximum suppression (NMS) processing; S8, obtaining the detection result, mapping the detection result obtained in S7 to the corresponding position of the original image and marking the predicted category information; The construction of the self-decoding module and the embedding training of the self-decoding module described in S1, S2 and S5, the design of the corresponding self-decoder for the backbone network of the detection model, and the use of the constructed self-decoder for collaborative training with the object detection model to enhance the feature extraction capability of the backbone network.

2. The feature enhancement network optimization method based on self-decoding according to claim 1, characterized in that: S3 selects labeled optical remote sensing images. Each batch will be preprocessed by scaling the images in the batch and then padding them into images of uniform size to form a dataset. 67% of the images are then used as a training set, and 33% of the images are used as a validation set and a test set.

3. The feature enhancement network optimization method based on self-decoding according to claim 1, characterized in that: In S4, the data in each mini-batch is horizontally flipped with a probability of 50%. Then, the length and width of the images in each mini-batch are fixed, the length is scaled to a value in the range of 400-600, and the width is padded to a value in the range of 400-600. The values after scaling and padding are all integer multiples of 32.

Images