Fast ship target detection method in wide-area remote sensing images based on in-domain transfer learning

Abstract

The present invention provides a method for rapid detection of ship targets in wide-format remote sensing images based on intra-domain transfer learning, which belongs to the field of satellite image recognition. First, a multi-spectral classification network model is created to pre-train a feature extraction network; second, a single-stage rotation target detection model is established, the feature extraction network is fine-tuned, and the weight of the target detection network is updated; third, a thin cloud removal network based on a style transfer algorithm is used to perform thin cloud removal training on remote sensing images; then, in the prediction stage, the global 30-meter land cover fine classification product is used to perform sea and land segmentation on remote sensing images; finally, the target detection network is lightweighted by combining sea surface and coastline masks and regional inactivation methods to improve the detection speed. Compared with previous detection methods, the present invention has made improvements in multiple angles such as feature extractor training, thin cloud removal, and model lightweighting to achieve higher accuracy and faster speed wide-format remote sensing image ship target detection.

Description

Technical Field

[0001] The present invention belongs to the field of satellite image recognition, and more specifically relates to a method for rapid detection of ship targets in wide-area remote sensing images based on intra-domain transfer learning. Background Art

[0002] With the rapid development of remote sensing technology, the temporal resolution, spatial resolution and spectral resolution of satellites have been greatly improved, resulting in an explosive growth in the amount of remote sensing data. In order to extract effective information from remote sensing big data, researchers at home and abroad have conducted a lot of research, among which ship target detection in remote sensing images is an important research direction. In recent years, the successful application of deep learning in the field of computer vision has brought new methods to remote sensing image processing. Most of the existing remote sensing image target detection methods are based on transfer learning methods, that is, using large-scale natural scene image data sets to pre-train feature extractors, so that convolutional neural networks have the ability to extract features such as edges, corners, textures, and semantics. Then, feature engineering design and dedicated loss function design are performed on this basis to achieve the ability of remote sensing target detection. However, this kind of transfer learning is cross-domain, using images from the natural image domain to train feature extractors in the remote sensing image domain. There are two problems. First, natural images are mostly stored in 8 bits, while remote sensing images are mostly stored in 16 bits. 16-bit bit-depth images contain much more information than 8-bit bit-depth images. Second, the target pattern of natural images is quite different from that of remote sensing images. The feature extractor pre-trained with natural images has an insufficient range of feature representation. In addition, the deep learning model has a large amount of computation and a slow detection speed. Summary of the invention

[0003] The purpose of the present invention is to propose a method for rapid detection of ship targets in wide-area remote sensing images based on intra-domain transfer learning, in order to address the drawbacks of cross-domain transfer learning and the low computational efficiency of deep learning models.

[0004] The technical solution adopted by the present invention is:

[0005] A method for rapid detection of ship targets in wide-area remote sensing images based on in-domain transfer learning, comprising:

[0006] Step 1: Create a multi-spectral classification network model and use the original remote sensing image to perform scene classification training on the multi-spectral classification network model to obtain a pre-trained feature extraction network;

[0007] Step 2: Construct a target detection network, use the output features of the pre-trained feature extraction network as the input of the target detection network, and establish a single-stage rotation target detection model. Use the ship target detection dataset to train the single-stage rotation target detection model for target detection, and obtain the trained feature extraction network and target detection network.

[0008] Step 3: Construct a thin cloud removal network based on the style transfer algorithm, use the cloud-free remote sensing images, cloud-containing remote sensing images and noise images as the input of the trained feature extraction network, use the output features of the feature extraction network as the input of the thin cloud removal network, train the thin cloud removal network model, and obtain the trained thin cloud removal network;

[0009] Step 4: In the prediction stage, the land cover fine classification product is used to segment the remote sensing image into land and sea to obtain the sea surface and coastline masks;

[0010] Step 5: Cascade the trained feature extraction network, thin cloud removal network and target detection network, input the sea surface and coastline masks and the remote sensing image to be detected into the feature extraction network, use the regional inactivation method to lightweight the feature extraction network, and perform feature extraction, thin cloud removal and target detection on ships near the coastline and on the sea surface.

[0011] Furthermore, step 1 is specifically as follows:

[0012] A multi-spectral classification network model was created. Except for the input layer and the output layer, the multi-spectral classification network model adopted the Darknet53 network structure. The input layer adopted 4 channels and input 16-bit original remote sensing images. The output layer adopted 45-dimensional cross entropy loss, where 45 represented the number of categories for scene classification. The scene labels were smoothed and regularized to obtain a pre-trained feature extraction network.

[0013] Furthermore, in step 2, the target detection network adopts the YOLOv4 detection algorithm. The detection box description method of the YOLOv4 detection algorithm is as follows: the detection box is described in the form of 4 vertices and 8 coordinate parameters: {x1, y1, x2, y2, x3, y3, x4, y4}; at the same time, an arbitrary quadrilateral IOU calculation method is used instead of a rectangular IOU calculation method.

[0014] Furthermore, step 3 is specifically as follows:

[0015] A thin cloud removal network based on the style transfer algorithm is constructed. The cloud-containing remote sensing image is used as the style image a, the cloud-free remote sensing image is used as the content image p, and the style image a, content image p and random noise image x are used as the input of the trained feature extraction network to extract the multi-scale style features of the real objects in the style image and the fuzzy texture features A of the clouds. l , the sharpened texture feature P of the content image l And the style feature G of the random noise image l and sharp texture feature F l , l represents the feature layer number; and establishes the loss function of style features between random noise image and style image And the loss function of sharpening texture features between random noise image and content image The output of the feature extraction network is used to train the thin cloud removal network. Through iterative training, the random noise image has the style features of the style image and the sharp texture features of the content image. At this time, the obtained random noise image is the image after the cloud-containing remote sensing image is declouded; where w l and v l Represents the weight coefficient of the lth layer of the loss function.

[0016] Furthermore, in step 5, the feature extraction network is lightweighted using the region inactivation method, specifically:

[0017] The remote sensing image to be detected and the sea surface and coastline masks are used as the input of the feature extraction network at the same time, and the convolution layer and pooling layer corresponding to the land part in the sea-land segmentation mask are regionally inactivated to reduce the computational complexity of the feature extraction network.

[0018] The advantages of the present invention over the prior art are:

[0019] (1) In the feature extraction network training stage, the domain transfer learning-based method is used to make the features extracted by the feature extraction network richer and more suitable for remote sensing image target detection.

[0020] (2) In the thin cloud removal stage, the style transfer network is used to remove thin clouds, without the need to create cloud-free image pairs, thus saving annotation costs.

[0021] (3) In the prediction stage, the model is lightweighted by using regional inactivation, which does not lose detection accuracy, reduces false alarms, and improves detection speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The present invention provides an overall technical flow chart of the method.

[0023] Figure 2 Schematic diagram of the thin cloud removal algorithm used in the present invention.

[0024] Figure 3 (a) is the remote sensing image to be detected, Figure 3 (b) is the 30-meter surface cover fine classification product corresponding to the remote sensing image to be detected, Figure 3 (c) is the result of registering the image to be detected with the 30-meter land cover fine classification product. Figure 3 (d) is a schematic diagram of the region deactivation operation based on the sea-land segmentation mask. Figure 3 (e) is a schematic diagram of the region deactivation operation based on the sea and land segmentation mask and the effective range of the image to be detected. DETAILED DESCRIPTION

[0025] The successful application of deep learning in the field of computer vision has brought new methods to remote sensing image processing. However, most of the existing remote sensing image target detection methods are based on cross-domain transfer learning methods. In view of the drawbacks of cross-domain transfer learning and the low computational efficiency of deep learning models, this paper designs a method for rapid detection of ship targets in wide-area remote sensing images based on intra-domain transfer learning. The specific flow chart is as follows: Figure 1 shown.

[0026] The specific implementation manner and basic principle of the present invention are further described below in conjunction with the accompanying drawings.

[0027] Step 1: Create a multi-spectral classification network model. Except for the input layer and the output layer, the multi-spectral classification network adopts the Darknet53 network structure. The input layer uses 4-channel, 16-bit original remote sensing images to avoid information loss caused by channel elimination and image stretching. The output layer uses a 45-dimensional cross entropy loss, where 45 represents the number of categories for scene classification, and the scene labels are smoothed and regularized to reduce the difficulty of network learning and speed up convergence. Then, the multi-spectral classification network model is trained for scene classification using 16-bit original remote sensing images to obtain a pre-trained feature extraction network, which enables it to have the ability to extract remote sensing image features. The pre-trained feature extraction network will be used in subsequent steps, as shown below. Figure 1 As shown;

[0028] Step 2: Construct a target detection network, and use the output features of the pre-trained feature extraction network as the input of the target detection network to establish a single-stage rotation target detection model; the target detection network adopts the improved YOLOv4 detection algorithm, and improves the detection box description method in the target positioning stage, using 4 vertices and 8 coordinate parameters to describe the detection box: {x1, y1, x2, y2, x3, y3, x4, y4}. At the same time, the arbitrary quadrilateral IOU calculation method is used to replace the rectangular IOU calculation method. Then, the single-stage rotation target detection model is trained for target detection using the ship target detection dataset, the feature extractor is fine-tuned, and the weights of the target detection network are updated. The feature extraction network after fine-tuning and training in this stage will still be used in subsequent steps. The target detection network trained in this stage will be directly used in the model prediction stage, such as Figure 1 As shown;

[0029] Step 3: construct a thin cloud removal network based on the style transfer algorithm, use the cloud-free remote sensing image, the cloud-containing remote sensing image and the noise image as the input of the trained feature extraction network, use the output features of the feature extraction network as the input of the thin cloud removal network, train the thin cloud removal network model, and obtain the trained thin cloud removal network; Figure 2As shown in the figure, a thin cloud removal network based on the style transfer algorithm is constructed. The cloud-containing remote sensing image is used as the style image a, the cloud-free remote sensing image is used as the content image p, and the style image a, content image p and random noise image x are used as the input of the trained feature extraction network to extract the multi-scale style features of the real objects in the style image and the fuzzy texture features A of the clouds and fog. l , the sharpened texture feature P of the content image l And the style feature G of the random noise image l and sharp texture feature F l , l represents the feature layer number; and establishes the loss function of style features between random noise image and style image And the loss function of sharpening texture features between random noise image and content image The output of the feature extraction network is used to train the thin cloud removal network. Through iterative training, the random noise image has the style features of the style image and the sharp texture features of the content image. At this time, the obtained random noise image is the image after the cloud-containing remote sensing image is declouded; where w l and v l Represents the weight coefficient of the lth layer of the loss function.

[0030] Step 4: In the prediction stage, the global 30-meter land cover fine classification product GLC_FCS30 is used to segment the remote sensing image into land and sea to obtain the sea surface and coastline masks. Figure 3 As shown in (a), the 30-meter surface cover fine classification product corresponding to the remote sensing image to be detected is as follows Figure 3 As shown in (b), Figure 3 (c) is the result of registering the image to be detected with the 30-meter land cover fine classification product;

[0031] Step 5: Cascade the trained feature extraction network, thin cloud removal network and target detection network, input the sea surface and coastline masks and the remote sensing image to be detected into the feature extraction network, use the regional inactivation method to lightweight the feature extraction network, and perform feature extraction, thin cloud removal and target detection on ships near the coastline and on the sea surface. Figure 1 As shown in , the remote sensing image to be detected and the sea-land segmentation mask are taken as input at the same time; the regional inactivation strategy is as follows Figure 3 As shown in , firstly, the region inactivation operation is performed according to the sea and land segmentation mask, as Figure 3 As shown in (d), the convolution and pooling operations corresponding to the land mask are deactivated so that they do not participate in the calculation; secondly, the regional deactivation operation is performed according to the land and sea segmentation mask and the effective range of the image to be detected, as shown in Figure 3 As shown in (e), the invalid areas in the input image are also deactivated, thereby reducing the amount of calculation, improving the prediction speed, and reducing false alarms.

[0032] In view of the drawbacks of cross-domain transfer learning and the low computational efficiency of deep learning models, the present invention designs a method for rapid detection of ship targets in wide-area remote sensing images based on intra-domain transfer learning. The method based on domain transfer learning provided by the present invention can enrich the features extracted by the feature extraction network. At the same time, the present invention adopts a style transfer network to remove thin clouds, without the need to create cloud-no-cloud image pairs, saving annotation costs. In addition, the present invention adopts a regional inactivation method to perform model lightweight processing, which does not lose detection accuracy, can reduce false alarms, and can improve detection speed.

Claims

1. A method for rapid detection of ship targets in wide-area remote sensing images based on in-domain transfer learning, characterized in that: include: Step 1: Create a multi-spectral classification network model and use the original remote sensing image to perform scene classification training on the multi-spectral classification network model to obtain a pre-trained feature extraction network; Step 2: Construct a target detection network, use the output features of the pre-trained feature extraction network as the input of the target detection network, and establish a single-stage rotation target detection model. Use the ship target detection dataset to train the single-stage rotation target detection model for target detection, and obtain the trained feature extraction network and target detection network. Step 3: Construct a thin cloud removal network based on the style transfer algorithm, use the cloud-free remote sensing images, cloud-containing remote sensing images and noise images as the input of the trained feature extraction network, use the output features of the feature extraction network as the input of the thin cloud removal network, train the thin cloud removal network model, and obtain the trained thin cloud removal network; Step 4: In the prediction stage, the land cover fine classification product is used to segment the remote sensing image into land and sea to obtain the sea surface and coastline masks; Step 5: Cascade the trained feature extraction network, thin cloud removal network and target detection network, input the sea surface and coastline masks and the remote sensing image to be detected into the feature extraction network, use the regional inactivation method to perform lightweight processing on the feature extraction network, and perform feature extraction, thin cloud removal and target detection on ships near the coastline and on the sea surface; Among them, step 1 is specifically as follows: A multi-spectral classification network model was created. The multi-spectral classification network model used the Darknet53 network structure except for the input layer and the output layer. The input layer used 4 channels and input 16-bit original remote sensing images. The output layer used a 45-dimensional cross entropy loss, where 45 represented the number of scene classification categories, and the scene labels were smoothed and regularized to obtain a pre-trained feature extraction network. Among them, in step 2, the target detection network adopts the YOLOv4 detection algorithm. The detection box description method of the YOLOv4 detection algorithm is as follows: the detection box is described in the form of 4 vertices and 8 coordinate parameters: {x1, y1, x2, y2, x3, y3, x4, y4}; at the same time, the arbitrary quadrilateral IOU calculation method is used to replace the rectangular IOU calculation method; Among them, step 3 is specifically as follows: A thin cloud removal network based on the style transfer algorithm is constructed. The cloud-containing remote sensing image is used as the style image a, the cloud-free remote sensing image is used as the content image p, and the style image a, content image p and random noise image x are used as the input of the trained feature extraction network to extract the multi-scale style features of the real objects in the style image and the fuzzy texture features A of the clouds. l , the sharpened texture feature P of the content image l And the style feature G of the random noise image l and sharp texture feature F l , l represents the feature layer number; and establishes the loss function of style features between random noise image and style image And the loss function of sharpening texture features between random noise image and content image The output of the feature extraction network is used to train the thin cloud removal network. Through iterative training, the random noise image has the style features of the style image and the sharp texture features of the content image. At this time, the obtained random noise image is the image after the cloud-containing remote sensing image is declouded; where w l and v l Represents the weight coefficient of the lth layer of the loss function.

2. According to claim 1, a method for rapid detection of ship targets in wide-area remote sensing images based on in-domain transfer learning is characterized in that: In step 5, the feature extraction network is lightweighted using the regional inactivation method, specifically: The remote sensing image to be detected and the sea surface and coastline masks are used as the input of the feature extraction network at the same time, and the convolution layer and pooling layer corresponding to the land part in the sea-land segmentation mask are regionally inactivated to reduce the computational complexity of the feature extraction network.

Images