A remote sensing image ship detection method based on self-supervised learning

Abstract

The application relates to the technical field of image recognition, and discloses a remote sensing image ship detection method based on self-supervised learning, which comprises the following steps: (1) according to the characteristics that the area of a ship to be detected in a sea surface remote sensing image is small and multiple ships may exist in one image, a self-supervised learning module is designed, a feature extraction network is constructed only by using unlabeled sea surface remote sensing images, and efficient feature extraction of the marine remote sensing image is realized; and (2) the FPN network of Mask R-CNN is improved, three size feature maps are fused, and the feature extractor of the self-supervised learning module is combined, so that the detection precision of small and medium-sized ships is significantly improved under the condition that the labeled samples are few.

Description

TECHNICAL FIELD

[0001] The application relates to the field of image recognition, and particularly relates to a ship detection method for sea surface remote sensing images based on self-supervised learning. BACKGROUND

[0002] Ship detection has a wide application in marine shipping, port management, marine rescue and the like, and has been widely concerned in the fields of remote sensing and computer vision. A high-altitude reconnaissance platform such as satellite remote sensing imaging is a mainstream platform for obtaining sea surface remote sensing image data. However, it is difficult to obtain large-scale labeled data because remote sensing images are complex to interpret and label. When there are few labeled samples, mainstream target detection methods are difficult to learn good picture representations.

[0003] Self-supervised learning can learn the intrinsic features of images from unlabeled image samples by setting proxy tasks, and obtain deep image representations that play a key role in downstream ship detection tasks. However, due to the small size of the ship to be detected in the sea surface remote sensing image and the relatively simple background, it is difficult to design a proper contrast task, and thus it is difficult to construct a proper negative sample pair, so that the self-supervised models such as MoCo and SimCLR cannot effectively extract the features of the ship to be detected.

[0004] Another predictive self-supervised learning task is the CutPaste method. It constructs an enhanced image through the cut-and-paste method, and the corresponding learning task classifies the image as two classes to determine whether the image has a cut-and-paste operation. When this self-supervised learning method is applied to ship detection in sea surface remote sensing images, there will be a problem of missing a large number of small ships and difficulty in accurately positioning the ship. In view of this, the application proposes a more-way cutpaste self-supervised method based on the improved CutPaste method for the characteristics of small target objects in remote sensing images, and improves the Mask R-CNN target detection module accordingly, which significantly improves the detection performance of small ships in sea surface remote sensing images. SUMMARY

[0005] The application aims to provide a remote sensing image ship detection method based on self-supervised learning to solve the problems in the background art. To achieve the above-mentioned purpose, the application provides the following technical solutions.

[0006] A remote sensing image ship detection method SS R-CNN based on self-supervised learning, and the specific steps are as follows:

[0007] S1, collect ship remote sensing images, and the images are color RGB images with a resolution of not less than 96 dpi;

[0008] S2, using the unlabeled sea surface image, according to the constructed self-supervised module more-way cutpaste, the training of the feature network is completed;

[0009] S3, inputting the image to be detected into the feature network, and detecting whether there is a ship in the remote sensing image and labeling the specific position of all ships according to the generated feature map by the improved Mask RCNN detection module;

[0010] The model in S2 is a feature network based on a self-supervised task, which comprises a feature extractor F and a classifier C, wherein the feature extractor F adopts a ResNet50 convolutional neural network, the network has 50 layers, and a total of 4 blocks, each block has 3, 4, 6 and 3 bottlenecks respectively (the meanings of Block and Bottonck are the same as defined in Deep Residual Learning for Image Recognition), and a total of 23524424 parameters are contained; the classifier C is a commonly used MLP module, which has 3 layers, the input dimension is 2048, the output dimension is 6, the hidden layer dimension is 300, the hidden layer uses a ReLU activation function, and the output layer uses a Sigmoid function.

[0011] As a preferred scheme of the application, the training process of the model in S2 comprises the following steps:

[0012] S21, setting a self-supervised task, obtaining an image I' containing a simulated ship through a sea surface picture without a ship;

[0013] S22, inputting the simulated image I' into the feature network, learning the features through the extractor F, and predicting the category c of the model by the classifier C;

[0014] S23, calculating the empirical loss and updating the model parameters;

[0015] The loss function used by the model is cross-entropy loss, and the empirical loss is:

[0016]

[0017] Wherein, N is the sample number of the training set, M is the number of categories output by the classifier, the classification label c=0 indicates that there is no ship in the corresponding image sample, p ic is the probability that the ith image is predicted as the cth category, y ic indicates whether the ith image belongs to the cth category, if the ith image belongs to the cth category, y ic =1, otherwise 0; the feature network based on the self-supervised task and the minimum empirical loss L more-way , the parameters are updated through error back propagation, and the network training is completed;

[0018] S24, freeze the classifier C and only take the feature extractor F as the feature extraction part of Mask RCNN.

[0019] As a preferred scheme of the present application, the specific steps of setting an agent task to obtain the image I' containing the simulated ship in S21 are as follows:

[0020] (a) Obtain the sea surface image I not containing the ship, and calculate the area of the sea surface image;

[0021] (b) Set the length-width ratio interval [R1, R2] of the simulated ship area and the percentage interval [P1, P2] of the image area, and the parameters R1, R2, P1, P2 are selected according to the shape and area of the ship in the labeled data set. The specific method is: select the values R1, R2, P1, P2 so that at least 70% of the ship area in the labeled data set is located in the interval [P1S, P2S], and the length-width ratio of more than 70% of the ship is located in the interval [R1, R2];

[0022] (c) Draw a rectangular frame with a uniformly distributed random point r i ∈[R1, R2], p i ∈[P1, P2] as the vertex at any coordinate point of the image. And obtain the corresponding image block I p ; If the drawn rectangle exceeds the image boundary, it is rejected and redrawn;

[0023] (d) Rotate the image block in (c) around its center point, and the rotation angle is randomly taken from the interval [-45°, 45°];

[0024] (e) Color dithering is performed on the image block in (d), and the image I' after color dithering is obtained p , the specific method includes that the brightness, contrast, saturation and hue of the image are randomly changed to 90%-110% of the original image;

[0025] (f) Randomly paste the image block in (e) in any area of the image I to cover the content in the original image I, and if it exceeds the boundary, select another area, to obtain a new image I' containing simulated ships;

[0026] (g) Repeat operations (c)-(f) c times to obtain the image I' containing the simulated ship, and let c be the corresponding classification label, where c is a random integer between 1 and M, and the parameter M is determined according to the number of ships in the data set. Specifically: select the smallest positive integer M that satisfies the following conditions, so that the proportion of images containing no more than M ships in the training image samples is > 90%;

[0027] The detection network in S3 is an improved Mask R-CNN network, and the main improvement is: the FPN network in Mask R-CNN is simplified, the deep feature maps of the original FPN network are removed, and the five sizes of the original architecture [200, 100, 50, 25, 13] are simplified to three sizes of [200, 100, 50], see Figure 4 Where C1-C5 represents the number of ResNet convolution layers, 1x1 and 3x3 represent 1-dimensional convolution and 3-dimensional convolution operations respectively, 2x represents 2 times upsampling, and 0.5x represents 0.5 times downsampling, i.e. max-pooling, The elementwise sum operation is represented, P2-P6 represents the output fused feature map, and 200, 100, 50, 25 and 13 are the sizes corresponding to P2-P5 feature mapping. The shaded area in the figure is the sub-module deleted by the application.

[0028] The main technical innovation points of the above training process include:

[0029] (1) A self-supervised module is constructed for ship detection of sea surface remote sensing images, and its features include:

[0030] (a) The self-supervised learning module only needs sea surface remote sensing image samples without ships, and does not need manually labeled images;

[0031] (b) Each augmented image includes multiple rectangular simulated ships with different areas, different aspect ratios, different angles, different positions and different colors, especially multiple rectangular simulated ships with small areas and large aspect ratios, which is conducive to the model learning the features of small ships;

[0032] (c) The prediction task is not to detect whether there is a simulated ship in the image, but to detect the number of simulated ships, which makes the model less likely to miss small ships;

[0033] (d) The cross-entropy loss function is used to calculate the empirical loss, which not only considers whether there is a ship in the image sample, but also considers the number of ships in the image. This loss function is conducive to improving the detection accuracy of small ships;

[0034] (2) An improved target detection module is used, and its features include:

[0035] (a) The detection module uses the feature extraction network trained by the self-supervised module as a feature extractor, which significantly reduces the manually labeled images needed to train the target detection module;

[0036] (b) The improved MaskR-CNN network is used for ship detection, which can more accurately determine the position of the detected ship;

[0037] (c) The FPN network in the improved MaskR-CNN is simplified, and two deep feature maps are removed. It is generally believed that deep feature maps can extract more abstract overall image features, so deep feature maps can obtain the overall features of large ships in the image. The use of the simplified FPN network significantly reduces the amount of calculation for training the target detection model, and at the same time, it is found that the detection accuracy of large ships is not negatively affected.

[0038] Compared with the prior art, the beneficial effects of the present application are:

[0039] (1) The number of remote sensing images manually labeled for feature extraction and target detection module parameter training is significantly reduced;

[0040] (2) The detection and positioning accuracy of small ships (occupying less than 32x32 image pixels) in sea surface remote sensing images is significantly improved;

[0041] (3) The structure of the target detection module is simplified, and the training time of the target detection module is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The framework diagram of the technical solution of the present application;

[0043] The method of the present application is recorded as SSR-CNN, which mainly consists of two networks: a self-supervised network (a) and a detection network (b), which are trained independently. The self-supervised network is trained using sea surface images without ships, so it does not need to input labeled samples. The label c is generated by the more-way cutpaste task, the features are extracted by the feature extractor F, and the classifier C realizes the multi-classification task to complete the prediction of the number of simulated ships in the image. The trained feature extractor F is parameter migrated as the feature extractor in the detection network. The detected image is extracted by the feature extractor, the multi-scale feature map is generated by the FPN network, and the binary classification prediction task of whether the detected image contains a ship and the detection frame regression task of determining the ship position are completed.

[0044] Figure 2 The flowchart of the self-supervised method of the present application is more-way cutpaste;

[0045] For the input sea surface remote sensing image, the construction of the simulated ship area is repeated c times, and the value range of c is in the interval [1, M]. The simulated ship image is extracted by the feature extractor F, and the classifier C realizes the multi-classification task to complete the prediction of the number of simulated ships in the image.

[0046] Figure 3 The method of the present application is the flow chart of SSR-CNN;

[0047] Wherein (a) is a self-supervised network, which is realized by parameter training of a more-way cutpaste multi-classification task on a sea surface remote sensing image not containing a ship; (b) is a detection network, the main structure of which is an improved Mask R-CNN network, wherein the feature network is obtained from the self-supervised learning module (a), the detection network generates dense candidate anchor boxes through a region proposal network (RPN), removes redundant anchor boxes through non-maximum suppression (NMS), obtains candidate ROIs through ROI alignment technology, and determines the detection frame through a linear layer.

[0048] Figure 4 The figure is for the improvement of the FPN structure of the detection network of the present application;

[0049] The five sizes of feature maps of the original framework [200, 100, 50, 25, 13] are simplified to three sizes of feature maps [200, 100, 50]. C1-C5 represent the number of ResNet convolution layers, 1x1 and 3x3 represent 1-dimensional convolution and 3-dimensional convolution operations respectively, 2x represents 2 times of up-sampling, 0.5x represents 0.5 times of down-sampling, i.e. max-pooling, Elementwise summation operation is represented. P2-P6 represents the output fused feature map, and 200, 100, 50, 25 and 13 are the sizes corresponding to P2-P5 feature mapping. The shaded area in the figure is the removed sub-module.

[0050] Figure 5 The figure is for the introduction of the data set used in the embodiment of the present application;

[0051] The Airbus data set is used in the embodiment. The ships occupy different sizes in the image area, (a) shows the images containing large ships with a size greater than 96x96 pixels, (b) shows the medium-sized ships with a size between large and small ships, (c) shows small ships with a size less than 32x32 pixels, and (d) is a sea surface image without ships.

[0052] Figure 6 The figure is for the performance comparison between the present application and the mainstream supervised method in the embodiment;

[0053] The leftmost column shows several typical remote sensing images, including scenes of multiple ships (first two rows), medium-sized ships (third row), small ships (fourth row), and large ships accompanied by small ships (fifth row). The rightmost columns show the ships detected by SS R-CNN, SSD, and Mask RCNN in different images. The detected ships are marked with rectangular boxes, and the rectangles in the first column are the actual annotation boxes of the dataset.

[0054] Figure 7 Comparison chart of improved CutPaste method ship detection accuracy in the embodiments of the present application.

[0055] The leftmost column shows several typical remote sensing images, and the rectangular boxes marked in the leftmost column of images are the actual annotation boxes of the dataset. The rightmost columns show the ships detected by the improved Mask R-CNN detection module using more-way, normal, scar, and 3-way as self-supervised tasks in different images, and the rectangular boxes represent the positions of the ships detected by each method. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0058] The present application will be further described below in combination with the drawings and examples. A dataset is selected to test the effectiveness of the disclosed method. To simulate the situation of scarce annotation data in real scenarios, a specified number of image samples are randomly selected from the airbus dataset (https: / / www.kaggle.com / competitions / airbus-ship-detection). Image examples are shown in Figure 5(a) is a large ship, (b) is a medium ship, (c) is a small ship, (d) is an unmarked sea surface image. The training method given by the present application is carried out in two stages: 1. Using a self-supervised module on unmarked sea surface images to train the feature network. 2. Training the target detection network: After the labeled image sample training set is learned by the trained feature network, the ship target detection task is realized by the improved Mask R-CNN module. The performance of the self-supervised task is reflected by the accuracy of the improved Mask R-CNN target detection downstream. The annotation box in the specification refers to the rectangular box in the sea surface remote sensing image data set that marks the position of the ship, and the detection box refers to the rectangular box given by various detection methods that marks the position of the ship.

[0059] The indicators commonly used in the COCO evaluation protocol: mAP, AP50, AP75, APs, APm and APl are used to evaluate the proposed model. The characteristics of these indicators are listed in Table 1, wherein mAP is a main indicator, and APs represents the AP value of small objects. For each of the listed indicators, the larger the indicator value, the better the detection performance.

[0060] Table 1 Target detection measurement indicators

[0061]

[0062] The self-supervised learning embodiment of the present application is implemented in a Linux server 190 with a Tesla V2 GPU using Python 3.7, and the parameters R1, R2, P1, P2 are set to R1=0.3, R2=3, P1=0.01, P2=0.05, the SGD training algorithm batch size is set to 64, and the learning rate is set to 0.03. Then the SSR-CNN model is trained on a Tesla P100 GPU Colab, using the SGD optimizer, and the SGD algorithm training batch size is set to 8, and the learning rate is set to 0.0005.

[0063] In the region proposal network (RPN), three anchor box sizes are selected: 128x128, 256x256 and 512x512, the anchor box step is set to 4, 8 and 16 respectively, and the length-width ratio of the anchor box is randomly selected from 0.5, 1.0, 2.0, if the calculated IoU indicator of the anchor box and the actual annotation box is less than 0.3, the anchor box is considered to be irrelevant background (sea surface), if the calculated IoU indicator of the anchor box and the actual annotation box is greater than 0.6, the anchor box is considered to be the foreground region of interest (ship), after the NMS operation, at most 2000 candidate anchor boxes are selected for each image.

[0064] First, the detection performance of SS R-CNN is compared with two supervised models, SSD and classic MaskR-CNN, where SSD is a single-stage object detection model and MaskR-CNN is a two-stage model. To train the self-supervised learning network, 7500 sea surface images without ships are randomly selected. For the training of the detection model, 1000 labeled ship images are randomly selected as the training set, and 300 labeled ship images are randomly selected as the test set. After the training process of the model reaches stability, the corresponding detection result accuracy is shown in Table 2, and part of the prediction results are shown in Figure 6 where the bold numbers represent the best performance on the corresponding indicators.

[0065] Table 2. Comparison of detection accuracy between the method of the present application and mainstream supervised methods

[0066]

[0067] From Table 2, it can be seen that the detection accuracy of SS R-CNN is better than the supervised methods Mask R-CNN and SSD in mAP, AP50, 236AP75, APs and APm indicators. Since the main difference between SS R-CNN and Mask R-CNN is the use of the designed self-supervised learning module, the above results show that the self-supervised learning module of SS R-CNN extracts useful semantic feature information from unlabeled sea surface images.

[0068] From Figure 6 it can be seen that (i) for images with multiple ships (the first two rows), SS R-CNN correctly detects multiple objects, to some extent solving the problem of missing small ships; (ii) SS R-CNN has better performance on small ships and medium ships. Compared with the supervised methods SSD and MaskR-CNN, SS R-CNN has better performance on small ships and medium ships because the predicted anchor box is more accurate; (iii) in terms of large ship detection, the detection box of SS R-CNN and Mask R-CNN is not as accurate as the SSD method. One of the main reasons is that SS R-CNN also adopts the MaskR-CNN module, and its detection ability for large objects is limited by the number of image samples containing large ships.

[0069] To test the effectiveness of the More-Way CutPaste self-supervised module, we compared it with the typical CutPaste task and other mainstream self-supervised contrastive learning modules, namely normal, scar, 3-way, MoCo, and SimCLR. We randomly selected 7,500 unlabeled ocean surface images to train the self-supervised learning network. The detection network used the improved Mask R-CNN proposed in this paper. Table 3 lists the accuracy of the detection network, where bold numbers indicate the best performance in the corresponding metric.

[0070] Table 3. Comparison of detection accuracy between the proposed method and the self-supervised method

[0071]

[0072] Table 3 shows that the self-supervised More-Way Cutpaste task used in our SS R-CNN achieves higher detection accuracy than the normal, Scar, 3-way, MOCO, and SimCLR tasks. Performance improved by approximately 10.7% in terms of mAP, from 0.56 to 0.62, demonstrating that the designed More-Way Cutpaste module more effectively captures the characteristics of the downstream ship detection task. This improvement is primarily due to improved accuracy for small ship detection. However, for large ship detection, all six self-supervised tasks tested failed to achieve satisfactory results, with AP1 metrics below 0.30.

[0073] from Figure 7 As can be seen in the figure, compared to the normal, scar, and 3-way tasks, the SSR-CNN using the more-way cutpaste task more accurately captures the target ship, with fewer missed and falsely detected ships. This demonstrates that the more-way cutpaste task designed in this paper is suitable for detecting multiple small ships in remote sensing imagery.

[0074] This invention is primarily designed for ship detection in ocean surface remote sensing imagery with limited annotations. The examples tested the performance of the proposed method against baseline methods on annotated training sets of various sizes. Specifically, 200, 400, 1000, 2000, and 5000 annotated training images were randomly selected, each containing one or more ships and their corresponding ground-truth bounding boxes. Table 4 lists the detection accuracy of Mask R-CNN trained with SS R-CNN, SSD, and MOCO pre-trained Mask R-CNN.

[0075] Table 4. Comparison of detection accuracy of various methods under different amounts of labeled data

[0076]

[0077] As can be seen from Table 4: (i) the detection accuracy of each method being tested generally increases with the increase of the training scale; (ii) the detection accuracy of SS R-CNN has a strong advantage over SSD and MaskR-CNN (pre-trained by MOCO) at each training scale.

[0078] The above description describes the specific embodiments of the present application in combination with the drawings, but is not a limitation on the protection scope of the present application. Various modifications or variations made by the related skilled person on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A self-supervised learning-based ship detection method for sea surface remote sensing images, SS R-CNN. This method consists of two parts: a self-supervised feature learning network and an improved Mask R-CNN detection network. The specific steps are as follows: S1, collect ship remote sensing images, which should be color RGB images with a resolution of no less than 96dpi; S2 uses unlabeled ocean surface images and completes the training of the feature network based on the set self-supervised task more-way cutpaste; S21, set the self-supervised task more-way cutpaste, generate an image I′ containing a simulated ship from a sea surface image without a ship; (a) Obtain a sea surface image I excluding ships and calculate the area S of the sea surface image; (b) Setting the length-to-width ratio interval [R1, R2] and the percentage interval [P1, P2] of the simulated ship area. The parameters R1, R2, P1, and P2 are selected based on the shape and area of ​​the ships in the annotated dataset. Specifically, the values ​​R1, R2, P1, and P2 are selected so that at least 70% of the ships in the annotated dataset have an area within the interval [P1S, P2S] and more than 70% of the ships have a length-to-width ratio within the interval [R1, R2]. (c) Take any image coordinate point as the vertex and randomly select r i ∈[R1, R2], p i ∈[P1, P2] draw a rectangular box and get its corresponding image block I p ; If the drawn rectangle exceeds the image boundary, it will be discarded and redrawn; (d) Rotate the image block in (c) around its center point, and the rotation angle is arbitrarily selected from the interval [-45°, 45°]; (e) Perform color dithering on the image block in (d) and obtain the color-dithered image I′ p ,The specific method includes randomly changing the brightness, contrast, saturation and hue of the image to 90% to 110% of the original image; (f) Randomly paste the image block in (e) to any area of ​​image I to cover the content of the original image I. If it exceeds the boundary, select another area to obtain a new image I′ containing the simulated ship; (g) Repeat operations (c)-(f) c times to obtain an image I′ containing simulated ships. Let c be its corresponding classification label, where c is a random integer between 1 and M. The parameter M is selected based on the number of ships in the dataset. Specifically, select the smallest positive integer M that satisfies the following conditions, such that: the proportion of images in the training image sample containing no more than M ships is greater than 90%; S22, the simulated image I' is input into the feature network, after the extractor F learns the features, the classifier C predicts the category c to which the model belongs, where the feature extractor The ResNet50 convolutional neural network is used, which has 50 layers and 4 blocks. The classifier C is a commonly used MLP module with an input dimension of 2048, an output dimension of 6, and a hidden layer dimension of 300. The hidden layer uses the ReLU activation function, and the output layer activation function is the Sigmoid function. S23, calculate the loss and update the model parameters; The empirical loss is calculated based on the cross entropy loss function, which is: Where N is the number of samples in the training set, M is the number of categories output by the classifier, and the classification label c = 0 means that there is no ship in the corresponding image sample, p ic is the probability that the i-th image is predicted to be the c-th class, y ic Indicates whether the i-th image belongs to the c-th category. If the i-th image belongs to the c-th category, then y ic =1, otherwise 0; based on the feature network of the self-supervised task and minimizing the experience loss L more-way , update the parameters through error back propagation and complete network training; S24, freeze the classifier C and use the feature extractor F as the feature extraction module of the detection network; S3, input the image to be detected containing the ship into the feature network, and the improved Mask R-CNN network completes the detection of the ship and determines the ship's position based on the generated feature map; S31, improvements to the Mask R-CNN network include: streamlining the FPN network in Mask R-CNN, removing the deep feature maps of the original FPN network, and simplifying the feature maps of the original architecture [200, 100, 50, 25, 13] with five sizes to feature maps of three sizes [200, 100, 50].

Images