A ship detection method based on remote sensing images

Abstract

The present application belongs to the field of remote sensing image processing, and particularly relates to a ship detection method based on remote sensing images. The present application aims at the problem of low recognition accuracy in the existing ship model detection task of remote sensing images, and proposes a ship detection method based on remote sensing images. The present application constructs a ship detection model based on remote sensing images for the fine-grained target recognition task of marine ships; inputs the ship remote sensing image of the to-be-recognized area into the trained ship detection model based on remote sensing images to obtain the ship model of the ship in the to-be-recognized area; decouples the fine-grained target detection task into a positioning subtask and a classification subtask, and simultaneously, in view of the problem of short supply of marine ship training samples, designs a data expansion strategy based on CycleGAN and a contrast learning module to improve the recognition accuracy. The present application solves the problem of poor recognition accuracy in the existing fine-grained target recognition technology of remote sensing images due to high visual distance, small targets and insufficient training data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing image processing, and particularly relates to a ship detection method based on remote sensing images. BACKGROUND

[0002] Processing of remote sensing images of ships at sea can identify and monitor ships on the sea surface, analyze the types, sizes and directions of the ships, which is very important for maritime traffic management, fishery management and rational use of marine resources.

[0003] Currently, the target detector for ships at sea does not make corresponding adjustments for fine-grained targets. In remote sensing pictures, only the top view of the object can assist in classification, and it is more difficult to identify due to fewer texture features under high view distance. For example, DOTA is the largest OBB annotated remote sensing dataset, which contains 15 categories. The appearance of these categories is obviously different. Even in remote sensing images, it is not difficult to distinguish different categories (such as bridges and airplanes).

[0004] However, in the HRSC2016 dataset, specific ship models need to be identified and distinguished (such as cargo ships and medical ships), and there are few effective features that can be used to distinguish between the top views of different ship models. In this case, the current general target detection method performs much better in identifying ships, cars and other categories with large differences than in identifying specific ship models.

[0005] Currently, there are many challenges in remote sensing fine-grained target detection for ships at sea. First, there are very few open-source fine-grained ship remote sensing datasets, and most remote sensing datasets only contain the coarse-grained recognition object of ships, without fine-grained labels of specific ship models. For example, the NWPUVHR-10 geographic remote sensing dataset contains only the coarse-grained label of ships among the 10 categories. The HRSC2016 dataset additionally annotates the fine-grained label of specific ship models on the basis of the coarse-grained label of ships. However, among all the pictures, only 617 pictures have fine-grained labels, and compared with other fine-grained remote sensing datasets, the training samples are fewer, and the ordinary target recognition model will have a serious overfitting phenomenon.

[0006] In the field of remote sensing, target detection is mainly divided into two categories: two-stage and one-stage target detection methods. High-performance two-stage target detection methods usually follow the RCNN architecture, which consists of a region proposal network and a region CNN detection head. In recent years, many variants of the RCNN framework have emerged, such as the directional RCNN, which introduces a box encoding system to address the problem of unstable training loss. A representative one-stage target detection method is YOLOv5, and the YOLO architecture has also been widely used in the field of remote sensing. For example, BiFA-YOLO uses bidirectional feature fusion and angle classification to detect targets in different directions.

[0007] Image enhancement has been shown to improve the accuracy of fine-grained image recognition and target detection tasks. Image enhancement includes traditional operations such as rotation, cropping, and exposure adjustment, as well as more advanced strategies such as image mixing and deletion, which provide greater diversity compared to traditional methods. Representative mixing strategies include AlignMixUp, in which two images are aligned and interpolated in feature space, and TokenMixup, which optimally matches images in the same mini-batch to increase important information in the output contrast.

[0008] Current target recognition usually identifies the entire object, while fine-grained target recognition focuses more on the detailed features of the object's interior to distinguish different subcategories within the same category. Fine-grained target recognition for remote sensing images faces significant challenges in accurately identifying the target's category while ensuring high recall rates due to high altitude, small targets, and limited training data. Meanwhile, image enhancement for fine-grained target recognition is basically an operation on the image itself, even with the addition of mixing strategies, which do not introduce new ship features and may not be sufficient to create representative diversity, making it difficult to capture complex transformations in real-world scenarios and further leading to model overfitting on certain specific changes, rather than adapting well to actual data.

[0009] In summary, the existing fine-grained target recognition technology for remote sensing images has the problem of low recognition accuracy due to high altitude, small targets, and limited training data. SUMMARY

[0010] To address the low recognition accuracy problem in existing remote sensing image-based ship model detection tasks, the present application proposes a ship detection method based on remote sensing images. The specific process includes:

[0011] S1: Construct a remote sensing image-based ship detection model; obtain a trained remote sensing image-based ship detection model;

[0012] S2: collect a ship remote sensing image of a to-be-identified region, input the ship remote sensing image of the to-be-identified region into the trained ship detection model based on a remote sensing image, and obtain a ship model of the ship in the to-be-identified region;

[0013] The S1 constructs a ship detection model based on a remote sensing image, obtains a trained ship detection model based on a remote sensing image, and the specific process is as follows:

[0014] S1.1: obtain a ship remote sensing image set;

[0015] S1.2: construct a ship detection model based on a remote sensing image, train the ship detection model based on a remote sensing image according to the ship remote sensing image set, and obtain a trained ship detection model based on a remote sensing image.

[0016] The ship remote sensing image set in the S1.1 is an HRSC2016 remote sensing image ship data set;

[0017] The ship detection model based on a remote sensing image in the S1.2 sequentially comprises a data enhancement module, a target detection module, and a fine-grained picture classification module;

[0018] The S1.2 trains the ship detection model based on a remote sensing image according to the ship remote sensing image set, and obtains a trained ship detection model based on a remote sensing image, and the specific process is as follows:

[0019] S1.2.1: train the data enhancement module according to the ship remote sensing image set, and obtain a trained data enhancement module;

[0020] Then, the images of the ship remote sensing image set are input into the trained data enhancement module, and an enhanced ship remote sensing image set is obtained;

[0021] S1.2.2: train the target detection module according to the enhanced ship remote sensing image set, and obtain a trained target detection module;

[0022] Then, the enhanced ship remote sensing image set is input into the trained target detection module, and a ship object in the enhanced ship remote sensing image and a position of the ship object are obtained;

[0023] The position of the ship object is a coordinate of a square range where the ship object is located in a picture;

[0024] S1.2.3: cut all ship objects in the enhanced ship remote sensing image according to the position of the ship object, and obtain a cut ship remote sensing image set;

[0025] Each cut picture in the cut ship remote sensing image set contains only one ship;

[0026] According to the set of cut ship remote sensing images, the fine-grained picture classification module is trained to obtain the trained fine-grained picture classification module.

[0027] The set of cut ship remote sensing images is input into the trained fine-grained picture classification module to obtain the specific model of each ship.

[0028] S1.2.4: The trained data enhancement module, the trained target detection module and the trained fine-grained picture classification module are combined to obtain the trained ship detection model based on remote sensing images.

[0029] The above steps are combined to introduce the inventive concept of the present application, that is, first, the target detection model based on remote sensing images is used to detect the target of the input picture to obtain the position coordinates of the ship, the original picture is cut using the position coordinates, so that each cut picture contains only one ship, and the cut picture is sent to the ship classification model based on remote sensing images to classify the ship in the picture.

[0030] The present application has the following advantages:

[0031] The present application designs a set of architecture for the fine-grained target recognition task of marine ships, decouples the fine-grained target detection task into positioning subtask and classification subtask, and designs an effective data expansion strategy and contrast learning module based on CycleGAN to solve the problem of short training sample of marine ships, which greatly improves the recognition accuracy. Good results are achieved on the data set, solving the problem of poor recognition accuracy in existing fine-grained target recognition technology of remote sensing images due to high view distance, small target, and insufficient training data. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 It is a fine-grained remote sensing ship detection example schematic diagram.

[0033] Figure 2 It is a fine-grained remote sensing ship detection system framework schematic diagram.

[0034] Figure 3 It is an image processing result schematic diagram in the data enhancement process. DETAILED DESCRIPTION

[0035] Specific implementation one: combined with Figure 1 The present application is described, including:

[0036] S1: Construct a ship detection model based on remote sensing images; obtain a trained ship detection model based on remote sensing images;

[0037] S2: collect a ship remote sensing image of a to-be-identified region, input the ship remote sensing image of the to-be-identified region into the trained ship detection model based on a remote sensing image, and obtain a ship model of the ship in the to-be-identified region;

[0038] The S1 constructs a ship detection model based on a remote sensing image, obtains a trained ship detection model based on a remote sensing image, and the specific process is as follows:

[0039] S1.1: obtain a ship remote sensing image set;

[0040] S1.2: construct a ship detection model of a remote sensing image, train the ship detection model based on a remote sensing image according to the ship remote sensing image set, and obtain a trained ship detection model based on a remote sensing image.

[0041] Specific implementation method two: the difference between the embodiment and the specific implementation method one is that,

[0042] The ship remote sensing image set in the S1.1 is an HRSC2016 remote sensing image ship data set;

[0043] The ship detection model of a remote sensing image in the S1.2 comprises, in sequence, a data enhancement module, a target detection module, and a fine-grained picture classification module.

[0044] The ship detection model based on a remote sensing image is trained according to the ship remote sensing image set, and a trained ship detection model based on a remote sensing image is obtained, and the specific process is as follows:

[0045] S1.2.1: train the data enhancement module according to the ship remote sensing image set, and obtain a trained data enhancement module;

[0046] Then, the images of the ship remote sensing image set are input into the trained data enhancement module, and an enhanced ship remote sensing image set is obtained;

[0047] S1.2.2: train the target detection module according to the enhanced ship remote sensing image set, and obtain a trained target detection module;

[0048] Then, the enhanced ship remote sensing image set is input into the trained target detection module, and a ship object in the enhanced ship remote sensing image and a position of the ship object are obtained.

[0049] The position of the ship object is a coordinate of a square range where the ship object is located in a picture;

[0050] S1.2.3: cut all ship objects in the enhanced ship remote sensing image according to the position of the ship object, and obtain a cut ship remote sensing image set;

[0051] Each of the cut pictures in the set of cut ship remote sensing images contains only one ship;

[0052] The set of cut ship remote sensing images is input into the trained fine-grained picture classification module to obtain the specific model of each ship.

[0053] The set of cut ship remote sensing images is input into the trained fine-grained picture classification module to obtain the specific model of each ship.

[0054] S1.2.4: The trained data augmentation module, the trained target detection module, and the trained fine-grained picture classification module are combined to obtain a trained ship detection model based on remote sensing images.

[0055] The above steps are combined to introduce the inventive concept of the present application, that is, first, the target detection model based on remote sensing images is used to detect the target of the input picture to obtain the position coordinates of the ship, the original picture is cut using the position coordinates, so that each of the cut pictures contains only one ship, and the cut picture is input into the ship classification model based on remote sensing images to classify the ship in the picture.

[0056] The other steps and parameters are the same as in the first embodiment.

[0057] Embodiment three: the difference between this embodiment and the first embodiment is that,

[0058] The data augmentation module in S1.2.1 includes an image segmentation network and an image style conversion network.

[0059] In S1.2.1, the data augmentation module is trained according to the set of ship remote sensing images to obtain the trained data augmentation module; the specific process is as follows:

[0060] S1.2.1.1: Each ship remote sensing image in the set of ship remote sensing images is input into the image segmentation network in the data augmentation module for extraction processing to obtain the mask of each ship remote sensing image in the set of ship remote sensing images.

[0061] S1.2.1.2: Based on the mask of each ship remote sensing image extracted in S1.2.1.1, the ship entity processing is performed on each ship image in the set of ship remote sensing images to obtain the ship entity image of each ship remote sensing image and the background image of each ship remote sensing image.

[0062] All ship entity images of the ship remote sensing images are combined to form an entity image set; all background images of the ship remote sensing images are combined to form a background image set.

[0063] S1.2.1.3: constructing a recombined ship image set according to the entity image set and the background image set obtained in S1.2.1.2;

[0064] S1.2.1.4: performing screening processing on the recombined ship image set obtained in S1.2.1.3 to obtain a screened ship image set; the specific process is as follows:

[0065] manually filtering out images that do not meet the requirements,

[0066] such as the overlap of ships and ports.

[0067] The present application obtains 200 new images with ship entity and background recombination through this method, thereby increasing the number of training samples

[0068] S1.2.1.5: training the image style conversion network according to the screened ship image set to obtain a trained data enhancement module

[0069] The other steps and parameters are the same as one of the first two embodiments.

[0070] Embodiment four: the difference between the present embodiment and the first four embodiments is that,

[0071] The image segmentation network in S1.2.1.1 is a mask2former image segmentation model;

[0072] The mask2former image segmentation model is a general image segmentation model

[0073] S1.2.1.3 in the present application: constructing a recombined ship image set according to the entity image set and the background image set obtained in S1.2.1.2; the specific process is as follows:

[0074] S1.2.1.3.1: randomly extracting one entity image and one background image from the entity image set and the background image set obtained in S1.2.1.2 for combination; obtaining one recombined ship image;

[0075] S1.2.1.3.2: repeating S1.2.1.3.1 N times to obtain N recombined ship images; N is a positive integer; constructing a recombined ship image set from the N recombined ship images;

[0076] The image style conversion network in S1.2.1.5 includes a feature extraction network and a CycleGAN network;

[0077] S1.2.1.5 in the present application: training the image style conversion network according to the screened ship image set to obtain a trained data enhancement module; the specific process is as follows:

[0078] S1.2.1.5.1: taking the image A in the screening ship image set and the background image A' corresponding to the image A as inputs of the data augmentation module;

[0079] The input image A is processed by the generator G in the CycleGAN model to obtain the style converted image B;

[0080] The style converted image B is output as the output of the data augmentation module;

[0081] S1.2.1.5.2: calculating the feature mask M(A) of the image A according to the image A and the background image A' corresponding to the image A,

[0082] Wherein, the feature mask of the i-th row and j-th column of the image A is represented as M ij , and is expressed by the formula as:

[0083]

[0084] In the formula, A ij represents the pixel point of the i-th row and j-th column of the image A, A' ij represents the pixel point of the i-th row and j-th column of the original background image A' of the image A; and M ij The feature masks of all pixel points of the image A are merged into M(A).

[0085] According to the input image A, the image B and the feature mask M(A) of the image A, the feature transformation loss L F in the style conversion process is calculated.

[0086] S1.2.1.5.3: using the discriminator D Y in the CycleGAN model to calculate the stylization loss L GAN-G of the image B.

[0087] L GAN-G is a standard adversarial loss known in the CycleGAN model, which is referred to as a stylization loss according to the role, and is known to those skilled in the art

[0088] S1.2.1.5.4: inputting the image B into the generator F in the CycleGAN model to obtain the style converted image C;

[0089] S1.2.1.5.5: using the discriminator D X in the CycleGAN model to calculate the stylization loss L GAN-F of the image C.

[0090] L GAN-Fis a standard adversarial loss known in CycleGAN model, here referred to as style loss, which is well known to those skilled in the art

[0091] S1.2.1.5.6: Cycle loss L calculated according to image A and image C cyc ;

[0092] L cyc is a standard cycle consistency loss known in CycleGAN model, here referred to as cycle loss, which is well known to those skilled in the art

[0093] S1.2.1.5.7: Total loss L calculated according to feature transformation loss L F , style loss L GAN-F , style loss L GAN-F and cycle loss L cyc style , repeat iterations S1.2.1.5.1 to S1.2.1.5.6, stop iteration when the total loss function is minimized, and obtain the trained data enhancement module.

[0094] Other steps and parameters are the same as one of the first to third embodiments.

[0095] Embodiment five: the difference between this embodiment and the first to fourth embodiments is that,

[0096] The formula for calculating the feature transformation loss L F in S1.2.1.5.2 is:

[0097]

[0098] Where, denotes expectation, A ~ Pdata(A) denotes the data distribution of image A, || ||2 denotes the two norm, ^ denotes the XOR operation, and M(A) denotes the feature mask obtained in S1.2.1.5.2.

[0099] The feature extraction module selects the part between the ship image set image A and the original background image A' of image A that has changed as the selected feature, i.e. the part where the newly added ship body is located.

[0100] The formula for calculating the total loss L style in S1.2.1.5.7 according to the feature transformation loss L F , style loss L GAN-F , style loss L GAN-F and cycle loss L cyc is:

[0101] L style = L GAN-G + L​GAN-F + λ1L cyc + λ2L F

[0102] wherein λ1, λ2 represent weight parameters

[0103] Other steps and parameters are the same as one of the first to fourth embodiments.

[0104] The sixth embodiment is different from the first to fifth embodiments in that,

[0105] The target detection module in S1.2.2 is an LSKNet network.

[0106] The target detection module in S1.2.2 is trained according to the enhanced ship remote sensing image set, to obtain a trained target detection module; the specific process is:

[0107] S1.2.2.1: Randomly determine a filling position for each image in the enhanced ship remote sensing image set; fill a fixed size rectangular region with Gaussian noise at all filling positions to obtain a filled ship remote sensing image set;

[0108] S1.2.2.2: Input the filled ship remote sensing image set into the LSKNet network to obtain the coordinates of the ship object in the ship remote sensing image and the square range in which the ship object is located in the picture,

[0109] S1.2.2.3: Calculate the training loss according to the coordinates of the ship object in the ship remote sensing image and the square range in which the ship object is located in the picture, repeat S1.2.2.1 to S1.2.2.2, and when the training loss is the smallest, the trained target detection module is obtained

[0110] ; other steps and parameters are the same as one of the first to fifth embodiments.

[0111] The seventh embodiment is different from the first to sixth embodiments in that,

[0112] The fine-grained picture classification module sequentially includes a vit visual conversion network and a HERBS network.

[0113] The fine-grained picture classification module in S1.2.3 is trained according to the cut ship remote sensing image set, to obtain a trained fine-grained picture classification module; the specific process is:

[0114] S1.2.3.1: Construct a negative sample image set and a positive sample image for each picture in the cut ship remote sensing image set;

[0115] S1.2.3.2: Input image A, the negative sample image set of image A, and the positive sample image of image A from the cropped remote sensing image set of the ship together into the VIT visual transformation network to obtain feature map A* and the feature map set A of the negative samples. — Feature map A of positive samples + ;

[0116] S1.2.3.3: Combine feature map A* and the feature map set A of negative samples. — Feature map A of positive samples + The inputs are fed into the HERBS network to obtain feature map B* and the feature map set B of its negative samples. — Feature map B of positive samples + ;

[0117] Feature map B* is used as the output of the fine-grained image classification module;

[0118] The trained fine-grained image classification module outputs this feature map B*, and the specific model of the ship in the input image D is obtained based on the feature map B*.

[0119] S1.2.3.4: Based on feature map B* and the feature map set B of its negative samples — Feature map B of positive samples + Calculate the contrast loss L con ;

[0120] S1.2.3.5: Based on the contrast loss L con Calculate the total loss L, and repeat steps S1.2.3.1 to S1.2.3.4. When the total loss is minimized, stop the iteration to obtain the trained fine-grained image classification module.

[0121] The HERBS network model is a well-known network model in the field. The following is a brief introduction to the structure of the HERBS network model, including the background suppression module and the high-temperature thinning module.

[0122] The first step in the background suppression (BS) module processing is to generate a classification map from the feature map. The feature map output by the HERBS network model of this invention is the classification map; then, based on the classification map Y... i Calculate the maximum score P max,i This process can be represented as:

[0123] Y i =W i hs i +b i

[0124] P max,i =max((Softmax(Y) i ))

[0125] Next, top-K feature selection is performed based on all predicted scores.

[0126] Then, the graph convolutional network of the background suppression module is used to fuse the selected features, and a prediction is made based on the fused features. The merged classification prediction is denoted as Y. m .

[0127] The predicted distribution P is calculated using cross-entropy. m The similarity between the merged label and the true label y. The loss after merging is calculated as follows:

[0128] P m =Softmax(Y m )

[0129]

[0130] Another purpose of the BS module is to suppress the loss of features in the classification map and increase the gap between the foreground and background.

[0131] Applying the hyperbolic tangent function tanh to the deleted classification map Y d The loss is calculated using the mean squared error between the prediction and the spurious target, which is 1.

[0132] P d =tanh(Y d )

[0133]

[0134] The high-temperature refinement module includes classifiers k1 and k2, with the goal of enabling classifier k1 to learn the output distribution of classifier k2. The output of classifier k1 is defined as Y. i1 The output of classifier k1 is Y i2 Refining the objective function helps the model learn more diverse and stronger representations in early layers, while allowing later layers to focus on finer details. The high-temperature refinement module enables classifier k1 to discover a wider range of regions, while classifier k2 focuses on learning fine-grained and discriminative features. The formula for calculating the refinement loss is as follows:

[0135] P i1 =LogSoftmax(Y i1 / T e )

[0136] P i2 =Softmax(Y i2 / T e )

[0137]

[0138] Compared with the knowledge distillation method, the initial temperature T is set to a higher value. The purpose of doing this is to encourage the model to explore various features, even if the initial prediction is not accurate. Then, as the training progresses, the temperature gradually decreases, making the model pay more attention to the target class and learn more discriminative features. By using this decay strategy, the model can obtain a diverse and fine-grained representation and make accurate predictions.

[0139] The other steps and parameters are the same as one of embodiments one to six.

[0140] Embodiment eight: The difference between this embodiment and embodiments one to seven is that,

[0141] In S1.2.3.1, a negative sample image set and a positive sample image are constructed for each cut picture in the set of cut ship remote sensing images after cutting; the specific process is:

[0142] S1.2.3.1.1: Determine the ship model of all cut pictures; the confused ship model of each ship model; the specific process is: according to the label of the data set determined by the HERBS network, determine the ship model of all cut pictures;

[0143] According to the confusion matrix of the initial classification result in the data set determined by the HERBS network, find 1 to 3 misclassified for each ship model as the confused ship model of the model;

[0144] The confused ship model is the top 3 in the error rate of the wrong model;

[0145] S1.2.3.1.2: Construct a negative sample image set and a positive sample image for all cut pictures according to the confused ship model;

[0146] The specific process of constructing a negative sample image set and a positive sample image for a cut picture is:

[0147] Randomly select a ship remote sensing image of the confused ship model corresponding to the cut picture from the enhanced ship remote sensing image set,

[0148] Randomly select one image for each confused ship model, and if there are not three types, repeat the selection until three remote sensing images are obtained as the negative sample images of the cut picture,

[0149] Randomly select a ship remote sensing image of the ship model corresponding to the cut picture from the enhanced ship remote sensing image set as the positive sample image of the cut picture

[0150] The other steps and parameters are the same as one of embodiments one to seven.

[0151] Specific implementation nine: the difference between this embodiment and specific implementation one to eight is that,

[0152] The feature map set B of the feature map B* and its negative sample in S1.2.3.4 according to feature map B — And the feature map B of the positive sample + The formula for calculating the contrast loss L con

[0153]

[0154] Wherein, B * represents the feature map, B + is the positive sample feature map, B - is the negative sample feature map set, Sim(.) represents the dot product processing of two feature map vectors; max(.) represents taking the maximum value; α is a constant margin, so max(Sim(z i ,z n )-α,0) makes only the negative sample with similarity greater than α in the negative sample cause loss, thereby improving the accuracy of network prediction

[0155] Other steps and parameters are the same as one of specific implementation one to eight.

[0156] Specific implementation ten: the difference between this embodiment and specific implementation one to nine is that,

[0157] The formula for calculating the total loss L in S1.2.3.5 according to the contrast loss L con

[0158] L=λ m L m +λ d L d +λ l L l +λ con L con

[0159] Wherein, L m is the classification map prediction loss in HERBS network, L d is the classification map discard loss in HERBS network, L l is the distillation refinement loss in HERBS network, L m , L d and L l are known loss functions in HERBS network, λ m , λ d , λ l , λ con are weight coefficients.

[0160] ​​Other steps and parameters are the same as one of embodiments one to nine.

[0161] Simulation analysis combined with embodiments one to ten

[0162] Since the present application is aimed at fine-grained target recognition of sea surface remote sensing pictures, and is divided into two sub-tasks of coarse-grained target recognition for ships and fine-grained picture classification for ship types, for the target detection sub-task, the first level label (i.e. ship label) in HRSC2016 is used for training, and for the fine-grained image classification sub-task, 436 images with third level labels (i.e. 23 labels of specific types) in the training set are used for training. Finally, the overall performance of the overall architecture is tested on 181 three-level label images in the HRSC2016 test set. The overall performance of the architecture is evaluated respectively. For the evaluation of the target detection sub-task and the final model, mAP is used as the evaluation index. For the fine-grained image classification sub-task, precision is used as the evaluation index.

[0163] As a comparison, the present application tests the performance of existing target detection methods on HRSC2016, a total of faster-RCNN, yolov5, LSKNet, DETReg and STD are tested for fine-grained target detection performance on sea surface remote sensing pictures. The performance of the model on the fine-grained object detection task is reported in Table 1.

[0164] Table 1 Comparison of performance of different target detection methods on HRSC2016

[0165]

[0166] It can be seen that the system proposed in the present application improves the mAP by 1.26% than the strongest baseline. This proves that the adopted architecture is superior to previous general target detection models in capturing the details of small targets in complex backgrounds. At the same time, the data enhancement and contrast learning module added also effectively solves the problems of scarcity of training samples and imbalance of class number in the current fine-grained target detection in the field of ships.

[0167] Replacement test of some key steps of the present application

[0168] Here, the present application replaces the implementation method of some key steps to observe the performance changes, and the optimization contribution results of each key step are shown in Table 2.

[0169] Removing the effect of data augmentation. This part studies the importance of data augmentation. The present application generates new images by entity background separation and random splicing. The pre-augmented dataset and the augmented dataset are used to train the subtasks of image classification, and then test on the unified test set. Take TransFG which performs well in fine-grained image classification as a control, and the experimental results are shown in Table 2. From the results, it can be seen that after removing the augmented data, the classification accuracy of the two models has decreased, among which TransFG has decreased by 1.9%, and the model proposed in the present application has decreased by 1.8%. This is because the addition of new images effectively alleviates the overfitting problem and introduces different background features.

[0170] Table 2 Ablation experiment results of data augmentation on HRSC2016 dataset

[0171]

[0172] Removing the effect of contrastive learning. This part studies the importance of contrastive learning, and the present application removes the contrastive learning module in the fine-grained image classification module. Table 3 shows the performance comparison of image classification models with and without contrastive loss to verify its effectiveness. From the results, it can be seen that in the case of removing the contrastive loss, the accuracy of transFG decreases by 1.0%, and the accuracy of our model decreases by 0.7%, because the contrastive loss can effectively expand the representation distance between similar subcategories and reduce the representation distance between the same subcategories.

[0173] Table 3 Ablation experiment results of contrastive learning on HRSC2016 dataset

[0174]

[0175] The above is only the preferred embodiment of the present application, it needs to be understood that the present application is not limited to the above specific embodiments, although the present application has been disclosed as above with the preferred embodiment, however, it is not intended to limit the present application, any skilled person in the art, without departing from the technical solution of the present application, can make some changes or modifications to the equivalent embodiments of equivalent changes, but as long as it does not deviate from the technical solution of the present application, according to the technical essence of the present application, within the spirit and principles of the present application, any simple modification, equivalent replacement and improvement of the above embodiments, all still belong to the protection scope of the technical solution of the present application.

Claims

1. A ship detection method based on remote sensing images, characterized in that, The method comprises the following steps: S1: constructing a ship detection model based on remote sensing images; obtaining a trained ship detection model based on remote sensing images; S2: collecting a ship remote sensing image to be identified, inputting the ship remote sensing image to be identified into the trained ship detection model based on remote sensing images, and obtaining a ship model of the ship remote sensing image to be identified; The S1 of constructing a ship detection model based on remote sensing images; obtaining a trained ship detection model based on remote sensing images; the specific process is: S1.1: obtaining a ship remote sensing image set; the ship remote sensing image set is an HRSC2016 remote sensing image ship data set; S1.2: constructing a ship detection model based on remote sensing images; the ship detection model based on remote sensing images comprises in sequence: a data enhancement module, a target detection module and a fine-grained picture classification module; According to the ship remote sensing image set, the ship detection model based on remote sensing images is trained to obtain a trained ship detection model based on remote sensing images; the specific process is: S1.2.1: training the data enhancement module according to the ship remote sensing image set to obtain a trained data enhancement module; Then, the images of the ship remote sensing image set are input into the trained data enhancement module to obtain an enhanced ship remote sensing image set; the data enhancement module comprises an image segmentation network and an image style conversion network; The specific process of training the data enhancement module according to the ship remote sensing image set to obtain a trained data enhancement module is: S1.2.1.1: inputting each ship remote sensing image of the ship remote sensing image set into the image segmentation network in the data enhancement module for extraction processing to obtain a mask of each ship remote sensing image in the ship remote sensing image set; S1.2.1.2: extracting each ship image in the ship remote sensing image set based on the mask of each ship remote sensing image extracted in S1.2.1.1 to obtain a ship entity image of each ship remote sensing image and a background image of each ship remote sensing image; All ship entity images of the ship remote sensing images form an entity image set; all background images of the ship remote sensing images form a background image set; S1.2.1.3: constructing a reorganized ship image set according to the entity image set and the background image set obtained in S1.2.1.2; S1.2.1.4: screening the reorganized ship image set obtained in S1.2.1.3 to obtain a screened ship image set; S1.2.1.5: training the image style conversion network according to the screened ship image set to obtain a trained data enhancement module S1.2.2: training the target detection module according to the enhanced ship remote sensing image set to obtain a trained target detection module; Then, the enhanced ship remote sensing image set is input into the trained target detection module; the ship object in the enhanced ship remote sensing image and the position of the ship object are obtained; The position of the ship object is the coordinate of the square range where the ship object is located in the picture; S1.2.3: cutting all ship objects in the enhanced ship remote sensing image according to the position of the ship object to obtain a cut ship remote sensing image set; Each of the cut images in the set of cut ship remote sensing images contains only one ship; The fine-grained picture classification module is trained according to the set of cut ship remote sensing images, and a trained fine-grained picture classification module is obtained. The fine-grained picture classification module sequentially includes a vit visual conversion network and a HERBS network. The fine-grained picture classification module is trained according to the set of cut ship remote sensing images, and a trained fine-grained picture classification module is obtained. S1.2.3.1: Construct a negative sample image set and a positive sample image for each image in the set of cut ship remote sensing images; S1.2.3.2: input the cut ship remote sensing image set of pictures D, the negative sample image set of picture D and the positive sample image of picture D into the vit visual conversion network together to obtain the feature map D*, the feature map set D of negative samples and the feature map D of positive samples — . + ; S1.2.3.3: input the feature map D*, the feature map set D — of negative samples, and the feature map D + of positive samples into the HERBS network respectively to obtain the feature map B* and the feature map set B — of negative samples of B + ​ The feature map B* is output by the fine-grained picture classification module, and the specific model of the ship in the input picture D is obtained according to the feature map B*; S1.2.3.4: according to the feature map set B of the feature map B* and its negative samples — and the feature map B of the positive sample + computing the contrastive loss L con ; S1.2.3.5: according to the contrast loss L con Calculate the total loss L, repeat S1.2.3.1 to S1.2.3.4, stop iteration when the total loss is the smallest, and get the trained fine-grained picture classification module. S1.2.4: Combine the trained data enhancement module, the trained target detection module and the trained fine-grained picture classification module to obtain a trained ship detection model based on remote sensing images.

2. The ship detection method based on remote sensing images according to claim 1, characterized in that, The image segmentation network in S1.2.1.1 is a mask2former image segmentation model; The recombined ship image set is constructed according to the entity image set and the background image set obtained in S1.2.1.2 in S1.2.1.3, and the specific process is as follows: S1.2.1.3.1: Randomly select an entity image and a background image from the entity image set and the background image set obtained in S1.2.1.2 to combine them to obtain a recombined ship image; S1.2.1.3.2: Repeat S1.2.1.3.1 N times to obtain N recombined ship images; N is a positive integer; The N recombined ship images are constructed into a recombined ship image set; The image style conversion network in S1.2.1.5 includes a feature extraction network and a CycleGAN network; The CycleGAN network comprises a generator G, a discriminator D Y , a generator F and a discriminator D X ; The trained data enhancement module is obtained by training the image style conversion network according to the filtered ship image set in S1.2.1.5, and the specific process is as follows: S1.2.1.5.1: Take the image A in the filtered ship image set and the background image A' in the background image set corresponding to the image A as inputs of the data enhancement module; The input image A is processed by the generator G in the CycleGAN network to obtain image B; The style converted image B is output by the data enhancement module; S1.2.1.5.2: Calculate the feature mask M(A) of image A according to image A and the background image A' in the background image set corresponding to image A, wherein the feature mask of the i-th row and j-th column of image A is denoted as M ij In formula, In the formula, A ij represents a pixel point of the i-th row and the j-th column of the image A of the screening ship image set, A' ij represents a pixel point of the i-th row and the j-th column of the background image A' Computing a feature transformation loss L in the style transfer process from the input image A, the image B, and a feature mask M(A) of the image A F ; S1.2.1.5.3: Using the discriminator D in the CycleGAN network Y Computing the stylization loss L on the image B GAN-G ; S1.2.1.5.4: Input image B into the generator F in the CycleGAN network to obtain image C; S1.2.1.5.5: Using the discriminator D in the CycleGAN model X Computing the stylization loss L on the image C GAN-F ; S1.2.1.5.6: Cycle loss L calculated from image A and image C cyc ; S1.2.1.5.7: according to the feature transformation loss L F , the stylization loss L GAN-G , the stylization loss L GAN-F and the cycle loss L cyc calculate the total loss L style , repeat iterations S1.2.1.5.1 to S1.2.1.5.6, stop iteration when the total loss function is minimum, and obtain the trained data enhancement module.

3. The ship detection method based on remote sensing images according to claim 2, wherein, The formula of the calculated feature transformation loss L in S1.2.1.5.2 is: F The formula is: wherein, represents expectation, A ~ Pdata(A) represents a data distribution to which image A is subject, || ||2 represents a two-norm, and represents an exclusive OR operation. The formula of the total loss L F , the stylization loss L GAN-G , the stylization loss L GAN-F and the cycle loss L cyc The formula of the total loss L style is: L style = L GAN-G + L GAN-F + λ1L cyc + λ2L F Wherein λ1, λ2 represent weight parameters.

4. The ship detection method based on remote sensing images according to claim 3, wherein, The target detection module in S1.2.2 is an LSKNet network. The S1.2.2.2 in the S1.2.2 is used for training the target detection module according to the enhanced ship remote sensing image set, and a trained target detection module is obtained; the specific process is: S1.2.2.1: a filling position is randomly determined for each image in the enhanced ship remote sensing image set; a fixed size rectangular region is filled with Gaussian noise at all filling positions to obtain a filled ship remote sensing image set; S1.2.2.2: input the filled ship remote sensing image set into the LSKNet network to obtain the coordinates of the ship object in the ship remote sensing image and the square range thereof in the picture, S1.2.2.3: calculate the training loss according to the coordinates of the ship object in the ship remote sensing image and the square range thereof in the picture, repeat S1.2.2.1 to S1.2.2.2, and obtain the trained target detection module when the training loss is minimum.

5. The ship detection method based on remote sensing images according to claim 4, wherein S1.2.3.1 in the S1.2.3 is used for constructing a negative sample image set and a positive sample image for each cut picture in the cut ship remote sensing image set; the specific process is: S1.2.3.1.1: determine the ship model of all cut pictures; the confused ship model of each ship model; the specific process is: according to the label of the data set determined by the HERBS network, determine the ship model of all cut pictures; According to the confusion matrix of the initial classification result of the data set by the HERBS network, find 1 to 3 misclassified ship models for each ship model as the confused ship model of the ship model; S1.2.3.1.2: construct a negative sample image set and a positive sample image for all cut pictures according to the confused ship model; The specific process of constructing a negative sample image set and a positive sample image for a certain cut picture is: randomly select a ship remote sensing image of the confused ship model corresponding to the cut picture from the enhanced ship remote sensing image set, randomly select one image of each confused ship model as the negative sample image of the cut picture, randomly select a ship remote sensing image of the ship model corresponding to the cut picture from the enhanced ship remote sensing image set as the positive sample image of the cut picture.

6. The ship detection method based on remote sensing images according to claim 5, wherein The feature map set B of the S1.2.3.4 according to the feature map B* and the feature map set B of the negative sample — and the feature map B of the positive sample + The formula of the contrast loss L con is: where B * represents a feature map, B + is a positive sample feature map, B - is a negative sample feature map set, Sim(.) represents dot product processing; max(.) represents taking the maximum value; and a is a constant margin.

7. The ship detection method based on remote sensing images according to claim 6, wherein In the S1.2.3.5, the contrast loss L con The formula for calculating the total loss L is: L = λ m L m + λ d L d + λ l L l + λ con L con where L m is the classification map prediction loss in the HERBS network, L d is the classification map dropout loss in the HERBS network, L l is the distillation refinement loss in the HERBS network, λ m , λ d , λ l , λ con are weight coefficients.

Images