Ship extraction method and system based on double tasks and double attention

Abstract

The invention relates to the technical field of remote sensing image ship extraction, in particular to a ship extraction method and system based on double tasks and double attention, which are used for acquiring remote sensing image data of typical port areas in a global range and completing manual annotation. Obtaining a ship semantic segmentation data set containing ship pixel-level binary labels and ship contour labels; a remote sensing image ship extraction model combining double tasks and double attention is constructed, the model comprises a double-task network module, and the double-task network module comprises a main task branch and an auxiliary task branch; the main task branch comprises a coding residual module and a double attention module, input data is a remote sensing image feature map, and output data is a ship pixel-level binary label; the input data of the auxiliary task branch comes from the coding residual module of the main task branch, and the output data is a ship contour label. According to the method, the model training effect is improved through the self-made ship semantic segmentation data set, and the ship region can be efficiently and accurately segmented by the remote sensing image ship extraction model.

Description

Technical Field

[0001] The present invention relates to the technical field of ship extraction from remote sensing images, and in particular to a ship extraction method and system based on dual tasks and dual attention. Background Art

[0002] In recent years, with the rapid development of satellite remote sensing imaging technology, ship extraction in remote sensing images has become an important application direction for remote sensing target recognition. Fully Convolutional Networks (FCN), as the first network to achieve end-to-end pixel-level prediction, replaces the fully connected layer in the traditional Convolutional Neural Networks (CNN) with a fully convolutional structure and uses deconvolution for feature upsampling. On this basis, Liu et al. proposed a method to obtain and fuse multi-scale contextual features using a spatial residual truncation module. Pan Xuran et al. proposed an improved method of replacing the convolutional layer in FCN with an Inception module with a multi-scale convolution kernel. However, these methods mainly focus on extracting shallow features of ships and are less sensitive to deep features.

[0003] In terms of data sets, the current data sets used for ship detection and recognition include the Fine-grained Ship Detection Dataset for Optical Remote Sensing Images (HRSC2016 Dataset) published by Liu et al. in 2016, the Large-Scale Horizontal Box Dataset for General Target Detection in Optical Remote Sensing Images (DIOR Dataset) published by Cheng et al. of Northwestern Polytechnical University in 2019, and the Large-Scale Rotation Box Dataset for Target Detection in Optical Remote Sensing Images (DOTA Dataset) published by Xia et al. of Wuhan University in 2018. However, both the DIOR and DOTA data sets use rectangular box annotations, which only provide the location and category information of ships in remote sensing images, and lack refined pixel-level annotations.

[0004] The target detection technology based on deep learning has the characteristics of high precision, high efficiency and wide applicability, and has achieved certain results in the task of ship detection in remote sensing images. However, current research still faces two problems in practical applications. The first is the problem of insufficient data reference. There are very few data sets suitable for ship extraction from remote sensing images, and the existing data sets are limited by the characteristics of small number of samples, incomplete scene categories, and overly dense regional distribution, which are not enough to represent the actual scene. The second is the problem of poor interpretation accuracy. In high-resolution remote sensing images, due to the interference of realistic conditions such as cloud and fog occlusion, complex background of ground objects, and diverse ship types, there are obvious omissions, false detections, and blurred edges in ship extraction, which leads to unsatisfactory interpretation accuracy. Summary of the invention

[0005] The present invention aims to solve the problems of insufficient data reference and poor interpretation accuracy in the current remote sensing image ship extraction task, and proposes a ship extraction method and system based on dual tasks and dual attention, which can efficiently and accurately segment the ship area.

[0006] To achieve the above purpose, the technical solution adopted is:

[0007] A ship extraction method based on dual tasks and dual attention, including:

[0008] Based on the public dataset, we obtain remote sensing image data of typical port areas around the world, complete manual annotation, and obtain a ship semantic segmentation dataset containing ship pixel-level binary labels and ship contour labels;

[0009] A remote sensing image ship extraction model with joint dual-task and dual-attention is constructed. The model includes a dual-task network module, which includes a main task branch and an auxiliary task branch. The main task branch includes a coding residual module and a dual attention module. The input data is the remote sensing image feature map, and the output data is the pixel-level binary label of the ship. The input data of the auxiliary task branch comes from the coding residual module of the main task branch, and the output data is the ship contour label.

[0010] According to the ship extraction method based on dual-task and dual-attention of the present invention, further, the remote sensing image data is sliced ​​and the images are uniformly cropped into a size of 256×256 pixels; a 10% overlap is designed during the slicing process; the ship semantic segmentation dataset includes offshore ships, docked ships and ships on land; the ship semantic segmentation dataset includes ships with different aspect ratios.

[0011] According to the ship extraction method based on dual tasks and dual attention of the present invention, further, the coding residual module includes a 7×7 convolution, a global maximum pooling layer, a relu activation function and four coding layers, and the remote sensing image feature map is sent to the 7×7 convolution for convolution operation, and global maximum pooling and activation operations are performed; then the output feature map is input into four coding layers, and four downsampling operations are performed.

[0012] According to the ship extraction method based on dual tasks and dual attention of the present invention, further, the dual attention module includes a multi-head self-attention module and a channel attention module.

[0013] According to the ship extraction method based on dual tasks and dual attention of the present invention, further, the main task branch also includes four decoding layers, 1 4×4 transposed convolution, 1 3×3 convolution and Sigmoid activation function, each decoding layer uses a jump connection to take the output of the corresponding encoding layer as its input, and introduces a dual attention module on the jump connection branch; the output of the last encoding layer of the encoding residual module is sent to four decoding layers for four upsampling operations, and the upsampled feature map is sequentially subjected to transposed convolution, convolution and activation operations to obtain the ship pixel-level binary label.

[0014] According to the ship extraction method based on dual tasks and dual attention of the present invention, further, the auxiliary task branch includes three decoding layers, one 4×4 transposed convolution, one 3×3 convolution and a Sigmoid activation function, each decoding layer uses a jump connection to take the output of the corresponding encoding layer as its input, and the output of the last encoding layer of the encoding residual module is sent to the three decoding layers for three upsampling operations. The upsampled feature map is sequentially subjected to transposed convolution, convolution and activation operations to obtain the ship contour label.

[0015] According to the ship extraction method based on dual tasks and dual attention of the present invention, further, the total loss function of the remote sensing image ship extraction model combining dual tasks and dual attention is defined as follows:

[0016] L total =L s +αL B

[0017] L=L Bce +L Dice

[0018] Among them, L total , L S , L B They represent the total loss, main task loss and auxiliary task loss respectively, α is a hyperparameter, L represents the main task loss or auxiliary task loss, and L Bce represents the binary cross entropy loss, L Dice Denotes Dice loss.

[0019] Furthermore, the present invention also proposes a ship extraction system based on dual tasks and dual attention, comprising a data set production module and a model construction module, wherein:

[0020] The dataset production module is used to obtain remote sensing image data of typical port areas around the world based on public datasets, complete manual annotation, and obtain a ship semantic segmentation dataset containing ship pixel-level binary labels and ship contour labels;

[0021] The model building module is used to build a remote sensing image ship extraction model with joint dual-task and dual-attention. The model includes a dual-task network module, which includes a main task branch and an auxiliary task branch. The main task branch includes a coding residual module and a dual attention module. The input data is the remote sensing image feature map, and the output data is the pixel-level binary label of the ship. The input data of the auxiliary task branch comes from the coding residual module of the main task branch, and the output data is the ship contour label.

[0022] The beneficial effects achieved by adopting the above technical solution are:

[0023] 1. The ship extraction method based on dual tasks and dual attention proposed in this paper constructs a ship semantic segmentation dataset (SSSD) with sufficient quantity, diverse ship types, complex scenes, and rich spatial distribution. Based on the public dataset, open source images of typical port areas are obtained worldwide, and datasets are produced through manual annotation, providing rich data support for model training. This makes the remote sensing image ship extraction model trained on SSSD more adaptable to various complex and extreme situations, showing stronger robustness and wide adaptability.

[0024] 2. This paper designs a remote sensing image ship extraction model that combines dual tasks and dual attention. Based on the codec network, the multi-head self-attention and channel attention are used to enhance the ability to capture ship features, thereby constructing a more effective feature space in network training and improving the accuracy of ship segmentation. The contour supervision auxiliary task branch is introduced to enhance the distinction between ships and backgrounds, further improving the detail cutting effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings of the embodiments of the present invention, wherein the drawings are only used to illustrate some embodiments of the present invention, but not to limit all embodiments of the present invention thereto.

[0026] Figure 1 is a flow chart of a ship extraction method based on dual tasks and dual attention according to an embodiment of the present invention;

[0027] Figure 2 is a typical port area distribution map of an embodiment of the present invention;

[0028] Figure 3 1 is a partial data set example image of an embodiment of the present invention.

[0029] Figure 4 is an example image of distribution of different ships according to an embodiment of the present invention;

[0030] Figure 5 are example images of ships with different aspect ratios according to an embodiment of the present invention;

[0031] Figure 6 is a structural diagram of a remote sensing image ship extraction model combining dual tasks and dual attention according to an embodiment of the present invention;

[0032] Figure 7 is a structural diagram of each decoding layer in an embodiment of the present invention;

[0033] Figure 8 is a structural diagram of a dual attention module according to an embodiment of the present invention;

[0034] Fig. 9 is a structural diagram of a multi-head self-attention module according to an embodiment of the present invention;

[0035] Fig.10 is a structural diagram of a channel attention module according to an embodiment of the present invention;

[0036] Fig.11 is an example of segmentation results of different models of ships according to an embodiment of the present invention;

[0037] Fig.12 is the large-scale Zhenhai Port application recognition result of an embodiment of the present invention;

[0038] Fig.13 is the large-scale San Diego port application recognition result of an embodiment of the present invention;

[0039] Fig.14 This is a diagram showing the effect of a test set without fog or clouds under an embodiment of the present invention;

[0040] Fig.15 This is a diagram showing the effect of an experiment with foggy conditions in the test set of an embodiment of the present invention;

[0041] Fig.16 It is a statistical chart of the experimental results of a small sample of an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings of specific embodiments of the present invention to clearly and completely describe the exemplary scheme of the embodiment of the present invention. Unless otherwise defined, the technical terms or scientific terms used in the present invention should be the common meanings understood by people with ordinary skills in the relevant field.

[0043] In order to overcome the limitations of the prior art in the ship extraction task, this embodiment discloses a ship extraction method based on dual tasks and dual attention, such as Figure 1 As shown, it contains the following contents:

[0044] Step S101: Based on the public dataset, remote sensing image data of typical port areas around the world are obtained, and manual annotation is completed to obtain a ship semantic segmentation dataset (SSSD) containing ship pixel-level binary labels and ship contour labels.

[0045] The ground object extraction technology based on remote sensing images requires a large number of samples to train the network model. The quality of the samples will directly affect the performance of the entire network model to a certain extent. Therefore, it is particularly important to establish a dataset based on high-resolution remote sensing images with accurate annotations, high ship boundary clarity, and strong scene diversity. Therefore, based on the HRSC2016 dataset with pixel-level annotations, typical port areas are selected worldwide, and combined with manual annotations, a ship semantic segmentation dataset with sufficient quantity, rich ship types, and diverse scenes is finally formed.

[0046] The typical port areas selected by SSSD are distributed as follows: Figure 2 As shown, it includes the Yokosuka port area in Japan, the Sasebo port area in Japan, and the Zhoushan port area in China on the west coast of the Pacific Ocean; the Bremerton port area in the United States and the San Diego port area in the United States on the east coast of the Pacific Ocean; the Brest port area in France, the Toulon port area in France, and the Norfolk port area in the United States on the Atlantic coast; and the Mumbai port area in India on the Indian Ocean coast, etc.

[0047] For the HRSC2016 dataset, the original multi-category pixel-level labels are post-processed into binary labels. Specifically, by traversing the label images in the HRSC2016 dataset, the grayscale values ​​of the pixels representing the ship labels are uniformly set to 255, and the grayscale values ​​of other pixels are set to 0, so as to obtain pixel-level binary label data that meets the requirements.

[0048] For the self-made data set, we first rely on the Shuijingzhu software to obtain remote sensing image data of various port areas around the world, and then perform manual annotation to obtain pixel-level binary labels for ships. In addition, ship contour labels are crucial to improving the accuracy and efficiency of ground object interpretation. It not only helps to accurately identify and classify ships, but also enhances the model's focus on the key features of ships during the learning process. Therefore, ship contour labels are introduced as new label data. First, the Canny operator is used to extract edges and find contours from the above binary label data. At the same time, considering the aggregation characteristics of the network prediction value, too narrow contours are not enough to provide sufficient supervision information, resulting in large loss values, which in turn affects the back propagation of the gradient and the optimization effect of the model, resulting in poor model training results. Therefore, a contour line with a width of 5 pixels is drawn on the basis of the original contour label to generate a binary contour label for the ship. Some data sets such as Figure 3 shown.

[0049] Since the ship images in the HRSC2016 dataset and the self-made dataset have large differences in size, it is not conducive to directly inputting the model for training. Therefore, the remote sensing image data is sliced ​​and the images are uniformly cut into 256*256 pixel sizes. At the same time, in order to enhance the application effect of the data in large scenes, a 10% overlap is designed in the slicing process (the upper and lower overlaps are both 10%). For the right and lower edges of the image, the size is less than 256*256 pixels, and it is directly discarded. For ship labels and contour labels, the same cutting method as the ship images is used to finally obtain a complete data set.

[0050] The spatial distribution of ships is of great significance for marine monitoring and security. According to the background of the ships, they are divided into three types: offshore, shore and land. Among the three types of ships, shore ships are the majority. Specifically, the number of ships of the three types is shown in Table 1. There are 67 offshore ships, 1,734 shore ships and 27 land ships.

[0051] Table 1 Ship distribution type and quantity

[0052]

[0053] Different backgrounds have different levels of difficulty in extraction. The background of offshore ships is simple, which has little impact on the model's extraction of ship features and is easy to extract. The background of ships on land is complex and has strong interference with ship extraction. This distribution is consistent with the reality that ships spend a lot of time at ports, and it also reflects that the data set contains a variety of application scenarios. Through quantitative analysis of the spatial characteristics of ships, we can better understand the distribution of ships in the data set. It can ensure that the model is exposed to a variety of scenarios and conditions during training, so that the model can better predict and make decisions when faced with unknown data. Example images such as Figure 4 shown.

[0054] In addition, the aspect ratio reveals the geometric shape and structural characteristics of the ship. Figure 5 These are some sample images of different aspect ratios in SSSD. From the figure, we can see that the aspect ratios of ships vary greatly. In addition, there are a large number of examples with large aspect ratios in SSSD. The image size is 256*256 pixels, and the resolution is 0.6 meters / pixel. Figure 5 It can be clearly seen that the length-to-width ratios of different types of ships are often different, especially some frigates whose shapes are close to strip-shaped ground objects, which puts forward requirements for the attention space in operations such as convolution.

[0055] By analyzing the spatial distribution and aspect ratio characteristics of ships in the dataset, SSSD is diverse and representative in the composition of ship data, making the model trained on SSSD more adaptable to various complex and extreme situations, thus showing stronger robustness and wide adaptability in practical applications.

[0056] Step S102, construct a remote sensing image ship extraction model with joint dual-task and dual-attention, the model includes a dual-task network module, the dual-task network module includes a main task branch and an auxiliary task branch; the main task branch includes a coding residual module and a dual attention module, the input data is the remote sensing image feature map, and the output data is the ship pixel-level binary label; the auxiliary task branch input data comes from the coding residual module of the main task branch, and the output data is the ship contour label.

[0057] A ship extraction model for remote sensing images based on dual-task and dual-attention (e.g. Figure 6 The construction basis of the main task branch is the encoder-decoder network. The main task of the main task branch is ship extraction. The encoding residual module contains a 7×7 convolution, a global maximum pooling layer, a relu activation function and four encoding layers. The remote sensing image feature map of size 256×256×3 is sent to the 7×7 convolution for convolution operation to obtain a 128×128×64 feature map; the 128×128×64 feature map is subjected to global maximum pooling and activation operations to obtain a 128×128×64 feature map; the 128×128×64 feature map is then input into four encoding layers for four downsampling operations. After the above operations, the number of channels of the original remote sensing image is increased from 3 to 512, and the size is reduced to 8*8 pixels, so as to fully extract the semantic information of the image. The encoding residual module sets a total of 5 downsamplings, and the input and output features of each downsampling are shown in Table 2.

[0058] Table 2 Basic information of downsampling input and output characteristics

[0059]

[0060] The main task branch also contains four decoding layers, one 4×4 transposed convolution, one 3×3 convolution and Sigmoid activation function. Each decoding layer uses a jump connection to take the output of the corresponding encoding layer as part of its input (the other part comes from the previous encoding layer / decoding layer) to avoid excessive information loss; and a dual attention module is introduced on the jump connection branch; the output of the last encoding layer of the encoding residual module is sent to four decoding layers for four upsampling operations. The upsampled feature map is sequentially subjected to transposed convolution, convolution and activation operations to obtain a ship extraction result consistent with the original remote sensing image size. The structure of each decoding layer is as follows: Figure 7 shown.

[0061] In order to improve the refinement of the ship segmentation results, a contour supervision branch (auxiliary task branch) is introduced on the basis of the main task branch. In order to strengthen the supervision of the auxiliary task branch on the main task branch, the auxiliary task branch and the main task branch share the coding residual module. The auxiliary task branch contains three decoding layers, one 4×4 transposed convolution, one 3×3 convolution and Sigmoid activation function. Each decoding layer uses skip connection to take the output of the corresponding encoding layer as part of its input (the other part comes from the previous encoding layer / decoding layer). The output of the last encoding layer of the coding residual module is sent to the three decoding layers for upsampling. The upsampled feature map is sequentially subjected to transposed convolution, convolution and activation operations to obtain the ship contour label. This branch further uses auxiliary information such as the shape and edge of the ship to enhance the distinction between the ship and the background. In the decoding layer part, the output features of the first and second encoding layers are used as input, and multi-level skip connections and decoding operations are performed sequentially to generate a decoding feature map of size 64×64×64. In the prediction part, the feature map is upsampled to a 256×256×32 feature map through transposed convolution, and the contour prediction result is obtained using 3×3 convolution and Sigmoid activation function. The contour prediction result is consistent with the size of the original remote sensing image, and the true label of the ship contour is used as the supervision signal.

[0062] Dual attention module: In order to enhance the network's ability to focus on key points and extract features in multiple dimensions in deep information extraction, a dual attention mechanism is introduced, which not only considers the dependencies within the sequence and the correlations between different channels at the same time, but also solves the problem of incomplete semantic information extraction in a single attention mechanism model, which greatly promotes the improvement of ship segmentation accuracy. The effective combination of the two mechanisms is an effective fusion of two dimensions. In deep information extraction, the focus of the model can be effectively maintained. In this task, key features can be extracted more accurately, which improves the accuracy and robustness of the segmentation task. This embodiment uses two attention mechanisms to superimpose, and the superposition method is as follows: Figure 8 As shown. The input data is subjected to the multi-head self-attention mechanism (KAM) and the channel attention mechanism (CAM) respectively to obtain two output data, and then the two output data are added to output the final value. The following is a detailed introduction to the two attention mechanisms.

[0063] (1) Multi-head self-attention module

[0064] Optical images contain rich feature information. Traditional self-attention mechanisms can only focus on certain features at different locations. However, for optical ship images, the feature distribution of image data varies greatly, and the degree of correlation between different features is different. A single self-attention mechanism is not enough to fully utilize the feature information of optical images, and may even generate additional redundant information or noise, which is not conducive to the final extraction results. In summary, it is proposed to use a multi-head self-attention module to adapt to the feature information of optical images. The module structure is as follows: Fig. 9 As shown, based on the combination of multiple groups of self-attention components, it aims to help the input features better capture information of other dimensions and levels of themselves, establish appropriate feature expressions, and thus enhance the model's ability to capture local or global information.

[0065] The specific implementation process of the multi-head self-attention module is: for each attention head H i , first use linear transformation to get an independent query vector Q i , key vector K i Sum value vector V i , as shown in Formula 1. Then, each head will perform self-attention calculation independently. Get its own attention output, as shown in Formula 2. Finally, concatenate the outputs of all heads and perform linear transformation to get the final multi-head attention output, as shown in Formula 3.

[0066] Q i =W Qi XK i =W Ki XV i =W Vi X (1)

[0067]

[0068] MultiHead(Q,K,V)=Concat(Attention 1 , Attention 2 ,...,Attention h )W 0 (3)

[0069] (2) Channel Attention Module

[0070] Channel attention is an attention mechanism for feature map channels in deep learning models. Its network structure is shown in the following figure: Fig.10The core idea is to enable the model to dynamically focus on different parts of the input when processing input data, rather than treating all inputs equally. The weights of each channel are dynamically adjusted according to the correlation between the channels, so that the model can focus on the channels that are more important to the current task, while reducing the attention to irrelevant or redundant channels, thereby improving the network model's ability to capture ship features in remote sensing images.

[0071] Channel attention weight matrix From the input feature map Calculated from, where H×W represents the spatial size of the feature map and C represents the number of channels. Perform matrix transformation to obtain Then and its transposed matrix A T Multiply them, and finally pass them through the softmax function to generate the channel attention weight matrix S, as shown in Formula 4.

[0072]

[0073] Then the attention weight matrix S and The transformed matrix Perform matrix multiplication and transform to a size of The feature map is weighted using the proportional coefficient β and superimposed with the original feature map to enhance the feature map E j , the formula is shown in Formula 5.

[0074]

[0075] Step S103: The remote sensing image ship extraction model combining dual tasks and dual attention designs two branches, the main task and the auxiliary task. The total loss function is calculated using the ship segmentation label and the ship contour label. The specific formula is shown in Formula 6:

[0076] L total =L S +αL B (6)

[0077] L=L Bce +L Dice (7)

[0078] L Bce =-(ylog(p(x)+(1-y)log(1-p(x)) (8)

[0079]

[0080] Among them, L total , L S , LB Denote the total loss, main task loss, and auxiliary task loss, respectively, and α is a hyperparameter for balancing the two tasks. Considering that both the main task and the auxiliary task have the problem of category imbalance in the dataset (i.e., the foreground target pixels are far less than the background pixels), using only binary cross entropy loss (BCE Loss) may cause the model to be too biased towards background prediction. In order to alleviate this problem and improve the performance of the model when dealing with rare categories, the Dice loss is combined to optimize the model's accuracy in details and its ability to restore the overall shape. Therefore, both the main task loss function and the auxiliary task loss function adopt a composite loss function that combines binary cross entropy loss and Dice loss. The specific loss function is shown in formula (7), where L represents the main task or auxiliary task loss, and L Bce represents the binary cross entropy loss (the specific formula is shown in formula (8)), y is the true label, which is 0 or 1, p(x) is the value predicted by the model, and L Dice represents the Dice loss (the specific formula is shown in formula (9)), where |x∩y| represents the number of elements in the intersection of sets x and y, and |x| and |y| represent the number of elements in sets x and y respectively.

[0081] Corresponding to the above method, this embodiment also proposes a dual-task and dual-attention ship extraction system, which includes a data set production module and a model construction module, wherein:

[0082] The dataset production module is used to obtain remote sensing image data of typical port areas around the world based on public datasets, complete manual annotation, and obtain a ship semantic segmentation dataset containing ship pixel-level binary labels and ship contour labels.

[0083] The model building module is used to build a remote sensing image ship extraction model with joint dual-task and dual-attention. The model includes a dual-task network module, which includes a main task branch and an auxiliary task branch. The main task branch includes a coding residual module and a dual attention module. The input data is the remote sensing image feature map, and the output data is the pixel-level binary label of the ship. The input data of the auxiliary task branch comes from the coding residual module of the main task branch, and the output data is the ship contour label.

[0084] In order to verify the effectiveness of this solution, further explanation is given below in combination with experimental data.

[0085] (I) Experimental configuration

[0086] All experiments were configured the same way, specifically: processor—Intel(R)Xeon(R)Silver 4216@2.10GHZ, graphics card—Nvidia Tesla T4 (memory 16GB), operating system—Windows 10, and implemented based on Python programming language and PyTorch deep learning framework. During model training, the Adam optimizer was used to dynamically adjust network parameters. During training, the batch size of the dataset was set to 4, and training was performed for 100 epochs. In addition, to avoid overfitting, data augmentation techniques such as random horizontal flipping and vertical flipping were used on the input data.

[0087] ① Experimental Dataset

[0088] The training images and test images in SSSD are randomly divided in a ratio of 4:1, and finally 3760 training images and 923 test images (all containing pixel-level labels) are obtained, and the size of all images is 256*256 pixels. The specific image data distribution is shown in Table 3. The total number of images in the public dataset of HRSC2016 is 2643, and the total number of images in the self-made dataset is 2040.

[0089] Table 3. Distribution of SSSD image data

[0090]

[0091] ② Experimental accuracy evaluation index

[0092] Multiple accuracy evaluation indicators are used to comprehensively evaluate the performance of the method, including precision (Precision, P), recall (Recall, R), F1 score, overall classification accuracy (Overall Accuracy, OA), intersection over Union (Intersection over Union, IoU) and average training time (Mean Time). The calculation formula is as follows:

[0093]

[0094] Among them, TP represents the number of correctly classified ship pixels, TN represents the number of correctly classified non-ship pixels, FP represents the number of non-ship pixels classified as ship pixels, FN represents the number of ship pixels classified as non-ship pixels, time represents the total training time, and epoch represents the number of training iterations.

[0095] (II) Hyperparameter Analysis

[0096] In order to achieve the best performance of the model and clarify the proportion of auxiliary tasks to the main task, the hyperparameters of the auxiliary task loss function are experimentally analyzed, and four hyperparameter ratios of 0.5, 0.8, 1.0, and 1.5 are set. The experimental results are shown in Table 4. The experimental results show that when the hyperparameter ratio is 1.0, the R, F1, OA, and IoU indicators have the highest accuracy. The R indicator is 2.44%, 5.48%, and 0.21% higher than when the hyperparameter ratio is 0.5, 0.8, and 1.5, respectively. The F1 indicator is 1.55%, 2.26%, and 0.47% higher than when the hyperparameter ratio is 0.5, 0.8, and 1.5, respectively. The OA indicator is 0.32%, 0.40%, and 0.11% higher than when the hyperparameter ratio is 0.5, 0.8, and 1.5, respectively. The IoU indicator is 0.6%, 1.8%, 2.1%, 3.9%, 4.8%, and 5.9% higher than when the hyperparameter ratio is 0.6, 1.8, and 2.9% higher. When the hyperparameter ratio is 0.5, 0.8, and 1.5, it is 2.51%, 3.62%, and 0.77% higher, and the P index is 1.53% lower than when the hyperparameter ratio is 0.8, but its recall rate is 5.48% higher than when the hyperparameter ratio is 0.8, indicating that although the hyperparameter of 0.8 can improve the accuracy of the extraction results, it also reduces the proportion of real ships extracted. Compared with the hyperparameter ratio of 0.5 and 1.5, both the precision and recall rate are improved, proving that under this network architecture, the loss constraints of the two tasks adopt an equal proportional relationship to achieve the best results.

[0097] Table 4. Comparison results of loss function hyperparameters

[0098]

[0099] (III) Comparative experiment

[0100] In order to verify the effectiveness of this method for ship extraction, it is tested on SSSD and compared with typical and advanced network models in recent years. The qualitative extraction effect of the experiment is as follows: Fig.11 The quantitative experimental results are shown in Table 5.

[0101] Fig.11Some qualitative visualization results of different methods on the test data set are shown. The backgrounds of the selected 5 images include different water environments, different ship distribution locations and different ship categories, which basically cover the key and difficult areas of ship extraction. Image (a) shows two ships moored at the dock in different directions. The background only involves the ocean and land. The position of the ship is close to the edge of the image and the ship is incomplete. From the extraction results, U-Net, DeepLabV3+, and MANet have large-scale missed detections, and ASPPUNet has a small part of the stern of a ship. The method in this case can achieve accurate extraction of ships and oceans. Image (b) is a docked ship, and its background includes the dock and the ocean. The contrast between the ship and the ocean background is low, and the contrast with the dock background is high. From the extraction results, all the comparison methods have obvious false detections at the bow position, and the method in this case can obtain more accurate visualization results. Image (c) covers part of the aircraft carrier image. The target accounts for a large proportion of the entire image. U-Net, ASPPUNet, and DeepLabV3+ all have large-scale missed detections, while MANet and the proposed method have fewer falsely detected pixels. Image (d) covers two ships. The ship area accounts for a large proportion of the entire image. ASPPUNet has a large number of missed extractions, U-Net and DeepLabV3+ have slight missed extractions, and MANet and the proposed method can completely extract the two ships in the image. The land background of image (e) is more complex, and the ships are only distributed in the lower left corner of the image. From the extraction results, the comparison methods all have missed detections, among which ASPPUNet and DeepLabV3+ are particularly serious. The proposed method can effectively avoid missed detections, but there is still the phenomenon of over-extraction of ship contours.

[0102] Table 5 Comparison of ship extraction accuracy of different network models

[0103]

[0104] As shown in Table 5, the proposed method has achieved numerical advantages in P, R, F1, OA, and IoU indicators. The P indicator is improved by 11.10%, 11.78%, 6.65%, and 1.57% compared with U-Net, ASPPUNet, DeepLabV3+, and MANet, respectively. The R indicator is improved by 3.83%, 12.43%, 9.29%, and 0.49% compared with U-Net, ASPPUNet, DeepLabV3+, and MANet, respectively. The F1 indicator is improved by 7.48%, 12.13%, 8.05%, and 1.74% compared with U-Net, ASPPUNet, DeepLabV3+, and MANet, respectively. The OA indicator is improved by 1.87%, 2.70%, 1.74%, and 0.24% compared with U-Net, ASPPUNet, DeepLabV3+, and MANet, respectively. The IoU index is improved by 11.44%, 17.84%, 12.25% and 1.64% compared with U-Net, ASPPUNet, DeepLabV3+ and MANet, respectively. This shows that the proposed method can show better segmentation performance than typical and advanced network models for semantic segmentation, although the proposed method takes more average training time than ASPPUNet and U-Net. However, compared with MANet, which has the second best accuracy, the proposed method has achieved a lead of 46.18 seconds in training time, indicating that the proposed method can take into account both model performance and computational efficiency, and has stronger practicality and competitiveness in the task of ship extraction based on optical images.

[0105] according to Fig.11 The comparative analysis of visual levels and the quantitative analysis of the specific data in Table 5 prove that the proposed method is superior to the comparative method in both qualitative visualization results and quantitative accuracy evaluation indicators, and the overall accuracy achieves the best effect, which fully verifies the superiority and robustness of the proposed method in the ship extraction task.

[0106] (IV) Ablation experiment

[0107] To prove the effectiveness of the module proposed in this case, the following ablation experiments are conducted. The specific cases mainly include the following four cases, where case 1 is the basic network, case 2 adds dual attention on the basis of case 1, case 3 adds contour branch on the basis of case 1, and case 4 is the method of this case. The accuracy statistics of the ablation experiment are shown in Table 6.

[0108] Table 6 Ablation experiment accuracy statistics

[0109]

[0110]

[0111] The following conclusions can be drawn from Table 6: (1) Compared with case 1 and case 2, case 3 and case 4 have a certain degree of improvement in the four indicators of R, F1, OA, and IoU. This shows that the addition of the dual attention mechanism has a promoting effect on the accuracy of ship extraction. (2) Compared with case 1 and case 3, case 2 and case 4 have improved in the five indicators of P, R, F1, OA, and IoU. Case 4 lags behind slightly in the P and R indicators, but has an advantage in the other three accuracy indicators, indicating that adding contour auxiliary tasks can greatly promote the proportion of real ships being extracted and can meet the needs of practical applications. (3) Compared with case 1, case 2, and case 3, case 4 has improved in the three indicators of F1, OA, and IoU. The addition of the two modules separately helps to improve the accuracy of ship extraction, and after the collaborative addition, the accuracy of the ship extraction results in quantitative evaluation is further improved, proving that the two modules proposed in this case have a promoting effect on improving the effect of ship extraction.

[0112] (V) Large-scale scenario experiment

[0113] In order to test the large-scene ship extraction effect of the method in this case, an image of the Zhenhai port area with a size of 3184pixel×3448pixel and an image of the San Diego port area with a size of 6512pixel×6512pixel, which are from different periods and have a large background difference from the training samples, were collected, and compared with the two network models (DeepLabV3+ and MANet) that performed better in the above experiments.

[0114] In the experiment, the two images of the port area were first sliced ​​to obtain slice images with an overlap of 10% and a size of 256×256 pixels. The slice images were then placed in the network model pre-trained with the SSSD dataset to obtain the prediction results. The prediction results were then spliced ​​with 10% overlap to obtain the ship extraction results of the large scene. The quantitative accuracy of the extraction results of each method in the Zhenhai port area is shown in Table 7.

[0115] Table 7 Comparison of the extraction accuracy of Zhenhai Port large scene with different network models

[0116]

[0117] It can be seen that the P, F1, OA and IoU of DeepLabV3+ are far behind those of the other two models. The method in this case is 1.28% and 1.04% higher than MANet in the two comprehensive indicators of F1 and IoU, and 27.01% and 18.89% higher than DeepLabV3+.

[0118] The quantitative accuracy of the extraction results of each method in the San Diego Port area is shown in Table 8.

[0119] Table 8 Comparison of extraction accuracy of San Diego Port large scene by different network models

[0120]

[0121] It can be seen that the proposed method is superior to the other two models in terms of R, F1, OA, and IoU. It is 4.69%, 1.38%, 0.01%, and 1.71% higher than MANet, and 5.78%, 33.06%, 1.66%, and 33.05% higher than DeepLabV3+. The detection results of the two bases show that the proposed method can better meet the requirements of large-scale ship extraction. Fig.12 Some qualitative visualization results of different methods on the Zhenhai port area image are shown.

[0122] Fig.13 The qualitative visualization results of different methods on the images of the San Diego port area are shown. From the extraction results, DeepLabV3+ has more false detection pixels for non-ship targets and the segmentation effect is not good. MANet has roughly the same false detection situation as the method in this case, but the method in this case has an advantage in terms of detection completeness. This shows the superiority of the method in this case in extracting ships in a large scene range.

[0123] (VI) Cloud and fog experiment

[0124] In order to verify the stability of this method in complex weather conditions, this case conducted two sets of comparative experiments under foggy conditions. The two sets of experiments were trained using the same training set with foggy processing, and the test set was divided into two cases: no foggy processing and foggy processing. Consistent with the large-scale scene experiment, two networks with better comprehensive performance (DeepLabV3+ and MANet) were selected for comparative experiments, trained on SSSD after foggy processing, and then verified using the SSSD test set.

[0125] ①Test set cloud and fog comparison experiment

[0126] Fig.14The following shows some qualitative visualization results of different methods under the condition of no cloud and fog processing in the test set. The backgrounds of the selected 5 images include different water environments, different ship distribution locations and different ship categories, which basically cover the key and difficult areas of ship extraction. Image (a) covers part of the aircraft carrier image, the target accounts for a large proportion of the entire image, and there is almost no ocean in the background. DeepLabV3+ has some errors in extracting the background and ships, MANet misses most of the ship area, and only the method in this case extracts the ship targets in the image more completely and accurately. Image (b) shows the distribution of two ships along the two ends of the pier. It can be clearly seen that DeepLabV3+ extracts incomplete ships, while MANet almost does not extract ships, and only the method in this case has only a small part of the error extraction. Image (c) shows the situation where multiple ships are distributed side by side. DeepLabV3+ mistakenly extracts the pier as a ship, while MANet and the method in this case do not have this situation. For the missed extraction of ships, the method in this case is obviously better than the other two comparison methods. The main body of image (d) is the bow of the aircraft carrier, and the proportion of ocean background is higher than that of image (a). DeepLabV3+ and MANet both mistakenly extracted the dock as a ship, while the method in this case only had a small amount of error extraction at the edge of the ship. Image (e) has a large proportion of ocean background. All three methods can extract ships well, but the method in this case is more advantageous in avoiding wrong extraction of ships. The accuracy statistics of the experimental results under the condition of no cloud and fog in the test image are shown in Table 9. It can be seen that the method in this case is 6.3%, 18.05%, 6.54%, and 21.88% higher than DeepLabV3+ in R, F1, OA and IoU indicators, respectively, and 23.28%, 10.95%, 1.65%, and 14.03% higher than MANet. Only the P indicator is slightly lower than MANet by 0.28%, indicating that the stability of the method in this case is better than the other two comparison methods under cloud and fog conditions, proving that the method in this case can meet the conditions of use in complex weather.

[0127] Table 9 Comparison of extraction accuracy of different network model test sets under fog-free conditions

[0128]

[0129] according to Fig.14 The comparative analysis of the visual level and the specific data in Table 9 prove that the method in this case is superior to the comparison method in terms of qualitative visualization results and quantitative accuracy evaluation indicators under complex weather conditions, and the overall accuracy achieves the best effect, which fully verifies the stability and universality of the method in the extraction task of ship fog conditions.

[0130] ②Test set has cloud and fog comparison experiment

[0131] The accuracy statistics of the experimental results under the condition of foggy test images are shown in Table 10. It can be seen that the method in this case is 7.61%, 6.68%, 1.49%, and 9.78% higher than DeepLabV3+ in P, F1, OA, and IoU indicators, and 0.76%, 0.09%, 0.03%, and 0.14% higher than MANet. Only in the R indicator, it is slightly lower than MANet by 0.56%. The extraction accuracy of the method in the foggy test set is better than that of the comparison method, which further proves the generalization and adaptability of the method in complex weather conditions.

[0132] Table 10 Comparison of extraction accuracy of different network model test sets under fog conditions

[0133]

[0134] Fig.15 Some qualitative visualization results of different methods under foggy conditions are shown. The backgrounds of the five selected images include different water environments, different ship distribution locations, and different ship categories, which basically cover the key and difficult areas of ship extraction. Image (a) covers part of the aircraft carrier image. The target accounts for a large proportion of the entire image, and there is almost no ocean in the background. DeepLabV3+ has a large number of incorrectly extracted backgrounds and ships. MANet only has a small number of incorrectly extracted ships above the aircraft carrier. Only the method in this case extracts the ship part of the image more completely and accurately. Image (b) is a ship distributed along the pier. It can be clearly seen that DeepLabV3+ mistakenly extracts the pier part as a ship. MANet and the method in this case can extract the ship more completely, but the method in this case is obviously more advantageous in terms of the completeness of ship extraction. Image (c) shows a small ferry next to the ship as interference. DeepLabV3+ mistakenly extracted the small ferry, while MANet and the method in this case did not have this problem. However, for missed extraction of the ship, the method in this case is obviously better than the other two comparison methods. The main body of image (d) is the bow of the ship under repair. DeepLabV3+ and MANet both mistakenly extracted the dock as the ship, while the method in this case only had a small amount of wrong extraction at the edge of the ship. Image (e) is a ship under construction in a dry dock, with only some seawater in the lower right corner of the background. DeepLabV3+ has some missed extraction, MANet missed the entire ship, and the results of the method in this case are ideal in terms of both accuracy and completeness.

[0135] according to Fig.15 The comparative analysis of the visual hierarchy and the specific data in Table 10 proves that the proposed method has an advantage over the comparative method under complex weather conditions, and the overall accuracy achieves the best effect, which fully verifies the stability and universality of the proposed method in the task of ship extraction when fog appears in ship images.

[0136] (VII) Small sample experiment

[0137] In order to test the effect of this method on ship feature extraction under small sample conditions, this case conducted comparative experiments under the conditions of 500, 1000, and 2000 samples. In order to ensure that the experimental conditions are only related to the number of samples, the large number of training samples all contain the small number of training samples. The experimental results are as follows Fig.16 shown.

[0138] By analyzing the line statistics, we can draw the following conclusions: (1) As the number of samples increases, the accuracy evaluation indicators all show an upward trend. (2) The method in this case has an accuracy advantage over DeepLabV3+ and MANet in terms of R, F1, OA, and IoU under different sample numbers. (3) When the number of samples exceeds 2000, the method in this case improves slowly in the two comprehensive indicators of F1 and IoU and tends to be stable.

[0139] In summary, in terms of dataset construction, the present invention integrates the HRSC2016 dataset (public dataset) and the self-made dataset to construct a ship semantic segmentation dataset that is widely distributed around the world; the overall scene of this dataset is complex, the types of ships are diverse, and the spatial distribution is rich and representative, providing rich data support for model training. In terms of model architecture, a remote sensing image ship extraction model with combined dual tasks and dual attention is proposed. The model combines a multi-head self-attention module and a channel attention module to enhance the ability to capture ship features, and introduces a contour supervision auxiliary task to improve the detail cutting effect.

[0140] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0141] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0142] The units and method steps of each example described in conjunction with the embodiments disclosed in this case can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0143] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0144] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A ship extraction method based on dual tasks and dual attention, characterized in that: Include: Based on the public dataset, we obtain remote sensing image data of typical port areas around the world, complete manual annotation, and obtain a ship semantic segmentation dataset containing ship pixel-level binary labels and ship contour labels; A remote sensing image ship extraction model with joint dual-task and dual-attention is constructed. The model includes a dual-task network module, which includes a main task branch and an auxiliary task branch. The main task branch includes a coding residual module and a dual attention module. The input data is the remote sensing image feature map, and the output data is the pixel-level binary label of the ship. The input data of the auxiliary task branch comes from the coding residual module of the main task branch, and the output data is the ship contour label.

2. The ship extraction method based on dual tasks and dual attention according to claim 1 is characterized in that: The remote sensing image data is sliced ​​and the images are uniformly cropped into a size of 256×256 pixels. A 10% overlap is designed during the slicing process. The ship semantic segmentation dataset includes offshore ships, docked ships, and ships on land. The ship semantic segmentation dataset includes ships with different aspect ratios.

3. The ship extraction method based on dual tasks and dual attention according to claim 1 is characterized in that: The coding residual module includes a 7×7 convolution, a global maximum pooling layer, a relu activation function and four coding layers. The remote sensing image feature map is sent to the 7×7 convolution for convolution operation, and global maximum pooling and activation operations are performed; then the output feature map is input into four coding layers for four downsampling operations.

4. The ship extraction method based on dual tasks and dual attention according to claim 3 is characterized in that: The dual attention module includes a multi-head self-attention module and a channel attention module.

5. The ship extraction method based on dual tasks and dual attention according to claim 4 is characterized in that: The main task branch also includes four decoding layers, one 4×4 transposed convolution, one 3×3 convolution and a Sigmoid activation function. Each decoding layer uses a skip connection to take the output of the corresponding encoding layer as its input, and a dual attention module is introduced on the skip connection branch. The output of the last encoding layer of the encoding residual module is sent to four decoding layers for four upsampling operations. The upsampled feature map is sequentially subjected to transposed convolution, convolution and activation operations to obtain the pixel-level binary label of the ship.

6. The ship extraction method based on dual tasks and dual attention according to claim 3 is characterized in that: The auxiliary task branch includes three decoding layers, one 4×4 transposed convolution, one 3×3 convolution and a Sigmoid activation function. Each decoding layer uses a skip connection to take the output of the corresponding encoding layer as its input. The output of the last encoding layer of the encoding residual module is sent to the three decoding layers for three upsampling operations. The upsampled feature map is sequentially subjected to transposed convolution, convolution and activation operations to obtain the ship contour label.

7. The ship extraction method based on dual tasks and dual attention according to claim 1 is characterized in that: The total loss function of the remote sensing image ship extraction model with dual tasks and dual attention is defined as follows: Among them, L total , L S , L B They represent the total loss, main task loss and auxiliary task loss respectively, α is a hyperparameter, L represents the main task loss or auxiliary task loss, and L Bce represents the binary cross entropy loss, L Dice Denotes Dice loss.

8. A ship extraction system based on dual tasks and dual attention, characterized in that: Contains dataset making module and model building module, including: The dataset production module is used to obtain remote sensing image data of typical port areas around the world based on public datasets, complete manual annotation, and obtain a ship semantic segmentation dataset containing ship pixel-level binary labels and ship contour labels; The model building module is used to build a remote sensing image ship extraction model with joint dual-task and dual-attention. The model includes a dual-task network module, which includes a main task branch and an auxiliary task branch. The main task branch includes a coding residual module and a dual attention module. The input data is the remote sensing image feature map, and the output data is the pixel-level binary label of the ship. The input data of the auxiliary task branch comes from the coding residual module of the main task branch, and the output data is the ship contour label.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images