Ship instance segmentation algorithm based on global and local attention mechanisms

Abstract

The application discloses a ship instance segmentation algorithm based on global and local attention mechanisms, which comprises the following steps: constructing a ship instance segmentation dataset, data enhancement, constructing a SOLOv2 ship instance segmentation network using a global and local attention mechanism GALA, network training and testing, etc. Two new VL ship datasets are collected and manually labeled, and considering the particularity of marine ship instance segmentation, a GALA attention mechanism is proposed, which uses 1D strip pooling and 2D global pooling to save the global and local information of the input image, so as to retain the global position and semantic information of the ship, thereby improving the segmentation accuracy by 4.3% on average compared with existing attention mechanisms. The new open source dataset and the proposed GALA mechanism will promote the research of VL ship applications and attract attention in other computer vision fields to use the GALA mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of ship target detection, and particularly relates to a ship instance segmentation algorithm based on global and local attention mechanisms. BACKGROUND

[0002] Image segmentation plays a crucial role in many vision understanding and object detection systems. It involves a process that divides an image into independently connected regions using the intensity (brightness) or other information (e.g., edges) of the image. Image segmentation can be at least divided into two categories, namely semantic segmentation and instance segmentation. Semantic segmentation performs pixel-level labeling using a set of colors (object classes), and instance segmentation extends semantic segmentation by simultaneously detecting and delineating each object of interest in an image. Compared with object detection that detects the location of an object and places a window on it, instance segmentation is more like a combination of object detection and semantic segmentation, which not only finds the location of all specific objects in an image, but also finds the contour and classification of each detected object. For marine ship segmentation, semantic segmentation classifies all ships in an image into one category, labels all ships with the same color, and classifies islands into another category, while instance segmentation detects individual ships and classifies them into different categories. The application of instance segmentation has been successfully landed in scenes such as unmanned vehicles, human-computer interaction, biomedical, video surveillance, and marine ship monitoring.

[0003] Since marine ships are the carriers of marine-related activities such as marine scientific research and education, cross-sea transportation, and marine fisheries, image and video analysis of marine ships, including instance segmentation of marine ships, has been increasingly concerned in the past few years. Instance segmentation of marine ships can provide important information such as the relative position of a ship relative to other ships or surrounding obstacles, which is crucial for the safe travel of ships. Today, various imaging technologies such as radars, infrared cameras, etc. have been equipped on modern marine ships or intelligent unmanned ships. The analysis of the (relative) position of ships in these imaging data can provide accurate environmental information to help ships autonomously avoid collisions with other ships and natural obstacles in the ocean. In addition, by segmenting the ships in the image relative to the background that may contain spatial information (e.g., known islands or cities), we can obtain accurate information about the identity of the ship and the absolute position of the ship on the earth. Therefore, instance segmentation of marine ships from complex marine backgrounds is crucial for many marine-related activities.

[0004] Currently, satellites, Synthetic Aperture Radar (SAR), Infrared Imaging (IR), and cameras using Visible Light (VL) are the main imaging tools for recording marine vessels, resulting in several open-source databases for marine vessel segmentation. These databases include satellite remote sensing images, SAR images, IR images, and VL images. Satellite images usually have very large fields of view covering vast spaces, but their image resolution is low, which cannot provide accurate vessel information (e.g., shape and type). SAR images are not affected by weather conditions, but SAR images usually contain a large amount of scattering noise, and the spectral information is not rich, which is not convenient for subsequent vessel segmentation. IR has strong penetration and is not easily affected by environmental conditions, but the contrast and signal-to-noise ratio of the obtained IR images are usually not high enough, resulting in a lack of color and texture information of vessels. In contrast, visible light images have unique advantages such as high resolution, containing color and texture information, high signal-to-noise ratio, high contrast, and rich details compared to other types of images, so visible light images can be a complementary part of satellite, SAR, and IR images. With these advantages, VL images can provide clear vessel feature information (e.g., shape), which is crucial for subsequent vessel detection, segmentation, and classification. In addition, VL images can be easily obtained with our regular mobile phones and cameras at very low cost, so they are suitable for building applications that require large-scale datasets. However, so far, there are limited open-source databases of VL marine vessel images, and there is a lack of suitable open-source visible light image vessel datasets, which can slow down the development of unmanned vessels. Although Zhang et al. and Sun et al. proposed two VL image datasets of marine vessels, respectively. These datasets are not publicly available, the size of the dataset is relatively small, and the labeling of the dataset in terms of vessel categories is not fine enough.

[0005] Instance segmentation of ships in VL images is a challenging image processing task. Overall, existing segmentation methods can be mainly divided into threshold-based, edge-based, region-based, superpixel-based, correlation-based and deep learning-based segmentation. Deep learning methods have recently received more and more attention. One challenge of instance segmentation based on deep learning is to obtain the location and semantic mask of each instance. Mask R-CNN implements a general framework that can effectively detect objects in images while generating high-quality segmentation masks for each instance. It extends the existing branch of the Faster R-CNN model bounding box recognition by adding a branch to predict object masks. Since two-stage instance segmentation methods are accurate but slow, one-stage instance segmentation methods have been proposed to improve segmentation efficiency. BlendMask is proposed to improve mask prediction by effectively combining instance-level information and semantic information with lower-level fine-grained information. PolarMask is proposed to formulate the instance segmentation problem as predicting instance contours through instance center classification and dense distance regression in polar coordinates, providing a new method for designing mask contours. CenterMask is a single-stage anchor-free instance segmentation method that designs a new spatial attention-guided mask branch. Unlike the above methods that rely on accurate edge detection, models such as SOLO and SOLOv2 directly segment instance masks and learn instance mask labels, thus achieving end-to-end optimization. SOLO series algorithms have been proven to outperform two-stage algorithms and one-stage algorithms. Instance segmentation of marine ships in VL images has also gradually attracted interest in the field. Zhang et al. proposed an integrated ship segmentation method based on discriminators and extractors to reduce interference factors from complex marine backgrounds. To preserve the global information of ships, Sun et al. proposed a method using precise RoI pooling and global mask heads, which improved the performance of ship instance segmentation. Therefore, this indicates that using global and complete information of ship appearance can improve the performance of instance segmentation. However, due to the extensive modification of the network architecture, this method is too complex, and there is still room for improvement in preserving the global and local information of ships, while it is not easy for existing deep learning models to effectively use global and local information.

[0006] Human attention mechanism can be a potential solution to preserve global and local information. In complex scenes, salient features and regions are easy to attract human attention. Inspired by this observation, attention mechanism is introduced into computer vision. Using attention mechanism in instance segmentation can guide segmentation to the most important regions in the image and ignore irrelevant parts. Attention mechanism amplifies the role played by key feature maps by assigning them greater weights. It is also worth noting that attention mechanism is a plug-and-play module that can be effectively included in many deep learning models, thus achieving great success in image classification, object detection, semantic segmentation, instance segmentation and 3D vision. In current deep neural networks, such as SENet and CBAM, attention mechanism is mainly used to convert 2D feature maps into single pixel feature maps by two-dimensional (2D) global pooling for feature map weight recalibration. These models using 2D global pooling mainly emphasize global information while ignoring local information. In contrast, one-dimensional strip pooling can accurately preserve local information along the spatial direction. Therefore, the combination of 2D global pooling and 1D strip pooling is a promising approach that can preserve both global and local information of the image. Therefore, the present application proposes a ship instance segmentation algorithm based on global and local attention mechanism. In this work, in order to meet the urgent need for open source ship VL database, we collect and label two ship VL datasets, and test the segmentation performance of multiple existing deep learning models on our VL datasets. In addition, we further propose a global and local attention mechanism (GALA) to re-extract global and local feature information by combining 2D global pooling and 1D strip pooling, thereby improving the performance of current instance segmentation models. The dataset and the proposed method are open source with this research. SUMMARY

[0007] The technical problem solved by the present application is to provide a ship instance segmentation algorithm based on global and local attention mechanism in view of the deficiencies of the prior art.

[0008] The technical scheme adopted by the present application to solve the above technical problem is: a ship instance segmentation algorithm based on global and local attention mechanism, comprising the following steps:

[0009] S1, constructing a ship instance segmentation dataset

[0010] A script for collecting image data from the Google Image platform is developed, using "ship" as a search keyword to search for marine vessel images, manually selecting real marine vessel images and segmenting and labeling the selected marine vessel images using LabelMe software, during the labeling process, first tracking and labeling the shape of the ship in the image without interruption, thereby drawing a polygon frame of different ships, then naming and classifying the labeled marine vessel images, after using the polygon annotation to depict all the ships in the image, generating an image annotation file in json format, obtaining a dataset, finally naming and annotating the dataset using the naming and annotation method consistent with the MS-COCO dataset, obtaining a ship instance segmentation dataset, named as MariBoats dataset, the MariBoats dataset contains all the labeled marine vessel images and ship segmentation annotations, all the labeled marine vessel images are named as self_boats dataset and assigned as "ship" category, and coco_boats dataset is incorporated into MariBoats dataset, the coco_boats dataset is extracted from the known MS-COCO dataset containing ship VL images;

[0011] The images and segmentation annotations of six types of ships in the MariBoats dataset are used as the dataset to construct the MariBoatsSubclass dataset, wherein the six types of ships are "Engineering Ship", "Cargo Ship", "Speedboat", "Passenger Ship", "Official Ship" and "Unknown Ship";

[0012] S2, data augmentation

[0013] At least one of horizontal flipping, scaling and multi-scale input is used to expand the MariBoats dataset and the MariBoatsSubclas dataset to obtain the processed ship instance segmentation dataset, wherein the specific operations of horizontal flipping, scaling and multi-scale input are as follows:

[0014] Horizontal flipping: the ship instance segmentation dataset is expanded by horizontal flipping, that is, the images in the ship instance segmentation dataset are flipped 180 degrees from left to right with a probability of 50%, to obtain the horizontal flipped image data and incorporate it into the corresponding ship instance segmentation dataset;

[0015] Scaling: the images in the ship instance segmentation dataset are scaled to obtain compressed image data and incorporate it into the corresponding ship instance segmentation dataset;

[0016] Multi-scale input: on the basis of scaling, multiple scaling scales are preset, and a scaling scale is randomly selected from the multiple preset scaling scales for input, obtaining multi-scale image data and incorporating into the corresponding ship instance segmentation dataset;

[0017] S3, constructing a SOLOv2 ship instance segmentation network using a global and local attention mechanism GALA

[0018] Constructing a SOLOv2 ship instance segmentation network using a global and local attention mechanism GALA, the ship instance segmentation network comprising a backbone network, a neck layer and a prediction head, and using the SOLOv2 ship instance segmentation network as a SOLOv2 ship instance segmentation model, wherein:

[0019] The backbone network refers to a convolutional neural network used to extract features at different stages, i.e. a ResNet-50 network;

[0020] The prediction head is used to predict the category and location of the ship target;

[0021] The neck layer is located between the backbone network and the prediction head, and the neck layer uses an enhanced feature pyramid network and adds a network layer for collecting feature maps at different stages;

[0022] S4, network training and testing

[0023] Taking 70% of the data in the MariBoatsSubclass dataset as the training set and 30% of the data as the test set, inputting the processed ship instance segmentation dataset obtained in step S2 into the SOLOv2 ship instance segmentation model constructed in step S3 for training, and in the initial stage of training, the backbone network uses transfer parameter learning to obtain a trained network; verifying the test set according to the trained network, and outputting the ship instance segmentation result.

[0024] As preferred, the backbone network comprises five feature extraction sections for extracting five different stage features in sequence, namely stage 0, stage 1, stage 2, stage 3 and stage 4, wherein: stage 0 comprises a convolution layer 0 and a corresponding BN batch normalization unit 0 and a ReLu rectified linear unit 0, and a pooling layer 0, wherein the convolution kernel of the convolution layer 0 has a size of 7x7 and a step of 2, and the pooling layer 0 adopts maximum pooling with a pooling kernel size of 3 and a step of 2; stage 1 comprises three residual blocks, each of which comprises a 1x1 convolution layer 1 and a corresponding BN batch normalization unit 1 and a ReLu rectified linear unit 1, a 3x3 convolution layer 2 and a corresponding BN batch normalization unit 2 and a ReLu rectified linear unit 2, and a 1x1 convolution layer 3 and a corresponding BN batch normalization unit 3 and a ReLu rectified linear unit 3, wherein the convolution layer 1, the convolution layer 2 and the convolution layer 3 are connected in sequence, and the input of each residual block is added to the output of the convolution layer 3; stage 2 comprises four residual blocks, stage 3 comprises six residual blocks, and stage 4 comprises three residual blocks, and each residual block contained in stage 2, stage 3 and stage 4 has the same structure as each residual block contained in stage 1;

[0025] The neck layer comprises a 4-layer network layer, which is respectively constructed by the last layer feature maps of stage 1, stage 2, stage 3 and stage 4 of the backbone network, and the construction method is as follows: for each feature map selected from the last layer feature maps of stage 1, stage 2, stage 3 and stage 4, a 1x1 convolution layer is respectively applied for channel scaling, thereby P1, P2, P3 and P4 layers are constructed, since the feature maps between two adjacent feature extraction sections are scaled by 2 times, the width and height scales of P1, P2, P3 and P4 layers are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original graph respectively; firstly, the feature maps of P4 layer are up-sampled by 2 times through a bilinear interpolation method, and the same width and height scales as the feature maps of P3 layer are maintained, and the feature maps of P4 layer and the feature maps of P3 layer are added; then the feature maps of P3 layer are up-sampled by 2 times through a bilinear interpolation method, and the same width and height scales as the feature maps of P2 layer are maintained, and the feature maps of P3 layer and the feature maps of P2 layer are added; then, firstly, the feature maps of P2 layer are up-sampled by 2 times through a bilinear interpolation method, and the same width and height scales as the feature maps of P1 layer are maintained, and the feature maps of P2 layer and the feature maps of P1 layer are added; for P1, P2, P3 and P4 layers, a 3x3 convolution layer is respectively applied, and a SOLOv2 ship instance segmentation model is connected, which comprises a global attention mechanism model and a local attention mechanism model, the global attention mechanism model comprises an entire space global average pooling layer, a full connection layer, a ReLu modified linear unit and a sigmoid activation layer in sequence, the global attention mechanism model performs point-by-point multiplication operation on the input feature maps and the output results, the local attention mechanism model comprises an X-direction global average pooling layer, a Y-direction global average pooling layer, a concatenate operation layer, a 1x1 convolution layer, a BN batch normalization unit, a nonlinear layer, a 1x1 convolution layer, a split operation layer, a 1x1 convolution layer and a sigmoid activation function layer, the local attention mechanism model performs point-by-point multiplication operation on the input feature maps and the output results;

[0026] The prediction head comprises a plurality of 3x3 convolution layers and corresponding GN batch normalization units and ReLu modified linear units, and a bilinear up-sampling layer.

[0027] Preferably, the construction process of the global and local attention mechanism GALA is as follows:

[0028] 1) The dimension of the entire 2D space information in the feature map is reduced through global average pooling, that is, the average of all pixel values of each channel map is obtained, thereby a new 1x1 channel feature map is obtained, and the output of the cth channel feature map is represented as:

[0029]

[0030] wherein x c is the input image of the cth channel, H is the height of the input image, and W is the width of the input image; z c is the output image of the cth channel;

[0031] After the activation function and the convolutional transformation processing, the channel correlation feature map is obtained, and the output image of the cth channel is represented as:

[0032]

[0033] wherein represents the input image of the cth channel that has already had channel correlation; σ represents a sigmoid activation function; T1 and T2 are nonlinear transformations for describing the importance of each channel; and δ represents a ReLu activation function;

[0034] 2) One-dimensional pooling is performed on the channel correlation feature map in the horizontal and vertical directions, respectively, and the output image of the cth channel with a height of h is represented as:

[0035]

[0036] the output image of the cth channel with a width of w is represented as:

[0037]

[0038] After the activation function and the convolutional transformation processing, the final channel correlation feature map with direction perception and position sensitivity is obtained, and the output image of the cth channel is represented as:

[0039]

[0040] wherein F h and F w represent two 1x1 convolutional transformations;

[0041] 3) Finally, the loss function loss is calculated, and the loss function is defined as:

[0042] L = Lcate + λLmas,

[0043] wherein Lcate is a regular Focal Loss for semantic classification, Lmask is a Dice Loss for mask prediction, and λ is a manually set parameter.

[0044] Compared with the prior art, the present application has the following advantages:

[0045] Marine vessel instance segmentation of VL images plays an important role in marine-related scientific research, education and commercial applications. However, there are almost no publicly available and suitable datasets containing vessel VL images for the purpose of vessel instance segmentation. To address this issue, we collect and manually label two new VL vessel datasets. Meanwhile, considering the particularity of marine vessel instance segmentation, we propose a GALA attention mechanism that utilizes 1D strip pooling and 2D global pooling to preserve the global and local information of the input image to retain the global position and semantic information of the vessel, thereby increasing the segmentation accuracy by an average of 4.3% compared to existing attention mechanisms. The new open-source dataset and the proposed GALA mechanism established by the invention will promote the research of VL vessel applications and attract attention in other computer vision fields to use the GALA mechanism. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 Distribution of the number of individual classes in the MS COCO dataset;

[0047] Figure 2 Distribution of image and object size for the "vessel" class in the MS COCO dataset;

[0048] Figure 3 Distribution of image size and vessel size for self_boats;

[0049] Figure 4 Scatter plot of image size and vessel scale distribution for MariBoats combined from self_boats and coco_boats;

[0050] Figure 5 Histogram of the number of vessels in each class in the MariBoatsSubclass dataset and depiction of representative images of each class in the MariBoats subclass;

[0051] Figure 6 Rough network structure diagram of the SOLOv2 vessel instance segmentation network using the global and local attention mechanism GALA;

[0052] Figure 7 Overall network structure diagram of the SOLOv2 vessel instance segmentation network using the global and local attention mechanism GALA;

[0053] Figure 8 Establishment process diagram of the GALA attention mechanism proposed by the invention;

[0054] Figure 9 Description diagram of the GALA attention mechanism proposed by the invention;

[0055] Figure 10Figures showing representative examples of existing Mask R-CNN, SOLO and SOLOv2 incomplete segmentation for several typical boats selected from MariBoatsSubclass.

[0056] Figure 11 Figures showing segmentation results comparison for several typical boat examples selected from MariBoatsSubclass. DETAILED DESCRIPTION

[0057] The application will be further described with reference to the drawings in which:

[0058] 1. Analysis of existing ship related databases

[0059] To investigate whether existing ship databases contain sufficient VL images that can be used for ship instance segmentation, we first explored the open-source datasets reported in previous studies, which cover several image types (as shown in Table 1). These datasets containing VL images of sea ships include the VL dataset Sea Ships, the IR dataset Distant sea ship, the SAR datasets SAR-Ship-Dataset and SSDD, the SAR dataset HRSID, and the MS COCO dataset. Sea Ships contains 31455 VL images covering six common ship types, mainly for object detection tasks. The Distant sea ship dataset consists of only 3132 long-wave infrared type images. The SAR-Ship-Dataset and SSDD datasets consist of only high-resolution SAR images. In addition, these datasets are mainly for object detection purposes, with object windows marking ships. They do not have the annotations required for instance segmentation purposes, which require marking the location, shape, and class of individual ships. HRSID can be used for ship instance segmentation as it contains the required annotations for instance segmentation, but it is of SAR type. The MS COCO dataset is a large open-source dataset commonly used for instance segmentation. To investigate whether this dataset contains specific ship VL images that can be used for instance segmentation, we performed a detailed analysis of the names, which can indicate all images in the MS COCO dataset. Figure 1 The number of individual classes in the MS COCO dataset is shown, indicating that the distribution of the number of individual classes is uneven. The “person” class accounts for 54% of the total number of images, while only about 2% of the images belong to the “boat” class.

[0060] We then developed an image extraction script to extract VL images of sea ships from the MS COCO dataset, which we named coco_boats. Since image size and object scale in images are two important factors that affect the performance of algorithms, we also analyzed these two factors for images in coco_boats Figure 2). We found that the length and width of the images cluster at 500 and 650 pixels (bottom left region, Figure 2 ), are uniformly distributed along a line, while the length and width of the boats are in [50, 650] and [50, 450] respectively (top right region, Figure 2 ). This analysis shows that the image sizes and the boat scales are not diverse enough. Overall, IR images and SAR images dominate the open-source marine vessel dataset, while there is currently a lack of open-source VL datasets for marine vessels that can be used for instance segmentation, from which we extracted the coco_boats dataset to facilitate the research of marine vessel instance segmentation in MS COCO.

[0061] Table 1. Overview of existing open-source marine vessel datasets

[0062]

[0063] 2、The ship instance segmentation dataset constructed by the present application

[0064] One reason for the lack of open-source VL datasets for marine vessel instance segmentation can be the difficulty of data collection and the relatively time-consuming and labor-intensive nature of labeling segmented data. To overcome the problem of dataset shortage, we developed a script for collecting image data from the Google Image platform, using "ship" as a search keyword to search for marine vessel images, manually selecting real marine vessel images and segmenting and labeling the selected marine vessel images using LabelMe software, during the labeling process, first tracking and labeling the shape of the ship in the image without interruption, thereby drawing a polygon mask of different ships, then naming and classifying the labeled marine vessel images, after using the polygon annotation to depict all the ships in the image, using json format to generate the image annotation file, we labeled a total of 6.2k images, which took about 400 hours, obtained the dataset, finally, we named and annotated the dataset using the same naming and annotation method as the MS-COCO dataset, obtained the marine vessel instance segmentation dataset, named MariBoats dataset, the MariBoats dataset contains all the labeled marine vessel images and ship segmentation annotations, all the labeled marine vessel images are named as self_boats dataset and assigned as "ship" category, while coco_boats dataset is incorporated into MariBoats dataset, the coco_boats dataset is extracted from the known MS-COCO dataset containing ship VL images. The final MariBoats dataset consists of 6,271 ship images and 15,777 ship segmentation annotations (i.e. labels), with only one category: "ship". The image part in this dataset is extracted from 13717 ship images searched on "Google Image" using keywords such as cargo ship, fishing boat, etc. We excluded those with low quality, blurred, irrelevant to ships and those with repetitive content.

[0065] To distinguish the images collected from Google Image from coco_boats, we named the former as "self_boats" dataset. The image size and ship size distribution of self_boats are shown in Figure 3 . Compared with coco_boats( Figure 2 ), the image size distribution of self_boats dataset (lower left corner area, Figure 3 ) is in the length of [50, 800] and the width of [50, 750], and the ship size (upper right corner area, Figure 3 ) is distributed in the length of [50, 550] and the width of [50, 500]. The image size and ship scale distribution scatter plot of MariBoats composed of self_boats and coco_boats Figure 4), we see that the image sizes and ship scales of the MariBoats dataset are more diverse than coco_boats.

[0066] To better distinguish the labeled ships, we construct the MariBoatsSubclass dataset with the images and segmentation annotations of the six ship classes in the MariBoats dataset, containing 3125 images and 4588 labels, where the six ship classes are: “Engineering Ship”, “Cargo Ship”, “Speedboat”, “Passenger Ship”, “Official Ship”, “Unknown Ship”. The number of labels in the MariBoatsSubclass dataset is higher than that in the VL image because a single image may contain multiple ships of different types. Figure 5 (a) shows the histogram of the number of ships in each category, where the “speedboat” category has the most images and labels (623 and 892, respectively). The “unknown ship” category has the fewest images (469). Overall, the distribution of each category and the accompanying segmentation annotations are relatively balanced, and the representation of each category is as follows Figure 5 (b) shows.

[0067] 3、The ship instance segmentation algorithm based on global and local attention mechanisms

[0068] The ship instance segmentation algorithm based on global and local attention mechanisms comprises the following steps:

[0069] S1, constructing a ship instance segmentation dataset

[0070] For details, see the content of the aforementioned Part 2.

[0071] S2, data augmentation

[0072] At least one of the following data augmentation methods is used to expand the MariBoats dataset and the MariBoatsSubclas dataset to obtain the processed ship instance segmentation dataset: horizontal flipping, scaling, and multi-scale input.

[0073] Horizontal flipping: the ship instance segmentation dataset is expanded by horizontal flipping, that is, a horizontal flipping operation of 180 degrees from left to right is performed on the images in the ship instance segmentation dataset with a probability of 50%, to obtain the horizontally flipped image data and incorporate it into the corresponding ship instance segmentation dataset.

[0074] scaling: scaling the images in the ship instance segmentation dataset to obtain compressed image data and incorporate into the corresponding ship instance segmentation dataset;

[0075] multi-scale input: on the basis of scaling, a plurality of scaling scales are preset, and a scaling scale is randomly selected from the plurality of preset scaling scales for input, to obtain multi-scale image data and incorporate into the corresponding ship instance segmentation dataset.

[0076] S3, constructing a SOLOv2 ship instance segmentation network using a global and local attention mechanism GALA

[0077] constructing a SOLOv2 ship instance segmentation network using a global and local attention mechanism GALA, the ship instance segmentation network comprising a backbone network, a neck layer and a prediction head, and using the SOLOv2 ship instance segmentation network as a SOLOv2 ship instance segmentation model, wherein:

[0078] the backbone network refers to a convolutional neural network for extracting features at different stages, i.e. a ResNet-50 network;

[0079] the prediction head is used to predict the category and position of the ship target;

[0080] the neck layer is located between the backbone network and the prediction head, and the neck layer uses an enhanced feature pyramid network and adds a network layer for collecting feature maps at different stages.

[0081] The backbone network comprises 5 feature extraction sections for extracting features at 5 different stages, i.e. stage 0, stage 1, stage 2, stage 3 and stage 4, which are connected in sequence, wherein: stage 0 comprises a convolution layer 0 and corresponding BN batch normalization unit 0 and ReLu rectified linear unit 0, and a pooling layer 0, wherein the convolution kernel of the convolution layer 0 has a size of 7x7 and a step of 2, and the pooling layer 0 uses maximum pooling with a pooling kernel size of 3 and a step of 2; stage 1 comprises 3 residual blocks, each of which comprises a 1x1 convolution layer 1 and corresponding BN batch normalization unit 1 and ReLu rectified linear unit 1, a 3x3 convolution layer 2 and corresponding BN batch normalization unit 2 and ReLu rectified linear unit 2, and a 1x1 convolution layer 3 and corresponding BN batch normalization unit 3 and ReLu rectified linear unit 3, wherein the convolution layer 1, the convolution layer 2 and the convolution layer 3 are connected in sequence, and the input of each residual block is added to the output of the convolution layer 3; stage 2 comprises 4 residual blocks, stage 3 comprises 6 residual blocks, and stage 4 comprises 3 residual blocks, and each residual block in stage 2, stage 3 and stage 4 has the same structure as each residual block in stage 1;

[0082] The neck layer comprises a 4-layer network layer, which is constructed by the last layer feature maps of stage 1, stage 2, stage 3 and stage 4 of the backbone network respectively (denoted as C1, C2, C3 and C4 respectively). Figure 6 The construction method is as follows: for each feature map selected from the last layer feature maps of stage 1, stage 2, stage 3 and stage 4, a 1x1 convolution layer is applied respectively for channel scaling, and P1, P2, P3 and P4 layers are constructed. Since the feature maps between adjacent two feature extraction sections have a 2-fold scale reduction, the width and height scales of P1, P2, P3 and P4 layers are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image respectively; first, the feature maps of P4 layer are 2-fold up-sampled by bilinear interpolation, and the same width and height scales as the feature maps of P3 layer are maintained, and the feature maps of P4 layer and the feature maps of P3 layer are added; then the feature maps of P3 layer are 2-fold up-sampled by bilinear interpolation, and the same width and height scales as the feature maps of P2 layer are maintained, and the feature maps of P3 layer and the feature maps of P2 layer are added; then first, the feature maps of P2 layer are 2-fold up-sampled by bilinear interpolation, and the same width and height scales as the feature maps of P1 layer are maintained, and the feature maps of P2 layer and the feature maps of P1 layer are added; for P1, P2, P3 and P4 layers, a 3x3 convolution layer is applied respectively, and a SOLOv2 ship instance segmentation model is connected, which comprises a global attention mechanism model and a local attention mechanism model. The global attention mechanism model comprises an entire space global average pooling layer, a full connection layer, a ReLu rectified linear unit and a sigmoid activation layer in sequence, and performs point-by-point multiplication operation on the input feature maps and the output results. The local attention mechanism model comprises an X-direction global average pooling layer, a Y-direction global average pooling layer, a concatenate operation layer, a 1x1 convolution layer, a BN batch normalization unit, a nonlinear layer, a 1x1 convolution layer, a split operation layer, a 1x1 convolution layer and a sigmoid activation function layer, and performs point-by-point multiplication operation on the input feature maps and the output results.

[0083] The prediction head comprises a plurality of 3x3 convolution layers and corresponding GN batch normalization units and ReLu rectified linear units, and a bilinear up-sampling layer. In this embodiment, the different scales of ship targets predicted by the prediction head are denoted as N1, N2, N3 and N4 respectively.

[0084] The network structure diagram of the constructed SOLOv2 ship instance segmentation network using global and local attention mechanism GALA is shown in Figure 6 and Figure 7 .

[0085] S4, network training and testing

[0086] The 70% data in the MariBoatsSubclass dataset is taken as the training set, and the 30% data is taken as the test set. The processed ship instance segmentation dataset obtained in step S2 is input into the SOLOv2 ship instance segmentation model constructed in step S3 for training. In the initial stage of training, the backbone network adopts the transfer parameter learning to obtain the trained network. The test set is verified according to the trained network, and the ship instance segmentation result is output.

[0087] Specifically, as shown in Figure 8 and Figure 9 , the establishment process of the global and local attention mechanism GALA is as follows:

[0088] 1) Reduce the dimension of the entire 2D space information in the feature map through global average pooling, that is, average all pixel values of each channel map, obtain a new 1×1 channel feature map, and the output of the cth channel feature map is expressed as:

[0089]

[0090] where x c is the input image of the cth channel, H is the height of the input image, and W is the width of the input image; z c is the output image of the cth channel;

[0091] After activation function and convolution transformation processing, the channel correlation feature map is obtained, and the output image of the cth channel is expressed as:

[0092]

[0093] where represents the input image of the cth channel with channel correlation; σ represents the sigmoid activation function; T1 and T2 are nonlinear transformations describing the importance of each channel; δ represents the ReLu activation function;

[0094] 2) One-dimensional pooling is performed on the feature map with channel correlation in the horizontal and vertical directions respectively, and the output image of the cth channel with height h is expressed as:

[0095]

[0096] The output image of the cth channel with width w is expressed as:

[0097]

[0098] After the activation function and convolutional transformation processing, the final feature map with channel correlation, direction perception and position sensitivity is obtained, and the output image of the cth channel is represented as:

[0099]

[0100] where F h and F w represent two 1x1 convolutional transformations;

[0101] 3) Finally, the loss function loss is calculated, and the loss function is defined as:

[0102] L = Lcate + λLmas,

[0103] where Lcate is the conventional Focal Loss of semantic classification, Lmask is the Dice Loss of mask prediction, and λ is a manually set parameter.

[0104] 4. Performance test

[0105] 4.1. Performance evaluation index

[0106] We follow the quantitative measurement system used by the MS COCO dataset to evaluate the performance of the ship instance segmentation model. The six indicators used are the intersection over union (IoU), average precision (AP), mean average precision (mAP), frames per second (FPS), parameters (Para.) and time complexity (TC). The IoU is defined as the degree of overlap between two segments. AP is the main indicator for accuracy determination. AP calculates the average IoU, which is averaged from 0.5 to 0.95 with an increment of 0.05. For example, AP 50 and AP 75 represent the calculation of the average IoU at thresholds 0.5 and 0.75, respectively. For multi-scale object detection capability, AP S , AP M , and AP L are used to represent small (area < 32 2 pixels), medium (32 2 < area < 64 2 pixels), and large (area > 64 2The average precision (mAP) is the average of the AP over the number of ship classes to be detected. In addition to evaluating accuracy, it is also important to compare the runtimes of the instance segmentation models tested. FPS is a measure of how many images a model can process in one second. Parameters are the number of parameters in a neural network model (parameters learned when training the network). In general, the number of parameters is directly proportional to the memory required to save the model and the hardware memory requirements. The metric memory access cost (MAC) is often used to measure the time complexity (TC) of a model or algorithm.

[0107] 4.2, Performance test

[0108] 4.2.1, Segmentation results of MariBoats with one ship class

[0109] We evaluate the performance of existing models (i.e., Mask R-CNN, SOLO, and SOLOv2) on our constructed ship instance segmentation dataset. We first evaluate the three models on coco_boats, and we find that the segmentation of images in coco_boats by the three models is unsatisfactory (AP: 9.4, 13.4, 13.6), which is mainly due to the fact that these models have not been trained on enough VL ship images. In fact, the poor performance of these models on coco_boats is the initial motivation for us to construct a larger dataset containing enough sea ship VL images, and we hope that retraining the models on this larger dataset MariBoats can improve the performance of these models. Then, we retrain the three models using 70% of MariBoats and test them using the remaining 30%. The results of the quantitative test are shown in Tables 2-4, which directly show the performance of the models on MariBoats and its two subsets coco_boats and self_boats (the last row in Table 2-4).

[0110] At the same time, we further test the segmentation accuracy of the three models by retraining the models using coco_boats or self_boats only and testing using the other two datasets (1st-2nd rows in Tables 2-4). Taking the existing SOLOv2 as an example (Table 4), the segmentation accuracy of coco_boats shows that the model trained on the MariBoats training set improves 2.4% compared to the result trained on the coco_boats training set. The result of self_boats shows that the model trained on the MariBoats training set improves 5.7% compared to the result trained on the self_boats training set. The result of MariBoats shows that the model trained on the MariBoats training set improves 7.9% compared to the result trained on the self_boats training set. Tables 2 and 3 further verify the advantage of the MariBoats dataset containing richer image data in improving the segmentation performance of Mask R-CNN and SOLO, respectively. These fully verify the necessity of constructing the self_boats dataset and the MariBoats dataset.

[0111] Table 2. Test results of Mask R-CNN in three instance segmentation datasets

[0112]

[0113] Table 3. Test results of SOLO in three instance segmentation datasets

[0114]

[0115] Table 4. Test results of SOLOv2 in three instance segmentation datasets

[0116]

[0117] 4.2.2. Segmentation results of MariBoatsSubclass 6 ship class

[0118] After solving the instance segmentation problem with six ship classes, we continued to test the segmentation performance and computational speed of the existing Mask R-CNN, SOLO, and SOLOv2 on MariBoatsSubclass. The experimental results are shown in Table 5. We first set the backbone network ResNet to 50 layers commonly used by Mask R-CNN, SOLO, and SOLOv2 (1-3 rows in Table 5), and the existing SOLOv2 scored the highest mAP (57.8%), while Mask R-CNN and SOLO were 42.9% and 55.5%, respectively. Compared with other classes, the segmentation accuracy of these models for the Unknown ship class is usually much lower, and we infer that the smaller size (fewer pixels) of the ships shown in this class affects the evaluation of the IoU index. However, the mAP of all 6 classes is higher. These further verify the diversity of the ship scale in MariBoatsSubclass. We also noticed that although these models have improved after retraining on MariBoatsSubclass, there is still considerable room for improvement in the performance of these models. For example, these models still have the problem of incomplete segmentation, as shown in Figure 10

[0119] Since the existing SOLOv2 has the best performance, we choose SOLOv2 to test the computational speed using different numbers of layers (4-5 rows). When the backbone network ResNet is set to 18 layers and 34 layers, the segmentation accuracy of SOLOv2 is 56.2% and 57.4% in mAP, respectively. Although the mAP values decrease by 1.6% and 0.4%, respectively, the computational speed of SOLOv2 increases by 35% and 20% in FPS, reaching 44.7 and 39.9, respectively. This shows that by reducing the number of remaining layers of the backbone network, the segmentation speed can be significantly improved with little loss in AP.

[0120] Table 5. Test results of different network structures on MariBoatsSubclass

[0121]

[0122] 4.2.3, Segmentation results of SOLOv2 ship instance segmentation model using global and local attention mechanism GALA

[0123] ​To demonstrate the performance of the proposed GALA attention mechanism, we introduce different classic attention mechanisms into the existing SOLOv2 as comparison (Table 6). As shown in Table 6, compared with SOLOv2 without using any attention mechanism, using several classic attention mechanisms ECA-Net, ScSE, Triplet Attention, SENet and CA has a certain improvement in mAP, and the improvement of using GALA mechanism is the largest, reaching 62.1%, which is 4.3% higher than SOLOv2, and 3.8%, 3.6% and 3.3% higher than CA, Triplet, respectively. Attention and SEnet. This verifies that the performance of the proposed GALA attention mechanism in the VL ship dataset is better than that of the existing attention mechanism.

[0124] We further analyze the computational complexity of the SOLOv2 model using GALA in terms of the increase in the number of parameters and FPS. The increase in the number of parameters of SOLOv2 is only 0.02M. Compared with the 30.94M parameters of SOLOv2, this increase is almost negligible. The increase in time complexity is only 0.03GMAC, and the FPS only decreases by 1.6. This analysis shows that introducing GALA into the feature pyramid network can significantly improve the performance of the instance segmentation model, while the time complexity and the number of parameters almost do not increase.

[0125] Table 6. Test results of different attention mechanisms on MariBoatsSubclass

[0126]

[0127] Visually, GALA also improves the segmentation performance of SOLOv2 on MariBoats. As shown in Figure 11 , SOLOv2, SOLOv2+SEnet and SOLOv2+CA networks cannot completely detect all ships, either partially missing ships Figure 11 (circles in a-c), or completely missing ships Figure 11 (i and Figure 11 k). In addition, SOLOv2, SOLOv2+SEnet and SOLOv2+CA networks mistakenly detect an airplane as part of a ship below it Figure 11 (e-g). However, SOLOv2+GALA can correctly separate the airplane from the ship and can completely correctly detect all ships Figure 11 (d, l and h). Therefore, this visual comparison together with the quantitative comparison verifies the superiority of the proposed GALA mechanism over the existing attention mechanisms.

[0128] 5. Conclusion and discussion

[0129] Marine vessel instance segmentation of VL images plays an important role in marine-related scientific research, education, and commercial applications. However, there are almost no publicly available and suitable datasets containing vessel VL images for the purpose of vessel instance segmentation. To address this issue, we collected and manually annotated two new VL vessel datasets using the developed data collection tool. These datasets and the accompanying image processing tools are available to the public for research on visual perception technology for marine scenes. To the best of our knowledge, there are few instance segmentation methods designed specifically for vessel instance segmentation. Therefore, considering the particularity of marine vessel instance segmentation, we propose a GALA attention mechanism that utilizes 1D strip pooling and 2D spatial pooling to preserve global and local information of input images. Experimental results demonstrate that GALA outperforms existing attention mechanisms on marine vessel datasets.

[0130] Sun et al. established a VL dataset MariShipInsSeg containing 4K images, but MariShipInsSeg is not open source and has a general ship category. In contrast, our dataset MariBoats has more images and is open source. Moreover, we further refined the categories of MariBoats into six ship categories, forming our second dataset MariBoatsSubclass. It will facilitate the research of precise segmentation and classification of marine vessels based on appearance, shape, and function. Furthermore, we introduced attention mechanisms into marine vessel instance segmentation of VL images and further proposed a new attention mechanism that can improve the performance of existing neural networks. It is worth noting that although attention mechanisms have been applied to the COCO dataset, there is no such attention model designed specifically for marine vessel instance segmentation of VL images. Marine vessel images are different from most images in the COCO dataset in terms of background, texture, and object contour. Therefore, this is the first time that attention mechanisms are introduced into marine vessel instance segmentation using VL images. Existing attention mechanisms, such as ECA-Net, ScSE, Triplet Attention, and SENet, mostly focus on global information, and only a few attention mechanisms, such as CA, focus on local information. Inspired by CA, we combined global and local mechanisms to preserve global and local feature information, achieving better segmentation results than existing attention mechanisms that use only global or local information. The GALA mechanism we proposed maintains the convenience of attention mechanisms, which can be universally applied to most neural networks, and the source code can be immediately made available to the public. Finally, we also disclosed the accompanying data extraction and analysis tools to facilitate research in the field of computer vision. In summary, we believe that the new open-source dataset we established and the GALA mechanism we proposed in this work will facilitate research in VL ship applications and attract attention in other computer vision fields to use the GALA mechanism.

Claims

1. A ship instance segmentation algorithm based on global and local attention mechanisms, characterized in that, Includes the following steps: S1. Constructing a ship instance segmentation dataset A script was developed to collect image data from the Google Image platform. Using "ship" as the search keyword, images of ocean vessels were searched. Real ocean vessel images were manually selected, and LabelMe software was used to segment and label the selected images. During the labeling process, the shapes of the vessels were continuously tracked and labeled in the images to draw polygonal bounding boxes for different vessels. Then, the labeled ocean vessel images were named and classified. After depicting all the vessels in the images using polygonal annotations, an image annotation file was generated in JSON format to obtain the dataset. Finally, the dataset was named and annotated using the same naming and annotation method as the MS-COCO dataset to obtain the ship instance segmentation dataset, named the MariBoats dataset. The MariBoats dataset contains all labeled ocean vessel images and ship segmentation annotations. All labeled ocean vessel images were named the self_boats dataset and assigned the "ship" category. The coco_boats dataset was merged into the MariBoats dataset. The coco_boats dataset was extracted from the known MS-COCO dataset, which contains ship VL images. Using images and segmentation annotations of six ship types from the MariBoats dataset as the dataset, the MariBoatsSubclass dataset was constructed. The six ship types are: "Engineering Ship", "Cargo Ship", "Speedboat", "Passenger Ship", "Official Ship", and "Unknown Ship". S2, Data Augmentation The MariBoats and MariBoatsSubclas datasets are augmented using at least one of the following data augmentation methods: horizontal flipping, scaling, and multi-scale input, to obtain the processed ship instance segmentation dataset. The specific operations for horizontal flipping, scaling, and multi-scale input are as follows: Horizontal flip: The ship instance segmentation dataset is expanded by horizontal flipping. That is, the images in the ship instance segmentation dataset are flipped 180 degrees from left to right with a 50% probability. The horizontally flipped image data is then incorporated into the corresponding ship instance segmentation dataset. Scaling: The images in the ship instance segmentation dataset are scaled to obtain compressed image data, which is then incorporated into the corresponding ship instance segmentation dataset; Multi-scale input: Based on scaling, multiple scaling scales are preset, and one scaling scale is randomly selected from the preset scaling scales for input to obtain multi-scale image data and incorporate it into the corresponding ship instance segmentation dataset; S3. Construct a SOLOv2 ship instance segmentation network using the global and local attention mechanism GALA. A SOLOv2 ship instance segmentation network using the global and local attention mechanism GALA is constructed. This network consists of a backbone network, a neck layer, and a predicted head. This SOLOv2 ship instance segmentation network is used as the SOLOv2 ship instance segmentation model, where: The backbone network refers to the convolutional neural network used to extract features at different stages, namely the ResNet-50 network; The predictive head is used to predict the type and location of ship targets; The neck layer is located between the backbone network and the predicted head. The neck layer uses an enhanced feature pyramid network and adds network layers for collecting feature maps at different stages. The backbone network comprises five sequentially connected feature extraction segments for extracting features from five different stages: stage 0, stage 1, stage 2, stage 3, and stage 4. The neck layer consists of four network layers, which are constructed from the last feature maps of stages 1, 2, 3, and 4 of the backbone network. The construction method is as follows: [The text abruptly ends here, so the translation stops.] For each feature map in the last layer of layer 4, a 1×1 convolutional layer is applied for channel scaling, constructing layers P1, P2, P3, and P4. Since the feature maps between two adjacent feature extraction segments are scaled by a factor of 2, the width and height scales of layers P1, P2, P3, and P4 are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. First, the feature map of layer P4 is upsampled by a factor of 2 using bilinear interpolation, while maintaining the same width and height scale as the feature map of layer P3. The feature map of layer P4 is then added to the feature map of layer P3. Next, the feature map of layer P3 is upsampled by a factor of 2 using bilinear interpolation, while maintaining the same width and height scale as the feature map of layer P2. The feature map of layer P3 is then added to the feature map of layer P2. Finally, the feature map of layer P2 is upsampled by a factor of 2 using bilinear interpolation, while maintaining the same width and height scale as the feature map of layer P1. The feature map is added to the feature map of layer P1. For layers P1, P2, P3, and P4, a 3×3 convolutional layer is applied and connected to the SOLOv2 ship instance segmentation model. This model includes a global attention mechanism model and a local attention mechanism model. The global attention mechanism model consists of a global average pooling layer, a fully connected layer, a ReLU corrected linear unit, and a sigmoid activation layer. This global attention mechanism model performs pointwise multiplication of the input feature map with the output result. The local attention mechanism model consists of a global average pooling layer in the X direction, a global average pooling layer in the Y direction, a concatenate operation layer, a 1×1 convolutional layer, a BN batch normalization unit, a nonlinear layer, a 1×1 convolutional layer, a split operation layer, a 1×1 convolutional layer, and a sigmoid activation function layer. This local attention mechanism model performs pointwise multiplication of the input feature map with the output result. The process of establishing the Global and Local Attention (GALA) mechanism is as follows: 1) Reduce the dimensionality of the entire 2D spatial information in the feature map by global average pooling, that is, average all pixel values ​​of each channel map to obtain a new 1×1 channel feature map. The output of the feature map of the c-th channel is represented as: Where x c Here, z is the input image of the c-th channel, H is the height of the input image, and W is the width of the input image; c This is the output image of the c-th channel; After activation function and convolution transformation, the feature map of channel correlation is obtained. The output image of the c-th channel is represented as: in This represents the input image of the c-th channel that already has channel correlation; σ represents the sigmoid activation function; T1 and T2 are nonlinear transformations describing the importance of each channel; δ represents the ReLU activation function. 2) Perform one-dimensional pooling on the feature maps with channel correlation in both the horizontal and vertical directions. The output image of the c-th channel with height h is represented as: The output image of the c-th channel with width w is represented as: After activation function and convolution transformation, the final feature map with channel correlation, orientation awareness, and position sensitivity is obtained. The output image of the c-th channel is represented as: Where F h and F w This represents two 1×1 convolution transformations; 3) Finally, calculate the loss function, which is defined as: L = Lcate + λLmas Where Lcate is the regular Focal Loss for semantic classification, and Lmask is the Dice Loss for mask prediction, which are manually set parameters; S4, Network Training and Testing Using 70% of the data in the MariBoatsSubclass dataset as the training set and 30% as the test set, the processed ship instance segmentation dataset obtained in step S2 is input into the SOLOv2 ship instance segmentation model constructed in step S3 for training. At the beginning of the training phase, the backbone network adopts transfer parameter learning to obtain a trained network. The test set is validated based on the trained network, and the ship instance segmentation results are output.

2. The ship instance segmentation algorithm based on global and local attention mechanisms according to claim 1, characterized in that, in: Stage 0 contains convolutional layer 0 with corresponding Batch Normalization (BN) units 0 and ReLU corrected linear units 0, and pooling layer 0. The convolutional layer 0 has a 7×7 kernel size and a stride of 2. The pooling layer 0 uses max pooling with a kernel size of 3 and a stride of 2. Stage 1 contains three residual blocks. Each residual block contains a 1×1 convolutional layer 1 with corresponding BN units 1 and ReLU corrected linear units 1, a 3×3 convolutional layer 2 with corresponding BN units 2 and ReLU corrected linear units 2, and a 1×1 convolutional layer 3 with corresponding BN units 3 and ReLU corrected linear units 3. Convolutional layers 1, 2, and 3 are connected sequentially, and the input of each residual block is added to the output of convolutional layer 3. Stage 2 contains four residual blocks, stage 3 contains six residual blocks, and stage 4 contains three residual blocks. Each residual block in stages 2, 3, and 4 is connected to the stage... 1 contains residual blocks with identical structures; The prediction header contains multiple 3×3 convolutional layers and corresponding GN batch normalization units, ReLU corrected linear units, and bilinear upsampling layers.

Images