A ship instance segmentation method based on spatially distributed attention

By introducing a spatially distributed attention method in the SOLOv2 algorithm, using feature pyramid network and spatial attention network, the problem of missegment background and missegment of similar instances in ship image instance segmentation is solved, and the instance sensitivity of segmented features is improved and anti-interference ability and accelerate the training process.

CN114581670BActive Publication Date: 2024-11-01HARBIN ENG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210304835.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-11-25
Filing Date
2022-03-23
Publication Date
2024-11-01
Estimated Expiration
2042-03-23

AI Technical Summary

Technical Problem

In the ship image instance segmentation task, the SOLOv2 algorithm has the problem of missegment between the missegment background and similar instances, and the training time is too long and the convergence is slow.

Method used

Using the ship instance segmentation method based on spatially distributed attention, the image feature extraction, the classification prediction of the grid and the position-sensitive dynamic convolution kernel are obtained, the relative relationship between the instances are calculated, the spatially distributed attention generation, the segmentation mask features and the final instance are generated. Segmentation masks use feature pyramid networks and spatial attention networks to enhance the instance sensitivity and anti-interference ability of segmentation features, and improve the training convergence speed.

Benefits of technology

It effectively solves the problem of missegment between similar ship instances caused by the insensitive SOLOv2 algorithm to instances, enhances the anti-interference ability to the background, and speeds up the training convergence speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114581670B_ABST
    Figure CN114581670B_ABST
Patent Text Reader

Abstract

The present invention provides a ship instance segmentation method based on spatial distribution attention, comprising the following steps: step 1: feature extraction of the image; step 2: obtaining the classification prediction of the grid and the position-sensitive dynamic convolution kernel; step 3: calculating the relative spatial distribution attention of the instance; step 4: generating the segmentation mask feature; step 5: generating the final instance segmentation mask. The present invention can effectively enhance the instance information in the feature map and reduce the interference of background information, while speeding up the network training speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of image segmentation of computer vision and relates to a ship instance segmentation method based on spatial distribution attention. Background Art

[0002] Image instance segmentation is a fundamental problem in computer vision, and is a combination of object detection and semantic segmentation. Given an input image, instance segmentation requires determining the location and semantic category of all objects in the image and performing pixel-level segmentation. For example, for a ship image input, all the ship pixels need to be segmented and labeled, and pixels of different ships need to be distinguished, labeled as ship 1 and ship 2.

[0003] Image instance segmentation algorithms are mainly divided into two-stage algorithms and single-stage algorithms. The two-stage algorithm is based on MaskR-CNN and mostly adopts the paradigm of first region screening and then segmentation. It has high accuracy but slow speed and is not suitable for real-time segmentation tasks. The single-stage algorithm does not use region candidates, so it greatly speeds up instance segmentation and is easy to apply to actual projects. SOLOv2, as a single-stage algorithm with better performance, has significant advantages in accuracy and speed. Since SOLOv2 divides instances in the image according to position and generates a full-image segmentation mask for each instance, it is difficult to effectively suppress background interference and it is not easy to distinguish instances with similar features in the image. In addition, the problems of SOLOv2's long training time and slow convergence also need to be solved.

[0004] Purpose of the Invention

[0005] The main problem solved by the present invention is to provide a ship instance segmentation method based on spatial distributed attention to address the mis-segmentation background problem and the mis-segmentation problem between similar instances that occur when SOLOv2 is used for ship image instance segmentation tasks.

[0006] The object of the present invention is achieved in that:

[0007] A ship instance segmentation method based on spatial distributed attention comprises the following steps:

[0008] Step 1: Image feature extraction;

[0009] Step 2: Obtain the classification prediction and position-sensitive dynamic convolution kernel of the grid;

[0010] Step 3: Calculate the relative spatial distribution attention of the instance;

[0011] Step 4: Generate segmentation mask features;

[0012] Step 5: Generate the final instance segmentation mask.

[0013] Furthermore, the step 1 is specifically as follows:

[0014] A two-dimensional image of any size is input into the residual convolutional feature extraction network to obtain multi-scale features of four stages. In order to make full use of the information of different stages, the high-level features rich in semantic information and the low-level features containing detail information are continuously integrated through the use of the feature pyramid network to obtain feature maps with rich semantic information at different scales, which are used to distinguish instances of different scales in the image.

[0015] Furthermore, the step 2 is specifically as follows:

[0016] The feature pyramid features of each layer output are bilinearly interpolated to obtain the SxS size grid feature representation S 2 The instance information of each location is continuously aggregated by stacking convolutional networks, and finally a prediction of the size of S*S*C is obtained to represent the probability of the existence of each location instance.

[0017] The dynamic convolution kernel generation network is also constructed by using the SxS features obtained by bilinear interpolation. The difference is that the input features expand the relative position information of x and y in two dimensions in the channel dimension, and the same number of convolutions as the classification network are superimposed to aggregate the feature information. Finally, each layer obtains S 2 Dynamic convolution kernel.

[0018] Furthermore, the step three is specifically as follows:

[0019] Perform attention modeling on the grid feature map generated in step 1, obtain q, k, v features and use them to calculate the relationship between different grids, and finally superimpose them on the original input grid features to obtain a clearer instance distribution grid feature.

[0020] Furthermore, the step 4 is specifically as follows:

[0021] The multi-stage pyramid feature maps of different scales generated in step 1 and the attention features with instance distribution information generated in step 4 are superimposed, and uniformly convolved and upsampled to 1 / 4 of the original image size. Finally, the features of different stages are added and fused into the final mask segmentation features using 1*1 convolution.

[0022] Furthermore, the step five is specifically as follows:

[0023] The position-sensitive dynamic convolution kernel generated in step 2 is applied to the segmentation features generated in step 4. For each convolution kernel, a prediction of an instance mask is generated, which corresponds one-to-one with the prediction of the instance classification in step 2.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention uses instance distributed attention of grid features to solve the problem of partial insensitivity to instances in the SOLOv2 algorithm, resulting in mis-segmentation between similar ship instances.

[0026] 2. The instance information in the segmentation features is clearer, thereby enhancing the SOLOv2 algorithm's ability to resist background interference.

[0027] 3. Using the attention features of the classification network to assist segmentation speeds up the training convergence of the SOLOv2 algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the present invention;

[0029] Figure 2 It is the feature extraction network and feature pyramid network diagram of the present invention;

[0030] Figure 3 It is a structural diagram of the segmentation feature fusion of the present invention;

[0031] Figure 4 is the spatial attention network diagram adopted by the present invention;

[0032] Figure 5 It is a corresponding relationship diagram between classification prediction and instance mask prediction of the present invention. DETAILED DESCRIPTION

[0033] The specific implementation modes of the present invention will be further described below in conjunction with the accompanying drawings.

[0034] Instance segmentation requires classification and segmentation of all targets in the image and effective distinction between different targets. From a visual point of view, different targets need to be masked with different colors. SOLOv2 cleverly uses the form of a grid to divide all areas in the image, and uses the grid as a sample point to predict on the multi-scale feature pyramid features. It only needs to generate an instance category probability for each grid (if all category probabilities are low, it means that there is no instance at this position) and generate a mask corresponding to the instance for this position to complete the instance segmentation task. In the SOLOv2 algorithm, the classification network and the segmentation feature are independent of each other. The present invention mainly utilizes the instance distribution information in the classification network. The instance information in the feature can be made more obvious by adjusting and introducing it into the segmentation feature through the spatial attention network. Since the classification feature includes the prediction information of the confidence (probability) of the instance category, the segmentation feature can be adjusted according to the instance information in its feature, so that its final segmentation feature is more sensitive to the instance. For similar instances, it will generate different activations according to the judgment of the classification to avoid different instances from being mis-segmented due to partial similar features (especially instances of the same category).

[0035] The steps of the present invention are as follows:

[0036] 1. Extraction of image features

[0037] The feature extraction process is as follows Figure 1 As shown in the figure, this method uses ResNet-101 as the feature extraction network (the ImageNet pre-trained ResNet model of pytorch is used during training to improve accuracy and accelerate convergence), and then uses the 4 scale features output by the feature extraction network to construct a feature pyramid network. The specific method is to continuously establish horizontal connections between high-level features and superimpose and convolve low-level features to construct 5 layers of multi-scale features from P2 to P6 with richer semantics (P6 is obtained by convolutional downsampling of P5 layer features).

[0038] 2. Generate classification predictions and position-sensitive dynamic convolution kernels

[0039] The output of the feature pyramid is used as the input feature for classification prediction and convolution kernel prediction. Since the principle of SOLOv2 is to divide the entire image into SxS grids to distinguish different instances in the image, the input features need to be unified into SxS size using bilinear interpolation. The grid features after bilinear interpolation are connected to a 4-layer convolutional network, in which Group Normalization and Relu activation functions are used after each convolution to adjust the output.

[0040] Since the dynamic convolution kernel needs to segment instances at different positions, it is necessary to break the translation invariance of the convolution itself. Therefore, Coordconv is introduced here. The specific operation is to add two dimensions to the input feature map and normalize them to the x and y relative position coordinates in the range of [0, 1], and then connect them to the full convolution network. The network depth and activation function configurations here are the same as those of the full convolution network of the classification network. Finally, the convolution kernel features of SxSxE dimensions will be obtained, where E indicates that the number of channels of the convolution kernel is the same as the number of input channels of the mask segmentation feature.

[0041] 3. Modeling the relationship between instances using spatial attention

[0042] The classification network is performed last. The corresponding grid features of the locations where instances exist will generate corresponding activations. Using the grid features before the classification prediction in the figure, the distribution of instances in the image can be easily obtained. Therefore, by using spatial attention to model the locations between instances and using this relationship to assist segmentation, the accuracy and anti-interference of the final segmentation can be improved to a certain extent. For the spatial attention network of grid features, Figure 4As shown in the figure, the input SxSxC (C is the number of channels, 512 channels in the figure) grid feature is convolved by 1*1 to obtain three feature vectors, namely query vector, key vector and value vector. The three-dimensional vector is converted into a two-dimensional vector by dimension conversion to facilitate the calculation of mutual self-attention. The calculation formula of self-attention is Attention=softmax(Q·K T )·V+F, where Q, K, and V represent query, key, and value vectors respectively, and F represents the input features.

[0043] 4. Generate attention fusion mask features

[0044] Fusion methods such as Figure 4 The main idea is to use the original features of the four-layer feature pyramid {P2, P3, P4, P5} to fuse the attention features generated by the spatial attention network to make the instance information in the segmentation features more obvious and reduce interference. Considering that the original SOLOv2 network uses the 4-layer pyramid feature fusion of P2-P4 as the segmentation feature, and the present invention needs to integrate the spatial attention features generated by the 5-layer classification network of P2-P6 into the final segmentation feature, the P5 layer grid feature is superimposed on the P6 layer feature by bilinear interpolation and adjusted by 1*1 convolution to merge into a layer of features as the input of the spatial attention network. In order to integrate the generated attention features into a unified segmentation feature, bilinear interpolation is first used to convert the generated attention grid features into the same size as the original pyramid features and superimpose them. At this time, the feature maps of the P2-P4 layers are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image size, respectively. Subsequently, in order to fuse the features of all layers into one layer of features, convolution, Relu, and bilinear interpolation 2x upsampling operations are repeated on images of different scales. The features of all layers are unified to 1 / 4 of the original image size. Finally, the features generated by the four layers are superimposed, and the final output channel is adjusted with 1*1 convolution to obtain the final fused segmentation feature map rich in instance distribution attention information.

[0045] 5. Generate final mask prediction

[0046] The final mask is obtained by convolving the fused segmentation features with an instance-sensitive 1*1 dynamic convolution kernel. Its principle is similar to that of a pixel classifier, which determines whether the pixel at each position belongs to the instance at that position. At the same time, the dynamic convolution kernel and the grid features of the classification network are one-to-one corresponding. The corresponding relationship can be referred to Figure 5 ,The prediction generated by the convolution kernel on the segmentation features,the order of the grid from left to right and top to bottom in the classification network is matched with the,prediction of the instance mask. Finally, only Matrix NMS is needed to be,screened based on the high classification prediction probability and the,quality of instance prediction to obtain the final prediction,of the image.

Claims

1. A ship instance segmentation method based on spatially distributed attention, Its characteristics are: The following steps are involved: Step 1: Image feature extraction; Step 2: Obtain the classification prediction and position-sensitive dynamic convolution kernel of the grid; The feature pyramid features of each layer output are bilinearly interpolated to obtain the SxS size grid feature representation S 2 The instance information of each location is continuously aggregated by stacking convolutional networks, and finally a prediction of the size of S*S*C is obtained to represent the probability of the existence of each location instance; The dynamic convolution kernel generation network is also constructed by using the SxS features obtained by bilinear interpolation. The difference is that the input features expand the relative position information of x and y in two dimensions in the channel dimension, and the same number of convolutions as the classification network are superimposed to aggregate the feature information. Finally, each layer obtains S 2 Dynamic convolution kernels; Step 3: Calculate the relative spatial distribution attention of the instance; Perform attention modeling on the grid feature map generated in step 1, obtain q, k, v features and use them to calculate the relationship between different grids, and finally superimpose them on the original input grid features to obtain grid features with clear instance distribution; Step 4: Generate segmentation mask features; Step 5: Generate the final instance segmentation mask.

2. A ship instance segmentation method based on spatial distributed attention according to claim 1, Its characteristics are: The step 1 is specifically as follows: A two-dimensional image of any size is input into the residual convolutional feature extraction network to obtain multi-scale features of four stages. In order to make full use of the information of different stages, the high-level features rich in semantic information and the low-level features containing detail information are continuously integrated through the use of the feature pyramid network to obtain feature maps with rich semantic information at different scales, which are used to distinguish instances of different scales in the image.

3. The ship instance segmentation method based on spatial distributed attention according to claim 1, Its characteristics are: The step 4 is specifically as follows: The multi-stage pyramid feature maps of different scales generated in step 1 and the attention features with instance distribution information generated in step 4 are superimposed, and uniformly convolved and upsampled to 1 / 4 of the original image size. Finally, the features of different stages are added and fused into the final mask segmentation features using 1*1 convolution.

4. The ship instance segmentation method based on spatial distributed attention according to claim 1, Its characteristics are: The step five is specifically as follows: The position-sensitive dynamic convolution kernel generated in step 2 is applied to the segmentation features generated in step 4. For each convolution kernel, a prediction of an instance mask is generated, which corresponds one-to-one with the prediction of the instance classification in step 2.