Method for detecting rotating ships based on feature extraction of remote sensing images

Abstract

This invention presents a method for detecting rotating ships based on remote sensing image feature extraction. First, remote sensing images are acquired and normalized. Then, a detection model is constructed, consisting of three networks: feature extraction, feature fusion, and prediction. The feature extraction network uses ResNet as its basic framework, replacing the 3×3 kernel convolutions in the bottleneck layers of the ResNet network with deformable dilated convolutions. The feature fusion network includes two deep supervised feature pyramids, each comprising a bottom-up downsampling branch and a top-down upsampling branch, with a self-attention module embedded within each pyramid. The prediction network includes an RPN network, a RoIPooling layer, and a detection head based on intersecting circles. This method can better extract features from rotating ships, unaffected by the rotation angle. The deep supervised feature pyramids address the impact of varying ship scales and background noise.

Description

Technical Field

[0001] This invention belongs to the field of ship detection technology in remote sensing images, specifically a rotating ship detection method based on remote sensing image feature extraction, which can be used for fishing vessel management, maritime ship identification and positioning, ship transportation, etc. Background Technology

[0002] As the main body of maritime transportation, accurate detection of ship positions at sea is of great significance in both military and civilian applications. In civilian applications, ship detection not only provides effective information for maritime vessel management but also provides accurate location information for maritime rescue, ensuring maritime traffic safety. In military applications, ship detection enables the timely detection of incoming vessels and dynamic monitoring of the distribution of ships at sea, which is crucial for strategic deployment.

[0003] In the field of computer vision, deep learning-based object detection technology has developed rapidly, achieving good recognition accuracy and speed in natural scenes. Remote sensing technology, with its advantages of macroscopic, high efficiency, and high resolution, provides new ideas for ship detection. Remote sensing images are an important carrier of ship position information. Compared with images in natural scenes, ship detection based on remote sensing images faces more challenges, specifically: 1) Complex image backgrounds due to differences between land and sea, ports, and day and night increase the difficulty of feature extraction; 2) Remote sensing images are generally taken by aerial or satellite, and due to different shooting angles and geographical locations, ships may appear rotated at arbitrary angles in the image, meaning that ships in the image are not neatly arranged in the horizontal or vertical direction; 3) To alleviate water resistance, ships are often designed as long and narrow shapes with a large length-to-width ratio, and the scale of ships varies due to factors such as image resolution and ship type.

[0004] To address the aforementioned issues, this application proposes a rotating ship detection method based on remote sensing image feature extraction. This method can not only detect rotating ships of varying scales and at arbitrary angles, but also preserves shallow information by utilizing the concept of depth supervision, thereby improving the detection accuracy of targets in complex backgrounds. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to propose a rotating ship detection method based on remote sensing image feature extraction.

[0006] The present invention solves the aforementioned technical problem by adopting the following technical solution:

[0007] A method for detecting rotating ships based on remote sensing image feature extraction, characterized in that the method includes the following:

[0008] Acquire remote sensing images and perform normalization processing on them;

[0009] A detection model is constructed. The detection model consists of three network parts: feature extraction, feature fusion, and prediction. The feature extraction network uses the ResNet network as the basic framework and uses deformable dilated convolution to replace the 3×3 convolution kernels in each bottleneck layer of the ResNet network. The normalized remote sensing image is input into the feature extraction network for feature extraction.

[0010] Deformable dilated convolution performs deformable convolution, dilated convolution, and average pooling on the input feature map. The feature map obtained by average pooling is then divided into two parts by convolution operation. The two parts of feature weights are multiplied by the feature maps obtained by deformable convolution and dilated convolution, respectively. The feature maps obtained by multiplication are then added together to obtain the output of deformable dilated convolution.

[0011] The feature fusion network comprises two deep supervised feature pyramids. Each deep supervised feature pyramid includes a bottom-up downsampling branch and a top-down upsampling branch. The input to the upsampling branch of the first deep supervised feature pyramid is the multi-scale feature map obtained by the feature extraction network. Each feature map obtained from the downsampling branch of the first deep supervised feature pyramid is processed by a self-attention module to obtain a feature attention map. Each feature attention map is multiplied by a feature map of the same level in the upsampling branch of the first deep supervised feature pyramid to obtain a feature map, which is then used as the input to the downsampling branch of the second deep supervised feature pyramid. Similarly, each feature map obtained from the upsampling branch of the first deep supervised feature pyramid is processed by a self-attention module to obtain a feature attention map. Each feature attention map is then multiplied by a feature map of the same level in the downsampling branch of the second deep supervised feature pyramid to obtain a feature map, which is then used as the input to the upsampling branch of the second deep supervised feature pyramid.

[0012] The prediction network includes an RPN network, an RoI Pooling layer, and a detection head based on intersecting circles. The four multi-scale feature maps output by the feature fusion network are input into the RPN network to generate candidate boxes. The four multi-scale feature maps output by the feature fusion network and the candidate boxes corresponding to each feature map are input into the RoI Pooling layer for feature mapping to generate feature vectors. The feature vectors are then input into the detection head based on intersecting circles for regression prediction to obtain the classification and position information of the rotating ship.

[0013] Furthermore, the self-attention module includes a series of average pooling layers, convolutional layers, ReLU activation functions, and Hard sigmoid activation functions.

[0014] Furthermore, the detection head based on intersecting circles includes a parallel fully connected detection head and a convolutional detection head. The fully connected detection head consists of two serial fully connected layers for classification prediction; the convolutional detection head consists of two serial convolutional layers for regression prediction.

[0015] Furthermore, both deformable convolution and dilated convolution have a kernel size of 3×3.

[0016] Furthermore, the ResNet network is ResNet18, ResNet34, ResNet50, or ResNet101.

[0017] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0018] 1. Conventional convolution kernels are square, and the window sliding method is either horizontal or vertical, making it difficult to extract rotation-invariant features. Considering that deformable convolution can achieve arbitrary point sampling and extract features of arbitrary shapes, and dilated convolution can expand the receptive field and extract more contextual information without changing the number of parameters, this invention proposes a deformable dilated convolution that can more stably extract feature information of rotating ships, unaffected by the ship's rotation angle, thus obtaining rotation-invariant features and enhancing the feature's ability to represent ship rotation information.

[0019] 2. In convolutional neural networks (CNNs), shallow neurons tend to have stronger activation responses to detailed features such as texture and edges of the target object, while deep neurons tend to have higher activation responses to the entire target object. This means that increasing the depth of the neural network can improve its representational ability and recognition accuracy to some extent. Furthermore, remote sensing images often contain significant noise, which necessitates shallow neurons to distinguish these noises. Therefore, this invention proposes a deep-supervised feature pyramid, adding an upsampling branch and a downsampling branch to the original feature pyramid, thus increasing the depth of the feature pyramid and aiming to enhance the representational ability of extracted features for multi-scale targets. However, as the network depth increases, the neural network becomes increasingly difficult to train, exhibiting phenomena such as gradient vanishing and gradient exploding. By borrowing the residual concept from ResNet, a self-attention module is added to the feature pyramid, preserving information from shallow FPNs while simultaneously providing deep supervision. This addresses the problems of inconsistent ship scales and low detection accuracy due to background noise. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of the present invention;

[0021] Figure 2 This is a structural diagram of the detection model of the present invention;

[0022] Figure 3 This is a diagram of the deformable hollow convolution structure of the present invention;

[0023] Figure 4 This is a structural diagram of the depth-supervised feature pyramid of the present invention;

[0024] Figure 5 This is a structural diagram of the self-attention module of the present invention;

[0025] Figure 6 The figure shows the test results of the method of the present invention on the DOTA dataset;

[0026] Figure 7 The figure shows the test results of the method of the present invention on the SSDD+ dataset. Detailed Implementation

[0027] Specific embodiments are given below with reference to the accompanying drawings. These specific embodiments are only used to illustrate the technical solutions of the present invention in detail, and are not intended to limit the scope of protection of this application.

[0028] This invention relates to a method for detecting rotating ships based on remote sensing image feature extraction (hereinafter referred to as the method, see [link]). Figures 1-7 The process includes the following steps:

[0029] The first step is to acquire remote sensing images and perform normalization processing on them.

[0030] Remote sensing datasets with OBB (Oriented bounding boxes) annotations, such as DOTA, HRSC2016, and SSDD+ remote sensing image ship datasets, were selected to perform data augmentation and normalization on ship images.

[0031] The second step is to construct a detection model consisting of three networks: feature extraction, feature fusion, and prediction.

[0032] The feature extraction network uses ResNet networks such as ResNet18, ResNet34, ResNet50, or ResNet101 as its basic framework. It replaces the 3×3 convolutions in the bottleneck layers of the ResNet network with deformable dilated convolutions and removes the final average pooling layer, fully connected layer, and output layer of the ResNet network. The feature extraction network outputs four multi-scale feature maps, which are the feature maps output by the last four modules of the feature extraction network.

[0033] Deformable dilated convolution combines deformable convolution and dilated convolution. It aims to use dilated convolution to increase the receptive field, while deformable convolution extracts features from irregular and rotating objects, thus ensuring the rotation invariance of the features. Deformable dilated convolution can more stably extract rotation-invariant features, enhancing the ability of features to represent ship rotation information. For example... Figure 3As shown, deformable dilated convolution performs deformable convolution, dilated convolution, and average pooling on the input feature map. The feature map obtained from average pooling is then convolved with a 1×1 kernel, and the feature weights are divided into two parts. These two parts are multiplied by the feature maps obtained from deformable convolution and dilated convolution, respectively, and then summed to obtain the output of deformable dilated convolution. The kernel size of both deformable and dilated convolution is 3×3, and the receptive field size of average pooling is 5×5. The 1×1 convolution operation is used for channel transformation to adapt to the output dimension of deformable and dilated convolution. The expression for deformable dilated convolution is:

[0034] F=S·Conv(APool(X))·Dconv(X,k,1)+(1-S)·Conv(APool(X))·Aconv(X,k,δ) (1)

[0035] Where X and F represent the input and output feature maps of deformable dilated convolution, respectively; Dconv(·) and Aconv(·) represent deformable convolution and dilated convolution, respectively; k represents the weight of the convolution kernel; δ represents the dilation rate; S represents the hyperparameter; APool(·) represents the average pooling operation; and Conv(·) represents the 1×1 convolution operation.

[0036] like Figure 4 As shown, the feature fusion network includes two Deep Feature Pyramid Networks (DFPNs). Each DFPN includes a bottom-up downsampling branch and a top-down upsampling branch. The input to the upsampling branch of the first DFPN is the multi-scale feature map obtained by the feature extraction network. Each feature map obtained from the downsampling branch of the first DFPN is processed by a self-attention (SA) module to obtain a feature attention map. Each feature attention map is multiplied by the feature map at the same level of the upsampling branch of the first DFPN to obtain the feature map, which is then used as the input to the downsampling branch of the second DFPN. The first DFPN... Each feature map obtained from the upsampling branch is processed by a self-attention module to obtain a feature attention map. Each feature attention map is multiplied by the feature map of the same level in the downsampling branch of the second deep supervised feature pyramid, and the resulting feature map is used as the input to the upsampling branch of the second deep supervised feature pyramid. The feature fusion network outputs a total of four multi-scale feature maps. DFPN uses multi-scale features for prediction and further propagates high-level features, so that each feature level contains richer high-level semantic information, which improves the performance of object detection, especially for small targets. DFPN mitigates the impact of different ship scales and improves the network's representation ability by increasing the network depth, aiming to extract scale-invariant features.

[0037] like Figure 5 As shown, the self-attention module (SA) includes an average pooling layer, a convolutional layer, a ReLU activation function, and a Hard sigmoid activation function. For example, the high-level features of the first deep supervised feature pyramid downsampling branch pass through an average pooling layer, a convolutional layer, a ReLU activation function, and a Hard sigmoid activation function in sequence to obtain the feature attention map. The Hard sigmoid activation function has a faster computation speed than the sigmoid activation function.

[0038] The prediction network includes an RPN network, an RoI Pooling layer, and an Intersecting Circle-based Detection Head (ICR-Head). The four feature maps output by the feature fusion network are input into the RPN network, generating multiple candidate boxes on each feature map. The RPN network is used to obtain candidate regions and perform non-maximum suppression on the candidate regions. The regression loss and classification loss of each candidate box are calculated using Equation (2) to initially determine the position and size of the candidate box. The classification loss is used to distinguish between the foreground and the background, and the regression loss is used to fine-tune the position of the candidate box. Each feature map undergoes this operation to obtain candidate boxes.

[0039]

[0040] Classification loss in equation (2) Regression loss Satisfying equations (3) and (4) respectively:

[0041]

[0042]

[0043] Where, p i This represents the probability that the i-th candidate box is predicted to be a true box; The true label of the candidate box is 1 for positive samples and 0 for negative samples; t i =[t x , t y , t w , t h ] represents the predicted value of the i-th candidate box. Represents the true value of the i-th candidate box, (t) x , t y ), Let t be the predicted coordinates and the ground truth coordinates of the geometric center of the i-th candidate box. w , Let t be the predicted width and the actual width of the i-th candidate box, respectively. h , These are the predicted height and the actual height of the i-th candidate box, respectively; Ncls N represents the number of samples contained in the minimum input sample size; reg The number of candidate boxes is represented by λ; the weight balancing coefficient is represented by λ.

[0044] In equation (4), The function is defined as:

[0045]

[0046] The four multi-scale feature maps output by the feature fusion network and the candidate boxes corresponding to each feature map are input into the RoIPooling layer for feature mapping to generate feature vectors. The feature vectors are then input into the intersecting circle-based detection head for regression prediction. The intersecting circle-based detection head includes a parallel fully connected detection head and a convolutional detection head. The fully connected detection head consists of two serial fully connected layers for classification prediction. The convolutional detection head consists of two serial convolutional layers for regression prediction. Regression prediction is achieved by predicting a horizontal rectangle circumscribed by the rotating ship and an intersecting circle. The intersecting circle is the circumscribed circle of the matrix formed by the four endpoints of the rotating ship. For details, see patent CN1 13284185B.

[0047] The regression prediction loss consists of two parts: the fully connected head loss and the convolutional head loss. The fully connected head loss includes the classification loss L. cls And the loss L in the tilt direction of the prediction box s ;

[0048] L cls = -log pu (6)

[0049] L S =-[s * log(s) + (1-s) * [log(1-s)] (7)

[0050] Where p is the classification probability, u is the true class label of the rotating ship to which the predicted box belongs, and s is the probability of predicting the candidate box as a true box. * This is a real label;

[0051] Convolutional Detector Head Loss L reg for:

[0052] L reg =L h + λL r (8)

[0053]

[0054]

[0055] Among them, L h For the regression loss of the horizontal rectangle, L r Let λ be the regression loss of the intersecting circles, λ be the hyperparameter, and r, r * v represents the predicted value and the true label of the intersecting circle, respectively. i , These are the predicted values ​​and the true labels for the horizontal rectangles, respectively.

[0056] The third step is to train the detection model and then use the trained model to detect rotating ships in remote sensing images. During the training process, the total loss L is calculated using the following formula. total ;

[0057] L total =L RPN +L reg +L cls +βL S (11)

[0058] Where β is a hyperparameter.

[0059] Figure 6 , 7 The images show the test results of the method of this invention on the DOTA dataset and the SSDD+ dataset, respectively. The white matrix boxes in the images represent the detected rotating ships.

[0060] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1.A method for detecting a rotating ship based on feature extraction of a remote sensing image, characterized in that, The method comprises the following contents: Obtaining a remote sensing image, and performing normalization processing on the remote sensing image; A detection model is constructed; the detection model comprises three network parts of feature extraction, feature fusion and prediction, the feature extraction network takes a ResNet network as a basic framework, a deformable dilated convolution is used to replace a convolution kernel with a size of 3*3 in each bottleneck layer of the ResNet network, and the normalized remote sensing image is input into the feature extraction network for feature extraction; The deformable dilated convolution respectively performs deformable convolution, dilated convolution and average pooling on the input feature map, the feature map obtained through the average pooling is subjected to a convolution operation to divide feature weights into two parts, the two parts of feature weights are multiplied with the feature maps obtained through the deformable convolution and the dilated convolution respectively, and the feature map obtained through the multiplication is subjected to addition to obtain the output of the deformable dilated convolution; The feature fusion network comprises two deep supervision feature pyramids, each deep supervision feature pyramid comprises a lower sampling branch from bottom to top and an upper sampling branch from top to bottom, the input of the upper sampling branch of the first deep supervision feature pyramid is a multi-scale feature map obtained through the feature extraction network; each feature map obtained through the lower sampling branch of the first deep supervision feature pyramid is subjected to a self-attention module to obtain a feature attention map, and the feature map obtained through multiplication of each feature attention map and a feature map at the same level of the upper sampling branch of the first deep supervision feature pyramid is taken as the input of the lower sampling branch of the second deep supervision feature pyramid; each feature map obtained through the upper sampling branch of the first deep supervision feature pyramid is subjected to a self-attention module to obtain a feature attention map, and the feature map obtained through multiplication of each feature attention map and a feature map at the same level of the lower sampling branch of the second deep supervision feature pyramid is taken as the input of the upper sampling branch of the second deep supervision feature pyramid; The prediction network comprises an RPN network, a RoIPooling layer and an intersection circle-based detection head; four multi-scale feature maps output by the feature fusion network are input into the RPN network to generate a candidate box, the four multi-scale feature maps output by the feature fusion network and the candidate box corresponding to each feature map are input into the RoIPooling layer for feature mapping to generate a feature vector; and then the feature vector is input into the intersection circle-based detection head for regression prediction to obtain classification information and position information of a rotating ship. The intersection circle-based detection head comprises a parallel full connection detection head and a convolution detection head, the full connection detection head is composed of two serial full connection layers and is used for classification prediction, and the convolution detection head is composed of two serial convolution layers and is used for regression prediction. 2.The method of claim 1, wherein, The self-attention module comprises a serial average pooling layer, a convolution layer, a Relu activation function and a Hard sigmoid activation function. 3.The method of claim 1, wherein, The convolution kernel sizes of the deformable convolution and the dilated convolution are both 3*3. 4.The method of claim 1 or 3, wherein, The ResNet network is ResNet18, ResNet34, ResNet50 or ResNet101.

Images