A method for detecting dock targets in remote sensing images

Abstract

The present invention relates to a method for detecting dock targets in remote sensing images, and belongs to the technical field of remote sensing image processing. Based on the YOLOv4 horizontal box detection algorithm, the present invention adds a PSA attention module to the feature fusion network, utilizes the PSA attention module to obtain receptive fields of different scales, and extracts the attention weights of channels to obtain the weights of attention of channels of different scales, thereby achieving fusion of contextual information of different scales and further improving the detection accuracy of the network. At the same time, a multi-dimensional corner point coordinate detection frame is used to calibrate the dock target, which can effectively represent the true boundary of the dock target, solve the problem of arbitrary dock direction, further improve the positioning accuracy of the target, and achieve accurate detection of the dock target.

Description

Technical Field

[0001] The invention relates to a remote sensing image dock target detection method, belonging to the technical field of remote sensing image processing. Background Art

[0002] With the rapid development of remote sensing technology, the use of remote sensing imagery for offshore target detection has become a research hotspot. Docks, as typical offshore targets, provide crucial insights for maritime battlefield environment development and marine economic development. However, dock targets in remote sensing imagery exhibit arbitrary orientation and diverse sizes, and are affected by surrounding vessels, man-made objects, and other environmental factors, making accurate dock target location challenging. Traditional dock target detection methods include edge detection, object-oriented dock recognition, and feature-based port detection. These methods primarily utilize information about the coastal waterline or shape features such as the dock's length and width to detect docks through a process of false alarm elimination and target confirmation. However, these methods are significantly influenced by subjective factors and struggle to accurately extract dock targets in the presence of interference from clouds, ships, and waves.

[0003] In recent years, deep learning, especially convolutional neural networks (CNNs), has achieved tremendous success in the field of computer vision. Object detection methods based on deep learning have received widespread attention and achieved remarkable success. Two-stage detection methods, such as R-FCN and Faster R-CNN, primarily involve two stages: region proposal and region classification. Two-stage detection networks first use a selective search algorithm or a region proposal network (RPN) to generate a series of candidate boxes, then perform object classification and bounding box regression. Unlike two-stage detection networks, single-stage detection networks, such as SSD and YOLO, simultaneously perform both classification and localization, offering speed advantages. Furthermore, YOLOv3 and YOLOv4 improve upon the YOLO network by utilizing a more powerful backbone network and multi-scale fusion strategies, significantly improving both model speed and accuracy. This has laid the foundation for using deep learning networks for object detection in remote sensing imagery.

[0004] These advanced object detection methods typically use horizontal rectangular boxes to describe the detected target. While suitable for detection in natural scene images, they cannot meet the requirements for dock target detection in remote sensing imagery. Dock targets in remote sensing imagery typically have a large aspect ratio and exhibit a certain degree of directionality. Using horizontal boxes for detection would include excessive redundant information, making it difficult to accurately locate the target. Furthermore, docks are often scattered throughout complex scenes, and the noise contained in the redundant areas of the horizontal detection box can interfere with feature extraction, significantly impacting dock target detection. Summary of the Invention

[0005] The purpose of the present invention is to provide a remote sensing image dock target detection method to solve the problem of poor positioning accuracy in the current use of horizontal detection to detect remote sensing image docks.

[0006] In order to solve the above technical problems, the present invention provides a remote sensing image dock target detection method, which includes the following steps:

[0007] 1) Obtain the remote sensing image to be detected;

[0008] 2) inputting the remote sensing image to be detected into the target detection model to output the dock target detection result;

[0009] The target detection model includes a backbone network, a feature fusion network and a prediction layer;

[0010] The backbone network is used to extract features from the input remote sensing image to obtain feature maps of different scales;

[0011] The feature fusion network includes an SPP module, a PSA attention module and a feature pyramid module. The SPP module is used to perform a maximum pooling operation on the features output by the backbone network and fuse feature maps of different scales. The PSA attention module is used to segment the output results of the SPP module and extract scale features and channel attention weights to obtain feature maps with multi-scale features and attention weights. The feature pyramid module is used to repeatedly extract and fuse the feature maps with multi-scale features and attention weights to obtain features with precise target location information and high-level semantic information.

[0012] The prediction layer locates the target to be detected by establishing a multi-dimensional corner point coordinate detection frame and obtaining features with precise target location information and high-level semantic information. The multi-dimensional corner point coordinates include the coordinates of the four corner points of the detection frame.

[0013] The present invention is based on the YOLOv4 horizontal box detection algorithm, adds a PSA attention module to the feature fusion network, uses the PSA attention module to obtain receptive fields of different scales, and extracts the attention weights of channels to obtain the attention weights of channels of different scales, thereby fusing contextual information of different scales and further improving the detection accuracy of the network. At the same time, a multi-dimensional corner point coordinate detection frame is used to calibrate the dock target, which can effectively represent the true boundary of the dock target, solve the problem of arbitrary dock direction, further improve the positioning accuracy of the target, and realize accurate detection of the dock target.

[0014] Furthermore, the PSA attention module includes an SPC module, an SE module and an output module; the SPC module is used to divide the feature map output by the SPP module into several parts in the channel dimension, and perform feature extraction of different scales on each part to obtain feature vectors of each part, and generate feature maps of the corresponding channels; the SE module is used to extract corresponding channel attention vectors from the feature maps of each channel, and re-characterize the attention vectors of each channel to obtain the corresponding attention weights after multi-scale channel interaction; the output module is used to perform weighted fusion processing on the feature maps of the corresponding channels according to the obtained attention weights to obtain feature maps with multi-scale features and attention weights.

[0015] The present invention first uses the SPC module to obtain receptive fields of different scales to better extract the multi-scale information of the image; secondly, the SE module is used to extract the attention weights of the channels to obtain the weights of the attention of channels of different scales, so that the PSA module can fuse contextual information of different scales.

[0016] Furthermore, the SPC module uses group convolution to extract features of different scales for each part. The convolution kernel of each group is related to the size of the group. The relationship between the convolution kernel size k and the group is:

[0017]

[0018] Among them, G is the size of the group and k is the size of the convolution kernel.

[0019] The present invention provides a method for generating huge parameters as the size of the convolution kernel increases continuously. Therefore, when performing group convolution on the feature vectors of each part after segmentation, the convolution kernel size k is defined to be related to the group size of each part.

[0020] Furthermore, the multi-dimensional corner coordinate loss function used in the training of the target detection model is:

[0021] L=L Pre +L Conf +L Cls

[0022]

[0023]

[0024]

[0025] S 2 Indicates the number of grids into which the input remote sensing image is divided, B indicates the number of prior frames in each grid, Pr(object) indicates whether the current prior frame contains the target. When the target is included, the value of Pr(object) is 1, and when there is no target, the value of Pr(object) is 0; x, y, x gt 、y gt Represent the detection box and the real box coordinates, c and c respectively gt Represent the predicted confidence and true confidence, p and p respectively gt denote the probability of the predicted category and the true probability, respectively.

[0026] In order to calibrate a detection frame that better fits the target for objects with directional information, the present invention uses multi-dimensional corner point coordinates to calibrate the target. Therefore, when training the model, a multi-dimensional corner point coordinate loss function is used to optimize the detection results of the corner point coordinate detection frame.

[0027] Furthermore, the backbone network adopts the CSPDarknet53 network.

[0028] The present invention selects CSPDarkNet53 as the backbone network and combines the backbone network DarkNet53 of YOLOv3 with the cross-stage local network to better extract the feature information of the image.

[0029] Furthermore, the prediction layers include 3, and each prediction layer includes 8 coordinate information, 1 bounding box confidence and 1 category confidence.

[0030] Furthermore, the calculation formula used for the border confidence is:

[0031]

[0032] Among them C ij is the jth prior box in the i-th grid; Pr(object) indicates whether the current prior box contains the target. When the target is contained, the value of Pr(object) is 1, and when the target is not contained, the value of Pr(object) is 0; It represents the intersection-over-union ratio of the predicted bounding box to the true bounding box, between 0 and 1. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a schematic diagram of the current YOLOv4 network structure;

[0034] Figure 2 This is a schematic diagram of the CSPDarkNet53 structure used;

[0035] Figure 3-a This is a schematic diagram of the CBM structure in the CSPDarkNet53 structure;

[0036] Figure 3-b This is a schematic diagram of the residual connection structure in the CSPDarkNet53 structure;

[0037] Figure 4 This is the structure diagram of the CSPn module in the CSPDarkNet53 structure;

[0038] Figure 5 This is a schematic diagram of the SPP module structure in the CSPDarkNet53 structure;

[0039] Figure 6 This is the PANet module structure diagram in the CSPDarkNet53 structure;

[0040] Figure 7 Schematic diagram of the improved YOLOv4 network structure adopted by the present invention;

[0041] Figure 8-a It is currently using horizontal boxes to mark schematic diagrams;

[0042] Figure 8-b This is a schematic diagram of the multi-dimensional corner point coordinate detection box annotation used in the prediction layer of the improved YOLOv4 adopted in the present invention;

[0043] Figure 9 This is a structural diagram of the PSA module used in the feature fusion network of the improved YOLOv4 adopted in the present invention;

[0044] Figure 10 Schematic diagram of the structure of the SPC module in the PSA module used in the present invention;

[0045] Figure 11-a This is a schematic diagram of a jetty;

[0046] Figure 11-b It is a schematic diagram of the extended dock;

[0047] Figure 11-c This is a schematic diagram of a shore-side dock;

[0048] Figure 12-a This is a schematic diagram of the dock recognition results using the SSD algorithm on dataset 1 during the experiment;

[0049] Figure 12-bThis is a schematic diagram of the dock recognition results using the YOLOv3 algorithm on dataset 1 during the experiment;

[0050] Figure 12-c This is a schematic diagram of the dock recognition results using the YOLOv4 algorithm on dataset 1 during the experiment;

[0051] Figure 13-a This is a schematic diagram of the dock recognition results using the SSD algorithm on dataset 2 during the experiment;

[0052] Figure 13-b This is a schematic diagram of the dock recognition results using the YOLOv3 algorithm on dataset 2 during the experiment;

[0053] Figure 13-c This is a schematic diagram of the dock recognition results using the YOLOv4 algorithm on dataset 2 during the experiment;

[0054] Figure 14-a This is a schematic diagram of the ground truth of the dock labels on dataset 1 during the experiment;

[0055] Figure 14-b This is a schematic diagram of the dock recognition results using the YOLOv4-M algorithm on dataset 1 during the experiment;

[0056] Figure 14-c This is a schematic diagram of the dock recognition results on dataset 1 using the algorithm of the present invention during the experiment;

[0057] Figure 15-a This is a schematic diagram of the ground truth of the dock labels on dataset 2 during the experiment;

[0058] Figure 15-b This is a schematic diagram of the dock recognition results using the YOLOv4-M algorithm on dataset 2 during the experiment;

[0059] Figure 15-c This is a schematic diagram of the dock recognition results on dataset 2 using the algorithm of the present invention during the experiment;

[0060] Figure 16 This is a comparison chart of the loss function loss curves of different backbone networks during the experiment;

[0061] Figure 17 The figure is a schematic diagram of the detection results of the dock identification in a large-scale remote sensing image using the target detection method of the present invention. DETAILED DESCRIPTION

[0062] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0063] Based on the existing YOLOv4, the present invention adds a PSA attention module to the feature fusion network, uses the PSA attention module to obtain receptive fields of different scales, and extracts the attention weights of the channels to obtain the weights of the attention of the channels of different scales, thereby fusing the contextual information of different scales and further improving the detection accuracy of the network. At the same time, a multi-dimensional corner point coordinate detection frame is used to calibrate the dock target, which can effectively represent the true boundary of the dock target, solve the problem of arbitrary dock direction, further improve the positioning accuracy of the target, and achieve accurate detection of the dock target. Before introducing the present invention, the YOLOv4 algorithm is first introduced.

[0064] YOLOv4 is a single-stage target detection algorithm that integrates the most advanced target detection ideas in recent years, thus achieving a better balance between network detection accuracy and speed. YOLOv4 mainly includes the backbone network, feature fusion network and prediction layer. Figure 1 As shown in the figure, the input image size is 480×480 pixels, and the feature extraction is performed through the CSPDarknet53 backbone network. Then the image features are further enhanced using the Space Pyramid Pool (SPP) module and the Path Aggregation Network (PANet). Finally, the final detection result is obtained through the prediction layer.

[0065] YOLOv4 selects CSPDarkNet53 as the backbone network. CSPDarkNet53 combines YOLOv3's backbone network DarkNet53 with the Cross Stage Paritial Network (CSPNet) to better extract the feature information of the image. The structure of CSPDarkNet53 is as follows: Figure 2 As shown in Figure 2. The CBM module includes the convolution layer Conv, batch normalization (BN) and Mish activation function, as shown in Figure 2. Figure 3-a As shown. The residual connection (Res_unit) structure is similar to the residual structure of the ResNet network, as shown Figure 3-b , through skip connections, the input information can be directly transmitted to the subsequent layers, simplifying the difficulty of network learning. CSPn is a CSPNet introduced on each residual block of Darknet53, which consists of CBM and n Res_unit modules, such as Figure 4 CSPNet mainly solves the problem of high computational complexity through network structure design. The CSP module divides the input feature map into two parts. One part obtains the residual result through residual convolution; the other part is directly fused with the residual result across stages, thereby reducing the computational complexity while still ensuring its accuracy.

[0066] The feature fusion network includes SPP and PANet feature pyramid structures, where the SPP module is used to free the input of the convolutional neural network from fixed size restrictions, improve the network's receptive field, and effectively extract important contextual information. Figure 5 As shown in the figure, the SPP module uses pooling kernel sizes of 1×1, 5×5, 9×9, and 13×13 to perform a maximum pooling operation on the last feature layer of CSPDarkNet53, and then fuses feature maps of different scales (Concat) to improve the scale invariance of the image. PANet is a further improvement of the Feature Pyramid Network (FPN). By repeatedly extracting and fusing image features, it improves the network's multi-scale feature extraction capabilities. Figure 6 As shown in Figure 1, FPN transfers rich semantic information from high-level networks to low-level networks through upsampling, and achieves feature fusion through lateral connections between corresponding feature layers. This top-down network structure focuses more on transmitting high-level semantic information. PANet further optimizes the FPN structure using a bottom-up path aggregation network (PAN). Through downsampling, it transfers the position information extracted by the low-level network to the high-level network, allowing the network to simultaneously obtain the target's precise position information and high-level semantic information, significantly improving the accuracy of target detection.

[0067] The prediction layer uses a horizontal detection frame to achieve target positioning, that is, the center coordinates of the detection frame, the confidence of the detection frame, and the confidence information of the target category in the detection frame are used. That is, the input size is 480×480×3 pixels, and the prediction layer obtains feature maps with dimensions of 60×60×3×(4+1+n), 30×30×3×(4+1+n), and 15×15×3×(4+1+n), where 60, 30, and 15 represent the output size of each layer, and 3×(4+1+n) represents that YOLOv4 assigns 3 prior frames to the feature layer of each scale, and also contains 4 coordinate information of the detection frame, confidence information, and n categories. Among them, YOLOv4 uses the prior frame to roughly locate the target to be detected, and then calculates the actual center point of the bounding box (b x ,b y ) and width and height (b w ,b h ):

[0068]

[0069]

[0070]

[0071]

[0072] Where, is the Sigmoid function, (t x ,t y ) is the horizontal and vertical coordinates of the center point of the detection frame relative to the upper left corner of the prior frame (c x ,c y ) offset, (t w ,t h ) is the width and height of the detection frame relative to the prior frame (p w ,p h ) scaling ratio. The confidence score calculation formula for the detection box is as follows:

[0073]

[0074] Among them C ij is the jth prior box in the i-th grid; Pr(object) indicates whether the current prior box contains the target. When the target is contained, the value of Pr(object) is 1, and when the target is not contained, the value of Pr(object) is 0; It represents the intersection-over-union ratio of the predicted bounding box to the true bounding box, and its value is between 0 and 1. Finally, non-maximum suppression is used to eliminate redundant detection boxes to obtain the final detection result.

[0075] The YOLOv4 object detection algorithm significantly improves both accuracy and speed by utilizing a higher-performance backbone network and multi-scale feature fusion strategies. However, the YOLOv4 object detection algorithm detects horizontal target boxes with a default tilt angle of 0. While it achieves good detection results for targets in natural scene images, it struggles to accurately describe the size and angle of arbitrarily oriented dock targets in remote sensing images.

[0076] To solve the above problems, the present invention proposes a dock target detection method based on the improved YOLOv4 algorithm (Im-YOLOv4 for short) on the basis of the existing YOLOv4 algorithm. This method uses multi-dimensional corner point coordinate detection boxes in the prediction process, so that the detection results can more accurately describe the dock target. The loss function of YOLOv4 is improved to make it suitable for dock target detection. At the same time, Im-YOLOv4 introduces the Pyramid Split Attention module (PSA) to fully extract the multi-scale spatial information and cross-dimensional important features of the feature vector, thereby improving the accuracy of dock target detection.

[0077] Specifically, the Im-YOLOv4 network used in the present invention is as follows: Figure 7 As shown, it is the same as the existing YOLOv4 network structure, including the backbone network, feature fusion network and prediction layer. The backbone network is the same as the existing backbone network, which uses the CSPDarkNet53 network. Its structure is as follows Figure 2 The improvements of the present invention are mainly in the feature fusion network and prediction layer, wherein a PSA module is added to the feature fusion network and a multi-dimensional corner point coordinate detection frame is used in the prediction layer to replace the existing horizontal detection frame.

[0078] like Figure 9 As shown in the figure, the added PSA module of the present invention is a lightweight and effective attention module for improving the network's multi-scale feature extraction capabilities. The PSA attention module includes an SPC module, an SE module, and an output module. First, the Split and Concat (SPC) module is used to obtain receptive fields of different scales to better extract the multi-scale information of the image; secondly, the SE module is used to extract the channel attention weights to obtain the weights of the channel attention at different scales, allowing the PSA module to fuse contextual information at different scales; finally, each group of channel attention vectors is Softmax normalized to output a feature map with multi-scale feature information.

[0079] Among them, the SPC module uses the method of segmenting feature maps to enable the network to extract the spatial information of each channel feature vector and perform feature fusion by establishing local cross-channel connections to obtain multi-scale information of the image. First, the SPC module inputs the feature vectors Figure X The channel dimension is divided into S parts, and the feature vectors of each part are represented by [X0, X1, ..., X S-1 ]express( Figure 10 The structure diagram of the SPC module when S=4). Secondly, different sizes of convolution are used to extract spatial feature information from each feature vector. Since the continuous increase of the convolution kernel size will bring a huge amount of parameters, the feature vectors of each part after segmentation are further grouped and convolved. The convolution kernel size k and the group size G of each part are defined as:

[0080]

[0081] Each part is grouped and convolved to generate a feature map F i , and F i The multi-scale feature map F is obtained by stacking.

[0082] F i =Conv(k i ×k i ,G i )(Xi ) (7)

[0083] F=Cat([F0,F1,...,F S-1 ]) (8)

[0084] Where i = 0, 1, 2,…, S-1.

[0085] The horizontal detection box used in the existing prediction layer usually uses a four-dimensional vector (x, y, w, h) to mark the detection target, such as Figure 8-a As shown, x, y are the coordinates of the center point of the detection frame, and w, h are the width and height of the detection frame. This labeling method uses the least parameters to describe the information of the detection frame, but for targets with large aspect ratios, especially when the angle of the target is arbitrary, this labeling method cannot provide accurate boundary information of the target. However, the dock target in the remote sensing image is a directional target, and the horizontal frame labeling method cannot represent the direction information and boundary information of the dock, resulting in a large amount of redundant information in the detection frame. In order to calibrate a detection frame that is more suitable for the target for an object with directional information, the present invention uses multi-dimensional corner point coordinates (x1, y1, x2, y2, x3, y3, x4, y4) to calibrate the target. As shown Figure 8-b As shown in the figure, x1, y1, x2, y2, x3, y3, x4, and y4 represent the coordinates of the four corner points of the detection frame respectively. This annotation method can well frame the target with direction information, making the detection frame fit the target more closely.

[0086] The present invention uses multi-dimensional corner coordinates to calibrate the target. In order to adapt to this optimization, the present invention also optimizes the loss function when training the Im-YOLOv4 model. It mainly constructs a multi-dimensional corner coordinate loss function to optimize the detection results of the corner coordinate detection frame. The total network loss function L is composed of the detection frame regression loss L Pre , confidence loss L Conf And the classification loss L Cls It consists of three parts, which are represented by Smooth L1 Loss loss function, Binary Cross Entropy Loss and Cross Entropy Loss respectively, namely:

[0087] L=L Pre +L Conf +L Cls (9)

[0088] Among them, the detection box regression loss L Pre , confidence loss L Conf And the classification loss L Cls The calculation formulas are as follows:

[0089]

[0090]

[0091]

[0092] Where S 2 Indicates that the target detection network divides the original image into S×S grids, B represents the number of prior boxes in each grid, and the prior box refers to a rectangular box of different sizes predefined at each position in the feature map, including different aspect ratios, which is used to match the rectangular box of the real object. Pr(object) indicates whether the current prior box contains the target. When it contains the target, the value of Pr(object) is 1, and when there is no target, the value of Pr(object) is 0; x, y, x gt 、y gt Represent the detection box and the real box coordinates, c and c respectively gt Represent the predicted confidence and true confidence, p and p respectively gt Represent the probability of the predicted category and the true probability respectively. Confidence loss L Conf And the classification loss L Cls Use the existing loss function.

[0093] Experimental verification

[0094] To further demonstrate the effectiveness of the present invention, we first conducted experimental simulations. Two sets of experiments were conducted. The first set, based on horizontal bounding box detection, tested the currently popular single-stage object detection algorithms SSD, YOLOv3, and YOLOv4 to explore their performance in the dock object detection task. The second set of experiments focused on object detection algorithms based on multi-dimensional corner point coordinate bounding box detection. This improved upon the horizontal bounding box YOLOv4 algorithm to enable dock object detection with directional information in remote sensing imagery.

[0095] All network experiments were conducted in the same environment, with consistent training parameters. To quantitatively evaluate model performance, we selected Average Precision (AP) as the evaluation metric for the terminal detection results. AP measures the overall detection accuracy of the network and is defined as the integral of the curve of Precision (P) versus Recall (R) over a range of R from 0 to 1, as shown in the following formula:

[0096]

[0097]

[0098]

[0099] Where TP is the number of correctly detected terminals, FP is the number of incorrectly detected terminals, and FN is the number of missed terminals.

[0100] 1) Select experimental data

[0101] Piers are relatively small objects in remote sensing images, and their grayscale, texture and other features are relatively simple. In order to better construct a pier dataset for pier detection, it is necessary to fully analyze the image features of the pier. There are many types of piers, mainly divided into pier-type, extended-bank type, and shore-type. It is difficult to have a single method to effectively extract all piers. Among them, pier-type piers extend out of the coast at right angles or obtuse angles, such as Figure 11-a As shown in Figure 11, the front line of this type of terminal presents a large angle with the natural shoreline, which is more common in seaports with large cargo volumes. 11(b) The front loading and unloading platform of the extended shore terminal is connected to the rear shoreline through an approach bridge (or an approach embankment), and has more obvious structural features, such as Figure 11-b As shown in the figure, there are huge differences in the structural characteristics of the shore-type wharf, pier-type wharf and approach bridge-type wharf. Figure 11-c As shown, the front line of the wharf is generally parallel to the natural coastline, with no obvious boundary with the land. The surface texture is similar to strip features such as roads, and it is difficult to distinguish from other reinforced coastlines. Therefore, the extraction problem of the shore-side wharf was not considered during the research process of this invention.

[0102] This experiment constructed two types of dock datasets based on publicly available remote sensing datasets, designated as Dataset 1 and Dataset 2. The basic information of each dataset is shown in Table 1. Dataset 1 selects images of military port areas from Google Earth. After manual labeling, it constructs a dock target detection dataset. Data augmentation is used to expand the training data to obtain 812 training images. Dataset 2 is taken from the DOTA dataset [9]. Through image filtering and cropping, 1080 images containing civilian docks are obtained, and the dataset is randomly divided at a 1:1 ratio. The horizontal box-based target detection algorithm uses the minimum bounding rectangle of the sample annotations in the dataset to describe the target.

[0103] Table 1

[0104]

[0105] 2) Horizontal detection frame experiment

[0106] In order to explore the detection ability of mainstream horizontal box target detection algorithms for dock targets, this experiment selected SSD, YOLOv3 and YOLOv4 single-stage horizontal box detection algorithms based on datasets 1 and 2 respectively. The dock in dataset 1 is a military type. Figure 12-a 、12-b 12-c and 12-d are the detection results of each algorithm in dataset 1. Among them, the SSD algorithm cannot overcome the interference caused by the ship targets around the dock, resulting in false detection and missed detection. Figure 12-a As shown in the figure. Since the military dock has a certain length and width, the YOLOv3 detection frame may have incomplete target detection, as shown in the figure. Figure 12-b As shown. Figure 12-c As shown in the figure, the YOLOv4 algorithm can accurately detect the dock target, but the detection frame also includes the docked ship target, which interferes with the detection results.

[0107] Most of the docks in dataset 2 are civilian types, with characteristics such as slender structure, small size, and disorderly distribution. Figure 13-a 、 13-b and 3-c are the detection results of each algorithm in dataset 2. Figure 13-a As shown in Figure 2, the SSD algorithm has a poor detection effect in dataset 2. When the dock is small and there are interference factors such as vegetation around it, it is easy to cause missed detection. In addition, the buildings around the dock will also interfere with the detection of dock targets, resulting in false detection. Figure 13-b The results of the YOLOv3 algorithm for dock detection in Data 2 are shown. Compared with the SSD algorithm, the detection results are significantly improved, but there are still some missed detections. Figure 13-c As shown in the figure, the YOLOv4 algorithm can detect dock targets more accurately. However, when the docks are densely distributed and slender, the detection frames of different dock targets will overlap, making it impossible to accurately locate the dock targets.

[0108] The accuracy evaluation results for the test set are shown in Table 2. The SSD algorithm, using VGG16 as its feature extraction network, achieved AP values ​​of 49.08% and 52.89% for the two datasets, respectively, the lowest among all compared networks, demonstrating that the SSD algorithm performs well in the dock object detection task. The YOLOv3 algorithm, using Darknet53 as its feature extraction network, achieved AP values ​​of 75.39% and 82.61% for the two datasets, respectively, significantly outperforming the SSD algorithm in detection accuracy. The YOLOv4 algorithm, a further improvement on the YOLOv3 algorithm, uses CSPDarknet53 as its feature extraction network. Its AP values ​​reached 81.84% and 84.78% for the two dock object detection datasets, respectively, representing improvements of 6.45% and 2.17% over the YOLOv3 algorithm. This demonstrates that the YOLOv4 horizontal bounding box detection algorithm achieves optimal performance in the dock object detection task, laying the foundation for the subsequent implementation of dock object detection in any orientation.

[0109] Table 2

[0110]

[0111]

[0112] 3) Multi-dimensional focus coordinate detection frame experiment

[0113] The previous experiment demonstrated that the YOLOv4 horizontal bounding box detection algorithm performs better in the dock object detection task. Based on the YOLOv4 algorithm, the following introduces a multi-dimensional corner coordinate detection method to detect dock objects with directional information in remote sensing imagery. This algorithm, denoted as the YOLOv4 multi-dimensional corner coordinate detection algorithm (YOLOv4-M), is further improved to address the specific characteristics of dock object detection, making it more suitable for detecting dock objects in remote sensing imagery. Dataset 1 contains a large number of military dock objects. Figure 14-a Visualization result of marking the dock in the image, Figure 14-b YOLOv4-M uses a multi-dimensional corner point coordinate detection frame to make up for the shortcomings of the horizontal detection frame. However, the target features of ships docked at the military dock are similar to those of the dock and are prone to shadows in the dock area, covering the dock features, which greatly interferes with the algorithm detection results, resulting in missed detections and inaccurate positioning in the YOLOv4-M detection results. Figure 14-c The detection results of the invented algorithm have been significantly improved compared with those of YOLOv4-M, effectively overcoming the interference of factors such as ships and realizing the accurate extraction of dock targets in military port images.

[0114] Next, the YOLOv4 multi-dimensional corner coordinate detection algorithm and the algorithm of the present invention are simulated in data set 2. Figure 15-a Visualization result of marking the dock in the image, Figure 15-b This is the detection result of YOLOv4-M. It can be seen that the YOLOv4-M multi-dimensional corner point coordinate detection frame can accurately reflect the size information of the dock target, and the positioning of the dock is more accurate, which is more in line with actual needs than the horizontal frame. However, the sizes of civil docks vary greatly, and YOLOv4-M misses the dock targets with slender structures and large length-to-width ratios. In addition, YOLOv4-M mistakenly detects interference factors such as yachts and vegetation around the dock as docks, which affects the dock detection results. The detection results of the algorithm of the present invention in data set 2 are shown in Figure 2. Figure 15-c As shown in the figure, the algorithm introduces the PSA module on the basis of YOLOv4-M, which enhances the multi-scale feature extraction capability of YOLOv4-M, reduces the missed detection rate of the network for slender dock targets, and can obtain accurate dock detection results even in civil port areas with complex backgrounds.

[0115] The test set accuracy evaluation results for the two datasets are shown in Table 3. To explore the effectiveness of the PSA attention module in the dock object detection task, we introduced the SE attention module and the Convolutional Block Attention Module (CBAM) on top of YOLOv4-M for comparative experiments, maintaining consistent parameters throughout the experiments. YOLOv4-M achieved AP values ​​of 59.19% and 73.04% on the two datasets, respectively. After introducing the PSA attention module to YOLOv4-M (the proposed algorithm), the AP values ​​for the two datasets improved by 7.58% and 5.78%, respectively, demonstrating that the PSA module can effectively improve dock object detection accuracy. Furthermore, after introducing the SE module to YOLOv4-M, the AP values ​​for the two datasets improved by 6.97% and 1.43%, respectively, compared to YOLOv4-M, and decreased by 0.61% and 4.35%, respectively, compared to the proposed algorithm. This demonstrates that the SE module also contributes to dock object detection, but its improvement is not as significant as that of the PSA module. The introduction of the CBAM module to YOLOv4-M improves its AP by 4.28% on Dataset 1, while decreasing its AP by 0.15% on Dataset 2. This demonstrates that the CBAM module improves the algorithm's detection capabilities for military docks, but is detrimental to civilian dock detection. These experimental results demonstrate that the PSA attention module effectively improves dock target detection accuracy and demonstrates that the PSA attention module, by extracting multi-scale features from images, is more suitable for dock target detection than the SE and CBAM attention modules.

[0116] Table 3

[0117]

[0118] 4) Backbone network ablation experiment

[0119] In addition, in order to explore the impact of the feature extraction backbone network on the detection performance of the target detection algorithm, the algorithm of the present invention is used to conduct comparative experiments on the MobileNet v2, MobileNet v3 and CSPDarknet53 backbone networks on Dataset 2. The loss function of different backbone networks on Dataset 2 changes with the number of iterations as shown in the following figure. Figure 16As shown in the figure, in the initial stage of model training, the loss values ​​of each curve remain at a high level. As the number of iterations increases, the loss values ​​continue to slowly oscillate and decrease. When the iteration reaches 250, the loss values ​​of each algorithm tend to stabilize. Among them, the loss values ​​using the MobileNet v2 and MobileNet v3 backbone networks ultimately stabilize at 15.02 and 16.16, respectively. The loss value of the algorithm of the present invention using the CSPDarknet53 backbone network ultimately stabilizes at 11.44, which is the lowest compared to the loss values ​​using other networks.

[0120] Table 4 shows the evaluation results of models using different backbone networks on Dataset 2. Frames Per Second (FPS) is used to evaluate the detection speed of each model. FPS is the number of images a model can detect in one second; higher FPS values ​​indicate faster detection speeds. As shown in Table 4, the YOLOv4-M algorithm, using the CSPDarknet53 backbone network, achieved an AP of 73.04% on Dataset 2, with a detection rate of 29.18 frames per second. When using the MobileNet v2 backbone network, the AP value on Dataset 2 was 71.86%, and the detection rate was 31.78 frames per second. When using the MobileNet v3 backbone network, the AP value was 69.10%, and the detection rate was 32.35 frames per second. When the present invention used CSPDarknet53 as the backbone network, the detection rate was 28.49 frames per second, which was slightly lower than the detection rate using Mobilnetv2 and Mobilnetv3 as the feature extraction backbone networks. However, the AP value on both datasets reached 78.80%, reaching the optimal level. In addition, compared with YOLOv4-M, which also uses CSPDarknet53 as the backbone network, the present invention's FPS was only reduced by 0.69 frames per second, demonstrating that PSA, as a lightweight attention module, achieves an effective balance between algorithm detection accuracy and speed.

[0121] Table 4

[0122]

[0123] 5) Application example of remote sensing image dock target detection

[0124] This example uses the Norfolk Harbor area in the United States to verify the dock detection performance of a large-scale Google Earth image. The image area has a pixel size of 5,559 × 6,447 pixels and a spatial resolution of 1 meter. There are 17 docks in the image.

[0125] The dock detection results obtained by applying the identification method of the present invention are as follows: Figure 17As shown in the figure, 16 dock targets were correctly detected, 3 were falsely detected (indicated by white wireframes), and 1 was missed (indicated by black wireframes), resulting in a precision of 84.21% and a recall of 94.12%. The falsely detected targets were at the port's land-sea boundary or artificial structures on land that closely resembled dock features. The missed dock structures were too slender, making them difficult for the algorithm to detect, resulting in missed detections.

[0126] In summary, based on the YOLOv4 horizontal box detection algorithm, this paper designs an Im-YOLOv4 algorithm that can detect dock targets in any direction. This algorithm uses a multi-dimensional corner point coordinate detection box to solve the problem of arbitrary dock directions, and introduces the PSA attention mechanism to enhance the network's ability to extract dock targets. The experiment selected two remote sensing image dock target detection datasets to verify the feasibility of the current mainstream horizontal box detection algorithm in the dock target detection task. In addition, an ablation experiment was conducted to analyze the impact of the attention mechanism and feature extraction backbone network on dock detection performance. The results show that the present invention can achieve accurate extraction of dock targets in remote sensing images and is also practical for large-scale images.

Claims

1. A remote sensing image dock target detection method, characterized in that: The method comprises the following steps: 1) Obtain the remote sensing image to be detected; 2) inputting the remote sensing image to be detected into the target detection model to output the dock target detection result; The target detection model includes a backbone network, a feature fusion network and a prediction layer; The backbone network is used to extract features from the input remote sensing image to obtain feature maps of different scales; The feature fusion network includes an SPP module, a PSA attention module and a feature pyramid module. The SPP module is used to perform a maximum pooling operation on the features output by the backbone network and fuse feature maps of different scales. The PSA attention module is used to segment the output results of the SPP module and extract scale features and channel attention weights to obtain feature maps with multi-scale features and attention weights. The feature pyramid module is used to repeatedly extract and fuse the feature maps with multi-scale features and attention weights to obtain features with precise target location information and high-level semantic information. The prediction layer locates the target to be detected by establishing a multi-dimensional corner point coordinate detection frame and obtaining features with precise target location information and high-level semantic information. The multi-dimensional corner point coordinates include the coordinates of the four corner points of the detection frame.

2. The remote sensing image dock target detection method according to claim 1, characterized in that: The PSA attention module includes an SPC module, an SE module and an output module; the SPC module is used to divide the feature map output by the SPP module into several parts in the channel dimension, and perform feature extraction of different scales on each part to obtain the feature vectors of each part and generate the feature map of the corresponding channel; the SE module is used to extract the corresponding channel attention vector from the feature map of each channel, and re-characterize the channel attention vector to obtain the corresponding attention weight after multi-scale channel interaction; The output module is used to perform weighted fusion processing on the feature maps of the corresponding channels according to the obtained attention weights to obtain feature maps with multi-scale features and attention weights.

3. The remote sensing image dock target detection method according to claim 2, characterized in that: The SPC module uses group convolution to extract features of different scales for each part. The convolution kernel of each group is related to the size of the group. The relationship between the convolution kernel size k and the group is: Among them, G is the size of the group and k is the size of the convolution kernel.

4. The remote sensing image dock target detection method according to claim 1, characterized in that: The multi-dimensional corner coordinate loss function used by the target detection model during training is: L=L Pre +L Conf +L Cls S 2 Indicates the number of grids into which the input remote sensing image is divided, B indicates the number of prior frames in each grid, and Pr(object) indicates whether the current prior frame contains the target. When the target is contained, the value of Pr(object) is 1, and when there is no target, the value of Pr(object) is 0; x, y, x gt 、y gt Represent the detection box and the real box coordinates, c and c respectively gt Represent the predicted confidence and true confidence, p and p respectively gt denote the probability of the predicted category and the true probability, respectively.

5. The remote sensing image dock target detection method according to claim 1, characterized in that: The backbone network adopts CSPDarknet53 network.

6. The remote sensing image dock target detection method according to claim 4, characterized in that: The prediction layers include 3, each of which contains 8 coordinate information, 1 bounding box confidence and 1 category confidence.

7. The method for detecting dock targets using remote sensing images according to claim 6, wherein: The calculation formula used for the border confidence is: Among them C ij is the jth prior box in the i-th grid; Pr(object) indicates whether the current prior box contains the target. When the target is contained, the value of Pr(object) is 1, and when the target is not contained, the value of Pr(object) is 0; It represents the intersection-over-union ratio of the predicted bounding box to the true bounding box, between 0 and 1.

Images