A SAR image ship iceberg detection and classification method based on a deep neural network

Abstract

The application relates to a SAR image ship iceberg detection and classification method based on a deep neural network, which is based on each sea surface area SAR image sample containing an iceberg target label and a ship target label, combines a target detection and classification deep neural network specifically constructed by including an image feature extraction network, a feature fusion network and a regression classifier, and obtains a target detection and classification model taking the sea surface area SAR image sample as input and taking the iceberg target label, the ship target label or no iceberg and ship target label in the sea surface area SAR image sample as output through network training, so that the detection and classification of the ship and iceberg targets can be accurately realized. In the whole design, the network training time is short, the convergence speed is fast, the accuracy is high, and the offshore situation awareness capability can be effectively improved.

Description

TECHNICAL FIELD

[0001] The application relates to a SAR image ship iceberg detection and classification method based on a deep neural network and belongs to the technical field of remote sensing image target detection and classification. BACKGROUND

[0002] Synthetic Aperture Radar (SAR) is a kind of high-resolution imaging radar, which has the imaging characteristics of all-weather, all-day, and being not affected by weather, can effectively identify disguises and penetrate coverings, and can long-time, dynamically, and seamlessly observe land and sea in real time. Among them, ship iceberg image detection and classification is one of the key problems of SAR image interpretation technology, and has a wide application prospect in various fields, especially in marine transportation safety, marine fishery management, illegal immigration, search and rescue, sunken ships and environment (oil leakage or pollution), and military monitoring, so it is of great significance to study SAR image ship iceberg detection and classification.

[0003] Traditional SAR image target detection and classification methods rely on artificial design features and are easily disturbed by complex backgrounds, and have the characteristics of low recognition accuracy, low recognition efficiency, and poor generalization ability. Based on deep learning, the method can learn parameters and automatically extract features, and is free from the dependence on artificial design features and modeling. Compared with traditional methods, the method has the advantages of strong anti-interference and high detection and classification accuracy, and has great development potential in the field of SAR image marine target detection and classification with complex and variable scenes and variable target poses.

[0004] In recent years, with the rise of artificial intelligence technology, for the field of remote sensing image target detection and classification, Faster R-CNN (Fast Region Convolutional Neural Network), YOLO (One Network) and Resnet (Residual Network) algorithms based on deep learning have been applied to SAR image target detection and classification, and have achieved excellent performance in the field of SAR image ship target detection and classification. However, the existing research methods have the disadvantages of long network model training time, low classification accuracy, and few applications in the field of SAR image ship iceberg target detection and classification. Therefore, for the field of SAR image ship iceberg target detection and classification, how to propose a SAR image ship iceberg target detection and classification method with short training time, fast convergence speed and high accuracy is a technical problem to be solved. SUMMARY

[0005] The technical problem solved by the present application is to provide a SAR image ship iceberg detection classification method based on a deep neural network, based on the application of a SAR image, and combined with the training and verification under the design network structure, to realize accurate detection and classification of SAR image ship icebergs, and to improve the sea situation awareness capability.

[0006] The present application adopts the following technical solutions to solve the above technical problems: the present application designs a SAR image ship iceberg detection classification method based on a deep neural network, and obtains a target detection classification model for identifying ships and icebergs on the sea surface according to the following steps A to step C; and then applies the target detection classification model to the target sea surface area to realize detection and classification of ships and icebergs.

[0007] Step A. Construct a predetermined number of sea surface area SAR image samples containing iceberg target labels and ship target labels, and then proceed to step B.

[0008] Step B. According to the image feature extraction network, the feature fusion network, and the regression classifier, construct a target detection classification deep neural network, and then proceed to step C.

[0009] Step C. Based on each sea surface area SAR image sample, input the sea surface area SAR image sample, and output the iceberg target label, the ship target label, or the no-iceberg and ship target label in the sea surface area SAR image sample, train the target detection classification deep neural network, and obtain the target detection classification model.

[0010] As a preferred technical solution of the present application, the following step i is further included to realize precision evaluation of the target detection classification model.

[0011] Step i. According to the following formula, obtain the F1 score corresponding to the target detection classification model, which is used to evaluate the precision of the target detection classification model.

[0012]

[0013]

[0014]

[0015] Wherein, P is the precision rate, i.e. the proportion of true examples in the samples predicted as positive examples, R is the recall rate, i.e. the proportion of true examples predicted as positive examples in all positive examples; TP is the true example, i.e. the prediction is correct, and the sample is positive; FP is the false positive example, i.e. the prediction is wrong, and the sample is predicted as positive, but the sample is actually negative; FN is the false negative example, i.e. the prediction is wrong, and the sample is predicted as negative, but the sample is actually positive.

[0016] As a preferred technical solution of the present application: the step A comprises steps A1 to A3.

[0017] Step A1. Based on different bands of the Sentinel 1 satellite, the horizontal polarization SAR image of the sea surface area containing a preset first number of iceberg target labels is captured by the Sentinel 1A satellite, and the horizontal polarization SAR image and the horizontal vertical cross-polarization SAR image of the sea surface area containing a preset second number of iceberg target labels are captured by the Sentinel 1B satellite;

[0018] At the same time, based on different bands of the Sentinel 1 satellite, the vertical horizontal cross-polarization SAR image of the sea surface area containing a preset third number of ship target labels is captured by the Sentinel 1A satellite, and the vertical horizontal cross-polarization SAR image and the vertical polarization SAR image of the sea surface area containing a preset fourth number of ship target labels are captured by the Sentinel 1B satellite; then step A2 is entered;

[0019] Step A2. The horizontal polarization SAR image containing the iceberg target label and the vertical horizontal cross-polarization SAR image containing the ship target label are fused to obtain an R channel SAR image;

[0020] The horizontal vertical cross-polarization SAR image containing the iceberg target label and the vertical polarization SAR image containing the ship target label are fused to obtain a G channel SAR image;

[0021] The horizontal polarization SAR image containing the iceberg target label and the vertical polarization SAR image containing the ship target label are fused to obtain a B channel SAR image;

[0022] Further, an RGB channel SAR image is obtained, and then step A3 is entered;

[0023] Step A3. The RGB channel SAR image is divided into a grid according to a preset grid size, and each RGB channel SAR image block is obtained, that is, a preset number of sea surface area SAR image samples containing the iceberg target label and the ship target label are obtained.

[0024] As a preferred technical solution of the present application: the target detection classification model is applied to realize detection and classification of ships and icebergs for the target sea surface area as follows:

[0025] Operation: First, based on different bands of the Sentinel 1 satellite, the horizontal polarization SAR image and the horizontal vertical cross-polarization SAR image of the target sea surface area are captured by the Sentinel 1A satellite; at the same time, based on different bands of the Sentinel 1 satellite, the vertical horizontal cross-polarization SAR image and the vertical polarization SAR image of the target sea surface area are captured by the Sentinel 1B satellite;

[0026] Then the horizontal polarization SAR image is fused with the vertical horizontal cross-polarization SAR image to obtain an R channel SAR image; the horizontal vertical cross-polarization SAR image is fused with the vertical polarization SAR image to obtain a G channel SAR image; the horizontal polarization SAR image is fused with the vertical polarization SAR image to obtain a B channel SAR image; and then an RGB channel SAR image corresponding to the target sea surface region is obtained.

[0027] Finally, a target detection classification model is applied to the RGB channel SAR image corresponding to the target sea surface region to realize detection and classification of ships and icebergs.

[0028] As a preferred technical solution of the present application: in step B, a target detection classification deep neural network is constructed according to an image feature extraction network Darknet53, a feature fusion network FPN, and a regression classifier YoloHead; wherein an input end of the image feature extraction network Darknet53 constitutes an input end of the target detection classification deep neural network, and the image feature extraction network Darknet53 is connected in series from its input end to an Input(640, 640, 3) layer, a Conv2BLR(32, 2, 1, 1) layer, a Conv2(64, 3, 2, 1) layer, a Resb1 residual block, a Conv2(128, 3, 2, 1) layer, a Resb2 residual group, a Conv2(256, 3, 2, 1) layer, a Resb3 residual group, a Conv2(512, 3, 2, 1), a Resb4 residual group, a Conv2(1024, 3, 2, 1) layer, and a Resb5 residual group;

[0029] The Resb1 residual block is connected in series from its input end to a Conv2BLR(32, 1, 1, 0) layer and a Conv2BLR(32, 2, 1, 1) layer;

[0030] The Resb2 residual group is connected in series from its input end to two Resb2 residual blocks with the same structure, and the Resb2 residual block is connected in series from its input end to a Conv2BLR(64, 1, 1, 0) layer and a Conv2BLR(128, 3, 1, 1) layer;

[0031] The Resb3 residual group is connected in series from its input end to eight Resb3 residual blocks with the same structure, and the Resb3 residual block is connected in series from its input end to a Conv2BLR(128, 1, 1, 0) layer and a Conv2BLR(256, 3, 1, 1) layer;

[0032] The Resb4 residual group is sequentially connected in series with eight Resb4 residual blocks of the same structure from its input end, and the Resb4 residual block is sequentially connected in series with a Conv2BLR(256, 1, 1, 0) layer and a Conv2BLR(512, 3, 1, 1) layer from its input end;

[0033] The Resb5 residual group is sequentially connected in series with four Resb4 residual blocks of the same structure from its input end, and the Resb5 residual block is sequentially connected in series with a Conv2BLR(512, 1, 1, 0) layer and a Conv2BLR(1024, 3, 1, 1) layer from its input end;

[0034] The Input(m, m, c) is an input layer, the Conv2(c, k, s, p) layer is a two-dimensional convolution layer, the Conv2BLR(c, k, s, p) layer is a Conv2 layer plus a batch normalization layer plus a linear activation function layer with a leakage correction, m represents the pixel size of image data, c is the number of channels, k represents the kernel size, s represents the step number, and p is the padding number;

[0035] The output end of the last Resb3 residual block in the Resb3 residual group, the output end of the last Resb4 residual block in the Resb4 residual group, and the output end of the last Resb5 residual block in the Resb5 residual group constitute the respective output ends of the image feature extraction network Darknet53, the respective output ends of the image feature extraction network Darknet53 are connected to the input ends of the feature fusion network FPN, the output end of the feature fusion network FPN is connected to the input end of the regression classifier YoloHead, and the output end of the regression classifier YoloHead constitutes the output end of the target detection and classification deep neural network.

[0036] As a preferred technical solution of the present application: the feature fusion network FPN includes a first Conv2BLR group, a Conv2BLR(256, 1, 1, 1) layer, an UpSample2D(2) layer, a second Conv2BLR group, a Conv2BLR(128, 1, 1, 1) layer, an UpSample2D(2) layer, and a third Conv2BLR group sequentially connected in series according to the data flow direction;

[0037] The first Conv2BLR group comprises, in sequence, a Conv2BLR (512, 1, 1, 1) layer, a Conv2BLR (1024, 3, 1, 1) layer, a Conv2BLR (512, 1, 1, 1) layer, a Conv2BLR (1024, 3, 1, 1) layer, and a Conv2BLR (512, 1, 1, 1) layer in the data flow direction; the second Conv2BLR group comprises, in sequence, a Conv2BLR (256, 1, 1, 1) layer, a Conv2BLR (512, 3, 1, 1) layer, a Conv2BLR (256, 1, 1, 1) layer, a Conv2BLR (512, 3, 1, 1) layer, and a Conv2BLR (256, 1, 1, 1) layer in the data flow direction; and the third Conv2BLR group comprises, in sequence, a Conv2BLR (128, 1, 1, 1) layer, a Conv2BLR (256, 3, 1, 1) layer, a Conv2BLR (128, 1, 1, 1) layer, a Conv2BLR (256, 3, 1, 1) layer, and a Conv2BLR (128, 3, 1, 1) layer in the data flow direction.

[0038] The input end of the first Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group, the input end of the first Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group, and the input end of the first Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group constitute the respective input ends of the feature fusion network FPN; and the output end of the last Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group, the output end of the last Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group, and the output end of the last Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group constitute the respective output ends of the feature fusion network FPN.

[0039] The input end of the first Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group in the feature fusion network FPN is connected to the output end of the last Resb5 residual block in the Resb5 residual group in the image feature extraction network Darknet53; the input end of the first Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group in the feature fusion network FPN is connected to the output end of the last Resb4 residual block in the Resb4 residual group in the image feature extraction network Darknet53; and the input end of the first Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group in the feature fusion network FPN is connected to the output end of the last Resb3 residual block in the Resb3 residual group in the image feature extraction network Darknet53.

[0040] The output ends of the feature fusion network FPN are connected to the input ends of the regression classifier YoloHead.

[0041] As a preferred technical scheme of the present application, the regression classifier YoloHead comprises a Yolohead1 group, a Yolohead2 group and a Yolohead3 group, wherein the Yolohead1 group comprises, in sequence from the input end, a Conv2BLR (256, 3, 1, 1) layer, a Conv2 (255, 1, 1, 1) layer and an Output (80, 80, 255) layer; the Yolohead2 group comprises, in sequence from the input end, a Conv2BLR (512, 3, 1, 1) layer, a Conv2 (255, 1, 1, 1) layer and an Output (40, 40, 255) layer; and the Yolohead3 group comprises, in sequence from the input end, a Conv2BLR (1024, 3, 1, 1) layer, a Conv2 (255, 1, 1, 1) layer and an Output (20, 20, 255) layer; and Output (m, m, c) is an output layer.

[0042] The input end of the Yolohead1 group, the input end of the Yolohead2 group and the input end of the Yolohead3 group constitute the input ends of the regression classifier YoloHead; the output end of the last Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group in the feature fusion network FPN is connected to the input end of the Yolohead3 group; the output end of the last Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group in the feature fusion network FPN is connected to the input end of the Yolohead2 group; and the output end of the last Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group in the feature fusion network FPN is connected to the input end of the Yolohead1 group.

[0043] The output end of the Output (80, 80, 255) layer in the Yolohead1 group, the output end of the Output (40, 40, 255) layer in the Yolohead2 group, and the output end of the Output (20, 20, 255) layer in the Yolohead3 group constitute the respective output ends of the regression classifier YoloHead.

[0044] As a preferred technical solution of the present application: in the process of step C, the K-means clustering algorithm is used for iterative solution, and the Adam gradient optimization algorithm is used for network training optimization.

[0045] As a preferred technical solution of the present application: in the process of step C, the following binary cross entropy is used as a loss function for network training optimization;

[0046]

[0047] In the formula, Loss is the loss value of each round of training, log represents the logarithm with e as the base, * represents summation, N is the number of training data set samples, p i is the probability that the i-th sample of the training data set is an iceberg, y i is the labeled value of the i-th sample of the training data set, i=1,2,...,N.

[0048] Compared with the prior art, the SAR image ship iceberg detection and classification method based on the deep neural network has the following technical effects:

[0049] (1) The SAR image ship iceberg detection and classification method based on the deep neural network is designed based on each sea surface area SAR image sample containing an iceberg target label and a ship target label, and a target detection and classification deep neural network specifically constructed by combining an image feature extraction network, a feature fusion network, and a regression classifier. Through network training, a target detection and classification model is obtained, which takes the sea surface area SAR image sample as input and outputs the iceberg target label, the ship target label, or no iceberg and ship target label in the sea surface area SAR image sample. The target detection and classification model is used to accurately realize the detection and classification of the ship and iceberg targets. In the entire design, the network training time is short, the convergence speed is fast, and the accuracy is high, which can effectively improve the maritime situation awareness capability. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is the processing flowchart of the SAR image ship iceberg detection and classification method based on the deep neural network designed by the present application;

[0051] Figure 2 Structure diagram of deep neural network for target detection classification;

[0052] Figure 3 Target detection classification result diagram of only ship (white circle) of the present application;

[0053] Figure 4 Target detection classification result diagram of only iceberg (black circle) of the present application;

[0054] Figure 5 Target detection classification result diagram of ship (white circle) and iceberg (black circle) of the present application. DETAILED DESCRIPTION

[0055] The specific embodiments of the present application will be further described in detail below in combination with the drawings of the specification.

[0056] The present application designs a SAR image ship and iceberg detection classification method based on deep neural network. In practical application, as shown in the figure, the specific design is as follows: Figure 1 Steps A to C to obtain a target detection classification model for identifying ships and icebergs on the sea surface.

[0057] Step A. Perform the following steps A1 to A3 to construct a predetermined number of each sea area SAR image sample containing an iceberg target label and a ship target label, and then proceed to step B.

[0058] Step A1. Based on different bands of Sentinel-1 satellite, apply Sentinel-1A satellite to capture horizontal polarization SAR images of sea areas containing a predetermined first number of iceberg target labels, and apply Sentinel-1B satellite to capture horizontal polarization SAR images and horizontal vertical cross-polarization SAR images of sea areas containing a predetermined second number of iceberg target labels.

[0059] At the same time, based on different bands of Sentinel-1 satellite, apply Sentinel-1A satellite to capture vertical horizontal cross-polarization SAR images of sea areas containing a predetermined third number of ship target labels, and apply Sentinel-1B satellite to capture vertical horizontal cross-polarization SAR images and vertical polarization SAR images of sea areas containing a predetermined fourth number of ship target labels; then proceed to step A2.

[0060] In the above-mentioned influence capture process based on different bands of Sentinel-1 satellite, such as selecting the sea surface of Greenland, designing and applying Sentinel-1A satellite to capture the horizontal polarization SAR image of the sea surface area containing 1148 iceberg target labels, and applying Sentinel-1B satellite to capture the horizontal polarization SAR image and the horizontal vertical cross-polarization SAR image of the sea surface area containing 607 iceberg target labels;

[0061] At the same time, selecting the sea surface of Denmark, applying Sentinel-1A satellite to capture the vertical horizontal cross-polarization SAR image of the sea surface area containing 196 ship target labels, and applying Sentinel-1B satellite to capture the vertical horizontal cross-polarization SAR image and the vertical polarization SAR image of the sea surface area containing 322 ship target labels.

[0062] Step A2. Fuse each horizontal polarization SAR image containing an iceberg target label with each vertical horizontal cross-polarization SAR image containing a ship target label to obtain an R-channel SAR image;

[0063] Fuse the horizontal vertical cross-polarization SAR image containing the iceberg target label with the vertical polarization SAR image containing the ship target label to obtain a G-channel SAR image;

[0064] Fuse each horizontal polarization SAR image containing an iceberg target label with the vertical polarization SAR image containing a ship target label to obtain a B-channel SAR image;

[0065] Further obtain an RGB-channel SAR image, and then proceed to step A3.

[0066] Step A3. Grid division is performed on the RGB-channel SAR image according to a preset grid size, and each RGB-channel SAR image block is obtained, that is, a preset number of sea surface area SAR image samples containing iceberg target labels and ship target labels are obtained; in actual application, such as grid division is performed according to a preset grid size such as 640*640 pixels, and 1609 sea surface area SAR image samples containing iceberg target labels and ship target labels are obtained.

[0067] Step B. According to the image feature extraction network Darknet53, the feature fusion network FPN, and the regression classifier Yolo Head, a target detection and classification deep neural network is constructed according to the following structure, and then proceeds to step C.

[0068] As Figure 2As shown, the input end of the image feature extraction network Darknet53 constitutes the input end of the target detection classification deep neural network, and the image feature extraction network Darknet53 is sequentially connected with an Input(640, 640, 3) layer, a Conv2BLR(32, 2, 1, 1) layer, a Conv2(64, 3, 2, 1) layer, a Resb1 residual block, a Conv2(128, 3, 2, 1) layer, a Resb2 residual group, a Conv2(256, 3, 2, 1) layer, a Resb3 residual group, a Conv2(512, 3, 2, 1), a Resb4 residual group, a Conv2(1024, 3, 2, 1) layer, and a Resb5 residual group from the input end thereof.

[0069] As to each Resb residual block, the specific design is as follows:

[0070] The Resb1 residual block is sequentially connected with a Conv2BLR(32, 1, 1, 0) layer and a Conv2BLR(32, 2, 1, 1) layer from the input end thereof.

[0071] The Resb2 residual group is sequentially connected with two Resb2 residual blocks of the same structure from the input end thereof, and the Resb2 residual block is sequentially connected with a Conv2BLR(64, 1, 1, 0) layer and a Conv2BLR(128, 3, 1, 1) layer from the input end thereof.

[0072] The Resb3 residual group is sequentially connected with eight Resb3 residual blocks of the same structure from the input end thereof, and the Resb3 residual block is sequentially connected with a Conv2BLR(128, 1, 1, 0) layer and a Conv2BLR(256, 3, 1, 1) layer from the input end thereof.

[0073] The Resb4 residual group is sequentially connected with eight Resb4 residual blocks of the same structure from the input end thereof, and the Resb4 residual block is sequentially connected with a Conv2BLR(256, 1, 1, 0) layer and a Conv2BLR(512, 3, 1, 1) layer from the input end thereof.

[0074] The Resb5 residual group is sequentially connected with four Resb4 residual blocks of the same structure from the input end thereof, and the Resb5 residual block is sequentially connected with a Conv2BLR(512, 1, 1, 0) layer and a Conv2BLR(1024, 3, 1, 1) layer from the input end thereof.

[0075] Wherein, Input(m, m, c) is an input layer, Conv2(c, k, s, p) layer is a two-dimensional convolution layer, Conv2BLR(c, k, s, p) layer is a Conv2 layer plus a batch normalization layer plus a linear activation function layer with a leaky correction, m represents the pixel size of image data, c is the number of channels, k represents the kernel size, s represents the step number, and p is the padding number.

[0076] Further as shown in the figure, Figure 2 The output end of the last Resb3 residual block in the Resb3 residual group, the output end of the last Resb4 residual block in the Resb4 residual group, and the output end of the last Resb5 residual block in the Resb5 residual group constitute the respective output ends of the image feature extraction network Darknet53, the respective output ends of the image feature extraction network Darknet53 are connected to the input ends of the feature fusion network FPN, the output end of the feature fusion network FPN is connected to the input end of the regression classifier YoloHead, and the output end of the regression classifier YoloHead constitutes the output end of the target detection and classification deep neural network.

[0077] Regarding the feature fusion network FPN, the application specifically designs the feature fusion network FPN to include, in the data flow direction, a first Conv2BLR group, a Conv2BLR(256, 1, 1, 1) layer, an UpSample2D(2) layer, a second Conv2BLR group, a Conv2BLR(128, 1, 1, 1) layer, an UpSample2D(2) layer, and a third Conv2BLR group connected in sequence; UpSample2D(n) is a two-dimensional up-sampling layer, and n is the up-sampling multiple.

[0078] The first Conv2BLR group comprises, in sequence, a Conv2BLR (512, 1, 1, 1) layer, a Conv2BLR (1024, 3, 1, 1) layer, a Conv2BLR (512, 1, 1, 1) layer, a Conv2BLR (1024, 3, 1, 1) layer, and a Conv2BLR (512, 1, 1, 1) layer in the data flow direction; the second Conv2BLR group comprises, in sequence, a Conv2BLR (256, 1, 1, 1) layer, a Conv2BLR (512, 3, 1, 1) layer, a Conv2BLR (256, 1, 1, 1) layer, a Conv2BLR (512, 3, 1, 1) layer, and a Conv2BLR (256, 1, 1, 1) layer in the data flow direction; and the third Conv2BLR group comprises, in sequence, a Conv2BLR (128, 1, 1, 1) layer, a Conv2BLR (256, 3, 1, 1) layer, a Conv2BLR (128, 1, 1, 1) layer, a Conv2BLR (256, 3, 1, 1) layer, and a Conv2BLR (128, 3, 1, 1) layer in the data flow direction.

[0079] The input end of the first Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group, the input end of the first Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group, and the input end of the first Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group constitute the respective input ends of the feature fusion network FPN; and the output end of the last Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group, the output end of the last Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group, and the output end of the last Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group constitute the respective output ends of the feature fusion network FPN.

[0080] As Figure 2As shown, the input end of the first Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group in the feature fusion network FPN is connected to the output end of the last Resb5 residual block in the Resb5 residual group in the image feature extraction network Darknet53; the input end of the first Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group in the feature fusion network FPN is connected to the output end of the last Resb4 residual block in the Resb4 residual group in the image feature extraction network Darknet53; the input end of the first Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group in the feature fusion network FPN is connected to the output end of the last Resb3 residual block in the Resb3 residual group in the image feature extraction network Darknet53; and the output ends of the feature fusion network FPN are connected to the input ends of the regression classifier YoloHead.

[0081] As to the regression classifier YoloHead, the regression classifier YoloHead is specifically designed in the application and includes a Yolohead1 group, a Yolohead2 group and a Yolohead3 group. The Yolohead1 group is connected in series from its input end to a Conv2BLR (256, 3, 1, 1) layer, a Conv2 (255, 1, 1, 1) layer and an Output (80, 80, 255) layer in sequence; the Yolohead2 group is connected in series from its input end to a Conv2BLR (512, 3, 1, 1) layer, a Conv2 (255, 1, 1, 1) layer and an Output (40, 40, 255) layer in sequence; and the Yolohead3 group is connected in series from its input end to a Conv2BLR (1024, 3, 1, 1) layer, a Conv2 (255, 1, 1, 1) layer and an Output (20, 20, 255) layer in sequence. The Output (m, m, c) is an output layer.

[0082] The input end of the Yolohead1 group, the input end of the Yolohead2 group and the input end of the Yolohead3 group constitute the input ends of the regression classifier YoloHead; the output end of the last Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group in the feature fusion network FPN is connected to the input end of the Yolohead3 group; the output end of the last Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group in the feature fusion network FPN is connected to the input end of the Yolohead2 group; and the output end of the last Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group in the feature fusion network FPN is connected to the input end of the Yolohead1 group.

[0083] The output end of the Output (80, 80, 255) layer in the Yolohead1 group, the output end of the Output (40, 40, 255) layer in the Yolohead2 group, and the output end of the Output (20, 20, 255) layer in the Yolohead3 group constitute the respective output ends of the regression classifier YoloHead.

[0084] Step C. In continuation of the foregoing embodiment, based on each sea surface area SAR image sample, a 5-fold cross-validation method is used, 20% (322) of the sea surface area SAR image samples in the data set are randomly selected as the network verification data set, and the remaining 80% (1288) is the network training data set. Then, taking the sea surface area SAR image sample as the input and the ice mountain target label, the ship target label, or the no ice mountain and ship target label in the sea surface area SAR image sample as the output, the target detection and classification deep neural network is trained to obtain the target detection and classification model.

[0085] In the above specific network training process, the 80% training data set is sent to the target detection and classification deep neural network, and the network is trained until the network converges. The specific steps are as follows:

[0086] (1) Set the training starting learning rate, training batch size, and training round number.

[0087] (2) In the training process, the K-means clustering algorithm is used for iterative solution.

[0088] (3) The Adam gradient optimization algorithm is used to optimize the network training, and the following binary cross-entropy is used as the loss function in the optimization process, and the calculation formula is as follows:

[0089]

[0090] In the formula, Loss is the loss value of each round of training, log represents the logarithm with e as the base, ∑ represents summation, N is the number of training data set samples, p i is the probability that the i-th sample of the training data set is an ice mountain, y i is the annotation value of the i-th sample of the training data set, i=1,2,...,N.

[0091] (4) Repeat steps (2) and (3) until the network converges, obtain the final weight of the target detection and classification model for ship and ice mountain target detection and classification, and save the training weight.

[0092] (5) Load the training weight of the target detection and classification model, send the 20% verification data set to the trained target detection and classification model, verify the target detection and classification model, output the ship and ice mountain target detection and classification result, and realize the verification.

[0093] After obtaining the target detection classification model, the accuracy of the target detection classification model can be further evaluated by the following step i.

[0094] Step i. The F1 score corresponding to the target detection classification model is obtained according to the following formula, which is used to evaluate the accuracy of the target detection classification model.

[0095]

[0096]

[0097]

[0098] Wherein, P is the precision rate, that is, the proportion of true examples in the samples predicted as positive examples; R is the recall rate, that is, the proportion of true examples predicted as positive examples in all positive examples; TP is the true example, that is, the sample is correctly predicted as positive; FP is the false positive example, that is, the sample is predicted as positive, but the sample is actually negative; FN is the false negative example, that is, the sample is predicted as negative, but the sample is actually positive.

[0099] Based on the above-mentioned obtaining of the target detection classification model, further design is made according to the following operation in actual application, that is, applying the target detection classification model to detect and classify the ships and icebergs in the target sea surface area.

[0100] Operation: Firstly, based on different bands of Sentinel 1 satellite, the horizontal polarization SAR image and the horizontal and vertical cross polarization SAR image of the target sea surface area are captured by Sentinel 1A (Sentinel-1A) satellite; at the same time, based on different bands of Sentinel 1 satellite, the vertical horizontal cross polarization SAR image and the vertical polarization SAR image of the target sea surface area are captured by Sentinel 1B (Sentinel-1B) satellite.

[0101] Then, the horizontal polarization SAR image and the vertical horizontal cross polarization SAR image are fused to obtain the R channel SAR image; the horizontal and vertical cross polarization SAR image and the vertical polarization SAR image are fused to obtain the G channel SAR image; the horizontal polarization SAR image and the vertical polarization SAR image are fused to obtain the B channel SAR image; and then the RGB channel SAR image corresponding to the target sea surface area is obtained.

[0102] Finally, the target detection classification model is applied to the RGB channel SAR image corresponding to the target sea surface area to realize the detection and classification of ships and icebergs.

[0103] The above design is applied in actual application, and the final ship and iceberg target detection and classification result is as follows Figure 3 , Figure 4 ,Figure 5 As shown (black circle is iceberg target, and white circle is ship target), the test results prove that the application can realize accurate detection and classification of SAR image ship and iceberg targets, the F1 score is high, reaching 0.530, while the FasterR-CNN, Resnet and YOLO methods are only 0.430, 0.437 and 0.442 respectively, so the application design is a SAR image ship and iceberg target detection and classification method with short training time, fast convergence speed and high accuracy, which can effectively improve the sea situation awareness capability.

[0104] The embodiments of the application are described in detail above with reference to the drawings, but the application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the purpose of the application.

Claims

1. A method for detecting and classifying ships and icebergs in SAR images based on deep neural networks, characterized in that: A target detection classification model for identifying ships and icebergs on the sea surface is obtained through the following steps A to C; and the target detection classification model is applied to the target sea surface area to achieve detection and classification of ships and icebergs; Step A. A preset number of sea surface area SAR image samples containing iceberg target labels and ship target labels are constructed, and then step B is entered; Step B. A target detection classification deep neural network is constructed according to an image feature extraction network, a feature fusion network, and a regression classifier, and then step C is entered; Step C. The target detection classification deep neural network is trained based on the sea surface area SAR image samples, with the sea surface area SAR image samples as input and the iceberg target labels, ship target labels, or no icebergs and ship target labels in the sea surface area SAR image samples as output, to obtain the target detection classification model; In step B, a target detection classification deep neural network is constructed according to an image feature extraction network Darknet53, a feature fusion network FPN, and a regression classifier YoloHead; wherein the input end of the image feature extraction network Darknet53 constitutes the input end of the target detection classification deep neural network, and the image feature extraction network Darknet53 is sequentially connected with an Input(640, 640, 3) layer, a Conv2BLR(32, 2, 1, 1) layer, a Conv2(64, 3, 2, 1) layer, a Resb1 residual block, a Conv2(128, 3, 2, 1) layer, a Resb2 residual group, a Conv2(256, 3, 2, 1) layer, a Resb3 residual group, a Conv2(512, 3, 2, 1), a Resb4 residual group, a Conv2(1024, 3, 2, 1) layer, and a Resb5 residual group from the input end of the image feature extraction network Darknet53; The Resb1 residual block is sequentially connected with a Conv2BLR(32, 1, 1, 0) layer and a Conv2BLR(32, 2, 1, 1) layer from the input end of the Resb1 residual block; The Resb2 residual group is sequentially connected with two Resb2 residual blocks of the same structure from the input end of the Resb2 residual group, and the Resb2 residual block is sequentially connected with a Conv2BLR(64, 1, 1, 0) layer and a Conv2BLR(128, 3, 1, 1) layer from the input end of the Resb2 residual block; The Resb3 residual group is sequentially connected with eight Resb3 residual blocks of the same structure from the input end of the Resb3 residual group, and the Resb3 residual block is sequentially connected with a Conv2BLR(128, 1, 1, 0) layer and a Conv2BLR(256, 3, 1, 1) layer from the input end of the Resb3 residual block; The Resb4 residual group is sequentially connected with eight Resb4 residual blocks of the same structure from the input end of the Resb4 residual group, and the Resb4 residual block is sequentially connected with a Conv2BLR(256, 1, 1, 0) layer and a Conv2BLR(512, 3, 1, 1) layer from the input end of the Resb4 residual block; The Resb5 residual group is sequentially connected in series with four Resb4 residual blocks of the same structure from the input end of the Resb5 residual block, and the Resb5 residual block is sequentially connected in series with a Conv2BLR (512, 1, 1, 0) layer, a Conv2BLR (1024, 3, 1, 1) layer from the input end of the Resb5 residual block; The Input (m, m, c) is an input layer, the Conv2 (c, k, s, p) layer is a two-dimensional convolution layer, the Conv2BLR (c, k, s, p) layer is a Conv2 layer plus a batch normalization layer plus a linear activation function layer with a leakage correction, m represents the pixel size of image data, c is the number of channels, k represents the kernel size, s represents the step number, and p is the padding number; The output end of the last Resb3 residual block in the Resb3 residual group, the output end of the last Resb4 residual block in the Resb4 residual group, and the output end of the last Resb5 residual block in the Resb5 residual group constitute the respective output ends of the image feature extraction network Darknet53, the respective output ends of the image feature extraction network Darknet53 are connected to the input ends of the feature fusion network FPN, the output end of the feature fusion network FPN is connected to the input end of the regression classifier Yolo Head, and the output end of the regression classifier Yolo Head constitutes the output end of the target detection and classification deep neural network; The feature fusion network FPN includes a first Conv2BLR group, a Conv2BLR (256, 1, 1, 1) layer, an UpSample2D (2) layer, a second Conv2BLR group, a Conv2BLR (128, 1, 1, 1) layer, an UpSample2D (2) layer, and a third Conv2BLR group sequentially connected in series according to the data flow direction; The first Conv2BLR group includes a Conv2BLR (512, 1, 1, 1) layer, a Conv2BLR (1024, 3, 1, 1) layer, a Conv2BLR (512, 1, 1, 1) layer, a Conv2BLR (1024, 3, 1, 1) layer, and a Conv2BLR (512, 1, 1, 1) layer sequentially connected in series according to the data flow direction; the second Conv2BLR group includes a Conv2BLR (256, 1, 1, 1) layer, a Conv2BLR (512, 3, 1, 1) layer, a Conv2BLR (256, 1, 1, 1) layer, a Conv2BLR (512, 3, 1, 1) layer, and a Conv2BLR (256, 1, 1, 1) layer sequentially connected in series according to the data flow direction; and the third Conv2BLR group includes a Conv2BLR (128, 1, 1, 1) layer, a Conv2BLR (256, 3, 1, 1) layer, a Conv2BLR (128, 1, 1, 1) layer, a Conv2BLR (256, 3, 1, 1) layer, and a Conv2BLR (128, 3, 1, 1) layer sequentially connected in series according to the data flow direction. The input end of the first Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group, the input end of the first Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group, and the input end of the first Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group constitute the respective input ends of the feature fusion network FPN; the output end of the last Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group, the output end of the last Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group, and the output end of the last Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group constitute the respective output ends of the feature fusion network FPN; The input end of the first Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group in the feature fusion network FPN is connected to the output end of the last Resb5 residual block in the Resb5 residual group in the image feature extraction network Darknet53; the input end of the first Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group in the feature fusion network FPN is connected to the output end of the last Resb4 residual block in the Resb4 residual group in the image feature extraction network Darknet53; and the input end of the first Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group in the feature fusion network FPN is connected to the output end of the last Resb3 residual block in the Resb3 residual group in the image feature extraction network Darknet53; The respective output ends of the feature fusion network FPN are connected to the input ends of the regression classifier Yolo Head; The regression classifier Yolo Head includes a Yolo head1 group, a Yolo head2 group, and a Yolo head3 group, wherein the Yolo head1 group is connected in series from the input end to a Conv2BLR (256, 3, 1, 1) layer, a Conv2 (255, 1, 1, 1) layer, and an Output (80, 80, 255) layer; the Yolo head2 group is connected in series from the input end to a Conv2BLR (512, 3, 1, 1) layer, a Conv2 (255, 1, 1, 1) layer, and an Output (40, 40, 255) layer; the Yolo head3 group is connected in series from the input end to a Conv2BLR (1024, 3, 1, 1) layer, a Conv2 (255, 1, 1, 1) layer, and an Output (20, 20, 255) layer; and the Output (m, m, c) is an output layer; The input end of the Yolo head1 group, the input end of the Yolo head2 group and the input end of the Yolo head3 group constitute respective input ends of the regression classifier Yolo Head; the output end of the last Conv2BLR (512, 1, 1, 1) layer in the first Conv2BLR group in the feature fusion network FPN is connected to the input end of the Yolo head3 group; the output end of the last Conv2BLR (256, 1, 1, 1) layer in the second Conv2BLR group in the feature fusion network FPN is connected to the input end of the Yolo head2 group; and the output end of the last Conv2BLR (128, 1, 1, 1) layer in the third Conv2BLR group in the feature fusion network FPN is connected to the input end of the Yolo head1 group. The output end of the Output (80, 80, 255) layer in the Yolo head1 group, the output end of the Output (40, 40, 255) layer in the Yolo head2 group and the output end of the Output (20, 20, 255) layer in the Yolo head3 group constitute respective output ends of the regression classifier Yolo Head.

2. The SAR image ship and iceberg detection and classification method based on deep neural network according to claim 1, characterized in that: The method further comprises the following step i, which realizes precision evaluation of the target detection classification model. Step i. Obtain the score corresponding to the target detection classification model according to the following formula, for evaluating the accuracy of the target detection classification model. score, for evaluating the accuracy of the target detection classification model. ； ； ； wherein, is the precision, i.e. the proportion of true positives among samples predicted as positive, is the recall, i.e. the proportion of true positives among all positives; is a true positive, i.e. the prediction is correct and the sample is positive; is a false positive, i.e. the prediction is wrong and the sample is predicted as positive, but the sample is actually negative; is a false negative, i.e. the prediction is wrong and the sample is predicted as negative, but the sample is actually positive. 3.The SAR ship and iceberg detection and classification method based on deep neural network according to claim 1, wherein: The step A comprises steps A1 to A3. Step A1. Based on different bands of the Sentinel 1 satellite, a horizontal polarization SAR image of a sea surface area containing a preset first number of ice mountain target labels is captured by using a Sentinel 1A satellite, and a horizontal polarization SAR image and a horizontal-vertical cross-polarization SAR image of a sea surface area containing a preset second number of ice mountain target labels are captured by using a Sentinel 1B satellite; At the same time, based on different bands of the Sentinel 1 satellite, a vertical-horizontal cross-polarization SAR image of a sea surface area containing a preset third number of ship target labels is captured by using a Sentinel 1A satellite, and a vertical-horizontal cross-polarization SAR image and a vertical polarization SAR image of a sea surface area containing a preset fourth number of ship target labels are captured by using a Sentinel 1B satellite; then step A2 is entered. Step A2. The horizontal polarization SAR images containing the ice mountain target labels and the vertical-horizontal cross-polarization SAR images containing the ship target labels are fused to obtain an R channel SAR image. The horizontal-vertical cross-polarization SAR images containing the ice mountain target labels and the vertical polarization SAR images containing the ship target labels are fused to obtain a G channel SAR image. The horizontal polarization SAR images containing the ice mountain target labels and the vertical polarization SAR images containing the ship target labels are fused to obtain a B channel SAR image. Then, an RGB channel SAR image is obtained, and then step A3 is entered. Step A3. The RGB channel SAR image is divided into grids according to a preset grid size, and each RGB channel SAR image block is obtained, that is, a preset number of sea surface area SAR image samples containing ice mountain target labels and ship target labels are obtained.

4. The SAR ship and iceberg detection and classification method based on deep neural network according to claim 3, characterized in that: The target detection classification model is applied to the target sea surface area to realize detection and classification of ships and icebergs according to the following operation. Operation: firstly, based on different bands of the Sentinel 1 satellite, the horizontal polarization SAR image and the horizontal and vertical cross-polarization SAR image of the target sea surface area are captured by the Sentinel 1A satellite; and based on different bands of the Sentinel 1 satellite, the vertical and horizontal cross-polarization SAR image and the vertical polarization SAR image of the target sea surface area are captured by the Sentinel 1B satellite; Then, the horizontal polarization SAR image and the vertical and horizontal cross-polarization SAR image are fused to obtain an R-channel SAR image; the horizontal and vertical cross-polarization SAR image and the vertical polarization SAR image are fused to obtain a G-channel SAR image; and the horizontal polarization SAR image and the vertical polarization SAR image are fused to obtain a B-channel SAR image; and then the RGB channel SAR image corresponding to the target sea surface area is obtained; Finally, the target detection classification model is applied to the RGB channel SAR image corresponding to the target sea surface area to realize detection and classification of ships and icebergs.

5. The method of claim 1, wherein the method is based on a deep neural network. In the process of training the target detection classification deep neural network in step C, the K-means clustering algorithm is used for iterative solution, and the Adam gradient optimization algorithm is used to optimize the network training.

6. The SAR ship and iceberg detection and classification method based on deep neural network according to claim 5, characterized in that: In the process of optimizing the network training in step C, the following binary cross-entropy is used as the loss function. ； wherein is the loss value for each round of training, denotes the natural logarithm with base e, denotes the summation, is the number of samples in the training dataset, is the probability that the th sample in the training dataset is an iceberg, is the label value for the th sample in the training dataset, .

Images