A method for ship target detection in SAR images based on multi-layer neural network

Abstract

The present invention discloses a SAR image ship target detection method based on a multi-layer neural network, comprising the steps of: dividing a SAR image data set containing SAR images and their corresponding ship target true values ​​into a training set, a validation set, and a test set; preprocessing the training set, including size standardization, data enhancement, and prior frame clustering of the SAR image; using the validation set and the preprocessed training set to construct an initial model of a SAR image ship target detection method SSTD-MNN based on a multi-layer neural network; using the validation set and the preprocessed training set to adjust the parameters of the initial model of the SSTD-MNN to obtain an optimized model of the SSTD-MNN, and using the test set to evaluate the detection accuracy of the optimized model; using the obtained SSTD-MNN optimized model to perform ship target detection on a SAR image to be detected in a SAR image data set or in an actual application. The present invention provides a SAR image ship target detection method with high detection accuracy, which has broad application prospects.

Description

Technical Field

[0001] The invention belongs to the technical field of SAR image ship target detection, and relates to a SAR image ship target detection method based on a neural network, in particular to a SAR image ship target detection method based on a multi-layer neural network. Background Art

[0002] SAR (Synthetic Aperture Radar), or synthetic aperture radar, is an active earth observation system that can be installed on aircraft, satellites, spacecraft and other flying platforms to observe the earth all day and all weather, and has a certain surface penetration ability. Therefore, the SAR system has unique advantages in disaster monitoring, environmental monitoring, ocean monitoring, resource exploration, crop yield estimation, mapping and military applications, and can play a role that other remote sensing methods cannot play. SAR image is a high-resolution image generated by synthetic aperture radar through the synthetic aperture principle and high-frequency electromagnetic waves to actively sense an area. Since SAR imaging is less affected by the outside world, it can monitor an area over a long distance and all weather. A certain number of SAR images and their corresponding ship target ground truth constitute a SAR image data set. SAR image ship target detection (synthetic aperture radar image ship target detection) uses SAR images as units to detect ship targets in SAR images, which is an important means of ship detection in coastal areas. The SAR image ship target detection method can mine the image features of ship targets in SAR images based on the data in the SAR image dataset, and perform ship target detection on new SAR images. There are many non-ship target interferences in SAR images, so the SAR image ship target detection method relies on effective image feature extraction and screening capabilities, otherwise it will affect the accuracy of SAR image ship target detection.

[0003] A small improvement in the accuracy of SAR image ship target detection will also bring huge economic, social and military benefits. In daily life, SAR image ship target detection plays an important role in ship search and rescue, fishing boat monitoring, territorial defense, anti-smuggling and illegal immigration, etc. In battlefield environments, most modern ships are equipped with electronic radars. SAR image ship target detection and precision strikes or anti-missiles through electronic radars can affect the battlefield situation and have great significance at the strategic and tactical levels.

[0004] Common methods for ship target detection in SAR images include detection methods based on artificially set target features and deep learning detection methods based on neural networks.

[0005] A typical target feature detection method based on artificial settings is a detection method based on SIFT (Scale Invariant Feature Transform) features, which includes three parts: region selection, feature extraction, and target classification. The region selection uses an object detection method based on a sliding window (see: Yanwu Xu, Dong Xu, Stephen Lin, Jiang Liu, Jun Cheng, Carol Yim-lui Cheung, Tin Aung, Tien Yin Wong: Sliding Window and Regression Based Cup Detection in Digital Fundus Images for Glaucoma Diagnosis. MICCAI (3) 2011: 1-8. https: / / doi.org / 10.1007 / 978-3-642-23626-6_1). The non-targeted nature of the sliding window method results in high time complexity and low detection accuracy.

[0006] One of the most advanced (state-of-the-art, SOTA) neural network-based deep learning prediction methods is YOLOv4 (see: Alexey Bochkovskiy, Chien-Yao Wang, Hong-Yuan Mark Liao: YOLOv4: Optimal Speed ​​and Accuracy of Object Detection. CoRR abs / 2004.10934 (2020) https: / / arxiv.org / abs / 2004.10934). YOLOv4 can learn image features by itself and perform multi-scale ship target detection. It has fast detection time and strong robustness. However, the prediction method has insufficient processing capabilities for image features, resulting in low detection accuracy.

[0007] In order to overcome the above-mentioned deficiencies of the prior art in the relevant technical field, the present invention intends to use deep learning technology based on neural networks to construct a SAR image ship target detection method. Such a method has strong image feature processing capabilities and therefore has high detection accuracy.

[0008] Deep learning based on neural networks is a new generation of machine learning technology. Attention mechanism and convolutional neural network (CNN) are both typical neural networks. The attention mechanism in the field of deep learning technology (see: Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio: Neural Machine Translation by Jointly Learning to Align and Translate. 3rd International Conference on Learning Representations, ICLR 2015, May 7-9, 2015, Conference Track Proceedings. http: / / arxiv.org / abs / 1409.0473) is similar to the selective visual attention mechanism of humans, which can ignore unimportant information from a large amount of important information, filter out a small amount of important information and focus on these important information. Convolutional neural network (see: Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner: Gradient-based learning applied to document recognition. Proceedings of the IEEE, Volume 86, Issue 11, 2278-2324, Nov. 1998. https: / / doi.org / 10.1109 / 5.726791) is a type of feed-forward neural network that includes convolution calculations and has a deep structure, which can effectively extract features from data. According to the specific needs of the technical problem to be solved, the attention mechanism can be organically combined with the convolutional neural network to form a specific multi-layer neural network.

[0009] The following is a brief description of several prior arts adopted in the technical solution of the present invention. These prior arts are only several single technical features in the technical solution of the present invention. They constitute the technical solution of the present invention only after being organically combined with other technical features of the present invention in a non-obvious manner.

[0010] The K-means clustering algorithm is used to cluster the ship targets in the SAR image dataset to obtain a priori boxes of appropriate sizes.

[0011] Two-dimensional convolution (2-D convolution) and one-dimensional convolution (1-D convolution) (see: Vincent Dumoulin, Francesco Visin: A guide to convolution arithmetic for deep learning.arXiv:1603.07285v2[stat.ML]11 Jan 2018.http: / / arxiv.org / abs / 1603.07285) are used to form a convolution block to extract features of the SAR image.

[0012] Residual network (resnet) (see: Kaiming He, Xiangyu Zhang, Shaoqing Ren, et al.: Deep Residual Learning for Image Recognition. CVPR 2016: 770-778. https: / / doi.org / 10.1109 / CVPR.2016.90) is used to solve the problems of gradient propagation and network degradation caused by stacked two-dimensional convolution.

[0013] External Attention (see: Meng-Hao Guo, Zheng-Ning Liu, atal.: Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks. CoRR abs / 2105.02358 (2021). https: / / arxiv.org / abs / 2105.02358) is a type of attention mechanism used to enhance the feature processing capabilities of the method.

[0014] The CIoU (Complete-IoU) loss function (see: Zhaohui Zheng, Ping Wang, Wei Liu, JinzeLi, Rongguang Ye, Dongwei Ren: Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression. AAAI 2020: 12993-13000. https: / / ojs.aaai.org / index.php / AAAI / article / view / 6999) is used to calculate the distance between the detected coordinate values ​​and the coordinate values ​​of the true value.

[0015] The Adam optimizer (see: Diederik P. Kingma, Jimmy Ba: Adam: A Method for Stochastic Optimization. ICLR (Poster) 2015. http: / / arxiv.org / abs / 1412.6980) is used to optimize the parameters of the SAR image ship target detection method (model).

[0016] When evaluating the detection accuracy of the SAR image ship target detection method, the present invention uses two performance evaluation indicators commonly used in the technical field: precision and recall. The calculation formulas of the two are as follows:

[0017]

[0018] Where TP is called true positive, which indicates the number of ground truth values ​​of ship targets in the detection results; FP is called false positive, which indicates the number of ground truth values ​​of non-ship targets in the detection results; FN is called false negative, which indicates the number of ground truth values ​​of ship targets that have not been detected.

[0019] In summary, in order to overcome the above-mentioned shortcomings in the prior art, namely: the image feature processing ability of the SAR image ship target detection method is weak and the detection accuracy is not high enough, the present invention organically combines the attention mechanism with the convolutional neural network to form a specific multi-layer neural network, thereby providing a SAR image ship target detection method based on multi-layer neural network (SAR Image Ship Target Detection Method based on Multi-layer NeuralNetwork, abbreviated as SSTD-MNN). Summary of the invention

[0020] The technical problem to be solved by the present invention is to provide a SAR image ship target detection method with high detection accuracy.

[0021] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0022] The present invention provides a SAR image ship target detection method based on a multi-layer neural network, comprising the following steps:

[0023] Step S1: dividing the SAR image data set containing the SAR images and the true values ​​of the corresponding ship targets into a training set, a validation set, and a test set;

[0024] Step S2: preprocessing the training set, including size standardization, data enhancement and prior box clustering of SAR images;

[0025] Step S3: using the validation set and the preprocessed training set to construct an initial model of a SAR image ship target detection method SSTD-MNN based on a multi-layer neural network;

[0026] Step S4: using the validation set and the preprocessed training set to adjust the parameters of the initial model of the SSTD-MNN to obtain an optimized model of the SSTD-MNN, and using the test set to evaluate the detection accuracy of the optimized model;

[0027] Step S5: Use the obtained SSTD-MNN optimization model to perform ship target detection on the SAR images to be detected in the SAR image dataset or in actual applications.

[0028] In the method, the step S2 further comprises:

[0029] Step S21: the sizes of all SAR images and their true values ​​of ship targets in the training set are standardized to 608 pixels × 608 pixels;

[0030] Step S22: first, for each SAR image in the training set, three other SAR images in the training set are randomly selected, and the four SAR images are randomly scaled and placed at four corners of a blank image to splice into a new SAR image. Then, the SAR image and the true values ​​of the ship targets corresponding to the original four SAR images are edge processed to obtain the preprocessed SAR image and the true values ​​of the ship targets.

[0031] Step S23: Perform K-means clustering on the true values ​​of all ship targets in the training set to obtain n prior boxes, where the hyperparameter n is taken from the elements in the set {6, 9, 12}.

[0032] In the method, step S3 further comprises:

[0033] Step S31: construct a feature extraction layer for the initial model of the detection method SSTD-MNN, which includes the following 7 sub-layers:

[0034] The first sublayer contains two 2D convolutional layers and one mish function activation layer. The number of convolution kernels in the two 2D convolutional layers is 32 and 64 respectively, and the sizes of convolution kernels are 1×1 and 3×3 / 2 respectively. The output of this sublayer is defined by the following formula:

[0035] Output=mish(conv2(conv1(Input))),

[0036] In the formula, Input represents the input of the sub-layer, conv1 and conv2 represent two two-dimensional convolutional layers respectively, and Output represents the output of the sub-layer;

[0037] The second sublayer contains three two-dimensional convolutional layers, one mish function activation layer and one residual network layer. The number of convolution kernels in the three two-dimensional convolutional layers is 32 and the size of the convolution kernel is 1×1. The residual network layer contains two two-dimensional convolutions, and the number of their convolution kernels is 32 and 64 respectively, and the size of their convolution kernels is 1×1 and 3×3 respectively.

[0038] The third sublayer contains three two-dimensional convolutional layers, one mish function activation layer and two residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 64 and the size of the convolution kernels is 1×1. The two residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 64 and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0039] The fourth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 128, and the size of the convolution kernels is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 128, and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0040] The fifth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 256, and the size of the convolution kernels is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 256, and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0041] The sixth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 256, and the size of the convolution kernel is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 512, and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0042] The outputs of the 2nd to 6th sublayers are defined by the following formulas:

[0043] Output=mish(conv3(concat(conv1(Input), Resblock n (conv2(Input))))),

[0044] In the formula, Resblock nrepresents the residual network layer in the nth sublayer, n∈{1, 2, ..., 6}, Input represents the input of the corresponding sublayer, conv1, conv2, conv3 represent the three two-dimensional convolutional layers in the corresponding sublayer, concat represents the connection operation, and Output represents the output of the corresponding sublayer;

[0045] The seventh sublayer contains three maximum pooling layers, whose pooling kernel sizes are 3×3, 5×5, and 7×7 respectively. The output of this sublayer is defined by the following formula:

[0046] Output=concat(Input, maxpool1(Input), maxpool2(Input), maxpool3(Input)),

[0047] In the formula, Input represents the input of the sub-layer, maxpool1, maxpool2, and maxpool3 represent three maximum pooling layers respectively, concat represents the connection operation, and Output represents the output of the sub-layer;

[0048] Step S32: construct a feature fusion layer for the initial model of the detection method SSTD-MNN, which includes the following 5 sub-layers:

[0049] The first sublayer contains four two-dimensional convolutional layers, whose convolution kernel numbers are 512, 1024, 512, and 256, and whose convolution kernel sizes are 1×1, 3×3, 1×1, and 1×1, respectively. The output of the first sublayer is defined by the following formula:

[0050] Output = conv 1-4 (Input),

[0051] In the formula, Input represents the input of the sublayer, conv 1-4 Represents the 1st to 4th two-dimensional convolutional layers, and Output represents the output of this sublayer;

[0052] The second sublayer contains 1 upsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 256, 256, 512, 256, 512, 256 respectively, and the sizes of convolution kernels are 1×1, 1×1, 3×3, 1×1, 3×3, 1×1 respectively. The sampling multiple of the upsampling layer is 4. The output of this sublayer is defined by the following formula:

[0053] Output = ea (conv 2-6 (concat(upsample(Input),conv1(Output S31-5 )))),

[0054] In the formula, Input represents the input of the sublayer, upsample represents the upsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S31-5 represents the output of the fifth sublayer in step S31, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer;

[0055] The third sublayer contains 1 upsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 128, 128, 256, 128, 256, 128, respectively, and the sizes of convolution kernels are 1×1, 1×1, 3×3, 1×1, 3×3, 1×1, respectively. The sampling multiple of the upsampling layer is 4. The output of this sublayer is defined by the following formula:

[0056] Output = ea (conv 2-6 (concat(upsample(Input),conv1(Output S31-4 )))),

[0057] In the formula, Input represents the input of this sublayer, conv1 represents the first two-dimensional convolutional layer, and Output S31-4 represents the output of the fourth sublayer in step S31, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer;

[0058] The fourth sublayer contains 1 downsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 256, 256, 512, 256, 512, 256 respectively, and the sizes of convolution kernels are 3×3, 1×1, 3×3, 1×1, 3×3, 1×1 respectively. The sampling multiple of the downsampling layer is 1 / 4. The output of this sublayer is defined by the following formula:

[0059] Output = ea (conv 2-6 (concat(conv1(downsample(Input))),Output S32-2 ))),

[0060] In the formula, Input represents the input of the sublayer, downsample represents the downsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S32-2 represents the output of the second sublayer in step S32, concat represents the connection operation, conv2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer;

[0061] The fifth sublayer contains one downsampling layer and six two-dimensional convolutional layers. The number of convolution kernels of the six two-dimensional convolutional layers are 512, 512, 1024, 512, 1024, 512, respectively, and the sizes of the convolution kernels are 3×3, 1×1, 3×3, 1×1, 3×3, 1×1, and 1×1, respectively. The sampling multiple of the downsampling layer is 1 / 4. The output of this sublayer is defined by the following formula:

[0062] Output = conv 2-6 (concat(conv1(downsample(Input))),Output S32-1 )),

[0063] In the formula, Input represents the input of the sublayer, downsample represents the downsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S32-1 represents the output of the first sublayer in step S32, concat represents the connection operation, conv 2-6 Represents the 2nd to 6th two-dimensional convolutional layer, and Output represents the output of this sublayer;

[0064] Step S33: construct an output layer for the initial model of the detection method SSTD-MNN, which includes the following three sub-layers:

[0065] The first sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 256 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer.

[0066] The second sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 512 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer.

[0067] The third sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 1024 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer.

[0068] Step S34: performing calculations based on the prior frame obtained in step S23 and all outputs of the output layer to obtain the actual value of the ship target detection coordinates, specifically including:

[0069] First, the n priori frames obtained in step S23 are equally divided into three parts. Then, the three sublayers in the output layer are sequentially associated with the three priori frames, and the n / 3 outputs of each sublayer are sequentially corresponded to the n / 3 priori frames associated with the sublayer. Finally, the initial value of the ship target detection coordinate in each output of each sublayer is sequentially calculated with the width and height of the priori frame corresponding to the output to obtain the actual value of the ship target detection coordinate. The calculation is defined by the following formula:

[0070]

[0071]

[0072]

[0073]

[0074] In the formula, t x ,t y ,t w ,t h They represent the abscissa, ordinate, width and height of the initial value of the ship target detection coordinates in the output of the sublayer, respectively. x , b y , b w , b h They represent the abscissa, ordinate, width and height of the actual values ​​of the ship target detection coordinates obtained by calculation, respectively. w 、p h Respectively represent the width and height of the corresponding prior box, c x 、c y They represent the horizontal and vertical distances between the output of the sublayer and the upper left corner of the preprocessed SAR image, respectively;

[0075] Step S35: Perform non-maximum suppression on the calculation result of step S34 with an intersection-over-union ratio threshold of threshold1, where the hyperparameter threshold1 takes values ​​in an open interval (0, 1).

[0076] In the method, step S4 further comprises:

[0077] Step S41: First, the intersection and union ratios of each output of the initial model of the SSTD-MNN and the true value of each ship target are calculated one by one, and then, for each true value of the ship target, the first output with the maximum intersection and union ratio with it is recorded as the correct output, and if the maximum value of the intersection and union ratios of a certain output with the true values ​​of all ship targets among the remaining outputs is less than a threshold value threshold2, then the output is recorded as an incorrect output, wherein the hyperparameter threshold2 takes values ​​in the open interval (0, 1);

[0078] Step S42: construct a loss function for the initial model of the SSTD-MNN, and calculate the loss of all outputs of the model before performing gradient propagation. The loss function is defined by the following formula:

[0079] loss=I obj CIoU(pred coord ,gt coord )+I obj BCE(pred cls ,gt cls )+BCE(pred conf ,gt conf ),

[0080] In the formula, I obj Indicates whether it is the correct output, its value is 1 for the correct output, and its value is 0 for the rest of the output; CIoU represents the CIoU loss function; BCE represents the binary cross entropy function; pred coord represents the actual value of the ship target detection coordinates in the output of the output layer, gt coord Indicates the coordinate value of the ship target in the real value, pred cls Indicates the type of ship target detection object in the output of the output layer, gt cls Indicates the type of ship target object in the true value of the ship target, pred conf Represents the confidence in the output of the output layer, gt conf Indicates whether it is an erroneous output, with a value of 0 indicating an erroneous output and a value of 1 indicating the remaining output;

[0081] Step S43: Use the Adam optimizer with a learning rate of η to adjust the parameters of the initial model of the SSTD-MNN, where the hyperparameter η is taken from the elements in the set {0.0005, 0.001, 0.0015, 0.002}, and calculate the loss function on the validation set. If the function converges, the parameter adjustment process is terminated to obtain the optimized model of the SSTD-MNN. Otherwise, steps S41, S42 and S43 are repeated in sequence.

[0082] Step S44: using the test set and two performance evaluation indicators, namely, precision and recall, to evaluate the detection accuracy of the obtained SSTD-MNN optimization model.

[0083] The beneficial technical effects of the present invention mainly include three aspects: (1) The present invention provides a SAR image ship target detection method with high detection accuracy, which has certain scientific significance for the development of SAR image ship target detection technology; (2) Compared with the existing SAR image ship target detection method, the method of the present invention has the advantage of high detection accuracy. The reason is that the technical solution of the present invention organically combines the attention mechanism with the convolutional neural network to form a specific multi-layer neural network, which has stronger graphic feature processing capabilities; (3) The method of the present invention has broad application prospects in many aspects such as disaster monitoring, environmental monitoring, ocean monitoring, resource exploration, crop yield estimation, surveying and mapping, and military.

[0084] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. Additional aspects and advantages of the present invention will be partially given in the following description, which will become apparent from the following description or be understood through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 A schematic flow chart of a method for detecting ship targets in SAR images based on a multi-layer neural network according to the technical solution of the present invention is shown;

[0086] Figure 2 A schematic diagram of the process of constructing an initial model of SSTD-MNN in a SAR image ship target detection method based on a multi-layer neural network according to the technical solution of the present invention is shown. DETAILED DESCRIPTION

[0087] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar concepts, objects, elements, etc. or concepts, objects, elements, etc. with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0088] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs and related fields. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0089] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0090] The present invention provides a method for detecting ship targets in SAR images based on a multi-layer neural network. Figure 1 As shown, the following steps S1 to S5 are included:

[0091] Step S1: Divide the SAR image dataset containing the SAR images and the true values ​​of the corresponding ship targets into a training set, a validation set, and a test set.

[0092] Step S2: preprocessing the training set, including size standardization, data enhancement and prior frame clustering of SAR images; step S2 further includes:

[0093] Step S21: the sizes of all SAR images and their true values ​​of ship targets in the training set are standardized to 608 pixels × 608 pixels;

[0094] Step S22: first, for each SAR image in the training set, three other SAR images in the training set are randomly selected, and the four SAR images are randomly scaled and placed at four corners of a blank image to splice into a new SAR image. Then, the SAR image and the true values ​​of the ship targets corresponding to the original four SAR images are edge processed to obtain the preprocessed SAR image and the true values ​​of the ship targets.

[0095] Step S23: Perform K-means clustering on the true values ​​of all ship targets in the training set to obtain n prior boxes, where the hyperparameter n is taken from the elements in the set {6, 9, 12}.

[0096] Step S3: using the validation set and the preprocessed training set to construct an initial model of a SAR image ship target detection method SSTD-MNN based on a multi-layer neural network; Figure 2 As shown, the step S3 further comprises:

[0097] Step S31: construct a feature extraction layer for the initial model of the detection method SSTD-MNN, which includes the following 7 sub-layers:

[0098] The first sublayer contains two 2D convolutional layers and one mish function activation layer. The number of convolution kernels in the two 2D convolutional layers is 32 and 64 respectively, and the sizes of convolution kernels are 1×1 and 3×3 / 2 respectively. The output of this sublayer is defined by the following formula:

[0099] Output=mish(conv2(conv1(Input))),

[0100] In the formula, Input represents the input of the sub-layer, conv1 and conv2 represent two two-dimensional convolutional layers respectively, and Output represents the output of the sub-layer;

[0101] The second sublayer contains three two-dimensional convolutional layers, one mish function activation layer and one residual network layer. The number of convolution kernels in the three two-dimensional convolutional layers is 32 and the size of the convolution kernel is 1×1. The residual network layer contains two two-dimensional convolutions, and the number of their convolution kernels is 32 and 64 respectively, and the size of their convolution kernels is 1×1 and 3×3 respectively.

[0102] The third sublayer contains three two-dimensional convolutional layers, one mish function activation layer and two residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 64 and the size of the convolution kernels is 1×1. The two residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 64 and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0103] The fourth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 128, and the size of the convolution kernels is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 128, and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0104] The fifth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 256, and the size of the convolution kernels is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 256, and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0105] The sixth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 256, and the size of the convolution kernel is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 512, and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0106] The outputs of the 2nd to 6th sublayers are defined by the following formulas:

[0107] Output=mish(conv3(concat(conv1(Input), Resblock n (conv2(Input))))),

[0108] In the formula, Resblock n represents the residual network layer in the nth sublayer, n∈{1, 2, ..., 6}, Input represents the input of the corresponding sublayer, conv1, conv2, conv3 represent the three two-dimensional convolutional layers in the corresponding sublayer, concat represents the connection operation, and Output represents the output of the corresponding sublayer;

[0109] The seventh sublayer contains three maximum pooling layers, whose pooling kernel sizes are 3×3, 5×5, and 7×7 respectively. The output of this sublayer is defined by the following formula:

[0110] Output=concat(Input, maxpool1(Input), maxpool2(Input), maxpool3(Input)),

[0111] In the formula, Input represents the input of the sub-layer, maxpool1, maxpool2, and maxpool3 represent three maximum pooling layers respectively, concat represents the connection operation, and Output represents the output of the sub-layer;

[0112] Step S32: construct a feature fusion layer for the initial model of the detection method SSTD-MNN, which includes the following 5 sub-layers:

[0113] The first sublayer contains four two-dimensional convolutional layers, whose convolution kernel numbers are 512, 1024, 512, and 256, and whose convolution kernel sizes are 1×1, 3×3, 1×1, and 1×1, respectively. The output of the first sublayer is defined by the following formula:

[0114] Output = conv 1-4 (Input),

[0115] In the formula, Input represents the input of the sublayer, conv 1-4 Represents the 1st to 4th two-dimensional convolutional layers, and Output represents the output of this sublayer;

[0116] The second sublayer contains 1 upsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 256, 256, 512, 256, 512, 256 respectively, and the sizes of convolution kernels are 1×1, 1×1, 3×3, 1×1, 3×3, 1×1 respectively. The sampling multiple of the upsampling layer is 4. The output of this sublayer is defined by the following formula:

[0117] Output = ea (conv 2-6 (concat(upsample(Input),conv1(Output S31-5 )))),

[0118] In the formula, Input represents the input of the sublayer, upsample represents the upsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S31-5 represents the output of the fifth sublayer in step S31, concat represents the connection operation, conv 2-6represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer;

[0119] The third sublayer contains 1 upsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 128, 128, 256, 128, 256, 128, respectively, and the sizes of convolution kernels are 1×1, 1×1, 3×3, 1×1, 3×3, 1×1, respectively. The sampling multiple of the upsampling layer is 4. The output of this sublayer is defined by the following formula:

[0120] Output = ea (conv 2-6 (concat(upsample(Input),conv1(Output S31-4 )))),

[0121] In the formula, Input represents the input of this sublayer, conv1 represents the first two-dimensional convolutional layer, and Output S31-4 represents the output of the fourth sublayer in step S31, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer;

[0122] The fourth sublayer contains 1 downsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 256, 256, 512, 256, 512, 256 respectively, and the sizes of convolution kernels are 3×3, 1×1, 3×3, 1×1, 3×3, 1×1 respectively. The sampling multiple of the downsampling layer is 1 / 4. The output of this sublayer is defined by the following formula:

[0123] Output = ea (conv 2-6 (concat(conv1(downsample(Input))),Output S32-2 ))),

[0124] In the formula, Input represents the input of the sublayer, downsample represents the downsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S32-2 represents the output of the second sublayer in step S32, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer;

[0125] The fifth sublayer contains one downsampling layer and six two-dimensional convolutional layers. The number of convolution kernels of the six two-dimensional convolutional layers are 512, 512, 1024, 512, 1024, 512, respectively, and the sizes of the convolution kernels are 3×3, 1×1, 3×3, 1×1, 3×3, 1×1, and 1×1, respectively. The sampling multiple of the downsampling layer is 1 / 4. The output of this sublayer is defined by the following formula:

[0126] Output = conv 2-6 (concat(conv1(downsample(Input))),Output S32-1 )),

[0127] In the formula, Input represents the input of the sublayer, downsample represents the downsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S32-1 represents the output of the first sublayer in step S32, concat represents the connection operation, conv 2-6 Represents the 2nd to 6th two-dimensional convolutional layer, and Output represents the output of this sublayer;

[0128] Step S33: construct an output layer for the initial model of the detection method SSTD-MNN, which includes the following three sub-layers:

[0129] The first sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 256 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer.

[0130] The second sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 512 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer.

[0131] The third sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 1024 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer.

[0132] Step S34: performing calculations based on the prior frame obtained in step S23 and all outputs of the output layer to obtain the actual value of the ship target detection coordinates, specifically including:

[0133] First, the n priori frames obtained in step S23 are equally divided into three parts. Then, the three sublayers in the output layer are sequentially associated with the three priori frames, and the n / 3 outputs of each sublayer are sequentially corresponded to the n / 3 priori frames associated with the sublayer. Finally, the initial value of the ship target detection coordinate in each output of each sublayer is sequentially calculated with the width and height of the priori frame corresponding to the output to obtain the actual value of the ship target detection coordinate. The calculation is defined by the following formula:

[0134]

[0135]

[0136]

[0137]

[0138] In the formula, t x ,t y ,t w ,t h They represent the abscissa, ordinate, width and height of the initial value of the ship target detection coordinates in the output of the sublayer, respectively. x , b y , b w , b h They represent the abscissa, ordinate, width and height of the actual values ​​of the ship target detection coordinates obtained by calculation, respectively. w 、p h Respectively represent the width and height of the corresponding prior box, c x 、c y They represent the horizontal and vertical distances between the output of the sublayer and the upper left corner of the preprocessed SAR image, respectively;

[0139] Step S35: Perform non-maximum suppression on the calculation result of step S34 with an intersection-over-union ratio threshold of threshold1, where the hyperparameter threshold1 takes values ​​in an open interval (0, 1).

[0140] Step S4: using the validation set and the preprocessed training set to adjust the parameters of the initial model of the SSTD-MNN to obtain an optimized model of the SSTD-MNN, and using the test set to evaluate the detection accuracy of the optimized model; the step S4 further includes:

[0141] Step S41: First, the intersection and union ratios of each output of the initial model of the SSTD-MNN and the true value of each ship target are calculated one by one, and then, for each true value of the ship target, the first output with the maximum intersection and union ratio with it is recorded as the correct output, and if the maximum value of the intersection and union ratios of a certain output with the true values ​​of all ship targets among the remaining outputs is less than a threshold value threshold2, then the output is recorded as an incorrect output, wherein the hyperparameter threshold2 takes values ​​in the open interval (0, 1);

[0142] Step S42: construct a loss function for the initial model of the SSTD-MNN, and calculate the loss of all outputs of the model before performing gradient propagation. The loss function is defined by the following formula:

[0143] loss=I obj CIoU(pred coord ,gt coord )+I obj BCE(pred cls ,gt cls )+BCE(pred conf ,gt conf ),

[0144] In the formula, I obj Indicates whether it is the correct output, its value is 1 for the correct output, and its value is 0 for the rest of the output; CIoU represents the CIoU loss function; BCE represents the binary cross entropy function; pred coord represents the actual value of the ship target detection coordinates in the output of the output layer, gt coord Indicates the coordinate value of the ship target in the real value, pred cls Indicates the type of ship target detection object in the output of the output layer, gt cls Indicates the type of ship target object in the true value of the ship target, pred conf represents the confidence in the output of the output layer, gt conf Indicates whether it is an erroneous output, with a value of 0 indicating an erroneous output and a value of 1 indicating the remaining output;

[0145] Step S43: Use the Adam optimizer with a learning rate of η to adjust the parameters of the initial model of the SSTD-MNN, where the hyperparameter η is taken from the elements in the set {0.0005, 0.001, 0.0015, 0.002}, and calculate the loss function on the validation set. If the function converges, the parameter adjustment process is terminated to obtain the optimized model of the SSTD-MNN. Otherwise, steps S41, S42 and S43 are repeated in sequence.

[0146] Step S44: using the test set and two performance evaluation indicators, namely, precision and recall, to evaluate the detection accuracy of the obtained SSTD-MNN optimization model.

[0147] Step S5: Use the obtained SSTD-MNN optimization model to perform ship target detection on the SAR images to be detected in the SAR image dataset or in actual applications.

[0148] The following is a specific example to further illustrate the specific implementation of the technical solution of the present invention, and through experiments, the technical solution of the present invention (i.e., the detection method SSTD-MNN) is compared with YOLOv4, one of the most advanced (state-of-the-art, SOTA) neural network-based deep learning prediction methods (see: Alexey Bochkovskiy, Chien-YaoWang, Hong-Yuan Mark Liao: YOLOv4: Optimal Speed ​​and Accuracy of Object Detection. CoRR abs / 2004.10934 (2020) https: / / arxiv.org / abs / 2004.10934) for detection accuracy. Specifically, two performance evaluation indicators are used: precision and recall.

[0149] The SAR image dataset used in the specific embodiment is called the SSDD dataset (see: Tianwen Zhang, Xiaoling Zhang, et al.: SAR Ship Detection Dataset (SSDD): Official Release and Comprehensive Data Analysis. Remote. Sens. 13 (18): 3690 (2021). https: / / doi.org / 10.3390 / rs13183690). The SSDD dataset was made public at the BIGSARDATA 2017 conference held in Beijing, China on December 1, 2017 (see: http: / / www.radi.ac.cn / BIGSARDATA2017 / ), and is the first public dataset in this technical field. The SSDD dataset contains a total of 1,160 SAR images and 2,456 true values ​​of ship targets. The scene types include sea surface, islands and reefs, ports, etc. The SAR images in this dataset are collected from three satellite-borne SARs: Sentinel-1, TerraSAR-X, and Radarsat-2, and the resolution of the SAR images is not fixed.

[0150] Furthermore, in order to compare the performance of the technical solution of the present invention (i.e., the detection method SSTD-MNN) with the YOLOv4 method in terms of detection accuracy by using the above SSDD data set, the inventors implemented these methods through programming, using Pycharm as the software development platform and Python as the programming language. The specific implementation techniques are as follows:

[0151] Implementation technology of YOLOv4: The implementation tools are Python 3.7 and Pytorch 1.2.0. This method is implemented by downloading the code directly from GitHub (see: https: / / github.com / Tianxiaomo / pytorch-YOLOv4).

[0152] Implementation technology of SSTD-MNN: The implementation tools are Python3.7 and Pytorch1.2.0. The implementation technology is as follows: In the process of building the feature extraction layer, the code is downloaded from GitHub (see: https: / / github.com / Tianxiaomo / pytorch-YOLOv4) and slightly modified to implement it; the deep learning framework Pytorch (see: https: / / github.com / pytorch / pytorch) is used to build the feature fusion layer and output layer. Among them, the feature fusion layer is implemented using the Linear class, functional class, Conv1d class, and Conv2d class of the nn module in the Pytorch framework, and the output layer is implemented using the Conv2d class and functional class of the nn module in the Pytorch framework.

[0153] The operating environment of the experiment: (1) Hardware platform: Intel(R) Core(TM) i5-10600KF CPU@4.10GHz, NVIDIA GTX1070 8GB, memory 16GB; (2) Operating system: Windows 10; (3) Software platform: Pycharm.

[0154] The experimental process is as follows:

[0155] The experimental process of YOLOv4 is as follows: (1) The parameters of YOLOv4 are adjusted using the training set and the validation set to obtain an optimized model. The hyperparameter configuration of the model is: learning rate η = 0.001, number of training rounds e = 120; (2) The detection accuracy of the optimized model is calculated on the test set, namely the precision and recall values.

[0156] The experimental process of SSTD-MNN is as follows: (1) Set the hyperparameter values ​​for the SSTD-MNN model; (2) Use the Adam optimizer to adjust the parameters of SSTD-MNN on the training set; (3) Calculate the loss function of SSTD-MNN on the validation set. If the function converges, the parameter adjustment process ends. Otherwise, repeat (2) and (3) in sequence; (4) Calculate the detection accuracy of the optimized model on the test set, that is, the precision and recall values.

[0157] With respect to the specific embodiment, the specific implementation methods of the technical solution of the present invention are as follows:

[0158] like Figure 1 As shown, a method for detecting ship targets in SAR images based on a multi-layer neural network includes the following steps S1 to S5:

[0159] Step S1: Divide a SAR image dataset containing SAR images and their corresponding ship target true values ​​into a training set, a validation set, and a test set. In this embodiment, the SAR image dataset adopts the SSDD dataset, which contains 1160 SAR images and 2456 ship target true values. According to the common practice in the technical field, the SAR image dataset is divided into a training set, a validation set, and a test set in a ratio of 8:1:1, that is, the SAR images contained in the training set, the validation set, and the test set are 928, 116, and 116, respectively, and the ship target true values ​​are also 8:1:1. should be included in the training set, validation set and test set; it should be further explained that if the total number of SAR images in some SAR image data sets cannot be strictly divided into the training set, validation set and test set in the ratio of 8:1:1, then the number of SAR images contained in the training set, validation set and test set respectively needs to be fine-tuned during the division process to maximize the ratio of 8:1:1; in addition, according to the common practice in the technical field, a SAR image data set can also be divided into the training set, validation set and test set in turn according to other division ratios, and the division ratio includes but is not limited to 7:2:1.

[0160] Step S2: preprocessing the training set, including size standardization, data enhancement and prior frame clustering of SAR images; step S2 further includes:

[0161] Step S21: the sizes of all SAR images and their true values ​​of ship targets in the training set are standardized to 608 pixels × 608 pixels;

[0162] Step S22: first, for each SAR image in the training set, three other SAR images in the training set are randomly selected, and the four SAR images are randomly scaled and placed at four corners of a blank image to splice into a new SAR image. Then, the SAR image and the true values ​​of the ship targets corresponding to the original four SAR images are edge processed to obtain the preprocessed SAR image and the true values ​​of the ship targets.

[0163] Step S23: After performing K-means clustering on the true values ​​of all ship targets in the training set, n prior boxes are obtained, where the hyperparameter n is taken from the elements in the set {6, 9, 12}; in this embodiment, the value of n is 9.

[0164] Step S3: using the validation set and the preprocessed training set to construct an initial model of a SAR image ship target detection method SSTD-MNN based on a multi-layer neural network; Figure 2 As shown, the step S3 further comprises:

[0165] Step S31: construct a feature extraction layer for the initial model of the detection method SSTD-MNN, which includes the following 7 sub-layers:

[0166] The first sublayer contains two 2D convolutional layers and one mish function activation layer. The number of convolution kernels in the two 2D convolutional layers is 32 and 64 respectively, and the sizes of convolution kernels are 1×1 and 3×3 / 2 respectively. The output of this sublayer is defined by the following formula:

[0167] Output=mish(conv2(conv1(Input))),

[0168] In the formula, Input represents the input of the sub-layer, conv1 and conv2 represent two two-dimensional convolutional layers respectively, and Output represents the output of the sub-layer;

[0169] The second sublayer contains three two-dimensional convolutional layers, one mish function activation layer and one residual network layer. The number of convolution kernels in the three two-dimensional convolutional layers is 32 and the size of the convolution kernel is 1×1. The residual network layer contains two two-dimensional convolutions, and the number of their convolution kernels is 32 and 64 respectively, and the size of their convolution kernels is 1×1 and 3×3 respectively.

[0170] The third sublayer contains three two-dimensional convolutional layers, one mish function activation layer and two residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 64 and the size of the convolution kernels is 1×1. The two residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 64 and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0171] The fourth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 128, and the size of the convolution kernels is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 128, and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0172] The fifth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 256, and the size of the convolution kernels is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 256, and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0173] The sixth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 256, and the size of the convolution kernel is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 512, and the sizes of the convolution kernels are 1×1 and 3×3 respectively.

[0174] The outputs of the 2nd to 6th sublayers are defined by the following formulas:

[0175] Output=mish(conv3(concat(conv1(Input), Resblock n (conv2(Input))))),

[0176] In the formula, Resblock n represents the residual network layer in the nth sublayer, n∈{1, 2, ..., 6}, Input represents the input of the corresponding sublayer, conv1, conv2, conv3 represent the three two-dimensional convolutional layers in the corresponding sublayer, concat represents the connection operation, and Output represents the output of the corresponding sublayer;

[0177] The seventh sublayer contains three maximum pooling layers, whose pooling kernel sizes are 3×3, 5×5, and 7×7 respectively. The output of this sublayer is defined by the following formula:

[0178] Output=concat(Input, maxpool1(Input), maxpool2(Input), maxpool3(Input)),

[0179] In the formula, Input represents the input of the sub-layer, maxpool1, maxpool2, and maxpool3 represent three maximum pooling layers respectively, concat represents the connection operation, and Output represents the output of the sub-layer;

[0180] Step S32: construct a feature fusion layer for the initial model of the detection method SSTD-MNN, which includes the following 5 sub-layers:

[0181] The first sublayer contains four two-dimensional convolutional layers, whose convolution kernel numbers are 512, 1024, 512, and 256, and whose convolution kernel sizes are 1×1, 3×3, 1×1, and 1×1, respectively. The output of the first sublayer is defined by the following formula:

[0182] Output = conv 1-4 (Input),

[0183] In the formula, Input represents the input of the sublayer, conv 1-4 Represents the 1st to 4th two-dimensional convolutional layers, and Output represents the output of this sublayer;

[0184] The second sublayer contains 1 upsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 256, 256, 512, 256, 512, 256 respectively, and the sizes of convolution kernels are 1×1, 1×1, 3×3, 1×1, 3×3, 1×1 respectively. The sampling multiple of the upsampling layer is 4. The output of this sublayer is defined by the following formula:

[0185] Output = ea (conv 2-6 (concat(upsample(Input),conv1(Output S31-5 )))),

[0186] In the formula, Input represents the input of the sublayer, upsample represents the upsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S31-5 represents the output of the fifth sublayer in step S31, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer;

[0187] The third sublayer contains 1 upsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 128, 128, 256, 128, 256, 128, respectively, and the sizes of convolution kernels are 1×1, 1×1, 3×3, 1×1, 3×3, 1×1, respectively. The sampling multiple of the upsampling layer is 4. The output of this sublayer is defined by the following formula:

[0188] Output = ea (conv 2-6(concat(upsample(Input),conv1(Output S31-4 )))),

[0189] In the formula, Input represents the input of this sublayer, conv1 represents the first two-dimensional convolutional layer, and Output S31-4 represents the output of the fourth sublayer in step S31, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer;

[0190] The fourth sublayer contains 1 downsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 256, 256, 512, 256, 512, 256 respectively, and the sizes of convolution kernels are 3×3, 1×1, 3×3, 1×1, 3×3, 1×1 respectively. The sampling multiple of the downsampling layer is 1 / 4. The output of this sublayer is defined by the following formula:

[0191] Output = ea (conv 2-6 (concat(conv1(downsample(Input))),Output S32-2 ))),

[0192] In the formula, Input represents the input of the sublayer, downsample represents the downsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S32-2 represents the output of the second sublayer in step S32, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer;

[0193] The fifth sublayer contains one downsampling layer and six two-dimensional convolutional layers. The number of convolution kernels of the six two-dimensional convolutional layers are 512, 512, 1024, 512, 1024, 512, respectively, and the sizes of the convolution kernels are 3×3, 1×1, 3×3, 1×1, 3×3, 1×1, and 1×1, respectively. The sampling multiple of the downsampling layer is 1 / 4. The output of this sublayer is defined by the following formula:

[0194] Output = conv 2-6 (concat(conv1(downsample(Input))),Output S32-1 )),

[0195] In the formula, Input represents the input of the sublayer, downsample represents the downsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S32-1 represents the output of the first sublayer in step S32, concat represents the connection operation, conv 2-6 Represents the 2nd to 6th two-dimensional convolutional layer, and Output represents the output of this sublayer;

[0196] Step S33: construct an output layer for the initial model of the detection method SSTD-MNN, which includes the following three sub-layers:

[0197] The first sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 256 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer.

[0198] The second sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 512 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer.

[0199] The third sublayer includes two two-dimensional convolutional layers, whose convolution kernel numbers are 1024 and 3×5+c respectively, and whose convolution kernel sizes are 3×3 and 1×1 respectively, wherein the hyperparameter c represents the number of ship target types in the SAR image, and its value is any positive integer; in this embodiment, the value of c is 1;

[0200] Step S34: performing calculations based on the prior frame obtained in step S23 and all outputs of the output layer to obtain the actual value of the ship target detection coordinates, specifically including:

[0201] First, the n priori frames obtained in step S23 are equally divided into three parts. Then, the three sublayers in the output layer are sequentially associated with the three priori frames, and the n / 3 outputs of each sublayer are sequentially corresponded to the n / 3 priori frames associated with the sublayer. Finally, the initial value of the ship target detection coordinate in each output of each sublayer is sequentially calculated with the width and height of the priori frame corresponding to the output to obtain the actual value of the ship target detection coordinate. The calculation is defined by the following formula:

[0202]

[0203]

[0204]

[0205]

[0206] In the formula, t x ,t y ,t w ,t h They represent the abscissa, ordinate, width and height of the initial value of the ship target detection coordinates in the output of the sublayer, respectively. x , b y , b w , b h They represent the abscissa, ordinate, width and height of the actual values ​​of the ship target detection coordinates obtained by calculation, respectively. w 、p h Respectively represent the width and height of the corresponding prior box, c x 、c y They represent the horizontal and vertical distances between the output of the sublayer and the upper left corner of the preprocessed SAR image, respectively;

[0207] Step S35: performing non-maximum suppression on the calculation result of step S34 with an intersection-over-union ratio threshold of threshold1, wherein the hyperparameter threshold1 takes a value in an open interval (0, 1); in this embodiment, the value of threshold1 is 0.75;

[0208] Step S4: using the validation set and the preprocessed training set to adjust the parameters of the initial model of the SSTD-MNN to obtain an optimized model of the SSTD-MNN, and using the test set to evaluate the detection accuracy of the optimized model; the step S4 further includes:

[0209] Step S41: First, the intersection and union ratio of each output of the initial model of the SSTD-MNN and the true value of each ship target is calculated one by one, and then, for each true value of the ship target, the first output with the maximum intersection and union ratio with it is recorded as the correct output, and if the maximum value of the intersection and union ratio of a certain output with the true values ​​of all ship targets among the remaining outputs is less than the threshold threshold2, then the output is recorded as an incorrect output, wherein the hyperparameter threshold2 takes values ​​in the open interval (0, 1); in this embodiment, the value of threshold2 is 0.5;

[0210] Step S42: construct a loss function for the initial model of the SSTD-MNN, and calculate the loss of all outputs of the model before performing gradient propagation. The loss function is defined by the following formula:

[0211] loss=I obj CIoU(pred coord ,gt coord )+I obj BCE(predcls ,gt cls )+BCE(pred conf ,gt conf ),

[0212] In the formula, I obj Indicates whether it is the correct output, its value is 1 for the correct output, and its value is 0 for the rest of the output; CIoU represents the CIoU loss function; BCE represents the binary cross entropy function; pred coord represents the actual value of the ship target detection coordinates in the output of the output layer, gt coord Indicates the coordinate value of the ship target in the real value, pred cls Indicates the type of ship target detection object in the output of the output layer, gt cls Indicates the type of ship target object in the true value of the ship target, pred conf represents the confidence in the output of the output layer, gt conf Indicates whether it is an erroneous output, with a value of 0 indicating an erroneous output and a value of 1 indicating the remaining output;

[0213] Step S43: Use the Adam optimizer with a learning rate of η to adjust the parameters of the initial model of the SSTD-MNN, where the hyperparameter η is taken from the elements in the set {0.0005, 0.001, 0.0015, 0.002}, and calculate the loss function on the validation set. If the function converges, the parameter adjustment process is terminated to obtain the optimized model of the SSTD-MNN, otherwise, steps S41, S42 and S43 are repeated in sequence; in this embodiment, the value of η is 0.001, and the parameter adjustment is completed after 120 repetitions of steps S41, S42 and S43 to obtain the optimized model of the SSTD-MNN;

[0214] Step S44: Use the test set and the two performance evaluation indicators of precision and recall to evaluate the detection accuracy of the obtained SSTD-MNN optimization model; in this embodiment, the detection accuracy results of the SSTD-MNN optimization model are shown in Table 1, which also lists the detection accuracy results of the YOLOv4 optimization model.

[0215] Table 1. Comparison of detection accuracy experimental results of detection methods SSTD-MNN and YOLOv4 on SSDD dataset

[0216] Detection Methods Precision Recall SSTD-MNN 90.94％ 92.91％ YOLOv4 89.31％ 89.31％

[0217] Step S5: using the obtained SSTD-MNN optimization model to perform ship target detection on the SAR images to be detected in the SAR image dataset or in actual applications; in this embodiment, SAR image ship target detection is performed on the test set.

[0218] The above experimental results show that compared with the YOLOv4 method, the SAR image ship target detection method SSTD-MNN based on a multi-layer neural network provided by the present invention has higher detection accuracy.

[0219] In general, it can be understood from the above technical scheme of the present invention and its specific implementation methods that the beneficial technical effects of the present invention mainly include three aspects: (1) The present invention provides a SAR image ship target detection method with high detection accuracy, which has certain scientific significance for the development of SAR image ship target detection technology; (2) Compared with the existing SAR image ship target detection method, the method of the present invention has the advantage of high detection accuracy. The reason is that the technical scheme of the present invention organically combines the attention mechanism with the convolutional neural network to form a specific multi-layer neural network, which has stronger graphic feature processing capabilities; (3) The method of the present invention has broad application prospects in many aspects such as disaster monitoring, environmental monitoring, ocean monitoring, resource exploration, crop yield estimation, surveying and mapping, and military.

[0220] The above is only part of the embodiments of the present invention. It should be noted that, except for the technical features described in the specification, all other technical features are known to those skilled in the art. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A method for detecting ship targets in SAR images based on a multi-layer neural network, comprising the following steps: Step S1: dividing the SAR image data set containing the SAR images and the true values ​​of the corresponding ship targets into a training set, a validation set, and a test set; Step S2: preprocessing the training set, including size standardization, data enhancement and prior box clustering of SAR images; Step S3: Using the validation set and the preprocessed training set, an initial model of the SAR image ship target detection method SSTD-MNN based on a multi-layer neural network is constructed; specifically as follows: Constructing a feature extraction layer for the initial model of the SSTD-MNN; Constructing a feature fusion layer for the initial model of the SSTD-MNN; Constructing an output layer for the initial model of the SSTD-MNN; Perform calculations based on the a priori box clustering result obtained in step S2 and all outputs of the output layer to obtain actual values ​​of the ship target detection coordinates; Performing non-maximum suppression on the operation result; Step S4: using the validation set and the preprocessed training set to adjust the parameters of the initial model of the SSTD-MNN to obtain an optimized model of the SSTD-MNN, and using the test set to evaluate the detection accuracy of the optimized model; Step S5: Use the obtained SSTD-MNN optimization model to perform ship target detection on the SAR images to be detected in the SAR image dataset or in actual applications.

2. The method according to claim 1, characterized in that The step S2 further comprises the following steps: Step S21: the sizes of all SAR images and their true values ​​of ship targets in the training set are standardized to 608 pixels × 608 pixels; Step S22: first, for each SAR image in the training set, three other SAR images in the training set are randomly selected, and the four SAR images are randomly scaled and placed at four corners of a blank image to splice into a new SAR image. Then, the SAR image and the true values ​​of the ship targets corresponding to the original four SAR images are edge processed to obtain the preprocessed SAR image and the true values ​​of the ship targets. Step S23: Perform K-means clustering on the true values ​​of all ship targets in the training set to obtain n prior boxes, where the hyperparameter n is taken from the elements in the set {6, 9, 12}.

3. The method according to claim 1 or 2, characterized in that: The step S3 further comprises the following steps: Step S31: construct a feature extraction layer for the initial model of the detection method SSTD-MNN, which includes the following 7 sub-layers: The first sublayer contains two 2D convolutional layers and one mish function activation layer. The number of convolution kernels in the two 2D convolutional layers is 32 and 64 respectively, and the sizes of convolution kernels are 1×1 and 3×3 / 2 respectively. The output of this sublayer is defined by the following formula: Output=mish(conv2(conv1(Input))), In the formula, Input represents the input of the sub-layer, conv1 and conv2 represent two two-dimensional convolutional layers respectively, and Output represents the output of the sub-layer; The second sublayer contains three two-dimensional convolutional layers, one mish function activation layer and one residual network layer. The number of convolution kernels in the three two-dimensional convolutional layers is 32 and the size of the convolution kernel is 1×1. The residual network layer contains two two-dimensional convolutions, and the number of their convolution kernels is 32 and 64 respectively, and the size of their convolution kernels is 1×1 and 3×3 respectively. The third sublayer contains three two-dimensional convolutional layers, one mish function activation layer and two residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 64 and the size of the convolution kernels is 1×1. The two residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 64 and the sizes of the convolution kernels are 1×1 and 3×3 respectively. The fourth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 128, and the size of the convolution kernels is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 128, and the sizes of the convolution kernels are 1×1 and 3×3 respectively. The fifth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 256, and the size of the convolution kernels is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 256, and the sizes of the convolution kernels are 1×1 and 3×3 respectively. The sixth sublayer contains three two-dimensional convolutional layers, one mish function activation layer, and eight residual network layers. The number of convolution kernels in the three two-dimensional convolutional layers is 256, and the size of the convolution kernel is 1×1. The eight residual network layers contain two two-dimensional convolutions, and the number of their convolution kernels is 512, and the sizes of the convolution kernels are 1×1 and 3×3 respectively. The outputs of the 2nd to 6th sublayers are defined by the following formulas: Output=mish(conv3(concat(conv1(Input),Resblock n (conv2(Input))))), In the formula, Resblock n represents the residual network layer in the nth sublayer, n∈{1, 2, ..., 6}, Input represents the input of the corresponding sublayer, conv1, conv2, conv3 represent the three two-dimensional convolutional layers in the corresponding sublayer, concat represents the connection operation, and Output represents the output of the corresponding sublayer; The seventh sublayer contains three maximum pooling layers, whose pooling kernel sizes are 3×3, 5×5, and 7×7 respectively. The output of this sublayer is defined by the following formula: Output = concat (Input, maxpool1 (Input), maxpool2 (Input), maxpool3 (Input)), where Input represents the input of the sub-layer, maxpool1, maxpool2, and maxpool3 represent three maximum pooling layers, concat represents the connection operation, and Output represents the output of the sub-layer; Step S32: construct a feature fusion layer for the initial model of the detection method SSTD-MNN, which includes the following 5 sub-layers: The first sublayer contains four two-dimensional convolutional layers, whose convolution kernel numbers are 512, 1024, 512, and 256, and whose convolution kernel sizes are 1×1, 3×3, 1×1, and 1×1, respectively. The output of the first sublayer is defined by the following formula: Output=conv 1-4 (Input), In the formula, Input represents the input of the sublayer, conv 1-4 Represents the 1st to 4th two-dimensional convolutional layers, and Output represents the output of this sublayer; The second sublayer contains 1 upsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 256, 256, 512, 256, 512, 256 respectively, and the sizes of convolution kernels are 1×1, 1×1, 3×3, 1×1, 3×3, 1×1 respectively. The sampling multiple of the upsampling layer is 4. The output of this sublayer is defined by the following formula: Output=ea(conv 2-6 (concat(upsample(Input),conv1(Output S31-5 )))), In the formula, Input represents the input of the sublayer, upsample represents the upsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S31-5 represents the output of the fifth sublayer in step S31, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer; The third sublayer contains 1 upsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 128, 128, 256, 128, 256, 128, respectively, and the sizes of convolution kernels are 1×1, 1×1, 3×3, 1×1, 3×3, 1×1, respectively. The sampling multiple of the upsampling layer is 4. The output of this sublayer is defined by the following formula: Output=ea(conv 2-6 (concat(upsample(Input),conv1(Output S31-4 )))), In the formula, Input represents the input of this sublayer, conv1 represents the first two-dimensional convolutional layer, and Output S31-4 represents the output of the fourth sublayer in step S31, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer; The fourth sublayer contains 1 downsampling layer, 6 2D convolutional layers and 1 external attention layer. The number of convolution kernels of the 6 2D convolutional layers are 256, 256, 512, 256, 512, 256 respectively, and the sizes of convolution kernels are 3×3, 1×1, 3×3, 1×1, 3×3, 1×1 respectively. The sampling multiple of the downsampling layer is 1 / 4. The output of this sublayer is defined by the following formula: Output=ea(conv 2-6 (concat(conv1(downsample(Input)),Output S32-2 ))), In the formula, Input represents the input of the sublayer, downsample represents the downsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output S32-2 represents the output of the second sublayer in step S32, concat represents the connection operation, conv 2-6 represents the 2nd to 6th 2D convolutional layers, ea represents the external attention layer, and Output represents the output of this sublayer; The fifth sublayer contains one downsampling layer and six two-dimensional convolutional layers. The number of convolution kernels of the six two-dimensional convolutional layers are 512, 512, 1024, 512, 1024, 512, respectively, and the sizes of the convolution kernels are 3×3, 1×1, 3×3, 1×1, 3×3, 1×1, and 1×1, respectively. The sampling multiple of the downsampling layer is 1 / 4. The output of this sublayer is defined by the following formula: Output=conv 2-6 (concat(conv1(downsample(Input)),Output S32-1 )), In the formula, Input represents the input of the sublayer, downsample represents the downsampling layer, conv1 represents the first two-dimensional convolutional layer, and Output s32-1 represents the output of the first sublayer in step S32, concat represents the connection operation, conv 2-6 Represents the 2nd to 6th two-dimensional convolutional layer, and Output represents the output of this sublayer; Step S33: construct an output layer for the initial model of the detection method SSTD-MNN, which includes the following three sub-layers: The first sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 256 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer. The second sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 512 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer. The third sublayer contains two 2D convolutional layers, whose convolution kernel numbers are 1024 and 3×5+c, and whose convolution kernel sizes are 3×3 and 1×1, respectively. The hyperparameter c represents the number of ship target types in SAR images, and its value can be any positive integer. Step S34: performing calculations based on the prior frame obtained in step S23 and all outputs of the output layer to obtain the actual value of the ship target detection coordinates, specifically including: First, the n priori frames obtained in step S23 are equally divided into three parts. Then, the three sublayers in the output layer are sequentially associated with the three priori frames, and the n / 3 outputs of each sublayer are sequentially corresponded to the n / 3 priori frames associated with the sublayer. Finally, the initial value of the ship target detection coordinate in each output of each sublayer is sequentially calculated with the width and height of the priori frame corresponding to the output to obtain the actual value of the ship target detection coordinate. The calculation is defined by the following formula: Where, t x ,t y ,t w ,t h They represent the abscissa, ordinate, width and height of the initial value of the ship target detection coordinates in the output of the sublayer, respectively. x 、b y 、b w 、b h They represent the abscissa, ordinate, width and height of the actual values ​​of the ship target detection coordinates obtained by calculation, respectively. w 、p h Respectively represent the width and height of the corresponding prior box, c x 、c y They represent the horizontal and vertical distances between the output of the sublayer and the upper left corner of the preprocessed SAR image, respectively; Step S35: Perform non-maximum suppression on the calculation result of step S34 with an intersection-over-union ratio threshold of threshold1, where the hyperparameter threshold1 takes values ​​in an open interval (0, 1).

4. The method according to claim 3, characterized in that The step S4 further comprises the following steps: Step S41: First, the intersection and union ratios of each output of the initial model of the SSTD-MNN and the true value of each ship target are calculated one by one, and then, for each true value of the ship target, the first output with the maximum intersection and union ratio with it is recorded as the correct output, and if the maximum value of the intersection and union ratios of a certain output with the true values ​​of all ship targets among the remaining outputs is less than a threshold value threshold2, then the output is recorded as an incorrect output, wherein the hyperparameter threshold2 takes values ​​in the open interval (0, 1); Step S42: construct a loss function for the initial model of the SSTD-MNN, and calculate the loss of all outputs of the model before performing gradient propagation. The loss function is defined by the following formula: loss=I obi CIoU(pred coord ,gt coord )+I obj BCE(pred cls ,gt cls )+BCE(pred conf ,gt conf ), where I obj Indicates whether it is the correct output, its value is 1 for the correct output, and its value is 0 for the rest of the output; CIoU represents the CIoU loss function; BCE represents the binary cross entropy function; pred coord represents the actual value of the ship target detection coordinates in the output of the output layer, gt coord Indicates the coordinate value of the ship target in the real value, pred cls Indicates the type of ship target detection object in the output of the output layer, gt cls Indicates the type of ship target object in the true value of the ship target, pred conf Represents the confidence in the output of the output layer, gt conf Indicates whether it is an erroneous output, with a value of 0 indicating an erroneous output and a value of 1 indicating the remaining output; Step S43: Use the Adam optimizer with a learning rate of η to adjust the parameters of the initial model of the SSTD-MNN, where the hyperparameter η is taken from the elements in the set {0.0005, 0.001, 0.0015, 0.002), and calculate the loss function on the validation set. If the function converges, the parameter adjustment process is terminated to obtain the optimized model of the SSTD-MNN. Otherwise, steps S41, S42 and S43 are repeated in sequence. Step S44: using the test set and two performance evaluation indicators, namely, precision and recall, to evaluate the detection accuracy of the obtained SSTD-MNN optimization model.

Images