A method and system for sea surface target detection and recognition based on deep learning

Abstract

The present application provides a method and system for detecting and identifying sea targets based on deep learning, which includes: obtaining multiple images as original image data and annotating the borders and categories; performing data enhancement on the original image data, and using the newly generated images and the collected images as training images; modifying some convolution processing layers in the feature extraction network Darknet53 to Inception modules and ResNet modules, and using the modified feature extraction network to extract feature maps of the training images; sending the feature maps to the multi-scale detection network, generating prediction frames of different scales and different aspect ratios on the feature maps to fit the target to be detected in the image, and the multi-scale detection network includes at least four detection scales; using NMS to remove redundant prediction frames from the prediction frames to obtain a unique identification frame of the target to be detected; iteratively training the entire model of the modified feature extraction network, the multi-scale detection network and the NMS, and finally outputting the entire model. Effectively improve the detection rate of small targets and reduce missed detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and specifically to a method and system for sea surface target detection and recognition based on deep learning. Background Art

[0002] Object detection technology is a key technology in the field of computer vision. Its main purpose is to locate objects in images and accurately output their locations. At present, commonly used methods include methods based on manual features and methods based on deep learning. Among them, the method based on deep learning is to use convolutional neural networks to learn the features of different objects from a large amount of image data, and perform fitting regression with the labeled data to achieve the purpose of detecting the required objects. The YOLO algorithm first normalizes the input image to a fixed scale, and then divides the image into S*S grids, such as 13*13. If the center of an object falls in this grid, then this grid is responsible for predicting the object. YOLOv3 uses multi-scale detection. After convolution, the confidence of the target and the probability that the border area belongs to different categories will be output as the detection result of each grid. The detection framework of traditional YOLOv3 is as shown in the attached manual. Figure 1 shown.

[0003] Since the resolution of ocean targets is generally smaller, they generally contain fewer pixels. In addition, the sea surface targets are closely arranged. If the grid is too large, a special detection problem may occur, that is, two or more small ship targets fall into the same grid because they are arranged too closely. In order to improve this situation without affecting other detections, it is necessary to modify the original network structure accordingly, add more scales, and use a larger scale input layer.

[0004] Therefore, it is necessary to further improve the sea surface target detection and recognition method based on deep learning to meet various needs in practical applications. Summary of the invention

[0005] The embodiment of the present application proposes a method and system for sea surface target detection and recognition based on deep learning to solve the technical problems mentioned in the above background technology section.

[0006] In a first aspect, an embodiment of the present application provides a method for detecting and identifying sea surface targets based on deep learning, the method comprising the following steps:

[0007] S110, obtaining multiple images as original image data, and marking the borders and categories of the detection targets in the original image data;

[0008] S120, performing data enhancement operation on the original image data, and using the newly generated image together with the image collected in step S110 as training images;

[0009] S130, modifying some convolution processing layers in the feature extraction network Darknet53 into Inception modules and ResNet modules, and using the modified feature extraction network to extract feature maps of the training images;

[0010] S140, sending the feature map extracted in step S130 to a multi-scale detection network, generating prediction boxes of different scales and different aspect ratios on the feature map to fit the target to be detected in the picture, wherein the multi-scale detection network includes at least four detection scales;

[0011] S150, using non-maximum suppression (NMS) to remove redundant prediction frames from the prediction frame to obtain a unique identification frame of the target to be measured; and

[0012] S160, repeating steps S130-S150 to iteratively train the entire model including the modified feature extraction network, the multi-scale detection network and the non-maximum suppression NMS, and finally outputting the entire model.

[0013] This method can effectively improve the detection rate of small targets and is suitable for sea surface ship detection scenarios.

[0014] In some embodiments, in step S130, the processing process of the Inception module and the ResNet module is: after fusing features of multiple different convolutional layers, use a 1*1 convolutional layer to unify the values ​​of all output channels, and finally use the ResNet residual structure to obtain the overall module.

[0015] In some embodiments, the ResNet residual structure specifically includes 2 CBR units and 2 ResNet units. Adding the ResNet residual structure in the detection part can avoid gradient vanishing and enhance feature reuse, which is more suitable for the detection of small and closely arranged ships.

[0016] In some embodiments, in step S140, the feature map extracted in step S130 is sent to a multi-scale detection network, and the multi-scale detection network adopts a first improved SSD model. The first improved SSD model retains the structure of conv 1_1 to conv 5_3 in the VGG 16 structure, and adds conv7, conv8, conv9 and conv10 convolution layers; fitting the target to be tested in the picture is performed by deep feature fusion, which specifically includes the following steps:

[0017] S141, perform a deconvolution operation on the feature map of the conv 4_3 layer to obtain an enlarged Deconv 4 image, and use feature mapping to add the feature correspondence of the feature map of the conv 4_3 layer to the Deconv 4 image;

[0018] S142, performing a pooling operation on the Deconv 4 image after feature mapping to obtain a Pool4 feature map;

[0019] S143, upsample the feature map of the conv 11_2 layer to obtain the conv'11 feature map;

[0020] S144, fuse the features of the conv'11 feature map and the Pool 4 feature map to the improved conv'4 feature map, and finally concatenate and fuse the conv'4 feature map with the feature maps of conv4_3, conv7, conv 8_2, conv9_2, and conv 10_2 extracted from the first improved SSD model.

[0021] Compared with the traditional SSD network model, this deep feature fusion method adds more semantic information, enhances the image feature information to a certain extent, and can use more complex feature maps to improve the performance of small objects.

[0022] In some embodiments, the feature map of the conv 4_3 layer is added to the deconv 4 image using feature mapping, and the following formula is used for feature mapping:

[0023] x i+1 =stride i *x i +((kernel_size-1) / 2-padding))

[0024] Among them, x i and x i+1 They respectively refer to the coordinate points of the input and output of the feature map of the i-th layer, and stridei is the step size of the i-th layer deconvolution.

[0025] Feature mapping is to map data from the original space to other feature spaces, keeping the same distribution unchanged, but with new features. This formula can be used to transform the coordinates of each pixel position in the original feature map.

[0026] In some embodiments, in step S140, the target to be tested in the fitting picture is to use the second improved SSD model to fuse adjacent layer features. The second improved SSD model removes conv 10 and conv 11 of the original SSD model, retains the shallow feature conv 4_3, and fuses the remaining layers in pairs to form new detection layers newconv 7, newconv7, newconv8 and newconv9.

[0027] Complex semantic information can be integrated into the detection image, and detection through the second improved SSD model can achieve better performance and improve the accuracy of small objects to a certain extent. At the same time, compared with the FasterR-CNN type that requires candidate boxes, this method reduces the training process of the region proposal network and is easier from a training perspective.

[0028] In some embodiments, in step S150, non-maximum suppression (NMS) is used to remove redundant prediction boxes from the prediction box. Specifically, the confidence is updated by weight attenuation. The formula used is as follows:

[0029]

[0030] Among them, ci represents the confidence, iou(I, b k ) represents the intersection and union ratio.

[0031] By using this formula to process the confidence level accordingly according to the different iou, the confidence level of false positive detection results can be effectively reduced, which helps to improve the accuracy of detection.

[0032] In some embodiments, in step S140, the multi-scale detection network adds at least one scale for detecting small targets on the basis of extracting three-scale feature information of one upsampling and two downsampling in the original YOLOv3 algorithm; the target object is identified by using the k-means clustering algorithm, wherein the objective function used by the anchor clustering is as follows:

[0033]

[0034] Among them, D refers to the sample data, G and K represent the center coordinates of each cluster and the total number of clusters, and IOU (D, G) is the intersection-union ratio between the center box of the cluster and the cluster box.

[0035] On the basis of the three detection scales, multiple detection scales are added to improve the recall rate and detection accuracy of the detection network for small targets. The comparative relationship between the number and the number can be obtained through cluster analysis, and the appropriate anchor that fits the small ship target can be found.

[0036] In some embodiments, in step S120, the data enhancement operation includes horizontal mirroring, image rotation, random cropping, random brightness and contrast enhancement. The data enhancement operation is performed on the original image data to increase the generalization ability.

[0037] In the second aspect, the present application provides a sea surface target detection and recognition system based on deep learning, which includes the following modules:

[0038] An acquisition module is used to acquire multiple images as raw image data, and to annotate the borders and categories of the detection targets in the raw image data; and

[0039] A preprocessing module is used to perform data enhancement operations on the original image data, and use the newly generated pictures together with the pictures collected in the acquisition module as training pictures; and

[0040] A feature extraction module, used to modify some convolution processing layers in the feature extraction network Darknet53 into Inception modules and ResNet modules, and use the modified feature extraction network to extract feature maps of the training images; and

[0041] A detection module, which is used to send the feature map extracted by the feature extraction module into a multi-scale detection network, and generate prediction boxes of different scales and different aspect ratios on the feature map to fit the target to be detected in the picture, wherein the multi-scale detection network includes at least four detection scales; and

[0042] A redundancy removal module is used to remove redundant prediction boxes using non-maximum suppression (NMS) to obtain a unique identification box of the target to be measured; and

[0043] The output module is used to run the feature extraction module, the detection module and the redundancy removal module, iteratively train the entire model composed of the modified feature extraction network, the multi-scale detection network and the non-maximum suppression NMS, and finally output the entire model.

[0044] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above methods is implemented.

[0045] The embodiment of the present application provides a method and system for sea surface target detection and recognition based on deep learning. On the basis of the original YOLOv3 network model, the darknet-53 network is optimized, and the convolutional layer in the original network is replaced by Inception v4 to make the training network deeper and wider; a ResNet structure is added to the final detection part to enhance feature reuse, and at the same time, a larger scale is added on the basis of retaining the original multi-target detection, so that the model can well preserve the features of large and small targets; the original NMS algorithm is improved by using an attenuation-based function, so that the average accuracy of the overall detection is significantly improved, and the detection effect on small targets of closely arranged ships is also correspondingly improved; and the existing SSD algorithm is improved, and an improved SSD model network structure based on feature map fusion is proposed, which combines the two fusion methods of deep feature fusion and adjacent layer feature fusion to improve the accuracy of algorithm calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0047] Figure 1 It is a schematic diagram of the detection framework of YOLOv3 in the prior art;

[0048] Figure 2 is a flow chart of a method for detecting and identifying sea surface targets based on deep learning according to a specific embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of an Inception module + ResNet module according to a specific embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a detection framework of an improved YOLOv3 according to a specific embodiment of the present invention;

[0051] Figure 5a This is a schematic diagram of the output structure of the original YOLO v3 network;

[0052] Figure 5b is a schematic diagram of the output structure of an improved YOLO v3 network according to a specific embodiment of the present invention;

[0053] Figure 5c is a schematic diagram of a ResNet structure according to a specific embodiment of the present invention;

[0054] Figure 6 is a schematic diagram of a first improved SSD model structure according to a specific embodiment of the present invention;

[0055] Figure 7 is a schematic diagram of a deep feature fusion structure according to a specific embodiment of the present invention;

[0056] Figure 8 is a schematic diagram of a second improved SSD model structure according to a specific embodiment of the present invention;

[0057] Fig. 9 is a schematic diagram of a feature fusion structure of adjacent layers according to a specific embodiment of the present invention;

[0058] Fig.10a is a schematic diagram of a loss value function curve according to a specific embodiment of the present invention;

[0059] Fig.10b is a schematic diagram of an Avg IOU curve according to a specific embodiment of the present invention;

[0060] Fig.11aThis is a schematic diagram of the PR curve of the original YOLO v3 network;

[0061] Fig.11b is a schematic diagram of a PR curve of an improved YOLO v3 network according to a specific embodiment of the present invention;

[0062] Fig.12a This is a schematic diagram of the effect of using the original NMS algorithm for detection;

[0063] Figure 12b This is a schematic diagram of the effect of using the Soft-NMS algorithm for detection;

[0064] Fig.12c is a schematic diagram of the effect of performing detection according to an improved NMS algorithm according to a specific embodiment of the present invention;

[0065] Fig.13 is a visualization feature diagram of each stage according to a specific embodiment of the present invention;

[0066] Fig.14 It is a schematic diagram of a method and system for detecting and identifying sea surface targets based on deep learning according to the present invention;

[0067] Fig.15 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0068] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0069] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0070] Figure 2 FIG. 1 is a flow chart of a method for detecting and identifying sea surface targets based on deep learning according to a specific embodiment of the present invention. Figure 2 As shown, the method includes

[0071] S110, obtaining multiple images as original image data, and marking the borders and categories of the detection targets in the original image data;

[0072] In this embodiment, taking a sample of a sea surface ship image as an example, a picture containing a sea surface ship is selected as original image data, and then the original image data is annotated with a border and a category using annotation software.

[0073] S120, performing data enhancement operation on the original image data, and using the newly generated image together with the image collected in step S110 as training images;

[0074] In this embodiment, the data enhancement operation includes horizontal mirroring, image rotation, random cropping, random brightness and contrast enhancement. The data enhancement operation is performed on the original image data to increase the generalization ability.

[0075] S130, modifying some convolution processing layers in the feature extraction network Darknet53 into Inception modules and ResNet modules, and using the modified feature extraction network to extract feature maps of the training images;

[0076] In this embodiment, Figure 3 FIG. 1 shows a schematic diagram of an Inception module + ResNet module according to a specific embodiment of the present invention. Figure 3 Replace the "Inception module + ResNet module" shown Figure 1 The left half of the original YOLOv3 detection framework shows some convolution processing layers, which changes part of the "Convolutional" in the original network into the "Inception" module. The specific processing process of the Inception module and the ResNet module is: after fusing the features of multiple different convolutional layers (including 1*1 convolutional layers and 3*3 convolutional layers), use the 1*1 convolutional layer to unify the values ​​of all output channels, and finally use the ResNet residual structure to obtain the overall module.

[0077] The detection framework of YOLOv3 after the overall improvement by replacing some convolutional processing layers with the "Inception module + ResNet module" is shown as follows Figure 4 As shown in the figure, the modified feature extraction network adds a detection scale compared to the traditional Darknet53 network, and adds a ResNet residual structure to the final detection part to adapt to the detection of small and closely arranged sea surface ships.

[0078] Figure 5a The output structure diagram of the original YOLO v3 network is shown in Figure 1. Figure 5a As shown in Figure 1, the target detection output layer of the original YOLOv3 network contains 6 CBR (Conv+BN+Relu) units and 1 1*1 convolution layer. This application changes the original 6 CBR (Conv+BN+Relu) units to 2 CBR (Conv+BN+Relu) units and 2 ResNet units, as shown in Figure 1. Figure 5bAs shown. Adding the ResNet residual structure in the detection part can avoid gradient vanishing and enhance feature reuse. For the specific ResNet residual structure, please refer to Figure 5c .

[0079] S140, sending the feature map extracted in step S130 to a multi-scale detection network, generating prediction boxes of different scales and different aspect ratios on the feature map to fit the target to be detected in the picture, wherein the multi-scale detection network includes at least four detection scales;

[0080] In this embodiment, in order to improve the recall rate and detection accuracy of the detection network for small targets, information of multiple scales is used in the detection network. Based on the three-scale feature information extraction of 1 upsampling and 2 downsampling in the original YOLOv3 algorithm, according to the clustering results of the data set of this application, a scale is added to detect small targets, which can continuously fuse the image information on the feature map to a certain extent, and help solve the problem of two different objects in the same YOLO small grid. The k-means clustering algorithm is used to identify the target object. Specifically, the formula (1) of the objective function S used by anchor clustering is as follows:

[0081]

[0082] Wherein, D refers to the sample data data, G and K represent the center coordinates of each cluster and the total number of clusters respectively. IOU(D,G) is the intersection-union ratio calculated between the center box of the cluster and the cluster box. After cluster analysis, the comparative relationship between the number and can be obtained. According to multiple experiments by the inventors of this application, it is found that when k>9, the numerical growth trend slows down, and it tends to be stable when k=12. Preferably, the anchor is set to 12 and four different scales are set, and the partial results (8) of the anchor box size K and the ratio size R used are shown in the following table.

[0083]

[0084] Compared with general target detection, the "smallness" of remote sensing small targets is more intuitive due to the perspective, and its semantic features should be easier to reflect on the shallow feature map. Remote sensing images have a high resolution, and the resolution will remain at a high value without a large amount of convolution pooling. Therefore, the commonly used method to construct complex feature maps is feature fusion.

[0085] In order to achieve good detection performance, this application proposes a feature fused SSD structure in two methods, also known as FFSSD (feature map fused SSD). The first method is to fuse the features at the deep layer with the shallow features, and the second method is the fusion between adjacent feature layers. Through these two fusion methods, the semantic information of the feature map can be well supplemented to achieve the purpose of improving the small target detection performance. The following article will use deconvolution operations to increase the resolution of the feature map in view of the property that the shallow feature map reflects the small target features, and use this method to fuse and establish a new feature map to further improve the accuracy of small target detection.

[0086] Different feature maps need to set a certain number of prior boxes. If the number of feature maps used for prediction is m, the initial default box size is 0.1 times the input size, the second is 0.2 times, and from the third onwards, the sizes of the remaining default boxes can be calculated using formula (2). The size needs to be linearly transformed according to the formula.

[0087]

[0088] Among them, S max and S min The values ​​of are 0.9 and 0.2, respectively, representing the scale of the highest and lowest feature maps. The number of feature maps of the two methods in this application is 6 and 4, respectively. According to formula (2), the experimental parameters corresponding to the two methods can be obtained.

[0089] Figure 6 FIG. 4 shows a schematic diagram of a first improved SSD model structure according to a specific embodiment of the present invention. Figure 7 FIG. 4 shows a schematic diagram of a deep feature fusion structure according to a specific embodiment of the present invention. Figure 6 and Figure 7 In the first method, the multi-scale detection network adopts the first improved SSD model. The first improved SSD model retains the structure of conv 1_1 to conv 5_3 in the VGG 16 structure and adds conv7, conv8, conv9 and conv10 convolutional layers. In the first method, the specific feature fusion steps are:

[0090] S141, perform a deconvolution operation on the feature map of the conv 4_3 layer with a size of 38*38 responsible for detecting small targets, obtain an enlarged Deconv 4 image with a size of 300*300, and use feature mapping to add the features of the feature map of the conv 4_3 layer to the Deconv 4 image. Specifically, use formula (3) for feature mapping:

[0091] x i+1 =stridei *x i +((kernel_size-1) / 2-padding))(3)

[0092] Among them, x i and x i+1 They respectively refer to the coordinate points of the input and output of the feature map of the i-th layer, and stridei is the step size of the i-th layer deconvolution.

[0093] S142, performing a pooling operation on the Deconv 4 image after coordinate feature mapping is completed using formula (3) to obtain a Pool4 feature map;

[0094] S143, upsampling the feature map of the conv 11_2 layer with rich semantic information to obtain the conv'11 feature map;

[0095] S144, fuse the features of the conv'11 feature map and the Pool 4 feature map to the improved conv'4 feature map, and finally concatenate and fuse the conv'4 feature map with the feature maps of conv4_3, conv7, conv 8_2, conv9_2, and conv 10_2 extracted from the first improved SSD model.

[0096] like Figure 7 As shown in the figure, at the same size (38*38), the convolutional layers with dimensions of 10 and 20 are concatenated to obtain a convolutional layer with dimension 30 (the size is still 38*38).

[0097] In this embodiment, the fused and improved detection network uses a total of 6 feature maps. Taking the experimental size of 800 as an example, the sizes are (100*100), (50*50), (25*25), (13*13), (7*7), and (100*100). Compared with the original SSD network, the improved method of deep feature fusion adds more semantic information, enhances the image feature information to a certain extent, and can use more complex feature maps to improve the performance of small targets.

[0098] Figure 8 FIG. 4 shows a schematic diagram of a second improved SSD model structure according to a specific embodiment of the present invention. Fig. 9 FIG. 4 shows a schematic diagram of a structure of adjacent layer feature fusion according to a specific embodiment of the present invention. Figure 8 and Fig. 9In the second method, the multi-scale detection network uses the second improved SSD model to fuse adjacent layer features. The second improved SSD model only retains four layers on the basis of the original SSD, removes conv 10 and conv 11 of the original SSD model, retains the shallow feature conv 4_3, and fuses the remaining layers in pairs to form new detection layers newconv 7, newconv7, newconv8 and newconv9.

[0099] In matrix operations, there is a multiplication method called Hadamard product, which means that each element is multiplied separately, while the feature map fusion in this embodiment adopts the addition operation of the elements at the same position in the two feature maps in the same dimension. Fig. 9 As shown in the dotted box in , the two 4*4 feature maps are fused. Since the calculation process is only the addition of corresponding elements, the scale of the final result remains unchanged. The corresponding element coordinates 5 and 8 are added to get 13, and another 4*4 feature map is generated with an element of 13.

[0100] Continue to refer Fig. 9 In this embodiment, the operation of the two layers fc7 and conv 4_3 is used as an example for feature fusion. The fc7 layer is first deconvolved to obtain fc 7_deconv, and then convolved to a feature layer with the same size and dimension as the conv4_3 layer. Finally, the feature fusion of the two layers is achieved by direct corresponding addition, and the result is newconv 4_3. The same is true for the remaining two adjacent layers, and finally 4 feature maps are generated, with sizes of (100x 100), (100x100), (50x 50), and (25x25).

[0101] In short, complex semantic information can be added to the detection image through deep feature fusion and adjacent layer feature fusion. Compared with the FasterR-CNN type method that requires candidate boxes, this method reduces the training process of the region proposal network and is easier from a training perspective.

[0102] S150, using non-maximum suppression (NMS) to remove redundant prediction boxes from the prediction box, and obtaining a unique identification box of the target to be measured;

[0103] Introducing multi-scale detection in YOLO v3 will obtain more detection candidate boxes. The overlapping candidate boxes generated by the same target may belong to the same or different categories at different scales, and each result will correspond to a confidence level. Due to the shared coordinate system, non-maximum suppression (NMS for short) is required for redundancy removal. The IOU threshold is set according to the detection box to filter out the boxes with low confidence to solve the problem and reduce the false alarm rate of the result. The original NMS algorithm will retain the confidence score of the detection during the detection process and filter it based on a certain threshold. However, in the case of densely arranged targets, ordinary non-maximum suppression will have some shortcomings in the detection of dense targets in the dataset of this article, causing adjacent detection boxes to "disappear".

[0104] In this embodiment, the present application uses weight decay to update the confidence detection result by improving the non-maximum suppression NMS. The principle of the present application to improve the non-maximum suppression NMS is that when the detection process is carried out to screen the detection results, the number of detection frames to be screened is large and closely arranged, and their confidence scores are sometimes suppressed. In order to avoid this problem, the NMS algorithm will be improved by using weight decay. Inspired by Soft-NMS, the present application proposes an improved non-maximum suppression algorithm that mainly focuses on the confidence of the detection results. When the detection IOU is greater than the threshold, a function is set to uniformly reduce the confidence to a certain extent through a formula. Since the larger the IOU, the greater the probability that the detection result is a false positive, setting a larger function to reduce the confidence of this detection result can improve the overall detection accuracy. In addition, it is worth noting that if a false detection occurs under this method, the confidence of the detection frame in the corresponding area will also decrease, which can improve the detection performance to a certain extent.

[0105] The original NMS algorithm uses formula (3) on the threshold processing to show that when the iou is greater than the threshold, the confidence interval becomes 0, and some detection results will be directly ignored.

[0106]

[0107] This application uses non-maximum suppression NMS to remove redundant prediction boxes from the prediction box. Specifically, the confidence is updated by weight decay. The formula (4) used is:

[0108]

[0109] Among them, ci represents the confidence, iou(I,b k ) represents the intersection and union ratio.

[0110] Through formula (4), the confidence is processed accordingly according to the different iou, so that the detection results with lower iou themselves remain unchanged, while the confidence of the detection results that may be false positives will be reduced.

[0111] S160, repeating steps S130-S150 to iteratively train the entire model including the modified feature extraction network, the multi-scale detection network and the non-maximum suppression NMS, and finally outputting the entire model.

[0112] This application uses the improved YOLOv3 network to detect and analyze sea surface targets. The specific experimental data are as follows:

[0113] The training phase was started with an initial learning rate of 0.001. Compared with YOLO v3 and its improved structure, the attenuation coefficient was set to 0.005. When the number of training iterations reached 16,000 and 18,000, the learning rate was reduced to 0.0001 and 0.00001 to further converge the loss function. The convergence curve and Avg IOU curve drawn by the loss function are shown in Figure 2. Fig.10a , Fig.10b As shown in the figure, after reaching the maximum training value of 2000 times, the final result is that Loss and Avg IOU remain at around 0.25 and 0.857 respectively, and the parameter training process slowly stabilizes.

[0114] Fig.11a The PR curve diagram of the original YOLO v3 network is shown. Fig.11b A schematic diagram of the PR curve of the improved YOLO v3 network according to a specific embodiment of the present invention is shown; through the convergence of parameters, it can be seen that the training result of the improved YOLO v3 network has good stability and the convergence trend changes little.

[0115] 1) Quantitative comparison

[0116] The following table compares the target detection results using different algorithms (%). It can be seen from the table that the accuracy of small target detection of the improved YOLO v3 network in this application is 16.4% higher than that of the original network, and the recall rate is increased by 6.2%. The network performance is also 4.5% higher than the detection accuracy of Faster R-CNN. If the improved NMS is used directly on the original YOLO v3 network for experiments, the results show that the accuracy can also be improved by 1%.

[0117]

[0118] 2) Qualitative comparison

[0119] Fig.12a , Figure 12b and Fig.12cThis is a schematic diagram of the effect of using the original NMS algorithm, Soft-NMS algorithm and improved NMS algorithm for detection. Fig.12a , Figure 12b and Fig.12c It can be seen that for the fourth ship from the top to the bottom on the right side of the three figures, the detection results of the original NMS algorithm have missed detections because the detection boxes are very close and are suppressed, while the detection results of the Soft-NMS algorithm not only missed the fourth ship, but also had multiple detection boxes for the detection of the fifth ship, indicating that under this detection condition, the soft-NMS algorithm cannot suppress the detection box. The improved NMS algorithm proposed in the present application updates a lower confidence for the misdetected box through weight attenuation, and realizes the detection of the fourth ship. Compared with the detection results of the Soft-NMS algorithm, the improved NMS algorithm of the present application does not have the phenomenon of multiple detection boxes existing at the same time. The confidence of NMS and soft-NMS on the detection boxes determined by the results is not high, and there is a phenomenon of missed detection. The improved NMS can show the determined high confidence and detect the missed targets suppressed by the original NMS.

[0120] In summary, the improved NMS algorithm provided by this application can maintain a higher confidence level for the originally accurately located targets, while reducing the confidence scores of some targets with missed detection and deviation. This approach of weighted attenuation of confidence rather than directly retaining the highest confidence level can avoid certain missed detections in the detection of compact small targets.

[0121] The present application provides a method for detecting and identifying sea targets based on deep learning. The darknet-53 network is optimized on the basis of the original YOLOv3 model. In the original network, the original convolutional layer is replaced by Inception v4 to make the training network deeper and wider. In the final detection part, the reuse of the ResNet structure enhancement feature is added. At the same time, on the basis of retaining the original multi-target detection, a larger scale is added, so that the model can well preserve the features of large and small targets. The original NMS algorithm is improved by using a decay-based function, so that the average accuracy of the overall detection is significantly improved, and the detection effect on the closely arranged small ship targets is also improved accordingly. The improved YOLO v3 network increases the mAP from 56.1% to 72.5% in ship detection under remote sensing images. In addition to single-category ship detection, the method provided by the present application can also be applied to multi-category small target detection, and can improve the detection performance to a certain extent.

[0122] The two improved SSD feature fusion methods proposed in this application are experimented in the remote sensing image ship dataset. The experiment will use Pytorch, modified in the SSD framework, and the GPU is NVIDIA GeForce GTX TITAN X. In the experiment, we will train the original SSD network and the improved SSD structure based on the fusion of two features respectively. The experimental results will be analyzed from three perspectives: visual feature map, quantitative and qualitative.

[0123] 1) Visualization feature map

[0124] Fig.13 FIG. 4 shows a visualization feature diagram of each stage according to a specific embodiment of the present invention, as shown in FIG. Fig.13 As shown, Fig.13 (a) is the shallow feature of the image, Fig.13 (b) is the location of the data annotation in the original image, Fig.13 (c) shows the detection results. As the number of layers increases, Fig.13 Figures (f), 13(g) and 13(h) show relatively obvious semantic information, while the high-level feature 13(i) has a small size, rough edges and a strong sense of grain. The shallow network extracts texture and detail features, while the deep network extracts contours and the strongest features (the area and shape of the ship). As the number of layers increases, the resolution of the image becomes smaller and smaller. Relatively speaking, as the number of layers increases, the number of features contained in the shallow layer will gradually decrease, and more representative features will be extracted in the deep layer.

[0125] The improved SSD adjacent feature fusion stage superimposes semantic information on the underlying features from top to bottom, which strengthens the location of the target. Feature fusion further narrows the scope of the semantic information of the target location and enhances its edges, making the features of the target location more convergent and condensed, and finally strengthening the feature information of small targets in the output multi-layer feature map.

[0126] 2) Quantitative comparison

[0127] The initial learning rate is set to 0.001 and BatchSize is set to 32. When the number of training iterations increases to 80,000 and 100,000, the learning rate is changed to 0.0001 and 0.00001 respectively to further converge the loss function.

[0128] The detection results of different methods on the remote sensing ship dataset are shown in the following table. When training the network, the FasterR-CNN used ResNet50 as the basic network, and used the more accurate Efficient Net network and the faster Mobile Net as comparative experiments. The above algorithms were trained for 120,000 iterations, and the initial learning rate was set to 0.001. The prediction time comparison results of a single image are shown in the last column of the following table.

[0129]

[0130] As can be seen from the table, the detection accuracy of the first-stage detection algorithm SSD is significantly lower than that of the second-stage RCNN algorithm. From the perspective of the accuracy index AP-0.5, the overall map of the two improved FFSSD networks is improved compared to the original network. The deep feature fusion method improves the network accuracy by 6.8%, and the network based on the fusion of adjacent features has a larger improvement in accuracy, reaching 10.9%. In particular, in the detection of small targets with a size of S, there is a corresponding improvement compared to the original network. Compared with the FasterR-CNN network, the improved FFSSD1 (the first improved SSD model) and FFSSD2 (the second improved SSD model) also have an improvement of 4.9% and 8.6%.

[0131] In addition, it can be seen from the table that in terms of computational overhead, the improved SSD algorithm model parameter volume only increased by 3.19MB and 4.27MB compared to SSD+VGG16, which is significantly lower than the 19.35MB of the Faster R-CNN method. In terms of time cost, since the method based on adjacent feature fusion FFSSD2 has much fewer network layers than FFSSD1, the detection speed is faster, and it only increases by 58ms compared to SSD+VGG16, which is also faster than the Faster R-CNN method.

[0132] Both SSD improvement methods can improve the phenomenon of missed detection. Overall, they are faster and less expensive than the Faster R-CNN algorithm, which can solve the problem of poor adaptability to small target data sets to a certain extent. The feature fusion method has certain effectiveness and better semantic and location information. In terms of target detection time overhead, FFSSD is faster than Faster R-CNN.

[0133] Further references Fig.14 As an implementation of the above method, the present application provides an embodiment of a sea surface target detection and recognition system based on deep learning. Figure 2 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices. The system 200 includes:

[0134] An acquisition module 210 is used to acquire multiple images as original image data, and to mark the borders and categories of the detection targets in the original image data;

[0135] A preprocessing module 220 is used to perform data enhancement operations on the original image data, and use the newly generated pictures together with the pictures collected in the acquisition module as training pictures;

[0136] A feature extraction module 230, used to modify some convolution processing layers in the feature extraction network Darknet53 into Inception modules and ResNet modules, and use the modified feature extraction network to extract feature maps of the training images;

[0137] A detection module 240 is used to send the feature map extracted by the feature extraction module to a multi-scale detection network, and generate prediction boxes of different scales and different aspect ratios on the feature map to fit the target to be detected in the picture, wherein the multi-scale detection network includes at least four detection scales;

[0138] A redundancy removal module 250 is used to remove redundant prediction frames from the prediction frames by using non-maximum suppression (NMS) to obtain a unique identification frame of the target to be measured; and

[0139] The output module 260 is used to run the feature extraction module, the detection module and the redundancy removal module, iteratively train the entire model composed of the modified feature extraction network, the multi-scale detection network and the non-maximum suppression NMS, and finally output the entire model.

[0140] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above methods is implemented.

[0141] like Fig.15 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage part 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the system 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0142] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0143] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above functions defined in the method of the present application are executed. It should be noted that the computer-readable medium described in the present application can be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus or a device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus or a device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0144] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments described in the present application may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an acquisition module, an analysis module, and an output module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0147] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for sea surface target detection and recognition based on deep learning, characterized in that: The following steps are involved: S110, obtaining multiple images as original image data, and marking the borders and categories of the detection targets in the original image data; S120, performing a data enhancement operation on the original image data, and using the newly generated image together with the image collected in step S110 as a training image; S130, modifying some convolution processing layers in the feature extraction network Darknet53 into Inception modules and ResNet modules, and using the modified feature extraction network to extract feature maps of the training images; S140, sending the feature map extracted in step S130 to a multi-scale detection network, generating prediction boxes of different scales and different aspect ratios on the feature map to fit the target to be detected in the picture, wherein the multi-scale detection network includes at least four detection scales; S150, using non-maximum suppression (NMS) to remove redundant prediction frames from the prediction frame to obtain a unique identification frame of the target to be measured; as well as S160, repeating steps S130-S150 to iteratively train the entire model including the modified feature extraction network, the multi-scale detection network and the non-maximum suppression NMS, and finally outputting the entire model.

2. The method for sea surface target detection and recognition based on deep learning according to claim 1, characterized in that: In step S130, the processing process of the Inception module and the ResNet module is: after fusing the features of multiple different convolutional layers, use a 1*1 convolutional layer to unify the values ​​of all output channels, and finally use the ResNet residual structure to obtain the overall module.

3. The method for sea surface target detection and recognition based on deep learning according to claim 2, characterized in that: The ResNet residual structure specifically includes 2 CBR units and 2 ResNet units.

4. The method for sea surface target detection and recognition based on deep learning according to claim 1, characterized in that: In step S140, the feature map extracted in step S130 is sent to a multi-scale detection network, and the multi-scale detection network adopts a first improved SSD model. The first improved SSD model retains the structure of conv 1_1 to conv 5_3 in the VGG 16 structure, and adds conv7, conv8, conv9 and conv10 convolution layers; the target to be measured in the fitting picture is performed by deep feature fusion, which specifically includes the following steps: S141, performing a deconvolution operation on the feature map of the conv 4_3 layer to obtain an enlarged Deconv4 image, and adding the feature correspondence of the feature map of the conv 4_3 layer to the Deconv 4 image by using feature mapping; S142, performing a pooling operation on the Deconv 4 image after feature mapping to obtain a Pool4 feature map; S143, upsample the feature map of the conv 11_2 layer to obtain the conv'11 feature map; S144, fuse the features of the conv'11 feature map and the Pool 4 feature map to the improved conv'4 feature map, and finally concatenate and fuse the conv'4 feature map with the feature maps of conv4_3, conv7, conv 8_2, conv9_2 and conv 10_2 extracted from the first improved SSD model.

5. The method for sea surface target detection and recognition based on deep learning according to claim 4, characterized in that: The feature map of the conv 4_3 layer is added to the deconv 4 image using feature mapping. Specifically, the following formula is used for feature mapping: x i+1 =stride i *x i +((kernel_size-1) / 2-padding)) Among them, x i and x i+1 They respectively refer to the coordinate points of the input and output of the feature map of the i-th layer, and stridei is the step size of the i-th layer deconvolution.

6. The method for sea surface target detection and recognition based on deep learning according to claim 1, characterized in that: In step S140, the target to be tested in the fitting picture adopts the second improved SSD model to fuse the adjacent layer features. The second improved SSD model removes the conv 10 and conv 11 convolutional layers of the original SSD model, retains the shallow feature conv 4_3, and fuses the remaining layers in pairs to form new detection layers newconv 7, newconv7, newconv8 and newconv9.

7. The method for sea surface target detection and recognition based on deep learning according to claim 1, characterized in that: In step S150, non-maximum suppression (NMS) is used to remove redundant prediction boxes from the prediction box. Specifically, the confidence is updated by weight attenuation. The formula used is as follows: Among them, ci represents the confidence, iou(I, b k ) represents the intersection and union ratio.

8. The method for sea surface target detection and recognition based on deep learning according to claim 1, characterized in that: In step S140, the multi-scale detection network adds at least one scale for detecting small targets on the basis of extracting three-scale feature information of one upsampling and two downsampling in the original YOLOv3 algorithm; the target object is identified by using the k-means clustering algorithm, wherein the objective function used by the anchor clustering is as follows: Where D refers to the sample data, G and K represent the center coordinates of each cluster and the total number of clusters respectively, and IOU (D, G) is the intersection-union ratio between the center box of the cluster and the cluster box.

9. The method for sea surface target detection and recognition based on deep learning according to claim 1, characterized in that: In step S120, the data enhancement operations include horizontal mirroring, image rotation, random cropping, random brightness and contrast enhancement.

10. A sea surface target detection and recognition system based on deep learning, characterized in that: Includes the following modules: An acquisition module is used to acquire multiple images as raw image data and annotate the borders and categories of the detection targets in the raw image data; as well as A preprocessing module, used to perform data enhancement operation on the original image data, and use the newly generated pictures together with the pictures collected in the acquisition module as training pictures; as well as A feature extraction module, used to modify some convolution processing layers in the feature extraction network Darknet53 into Inception modules and ResNet modules, and use the modified feature extraction network to extract feature maps of the training images; as well as A detection module, configured to send the feature map extracted by the feature extraction module into a multi-scale detection network, and generate prediction boxes of different scales and different aspect ratios on the feature map to fit the target to be detected in the picture, wherein the multi-scale detection network includes at least four detection scales; and A redundancy removal module, used to remove redundant prediction frames from the prediction frame by using non-maximum suppression (NMS) to obtain a unique identification frame of the target to be measured; and The output module is used to run the feature extraction module, the detection module and the redundancy removal module, iteratively train the entire model composed of the modified feature extraction network, the multi-scale detection network and the non-maximum suppression NMS, and finally output the entire model.

11. A computer-readable storage medium, wherein a computer program is stored in the medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images