A Multi-scale Ship Target Detection Method Based on Deep Learning

Abstract

The present invention proposes a multi-scale ship target detection method based on deep learning. This method can extract the features of images of two different scales, and the two channels complement each other, improving the fault tolerance of the network; it can quickly detect ship targets and automatically identify ship types. Specifically, images of different sizes are respectively input into two channels for feature extraction, and the features extracted by the model from different datasets are different. The original image contains foreground and ship information. The information extracted by the FasterRCNN channel is numerous and miscellaneous, and there is also a problem of information loss. The graph convolutional neural network only extracts the information containing ship features, the features are accurate, and it can extract features that the FasterRCNN channel fails to extract. Then the two parts of features are fused to complement each other, further improving the accuracy of ship type recognition.

Description

Technical Field

[0001] The present invention relates to the technical fields of image processing and deep learning, and particularly relates to a multi-scale ship target detection method based on deep learning. Background Art

[0002] In modern intelligent maritime systems, high-definition cameras have been applied to various ports and sea areas for viewing and supervising channel information, etc. Due to the non-intuitive use of radar and the limited information in radar images, high-definition cameras are used for the important parts of maritime channels to visually judge the channel conditions. It is necessary to manually check the types of passing ships to identify key objects of concern. With the busy shipping in recent years, manual checking is time-consuming and laborious, and is also prone to errors. By means of technologies such as deep learning, machines can replace humans to perform target detection on ship images, identify and label ships, greatly reducing the manual input and saving costs. Therefore, ship image classification and recognition based on deep learning will become an important part of the water traffic system.

[0003] Traditional ship detection models all use a single channel and can only extract relevant features for images of one scale, resulting in relatively single extracted features and poor fault tolerance. Summary of the Invention

[0004] A multi-scale ship target detection method based on deep learning, based on a dual-channel detection model, can extract features of two different scale images, and the two channels complement each other, improving the fault tolerance of the network, enabling rapid detection of ship targets, and further improving the accuracy of ship type recognition. To achieve the above object, the following technical solutions are adopted:

[0005] A multi-scale ship target detection method based on deep learning includes the following steps:

[0006] S1. Obtain a data set: The data set includes images O2 containing only ship information and original images O1; the original images O1 contain ship information and background information;

[0007] S2. Construct and train a graph convolutional neural network and a Faster R-CNN network;

[0008] The Faster R-CNN network is used to detect the image type and extract features of the image, and includes an RPN layer, a first max pooling layer, and an R-CNN layer connected in sequence;

[0009] The graph convolutional neural network is used to extract features of the image and includes an attention-pooling layer;

[0010] S3. Construct a dual-channel network model:

[0011] The Faster R-CNN network and the graph convolutional neural network share a fusion layer;

[0012] The input end of the fusion layer is connected to the output end of the first max pooling layer in the Faster R-CNN network and the output end of the attention-pooling layer in the graph convolutional neural network;

[0013] The output end of the fusion layer is connected to the R-CNN layer of the Faster R-CNN network;

[0014] S4. Input the original image O1 into the Faster R-CNN network, and obtain the first feature map F1 at the output end of the first max pooling layer of the Faster R-CNN network. In the first feature map F1, the ship information and background information in the original image O1 are classified and extracted respectively;

[0015] Input the image O2 containing only ship information into the graph convolutional neural network, and obtain the second feature map F2 at the output end of the graph convolutional neural network; the second feature map F2 classifies and extracts the ship information in the image O2 containing only ship information;

[0016] S5. Input the first feature map F1 and the second feature map F2 into the fusion layer, and use the fusion layer to generate the target detection map T1 with ship information;

[0017] S6. Input the target detection map T1 through the fusion layer into the R-CNN layer of the Faster R-CNN network, and output the target detection and recognition result map T2.

[0018] Preferably, in step S5, the method for performing the fusion process is specifically as follows:

[0019] Adjust the first feature map F1 and the second feature map F2 to the same size respectively, and then fuse the features in the two feature maps into one pixel layer to obtain the target detection map T1.

[0020] Preferably, the Faster R-CNN network further includes a fourth convolutional layer, a third max pooling layer, a third convolutional layer, a second convolutional layer, a second max pooling layer, and a first convolutional layer connected in sequence; the output end of the first convolutional layer is connected to the input end of the RPN layer.

[0021] Preferably, the graph convolutional neural network further includes three graph convolutional layers connected in sequence; the output end of the last graph convolutional layer is connected to the input end of the attention-pooling layer.

[0022] Preferably, the image O2 containing only ship information and the original image O1 have different scales.

[0023] Preferably, the graph convolutional neural network adopts the third-generation graph convolutional model.

[0024] Compared with the prior art, the advantages of the present invention are as follows:

[0025] (1) The present invention can extract the features of two different scales of images, and the two channels complement each other, improving the fault tolerance of the network; it can quickly detect ship targets and automatically identify ships. Specifically, datasets (images) of different sizes are respectively input into the two channels for feature extraction, and the features extracted by the model from different datasets are different. The original image contains foreground and ship information. The information extracted by the FasterRCNN channel is numerous and miscellaneous, and there is also a problem of information loss. The graph convolutional neural network only extracts the information containing ship features, with accurate features, and can extract features that the FasterRCNN channel fails to extract. Then, the two parts of the features are fused to complement each other, further improving the accuracy of ship type recognition.

[0026] In addition, the two channels complement each other, and one channel only inputs images containing pure ships, making the discrimination ability of the model for ships stronger, endowing the model with good fault tolerance, enabling it to handle problems in complex environmental information and unclear inference structures, and allowing for defects in the original image.

[0027] (2) The two-channel network model inputs small-size images through one of the channels, resulting in a relatively small difference in the training time between the two-channel network model and the single-channel model of the same number of layers, saving training resources, and thus having broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flowchart of a multi-scale ship target detection method based on deep learning according to an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of a two-channel network model according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The present invention will be described in more detail below with reference to the schematic diagrams, which show the preferred embodiments of the present invention. It should be understood that those skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as a broad guidance for those skilled in the art and not as a limitation to the present invention.

[0031] Figures 1-2 As shown, a multi-scale ship target detection method based on deep learning is used to automatically detect and identify ship targets on water, and includes the following steps:

[0032] S1. Obtain the dataset: The dataset includes the image O2 that only contains ship information and the original image O1. The original image O1 contains ship information and background information.

[0033] The images O2 that only contain ship information and the original image O1 have different scales.

[0034] In this embodiment, first, a large number of ship images at positions such as inland river ports and the Yangtze River entrances and exits are collected as the database. The images are manually classified as the dataset for the graph convolutional neural network, that is, the image O2 that only contains ship information; and as the dataset for Faster R-CNN, that is, the original image O1.

[0035] A total of 3,875 waterborne ship pictures are collected. Through processing, they are divided into two-scale ship images, namely the image O2 that only contains ship information and the original image O1. The image O2 that only contains ship information has a size of 64 * 64, and the original image O1 has a size of 128 * 128.

[0036] S2. Construct and train the graph convolutional neural network and the Faster R-CNN network.

[0037] The Faster R-CNN network is used to detect the image type and extract the features of the image, including the RPN layer, the first max pooling layer, and the R-CNN layer connected in sequence.

[0038] The graph convolutional neural network adopts the third-generation graph convolutional model and is used to extract the features of the image, including the attention-pooling layer.

[0039] In this embodiment, the Faster R-CNN network further includes the fourth convolutional layer, the third max pooling layer, the third convolutional layer, the second convolutional layer, the second max pooling layer, and the first convolutional layer connected in sequence; the output end of the first convolutional layer is connected to the input end of the RPN layer.

[0040] The graph convolutional neural network further includes three graph convolutional layers connected in sequence; the output end of the last graph convolutional layer is connected to the input end of the attention-pooling layer.

[0041] S3. Construct a dual-channel network model, which is a dual-input single-output model.

[0042] The Faster R-CNN network and the graph convolutional neural network share a fusion layer.

[0043] The input end of the fusion layer is connected to the output end of the first max pooling layer in the Faster R-CNN network and the output end of the attention-pooling layer in the graph convolutional neural network.

[0044] The output end of the fusion layer is connected to the R-CNN layer of the Faster R-CNN network.

[0045] S4. Input the original image O1 into the Faster R-CNN network, and obtain the first feature map F1 at the output end of the first max-pooling layer of the Faster R-CNN network. In the first feature map F1, the ship information and background information in the original image O1 are classified and extracted respectively.

[0046] The Faster R-CNN network outputs the extraction of ship features and background features from the original image O1, that is, the first feature map F1 is formed. The information extracted by the Faster R-CNN network is numerous and miscellaneous, and there is also a problem of information loss.

[0047] Ship features such as: significant features such as the color, size of the ship, and the attitude of the ship in the water area.

[0048] Background features such as: features such as the color of the water, texture, and the color of the trees in the background.

[0049] Input the image O2 containing only ship information into the graph convolutional neural network, and obtain the second feature map F2 at the output end of the graph convolutional neural network; the second feature map F2 classifies and extracts the ship information in the image O2 containing only ship information.

[0050] The graph convolutional neural network only outputs ship features such as the color, size of the ship, and the attitude of the ship in the water area. Compared with the features extracted by the Faster R-CNN network, the features are purer and only contain ship features.

[0051] The graph convolutional neural network extracts features accurately, and can extract features that the Faster R-CNN channel does not extract. Then, the two parts of features are fused to complement each other, further improving the accuracy of ship type recognition.

[0052] Since Faster R-CNN contains a detection module (RPN module, draw a detection box and discriminate through softmax classification), if only the image dataset containing ship information is input, the ship detection function is lost (that is, there is only a ship in the image and no detection is required), so only the original image dataset can be input; the graph convolutional channel inputs the image dataset containing only ship information to extract ship features for supplementing the features extracted by the Faster R-CNN channel.

[0053] S5. Input the first feature map F1 and the second feature map F2 into the fusion layer, and use the fusion layer to generate the target detection map T1 with ship features.

[0054] The specific method for performing the fusion process is as follows:

[0055] Adjust the first feature map F1 and the second feature map F2 to the same size respectively, and then fuse the features in the two feature maps into one pixel layer to obtain the target detection map T1.

[0056] S6. Input the target detection graph T1 into the RCNN layer of the Faster RCNN network through the fusion layer, and output the detection and recognition result graph T2.

[0057] The above are only the preferred embodiments of the present invention and do not impose any limitation on the present invention. Any person skilled in the art within the technical field, without departing from the technical solution of the present invention, makes any form of equivalent substitution or modification and other changes to the technical solution and technical content disclosed by the present invention, which are all within the content of the technical solution of the present invention and still fall within the protection scope of the present invention.

Claims

1. A multi-scale ship target detection method based on deep learning, characterized in that, It includes the following steps: S1. Obtain a data set: The data set includes an image O2 containing only ship information and an original image O1; the original image O1 contains ship information and background information. S2. Construct and train a graph convolutional neural network and a Faster R-CNN network. The Faster R-CNN network is used to detect the image type and extract the features of the image, including an RPN layer, a first max pooling layer, and an R-CNN layer connected in sequence. The graph convolutional neural network is used to extract the features of the image, including an attention-pooling layer. S3. Construct a dual-channel network model: The Faster R-CNN network and the graph convolutional neural network share a fusion layer. The input end of the fusion layer is connected to the output end of the first max pooling layer in the Faster R-CNN network and the output end of the attention-pooling layer in the graph convolutional neural network. The output end of the fusion layer is connected to the R-CNN layer of the Faster R-CNN network. S4. Input the original image O1 into the Faster R-CNN network, and obtain a first feature map F1 at the output end of the first max pooling layer of the Faster R-CNN network. In the first feature map F1, the ship information and background information in the original image O1 are classified and extracted respectively. Input the image O2 containing only ship information into the graph convolutional neural network, and obtain a second feature map F2 at the output end of the graph convolutional neural network. The second feature map F2 classifies and extracts the ship information in the image O2 containing only ship information. S5. Input the first feature map F1 and the second feature map F2 into the fusion layer, and use the fusion layer to generate a target detection map T1 with ship information. S6. Input the target detection map T1 through the fusion layer into the R-CNN layer of the Faster R-CNN network, and output a target detection and recognition result map T2.

2. The multi-scale ship target detection method based on deep learning according to claim 1, wherein In step S5, the specific method for fusion processing is as follows: Adjust the first feature map F1 and the second feature map F2 to the same size respectively, and then fuse the features in the two feature maps into one pixel layer to obtain the target detection map T1.

3. The multi-scale ship target detection method based on deep learning according to claim 1, wherein, The Faster R-CNN network further includes a fourth convolutional layer, a third max pooling layer, a third convolutional layer, a second convolutional layer, a second max pooling layer, and a first convolutional layer connected in sequence; the output end of the first convolutional layer is connected to the input end of the RPN layer.

4. The multi-scale ship target detection method based on deep learning according to claim 1, characterized in that The graph convolutional neural network further includes three graph convolutional layers connected in sequence; the output end of the last graph convolutional layer is connected to the input end of the attention-pooling layer.

5. The multi-scale ship target detection method based on deep learning according to claim 1, characterized in that The scales of the image O2 containing only ship information and the original image O1 are different.

6. The multi-scale ship target detection method based on deep learning according to claim 1, characterized in that The graph convolutional neural network adopts a third-generation graph convolutional model.

Images