A ship target detection method based on vision and radar fusion

Abstract

The present invention discloses a ship target detection method based on vision and radar fusion, including: obtaining a visual image to be processed and corresponding radar data, converting the latitude and longitude coordinates of the ship to the pixel coordinates of the image, and using a selective radar mask to generate a radar mask map; constructing a dual-stream feature extraction network to extract three feature maps of different sizes from the visual image, and obtaining three mask maps of corresponding sizes to the feature map from the radar mask map; constructing a dual-modal feature fusion module to fuse feature maps and mask maps of the same size, constructing a neck network to enhance feature information; and constructing a prediction network to realize target detection classification and position regression. The present invention generates different types of radar mask images according to the characteristics of the ship and the size of the ship, which can improve the detection accuracy of large and medium-sized ships, and can effectively use radar information through cross-fusion to significantly improve the detection accuracy.

Description

Technical Field

[0001] The invention belongs to the technical field of ship target detection, and in particular relates to a ship target detection method based on vision and radar fusion. Background Art

[0002] Ship detection has important application value in the fields of ocean detection, port management, navigation safety, etc. In the field of port management, ship detection can supervise and assist in the management of maritime traffic and transportation; in terms of marine monitoring, ship detection has a strong supervisory effect on fishery dumping of pollutants and illegal smuggling; in terms of navigation safety, ship detection can determine whether there are abnormal navigation behaviors such as landing and stagnation, thereby ensuring coastal and maritime safety. Therefore, studying ship detection technology and realizing intelligent ship automatic detection is of great significance to ensure the rapid development of ship transportation.

[0003] According to the different image generation sources, image-based ship target detection methods can be roughly divided into the following four categories: ship detection based on synthetic aperture radar (SAR) images, ship detection based on optical remote sensing images, ship detection based on infrared images, and ship detection based on visual images. The acquisition and preprocessing of SAR images and remote sensing images often take time, and it is impossible to achieve real-time detection of sailing ships. Visual images and infrared images can be acquired in real time based on monitoring video equipment, which can achieve real-time detection of ships. Infrared images have the advantage of not being affected by lighting conditions and weather conditions, but compared with visual images, the resolution is low and they are more suitable for nighttime ship detection. Visual images can not only be extracted from video equipment for real-time detection, but also contain the advantage of rich semantic information, which can more accurately identify the type of ship. However, due to the complex marine environment in which visual images are located and the special lighting conditions at sea. There are many challenges in accurately detecting ship targets from surveillance videos, such as light reflections on the water surface. Strong light reflections on the water surface can lead to high illumination and overexposed images; interference from surrounding scenes. In some cases, buildings and shore vegetation increase the complexity of separating targets from the background; and short detection range. Some ships are small in size. When the target is far away from the camera, the number of pixels occupied by the target in the visual image is greatly reduced.

[0004] Compared with single vision-based target detection, radar is more robust to complex marine conditions and provides the possibility of long-distance observation. However, radar-based target detection also has great limitations. The most important reason is that radar can only provide position information but not semantic information, that is, it cannot detect multiple types of targets. Therefore, there is a bottleneck in the performance of target detection through a single sensor. The feature-level fusion of visual images rich in semantic information and radar containing position information to solve the problems of small target detection and poor robustness can achieve better detection results.

[0005] The fusion strategies of multimodal data include data-level fusion, feature-level fusion and decision-level fusion. Data-level fusion, also known as pixel-level fusion, belongs to the bottom-level fusion. It directly fuses the raw data observed by multiple sensors, and then extracts the feature vector for identification. Although this method will not cause data loss, the amount of fused tensor data is large, and the convolution processing takes a long time, which is difficult to meet the real-time requirements and will reduce the fusion performance. Decision-level fusion belongs to high-level fusion. Two independent convolutional neural networks are used to process visual images and depth images to obtain detection results. The final decision depends on a joint decision result, but it is affected by the possible mutual exclusion of the detection results of the two networks, which can easily lead to poor final detection performance. In contrast, feature-level fusion extracts features from the source data collected by multiple sensors and fuses them into a single feature vector, which is detected and identified through convolution processing. The extracted feature map has a smaller amount of data than the unprocessed original image, so the processing time is short, and it can be complementary fused according to the feature information actually required for extraction, thereby improving the high detection accuracy brought by fusion. Summary of the invention

[0006] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide a ship target detection method based on vision and radar fusion. The feature layer fusion method is adopted, which can effectively reduce the amount of data calculation and improve the real-time performance of the algorithm. At the same time, the fusion strengthens the correlation of multimodal data, improves the richness of the contained information and the accuracy of detection.

[0007] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0008] A ship target detection method based on vision and radar fusion, comprising:

[0009] S1. Acquire a visual image to be processed and corresponding radar data, wherein the radar data includes latitude and longitude coordinates and size information of the ship;

[0010] S2, converting the latitude and longitude coordinates of the ship into the pixel coordinates of the image, and then generating a radar mask map using a selective radar mask according to the size information of the ship;

[0011] S3. Construct a dual-stream feature extraction network including a first branch network and a second branch network. The visual image is extracted into three feature maps of different sizes through the first branch network. The radar mask map is downsampled through the second branch network to obtain three mask maps of sizes corresponding to the feature map.

[0012] S4. Construct a dual-modal feature fusion module to fuse feature maps and mask maps of the same size, and build a neck network to enhance the feature information of the fused data;

[0013] S5. Build a prediction network and input the output data of the neck network to achieve target detection classification and position regression.

[0014] To optimize the above technical solutions, the specific measures taken also include:

[0015] The formula for converting the longitude and latitude coordinates of the ship to the pixel coordinates of the image in the above S2 is:

[0016]

[0017] Where (x0, y0) is the latitude and longitude coordinates of the ship, (x1, y1) is the latitude and longitude coordinates of the target point, (x w ,y w ,z w ) is the three-dimensional coordinate of the target point on the earth, h0 is the camera height, h1 is the height of the target point, and R is the radius of the earth;

[0018]

[0019] Where (X c ,Y c ,Z c ) is the three-dimensional coordinate of the earth with the camera as the origin, T is the translation matrix, R x ,R y and R z Respective three-dimensional rotation matrices:

[0020]

[0021] Where θ, ρ and β are the rotation angles of the camera around the X-axis, Y-axis and Z-axis respectively;

[0022]

[0023] Where (x, y) is the coordinate in the image coordinate system, that is, the pixel coordinate of the image, f is the focal length of the camera, d x ,d y is the pixel size on the camera sensor, and u0 and v0 represent the center pixel coordinates relative to the pixel coordinate origin.

[0024] The process of generating a radar mask map using the selective radar mask in the above S2 is as follows:

[0025] K-means (K=3) is used to cluster the ship frames, and the formula is as follows:

[0026]

[0027] Among them, shipbox represents the ship box synthesized by the ship size, and cluster represents the cluster center box generated in each iteration process;

[0028] After clustering, the ship frames are divided into three types: L, M, and S, and the corresponding mask map is designed for each type of ship frame. The specific process includes:

[0029] First, construct an empty image with background pixels of δ (0<δ<255) and size of W×H;

[0030] Then, the different types of ships are processed in the corresponding ship frame of the empty image. For L-type ships, they are divided into three parts at a ratio of 1:2:1, and the pixel values ​​are α:β:α (0<β<α≤255, α≠δ, β≠δ); for M-type ships, they are divided into three parts at a ratio of 1:1:1, and the pixel values ​​are α:β:α; for S-type ships, they are regarded as a whole, and the pixel values ​​are all α;

[0031] Finally, the classified image is fused with the visual image to obtain the radar mask.

[0032] The specific formula for fusing the classified image with the visual image to obtain the radar mask is:

[0033]

[0034] Where α and β are pixel values ​​between 0 and 255, and α is greater than β;

[0035] Rect is a rectangular function, which is used to draw a mask image for the ship frame part;

[0036] x and y represent the coordinate values ​​of the upper left corner of the ship box, and w and h represent the width and length of the ship box.

[0037] The dual-stream feature extraction network constructed by S3 above changes the single-input feature extraction network into a dual-input network of visual image and radar mask image, including a first branch network and a second branch network;

[0038] The first branch network is a visual image feature extraction network, using CSPDarknet53. The image is downsampled 32 times when passing through the network for feature extraction, and the feature maps of the last three layers are output for subsequent operations;

[0039] The second branch network is the radar mask extraction network, which uses five pooling layers to downsample the mask image so that the output mask image size is consistent with the features output by the visual image branch. Figure 1 To.

[0040] The first branch network above outputs three different sizes of image features when the input image resolution is W×H×3, and the output feature map scale is

[0041] The mask map sizes output by the second branch network are

[0042] In the above S4, three bimodal feature fusion modules are constructed to fuse the visual feature maps and mask maps of the same size in three different sizes obtained in S3 respectively; the fusion process of the bimodal feature fusion module is as follows:

[0043] (1) First, the spatial information extracted from the visual feature map is fused with the radar mask map to enhance the real target data and weaken the clutter data. The fusion operation is as follows:

[0044]

[0045] Among them, the visual image feature map Radar Mask f 7×7 represents a convolution operation with a convolution kernel of 7, σ represents the sigmoid activation function, AvgPool represents average pooling, and MaxPool represents maximum pooling;

[0046] (2) The radar mask image is processed into a spatial attention form as position information to guide the visual image feature map, as shown below:

[0047]

[0048] Among them, f 1×1 represents a convolution operation with a kernel size of 1, and BN stands for batch normalization.

[0049] The neck network constructed by S4 above adopts the joint structure of FPN+PANet. FPN transmits high-level semantic feature information from top to bottom, and PANet transmits low-level positioning feature information from bottom to top.

[0050] The prediction network constructed by the above S5 adopts the head structure of the YOLO series and predicts the category and position of the target to be detected at the same time; the number of output channels is 3 (K+5), 3 means that each predicted position has 3 prediction boxes of different sizes, K is the number of prediction categories, and 5 means 4 coordinates and 1 confidence of the prediction box.

[0051] The present invention has the following beneficial effects:

[0052] The present invention is implemented by convolutional neural network, and on this basis, the selective radar mask generation algorithm and dual-modal feature fusion module proposed by the present invention are added, specifically including: using cameras and radars to collect visual images and radar data of ships, and converting the longitude and latitude coordinates of ships into pixel coordinates through radar data analysis; using selective radar mask generation algorithm to generate radar mask map; constructing a dual-stream feature extraction network to obtain feature maps and mask maps of three different sizes respectively; constructing a dual-modal feature fusion module to fuse feature maps and mask maps of the same size; constructing a neck network to enhance the feature information of fused data; constructing a prediction network to realize target detection classification and position regression. The visual radar fusion network proposed by the present invention combines the advantages of traditional image target detection and radar information to make the detection results more accurate. The present invention generates different types of radar mask images according to the characteristics and size of the ship, which can improve the detection accuracy of large and medium-sized ships. The present invention studies the fusion method of visual and radar information, and proposes a new visual radar fusion module, which can effectively utilize radar information through cross fusion to significantly improve the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is an overall flow chart of an embodiment of the present invention;

[0054] Figure 2 Radar mask images of three types of ships according to an embodiment of the present invention;

[0055] Figure 3 A dual-modal feature fusion module according to an embodiment of the present invention;

[0056] Figure 4 is the overall network structure of an embodiment of the present invention;

[0057] Figure 5 2 is a diagram showing the detection results of an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0060] Example:

[0061] like Figure 1-5 As shown, a ship target detection method based on vision and radar fusion according to an embodiment of the present invention includes:

[0062] S1. Use cameras and radars to collect the color camera image data and corresponding radar data to be processed, where the radar data mainly includes the latitude and longitude coordinates and length and width information of the ship. The self-made dataset contains 9513 color images, corresponding labels and corresponding radar data, and randomly divides the dataset into training set and test set in a ratio of 8:2.

[0063] S2, converting the latitude and longitude coordinates of the ship into the pixel coordinates of the image, and then generating a radar mask map using a selective radar mask according to the size information of the ship;

[0064] This step converts the latitude and longitude coordinates of the ship into pixel coordinates through radar data analysis, and then generates a radar mask map using a selective radar mask generation algorithm, as follows.

[0065] (1) The longitude and latitude points of the ship are converted to the pixel coordinates of the image. The main conversion process includes: first, according to formula 1, the longitude and latitude coordinates of the ship are converted to the three-dimensional coordinates of the earth with the camera as the coordinate origin; according to formulas 2 and 3, the three-dimensional coordinates of the earth are converted to the three-dimensional coordinates under the current camera posture; according to formula 4, the three-dimensional coordinates of the ship under the current camera posture are converted to the pixel coordinates of the image.

[0066]

[0067] Where (x0, y0) is the latitude and longitude coordinates of the ship, (x1, y1) is the latitude and longitude coordinates of the target point, (x w ,y w ,z w ) is the three-dimensional coordinate of the target point on the earth, h0 is the camera height, h1 is the target point height, and R is the radius of the earth.

[0068]

[0069] Where (X c ,Y c ,Z c ) is the three-dimensional coordinate of the earth with the camera as the origin, T is the translation matrix, R x ,R y and R z The three-dimensional rotation matrices are as follows.

[0070]

[0071] Where θ, ρ and β are the rotation angles of the camera around the X-axis, Y-axis and Z-axis respectively.

[0072]

[0073] Where (x, y) is the coordinate in the image coordinate system, f is the focal length of the camera, and d x ,d y is the pixel size on the camera sensor. u0 and v0 represent the center pixel coordinates relative to the pixel coordinate origin.

[0074] (2) The radar mask image is classified based on the distance and size information of the ship collected by the radar using the selective radar mask generation algorithm proposed in the present invention, and is drawn on the RGB image.

[0075] In this step, the selective radar mask generation algorithm will draw a mask map on the visual image based on the position and size information of the ship collected by the radar. The type of mask map will be designed according to the size of the ship.

[0076] The basis for classifying ship sizes comes from the radar ship length information collected in the training set. K-means clustering is performed on all ship lengths, and the IOU method is used to replace the Euclidean distance, as shown in Formula 5.

[0077]

[0078] Among them, shipbox represents the ship box synthesized by the ship size, and cluster represents the cluster center box generated in each iteration process.

[0079] The specific method for finding the three ship size boundaries is:

[0080] Randomly select 3 ship frames as cluster centers from all the ship frames; calculate the distance between each ship frame and each cluster center; calculate the cluster center closest to each ship frame and assign it to the nearest cluster; recalculate the cluster center based on the ship frames in each cluster; repeat the first two steps until the elements in each cluster no longer change. Based on the self-made ship dataset, this paper divides the ship frames into three main types: L, M, and S.

[0081] Because the clustering results are more sensitive to the length of the ship, the dividing boundaries of the mask map use the lengths in the clustering results, that is, 65 and 223. When the length of the ship box is less than 65, it is recorded as an S-type ship. When the length of the ship box is greater than 65 and less than 223, it is recorded as an M-type ship. When the length of the ship box is greater than 223, it is recorded as an L-type ship.

[0082] After clustering, the ship frames are divided into three types: L, M, and S, and a corresponding mask map is designed for each type of ship frame. The main process includes:

[0083] First, construct an empty image with a background pixel of δ (0<δ<255) and a size of W×H. For example, construct an empty image with a background pixel of 128 and a size of 640x640.

[0084] Then, according to the different types, they are processed in the corresponding ship frame of the empty image. For L-type ships, they are divided into three parts at a ratio of 1:2:1, and the pixel values ​​are α:β:α (0<β<α≤255,α≠δ,β≠δ); for M-type ships, they are divided into three parts at a ratio of 1:1:1, and the pixel values ​​are α:β:α; for S-type ships, they are regarded as a whole, and the pixel values ​​are all α. For example, according to the different types of ships, they are processed in the corresponding target frame area of ​​the empty image. For L-type ships, they are divided into three parts at a ratio of 1:2:1, and the pixel values ​​are 255:192:255; for M-type ships, they are divided into three parts at a ratio of 1:1:1, and the pixel values ​​are 255:192:255; for S-type ships, they are regarded as a whole, and the pixel values ​​are all 255;

[0085] Finally, the classified image is fused with the visual image (multiplication operation) to obtain the radar mask image. The specific fusion formula is:

[0086]

[0087] Where α and β are pixel values ​​between 0 and 255, and α is greater than β;

[0088] Rect is a rectangular function, which is used to draw a mask image for the ship frame part;

[0089] x and y represent the coordinate values ​​of the upper left corner of the ship box, and w and h represent the width and length of the ship box.

[0090] S3. Construct a dual-stream feature extraction network including a first branch network and a second branch network. The visual image is extracted into three feature maps of different sizes through the first branch network. The radar mask map is downsampled through the second branch network to obtain three mask maps of sizes corresponding to the feature map.

[0091] The dual-stream feature extraction network constructed in this step changes the single-input feature extraction network into a dual-input network of visual image and radar mask image, including a first branch network and a second branch network.

[0092] The first branch network is a visual image feature extraction network (the feature extraction network of the RGB image branch), which is consistent with the CSPDarknet53 used in YOLOv5. The image is downsampled by 32 times when it passes through the network for feature extraction, and the final three-layer feature map is output for subsequent operations. When the input image resolution is W×H×3, three different sizes of image features are output, and the output feature map scale is For example, when the input image resolution is 640×640, the output feature map scale is 256×80×80, 512×40×40, and 1024×20×20.

[0093] The second branch network is the radar mask extraction network (radar mask image branch extraction network). The radar mask image does not contain complex semantic information, but only contains the location information of the target. Therefore, the radar mask image branch extraction network mainly uses five pooling layers to downsample the mask image, so that the output mask image size is similar to the feature map output by the RGB image branch. Figure 1 This is convenient for subsequent fusion operations, that is, the output mask map sizes are For example, the scales of the extracted feature maps are 3×80×80, 3×40×40, and 3×20×20.

[0094] Furthermore, the feature extraction network for RGB images includes convolutional layers, residual modules, C3 modules and SPPF modules. The convolutional layer includes convolution operations, batch normalization and SiLU function activation; the residual module reduces the number of channels of the input by half through a 1x1 convolutional layer, and then increases the number of channels to the original number through a 3x3 convolutional layer, and finally adds it to the input; the C3 module divides the input into two branches, the first branch enters N residual modules after reducing the number of channels by half through a 1x1 convolutional layer, and the other branch is concatenated with the result of the first branch after reducing the number of channels by half through a 1x1 convolutional layer, and finally enters the 1x1 convolutional layer; the SPPF module first reduces the number of channels of the input through a 1x1 convolutional layer, and then divides it into two branches, the first branch passes through three cascaded 5x5 maximum pooling, and the second branch reduces the number of channels by half through a 1x1 convolutional layer, and finally concatenates the four outputs and enters the 1x1 convolutional layer. Specific network structure: the first layer is a convolution layer with a kernel size of 6x6, a stride of 2, a padding of 2, a number of channels of 64, and an output size of 320x320x64; the second layer is a convolution layer with a kernel size of 3, a stride of 2, a number of channels of 128, and an output size of 160x160x128; the third layer is a C3 module, which consists of 3 residual modules, and an output size of 160x160x128; the fourth layer is a convolution layer with a kernel size of 3, a stride of 2, a number of channels of 256, and an output size of 80x80x256; the fifth layer is a C3 module, which consists of 6 residual modules. The output size is 80x80x256; the sixth layer is a convolutional layer with a kernel size of 3, a stride of 2, a number of channels of 512, and an output size of 40x40x512; the seventh layer is a C3 module, which consists of 9 residual modules, and the output size is 40x40x512; the eighth layer is a convolutional layer with a kernel size of 3, a stride of 2, a number of channels of 1024, and an output size of 20x20x1024; the ninth layer is a C3 module, which consists of 3 residual modules, and the output size is 20x20x1024; the tenth layer is a SPPF module, and the output size is 20x20x1024.

[0095] There are five maximum pooling layers in the pooling layer network for radar mask images. If the input is a radar mask image of 640x640x3, the output sizes are 320x320x3, 160x160x3, 80x80x3, 40x40x3, and 20x20x3 respectively.

[0096] S4. Construct a dual-modal feature fusion module to fuse feature maps and mask maps of the same size, and use the position information provided by the radar to improve the target detection accuracy; construct a neck network to enhance the feature information of the fused data to improve the small target detection accuracy; the details are as follows:

[0097] (1) Construct three bimodal feature fusion modules to process three RGB feature maps of different sizes: 80x80x256, 40x40x512, and 20x20x1024, and three radar mask images of corresponding sizes: 80x80x3, 40x40x3, and 20x20x3.

[0098] The fusion process of the bimodal feature fusion module is:

[0099] 1) This module needs to first extract spatial information from the visual feature map and fuse it with the radar mask map to enhance the real target data and weaken the clutter data. The fusion operation is as follows:

[0100]

[0101] Among them, the visual image feature map Radar Mask f 7×7 represents a convolution operation with a convolution kernel of 7, σ represents the sigmoid activation function, AvgPool represents average pooling, and MaxPool represents maximum pooling;

[0102] 2) The radar mask image is processed into a form of spatial attention as position information to guide the visual image feature map. This operation can guide the model to focus on the location where the ship may appear. The specific operation is as follows:

[0103]

[0104] Among them, f 1×1 represents a convolution operation with a kernel size of 1, and BN stands for batch normalization.

[0105] The specific fusion process is as follows: the input RGB feature map (size is W1×H1×C1) is subjected to maximum pooling and average pooling respectively to obtain two feature maps of size W1×H1×1; then concatenated, the output is W1×H1×2; then a 7x7 convolution operation is performed, with a step size of 1, a padding of 3, a channel number of 1, and an output size of W1×H1×1; finally, it is activated by the Sigmoid function and multiplied and fused with the input radar mask map. After that, the radar mask image fused with RGB image features is used as input, with a size of W2×H2×3; first, a 1x1 convolution operation is performed, with a step size of 1, no padding, a channel number of 1, and an output size of W2×H2×1; then batch normalization and Sigmoid activation function are performed; the output radar mask map is multiplied with the original input RGB feature map, and the result is once again added to the original RGB feature map, and finally activated by the ReLu function. Through fusion, the output generates three types of fused data: 80x80x256, 40x40x512, and 20x20x1024.

[0106] (2) The neck network used is consistent with that of Yolov5, and both use the joint structure of FPN+PANet. The input source is the data of three different sizes after fusion in the previous step. FPN transmits high-level semantic feature information from top to bottom, and PANet transmits low-level positioning feature information from bottom to top. The network enhances the detection accuracy of small targets by processing the fused data of three different sizes. The structure mainly includes a convolution layer, an upsampling layer (using the nearest neighbor interpolation method), a splicing operation, and a C3 module without shortcut operation. In order to better describe this network structure, this embodiment splits this network into four modules. The first two modules are FPN structures, which are used for upsampling fusion and contain two output feature maps, which are defined as the first output and the second output respectively. The last two modules are PANet structures, which are used for downsampling fusion. The splicing operation requires the two output feature maps of the first two modules. The specific process is as follows: first, the input source 20x20x1024 is input into the network, and the output is 20x20x512 (the first output of the first module) through the first convolution layer of the first module (channel number 512, kernel size 1, step size 1), and then the output is 40x40x512 after the upsampling layer, and then it is spliced ​​with the input source 40x40x512 to output 40x40x1024, and finally it is output through the C3 module containing 3 residual modules to output 40x40x512 (the second output of the first module); the next second module takes the second output of the previous module as input, first passes through the convolution layer (channel number 256, kernel size 1, step size 1) to output 40x40x256 (the first output of the second module), and then passes through the upsampling layer to output 80x80x256, and then it is spliced ​​with the input source 80x80x256 to output 80x80x512, and finally Then it passes through the C3 module containing 3 residual modules to output 80x80x256 (the second output of the second module); the next third module takes the second output of the previous module as input, first passes through the convolution layer (channel number 256, kernel size 3, step size 2) to output 40x40x256, then concatenates with the first output 40x40x256 of the second module to output 40x40x512, and finally passes through the C3 module containing 3 residual modules to output 40x40x512; the next fourth module takes the output of the previous module as input, first passes through the convolution layer (channel number 512, kernel size 3, step size 2) to output 20x20x512, then concatenates with the first output 20x20x512 of the first module to output 20x20x1024, and finally passes through the C3 module containing 3 residual modules to output 20x20x1024.

[0107] S5. Build a prediction network and input the output data of the neck network to achieve target detection classification and position regression.

[0108] The prediction network uses the head structure of Yolov5 and predicts the category and position of the target to be detected. The input is the second output 80x80x256 of the second module in the previous step, the output 40x40x512 of the third module, and the output 20x20x1024 of the fourth module. The number of output channels is 3 (K+5), 3 means that each predicted position has 3 prediction boxes of different sizes, K is the number of prediction categories, and 5 means 4 coordinates and 1 confidence of the prediction box. The RGB images and corresponding depth images in the training set are input into the constructed network model for training and verification. After the model converges, it is tested with the test set and compared with other algorithms. The comparison algorithms include:

[0109] Faster-RCNN, please refer to Ren S, He K, Girshick R, et al. Faster r-cnn: Towards real-time object detection with region proposal networks [J]. Advances inneural information processing systems, 2015, 28.

[0110] Yolov3, see Redmon J, Farhadi A. Yolov3: An incremental improvement [J]. arXiv preprint arXiv:1804.02767, 2018.

[0111] Yolov4, please refer to Bochkovskiy A, Wang CY, Liao HY M.Yolov4: Optimal speed and accuracy of object detection[J].arXiv preprint arXiv:2004.10934,2020.

[0112] Yolov5, please refer to Glenn Jocher, Alex Stoken, Jirka Borovec, NanoCode012, AyushChaurasia, TaoXie, Liu Changyu, Abhiram V, Laughing, tkianai, yxNONG, Adam Hogan, lorenzomammana, AlexWang1900, Jan Hajek, Laurentiu Diaconu, Marc, Yonghye Kwon,oleg,wanghaoyang0106,Yann Defretin,Aditya Lohia,ml5ah,Ben Milanko,BenjaminFineran,Daniel Khromov,Ding Yiwei,Doug,Durgesh,andFranciscoIngham.ultralytics / yolov5:v5.0-YOLOv5-P6 1280models,AWS,Supervise.lyand YouTube integrations,Apr.2021.

[0113] The final test results are shown in Table 1:

[0114] Table 1 Detection results of the present invention and other target detection networks on the self-made ship data set

[0115]

[0116] Table 1 shows the detection results of the present invention and other target detection networks on a self-made ship data set. It can be seen that the present invention has the highest detection accuracy in AP for each type of ship, and the overall indicator mAP reaches 83.31%, which is better than other target detection algorithms. Moreover, the improvement for small ships that are difficult to detect is very high, with an improvement of more than 10%. The improvement for large dredging ships is also close to 10%, and the detection accuracy for other types of ships has also been improved, which is enough to prove the superiority of the present invention.

[0117] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0118] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A ship target detection method based on vision and radar fusion, characterized in that: include: S1. Acquire a visual image to be processed and corresponding radar data, wherein the radar data includes latitude and longitude coordinates and size information of the ship; S2, converting the latitude and longitude coordinates of the ship into the pixel coordinates of the image, and then generating a radar mask map using a selective radar mask according to the size information of the ship; S3. Construct a dual-stream feature extraction network including a first branch network and a second branch network. The visual image is extracted into three feature maps of different sizes through the first branch network. The radar mask map is downsampled through the second branch network to obtain three mask maps of sizes corresponding to the feature map. S4. Construct a dual-modal feature fusion module to fuse feature maps and mask maps of the same size, and build a neck network to enhance the feature information of the fused data; S5. Build a prediction network and input the output data of the neck network to achieve target detection classification and position regression; The process of S2 generating a radar mask map using a selective radar mask is as follows: K-means is used to cluster ship frames, and the formula is as follows: Among them, shipbox represents the ship box synthesized by the ship size, and cluster represents the cluster center box generated in each iteration process; After clustering, the ship frames are divided into three types: L, M, and S, and the corresponding mask map is designed for each type of ship frame. The specific process includes: First, construct an empty image with background pixels δ, 0<δ<255 and size W×H; Then, the different types of ships are processed in the corresponding ship frame of the empty image. For L-type ships, they are divided into three parts at a ratio of 1:2:1, with pixel values ​​of α:β:α, 0<β<α≤255, α≠δ, β≠δ; for M-type ships, they are divided into three parts at a ratio of 1:1:1, with pixel values ​​of α:β:α; for S-type ships, they are regarded as a whole, with all pixel values ​​of α; Finally, the classified image is fused with the visual image to obtain the radar mask image; In S4, three bimodal feature fusion modules are constructed to fuse the visual feature maps and mask maps of the same size in three different sizes obtained in S3 respectively; the fusion process of the bimodal feature fusion module is as follows: (1) First, the spatial information extracted from the visual feature map is fused with the radar mask map to enhance the real target data and weaken the clutter data. The fusion operation is as follows: Among them, the visual image feature map Radar Mask f 7×7 represents a convolution operation with a convolution kernel of 7, σ represents the sigmoid activation function, AvgPool represents average pooling, and MaxPool represents maximum pooling; (2) The radar mask image is processed into a spatial attention form as position information to guide the visual image feature map, as shown below: Among them, f 1×1 represents a convolution operation with a kernel size of 1, and BN stands for batch normalization.

2. According to claim 1, a ship target detection method based on vision and radar fusion is characterized in that: The formula for converting the longitude and latitude coordinates of the ship to the pixel coordinates of the image in S2 is: Where (x0, y0) is the latitude and longitude coordinates of the ship, (x1, y1) is the latitude and longitude coordinates of the target point, (x w ,y w ,z w ) is the three-dimensional coordinate of the target point on the earth, h0 is the camera height, h1 is the height of the target point, and R is the radius of the earth; Where (X c ,Y c ,Z c ) is the three-dimensional coordinate of the earth with the camera as the origin, T is the translation matrix, R x ,R y and R z Respective three-dimensional rotation matrices: Where θ, ρ and β are the rotation angles of the camera around the X-axis, Y-axis and Z-axis respectively; Where (x, y) is the coordinate in the image coordinate system, that is, the pixel coordinate of the image, f is the focal length of the camera, d x ,d y is the pixel size on the camera sensor, and u0 and v0 represent the center pixel coordinates relative to the pixel coordinate origin.

3. The ship target detection method based on vision and radar fusion according to claim 1 is characterized in that: The specific formula for fusing the classified image with the visual image to obtain the radar mask is: Where α and β are pixel values ​​between 0 and 255, and α is greater than β; Rect is a rectangular function, which is used to draw a mask image for the ship frame part; x and y represent the coordinate values ​​of the upper left corner of the ship box, and w and h represent the width and length of the ship box.

4. The ship target detection method based on vision and radar fusion according to claim 1 is characterized in that: The dual-stream feature extraction network constructed by S3 changes the single-input feature extraction network into a dual-input network of visual image and radar mask image, including a first branch network and a second branch network; The first branch network is a visual image feature extraction network, using CSPDarknet53. The image is downsampled 32 times when passing through the network for feature extraction, and the feature maps of the last three layers are output for subsequent operations; The second branch network is the radar mask map extraction network, which uses five pooling layers to downsample the mask map so that the output mask map size is consistent with the feature map output by the visual image branch.

5. The ship target detection method based on vision and radar fusion according to claim 4 is characterized in that: When the input image resolution is W×H×3, the first branch network outputs three image features of different sizes, and the output feature map scale is The mask map sizes output by the second branch network are 6. The ship target detection method based on vision and radar fusion according to claim 1 is characterized in that: The neck network constructed by S4 adopts a joint structure of FPN+PANet, where FPN transmits high-level semantic feature information from top to bottom, and PANet transmits low-level positioning feature information from bottom to top.

7. The ship target detection method based on vision and radar fusion according to claim 1 is characterized in that: The prediction network constructed by S5 adopts the head structure of the YOLO series and predicts the category and position of the target to be detected at the same time; the number of output channels is 3×(K+5), 3 means that each predicted position has 3 prediction boxes of different sizes, K is the number of prediction categories, and 5 means 4 coordinates and 1 confidence of the prediction box.

Images