A video-based method for detecting moving targets at sea

Abstract

This invention provides a video-based method for detecting moving targets at sea, specifically in the field of intelligent video surveillance. First, addressing the lack of a published dataset of images of ships at sea, a training set and a test set of ship images are established. The ground truth values ​​of the ship targets in the images are annotated, allowing for direct training and testing of convolutional networks of varying depths. To preserve the network's feature extraction capabilities while improving its speed, the VGGNet16 feature extraction network in Faster R-CNN is improved. Finally, this network is trained to detect and identify moving targets at sea.

Description

Technical Field

[0001] This invention belongs to the field of intelligent video surveillance and uses the Faster Regional Convolutional Neural Network (Faster R-CNN) in deep learning to detect and identify maritime ship targets. This is the first application of an intelligent video surveillance system to detect moving targets at sea. Fixed cameras onshore can be used to automatically monitor the waters near military ports and islands, while ships and unmanned boats equipped with cameras can be deployed to monitor sensitive areas such as offshore military exercise areas and the 12-nautical-mile territorial waters around islands. Background Art

[0002] Solving the problem of moving target detection and recognition in maritime surveillance is particularly challenging. Unmanned aerial vehicles (UAVs), a new type of surface warfare equipment, can quickly, efficiently, and cost-effectively perform dangerous, tedious, and unsuitable maritime missions in swarms. Autonomous collision avoidance by UAVs also requires addressing the challenges of maritime target detection and recognition. Furthermore, the use of cameras to assist ships in navigation and collision avoidance is an effective auxiliary measure, but this also requires breakthroughs in target detection and recognition technology.

[0003] Currently, maritime monitoring primarily relies on infrared imagery, satellite remote sensing imagery, radar imagery, and visible light imagery. Infrared imaging offers excellent concealment and can be used in low-light conditions and at night. However, its drawbacks include limited target information and low resolution. Satellite remote sensing imagery-based detection methods have long detection cycles, lack real-time detection capabilities, and produce low-resolution images. Radar-based detection methods are more common, utilizing microwave imaging. Their advantages include the ability to detect beyond-visual-range targets, strong distance calculation capabilities, and all-weather, real-time imaging. However, their drawbacks include high system cost, and the fact that detected moving targets are typically only a few pixels in size. In most cases, they can only determine the presence of a target, not its type. Detection accuracy for small, close-range targets is low, and scanning imaging has blind spots. Furthermore, they are unsuitable for environments requiring radio silence or strong electromagnetic interference. Visible light imagery-based methods can capture rich, detailed target information. Furthermore, as passive visual sensors, they do not reveal the location of the monitoring device, allowing them to function even in environments requiring radio silence, thus possessing significant military application value.

[0004] However, existing mature intelligent video surveillance systems are mainly used in land scenes. This is because compared with land scenes, the detection of moving targets in complex sea backgrounds has the following challenges:

[0005] (1) A large number of constantly moving waves in the sea background are often misjudged as moving targets.

[0006] (2) Rainy and foggy weather often occurs at sea, which makes the background of moving target detection more complex and changeable. Even in sunny weather, sudden changes in light intensity often occur. Direct sunlight hitting the sea surface will produce strong specular reflections. The background with sudden changes in light intensity and the constantly moving light of fish scales are often misjudged as moving targets.

[0007] (3) Since some sea areas that need to be monitored are far from the shore, the camera needs to be installed on a ship or unmanned boat. This camera movement situation puts higher requirements on the robustness of the detection method. Summary of the Invention

[0008] The purpose of this invention is to propose a video-based method for detecting moving targets at sea. This method can effectively overcome interference from complex backgrounds such as waves and vehicle motion, and can detect and identify the presence of ships in video footage in real time. The method is particularly effective for detecting small ships at sea.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] A video-based method for detecting moving targets at sea was developed. Addressing the lack of a published dataset of images of ships at sea, a training and test set of ship images was constructed. The ground truth of the ship targets in the images was annotated, allowing for direct training and testing of convolutional networks of varying depths. To maintain the network's feature extraction capabilities, improve the accuracy of small ship detection at sea, and increase network speed, the VGGNet16 feature extraction network and Fast R-CNN network in Faster R-CNN were modified. Finally, this network was trained to detect and recognize moving targets at sea.

[0011] The specific steps are as follows:

[0012] A video-based method for detecting moving targets at sea, the steps are as follows:

[0013] Step 101: Create a training set and a test set of marine ship images;

[0014] Step 102: Label the ships in the image: Use rectangular frames to outline the locations of the ships in the image, and name the aircraft carrier "hang boat," other warships "jun boat," and civilian ships "min boat." After labeling, a text file corresponding to each image is generated. The text information includes the image size, target type, and specific location of the target in the image.

[0015] Step 103: Improve VGGNet16 to extract features of maritime ship targets. It is necessary to remove the last pooling layer and the last three fully connected layers in VGGNet16 to use them as the feature extraction network in Faster R-CNN.

[0016] Step 104: Improve Fast R-CNN to achieve feature fusion in the two convolutional layers: convolutional layer 3_3 and convolutional layer 5_3.

[0017] Step 105: Perform general training on the improved VGGNet16 in ImageNet, and use the parameters obtained from the training in the ImageNet database as initial parameters to transfer to the training of marine ship images; then use the marine ship images and annotated text information files in the training set to perform special training on the improved VGGNet16 for ship target detection and recognition.

[0018] Step 106: The parameters of the improved VGGNet16 obtained after training in step 105 are used as initial parameters, and then the improved Fast R-CNN is specially trained using the marine ship images and the annotated text information files in the training set.

[0019] Step 107: Use the parameters obtained after training the improved Fast R-CNN in step 106 as initial parameters, and then use the marine ship images and annotated text information files in the training set to perform secondary dedicated training on the improved VGGNet16 for ship target detection and recognition.

[0020] Step 108: Using the parameters obtained after training the improved VGGNet16 in step 107 as initial parameters, the improved Fast R-CNN is trained twice using the marine ship images and the annotated text information files in the training set.

[0021] Step 109: Detect and identify the ship target in each frame of the video image through the trained network.

[0022] Furthermore, the marine ship images need to take into account the different types, sizes, and angles of the ship targets.

[0023] Beneficial effects of the present invention:

[0024] For the first time, the fast regional convolutional neural network technology was used to solve the practical problem of detecting moving targets at sea. It can effectively overcome the influence of complex sea surface backgrounds such as waves and fish scales. It is of great significance for enriching the maritime defense system and responding quickly in future wars and seizing the initiative in wars. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a labeled schematic diagram.

[0026] Figure 2 It is a flowchart for improving VGGNet16 feature extraction.

[0027] Figure 3 This is a schematic diagram of the ship target detection network structure based on the improved VGGNet16 and the improved Fast R-CNN.

[0028] Figure 4 It is a flow chart of marine moving target detection.

[0029] Figure 5 The following are the results of detecting moving targets of an aircraft carrier at sea in a sunny scene with a stationary camera. (a) Initial frame, (b) 300th frame, (c) 600th frame, (d) 900th frame, (e) 1150th frame, and (f) 1300th frame.

[0030] Figure 6 The following are the results of detecting small moving targets of multiple fast civilian ships at sea in a cloudy scene with the camera stationary. (a) Initial frame, (b) Frame 16, (c) Frame 36, (d) Frame 46, (e) Frame 56, and (f) Frame 75.

[0031] Figure 7 The results of detecting moving civilian ships at sea in a foggy scene with a stationary camera are shown in Figure 1. (a) Initial frame, (b) 300th frame, (c) 600th frame, (d) 900th frame, (e) 1200th frame, and (f) 1500th frame.

[0032] Figure 8 The following are the detection results of moving targets of an aircraft carrier at sea in the evening scene under camera motion conditions. (a) Initial frame, (b) 500th frame, (c) 1000th frame, (d) 1500th frame, (e) 2000th frame, and (f) 2480th frame.

[0033] Figure 9 This is the camera motion detection of other warships at sea in a sunny scene. Among them, (a) the initial frame, (b) the 100th frame, (c) the 200th frame, (d) the 300th frame, (e) the 400th frame, and (f) the 553rd frame.

[0034] Figure 10 This is the camera motion detection of multiple other warships at sea in a sunny scene. Among them, (a) the initial frame, (b) the 40th frame, (c) the 80th frame, (d) the 120th frame, (e) the 160th frame, and (f) the 208th frame. DETAILED DESCRIPTION

[0035] This video-based method for detecting moving targets at sea uses a fast regional convolutional neural network (FRN) from deep learning to detect and identify ships at sea. This method introduces new computer vision techniques to the field of moving target detection at sea, opening up a new avenue for research in this area.

[0036] The present invention will be further described below in conjunction with the accompanying drawings. Figure 4 The following is the working flow chart of the system.

[0037] Step 101: Create a marine ship image database by downloading it through web crawler software and taking it yourself.

[0038] The dataset consists of 5,000 representative images of maritime vessels, selected from the database, taking into account the diversity of vessel types, sizes, and angles. The dataset includes 1,000 images of aircraft carriers, primarily from ten countries; 2,000 images of other warships, primarily including destroyers, frigates, landing ships, missile boats, and supply ships; and 2,000 images of civilian vessels, primarily including passenger ships, dry cargo ships, liquid cargo ships, barges, container ships, roll-on / roll-off ships, engineering vessels, workboats, and fishing vessels.

[0039] Because the training set and test set need to come from the same distribution to better verify the algorithm performance, the present invention finally randomly selects 4000 pictures from 5000 pictures as the training set and the remaining 1000 pictures as the test set.

[0040] Step 102: Use ImagineNet software to mark the ship targets in each picture.

[0041] The locations of the ships were outlined with rectangles, and aircraft carriers were named "hang boats," other warships "jun boats," and civilian vessels "min boats." After annotation, the software automatically generated a corresponding "xml" text file for each image. This file contained information such as the image's dimensions, object type, and specific location within the image. 5,000 text files were generated after manual annotation of 5,000 images in the training and test sets.

[0042] Step 103: Remove the last pooling layer and the last three fully connected layers in VGGNet16 and use them as the region generation network in Faster R-CNN.

[0043] During the research process for this invention, while using VGGNet16 to perform feature extraction experiments on a large number of ship images, we discovered that the large number of parameters generated in the fully connected layers significantly impacted the network's operating speed. While VGGNet16 can detect and recognize 20 target categories in the "PASCAL VOC" image dataset, this invention only detects and recognizes three target categories: aircraft carriers, other warships, and civilian ships. Therefore, improvements were made to preserve the original VGGNet16's feature extraction capabilities while effectively reducing the network size, the number of parameters, and improving its operating speed.

[0044] Step 104: Use the improved VGG16 as the feature extraction network in Faster R-CNN. Experimental results show that although the ship target features extracted by the final layer of Fast R-CNN are the deepest and most representative features, their feature maps are small and contain less information, which can easily lead to missed detections when detecting smaller objects. Because ships at sea are often far from the camera when monitoring them, the targets occupy a small number of pixels in the video image. Therefore, the Fast R-CNN architecture has been improved to better detect small, distant ships.

[0045] In the experiment of feature extraction on a large number of ship images, the present invention found that the first three groups of convolutional pooling layers of VGG16 can already extract the overall outline, superstructure and other features of the ship. Therefore, the present invention first changes the input of the feature map generated in conv5_3 in Fast R-CNN into the RoI pooling layer to the input of the feature maps generated in conv3_3 and conv5_3 into the RoI pooling layer respectively, and then normalizes the two output feature maps, and connects the two normalized feature maps and inputs them into the subsequent Fast R-CNN, realizing the fusion of features in the two convolutional layers conv3_3 and conv5_3. The schematic diagram of the ship target detection network structure based on the improved Fast R-CNN is shown in the figure. Figure 3 shown.

[0046] Step 105: Perform general training on the improved VGGNet16 in ImageNet, and use the obtained weights and other parameters as initial parameters to migrate to the training of marine ship images; then use the marine ship images in the training set and the text information files generated after annotation to perform special training on the improved VGGNet16 for ship target detection and recognition.

[0047] Step 106: Continue to use the parameters obtained after training the improved VGGNet16 in ImageNet as initial parameters, and use the marine ship images and annotated text information files in the training set to perform dedicated training on the improved Fast R-CNN.

[0048] Step 107: Use the parameters obtained after training the improved Fast R-CNN in step 106 as initial parameters, and then use the marine ship images and annotated text information files in the training set to perform special training on the improved VGGNet16 for ship target detection and recognition.

[0049] Step 108: Use the maritime ship images and the annotated text information files in the training set to perform secondary dedicated training on the improved Fast R-CNN.

[0050] At this point, the training of the entire maritime ship target detection network has been completed through the alternating training of the two sub-networks, the improved VGGNet16 and the improved Fast R-CNN.

[0051] Step 109: Install the OpenCV development kit and call the functions therein to read each frame of video. Use the trained network to detect and identify the ship target in each frame of video image.

[0052] like Figure 5 and Figure 10 As shown, the invention was verified by video sequences under conditions of both still and moving cameras, such as sunny days, cloudy days, foggy days, dusk, small targets at long distances, and partially blocked ships, and achieved good detection and recognition effects.

[0053] The advantages of the present invention are: in response to the problem that there is currently no published data set of maritime ship images, a set of diverse maritime ship image training sets and test sets has been established, which can be directly used for training and testing of convolutional networks of different depths. For the first time, the fast regional convolutional neural network (Faster R-CNN) is applied to the detection of moving targets at sea, with strong robustness. For the problem of small target detection, the sub-network Fast R-CNN is improved; for the real-time problem, the sub-network VGGNet16 is improved. It can be applied not only to the situation where the camera is stationary, but also to the situation where the camera carrier such as unmanned boats is moving, which is beneficial for the on-duty personnel and relevant departments in the camp to grasp the situation of military restricted areas and coastal defense fronts in real time and intuitively and to deal with emergencies in a timely manner.

[0054] In summary, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting moving targets at sea based on video, characterized in that: Here are the steps: Step 101: Create a training set and a test set of marine ship images; Step 102: Label the ships in the image: Use rectangular frames to outline the locations of the ships in the image, and name the aircraft carrier "hang boat," other warships "jun boat," and civilian ships "min boat." After labeling, a text file corresponding to each image is generated. The text information includes the image size, target type, and specific location of the target in the image. Step 103: Select VGGNet16 to extract the features of the maritime ship target, but remove the last pooling layer and the last three fully connected layers in VGGNet16 to use as the feature extraction network in Faster R-CNN. Step 104: Perform general training on the improved VGGNet16 in ImageNet, and use the parameters obtained from the training in the ImageNet database as initial parameters to migrate to the training of marine ship images; Then, the improved VGGNet16 is trained for ship target detection and recognition using the sea ship images and annotated text information files in the training set; Step 105: Using the parameters obtained after training the improved VGGNet16 in step 104 as initial parameters, and then using the marine ship images and annotated text information files in the training set to perform dedicated training on the Fast R-CNN in Faster R-CNN; Step 106: Using the parameters obtained after Fast R-CNN training in step 105 as initial parameters, the improved VGGNet16 is then trained for ship target detection and recognition using the maritime ship images and annotated text information files in the training set. Step 107: Using the parameters obtained after the training of the improved VGGNet16 in step 106 as initial parameters, the Fast R-CNN in the Faster R-CNN is trained twice using the marine ship images and the annotated text information files in the training set; Step 108: Detect and identify the ship target in each frame of the video image through the trained network.

2. The method for detecting moving targets at sea based on video according to claim 1, characterized in that: The marine ship images need to take into account the different types, sizes, and angles of the ship targets.

Images