Ship target detection and ship rough intention prediction method

Abstract

The embodiment of the invention provides a ship target detection and ship rough intention prediction method based on a multi-head deep neural network of an SCNN, and the method comprises the steps: obtaining an original image collected by an airplane, and carrying out the noise reduction of the original image; setting software is adopted to carry out ship wake flow image annotation on the original image after noise reduction processing; and carrying out detection and identification through a multi-head deep neural network, and carrying out intention prediction to obtain a prediction result. Wake flow information existing in pictures shot by the unmanned aerial vehicle photoelectric load can be separated from other information such as ocean, reefs and islands in a scene, and foreground information with a small data size is expanded to be more than 10 times of the original data size through a data enhancement method. While original data features are reserved, richer artificial data are added, so that the problem of high data acquisition and labeling cost is solved.

Description

Technical Field

[0001] The present invention relates to the field of target detection methods and intention prediction technology in the field of machine learning, and in particular to a method for ship target detection and ship rough intention prediction based on a multi-head deep neural network (SCNN). Background Art

[0002] When an airplane takes photos from a high altitude, the scale of the ship targets captured is usually very small. When using traditional computer vision algorithms to detect and identify them, it is often difficult to extract subtle features such as the texture and color of the ship, which makes it very difficult to detect and identify the ship targets. The workload brought by manual identification is very heavy, and it is also difficult to complete the identification of ships with high accuracy and efficiency.

[0003] In order to improve the efficiency of ship intentions and ensure the prediction of the ship's real-time speed and heading in the absence of AIS data support, the application of traditional single-task deep neural networks may bring about the problem of relatively high recognition accuracy but long time consumption.

[0004] With the popularity of multi-head deep networks, this paper deals with a multi-head deep neural network ship detection and intention prediction method based on SCNN (Super-Resolution Convolutional Neural Network, referred to as super-resolution network model). Ship target detection is performed on the input 1920*1080 pixel image. In the image where the ship is detected, the ship's heading and speed are predicted to achieve a rough prediction of the ship's intention. Summary of the invention

[0005] In view of the above problems existing in the prior art, an embodiment of the present invention provides a method for ship target detection and rough ship intention prediction, which is aimed at the photoelectric images collected by the drone at high altitude. The method can separate the wake information (called "foreground" or impurities) in the picture taken by the drone photoelectric payload from other information such as the ocean, reefs, and islands in the scene (called "background"), and expand the foreground information with less data volume to more than 10 times the original data volume through the data enhancement method. While retaining the original data features, more abundant artificial data is added to solve the problem of high data collection and annotation costs. At the same time, this method introduces the SCNN network structure with very good lane line detection effect in the field of autonomous driving, and extracts and identifies the features of the ship's wake with geometric features tending to linear structures, solving the problem of difficulty in identifying the ship's wake caused by too large a background. And adopt a multi-detection head method, by reusing the feature extraction network to reduce the amount of calculation, to achieve the rough intention of the ship, including the prediction of the ship's heading and speed, to solve the detection and identification of the target ship and the intention prediction in the absence of AIS data.

[0006] An embodiment of the present invention provides a method for ship target detection and ship rough intention prediction, comprising:

[0007] Obtaining the original image collected by the aircraft, and performing noise reduction processing on the original image;

[0008] Using the setting software, the original image after noise reduction is annotated with the ship wake image;

[0009] Through multi-head deep neural network, detection and recognition are carried out, and intention prediction is performed to obtain prediction results.

[0010] In some embodiments of the present invention, the denoising process of the original image includes:

[0011] The high-frequency noise in the image is eliminated by using the wavelet threshold shrinkage denoising algorithm;

[0012] By selecting the wavelet basis, the number of decomposition layers, and the noise reduction threshold, the parameters of the wavelet threshold shrinkage noise reduction algorithm are initialized;

[0013] Use the wavelet basis to perform wavelet decomposition of the original image with the selected decomposition layer, and obtain several low-frequency signals and a high-frequency signal after wavelet decomposition. Filter the high-frequency signal according to the selected noise reduction threshold, and assign the part less than the threshold to 0 to eliminate the high-frequency noise;

[0014] The several low-frequency signals and one high-frequency signal obtained by decomposition are subjected to inverse wavelet decomposition to obtain a denoised image.

[0015] In some embodiments of the present invention, the step of labeling the ship wake image on the original image after noise reduction processing includes:

[0016] The ship wake is manually marked with polygons. The marking principle is that the polygon covers the entire wake.

[0017] The marked foreground data and background data are stored separately as two Json files;

[0018] The sample is enriched by performing data enhancement on the foreground in the background, including at least flip transformation, rotation transformation, linear brightness shift, nonlinear brightness shift, contrast map adjustment, and image perspective.

[0019] In some embodiments of the present invention, the step of labeling the ship wake image on the original image after noise reduction processing further includes:

[0020] In order to balance the ratio of foreground and background, the foreground data is randomly copied to the background data, and the data volume is expanded to more than 10 times by the aforementioned data enhancement method;

[0021] The new data boundary is softened by saving the original gradient change matrix of the ship wake boundary.

[0022] In some embodiments of the present invention, the detection and identification by a multi-head deep neural network includes:

[0023] The ship wake feature extraction is performed through the SCNN network structure to fully extract the ship wake features and prepare for ship detection.

[0024] In some embodiments of the present invention, the detection and recognition by a multi-head deep neural network and the intention prediction include:

[0025] The SCNN module is used as a feature extraction module for ship detection and ship speed and heading prediction. A classifier for ship detection and two regressors for ship speed prediction and heading prediction are set respectively.

[0026] Compared with the prior art, the ship target detection and rough ship intention prediction method provided by the embodiment of the present invention has the beneficial effects of: it solves the problem of misidentification caused by insufficient feature extraction of traditional algorithms when high-altitude pictures are used for ship identification; at the same time, the wavelet threshold denoising method is used to solve the problem of data quality degradation caused by sensor quality and interference in the transmission process; further, in view of the problem that the amount of ship target data is too small compared with the background (sea, islands, reefs, etc.), the problem of sample imbalance during training is solved by a data enhancement method; in view of the problem that it takes a long time to use multiple feature extraction modules for complex problems, the time is greatly reduced by reusing feature extraction modules and adding classification heads and regression heads, thereby improving the model efficiency; in view of the large length-width ratio of ship wake data, the SCNN module is used to enhance the feature extraction effect; in addition, it also solves the problem of rough prediction of ship target intention when AIS data is missing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A high-altitude ship wake diagram in the ship target detection and ship rough intention prediction method provided by an embodiment of the present invention;

[0028] Figure 2 Another high-altitude ship wake diagram in the ship target detection and ship rough intention prediction method provided by an embodiment of the present invention;

[0029] Figure 3 A diagram of the annotation effect in the method for ship target detection and ship rough intention prediction provided by an embodiment of the present invention;

[0030] Figure 4 A data expansion effect diagram of the ship target detection and ship rough intention prediction method provided by an embodiment of the present invention;

[0031] Figure 5 This is a diagram showing the effect of edge smoothing in the method for ship target detection and ship rough intention prediction provided by an embodiment of the present invention;

[0032] Figure 6 A block diagram of the SCNN module in the ship target detection and ship rough intention prediction method provided by an embodiment of the present invention;

[0033] Figure 7 A schematic diagram of the multi-head model training process in the ship target detection and ship rough intention prediction method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0036] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0037] It should also be understood that, although the present application has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present application that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0038] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0039] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to determine the true intent based on the user's historical operations and to avoid unnecessary or redundant details that make the present application unclear. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but are merely used as the basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0040] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0041] The embodiment of the present invention provides a multi-head deep neural network ship detection and intention prediction method based on SCNN, which can complete the detection of ships and rough intention prediction by quickly and efficiently identifying the wake of ships. Figures 1 to 7 As shown, the process of ship detection and rough intention prediction method is as follows:

[0042] Step 1: Obtain the original image collected by the aircraft and perform noise reduction on the original image;

[0043] Specifically, the visible light images collected by the aircraft are first subjected to denoising operations, and the high-frequency noise in the images is eliminated by using the wavelet threshold shrinkage denoising algorithm;

[0044] By selecting the wavelet basis, the number of decomposition layers, and the noise reduction threshold, the parameters of the wavelet threshold shrinkage noise reduction algorithm are initialized;

[0045] Use the wavelet basis to perform wavelet decomposition of the original image with the selected decomposition layer, and obtain several low-frequency signals and a high-frequency signal after wavelet decomposition. Filter the high-frequency signal according to the selected noise reduction threshold, and assign the part less than the threshold to 0 to eliminate the high-frequency noise;

[0046] Finally, several low-frequency signals and one high-frequency signal obtained by decomposition are subjected to inverse wavelet decomposition to obtain a denoised image.

[0047] Step 2: Use the setting software to annotate the ship wake image on the original image after noise reduction;

[0048] That is, the denoised data image is enhanced. Specifically, the denoised data can be annotated with the ship wake image in the image using the open source Labelme software.

[0049] First, polygons are used to manually mark the wake of the ship. The marking principle is that the polygon wraps the entire wake. The marked foreground and background data are stored separately as two Json files.

[0050] In order to solve the problem of small data volume, samples are enriched through various data enhancement methods such as flipping transformation, rotation transformation, linear brightness offset, nonlinear brightness offset, contrast map adjustment, image perspective, etc.

[0051] Since the ship wake content contained in the data is relatively small compared to the ocean, reefs, etc., in order to balance the ratio of foreground and background, the foreground data is randomly copied to the background data. Combined with the aforementioned data enhancement method, the data volume can be expanded by more than 10 times.

[0052] The original gradient change matrix of the ship wake boundary is saved to soften the boundary of the new data, thereby improving the naturalness of the artificial data.

[0053] Step 3: Use a multi-head deep neural network to detect and identify, and perform intent prediction to obtain prediction results;

[0054] Specifically, the ship wake images taken from high altitudes tend to be linear structures (straight or arc shapes) in terms of geometric characteristics, that is, the aspect ratio of the feature map size is quite different. The application of traditional deep learning frameworks for feature extraction often fails to fully extract its slender features, and the detection and recognition effect of the target is not good. Therefore, this method extracts ship wake features by introducing the SCNN network structure that has performed well in lane line detection tasks in the field of autonomous driving in recent years. Since the ship wake and lane lines are very similar in shape, both are feature maps with relatively large length and width. The application of the SCNN module for feature extraction can fully extract the ship wake features and prepare for the detection of ships.

[0055] Setting a feature extraction module for each task may get good results in terms of accuracy, but there will be certain problems in the efficiency of the model. However, the ship detection and rough intention recognition methods have strong requirements for efficiency. Therefore, for ship detection and rough intention prediction tasks, the reuse of feature extraction modules is adopted to reduce the amount of model calculation and improve the model efficiency. After the feature extraction module, different detection recognition or predictors are linked to complete different tasks. In the model of the method, the SCNN module is used as the feature extraction module for ship detection and ship speed and heading prediction. A classifier (for ship detection) and two regressors (for ship speed and heading prediction) are set for the three tasks respectively.

[0056] In order to facilitate the understanding of the above technical solution, the following is a detailed description with reference to the accompanying drawings and specific examples, as follows:

[0057] Since the actual ship wake images obtained are often affected by sensor performance, transmission interference, sea clutter interference and other issues, they usually contain extensive noise. It is very important to reduce the noise of the original data. Figure 1 and Figure 2 Shows the original data style.

[0058] When performing noise reduction, we first randomly sample the data, select 10% of the images for observation, and perform wavelet noise reduction on the selected 10% of the data. We then select the appropriate wavelet basis, number of decomposition layers, and noise reduction threshold. The noise reduction threshold selection principle should be such that the noise can be eliminated as much as possible while preserving the shape and contour of the original data.

[0059] Afterwards, the selected wavelet denoising parameters are applied to the entire data set to implement the wavelet threshold denoising method.

[0060] Furthermore, Labelme was used to manually annotate the denoised images and the tail flow data was framed using polygons. Figure 3 The Peugeot after-effect is shown.

[0061] Since manual labeling is not pixel-by-pixel, labeling errors may occur at the edge. Therefore, the boundary can be corrected through the expansion and erosion algorithm to improve the label quality. Usually, the IoU value before and after expansion and erosion (calculation formula: the ratio of the intersection of the two labeled areas before and after fine-tuning to the union of the two labeled areas before and after fine-tuning) increases by about 5%.

[0062] Afterwards, the data is enhanced to expand the dataset so that subsequent model training is more sufficient. The purpose of expanding the dataset can be achieved by randomly copying the previously annotated wake data multiple times or once in multiple or single dataset images, and by cropping, rotating, and resizing the wake images. Figure 4 Shows the Figure 1 The tail flow data is copied once on the original image, and then mirrored and resized. However, since the new data has obvious mutations at the boundary, the boundary of the new data is softened by saving the original gradient change matrix of the boundary, and the data is processed twice to improve the naturalness of the artificial data. The effect is shown in the figure below. Figure 5 shown.

[0063] In order to extract the features of wake data more efficiently during modeling, the SCNN module is used to build a common feature extraction module. When in use, the image is input into the model, the SCNN module is used to extract features, and the classifier is used to make a judgment. Finally, the output is whether wake data is detected in the image. The SCNN module extracts the "slender" features of wake data by performing slice convolution and sum operations on the feature map in accordance with the length and width dimensions. The structure diagram of the SCNN module is shown in the figure below. Figure 6 shown.

[0064] If it is detected as streaming data, the feature extraction results of the image are saved and used to train the ship speed and heading predictor. When training the ship speed and heading predictor, the highly abstract features extracted by the common feature extraction module are first used as input, and the actual annotated ship speed and heading are used as labels to train the predictor. Figure 7 The full-process training method of this method is demonstrated, that is, first training the feature extraction module and then training the prediction head.

[0065] Finally, it was packaged into interactive software to form a platform for ship target detection and rough intention prediction. By inputting images taken at high altitudes, it can automatically detect and identify ship targets in the image. If the target is identified, the speed and course of the target ship can be automatically predicted.

[0066] The algorithm platform requirements are as follows:

[0067] The programming software used is PyCharm 2021.3.1 (Community Edition), Anaconda 4.8.3 and Python3.9. The selected cuda acceleration version is 11.2. The Python libraries used include: numpy 1.19.4, Pillow8.0.0, matplotlib3.3.2, scikit-learn 0.23.2, Pytorch 1.12, etc.

[0068] The hardware platform requirements are as follows:

[0069] Processor computing power cannot be lower than AMD Ryzen 9 5900HS with Radeon Graphics 3.30GHz

[0070] Graphics card should be no less than NVIDIA GeForce RTX 3060

[0071] Memory should be no less than 16GB

[0072] Solid-state hard disk should be no less than 1TB

[0073] The operating system is Windows 10 with NVIDIA CUDA Drivers and Toolkit

[0074] It can be seen from the above technical scheme that the beneficial effects of the ship target detection and rough ship intention prediction method provided by the above embodiment of the present invention are: it solves the problem of misidentification caused by insufficient feature extraction of traditional algorithms when high-altitude pictures are used for ship identification; at the same time, the wavelet threshold denoising method is used to solve the problem of data quality degradation caused by sensor quality and interference during transmission; further, in view of the problem that the amount of ship target data is too small compared to the background (sea, islands, reefs, etc.), the data enhancement method is used to solve the problem of sample imbalance during training; in view of the problem that it takes a long time to use multiple feature extraction modules for complex problems, the time is greatly reduced by reusing feature extraction modules and adding classification heads and regression heads, thereby improving the model efficiency; in view of the large length-width ratio of ship wake data, the SCNN module is used to enhance the feature extraction effect; in addition, it also solves the problem of rough prediction of ship target intention when AIS data is missing.

[0075] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A method for ship target detection and ship rough intention prediction, characterized in that: include: Obtaining the original image collected by the aircraft, and performing noise reduction processing on the original image; Using the setting software, the original image after noise reduction is annotated with the ship wake image; Through multi-head deep neural network, detection and recognition are carried out, and intention prediction is performed to obtain prediction results.

2. The method for ship target detection and ship rough intention prediction according to claim 1, characterized in that: The denoising process of the original image comprises: The high-frequency noise in the image is eliminated by using the wavelet threshold shrinkage denoising algorithm; By selecting the wavelet basis, the number of decomposition layers, and the noise reduction threshold, the parameters of the wavelet threshold shrinkage noise reduction algorithm are initialized; The original image is decomposed by wavelet with a selected number of decomposition layers using a wavelet basis. After wavelet decomposition, several low-frequency signals and a high-frequency signal are obtained. The high-frequency signal is filtered according to the selected noise reduction threshold, and the part less than the threshold is assigned to 0 to eliminate high-frequency noise. The several low-frequency signals and one high-frequency signal obtained by decomposition are subjected to inverse wavelet decomposition to obtain a denoised image.

3. The method for ship target detection and ship rough intention prediction according to claim 2, characterized in that: The process of labeling the ship wake image on the original image after noise reduction processing comprises: The ship wake is manually marked with polygons. The marking principle is that the polygon covers the entire wake. The marked foreground data and background data are stored separately as two Json files; The sample is enriched by performing data enhancement on the foreground in the background, including at least flip transformation, rotation transformation, linear brightness shift, nonlinear brightness shift, contrast map adjustment, and image perspective.

4. The method for ship target detection and ship rough intention prediction according to claim 3, characterized in that: The process of labeling the ship wake image on the original image after the noise reduction process further includes: In order to balance the ratio of foreground and background, the foreground data is randomly copied to the background data, and the data volume is expanded to more than 10 times by the aforementioned data enhancement method; The new data boundary is softened by saving the original gradient change matrix of the ship wake boundary.

5. The method for ship target detection and ship rough intention prediction according to claim 4, characterized in that: The detection and identification by a multi-head deep neural network includes: The ship wake feature extraction is performed through the SCNN network structure to fully extract the ship wake features and prepare for ship detection.

6. The method for ship target detection and ship rough intention prediction according to claim 5, characterized in that: The detection and recognition by a multi-head deep neural network and the intention prediction include: The SCNN module is used as a feature extraction module for ship detection and ship speed and heading prediction. A classifier for ship detection and two regressors for ship speed prediction and heading prediction are set respectively.

Images