Marine target identification method based on multi-scale feature fusion of deep learning

Abstract

The invention discloses a deep learning-based multi-scale feature fusion marine target identification method, which comprises the following steps of: acquiring picture and video data of various marine targets of a marine ship, and acquiring and generating a marine target detection data set, ocean targets are divided into 10 types of yachts, warships, passenger ships, cargo ships, sailboats, tugboats, kayaks, navigation marks, lighthouses, fishing boats and the like, a multi-scale feature fusion module is added in a deep learning network for the problem that the detection precision of a self-built ocean target data set is insufficient, and the detection precision is improved by fusing multi-layer features and injecting global information into a higher layer. An improved deep learning network model used for sea target identification is constructed, the method has the characteristic of having a good detection effect on small sea targets, multiple targets and overlapped targets, the model can be deployed in navigation cameras of ships after iteration training is finished, a monitoring system is established in a matched mode, and the detection accuracy is improved. And various targets on the sea can be monitored in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision multi-target detection, and in particular to a method for marine target recognition based on multi-scale feature fusion based on deep learning. Background Art

[0002] As an important carrier of marine resource development and economic activities, accurate monitoring of marine targets is becoming increasingly important. Carrying out research on automated marine target monitoring is of great significance to strengthening marine management. Whether illegal targets can be discovered and located in a timely manner is the focus of marine target monitoring.

[0003] In the traditional marine target detection algorithm, Fefilatyev et al. used a camera system installed on a buoy to quickly detect ship targets and proposed a new deep-sea visual ocean surveillance algorithm. In view of the problem of ship target detection, a new horizon detection scheme suitable for complex sea areas was proposed. Shi Wei et al. combined morphological filtering with multi-structural elements and improved median filtering to effectively suppress the noise of the background image and detect the ship target. The influence of sea clutter on ship target detection was eliminated by connected domain calculation. Chen Z et al. proposed a ship target detection algorithm suitable for marine video surveillance, aiming to reduce the influence of clutter in the background and improve the accuracy of ship target detection. The main steps of background modeling, model training and updating, and foreground segmentation in the detector are all based on Gaussian mixture model (GMM). The algorithm not only improves the accuracy of the target, but also greatly reduces the probability of false alarms and reduces the impact of dynamic scene changes.

[0004] In recent years, deep learning technology has been increasingly applied to target detection tasks and video surveillance, autonomous driving, face recognition and other fields. The target detection algorithm based on deep learning has obtained better detection results than traditional methods based on big data and has stronger robustness. The target detection algorithm is mainly divided into target detection algorithm based on region proposal, target detection algorithm based on regression and target detection algorithm based on anchor-free. The method based on region proposal is also called two-stage target detection, while the method based on regression is called one-stage target detection. The two-stage detection algorithms mainly include SPP-Net, Fast R-CNN, Faster R-CNN, R-FCN, etc. The one-stage target detection algorithm does not require the step of generating region proposals. It can directly obtain the location and category information of the target, and the detection speed is greatly improved. The one-stage detection algorithm mainly includes SSD, YOLO, etc. The algorithms based on anchor-free target detection are mainly CornerNet and CenterNet, and no prior boxes are used. Many researchers have applied deep learning technology to marine target detection. Gu D proposed a marine ship recognition algorithm based on Faster-R-CNN, which adds a dropout layer after the first fully connected layer of the region generation network to improve the recognition accuracy. QiL et al. proposed an improved target detection algorithm based on Faster R-CNN, which enhances the useful information of ship images by image downsampling and improves the detection speed by using scene reduction technology. Zou Y et al. proposed an improved SSD algorithm based on Mobilenet v2 convolutional neural network for target detection and recognition in ship images. In order to improve the accuracy and robustness of ship target detection, Huang H et al. proposed an improved target detection method based on YOLO. First, the input image is preprocessed by guided filtering and grayscale enhancement, then k-means++ clustering is used to obtain the prior box, and finally the feature redundancy is reduced by reducing some convolution operations and adding skip connection mechanism, which improves the accuracy of ship recognition while ensuring real-time performance. Chen X et al. proposed a framework based on integrated YOLO for detecting ships from ocean images. This method can accurately detect ships and successfully identify the historical behavior of ships. Huang Z et al. proposed an improved YOLO network for intelligent detection and classification of ship images / videos. Although the existing methods have achieved good results, facing the complex and changeable marine environment with more noise interference, the simple YOLO detection algorithm has problems such as low detection accuracy and poor robustness. Therefore, the basic YOLOv8 monitoring method has great limitations in practical applications. Summary of the invention

[0005] 1. Technical issues

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to propose a marine target recognition method based on multi-scale feature fusion based on deep learning, which is used to solve the technical problem that the prior art is difficult to meet the actual application needs due to the complex and changeable marine environment and the large amount of noise interference, resulting in reduced detection accuracy.

[0007] (II) Technical content

[0008] In order to solve the above technical problems, the technical solution of the present invention is: a method for identifying marine targets based on multi-scale feature fusion based on deep learning, comprising the following steps:

[0009] Collect 10 types of surveillance video data, including speedboats, warships, passenger ships, cargo ships, sailboats, tugboats, kayaks, navigation marks, lighthouses, fishing boats, etc., and pre-process these data to build a basic maritime target recognition data set;

[0010] The original YOLOv8 network model is optimized to obtain an improved YOLOv8 network model;

[0011] The improved YOLOv8 network model is trained based on the preprocessed basic data set to obtain a maritime target recognition model;

[0012] The trained YOLOv8 improved algorithm model is deployed in the surveillance cameras of ships at sea to conduct real-time monitoring of various targets at sea.

[0013] Furthermore, all data sets are preprocessed, including rotating and cropping images, adjusting image brightness and contrast, and improving the quality of the initial data set; each image is manually labeled according to the pre-set labeling tools; on this basis, the labeled images are divided into training sets, validation sets, and test sets in proportion, and these sets are used to iteratively optimize the model and evaluate the model.

[0014] Furthermore, the original YOLOv8 network model is optimized. The original YOLOv8 network model includes: five CBS modules in the BackBone network, namely the convolutional modules, four C 2 f module and a SPPF module, namely the pyramid pooling module, Neck, namely the four C in the connection fusion layer part 2 f module, four Concat modules, two CBS modules and two Upsample modules, namely upsampling modules, Head, namely the decoupling head for large targets, the decoupling head for medium targets and the decoupling head for small targets in the detection head layer.

[0015] Furthermore, the optimization of the scale fusion part of the YOLOv8 network model is achieved by introducing the Gather-and-Distribute mechanism. By globally fusing multi-scale features, the information transfer problem in the traditional FPN structure is solved. The Gather-and-Distribute mechanism includes shallow Gather-and-Distribute branches and deep Gather-and-Distribute branches. The shallow Gather-and-Distribute branches and the deep Gather-and-Distribute branches respectively extract and fuse feature information of different scales. At the same time, a lightweight adjacent layer fusion module (FAM) is introduced to enhance the information flow between different levels. In the data preprocessing stage of the YOLO series model, MAE-style pre-training is implemented to improve the convergence speed and accuracy of the model.

[0016] Furthermore, the adjacent layer fusion part of the YOLOv8 network model is optimized by introducing a lightweight adjacent layer fusion module (LAF), which resizes the feature maps of adjacent layers to make them of the same size, and performs a concat operation on the channel dimension on the aligned feature maps, thereby enhancing the information flow between different layers.

[0017] Furthermore, the pre-training of the YOLOv8 network model is optimized by introducing the MAE-style pre-training method. The MAE-style pre-training method is an unsupervised learning method that blocks a part of the input image and allows the model to predict the occluded part, thereby learning the overall structure of the image.

[0018] Furthermore, the training speed of the YOLOv8 network model is optimized by replacing the original CIoU loss function with the WIoU loss function.

[0019] Furthermore, the structure of the LAF module is as follows:

[0020] Input: The input of the LAF module is the feature map from the adjacent layer;

[0021] Operation: Resize the feature maps of adjacent layers to make them of the same size;

[0022] Perform concat operation on the aligned feature maps in the channel dimension;

[0023] Then use 1x1 convolution to adjust the channel dimension of the concat feature map;

[0024] Finally, the adjusted feature map is output.

[0025] Furthermore, the improved YOLOv8 network model is trained based on the preprocessed marine target detection dataset to obtain a preliminary detection model, and the training process is iterated until the training loss function fully converges, the model parameters are saved, and the trained model is saved as the basic detection model for marine ships.

[0026] Furthermore, the method is to configure a trained target recognition algorithm on the surveillance camera of a marine vessel to realize real-time monitoring of various marine targets; deploy a trained target detection algorithm on the surveillance camera to obtain the confidence corresponding to each frame of the video in real time; judge whether the confidence of the current frame of the video is greater than the confidence threshold, if it is greater than the confidence threshold, it is determined that there is a monitoring target in the current monitoring area; upload the current frame image that needs to be monitored, the confidence of the frame image, the time, the location name of the monitoring area and the four corner coordinates of the detection box in the frame image to the server database; the server backend sends the above message to the staff, notifying the staff to take corresponding processing measures in time.

[0027] (III) Technical Effect

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] 1. The present invention constructs a basic data set by collecting 10 types of pictures and video frames in surveillance videos, such as speedboats, warships, passenger ships, cargo ships, sailboats, tugboats, kayaks, navigation marks, lighthouses, and fishing boats, and performs preprocessing such as rotating and cropping images, adjusting image brightness and contrast, etc., to increase the diversity of training data and enhance the generalization ability of the model.

[0030] 2. The present invention designs an improved YOLOv8 feature fusion module, adding a FAM module to collect and align feature maps from different layers for subsequent information fusion. FAM uses an average pooling (AvgPool) operation to downsample the input feature map to a uniform size, and then performs a resize operation to make the size of all feature maps consistent. The purpose of this is to reduce the computational complexity and ensure the efficiency of information fusion.

[0031] 3. The present invention designs a Gather-and-Distribute (GD) mechanism, which introduces new shallow Gather-and-Distribute branches and deep Gather-and-Distribute branches. The low-level GD branch (Low-GD) is mainly used to detect small targets. It includes Low-FAM, Low-IFM and Inject modules, which are responsible for fusing feature maps from level-1, level-2 and level-3. The high-level GD branch (High-GD) is mainly used to detect large targets. It includes High-FAM, High-IFM and Inject modules.

[0032] 4. The present invention designs a FAM module, which is a lightweight adjacent layer fusion module. Through simple concat and 1x1 convolution operations, the fusion of feature maps of adjacent layers is realized, and the information flow between different layers is enhanced, thereby improving the detection accuracy of the model. In addition, the FAM module has a simple structure, small amount of calculation, and will not significantly increase the delay of the model.

[0033] 5. The present invention uses a MAE-style pre-training method in the improved YOLOv8 network model architecture. In the MAE-style pre-training method, the model learns to predict the occluded image part, which can be regarded as a self-supervised learning. This self-supervised learning can enhance the model's perception of images and improve the model's adaptability to complex scenes.

[0034] 6. The present invention uses the WIoU loss function to replace the CIoU loss function of the original YOLOv8 network model, which speeds up the training speed of the model, balances the imbalance problem of positive and negative samples, accelerates model convergence, improves regression accuracy, and improves the overall detection performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of a method for constructing a marine target recognition model according to an embodiment of the present invention;

[0036] Figure 2 is a diagram of the IFM network architecture implemented by the present invention;

[0037] Figure 3 It is a schematic diagram of the FAM structure implemented by the present invention;

[0038] Figure 4 It is a schematic diagram of the traditional Neck information fusion structure;

[0039] Figure 5 It is a schematic diagram of the improved Neck information fusion structure; DETAILED DESCRIPTION

[0040] The embodiment of the present invention provides a method for identifying marine targets based on multi-scale feature fusion based on deep learning. The flowchart of the method for constructing the YOLOv8 marine target recognition model of the embodiment of the present invention is as follows: Figure 1 As shown, the process includes the following steps:

[0041] Step S1, collect pictures and video data of various targets at sea, and pre-process these data to build a basic marine target recognition data set, which will be continuously expanded in subsequent iterations.

[0042] Specifically, the above step S1 includes:

[0043] Step S11, use the surveillance camera of the ship at sea to take pictures or videos of various targets to be identified from the sea, then capture video frames containing multiple targets and small targets from the dynamic video and intercept pictures containing 10 types of targets to be trained, and select a dataset that meets the requirements from the existing open source marine datasets to construct an initial dataset.

[0044] Step S12, preprocessing all images in the data set, including rotating and cropping images, and adjusting image brightness and contrast.

[0045] Step S13, using the LabelImg annotation tool, set to YOLO format, to manually annotate all images.

[0046] Step S14, dividing the labeled 10 types of target images, including speedboats, warships, passenger ships, cargo ships, sailboats, tugboats, kayaks, navigation marks, lighthouses, and fishing boats, into a training set, a validation set, and a test set according to the ratio.

[0047] Exemplarily, the division ratio adopted in this embodiment is 7:2:1, that is, 70% of the data in the data set is divided into a training set for model training, 20% of the data is divided into a validation set for model validation, and 10% of the data is divided into a test set for model testing. The specific division ratio can be adjusted and optimized according to actual training.

[0048] Step S2, performing multi-scale feature fusion processing on the original YOLOv8 network model to obtain an improved YoLOv8 network model.

[0049] Specifically, the above step S2 includes:

[0050] In step S21, the recursive method of the traditional FPN structure is deleted, and a new Gather-and-Distribute (GD) forwarding distribution mechanism is constructed. By using a unified module to collect and fuse information from various levels and then distribute it to different levels, this avoids the information loss inherent in the traditional FPN structure and also enhances the partial information fusion capability of the neck without significantly increasing the delay.

[0051] Specifically, the above step S21 includes:

[0052] Step S211, in the low-stage feature alignment module (Low-FAM), we use the average pooling (AvgPool) operation to downsample the input features and achieve a uniform size. By mapping the features to the minimum feature size of the group. The Low-FAM technique ensures the effective aggregation of information while minimizing the computational complexity of subsequent processing through the Transformer module.

[0053] In step S212, the high-level feature alignment module (High-FAM) consists of AvgPool, which is used to reduce the dimension of the input features to a uniform size, and includes a Transformer module and a split operation. Since the Transformer module extracts high-level information, the pooling operation helps to aggregate the information.

[0054] The neck of the YOLO series usually uses feature pyramid networks (FPN) and their variants to fuse multi-level features. These neck modules basically follow Figure 3 However, current information fusion methods have a significant flaw: when cross-layer information integration is required (for example, level-1 and level-3 are fused), traditional FPN-like structures cannot transmit information losslessly, which prevents YOLO from performing better information fusion.

[0055] It is understandable that the Gather-and-Distribute (GD) mechanism implemented by the present invention achieves efficient exchange of YOLOv8 information by globally fusing multi-level features and injecting global information into higher levels. This significantly enhances the information fusion capability of the neck without significantly increasing latency.

[0056] In step S22, a LAF module is designed. The LAF module is a lightweight adjacent layer fusion module.

[0057] Specifically, the above step S22 includes:

[0058] Step S221, in order to strike a balance between speed and accuracy, two LAF models are added to the model: the LAF low-level model and the LAF high-level model, which are used for low-level injection (merging features of two adjacent layers) and high-level injection (merging features of one adjacent layer), respectively.

[0059] In step S222, to ensure that feature maps from different levels are aligned with the target size, the two LAF models in our implementation only use three operators: bilinear interpolation to upsample features that are too small, average pooling to downsample features that are too large, and 1x1 convolution to adjust features that are different from the target channel.

[0060] In the improved model, the combination of the LAF module and the information injection module effectively balances accuracy and speed. By using simplified operations, we are able to increase the number of information flow paths between different levels, thereby improving performance without significantly increasing latency.

[0061] Step S23, using the MAE-style pre-training method. In the MAE-style pre-training method, the model learns to predict the occluded image part, which can be regarded as a kind of self-supervised learning. This kind of self-supervised learning can enhance the model's perception ability of the image and improve the model's adaptability to complex scenes.

[0062] Specifically, the above step S23 includes:

[0063] Step S231, MIM pre-training following the SparK method, which successfully extends MAE-style pre-training to convolutional networks (convnets).

[0064] In step S232 , the unmasked pixels are regarded as sparse voxels of the 3D point cloud and encoded using sparse convolution.

[0065] In step S233, a hierarchical decoder is further developed to reconstruct the image from the multi-scale encoded features. The framework adopts a UNet-style architecture to decode the multi-scale sparse feature map, where all spatial locations are filled with embedded masks;

[0066] The MAE-style pre-training method allows convolution operations to handle irregular and randomly masked input images, and solves the inconsistency between the single-scale nature of BERT pre-training and the hierarchical structure of the convolutional network. It is helpful to solve problems such as complex background and severe occlusion of small targets, and improve the robustness of the model.

[0067] Step S24, add the FAM module to complete the collection and arrangement of feature maps at each level, providing a basis for further information fusion. FAM uses an average pooling operation to uniformly downsample the input feature map, and then resizes it to make each feature map consistent. Its goal is to reduce the amount of calculation while ensuring efficient information fusion.

[0068] Step S3, training the improved YOLOv8 network model by using the constructed basic marine target recognition data set to obtain a marine target recognition model, and fine-tuning the model according to the effect.

[0069] Specifically, the above step S3 includes:

[0070] Step S31, using the WIoU loss function to replace the original CIoU loss function.

[0071] Step S32, the pre-processed basic dataset of marine target images is input into the improved YOLOv8 network model for iterative optimization. After each training cycle, the model's indicators are calculated on the validation set, and the hyperparameters in the training process are appropriately adjusted.

[0072] Exemplarily, this embodiment evaluates the model performance through three indicators: precision (Precision, P), recall (Recall, R), and mAP.

[0073] Step S33, iterate the training process until the training loss function is fully converged, save the model parameters, and save the trained model.

[0074] The CIoU function introduces multiple additional operation dimensions in training, which increases the complexity of learning. However, when computing resources are limited, the efficiency of the CIoU function is limited. Based on the dynamic non-monotonic mechanism of the WIoU loss function, combined with the gradient gain distribution strategy, the excessive influence of high-quality anchor boxes on model optimization is avoided. At the same time, the algorithm can also effectively reduce the adverse effects of low-quality anchor boxes on model learning. This method can effectively solve the imbalance problem between positive and negative samples, accelerate the convergence speed of the model, and improve the prediction accuracy of the model, thereby improving the overall detection capability of the entire model.

[0075] Step S4, the trained marine target recognition algorithm is deployed in the monitoring camera of the marine ship to monitor the marine targets in real time.

[0076] Specifically, the above step S4 includes:

[0077] Step S41, deploying the trained marine target recognition algorithm in the surveillance camera of the ship.

[0078] Step S42, obtaining the confidence level corresponding to each frame of the video being monitored in real time.

[0079] Step S43, determining whether the confidence of the target to be detected in the current frame of the video is greater than a confidence threshold. If it is greater than the confidence threshold, it is determined that there is a marine target in the current monitoring area.

[0080] Step S44, uploading the current frame image that may contain the target, the confidence of the frame image, the time, the location name of the monitoring area and the coordinate values ​​of the four corners of the detection box in the frame image to the onboard or offshore server database.

[0081] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.

Claims

1. A method for marine target recognition based on multi-scale feature fusion based on deep learning, characterized in that: The following steps are involved: Collect 10 types of surveillance video data, including speedboats, warships, passenger ships, cargo ships, sailboats, tugboats, kayaks, navigation marks, lighthouses, fishing boats, etc., and pre-process these data to build a basic maritime target recognition data set; The original YOLOv8 network model is optimized to obtain an improved YOLOv8 network model; The improved YOLOv8 network model is trained based on the preprocessed basic data set to obtain a marine target recognition model; The trained YOLOv8 improved algorithm model is deployed in the surveillance cameras of ships at sea to conduct real-time monitoring of various targets at sea.

2. The method for marine target recognition based on multi-scale feature fusion based on deep learning according to claim 1, characterized in that: All data sets are preprocessed, including rotating and cropping images, adjusting image brightness and contrast, and improving the quality of the initial data set; each image is manually labeled according to the pre-set labeling tools; on this basis, the labeled images are divided into training sets, validation sets, and test sets in proportion, and these sets are used to iteratively optimize the model and evaluate the model.

3. The method for marine target recognition based on multi-scale feature fusion based on deep learning according to claim 1, characterized in that: The original YOLOv8 network model is optimized. The original YOLOv8 network model includes: five CBS modules, i.e. convolution modules, four C2f modules and one SPPF module, i.e. pyramid pooling module, in the BackBone network part; four C2f modules, four Concat modules, two CBS modules and two Upsample modules, i.e. upsampling modules, in the Neck, i.e. connection fusion layer part; and the Head, i.e. decoupling head for large targets, decoupling head for medium targets and decoupling head for small targets in the detection head layer part.

4. The method for marine target recognition based on multi-scale feature fusion based on deep learning according to claim 1, characterized in that: The optimization of the scale fusion part of the YOLOv8 network model is to solve the information transfer problem in the traditional FPN structure by introducing the Gather-and-Distribute mechanism and globally fusing multi-scale features. The Gather-and-Distribute mechanism includes shallow Gather-and-Distribute branches and deep Gather-and-Distribute branches. The shallow Gather-and-Distribute branches and the deep Gather-and-Distribute branches extract and fuse feature information of different scales respectively. At the same time, a lightweight adjacent layer fusion module (FAM) is introduced to enhance the information flow between different levels. In the data preprocessing stage of the YOLO series model, MAE-style pre-training is implemented to improve the convergence speed and accuracy of the model.

5. The method for marine target recognition based on multi-scale feature fusion based on deep learning according to claim 1, characterized in that: The optimization of the adjacent layer fusion part of the YOLOv8 network model is achieved by introducing a lightweight adjacent layer fusion module (LAF), resizing the feature maps of adjacent layers to make them of the same size, and performing a concat operation on the channel dimension on the aligned feature maps, thereby enhancing the information flow between different layers.

6. The method for marine target recognition based on multi-scale feature fusion based on deep learning according to claim 1, characterized in that: The optimization of YOLOv8 network model pre-training is achieved by introducing the MAE-style pre-training method. The MAE-style pre-training method is an unsupervised learning method that blocks part of the input image and allows the model to predict the blocked part, thereby learning the overall structure of the image.

7. The method for marine target recognition based on multi-scale feature fusion based on deep learning according to claim 1, characterized in that: The training speed of the YOLOv8 network model is optimized by replacing the original CIoU loss function with the WIoU loss function.

8. The method for marine target recognition based on multi-scale feature fusion based on deep learning according to claim 5, characterized in that: The structure of the LAF module is as follows: Input: The input of the LAF module is the feature map from the adjacent layer; Operation: Resize the feature maps of adjacent layers to make them of the same size; Perform concat operation on the aligned feature maps in the channel dimension; Then use 1x1 convolution to adjust the channel dimension of the concat feature map; Finally, the adjusted feature map is output.

9. The method for marine target recognition based on multi-scale feature fusion based on deep learning according to claim 1, characterized in that: The improved YOLOv8 network model is trained based on the preprocessed marine target detection dataset to obtain a preliminary detection model. The training process is iterated until the training loss function is fully converged, the model parameters are saved, and the trained model is saved as the basic detection model for marine ships.

10. The method for marine target recognition based on multi-scale feature fusion based on deep learning according to claim 1, characterized in that: The method is to configure a trained target recognition algorithm on the surveillance camera of a ship at sea to realize real-time monitoring of various targets at sea; deploy a trained target detection algorithm on the surveillance camera to obtain the confidence corresponding to each frame of the video in real time; judge whether the confidence of the current frame of the video is greater than the confidence threshold, if it is greater than the confidence threshold, it is determined that there is a monitoring target in the current monitoring area; upload the current frame image that needs to be monitored, the confidence of the frame image, the time, the location name of the monitoring area and the four corner coordinates of the detection box in the frame image to the server database; The server backend sends the above message to the staff, notifying them to take appropriate measures in a timely manner.

Images