YOLOv8-based marine ship target rapid identification method

Abstract

The invention discloses a rapid marine ship target identification method based on YOLOv8, and is applied to the technical field of target identification. Comprising the following steps of collecting an image sample of a ship navigation water area in a digital vehicle, cleaning and marking the image sample, performing data enhancement on the image sample to obtain a data set which can be used for training and verification, constructing a target identification network model based on a YOLOv8 deep neural network, and obtaining a target identification result by setting hyper-parameters such as a learning rate and a batch processing size. And carrying out transfer learning by using the pre-training model, removing a repeated detection frame by using a non-maximum suppression algorithm and carrying out result filtering by using a confidence coefficient threshold to carry out post-processing on a reasoning result of the model so as to complete prediction of the ship target, and carrying out sparse pruning training on the trained model neural network so as to obtain the ship target. And carrying out calibration and quantification on the pruned model. According to the method, on the premise that the model detection precision is reserved to the maximum, memory consumption during equipment operation is reduced, and meanwhile the model detection speed is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and more specifically to a method for quickly recognizing marine ship targets based on YOLOv8. Background Art

[0002] The ship digital carrier is a comprehensive ship maneuvering control system, of which the three-dimensional visual simulation system and the ship intelligent collision avoidance decision system are its subsystems, providing simulated navigation display and obstacle avoidance path planning respectively. This system aims to use digital technology to optimize the operation efficiency and safety of ships. Based on the growing demand for efficiency, environmental protection and intelligence of modern ships, through real-time monitoring and analysis of the surrounding environment of ships and maritime traffic conditions, using digital twin technology and integrating the Internet of Things, big data analysis and artificial intelligence algorithms, it can realize multiple functions such as real-time monitoring of ship operation status, fault prediction, performance optimization, target identification and route planning, thereby improving the transportation efficiency of ships, reducing operating costs, and enhancing the adaptability and safety of ships in complex sea conditions.

[0003] YOLOv8 is the latest real-time object detector developed by Ultralytics. It is the latest iteration of the YOLO series and has cutting-edge performance in both accuracy and speed. YOLOv8 introduces a number of new features and optimizations, including advanced backbone and neck architecture, anchor-free split head, optimized trade-off between accuracy and speed, and multiple pre-trained models, making it an ideal choice for a wide range of applications. However, the existing YOLOv8 algorithm requires high device performance, otherwise it cannot operate efficiently. Therefore, how to provide a method for rapid identification of marine ship targets based on YOLOv8 is an urgent problem that technicians in this field need to solve. Summary of the invention

[0004] In view of this, the present invention provides a method for rapid identification of marine ship targets based on YOLOv8, which realizes the real-time and efficient operation of the YOLOv8 algorithm under the condition of limited computing equipment performance, quickly and efficiently identifies and monitors target ships, and provides effective reference information for the ship intelligent collision avoidance decision-making system, thereby improving the safety, efficiency and response speed of maritime traffic management.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for rapid identification of marine ship targets based on YOLOv8, comprising the following steps:

[0007] S1, collecting images of various types of ships in the three-dimensional simulation system;

[0008] S2. Cleaning the collected image data and marking the target type of ships therein;

[0009] S3, performing data enhancement processing on the annotated image data;

[0010] S4, dividing the image data after data enhancement processing into a training set and a validation set, and using the YOLOv8 neural network pre-training model to perform transfer learning based on the training set to obtain a preliminary training model;

[0011] S5, fine-tuning the parameters of the preliminary training model to obtain a preliminary detection model;

[0012] S6. Use the deep learning tool TensorRT to calibrate and quantize the preliminary detection model to obtain the final detection model;

[0013] S7. Use the final detection model to identify marine ship targets.

[0014] Optionally, S1 specifically comprises: rotating each type of ship in the three-dimensional simulation system multiple times, collecting images of each rotation angle of all ships at multiple time points in a day, and on this basis collecting images of different rotation angles at different time points in different weather conditions to obtain images of all rotation angles of all types of ships at all weather conditions and at all time points.

[0015] Optionally, S1 divides the collected ship images into two types: military ships and civilian ships according to their uses.

[0016] Optionally, S2 specifically includes: clearing irrelevant images and images that do not meet the requirements during the acquisition process, and using the labelimg annotation tool to perform classification annotation.

[0017] Optionally, S3 specifically comprises: performing data enhancement processing on the annotated image data five times, each data enhancement adopts random numbers to randomly mix several enhancement methods and adjust the target bounding box accordingly, and the data enhancement processing includes randomly adding Gaussian noise, random area masking, random angle rotation of the image, horizontal random translation of the image, vertical random translation of the image, random image cropping except the target, random horizontal flipping, random vertical flipping, random diagonal flipping and random adjustment of image brightness.

[0018] Optionally, in S4, the original samples without data augmentation are divided into the validation set of the data set, and the enhanced samples of the original samples after five data augmentations are divided into the training set of the data set. The ratio of the training set to the validation set is 5:1. On this basis, the background image is added to the training set, and the training set is put into the YOLOv8 pre-trained neural network model for migration training to obtain a preliminary training model.

[0019] Optionally, the sample allocation strategy in the transfer training in S4 uses the Task-AlignedAssigner positive and negative sample allocation strategy, dynamically adjusts the allocation ratio of positive and negative samples during training, assigns a weight to each sample, and selects positive samples in the positive and negative sample allocation strategy based on the classification and regression scores as weighted scores.

[0020] Optionally, S5 performs sparse pruning training on the preliminary training model, specifically:

[0021] S51 constrained training adds L1 constraints to the weights and biases of the batch normalization layer in the preliminary training model, with lambda taking the value of 1e-3:

[0022]

[0023] Where l1_lambda is the coefficient that controls the regularization strength, epoch represents a complete cycle in the model training process, and self.epochs represents the total number of training cycles for the current model instance.

[0024] S52. Use the model obtained in S51 as the pruning object, traverse the batch normalization layer in the model, collect the absolute values ​​of weights and biases, concatenate the absolute values ​​of the weights of the batch normalization layer, sort the concatenated weight absolute value tensors, and calculate a threshold. If the absolute value of the weight is greater than or equal to the threshold, it is retained; if the absolute value of the weight is less than the threshold, it is pruned.

[0025] S53, fine-tune the model. Use the pruned model obtained in S52 to perform 100 epochs of migration training to restore the model accuracy and performance, and obtain the final fine-tuned model.

[0026] Optionally, S6 specifically includes: separating the validation set part in the data set as a model calibration data set, using the entropy calibrator Int8EntropyCalibrator2 of the deep learning tool TensorRT, running the original FP32 model on the calibration data set, collecting the activation value histogram of each layer, and finding the threshold T that minimizes the KL divergence for quantization. Based on the calibration results, the deep learning tool TensorRT will generate a final detection model after quantization, and finally perform a performance test on the final detection model.

[0027] It can be seen from the above technical solution that compared with the prior art, the present invention provides a method for rapid identification of marine ship targets based on YOLOv8, which has the following beneficial effects: the present invention performs sparse pruning training on the trained model neural network, calibrates the pruned model and then quantizes it, which can reduce the loss of model accuracy in the pruning and quantization process, and can provide a fast, reasonable and effective method for marine ship target detection, and also provide effective auxiliary information for subsequent marine collision avoidance decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0029] Figure 1 This is a flow chart of a method for rapid identification of marine ship targets of the present invention;

[0030] Figure 2 It is the YOLOv8 network structure diagram of the present invention;

[0031] Figure 3 This is the convolution structure diagram of the initial training model of the present invention;

[0032] Figure 4 This is the convolution structure diagram of the model after pruning of the present invention;

[0033] Figure 5 It is the PR curve diagram of the model after the initial training of the present invention;

[0034] Figure 6 It is the PR curve diagram of the pruned model of the present invention;

[0035] Figure 7 It is the PR curve diagram of the fine-tuned model of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The embodiment of the present invention discloses a method for rapid identification of marine ship targets based on YOLOv8. Figure 1 As shown, the following steps are included:

[0038] S1, collecting images of various types of ships in the three-dimensional simulation system;

[0039] S2. Cleaning the collected image data and marking the target type of ships therein;

[0040] S3, performing data enhancement processing on the annotated image data;

[0041] S4. The image data after data enhancement is divided into a training set and a validation set. Based on the training set, the YOLOv8 neural network pre-training model is used for transfer learning to obtain a preliminary training model. The YOLOv8 network structure is as follows: Figure 2 As shown;

[0042] S5. Fine-tune the parameters of the preliminary training model to obtain a preliminary detection model. The preliminary training model and the pruned preliminary detection model are respectively as follows: Figure 3 and Figure 4 As shown;

[0043] S6. Use the deep learning tool TensorRT to calibrate and quantize the preliminary detection model to obtain the final detection model;

[0044] S7. Use the final detection model to identify marine ship targets.

[0045] Furthermore, S1 is specifically as follows: each type of ship in the three-dimensional simulation system is rotated multiple times, and images of each rotation angle of all ships are collected at multiple time points in a day, and on this basis, images of different rotation angles at different time points in different weather conditions are collected to obtain images of all rotation angles of all types of ships at all weather conditions and at all time points.

[0046] In the embodiment of the present invention, each ship type is rotated 12 times, each time by 30°, namely 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, and 330°; each angle of each target ship is recorded according to the scene time of 06:00, 12:00, 18:00, and 24:00; all the target ship angles are recorded once at the previous time points according to sunny days, rainy days, foggy days, and snowy days, and one image not containing the target ship at all the above time points is saved as a background image so that the trained detection model can learn the typical environmental features when no target ship appears, which is helpful to exclude false alarms similar to the background during target detection, and obtain material recordings of all target ship angles at various weather and time points.

[0047] Furthermore, S1 divides the collected ship images into two types according to their uses: military ships and civilian ships.

[0048] Furthermore, S2 specifically includes: clearing irrelevant images and images that do not meet the requirements during the acquisition process, and using the labelimg annotation tool to classify and annotate them.

[0049] Furthermore, S3 is specifically as follows: performing data enhancement processing on the annotated image data five times, each data enhancement uses random numbers to randomly mix several enhancement methods and adjust the target bounding box accordingly, and the data enhancement processing includes randomly adding Gaussian noise, random area masking, random angle rotation of the image, horizontal random translation of the image, vertical random translation of the image, random image cropping except the target, random horizontal flipping, random vertical flipping, random diagonal flipping and random adjustment of image brightness processing.

[0050] Furthermore, in S4, the original samples without data augmentation are divided into the validation set of the data set, and the enhanced samples after five data augmentations are divided into the training set of the data set. The ratio of training set to validation set is 5:1. On this basis, the background image is added to the training set, and the training set is put into the YOLOv8 pre-trained neural network model for migration training to obtain a preliminary training model.

[0051] In one embodiment of the present invention, S2 deletes images that are not completely contained, blurred, overexposed, or deformed, and retains approximately 12,000 high-quality images. The image annotation tool Labeling is used to annotate the images to mark the location and category of the ship, where the 36 categories of ships are divided into two categories: military ships and civilian ships. Data enhancement is applied to the annotated images, such as five mixed enhancements of seven enhancement methods such as rotation, scaling, and brightness adjustment. The enhanced samples reach a total of approximately 72,000, of which approximately 12,000 are initial samples. The images after data enhancement are randomly divided into a training set (60,000 images), and the initial samples before enhancement are divided into a validation set (12,000 images). The data set is placed in the pre-trained model of YOLOv8 for 300 epochs of transfer learning.

[0052] Furthermore, the sample allocation strategy in the transfer training in S4 uses the Task-Aligned Assigner positive and negative sample allocation strategy, which dynamically adjusts the allocation ratio of positive and negative samples during training and assigns a weight to each sample. In the positive and negative sample allocation strategy, the classification and regression scores are used as weighted scores to select positive samples.

[0053] In the embodiment of the present invention, the weighted score is:

[0054] t=s α +u β

[0055] In the formula, s represents the score of the predicted box corresponding to the true category, and u represents the intersection over union (IoU) between the predicted box and the true bounding box (Ground Truth, GT Box). Multiplying these two values ​​​​can get an indicator that measures the degree of alignment between the prediction and the true annotation, called task alignment. In this process, α and β are weight hyperparameters used for balance and influence. By adjusting the parameters, the classification score s and IoU value u can be optimized at the same time to achieve task alignment, thereby prompting the network to focus on those high-quality anchors. Specifically, when the category score of the predicted box is high and the score with the true bounding box is high, the value of will be closer to 1, indicating that the prediction result is highly consistent with the true annotation.

[0056] The bounding box regression loss CIOU loss function of Yolov8 is:

[0057]

[0058] In the formula, b and bgt represent the center point coordinates of the predicted rectangular box and the ground truth box, respectively, ρ represents the Euclidean distance between the center points of the two rectangular boxes, c represents the diagonal length of the closed area of ​​the two rectangular boxes, which is used to measure the spatial proximity of the two boxes, and v is a new indicator used to measure the consistency of the size ratio of the two rectangular boxes, that is, their shape similarity. α is a weight coefficient used to adjust the influence of v during the calculation process to balance the contribution of different factors to the final result. Through these metrics, the degree of match between the predicted rectangular box and the ground truth box in position, size and shape can be more comprehensively evaluated. The introduction of α allows the emphasis on size ratio consistency to be adjusted according to different application scenarios during the evaluation process.

[0059] Furthermore, S5 performs sparse pruning training on the preliminary training model. Specifically:

[0060] S51 constrained training adds L1 constraints to the weights and biases of the batch normalization layer in the preliminary training model, with lambda taking the value of 1e-3:

[0061]

[0062] Where l1_lambda is the coefficient that controls the regularization strength, epoch represents a complete cycle in the model training process, that is, the process in which the model performs a forward propagation and a backward propagation on the entire training data set, and self.epochs represents the total number of training cycles for the current model instance, indicating how many epochs the model needs to train to complete the entire training process.

[0063] l1_lambda is a coefficient that decreases as the number of epochs increases. At the beginning of training, epoch = 0, the value is 0.001, that is, 1e-3. As the number of epochs increases, it gradually decreases to 0.0001, that is, 1e-4, when the epoch reaches 90% of self.epochs. If self.epochs is set to 10, the model will perform 10 complete passes on the training data set. After each epoch, the model's weights and biases will be updated according to the training data and loss function.

[0064] The regularization of weights pushes the weights toward zero by adding a term proportional to the sign of the weight to the gradient of the weights, prompting the model to learn a sparser representation. Similar to the weights, the L1 regularization of the bias prevents the bias term from being too large to maintain the generalization ability of the model. As the epoch increases, l1_lambda gradually decreases to 10% of its initial value.

[0065] S52. Use the model obtained in S51 as the pruning object, traverse the batch normalization layer in the model, collect the absolute values ​​of weights and biases, concatenate the absolute values ​​of the weights of the batch normalization layer, sort the concatenated weight absolute value tensors, and calculate a threshold. If the absolute value of the weight is greater than or equal to the threshold, it is retained; if the absolute value of the weight is less than the threshold, it is pruned.

[0066] S53, fine-tune the model. Use the pruned model obtained in S52 to perform 100 epochs of migration training to restore the model accuracy and performance, and obtain the final fine-tuned model.

[0067] For each batch normalization layer (BN layer), extract the absolute values ​​of weights and biases, and print out the maximum and minimum values. Then, concatenate the absolute values ​​of the weights of all BN layers into a large tensor, sort the concatenated absolute value tensors of weights, and obtain a tensor sorted from large to small. According to the given factor (0.8 in this embodiment), the number of weights to be retained is calculated, which is factor times the total number of absolute values ​​of weights of all BN layers. The threshold is the value at that position in the sorted tensor, that is, the threshold is the value ranked in the factor ratio position among the absolute values ​​of weights of all BN layers. This threshold is used for subsequent pruning operations, that is, convolution kernels with absolute values ​​of weights greater than this threshold are retained. In this embodiment, factor is set to 0.8, and the threshold will be the value ranked at 80% of the absolute values ​​of weights of all BN layers. During the pruning process, only convolution kernels with absolute values ​​of weights greater than or equal to this threshold will be retained.

[0068] like Figure 5-7As shown in the figure, the comparison of the PR curves after pruning and fine-tuning of the initial model shows that the accuracy of the model obtained after fine-tuning has improved and performs well on the validation set.

[0069] Furthermore, S6 is specifically as follows: separating the validation set part in the data set as a model calibration data set, using the entropy calibrator Int8EntropyCalibrator2 of the deep learning tool TensorRT, running the original FP32 model on the calibration data set, collecting the activation value histogram of each layer, and finding the threshold T that minimizes the KL divergence for quantization. Based on the calibration results, the deep learning tool TensorRT will generate a quantized final detection model, and finally perform a performance test on the final detection model.

[0070] In an embodiment of the present invention, the final detection model is calculated using INT8, thereby reducing memory usage and improving computing performance.

[0071] Model quantization is one of the important optimization technologies for model optimization in the field of deep learning. It compresses the FP32 floating point numbers in the model parameters into smaller INT8, with the aim of reducing the model storage requirements and memory usage. In addition, quantization can significantly improve the inference speed of the model, especially on devices with limited hardware resources. These performance improvements are particularly important.

[0072] TensorRT is a high-performance API ecosystem for deep learning reasoning launched by NVIDIA. It includes reasoning runtime and model optimization, providing low latency and high throughput for production applications. TensorRT is specially optimized for NVIDIA GPUs, which can significantly improve the reasoning speed, reaching a reasoning speed 36 times faster than using only CPU platforms. TensorRT supports a variety of deep learning frameworks, such as TensorFlow, PyTorch, ONNX, etc. Through the integrated interfaces of these frameworks, the process of optimizing and deploying models to the TensorRT runtime environment can be simplified. It also provides rich functions, including layer fusion, kernel automatic tuning, quantization, etc., which can further optimize the reasoning performance of deep learning models. TensorRT has a wide range of application scenarios, including edge computing, vision and image processing, speech and voice recognition, natural language processing, and intelligent video analysis. It is particularly suitable for applications that require real-time response, such as self-driving cars, intelligent monitoring systems, etc. First, TensorRT builds an original model engine represented by 32-bit floating point numbers. Next, inference is performed on the engine using a calibration data set to collect statistics for the activation values ​​of each tensor in the model and generate a histogram. Using the collected histogram data, TensorRT builds a calibration table that contains the quantization scale factors for each tensor. Finally, based on the calibration table and the original definition of the model, TensorRT builds an optimized INT8 model engine to accelerate the inference process. Through quantization, the model can achieve faster response times and more efficient operation while maintaining a similar level of accuracy. The final model tested under the same conditions has an inference speed increase of about 66%, while the model size has been reduced by 60%, which is suitable for resource-constrained devices. The model can accurately detect ship targets in the ship intelligent collision avoidance decision-making system in real time, and can provide timely and effective target information for intelligent collision avoidance decisions.

[0073] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0074] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for rapid identification of marine ship targets based on YOLOv8, characterized in that: The following steps are involved: S1, collecting images of various types of ships in the three-dimensional simulation system; S2. Cleaning the collected image data and marking the target type of ships therein; S3, performing data enhancement processing on the annotated image data; S4, dividing the image data after data enhancement processing into a training set and a validation set, and using the YOLOv8 neural network pre-training model to perform transfer learning based on the training set to obtain a preliminary training model; S5, fine-tuning the parameters of the preliminary training model to obtain a preliminary detection model; S6. Use the deep learning tool TensorRT to calibrate and quantize the preliminary detection model to obtain the final detection model; S7. Use the final detection model to identify marine ship targets.

2. According to a method for rapid identification of marine ship targets based on YOLOv8 according to claim 1, it is characterized in that: S1 is specifically as follows: each type of ship in the three-dimensional simulation system is rotated multiple times, and images of each rotation angle of all ships are collected at multiple time points in a day. On this basis, images of different rotation angles at different time points in different weather conditions are collected to obtain images of all rotation angles of all types of ships at all weather conditions and at all time points.

3. According to a method for rapid identification of marine ship targets based on YOLOv8 according to claim 1, it is characterized in that: In S1, the collected ship images are divided into two types according to their uses: military ships and civilian ships.

4. According to a method for rapid identification of marine ship targets based on YOLOv8 according to claim 1, it is characterized in that: S2 specifically includes: clearing irrelevant images and images that do not meet the requirements during the acquisition process, and using the labelimg annotation tool for classification and annotation.

5. According to a method for rapid identification of marine ship targets based on YOLOv8 according to claim 1, it is characterized in that: S3 is specifically as follows: the annotated image data is subjected to five data enhancement processes. Each data enhancement uses a random number to randomly mix several enhancement methods and adjust the target bounding box accordingly. The data enhancement processes include randomly adding Gaussian noise, randomly masking areas, rotating images at random angles, randomly translating images horizontally, randomly translating images vertically, randomly cropping images except for the target, randomly flipping horizontally, randomly flipping vertically, randomly flipping diagonally, and randomly adjusting image brightness.

6. A method for rapid identification of marine ship targets based on YOLOv8 according to claim 1, characterized in that: In S4, the original samples without data augmentation are divided into the validation set of the data set, and the enhanced samples after five data augmentations are divided into the training set of the data set. The ratio of training set to validation set is 5:

1. On this basis, the background image is added to the training set, and the training set is put into the YOLOv8 pre-trained neural network model for migration training to obtain a preliminary training model.

7. The method for rapid identification of marine ship targets based on YOLOv8 according to claim 1, characterized in that: The sample allocation strategy in the transfer training in S4 uses the Task-AlignedAssigner positive and negative sample allocation strategy. During training, the allocation ratio of positive and negative samples is dynamically adjusted, and a weight is assigned to each sample. In the positive and negative sample allocation strategy, the classification and regression scores are used as weighted scores to select positive samples.

8. According to a method for rapid identification of marine ship targets based on YOLOv8 according to claim 1, it is characterized in that: S5 performs sparse pruning training on the preliminary training model. Specifically: S51 constrained training adds L1 constraints to the weights and biases of the batch normalization layer in the preliminary training model, with lambda taking the value of 1e-3: Where l1_lambda is the coefficient that controls the regularization strength, epoch represents a complete cycle in the model training process, and self.epochs represents the total number of training cycles for the current model instance; S52, taking the model obtained in S51 as a pruning object, traversing the batch normalization layer in the model, collecting the absolute values ​​of weights and biases, concatenating the absolute values ​​of weights of the batch normalization layer, sorting the concatenated absolute value tensors of weights, and calculating a threshold. If the absolute value of the weight is greater than or equal to the threshold, it is retained; if the absolute value of the weight is less than the threshold, it is pruned; S53, fine-tune the model. Use the pruned model obtained in S52 to perform 100 epochs of migration training to restore the model accuracy and performance, and obtain the final fine-tuned model.

9. The method for rapid identification of marine ship targets based on YOLOv8 according to claim 1, characterized in that: S6 is specifically as follows: the validation set part of the data set is separated as the model calibration data set, and the entropy calibrator Int8EntropyCalibrator2 of the deep learning tool TensorRT is used to run the original FP32 model on the calibration data set, collect the activation value histogram of each layer, and find the threshold T that minimizes the KL divergence for quantization. Based on the calibration results, the deep learning tool TensorRT will generate the final detection model after quantization, and finally perform a performance test on the final detection model.

Images