Ship type detection method and device based on improved YOLOv5

Abstract

The invention provides a ship type detection method and device based on improved YOLOv5. A lighter GhostConv convolution module is adopted as the main convolution operation of YOLOv5, meanwhile, a CA attention module is introduced, channel information and position information related to the direction are fully considered, the parameter quantity and the calculated quantity of the model are remarkably reduced, sufficient flexibility and light weight are guaranteed, and different types of operation ships on the sea can be accurately detected in real time by means of the device.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a ship type detection method and device based on improved YOLOv5. Background Art

[0002] Compared with land traffic congestion, shipping on water plays an important role in maritime transportation due to its large carrying capacity and low energy consumption. However, with the increasing complexity of water transportation scene environment, it faces many problems, such as low ship target clarity and diversified types, making it relatively difficult to identify ships of different types and sizes in intelligent monitoring and analysis. Therefore, using computer vision technology, studying deep learning algorithms suitable for ship type detection and edge computing intelligent devices suitable for target detection algorithms is of great significance for serving various maritime applications.

[0003] In recent years, with the combination of artificial intelligence technology and edge computing smart devices, target detection algorithms have achieved remarkable intelligent detection effects in the field of video surveillance. At the same time, transplanting the algorithm model trained on the server side and deploying it on the edge computing smart device can better perform target detection tasks in actual application scenarios.

[0004] At present, the research methods for marine ship target detection mainly include the YOLO series, SSD series and FasterR-CNN series. In view of the high efficiency of the current domestic edge computing intelligent devices in algorithm transplantation and deployment, the YOLO series algorithm, with its complete and systematic feature engineering, significantly shortens the training time of the model on the local side and the migration and deployment time on the edge side, thereby improving the accuracy and efficiency of target detection. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a ship type detection method and device based on improved YOLOv5. A more lightweight GhostConv convolution module is used as the main convolution operation of YOLOv5, and a CA attention module is introduced at the same time, which fully considers the channel information and direction-related position information, and significantly reduces the number of parameters and calculations of the model, ensuring sufficient flexibility and lightness. Using the device of the present invention, different types of operating ships at sea can be detected in real time and accurately.

[0006] A ship type detection method based on improved YOLOv5, characterized by the following steps:

[0007] Step 1, data collection: collect video surveillance data of ships at sea in different time periods and weather conditions as the initial data set;

[0008] Step 2, data annotation and image preprocessing: convert the videos in the initial data set into image format, and annotate the different types of ships contained in the images respectively; expand the annotated images, and divide the expanded image data set into training data set, verification data set and test data set;

[0009] Step 3, build an improved YOLOv5 detection model: first, use the GhostConv convolution module to replace the convolution operation in the backbone network of the original YOLOv5 model; then, add the CA attention module in the layer before the SSPF module of the backbone network to obtain the improved YOLOv5 detection model;

[0010] Step 4, model training: using the training data set obtained in step 2 as input and the target bounding box and category probability as output, the improved YOLOv5 detection model constructed in step 3 is trained to obtain a trained detection model;

[0011] Step 5, model application: input the verification data set into the trained detection model for model verification; input the test data set into the trained detection model, and output the target bounding box and category probability.

[0012] Specifically, the image format is JPEG format, and the video data is converted into a JPEG image set through the mp4ToJpeg.py file.

[0013] Specifically, the labelimg tool is used to label the different types of ships contained in the image.

[0014] Specifically, the ship types include passenger ships, fishing boats, container ships and general cargo ships.

[0015] Specifically, the expansion processing includes mirror flipping, picture rotation, scaling and brightness adjustment operations.

[0016] Specifically, the ratio of the number of images contained in the training data set, the validation data set, and the test data set is 7:2:1.

[0017] Specifically, the YOLOv5 model mainly includes a backbone network Backbone, a neck network Neck and a prediction module Prediction, wherein the backbone network Backbone includes a standard convolutional layer, a CSP module and an SSPF module, and uses a deep convolutional neural network to extract features from the input image; the neck network Neck integrates multi-scale features from the backbone network Backbone and performs feature fusion; the prediction module Prediction performs target detection on the feature map processed by the neck network Neck, and outputs the target bounding box and category probability.

[0018] Specifically, the specific processing process of the GhostConv convolution module is as follows: first, the input feature map is processed by the convolution layer and is divided into multiple channel groups. Each channel group uses different convolution kernels to perform independent convolution operations to extract local features of different channel groups. At the same time, the input feature map is directly passed to the output layer through the residual connection, and the local feature map after convolution processing is combined with the feature map of the residual connection to obtain the final output feature map.

[0019] Specifically, the specific processing process of the CA attention module is as follows: first, the original input feature map is directly passed to the output layer through a residual connection, and the input feature map size is C×H×W, where C represents the number of channels, H represents the height, and W represents the width; then, the feature map is subjected to X-direction average pooling and Y-direction average pooling operations respectively to obtain feature maps of sizes C×1×W and C×H×1, respectively, and the pooled feature maps are concatenated through the Concat operation to obtain a feature map of size C / r×1×(W+H), where r represents the channel compression rate, and then through the convolution layer, batch normalization and nonlinear activation function processing, the obtained feature map is divided into two parts, and is processed by the convolution layer and the Sigmoid activation function respectively to generate weight matrices for different dimensions, with sizes C×1×W and C×H×1, respectively; finally, the generated weight matrix is ​​weighted with the original input feature map channel by channel, and the weight-adjusted feature map is output, and the size is still C×H×W.

[0020] Specifically, when training the improved YOLOv5 detection model, the Adam optimizer is used, the initial learning rate is set to lr=0.003, the training batch size is set to batch size=20, the momentum is set to momentum=0.9, and the number of training iterations is set to 400.

[0021] Specifically, the specific process of step 3 is: first write the GhostConv module and CA attention module code into the common.py configuration file of the yolov5 model, then create a new yolov5_Ghost.yaml file, change the corresponding convolution structure in the backbone network of the yolov5 model to the GhostConv mode, add judgment conditions in the yolo.py file, and add a CA attention module in the layer before the SSPF module of the backbone network to obtain an improved YOLOv5 detection model.

[0022] Specifically, the specific process of step 4 is: use the improved YOLOv5 detection model as a trainer, and transfer it to the local server together with the data set obtained in step 2, use the NVIDIA GPU graphics card to complete the training process, and generate the pt best detection model file boat_best.pt.

[0023] Specifically, the specific process of step 5 is: convert the trained best detection model pt file into a model in onnx format, and then use the model conversion tool to convert it into a Bmodel format file to generate a boat_best.bmodel model file, use the bmodel format model file to infer the validation set, verify the inference results, and detect the test data set to obtain the target bounding box and category probability.

[0024] The present invention also provides a ship type detection device based on the improved YOLOv5 based on the above method, which is characterized by comprising:

[0025] Data collection equipment, including cameras with video and photo functions and monitoring devices, used to record videos to obtain video data sets;

[0026] The server is used for data annotation and image preprocessing, building an improved YOLOv5 detection model, model training, and using the model conversion tool to convert the trained model into the model file required by the edge smart device;

[0027] The edge intelligent device is used to load the model file and run the program to realize ship target detection, visualize the detection results, and store the detection results.

[0028] Specifically, the edge intelligent device is an edge computing intelligent device including an embedded Linux system.

[0029] The beneficial effects of the present invention are as follows: since an improved YOLOv5 detection model based on a CA attention module and a lightweight GhostConv module is constructed, the significance of each channel can be better captured, so that the model can more effectively utilize useful information and reduce the interference of irrelevant information; at the same time, the GhostConv module introduces a "ghost" operation to split the complex convolution operation into multiple simpler operations, which can significantly reduce the amount of parameters and calculations. Compared with traditional convolutional neural networks, this not only reduces storage requirements, but also reduces training time and resource consumption; in addition, the design based on GhostConv is very suitable for deployment on edge computing intelligent devices, can achieve efficient edge computing, and enable deep learning algorithms to run efficiently on embedded devices, thereby effectively ensuring the accuracy and efficiency of real-time detection of ships at the edge. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a ship type detection method based on improved YOLOv5 of the present invention;

[0031] Figure 2 is a frame of image according to an embodiment of the present invention;

[0032] Figure 3 It is a structural diagram of the YOLOv5 detection model after the GhostConv convolution module of the present invention is replaced;

[0033] Figure 4 Schematic diagram of the GhostConv convolution module of the present invention;

[0034] Figure 5 It is a schematic diagram of the CA attention mechanism module of the present invention;

[0035] Figure 6 It is a detection effect diagram of the improved YOLOv5 detection model of the present invention;

[0036] Figure 7 It is a schematic diagram of a ship type detection device based on improved YOLOv5 of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The present invention includes but is not limited to the following embodiments.

[0038] The present invention provides a ship type detection method based on improved YOLOv5, such as Figure 1 As shown, the specific implementation process is as follows:

[0039] Step 1, data collection: Collect ship video surveillance data at sea in different time periods and under different weather conditions as the initial data set. Figure 2 A certain frame of visualized image according to an embodiment of the present invention is provided.

[0040] Step 2, data labeling and image preprocessing: Convert the video data in the initial data set into JPEG format images through the mp4ToJpeg.py file, and label the different types of ships contained in the images separately. You can label them manually or use the labelimg tool to label all the ships contained in the images. The labeled ship types include passenger ships, fishing boats, container ships and general cargo ships. In this way, a labeled data set is constructed.

[0041] The images in the labeled dataset are expanded, including mirror flipping, image rotation, scaling, and brightness adjustment. The dataset is then divided into training set, validation set, and test set in a ratio of 7:2:1.

[0042] Step 3: Build an improved YOLOv5 detection model: First, use the GhostConv convolution module to replace the convolution operation in the original YOLOv5 model backbone network, such as Figure 3 As shown, the specific structure is as follows:

[0043] a) Backbone network: This part is responsible for extracting the basic features of the input image. Starting from the input layer, the image passes through the Focus module, which downsamples the image information. Then, it passes through multiple layers of Conv (convolutional layer) and C3 modules (including three layers of convolution and feature fusion of cross-stage partial networks, often used for feature extraction). The GhostConv (lightweight convolution) module is then used to replace the original convolution operation to improve computational efficiency while retaining feature extraction capabilities. Finally, the SPPF (spatial pyramid pooling) module is used to aggregate features of different scales.

[0044] b) Neck network: This part builds a feature pyramid to fuse features of different scales. After upsampling the feature map, Concat is used between layers to integrate feature maps of different scales, and further processed by the C3 module. Through this multi-scale feature pyramid structure, the model can detect objects of different sizes.

[0045] c) Prediction module: This part generates the final detection output. The multi-scale feature map is processed by convolution to obtain outputs of 80×80×11, 40×40×11 and 20×20×11 sizes, respectively. Each output corresponds to the detection of objects of different scales. Finally, through these outputs of different scales, large, medium and small objects can be detected.

[0046] Figure 3 In the figure, C3 represents the cross-stage partial network (CSP) module, which is responsible for extracting and fusing multi-level features; Concat: concatenating feature maps from different layers; Conv: standard convolution operation for feature extraction; Upsample: upsampling operation, used to enlarge the feature map to match the size of feature maps in other layers.

[0047] The structure of the GhostConv convolution module of the present invention is as follows Figure 4 As shown in Figure 1, the specific processing process is as follows: the input feature map is processed by the convolution layer (Conv) and divided into multiple channel groups, each of which uses different convolution kernels ω1, ω2, …, ω p Independent convolution operations are performed to extract local features in different channel groups. At the same time, the residual connection (Identity) passes the input directly to the output layer (Output) to retain the original feature information. The feature map after convolution processing is combined with the feature map of the residual connection to form the final output feature map. This structure reduces the amount of calculation through grouped convolution and uses residual connections to avoid information loss, thereby improving the network's feature extraction ability and training efficiency.

[0048] In order to improve the feature expression capability, the present invention adds a CA attention module in the layer before the SSPF module of the YOLOv5 model backbone network Backbone to obtain the final improved YOLOv5 detection model. Figure 5 As shown in the figure, the specific processing process is as follows: First, the input feature map [C, H, W] of size C×H×W is globally averaged pooled in the X and Y directions, and average pooling is performed along different dimensions, X Avg Pool is pooled along the height direction, and Y AvgPool is pooled along the width direction, so as to extract global spatial information. After average pooling in the X direction, a feature layer [C, H, 1] is generated, and after average pooling in the Y direction, a feature map size of [C, 1, W] is generated, where C represents the number of channels, H represents the height of the feature map, and W represents the width of the feature map. Then, the pooled feature maps are concatenated through the Concat operation to obtain a feature map of size C / r×1×(W+H) (r represents the channel compression ratio), and then further extracted through the convolution layer (Conv2d, parallel 1×H and W×1 convolution operations). The convolutional feature map is processed by batch normalization (BatchNorm) and nonlinear activation function to generate a new feature map to enhance the expressiveness of the model. Next, the new feature layer is split along the spatial dimension, processed by the convolution layer and the Sigmoid activation function respectively, and the weight matrices for different dimensions are generated, with sizes of [C,H,1] and [C,1,W] respectively; finally, the generated weight matrix is ​​re-weighted with the original input feature map channel by channel, and the original feature map is adaptively adjusted. The weighted feature map is output (Output), and the size is still C×H×W, but the expressiveness of the feature is improved by re-weighting.

[0049] Furthermore, when building an improved YOLOv5 detection model, first write the GhostConv module and CA attention module codes into the common.py configuration file of the yolov5 model, then create a new yolov5_Ghost.yaml file, change the corresponding convolution structure in the backbone network of the yolov5 model to the GhostConv mode, add judgment conditions in the yolo.py file, and add a CA attention module in the layer before the SSPF module of the backbone network to obtain an improved YOLOv5 detection model.

[0050] Step 4, model training: Take the training data set obtained in step 2 as input, and the target bounding box and category probability as output, and train the improved YOLOv5 detection model constructed in step 3. That is, use the improved YOLOv5 detection model as a trainer, and pass it to the local server together with the data set obtained in step 2, and use the NVIDIA GPU graphics card to complete the training process. During training, use the Adam optimizer, the initial setting of the learning rate lr = 0.003, the training batch size is set to batch size = 20, the momentum is set to momentum = 0.9, and the number of training iterations is set to 400. After training, the improved YOLOv5 detection model with the best detection effect is obtained, and the pt detection model file boat_best.pt is generated.

[0051] Step 5, model application: Convert the trained best detection model pt file into a model in onnx format, and then use the model conversion tool to convert it into a Bmodel format file to generate the boat_best.bmodel model file. Use the bmodel format model file to infer the validation set, verify the inference results, and detect the test data set to obtain the target bounding box and category probability. Figure 6 The detection effect diagram of the improved YOLOv5 detection model of the present invention is given, including the detection frame, ship type category and category probability. For example, the category probability of detecting a general cargo ship is 89.5%.

[0052] The present invention also provides a ship type detection device based on the improved YOLOv5 based on the above method, such as Figure 7 As shown, it includes: data acquisition equipment, server and edge intelligent devices.

[0053] Among them, data acquisition equipment is mainly used to record videos to obtain video data sets, including cameras with video and photo functions, monitoring devices, etc. For example, the video recorded by the acquisition equipment is organized into a video data set, and the video data set contains video data of different image acquisition devices at different times of the day, with a video resolution of 1920×1080, a frame rate of 25fps, and H.264 for video compression.

[0054] The server is mainly used for data annotation and image preprocessing, building an improved YOLOv5 detection model, model training, and using the model conversion tool to convert the trained model into the model file required by the edge smart device.

[0055] The edge intelligent device is mainly used to load model files and run programs to realize ship target detection. At the same time, the edge device visualizes the detection results and stores the detection results in video or picture format to the edge for real-time monitoring.

[0056] Specifically, the edge intelligent device is an edge computing intelligent device that includes an embedded Linux system and is connected to an external network camera. The edge detect mode is started, and the captured images are transmitted to the edge for detection reasoning. According to the classification results, it is determined whether the ship is a passenger ship, fishing boat, container ship or general cargo ship; at the same time, the position, target category and confidence of the target box Box are drawn on the original image to visualize the detection results, and the detection results are stored in the edge in video format for download and viewing. Box is represented as follows:

[0057] Box i ={x i ,y i , w i ,hi,class i ,confidence i} (1)

[0058] class i ∈{passenger ship, fishing vessel, container ship, general cargo ship} (2)

[0059] Among them, the detection box obtained by detecting the i-th frame image, x i is the horizontal coordinate of the box, y i is the vertical coordinate of the box, w i is the width of the box, h i is the height of the box, class i The target category is a passenger ship, fishing vessel, container ship or general cargo ship. i is the confidence of the box.

[0060] The present invention discloses a ship type detection method and device based on improved YOLOv5, which utilizes lightweight GhostConv convolution as the main convolution operation of YOLOv5, and introduces Coordinate Attention (CA) module to enhance the processing capability of channel and direction related position information. The detection model generated by training is migrated and quantized through conversion tools and deployed to edge computing intelligent devices. After optimization, the model has significant improvements in accuracy and efficiency, and is particularly suitable for real-time detection of different types of operating ships at sea. The advantage of the present invention is that, through the introduction of lightweight networks and attention mechanisms, the number of parameters and the amount of calculation of the model are significantly reduced, so that it can run efficiently on embedded devices, ensuring the accuracy and efficiency of real-time detection at the edge.

Claims

1. A ship type detection method based on improved YOLOv5, characterized in that Here are the steps: Step 1, data collection: collect video surveillance data of ships at sea in different time periods and weather conditions as the initial data set; Step 2: Data labeling and image preprocessing: Convert the videos in the initial dataset into image format and label the different types of ships contained in the images. Expand the annotated images and divide the expanded image dataset into training dataset, verification dataset and test dataset; Step 3: Build an improved YOLOv5 detection model: First, use the GhostConv convolution module to replace the convolution operation in the original YOLOv5 model backbone network; then, add the CA attention module before the SSPF module in the backbone network to obtain the improved YOLOv5 detection model; Step 4: Model training: Using the training dataset obtained in step 2 as input and the target bounding box and category probability as output, the improved YOLOv5 detection model constructed in step 3 is trained to obtain a trained detection model. Step 5, model application: input the validation dataset into the trained detection model for model validation; The test dataset is input into the trained detection model, and the output is the target bounding box and category probability.

2. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: The image format is JPEG format, and the video data is converted into a JPEG image set through the mp4ToJpeg.py file.

3. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: Use the labelimg tool to label the different types of ships contained in the image.

4. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: The types of ships mentioned include passenger ships, fishing vessels, container ships and general cargo ships.

5. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: The expansion processing includes mirror flipping, image rotation, scaling and brightness adjustment operations.

6. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: The ratio of the number of images contained in the training dataset, validation dataset, and test dataset is 7:2:

1.

7. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: The YOLOv5 model mainly includes a backbone network, a neck network, and a prediction module. The backbone network includes a standard convolutional layer, a CSP module, and a SSPF module, and uses a deep convolutional neural network to extract features from the input image; the neck network integrates multi-scale features from the backbone network and performs feature fusion; the prediction module performs target detection on the feature map processed by the neck network, and outputs the target bounding box and category probability.

8. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: The specific processing process of the GhostConv convolution module is as follows: first, the input feature map is processed by the convolution layer and divided into multiple channel groups. Each channel group uses a different convolution kernel to perform independent convolution operations to extract local features of different channel groups. At the same time, the input feature map is directly passed to the output layer through the residual connection. The local feature map after convolution processing is combined with the feature map of the residual connection to obtain the final output feature map.

9. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: The specific processing process of the CA attention module is as follows: first, the original input feature map is directly passed to the output layer through a residual connection, and the input feature map size is C×H×W, where C represents the number of channels, H is the height, and W is the width; then, the feature map undergoes X-direction average pooling and Y-direction average pooling operations respectively to obtain feature maps of sizes C×1×W and C×H×1, respectively. The pooled feature maps are concatenated through a Concat operation to obtain a feature map of size C / r×1×(W+H), where r represents the channel compression rate. The feature map is then divided into two parts through a convolutional layer, batch normalization, and a nonlinear activation function, and is processed through a convolutional layer and a Sigmoid activation function respectively to generate weight matrices for different dimensions, with sizes C×1×W and C×H×1, respectively; finally, the generated weight matrix is ​​weighted with the original input feature map channel by channel, and the weight-adjusted feature map is output, and the size is still C×H×W.

10. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: When training the improved YOLOv5 detection model, the Adam optimizer is used, the initial learning rate is set to lr = 0.003, the training batch size is set to batch size = 20, the momentum is set to momentum = 0.9, and the number of training iterations is set to 400.

11. The ship type detection method based on improved YOLOv5 according to claim 1, characterized in that: The specific process of step 3 is: first write the GhostConv module and CA attention module code into the common.py configuration file of the yolov5 model, then create a new yolov5_Ghost.yaml file, change the corresponding convolution structure in the backbone network of the yolov5 model to the GhostConv mode, add judgment conditions in the yolo.py file, and add the CA attention module in the layer before the SSPF module of the backbone network to obtain the improved YOLOv5 detection model.

12. The ship type detection method based on improved YOLOv5 according to claim 11, characterized in that: The specific process of step 4 is as follows: the improved YOLOv5 detection model is used as a trainer, and it is transferred to the local server together with the dataset obtained in step 2. The NVIDIA GPU graphics card is used to complete the training process and generate the pt best detection model file boat_best.pt.

13. The ship type detection method based on improved YOLOv5 according to claim 12, characterized in that: The specific process of step 5 is as follows: convert the trained best detection model pt file into a model in onnx format, then use the model conversion tool to convert it into a Bmodel format file to generate the boat_best.bmodel model file, use the bmodel format model file to infer the validation set, verify the inference results, and detect the test data set to obtain the target bounding box and category probability.

14. A ship type detection device based on improved YOLOv5, characterized in that include: Data acquisition equipment, including cameras with video and photo functions and monitoring devices, used to record videos to obtain video data sets; The server is used for data annotation and image preprocessing, building an improved YOLOv5 detection model, model training, and using model conversion tools to convert the trained model into the model file required by edge smart devices; Edge-end intelligent devices are used to load model files and run programs to detect ship targets, visualize the detection results, and store the detection results.

15. The ship type detection device based on improved YOLOv5 according to claim 14, characterized in that: The edge intelligent device is an edge computing intelligent device that includes an embedded Linux system.

Images