Ship detection method suitable for extreme weather at sea, electronic equipment and computer storage medium

Abstract

The invention discloses a ship detection method suitable for offshore extreme rain and snow weather, electronic equipment and a computer storage medium, and belongs to the field of offshore ship detection. Comprising the following steps: processing a ship image data set through a dark channel prior defogging algorithm, ensuring equalization of various ship samples through data diffusion, extracting ship features, performing effective information fusion among features of different scales by a feature fusion network, classifying and positioning deep and shallow features fused with different levels, and obtaining a ship image data set; therefore, the classification and the position of the target ship are obtained, the detection frames which are low in confidence coefficient and are excessively overlapped and the position and category information of the ship in each frame are screened out through a maximum suppression algorithm, and the real-time detection of the marine ship video stream is completed.

Description

Technical Field

[0001] The present invention belongs to the field of marine ship detection, and more specifically, relates to a ship detection method, electronic equipment and computer storage medium suitable for extreme weather at sea. Background Art

[0002] With the development of deep learning technology, smart water transportation uses intelligent technology to manage ships and applies computer vision technology to monitor ship driving, which can effectively reduce shipping supervision costs and improve the intelligent supervision capabilities of key sensitive areas. At present, the target detection method based on deep learning has made breakthrough progress in the construction of smart ports. However, in actual applications, real-time marine ship detection in complex scenarios such as heavy fog, rain and snow, weak lighting conditions, and multi-scale ships still has missed detections and false detections. At the same time, this complex use environment at sea also puts higher requirements on the calculation speed and convenience of the ship target detection system. Summary of the invention

[0003] In view of the defects of the prior art, the purpose of the present invention is to provide a ship detection method, electronic equipment and computer storage medium that are suitable for extreme weather at sea, aiming to solve the problems of slow detection speed, poor anti-interference ability and poor convenience of ship detection system devices in complex weather conditions in current ship detection models.

[0004] To achieve the above object, the present invention provides a ship detection method suitable for extreme weather at sea, comprising the following steps:

[0005] Step 1: Collect ship images under extreme marine weather conditions and classify the ship types in all ship images to build a ship image dataset;

[0006] Step 2: Dehaze the ship image dataset to remove background noise;

[0007] Step 3: Perform data augmentation on the dehazed ship image dataset;

[0008] Step 4: Select a first preset number of ship images from the ship data set after data amplification and input them into the neural network model for pre-training to obtain an initial pre-trained network model;

[0009] Step 5: Modify the initial pre-trained network model, use the silm neck as its feature fusion network to extract and fuse features of different scales, and select a second preset number of ship images for a second training;

[0010] Step 6: Input the collected video frame data into the ship target detection model obtained by the second training, and output the detection frame of the ship;

[0011] Step 7: Set the confidence threshold and IOU threshold, remove the detection frames with confidence less than the confidence threshold, and perform IOU matching on the remaining detection frames. The detection frame with IOU matching results greater than the IOU threshold is used as a single valid detection frame to obtain the location and category information of the ship.

[0012] Furthermore, the defogging process of the ship image dataset in step 2 includes:

[0013] For each original image in the ship image data set, the minimum value of the three channels of RGB is obtained, and then a grayscale image, i.e., a dark channel, is formed after minimum value filtering;

[0014] Extract pixels with brightness greater than the preset value from all dark channels, record the positions of all pixels, find the value corresponding to the point with the brightest brightness at the corresponding position in the ship image dataset as the atmospheric light value, and obtain the transmittance from the ratio of the brightness of the original image to the atmospheric light value, and use the formula Calculate the dehazed image;

[0015] Where t(x) is the transmittance, t0 is the transmittance threshold, A is the atmospheric light value, I(x) is the original image, J(x) is the dehazed image, and x is the spatial coordinate of the pixel.

[0016] Furthermore, the neural network model in step 4 is a yolov5s network model.

[0017] Furthermore, the computing framework of the feature fusion network includes a GSConv module.

[0018] Furthermore, the use of the silm neck as a feature fusion network for extracting and fusing features of different scales includes:

[0019] The input is features of different levels, which are divided into shallow features, middle features and deep features;

[0020] The deep features are input into the GSConv module, one output of the GSConv module is connected to the upsample module and Concat, the upsample branch is used to perform feature fusion with the middle-layer features through concat, and the other output of the GSConv module is used to perform concat with the downsampled result of the fused features;

[0021] After the deep features and the middle-level features are fused through concat, they enter the VOVGSCSP module for further extraction and fusion, and then are fused with the shallow features through the upsample branch. After the fusion with the shallow features is completed, they enter the VOVGSCSP module for the final extraction and output one channel of data. At the same time, the output of the VOVGSCSP module is connected to the GSConv module for downsampling, thereby outputting data of three different scales on the downsampling branch for subsequent detection.

[0022] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned method for detecting ships in extreme weather at sea when executing the computer program.

[0023] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0024] Compared with the existing solutions, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0025] 1. The present invention provides a ship detection method suitable for extreme weather at sea, which adopts a lightweight and flexible deep learning algorithm model for detection. The feature fusion network effectively fuses information between features of different scales, classifies and locates deep and shallow features that are integrated at different levels, thereby obtaining the classification and location of the target ship.

[0026] 2. In the feature fusion stage of the algorithm model, the present invention uses the GSConv module to replace the original Conv module, and reduces the overall calculation amount and the calculation time of the model by adding the deep separable convolution of channel sparse calculation, thereby improving the detection speed.

[0027] 3. The ship detection method adapted to extreme weather at sea provided by the present invention incorporates a defogging image enhancement algorithm, which solves the problem of unclear ship images under extreme weather at sea. In a marine environment with limited computing resources, it has the application advantages of low power consumption, long endurance, and high efficiency, and plays an intelligent auxiliary role in maritime traffic management, accident investigation, combating illegal sand mining, ship lock charging, and combating maritime smuggling activities.

[0028] 4. The present invention provides an electronic device and storage medium for ship detection that is suitable for extreme weather at sea. The zcu104 FPGA development board is used as the deployment platform for the overall algorithm. The low power consumption, long battery life and stable computing characteristics of the FPGA are utilized. It does not require remote connection control and can flexibly work in various scenarios such as docks and ports. It has more practical advantages in terms of power consumption, cost, efficiency and convenience, and can meet the practical needs of the scenario of ship detection under extreme weather at sea. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall structure provided by an embodiment of the present invention;

[0030] Figure 2 Schematic diagram of a dark channel priori defogging algorithm provided by an embodiment of the present invention;

[0031] Figure 3 is a schematic diagram of a feature fusion network structure provided by an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of GSConv calculation provided by an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of GSConv Bottleneck calculation provided in an embodiment of the present invention;

[0034] Figure 6 A schematic diagram of VoVGSCSP calculation provided by an embodiment of the present invention;

[0035] Figure 7 It is an overall schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] In view of the defects of the prior art, the purpose of the present invention is to provide a ship detection method and electronic equipment suitable for extreme weather at sea, which has the characteristics of high accuracy, strong anti-interference ability, and convenient use.

[0039] The overall structure of the present invention is as follows Figure 1 As shown, the following steps are included:

[0040] Step 1: Collect ship images under extreme marine weather conditions and build image datasets of different ship types;

[0041] Step 2: Use the dark channel prior algorithm to defog the ship image dataset to obtain a clear ship image. The algorithm relies on the atmospheric scattering model for defogging. By observing and summarizing a large number of foggy and non-fog images, some mapping relationships are obtained. Then, the inverse operation is performed according to the formation process of the foggy image to restore the clear image.

[0042] Step 3: The dehazed image data is cropped, flipped, and noise is added to ensure the balance of data samples of various types of ships;

[0043] Step 4: Select a first set number of images from the amplified ship data set and input them into the yolov5s network model for pre-training to obtain an initial pre-trained network model;

[0044] Step 5: Modify the initial pre-trained network model and use the lightweight and efficient silm neck as its feature fusion network to extract and fuse features of different scales, thereby reducing the computational complexity and time of the algorithm model itself, and select a second set number of ship images to input into the ship target detection model with the silm neck framework added for the second training;

[0045] Step 6: Input the video stream data collected by the camera into the ship target detection model obtained by the second training, and output the detection frame of the ship;

[0046] Step 7: Set the confidence threshold and IOU threshold, remove the detection frames with confidence less than the confidence threshold, perform IOU matching on the detection frames with high confidence, filter the detection frames with IOU matching results greater than the IOU threshold, leave a single valid detection frame, and finally output the location and category information of the ship.

[0047] Furthermore, the dark channel of the image refers to the fact that in most non-sky local areas, some pixels will always have at least one color channel with a very low value. In general pictures, except for the sky area, the value of one of the three RGB channels of the vast majority of pixels in the picture is relatively small. Therefore, it is necessary to obtain the minimum value of the three channels of each pixel in the picture, and filter the channel minimum values ​​of these pixels to form a grayscale image. At this time, each pixel has only one channel. This grayscale image is called the dark channel of the original image. For an image I(x), the mathematical definition of its dark channel is as follows:

[0048]

[0049] Among them, Ω(x) represents the local area centered on pixel x, the superscript c represents the three channels of RGB, and the dark channel of the image can be obtained by taking the minimum value of the three channels.

[0050] Furthermore, solving the atmospheric light value refers to taking the first 0.1% of pixels from the dark channel image according to the brightness, and recording the positions of these pixels, and finding the value corresponding to the point with the brightest brightness at the corresponding position in the original foggy image as the atmospheric light value A.

[0051] Furthermore, the transmittance calculation refers to estimating the transmittance through the original image and the obtained atmospheric light value. The transmittance formula is as follows:

[0052]

[0053] After the previous two steps, the dark channel and atmospheric light value A are obtained, and the transmittance can be obtained through the calculation formula.

[0054] Furthermore, the image restoration refers to the process of describing the physical mapping relationship between the original image and the defogging image by transmittance, and performing inverse operation to obtain the defogging image. The relationship between the defogging image and the original image is as follows:

[0055]

[0056] Among them, t(x) refers to the transmittance, A refers to the atmospheric light value, I(x) refers to the original image, and J(x) is the dehazed image. t0 is the transmittance threshold. When the transmittance t(x) is very small, the value of J(x) will be too large, which will cause some parts of the image to be overexposed. Therefore, a threshold can generally be set to limit it. t0 is a set value of about 0.1 to prevent the image from being overexposed.

[0057] The YOLOV5 algorithm training includes building a network structure, using the gradient descent method for training, and saving the trained weight parameters.

[0058] The feature fusion network introduces GSConv operators and module elements with lower computational complexity but higher computational accuracy, and their design aims to reduce computational costs while maintaining or improving the learning ability of the model.

[0059] Furthermore, the GSConv operator is designed to solve the problem of slow ship detection in the original YOLOV5 model. In the backbone network of YOLOV5, the input image always passes through the process of spatial information to the channel. Each spatial compression and channel expansion of the feature map will lead to partial loss of semantic information. While maintaining a low time complexity, GSConv retains the hidden connections between channels as much as possible. This calculation method is particularly beneficial for environments with limited computing resources.

[0060] Furthermore, the feature fusion network, through modular design, flexibly constructs feature fusion networks of different numbers of layers in combination with the needs of ship target detection, thereby reducing computational complexity and inference time while maintaining accuracy.

[0061] The dark channel priori algorithm structure is as follows Figure 2 As shown. First, input a low-light or foggy input image. Secondly, perform dark channel extraction on the input image, and for each pixel of the input image, calculate the minimum value of the three color channels to obtain a dark channel image. Then determine the atmospheric light value. Solving the atmospheric light value means taking the first 0.1% of pixels from the dark channel image according to the brightness, and recording the positions of these pixels, and finding the value corresponding to the point with the brightest brightness at the corresponding position in the original foggy image as the atmospheric light value A; then, perform transmittance calculation, and use the atmospheric light estimation value to calculate the transmission map. The transmission map is an image that represents the degree of atmospheric occlusion of each pixel. The calculation method is to divide each pixel value in the dark channel image by the atmospheric light value. Finally, restore the defogging image, use the transmittance and the original image, and perform element multiplication and subtraction operations to remove the influence of atmospheric light and restore the original brightness and contrast of the image. Finally, post-processing operations such as color correction and detail enhancement are required to improve the quality of the final image.

[0062] The silm neck feature fusion calculation architecture is as follows Figure 3As shown. The input is features of different levels, which can be divided into shallow features, middle features and deep features. The deep features will be input into the GSConv module, and the output of GSConv will be connected to the upsample module and Concat. The upsample branch is used to fuse the features with the middle features through concat, and the other output is used to concat with the downsampled results of the fused features; after the deep features are fused with the middle features through concat, they will enter the VOVGSCSP module for further extraction and fusion, and then fuse with the shallow features through the upsample branch. After the fusion with the shallow features is completed, it will enter the VOVGSCSP module for the final extraction and output one data. At the same time, the output of the VOVGSCSP module will be connected to the GSConv module for downsampling. This is to obtain features of different sizes after fusion, so as to facilitate the detection of ship targets of different scales. Therefore, 3 data of different scales will be output on the downsampling branch for subsequent detection.

[0063] The calculation process of the GSConv module in the Silm neck framework is as follows Figure 4 As shown in the figure. The feature map with input channel C2 is first input into a convolution layer, and the number of output channels of this layer is C2 / 2; then the output result of the convolution layer is subjected to depthwise separable convolution, that is, convolution is performed independently on each channel of the input feature map; then, the output result of the convolution layer is concatenated with the output result of the depthwise separable convolution, and the concatenated result is rearranged to disrupt each feature channel to improve the information flow between features, and finally obtain the feature map with output channel C2; compared with the standard convolution, GSConv reduces the overall amount of calculation by adding depthwise separable convolution with channel sparse calculation, and maintains sufficient accuracy through channel rearrangement operation.

[0064] The calculation process of the GSConv Bottleneck module described in the Silm neck framework is as follows Figure 5 As shown. The GSConvBottleneck module consists of a GSConv module, a Conv module, and an adder. The feature map with input channel C2 is input into the GSConv module and the Conv module. The output result of the GSConv module is a feature map with a channel number of C2 / 2. This feature map will be input into the next GSConv module again, and the output channel number is a feature map of C2. The feature map of the GSConv module branch will be added to the feature map of the Conv module branch, and the result of the addition is the final output result.

[0065] The calculation process of the VoVGSCSP module described in the Silm neck framework is as follows Figure 6As shown. The overall calculation consists of the GSConvBottleneck module, the Conv module and the Concat module. The input feature map is divided into two branches for calculation. The branch where the GSConvBottleneck is located will first perform a standard convolution calculation on the input feature map, and input the result of the standard convolution calculation into the GSConv Bottleneck. The other branch will perform a standard convolution calculation on the input feature map. The result of the standard convolution calculation will be spliced ​​with the output result of the GSConv Bottleneck. The result of the splicing operation will be input into the Conv module. The output result of the Conv module is the output result of the VoVGSCSP module.

[0066] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Based on the above-mentioned ship detection method, the present invention also provides a ship detection electronic device that can be used in extreme weather at sea, the electronic device includes a USB interface camera, a core processing component zcu104 FPGA development board, a DP interface display screen and related mechanical components. Figure 7 Only some components of the electronic device are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0067] The camera is used to input the original video stream of the marine ship, and communicate data with the core processing component zcu104FPGA development board through the USB interface, and input the video into the subsequent ship target detection program.

[0068] The core processing component zcu104 FPGA development board includes a memory and a processor. The memory in the embodiment of the present invention refers to a DDR3 high-bandwidth memory. In some other embodiments, it can also be an external storage device in the ship detection equipment, such as a plug-in hard disk, a smart memory card, and a flash memory card. The processor in the embodiment of the present invention is a Cortex ARM A9 dual-core processing CPU. In some other embodiments, it can also be a microprocessor, a digital signal processor, or other data processing chip, which is used to run the program code stored in the memory, such as executing the above-mentioned ship detection method.

[0069] The display is used to display the location and category information of the ship detection and to display a visual user interface. The display communicates with the core processing component zcu104 FPGA development board through the DISPLAYPORT interface.

[0070] In an embodiment of the present invention, a computer storage medium is further provided, on which a ship detection program under extreme weather conditions is stored. When the ship detection program is executed by a processor, the following steps are included:

[0071] The dark channel prior algorithm is used to dehaze the ship image dataset to obtain clear ship images;

[0072] The data amplification methods such as cropping, flipping and adding noise to the dehazed image data ensure the balance of data samples of various types of ships;

[0073] Select a first set number of images from the amplified ship data set and input them into the yolov5s network model for pre-training to obtain an initial pre-trained network model;

[0074] The initial pre-trained network model is modified, and the lightweight and efficient silm neck is used as its feature fusion network to extract and fuse features of different scales, thereby reducing the computational complexity and computational time of the algorithm model itself, and a second set number of ship images are selected and input into the ship target detection model with the silm neck framework added for the second training;

[0075] The video stream data collected by the camera is input into the ship target detection model obtained by the second training, and the detection frame of the ship is obtained as output;

[0076] The confidence threshold and IOU threshold are set, the detection frames with confidence less than the confidence threshold are removed, and the detection frames with high confidence are matched by IOU. The detection frames with IOU matching results greater than the IOU threshold are screened, leaving a single valid detection frame, and finally the location and category information of the ship are output.

[0077] In summary, the present embodiment provides a ship detection method, electronic device and computer storage medium suitable for extreme weather at sea, the method comprising: using a dark channel prior algorithm to perform defogging on a ship image data set to obtain a clear ship image; performing data amplification methods such as cropping, flipping and adding noise to the defogged image data to ensure the balance of various types of ship data samples; selecting a first set number of images from the amplified ship data set and inputting them into the yolov5s network model for pre-training to obtain an initial pre-trained network model; modifying the initial pre-trained network model and using a lightweight and efficient silm neck as its feature fusion network to extract and fuse features of different scales, thereby reducing the computational complexity and computational time of the algorithm model itself, and selecting a second set number of ship images and inputting them into the silm added network model. The second training is performed in the ship target detection model of the neck framework; the video frame data collected by the camera is input into the ship target detection model obtained by the second training, and the detection frame of the ship is output; the confidence threshold and the IOU threshold are set, the detection frames with confidence less than the confidence threshold are removed, and the detection frames with high confidence are matched by IOU, and the detection frames with IOU matching results greater than the IOU threshold are screened, leaving a single valid detection frame, and finally the position and category information of the ship are output;

[0078] The embodiment of the present invention achieves an mAP accuracy of 98.19% for ship detection under extreme weather conditions at sea, an accuracy rate of 97.068%, and a recall rate of 94.63%. Compared with the traditional YOLOV5 algorithm model, the algorithm model optimized by the present invention processes 80.5 frames per second, and the acceleration ratio reaches 1.41, thanks to the lightweight and efficient design concept of the silm neck. At the same time, after combining the dark channel prior algorithm, the overall anti-interference ability of the detection system in rainy and snowy weather at sea is greatly improved. In addition, the present invention is based on the ZCU104 development board to complete integrated deployment and use. Compared with traditional terminal detection solutions, the solution of the present invention has more practical advantages in efficiency, convenience, and cost, and can meet the needs of marine ship detection.

[0079] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A ship detection method suitable for extreme weather at sea, characterized in that: The following steps are involved: Step 1: Collect ship images under extreme marine weather conditions and classify the ship types in all ship images to build a ship image dataset; Step 2: Dehaze the ship image dataset to remove background noise; Step 3: Perform data augmentation on the dehazed ship image dataset; Step 4: Select a first preset number of ship images from the ship data set after data amplification and input them into the neural network model for pre-training to obtain an initial pre-trained network model; Step 5: Modify the initial pre-trained network model, use the silm neck as its feature fusion network to extract and fuse features of different scales, and select a second preset number of ship images for a second training; Step 6: Input the collected video frame data into the ship target detection model obtained by the second training, and output the detection frame of the ship; Step 7: Set the confidence threshold and IOU threshold, remove the detection frames with confidence less than the confidence threshold, and perform IOU matching on the remaining detection frames. The detection frame with IOU matching results greater than the IOU threshold is used as a single valid detection frame to obtain the location and category information of the ship.

2. The method according to claim 1, characterized in that The defogging process for the ship image dataset in step 2 includes: For each original image in the ship image data set, the minimum value of the three channels of RGB is obtained, and then a grayscale image, i.e., a dark channel, is formed after minimum value filtering; Extract pixels with brightness greater than the preset value from all dark channels, record the positions of all pixels, find the value corresponding to the point with the brightest brightness at the corresponding position in the ship image dataset as the atmospheric light value, and obtain the transmittance from the ratio of the brightness of the original image to the atmospheric light value, and use the formula Calculate the dehazed image; Where t(x) is the transmittance, t0 is the transmittance threshold, A is the atmospheric light value, I(x) is the original image, J(x) is the dehazed image, and x is the spatial coordinate of the pixel.

3. The method according to claim 1, characterized in that The neural network model in step 4 is the yolov5s network model.

4. The method according to claim 3, characterized in that The computing framework of the feature fusion network includes a GSConv module.

5. The method according to claim 4, characterized in that The silm neck is used as its feature fusion The network is used to extract and fuse features of different scales, including: The input is features of different levels, which are divided into shallow features, middle features and deep features; The deep features are input into the GSConv module, one output of the GSConv module is connected to the upsample module and Concat, the upsample branch is used to perform feature fusion with the middle-layer features through concat, and the other output of the GSConv module is used to perform concat with the downsampled result of the fused features; After the deep features are fused with the middle-level features through concat, they enter the VOVGSCSP module for further extraction and fusion, and then are fused with the shallow features through the upsample branch. After the fusion with the shallow features is completed, they enter the VOVGSCSP module for final extraction and output one channel of data. At the same time, the output of the VOVGSCSP module is connected to the GSConv module for downsampling, thereby outputting data of three different scales on the downsampling branch for subsequent detection.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for detecting ships in extreme weather at sea described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images