Method for detecting fishing vessels based on improved YOLOv5 network from noctilucent remote sensing data

Abstract

The application discloses a kind of luminescent remote sensing data light lure fishing boat detection methods based on improved YOLOv5 network, using ship monitoring system data to the light lure fishing boat is verified, the light lure fishing boat on luminescent remote sensing image is marked, constructs a fishing boat target detection data set including various complex background noise;Proposed a kind of attention mechanism multi-scale fusion algorithm based on YOLOv5;And using the algorithm to luminescent remote sensing image data is detected, including light lure fishing boat target detection data set construction;YOLOv5-based attention mechanism multi-scale fusion light lure fishing boat detection algorithm;Light lure fishing boat position information product production method.The proposed light lure fishing boat target detection algorithm will provide technical support for the fishing effort estimation of open-sea and near-shore light lure fishery, fishing ground spatio-temporal variation, monitoring cross-border fishing applications, etc.

Description

Technical Field

[0001] This invention relates to the field of fisheries remote sensing information technology, specifically to a method for detecting fishing vessels using nighttime light remote sensing data based on an improved YOLOv5 network. Background Technology

[0002] Foreign methods for detecting light-lured fishing boats based on nighttime light remote sensing data include: using Wiener filtering and median filtering to convolve the image; removing noise such as smiley face noise and salt-and-pepper noise; employing a series of automatic methods such as peak detection algorithm and sharpness index; and using threshold segmentation to detect the brightness characteristics of the light-lured fishing boats on the nighttime light remote sensing image.

[0003] Existing methods for detecting fishing vessels using light-induced cull techniques based on nighttime remote sensing data have limitations. While these methods perform well under low lunar illumination, they become highly susceptible to false detections under full moon conditions due to variations in cloud brightness.

[0004] The complexity of the environment in which light-lured fishing vessels operate is mainly manifested in weather changes, cloud cover (affected by cyclones, clouds, and fog), and close operating distances. When affected by cloud scattering, these vessels tend to cluster together, resulting in the loss of significant feature information and a high false negative rate. Traditional threshold-based detection algorithms cannot extract effective features under complex conditions, leading to incomplete feature representation. Furthermore, the sensor's imaging incident angle, terminator, and spatial resolution significantly affect the extraction of light-lured fishing vessels from nighttime remote sensing images using traditional algorithms, as these are greatly influenced by moonlight intensity and cloud optical thickness. Summary of the Invention

[0005] This invention provides a method for detecting light-lured fishing vessels based on nighttime light remote sensing data using an improved YOLOv5 network. The purpose is to accurately extract light-lured fishing vessels amidst complex background noise such as weather changes, dense operations, and cloud scattering, thereby solving the problem of high-precision small target detection under complex background noise.

[0006] This application is achieved through the following technical solution:

[0007] A method for detecting light-lured fishing vessels based on nighttime remote sensing data using an improved YOLOv5 network is proposed. This method verifies the light-lured fishing vessels using Vessel Monitoring System (VMS) data, marks the light-lured fishing vessels on nighttime remote sensing images, and constructs a target detection dataset for fishing vessels containing various complex background noises. A multi-scale fusion algorithm based on the YOLOv5 attention mechanism is proposed, and this algorithm is used to perform target detection on nighttime remote sensing image data, specifically including the following three steps:

[0008] (1) Data structure for optically induced fishing vessel target detection;

[0009] (2) A multi-scale fusion light-induced fishing boat detection algorithm based on the attention mechanism of YOLOv5;

[0010] (3) Production method of light-induced fishing vessel position information products.

[0011] As a preferred embodiment, in order to achieve better target detection results, step (1) requires the construction of a specialized dataset for various complex background noise target detection based on the characteristics of day / night band (DNB) nighttime remote sensing images. The specific operation steps are as follows:

[0012] (11) Data preprocessing of DNB nighttime light remote sensing images;

[0013] (12) Annotation of images in the dataset.

[0014] As a preferred embodiment, the specific steps of the data preprocessing of the DNB nighttime light remote sensing image in step (11) are as follows:

[0015] (111) First, all DNB images are preprocessed;

[0016] (112) Multiply the DNB radiance value by one billion (10^9) and take the logarithm. This step is to enhance the contrast of the features and reduce the complexity of subsequent model calculations, thereby improving computational efficiency.

[0017] (113) Secondly, the VMS information of the light-induced fishing boat is cleaned and filtered to obtain the VMS record (time, longitude, latitude) that is closest to the DNB sensor shooting time (within 1 hour).

[0018] (114) The original DNB image of 3072×4064 pixels is cropped to obtain a 256×256 pixel slice containing the target of the light-induced fishing boat. In order to mark the position of the boat later, these boat slices will be converted into grayscale images (8-bit depth).

[0019] As a preferred embodiment, the specific steps for annotating the dataset images in step (12) are as follows:

[0020] (121) Using the latitude and longitude information provided by VMS information, the LabelImg tool is used to label the light-induced fishing boat targets on the grayscale image;

[0021] (122) Taking into account factors such as safe operating distance of vessels and spatial resolution of images, a 5×5 rectangular box was selected for marking. This dataset is similar to the PASCAL VOC dataset.

[0022] (123) Each light-attracting fishing vessel has a corresponding XML file for its slice, which marks the vessel's location, the types of objects contained in the slice, and the size of the vessel slice;

[0023] (124) The annotation format of this dataset can be used for object detection models such as SSD and Faster R-CNN;

[0024] (125) The dataset annotation format used by the YOLO series algorithms is different from the XML format mentioned above. After the series of algorithms are annotated, a TXT file is generated to store information such as the target category ID number, the x-coordinate of the target's center point (horizontal) / total image width, the y-coordinate of the target's center (vertical) / total image height, the width of the target bounding box / total image width, and the height of the target bounding box / total image height.

[0025] (126) In order to make the constructed dataset applicable to the YOLO series of algorithm models, the labeled documents of the dataset were transformed.

[0026] As a preferred embodiment, step (2) addresses the limitation that the convolution of the YOLOv5 network structure causes the loss of key image information in the deep feature map, resulting in incomplete and insufficient extraction of key image feature information. The attention mechanism module and the weighted bidirectional feature pyramid (BiFPN) are added to the YOLOv5 network structure to perform feature fusion. At the same time, a detection layer is added to improve the accuracy of small target detection in complex backgrounds.

[0027] As a preferred embodiment, step (3) takes into account the characteristics of the large swath of the night light remote sensing image and the relatively small size of the light-induced fishing boat, and designs a detection method for the actual night light remote sensing image according to the idea of ​​"segmentation-detection-merging".

[0028] As a preferred embodiment, the specific operation steps of step (3), the method for producing the light-induced fishing vessel position information product, are as follows:

[0029] (31) First, the DNB, h5 format image (3072×4064) is preprocessed for the detection model input. It is then cropped according to the rule grid (each grid is 256 rows × 256 columns) into 12 rows × 15 columns, for a total of 180 slices.

[0030] (32) Preprocess the slices using the same method as the constructed detection dataset;

[0031] (33) First, multiply the DNB radiance value by one billion (10^9) and take the logarithm;

[0032] (34) The 180 slices were then converted into grayscale images (8-bit depth);

[0033] (45) Call the trained object detection model to perform object detection;

[0034] (36) Output the detected images and the normalized center coordinates of each detected target and the confidence level of the detected target;

[0035] (37) Based on the row and column numbers of the original DNB image slices and the latitude and longitude information, the coordinates of the normalized center position point of each ship target are converted into latitude and longitude coordinates.

[0036] (38) Subsequently, through masking, erroneous targets on land were removed, and the latitude and longitude data of ships in the study area were obtained.

[0037] Beneficial effects: This application proposes a target detection algorithm for light-lured fishing vessels based on nighttime light remote sensing data using an improved YOLOv5 network. This algorithm will provide technical support for estimating fishing effort, monitoring spatiotemporal changes in fishing grounds, and monitoring cross-border fishing in light-lured fisheries in both distant and near-shore waters. Attached Figure Description

[0038] Figure 1 This is a schematic diagram illustrating the construction process of the target detection dataset for light-induced fishing vessels in this embodiment.

[0039] Figure 2 This is a schematic diagram of the network structure model in this embodiment.

[0040] Figure 3 This is a schematic diagram of the production process of the light-induced fishing vessel position information product in this embodiment.

[0041] Figure 4 This is a schematic diagram comparing different data and product test results under cloudless or thin cloud conditions in this embodiment.

[0042] Figure 5 This diagram illustrates a comparison of detection results between the VIIRS BoatDetection (VBD) algorithm and the deep learning algorithm in this embodiment.

[0043] Figure 6 This is a schematic diagram comparing different data and product test results under complex background noise in this embodiment.

[0044] Figure 7 This is a schematic diagram comparing the detection results of the VBD algorithm and the deep learning algorithm in this embodiment. Detailed Implementation

[0045] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to the following embodiments.

[0046] A method for detecting light-lured fishing vessels based on nighttime remote sensing data using an improved YOLOv5 network is proposed. This method verifies the light-lured fishing vessels using Vessel Monitoring System (VMS) data, marks the light-lured fishing vessels on nighttime remote sensing images, and constructs a target detection dataset for fishing vessels containing various complex background noises. A multi-scale fusion algorithm based on the YOLOv5 attention mechanism is proposed, and this algorithm is used to perform target detection on nighttime remote sensing image data, specifically including the following three steps:

[0047] (1) Data structure for optically induced fishing vessel target detection;

[0048] (2) A multi-scale fusion light-induced fishing boat detection algorithm based on the attention mechanism of YOLOv5;

[0049] (3) Production method of light-induced fishing vessel position information products.

[0050] like Figure 1 As shown, step (1), the construction of the target detection dataset for light-induced fishing vessels, is necessary to achieve better target detection results. This requires constructing specialized target detection datasets for various complex background noise conditions specific to the characteristics of DNB nighttime light remote sensing images. In this embodiment, the target detection dataset construction scheme for light-induced fishing vessels mainly includes two aspects:

[0051] (11) Data preprocessing of DNB nighttime light remote sensing images

[0052] First, all DNB images are preprocessed. The DNB radiance values ​​are multiplied by one billion (10^9) and the logarithm is taken. This step enhances feature contrast and reduces the complexity of subsequent model calculations, improving computational efficiency. Second, the VMS information of the light-lured fishing vessels is cleaned and filtered to obtain the VMS records (time, longitude, latitude) most recently captured by the DNB sensor (within 1 hour). The original 3072×4064 pixel DNB images are cropped to obtain 256×256 pixel slices containing the light-lured fishing vessel targets. These vessel slices will be converted into grayscale images (8-bit depth) for subsequent vessel location marking.

[0053] (12) Annotation of images in the dataset

[0054] Using the latitude and longitude information provided by VMS, the LabelImg tool was used to label the light-luring fishing boat targets on the grayscale images. Taking into account factors such as the safe operating distance of the boats and the spatial resolution of the image, 5×5 rectangular boxes were selected for labeling. This dataset is similar to the PASCAL VOC dataset. Each slice of light-luring fishing boat has a corresponding XML file, which labels the boat's location, the type of object contained in the slice, and the size of the boat slice. This dataset annotation format can be used for object detection models such as SSD and Faster R-CNN. The dataset annotation format used by the YOLO series algorithms differs from the XML format mentioned above. After annotation, the YOLO series algorithms generate TXT files storing information such as the target category ID, the target's center point x-coordinate (horizontal) / total image width, the target's center y-coordinate (vertical) / total image height, the width of the target bounding box / total image width, and the height of the target bounding box / total image height. To ensure that the constructed dataset can be applied to YOLO series algorithm models, the dataset's labeled documents underwent conversion processing.

[0055] like Figure 2 As shown, step (2) is a multi-scale fusion light-induced fishing boat detection algorithm based on the attention mechanism of YOLOv5. This algorithm addresses the limitation that the convolution of the YOLOv5 network structure causes the loss of key image information in the deep feature map, resulting in incomplete and insufficient extraction of key image feature information. The attention mechanism module and the weighted bidirectional feature pyramid (BiFPN) are added to the YOLOv5 network structure to perform feature fusion. At the same time, a detection layer is added to improve the accuracy of small target detection in complex backgrounds.

[0056] like Figure 3As shown, step (3) of the method for producing information on the position of light-induced fishing vessels is designed based on the characteristics of large swath width of nighttime light remote sensing images and relatively small light-induced fishing vessels. This embodiment follows the idea of ​​"segmentation-detection-merging" and designs a detection method for actual nighttime light remote sensing images. First, the DNB, h5 format image (3072×4064) is preprocessed as input to the detection model. It is cropped according to the regular grid (each grid is 256 rows × 256 columns) into 12 rows × 15 columns, for a total of 180 slices. The slices are preprocessed using the same method as the constructed detection dataset. First, the DNB radiance value is multiplied by one billion (10^9) and the logarithm is taken. Then, the 180 slices are converted into grayscale images (8-bit depth). The trained target detection model is called to perform target detection. The detected image and the normalized center coordinates of each detected target and the confidence score of the detected target are output. Based on the row and column numbers of the original DNB image slices, as well as latitude and longitude information, the coordinates of the normalized center position point of each ship target are converted into latitude and longitude coordinates. Then, through masking processing, erroneous targets on land are removed to obtain the latitude and longitude data of ships in the study area.

[0057] After adding three different attention mechanism modules to YOLOv5, significant improvements were observed in both mAP and F1 score compared to the original YOLOv5 model. SENet and ECA modules showed similar improvements in model accuracy, while CBAM provided the highest improvement, increasing mAP by 9 percentage points and F1 score by 8 percentage points. These are shown in Tables 1 and 2 below.

[0058] Table 1 Comparison results of different attention modules on the test set

[0059]

[0060] Table 2 Ablation Experiment Results

[0061]

[0062] In the 8th experiment, the mAP value reached the highest at 0.949 on the test set, and the F1 score was also the highest at 0.938. This indicates that the proposed object detection method, by improving the feature pyramid structure, fusing the features of the backbone network and the neck, adding a detection head, and also adding an attention mechanism in the neck, can effectively reduce false detections of small objects, and the model achieves the best detection performance.

[0063] like Figure 4As shown, nighttime light remote sensing images under cloudless conditions on January 1, 2021, and nighttime light remote sensing images under complex cloud backgrounds on January 31, 2021, were selected respectively. The target detection algorithm of this embodiment was used to detect ship positions, and the data were compared with ship position data obtained by VBD products and VMS ship position monitoring systems during the same period.

[0064] Under cloudless conditions, after data cleaning, within one hour of satellite transit, the VMS vessel location system reported a total of 69 fishing vessels operating or drifting within the study area (e.g., Figure 4 (b) As shown, visual interpretation revealed a large number of unregistered or unlicensed fishing vessels of other nationalities within the study area that were not detected by the VMS system. Due to its limited coverage, the VBD vessel location product only detected light-induced fishing vessels within the EEZ area and near the EEZ boundary. Vessels far from the EEZ boundary were not included in the product report. The VBD vessel location product report indicated that 127 fishing vessels were operating within the study area (e.g., ...). Figure 4 (c) shows the target detection algorithm proposed in this embodiment, which displays 151 fishing vessels operating within the study area (e.g., Figure 4 (d) is shown. Figure 4 Comparison of detection results for different data and products under cloudless or thin cloud conditions (a. Original image; b. VMS data product (red annotation); c. US VBD ship position detection algorithm (blue annotation); d. Deep learning-based detection algorithm (yellow annotation)).

[0065] like Figure 5 As shown, within the detection area of ​​the VBD product, comparing the VBD vessel position detection algorithm with the target detection algorithm proposed in this embodiment, it was found that this embodiment can detect the light-induced fishing vessels detected by VBD. In addition, it also detected 9 more fishing vessels. Among them, 4 vessels are consistent with the vessel positions obtained by the VMS system. 4 vessels can be identified as having fishing operations through visual interpretation. 1 vessel cannot be verified, that is, there is no VMS record and it cannot be determined by visual interpretation. Figure 5 Comparison of detection results between VBD algorithm and deep learning algorithm (US VBD vessel position detection algorithm (blue annotation), vessel positions detected by the algorithm in this study (yellow annotation), red circles indicate vessel positions with VMS records, green circles indicate vessel positions where operation can be determined by visual interpretation, and yellow circles indicate vessel positions where there are no VMS records and it is impossible to determine whether there are fishing vessels operating in the area by visual interpretation).

[0066] like Figure 6 As shown, under complex cloud cover, through data cleaning, within one hour of satellite transit, the VMS vessel location system reported a total of 81 fishing vessels operating or drifting within the study area (e.g., Figure 6(b) As shown, due to cloud cover, it was impossible to visually determine whether there were any unregistered or unlicensed fishing vessels or vessels of other nationalities within the study area. Similarly, it was found that the VBD vessel location product, due to its limited coverage, only detected light-induced fishing vessels within the EEZ area and near the EEZ boundary. Vessels far from the EEZ boundary were not included in the product report. The VBD vessel location product reported 120 fishing vessels operating within the study area (e.g., ...). Figure 6 (c) shows the target detection algorithm proposed in this embodiment, which displays 146 fishing vessels operating in the study area (e.g., Figure 6 (d) is shown. Figure 6 Comparison of detection results for different data and products under complex background noise (a. Original image; b. VMS data product (red annotation); c. US VBD ship position detection algorithm (blue annotation); d. Deep learning-based detection algorithm (yellow annotation)).

[0067] like Figure 7 As shown, within the detection area of ​​the VBD product, comparing the VBD vessel position detection algorithm with the target detection algorithm proposed in this embodiment, it was found that this embodiment can detect the light-induced fishing vessels detected by VBD. In addition, 12 more fishing vessels were detected, of which 6 vessels were consistent with the vessel positions obtained by the VMS system, and 6 vessels could not be verified, that is, there were no VMS records and they could not be determined by visual interpretation. Figure 7 Comparison of detection results between VBD algorithm and deep learning algorithm (US VBD vessel position detection algorithm (blue annotation), vessel positions detected by the algorithm in this study (yellow annotation), red circles indicate vessel positions with VMS records, yellow circles indicate vessel positions without VMS records and where it is impossible to determine whether there are fishing vessels operating in the area by visual interpretation)

[0068] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting fishing vessels using light-induced remote sensing data at night based on an improved YOLOv5 network, characterized in that, The data from the Vessel Monitoring System (VMS) was used to verify the light-luring fishing vessels, and the light-luring fishing vessels on the nighttime light remote sensing imagery were marked to construct a target detection dataset for fishing vessels containing various complex background noises. A multi-scale fusion algorithm based on the YOLOv5 attention mechanism was proposed, and the algorithm was used to perform target detection on the nighttime light remote sensing imagery data, specifically including the following three steps: (1) Data structure for optically induced fishing vessel target detection; (2) A multi-scale fusion light-induced fishing boat detection algorithm based on YOLOv5 attention mechanism; (3) Production method of light-induced fishing vessel position information products; In order to achieve better target detection results in step (1), it is necessary to construct a special dataset for target detection in various complex background noise environments, taking into account the characteristics of DNB nighttime light remote sensing images. The specific operation steps are as follows: (11) Data preprocessing of DNB nighttime light remote sensing images; (12) Image annotation in the dataset; The specific steps for data preprocessing of the DNB nighttime light remote sensing image in step (11) are as follows: (111) First, all DNB images are preprocessed; (112) Multiply the DNB radiance value by one billion and take the logarithm. This step is to enhance the contrast of the features and reduce the complexity of subsequent model calculations, thereby improving computational efficiency. (113) Secondly, the VMS information of the light-induced fishing boat is cleaned and filtered to obtain the VMS record closest to the time of the DNB sensor capture. (114) The original DNB image of 3072×4064 pixels is cropped to obtain a 256×256 pixel slice containing the target of the light-induced fishing boat. In order to mark the position of the boat later, these boat slices will be converted into grayscale images. Step (113) obtains VMS records within the last hour of the DNB sensor's capture time, including time, longitude, and latitude; The specific steps for annotating the dataset images in step (12) are as follows: (121) Using the latitude and longitude information provided by VMS information, the LabelImg tool is used to label the light-induced fishing boat targets on the grayscale image; (122) Taking into account factors such as safe operating distance of vessels and spatial resolution of images, 5×5 rectangular boxes were selected for marking. This dataset is similar to the PASCAL VOC dataset. (123) Each light-attracting fishing vessel slice has a corresponding XML file, which marks the vessel's location, the type of objects contained in the slice, and the size of the vessel slice; (124) The annotation format of this dataset can be used for SSD and Faster R-CNN object detection models; (125) The dataset annotation format used by the YOLO series algorithms is different from the XML format mentioned above. After the annotation of this series of algorithms is completed, a TXT file is generated to store the target category ID number, the x-coordinate of the target's center point / the total width of the image, the y-coordinate of the target's center / the total height of the image, the width of the target box / the total width of the image, and the height of the target box / the total height of the image. (126) In order to make the constructed dataset applicable to the YOLO series of algorithm models, the labeled documents of the dataset were transformed; Step (2) addresses the limitations of the YOLOv5 network structure, which suffers from the loss of key image information in deep feature maps due to convolution, resulting in incomplete and insufficient extraction of key image features. It adds an attention mechanism module and a weighted bidirectional feature pyramid to the YOLOv5 network structure for feature fusion and adds a detection layer to improve the accuracy of small target detection in complex backgrounds. In step (3), considering the characteristics of large swath width of nighttime light remote sensing images and relatively small light-induced fishing boats, a detection method for actual nighttime light remote sensing images was designed according to the idea of ​​"segmentation-detection-merging". The specific steps for producing the light-induced fishing vessel position information product in step (3) are as follows: (31) First, the DNB, h5 format image is preprocessed to input the detection model; (32) Preprocess the slices using the same method as the constructed detection dataset; (33) First, multiply the DNB radiance value by one billion and take the logarithm; (34) The 180 slices were then converted into grayscale images; (45) Call the trained object detection model to perform object detection; (36) Output the detected image and the normalized center coordinates of each detected target and the confidence score of the detected target; (37) Based on the row and column numbers of the original DNB image slices and the latitude and longitude information, convert the coordinates of the normalized center position point of each ship target into latitude and longitude coordinates; (38) Subsequently, through masking, erroneous targets on land were removed, and the latitude and longitude data of ships in the study area were obtained.

2. The method for detecting fishing vessels using nighttime light remote sensing data based on an improved YOLOv5 network according to claim 1, characterized in that, In step (31), each grid cell is cut into 256 rows × 256 columns according to the rule grid, and then cut into 12 rows × 15 columns, for a total of 180 slices.

3. The method for detecting fishing vessels using nighttime light remote sensing data based on an improved YOLOv5 network according to claim 1, characterized in that, The grayscale image in step (34) has an 8-bit depth.

Images