Night fishing boat detection method based on artificial intelligence

Abstract

The present application relates to a kind of night light image light fishing boat detection method based on artificial intelligence, comprising the following steps: the single-band night light image obtained is preprocessed, and the annotation of light fishing boat is carried out;The single-band night light image after pre-processing is handled using two different stretching methods, and the image after stretching processing is combined with the image before stretching, to form three-band image;Three-band image is normalized and mapped to preset space, to generate N*N size slice image;Slice image is divided into training set, verification set and test set, to obtain light fishing boat sample set;On the basis of YOLO-V5 network model, target recognition model is constructed, and light fishing boat sample set is used to train target recognition model, to obtain light fishing boat recognition model;Single-band night light image to be identified is input into light fishing boat recognition model, to obtain the identification result of night light fishing boat.The present application can accurately identify the light fishing boat of night operation.

Description

Technical Field

[0001] The present invention relates to the technical field of light fishing vessel recognition applications, and in particular to an artificial intelligence-based nighttime light image light fishing vessel detection method. Background Art

[0002] The rapid development of nighttime light remote sensing provides effective technical support for real-time spatial location monitoring of light-emitting fishing vessels, estimation of fishing intensity, and illegal fishing management, significantly contributing to the sustainable development of fishery resources. Nighttime light remote sensing not only offers advantages such as wide detection range, high efficiency, and low cost, but also provides a direct view of vessel distribution at night.

[0003] Currently, the detection of fishing boat targets based on nighttime light images mainly relies on the threshold method, but its robustness and transferability are poor, and the detection performance drops sharply in complex scenes and with big data. Currently, deep learning methods in the field of target detection are mainly divided into two categories. One category mainly includes two-stage algorithms such as Fast R-CNN, Faster RCNN, Mask R-CNN, SPP-net, and FPN. The other category mainly includes the YOLO series. Figure 1 As shown in the figure, the ship targets in remote sensing images are small and easily affected by the surrounding environment, and night light images are generally single-band images. Therefore, it is difficult for the existing target detection network to extract target features, resulting in low recognition accuracy. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a night light image light fishing vessel detection method based on artificial intelligence to improve the recognition accuracy of light fishing vessels.

[0005] The technical solution adopted by the present invention to solve the technical problem is to provide a night light image light fishing boat detection method based on artificial intelligence, comprising the following steps:

[0006] (1) Preprocess the acquired single-band nighttime light images and mark the fishing vessels with lights;

[0007] (2) The pre-processed single-band night light image is processed using two different stretching methods, and the stretched image is combined with the unstretched image to form a three-band image;

[0008] (3) The combined image is normalized and mapped to a preset space, and an N*N size slice image is generated;

[0009] (4) Divide the obtained slice images into a training set, a validation set, and a test set to obtain a light fishing vessel sample set;

[0010] (5) constructing a target recognition model based on a YOLO-V5 network model, and training the target recognition model using the light fishing boat sample set to obtain a light fishing boat recognition model;

[0011] (6) processing the to-be-identified single-band night light image in the manner of steps (2)-(3), and inputting the processed to-be-identified single-band night light image into the light fishing boat recognition model to obtain a recognition result of the night light fishing boat.

[0012] The preprocessing in step (1) refers to radiation correction of the single-band night light image, and batch removal of abnormal values and negative values in the image.

[0013] The two different stretching manners in step (2) are 2% linear stretching and square root stretching, and the image after stretching is represented as: g(x, y) = [(d-c) / (b-a)]*(f(x, y)+c), wherein g(x, y) represents the pixel value of the image after stretching, f(x, y) is the pixel value of the preprocessed single-band night light image, d and c are the maximum value and the minimum value of the output image respectively; when the stretching manner is 2% linear stretching, b and a represent the pixel value of the single-band night light image with cumulative frequency of 98% and 2% respectively, when the stretching manner is square root stretching, b and a represent the square root of the maximum pixel value and the square root of the minimum pixel value of the single-band night light image respectively.

[0014] Step (3) is specifically: using normalization to the range of [0, 1] to make the pixel values of the image before and after processing form a one-to-one mapping relationship, then multiplying by 255 and storing using 8 bits to make the image map to [0, 255], setting the step size for images of different widths, and cutting to generate N*N slice images; wherein g(x, y)'' represents the pixel value of the normalized three-band image, g(x, y)' is the pixel value of the three-band image, min and max represent the minimum pixel value and the maximum pixel value in the three-band image respectively. 8 -1], setting the step size for images of different widths, and cutting to generate N*N slice images; wherein g(x, y)'' represents the pixel value of the normalized three-band image, g(x, y)' is the pixel value of the three-band image, min and max represent the minimum pixel value and the maximum pixel value in the three-band image respectively.

[0015] The target recognition model in the step (5) comprises: an input layer, which is used for Mosaic data enhancement, adaptive picture zooming and anchor point coordinate calculation on an input picture, and adds a detection head for detecting small weak targets; a backbone network, which aggregates different fine-grained images and forms an image feature map; a neck module, which is used for up-sampling the image feature map to obtain feature maps of different sizes, and merging and fusing the feature maps of different sizes with feature maps of different layers in the backbone network to obtain multi-scale feature maps; and a prediction layer, which is used for generating a detection frame, detecting targets by using the multi-scale feature maps, and detecting small targets by large images and detecting large targets by small images.

[0016] The backbone network down-samples the image by a focal module, and extracts features in the image by a CSP module and an SPP module.

[0017] The neck module adopts an FPN+PAN structure, wherein the up-sampling process is completed by connecting 3 groups of bottleneck CSP modules, a Conv module with a size of 1 and a step of 1, an up-sampling module and a concatenation module.

[0018] The prediction layer obtains four feature maps of different sizes by a Conv2d convolution operation on the multi-scale feature maps, wherein the largest feature map is used for detecting small targets.

[0019] The target recognition model in the step (5) is trained by using the light fishing boat sample set, and the training specifically comprises:

[0020] Converting the annotation information in the light fishing boat sample set into a txt text format;

[0021] Inputting a training set in the light fishing boat sample set into the target recognition model for training;

[0022] Inputting a verification set in the light fishing boat sample set into the trained target recognition model for verification, and when the verification result meets the requirements, the trained target recognition model is used as a light fishing boat recognition model.

[0023] Advantages

[0024] Compared with the prior art, the present application has the following advantages and positive effects: the present application uses a stretching method to form a multi-band image from a single-band image with single night light image information, effectively enhances the extraction of feature information, and improves the precision of the model. The present application takes the light fishing boat in the night light image as the sample data of the model, and introduces a small target detection layer into the detection layer of the model, which can more accurately identify the position information of the light fishing boat. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 This is a schematic diagram of a fishing boat target in a night light image;

[0026] Figure 2 A flowchart of an embodiment of the present invention;

[0027] Figure 3 2 is a schematic diagram of the network structure of the target recognition model in the embodiment;

[0028] Figure 4 This is an example of performing three-band composite imaging after stretching in the embodiment;

[0029] Figure 5 It is a test effect diagram of the embodiment. DETAILED DESCRIPTION

[0030] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0031] The embodiment of the present invention relates to a method for detecting fishing boats with nighttime light images based on artificial intelligence, such as Figure 2 As shown, the method includes the following steps: pre-processing the acquired single-band night light image and marking the light fishing vessel; processing the pre-processed single-band night light image using two different stretching methods, and combining the stretched image with the unstretched image to form a three-band image; normalizing the three-band image and mapping it to a preset space, and generating N*N sized slice images; dividing the obtained slice images into a training set, a validation set, and a test set to obtain a light fishing vessel sample set; constructing a target recognition model based on the YOLO-V5 network model, and using the light fishing vessel sample set to train the target recognition model to obtain a light fishing vessel recognition model; using the above-mentioned combination step and slicing step to process the single-band night light image to be identified, and inputting the processed single-band night light image to be identified into the light fishing vessel recognition model to obtain the recognition result of the night light fishing vessel. This embodiment can accurately identify light fishing vessels operating at night.

[0032] The present invention is further described below through a specific embodiment.

[0033] 1. Data Source and Preprocessing

[0034] The LJ1-01 imagery of the Chinese coastal waters was acquired from the Hubei Data and Application Center of the High-Resolution Earth Observation System. 104 nighttime images containing illuminated fishing vessels were captured. The acquired images were batch preprocessed, first by radiometric calibration, then by removing outliers. Finally, the images were annotated using vessel position data combined with visual interpretation to identify illuminated fishing vessels.

[0035] 2. Image stretching and synthesis

[0036] The pre-processed image was subjected to 2% linear stretching and square root stretching respectively, and the unstretched image and the two stretched images were combined into a three-band image. The image was then normalized and stored in 8 bits. The results before and after stretching and the synthesized three-band image are shown in Figure 2. Figure 4 As shown, Figure 4 In the image, RAW is the original image, Liner is the image after 2% linear stretching, Square is the image after square root stretching, and Stacked is the three-band image.

[0037] The image after stretching is expressed as: g(x,y)=[(dc) / (ba)]*(f(x,y)+c), where g(x,y) represents the pixel value of the stretched image, f(x,y) is the pixel value of the preprocessed single-band night light image, d and c are the maximum and minimum values ​​of the output image, respectively. When the stretching method is 2% linear stretching, b and a represent the pixel values ​​with a cumulative frequency of 98% and 2% of the single-band night light image, respectively. When the stretching method is square root stretching, b and a represent the square root of the maximum pixel value and the square root of the minimum pixel value of the single-band night light image, respectively.

[0038] After the image is normalized, 8-bit storage is adopted. Specifically, the pixel values ​​of the three-band image are processed using Normalize to the range of [0,1] so that the pixel values ​​of the image before and after processing form a one-to-one mapping relationship, then multiply by 255 and store in 8 bits to map the image to [0,2 8 -1]; where g(x,y)″ represents the pixel value of the normalized three-band image, g(x,y)′ represents the pixel value of the three-band image, and min and max represent the minimum and maximum pixel values ​​in the three-band image, respectively.

[0039] 3. Sample Data

[0040] The synthesized three-band image was cut into 128x128 slices. The step size was set to 64, and a total of 1364 images of 1281 fishing vessels were acquired. The generated XML was converted into a txt file containing the labeled categories and corresponding location coordinates. The experimental data was divided into training, validation, and test sets in an 8:1:1 ratio, forming a standard Coco128 format dataset.

[0041] 4. Target Recognition Model

[0042] In this embodiment, it is proposed to add a small target detection layer on the basis of the YOLO-V5 network model to perform detection after splicing the shallow feature map with the deep feature map. The improved model structure is as follows Figure 3 As shown in the figure, the model is mainly composed of four parts: input layer (Input), backbone network (Backbone), neck module (Neck), and prediction layer (Prediction).

[0043] Input layer: It is used to perform Mosaic data enhancement, adaptive image scaling, and anchor coordinate calculation on the input 128*128*3 sample data set. It adds a detection head Anchor1 to detect small targets.

[0044] Backbone network: gathers different fine-grained images and forms an image feature map, downsamples the image through the focus module, and uses the CSP module and SPP module to extract features in the image. In this embodiment, the focus module of the backbone network slices the image and splices it in the channel dimension. For a 3-channel input image, a value is taken for every other pixel in the w and h dimensions, thereby constructing 12 feature maps. These 12 feature maps are reduced to half of the original in width and height, but the channel dimension is expanded by 4 times. At the same time, these 12 feature maps contain all the information of the input image, so the focus module not only achieves 2x downsampling while reducing information loss, but also reduces the number of parameters (params) and computational complexity (FLOPs), reduces the consumption of CUDA video memory, and thus improves the speed of forward and backward transmission.

[0045] Neck module: used for upsampling the image feature map to obtain feature maps of different sizes, and merging and fusing the feature maps of different sizes with the feature maps of different layers in the backbone network to obtain multi-scale feature maps. The neck module of the embodiment adopts the structure of FPN+PAN, generates a feature pyramid based on PANet, and performs upsampling processing on the feature map to continue to expand the feature map, and at the same time, the feature maps of sizes 20*20, 40*40, 80*80 and 160*160 obtained are respectively merged and fused with the feature maps obtained by the SPPF and CBS, CSP1_3, CSP1_2, CSP1_1 of the backbone network, so as to obtain larger feature maps and input them to the prediction layer.

[0046] The BottleneckCSP module in PANet mainly performs convolution on one of the paths to reduce the number of channels of the feature map by half, extracts features, doubles the number of channels, and adds the output feature map of the first path through shortcut residual connection to realize feature fusion. BottleneckCSP1 and BottleneckCSP2 are adopted to ensure accuracy while improving network speed. The upsampling process is completed by 3 groups of BottleneckCSP2, Conv with a size of 1 and a step of 1, Upsample and Concat connection. The obtained feature map is further processed by downsampling operation to obtain four feature maps of different sizes.

[0047] Prediction layer: used for generating a detection frame, detecting targets by using multi-scale feature maps, and detecting small targets by large images and detecting large targets by small images. In the embodiment, the prediction layer uses GIOU_Loss as the loss function, and the feature maps of sizes 20*20, 40*40 and 80*80 are respectively used to detect targets with sizes of 32*32, 16*16 and 8*8 or above. The added detection head Anchor1 is used to detect small targets of 4*4. The above prediction results are suppressed by non-maximum suppression to solve the problem of multiple candidate frames for one target. The anchor frame is applied to the multi-scale feature mapping of the neck module to indicate the category, coordinates and confidence.

[0048] In the embodiment, the three different scale feature maps of the neck module are finally obtained by Conv2d convolution operation to obtain four feature maps of sizes 160*160*255, 80*80*255 160*160*255, 40*40*255 and 20*20*255. Among them, 160*160*255 corresponds to the newly added small target detection layer for detecting weak small targets.

[0049] The target recognition model is trained using the training set data, and the batch size is set to 16, the number of iterations is set to 500, and the size of the input image is set to 128*128. Other parameters are the same as the default values, and the experimental environment is shown in Table 1.

[0050] Table 1

[0051] Item Value CPU Intel(R) Xeon Silver 4210R CPU RAM 32G GPU NVIDIA Tesla V100 Operating System Ubuntu 18.04 Cuda Cuda 10.2 with Cudann Data Processing Python 3.8 Deep Learning Framework Pytorch 1.9.0

[0052] During the verification process of the verification set data, if the verification is passed, the trained model is obtained, and at this time, the test set data is input to obtain the position information of the light fishing boat. The test parameters are set as follows: the batch size is 1, the input image is 128*128, the confidence threshold of the prediction box is 0.35, and the NMS is 0.1. Other parameters are the same as the default values. Figure 5 The test result is shown in the following table.

[0053] In the experiment, the precision, recall, mean average precision (mAP@0.5), and F1 value are used to evaluate the performance of the model. The precision is used to evaluate the prediction results of the model, which represents the number of true positive samples (TP) among the predicted positive samples. The recall is used to measure the number of positive samples that are classified as positive samples, and the mAP@0.5 is the intersection over union (IoU) of the predicted and true bounding boxes. When the IoU is set to 0.5, the average precision of all images for each class is calculated, and then the average of all classes is calculated. Since P and R are a pair of contradictory values, the F1 value is introduced to reconcile the two, which can have both precision and recall. The maximum value of F1 is 1, and the minimum value is 0. It can be said that the larger the F1 value, the better the overall precision.

[0054] The results of different models are compared in the embodiment, and different versions of the original YOLO-V5 model YOLO-V5x V5.0, YOLO-V5s V5.0 and YOLO-V5s V6.0 are compared respectively. On this basis, YOLO-V5s V6.0 is selected as the basic architecture to detect night light images in different processing modes, including Raw_YOLO-V5s V6.0 without preprocessing, Preprocessed_YOLO-V5s V6.0 after preprocessing, 2%liner_YOLO-V5s V6.0, squ_YOLO-V5s V6.0 and Stacked_YOLO-V5s V6.0 after two stretching and three band synthesis. It is found that the synthesized three-band image can significantly improve various evaluation indexes. Based on this, a small target recognition layer is added to improve the recognition accuracy of weak and small targets. The model detection results are shown in Table 2.

[0055] Table 2 Comparison of detection results of different models

[0056] Methods Precision Recall Map@0.5 F1 YOLO-V5x V5.0 0.79 0.736 0.730 0.762 YOLO-V5s V5.0 0.82 0.727 0.722 0.770 YOLO-V5s V6.0 0.843 0.85 0.854 0.846 Raw_YOLO-V5s V6.0 0.843 0.85 0.854 0.846 Preprocessed_YOLO-V5s V6.0 0.936 0.906 0.924 0.920 2% liner_YOLO-V5s V6.0 0.959 0.907 0.938 0.932 squ_YOLO-V5s V6.0 0.947 0.9 0.937 0.923 Stacked_YOLO-V5s V6.0 0.953 0.926 0.947 0.939 Stacked_YOLO-V5_Tiny V6.0 0.96 0.896 0.938 0.927

[0057] As shown in Table 2, the overall effect is significantly improved after normalization and other preprocessing than directly converting to 8-bit data, wherein the P value is increased by nearly 10%, the R value is increased by 5%, and the Map@0.5 value is increased by 7%, which indicates that it is particularly important to perform appropriate preprocessing on LJ1-01 remote sensing data. Linear stretching and square root stretching of the preprocessed single-band image can effectively improve the precision, recall rate and Map@0.5 value, which shows that stretching the single-band image can effectively highlight the target and identify the target.

[0058] The present application proposes to stack three bands to significantly improve the values of various indicators, which is better than the results of the original model and the results after stretching. The reason mainly lies in that the addition of multiple bands can increase the features and enhance the feature extraction capability. Since the recognition error of the original model mainly occurs in the area with small targets, the present application introduces a small target layer to enhance the extraction capability of small targets by increasing 160x160 feature maps. However, the increase of the small target layer also leads to the decrease of the recall rate and the F1 value, which is because the addition of the small target layer improves the model detection accuracy, but also misrecognizes part of the non-fishing boat targets as fishing boats, thereby leading to a large decrease of the recall rate and the F1 value.

Claims

1. A nighttime light image fishing boat detection method based on artificial intelligence, characterized in that: The following steps are involved: (1) Preprocess the acquired single-band nighttime light images and mark the fishing vessels with lights; (2) The pre-processed single-band night light image is processed using two different stretching methods, and the stretched image is combined with the unstretched image to form a three-band image; (3) Normalize the three-band image and map it to the preset space, and generate an N*N size slice image; (4) Divide the obtained slice images into a training set, a validation set, and a test set to obtain a light fishing vessel sample set; (5) constructing a target recognition model based on the YOLO-V5 network model, and using the light fishing boat sample set to train the target recognition model to obtain a light fishing boat recognition model; (6) The single-band night light image to be identified is processed in the manner of steps (2) to (3), and the processed single-band night light image to be identified is input into the light fishing vessel recognition model to obtain the recognition result of the night light fishing vessel.

2. The artificial intelligence-based nighttime light image fishing boat detection method according to claim 1 is characterized in that: The preprocessing in step (1) refers to performing radiation correction on the single-band night light image and removing abnormal values ​​and negative values ​​in the image in batches.

3. The method for detecting fishing boats by nighttime light images based on artificial intelligence according to claim 1 is characterized in that: The two different stretching methods in step (2) are a 2% linear stretching method and a square root stretching method, respectively. The image after stretching is expressed as: g(x,y)=[(dc) / (ba)]*(f(x,y)+c), wherein g(x,y) represents the pixel value of the image after stretching, f(x,y) is the pixel value of the pre-processed single-band night light image, and d and c are the maximum and minimum values ​​of the output image, respectively. When the stretching method is the 2% linear stretching method, b and a represent the pixel value with a cumulative frequency of 98% and 2% of the single-band night light image, respectively. When the stretching method is the square root stretching method, b and a represent the square root of the maximum pixel value and the square root of the minimum pixel value of the single-band night light image, respectively.

4. The artificial intelligence-based nighttime light image fishing boat detection method according to claim 1 is characterized in that: The step (3) is specifically as follows: the pixel values ​​of the three-band image are used Normalize to the range of [0,1] so that the pixel values ​​of the image before and after processing form a one-to-one mapping relationship, then multiply by 255 and store in 8 bits to map the image to [0,2 8 -1], set the step size for images of different widths, and perform cropping to generate N*N slice images; where g(x,y)″ represents the pixel value of the normalized three-band image, g(x,y)′ represents the pixel value of the three-band image, and min and max represent the minimum and maximum pixel values ​​in the three-band image, respectively.

5. The artificial intelligence-based nighttime light image fishing boat detection method according to claim 1 is characterized in that: The target recognition model in step (5) includes: an input layer, which is used to perform mosaic data enhancement, adaptive image scaling, and anchor coordinate calculation on the input image, and add a detection head for detecting small and weak targets; a backbone network, which aggregates different fine-grained images and forms an image feature map; a neck module, which is used to upsample the image feature map to obtain feature maps of different sizes, and merge the feature maps of different sizes with the feature maps of different layers in the backbone network to obtain a multi-scale feature map; a prediction layer, which is used to generate a detection frame, use the multi-scale feature map to detect the target, detect small targets through large images, and detect large targets through small images.

6. The artificial intelligence-based nighttime light image fishing boat detection method according to claim 5 is characterized in that: The backbone network downsamples the image through the focus module and extracts features from the image using the CSP module and the SPP module.

7. The method for detecting fishing boats by nighttime light images based on artificial intelligence according to claim 5 is characterized in that: The neck module adopts the FPN+PAN structure, in which the upsampling process is completed by connecting three groups of BottleneckCSP modules, a Conv module with a size of 1 and a step size of 1, an Upsample module and a Concat module.

8. The artificial intelligence-based nighttime light image fishing boat detection method according to claim 5 is characterized in that: The prediction layer performs a Conv2d convolution operation on the multi-scale feature map to obtain feature maps of four sizes, where the largest feature map is used to detect small objects.

9. The method for detecting fishing boats by nighttime light images based on artificial intelligence according to claim 5, characterized in that: The step (5) of using the light fishing boat sample set to train the target recognition model is specifically as follows: Convert the annotation information in the light fishing vessel sample set into txt text format; Input the training set in the light fishing vessel sample set into the target recognition model for training; The verification set in the light fishing vessel sample set is input into the trained target recognition model for verification. When the verification result meets the requirements, the trained target recognition model is used as the light fishing vessel recognition model.

Images