A water surface floating object identification method under a complex scene

Abstract

The application relates to the technical field of water surface floating object processing, in particular to a water surface floating object identification method in a complex scene. The specific steps are as follows: network model structure design: the Yolov7 algorithm is a target detection algorithm proposed in 2022, and test results on a COCO data set show that the algorithm is superior to YOLOR, YOLOX, Scaled-YOLOv4 and YOLOv5 in terms of speed and accuracy. The detection problem of the water surface floating object is further analyzed, and a target detection algorithm Yolov7-FC fusing an attention mechanism is proposed, the network structure of the algorithm comprises a Backbone layer, an Attention layer, an FPN layer and a Head layer; the network optimization method comprises a loss function and image preprocessing; the application provides a water surface floating object identification algorithm fusing an attention mechanism, the water surface floating objects in different scenes are subjected to identification test, the identification accuracy and recall rate are improved, the test is carried out by using an offline small mobile platform, and the water surface floating object identification method in the complex scene meets the demand of a shipborne platform for realizing real-time detection.

Description

Technical Field

[0001] This invention relates to the field of water surface floating object processing technology, specifically a method for identifying water surface floating objects in complex scenarios. Background Technology

[0002] Marine debris pollution has become a global environmental problem, seriously threatening the balance of the ecosystem, both in inland rivers and in the ocean. Most of this debris is transported to the ocean via rivers flowing through major cities. During this process, large plastics are broken down into microplastics, further exacerbating the impact on aquatic ecosystems. Cleaning up as much plastic waste as possible can mitigate its impact on marine ecosystems.

[0003] Many coastal countries have increasingly emphasized and undertaken some degree of cleanup efforts to address marine debris, but manual cleaning remains the primary method. Given the large scale and quantity of marine debris, the low efficiency and high cost of manual cleaning remain significant issues. Therefore, an automated cleaning solution integrating multiple sensors has been designed for debris removal. This solution first analyzes remote sensing imagery to determine the distribution range of floating debris on the water surface, thus identifying different levels of pollution. Then, drones are used to search for polluted areas and capture targets that are difficult to collect manually. For areas with a large number of floating debris, unmanned surface vessels (USVs) equipped with cleaning robots can be used for autonomous cleaning. For underwater debris, unmanned underwater vehicles (UUVs) can be used for collection. The high efficiency of these unmanned devices is inseparable from the rapidly developing deep learning technology in recent years. This technology has now made significant progress in the field of marine environmental perception, and the intelligence of unmanned robots is becoming increasingly sophisticated. Using unmanned equipment to replace manual labor in marine debris cleanup has become a trend.

[0004] However, whether it's drones, unmanned surface vessels, or unmanned underwater vehicles, the prerequisite for target cleaning operations is accurate identification of the target. This requires a reliable automatic identification system to detect the target. Deep learning-based target recognition technology, which has been widely used in many fields, can provide several feasible and reliable solutions to achieve accurate and real-time detection of the target.

[0005] Detecting small targets is more difficult than detecting large surface targets. These small floating objects also come in different types, such as wood blocks, aquatic plants, plastic bottles, and plastic bags. Among these, only plastic products are floating debris targets that unmanned surface vessels need to retrieve.

[0006] This paper focuses on floating objects on the water surface. Small floating objects, such as plastic bottles, are easily blended into the background due to their small size and transparent color, making them almost indistinguishable from other objects by radar. However, they can be captured by visual sensors. Therefore, a target detection algorithm based on visible light sensors combined with visual information is needed to identify small floating targets. The algorithm must not only achieve high accuracy but also high operating speed to enable autonomous cleaning operations by unmanned surface vessels (USVs).

[0007] The complex and ever-changing aquatic environment, such as light reflection on the water surface, insufficient light, and interference, has a significant impact on the recognition system, leading to a decrease in the accuracy of target recognition. Therefore, it is necessary to design a method for recognizing small floating objects on the water surface that can meet the requirements of different complex scenarios.

[0008] Furthermore, current small target recognition algorithms are all based on public datasets, such as the COCO and PASCAL VOC datasets. However, the patterns of small targets in these public datasets still differ from those of floating objects on the water surface, and many excellent small target recognition algorithms have not been optimized for water surface applications. This invention uses the FloW dataset for training and testing, which is a dataset specifically designed for floating objects on the water surface. By comparing and evaluating various algorithms, the advantages of the algorithm proposed in this invention are verified. Summary of the Invention

[0009] This invention provides a floating object recognition algorithm that integrates an attention mechanism. The algorithm has been tested for recognition of floating objects in different scenarios, improving the recognition accuracy and recall rate. The algorithm was tested using an offline small mobile platform, which meets the requirements of shipborne platforms for real-time detection in complex scenarios.

[0010] The technical solution adopted in this invention is: a method for identifying floating objects on the water surface in complex scenarios, comprising the following specific steps:

[0011] S1: Network Model Structure Design: The YOLOv7 algorithm, proposed in 2022, is an object detection algorithm. Test results on the COCO dataset show that it outperforms multiple object detectors such as YOLOR, YOLOX, Scaled-YOLOv4, and YOLOv5 in both speed and accuracy. Further analysis was conducted on the detection of floating objects on the water surface, and a YOLOv7-FC object detection algorithm incorporating an attention mechanism was proposed. Its network structure includes Backbone, Attention, FPN, and Head layers, representing the framework structure of different computational stages from shallow to deep. It also includes CBS to represent convolutional blocks, all composed of convolutional layers (Conv), batch normalization layers (BN), and activation functions (silu). Different CBS colors indicate different kernel sizes (k) and stride sizes (s). CBM also represents convolutional blocks, but unlike CBS, it uses the sigmoid activation function. The calculation methods for the two activation functions are shown in Equations 1 and 2.

[0012]

[0013]

[0014] UPS represents the upsampling layer, which uses the nearest neighbor interpolation method for calculation; ELAN represents the multi-branch stacking module, where concat represents concat connection calculation, o=i means the number of output channels equals the number of input channels, and o=i / 2 means the number of output channels equals half the number of input channels; MP / MP* represents the downsampling transition module, MaxPool represents max pooling, and * means that the CBS* block uses the o=i / 2 method for output calculation.

[0015] S2: Network optimization methods include loss functions and image preprocessing;

[0016] The loss function is specifically defined as follows: The loss function of the Yolov7-FC algorithm comprises three parts: the regression loss of the target object (Loss...). reg ), classification loss cls ) and location loss (Loss) loc As shown in equations 10-14:

[0017] Loss=λ reg Loss reg +λ cls Loss cls +λ loc Loss loc (10)

[0018]

[0019]

[0020]

[0021]

[0022] Where, λ reg , λ cls and λ loc These represent the weights of the three different categories of loss in the loss function. The Yolov7-FC network first divides each input image into K×K cells, generating M anchor boxes in each cell. After the network performs preprocessing on each anchor box, it obtains an adjusted bounding box, with a total of K×K×M. ij obj and I ij noobj This is used to determine whether the center coordinates of the target object are in the j-th anchor frame of the i-th grid. If they are, the former is equal to 1 and the latter is equal to 0; otherwise, they are not. C i Let be the confidence score of the true frame in the i-th cell. p represents the confidence level of the prediction box within the i-th cell. i (k) represents the conditional probability that the true bounding box in the i-th cell contains an object of the k-th type. This represents the conditional probability that the prediction box in the i-th cell contains an object of the k-th type.

[0023] The image preprocessing specifically involves the mosaic image enhancement method, a new data augmentation algorithm that extends the mixup method. Unlike the two-image overlay fusion method, it uses four images for cropping and stitching to form a new image. This method can better enrich the background of the target object and prevent the network's generalization ability from decreasing due to similar backgrounds in the training set. However, since the distribution of the generated training images differs significantly from that of natural images, it needs to be disabled after z iterations to allow the network to deepen its understanding of natural images.

[0024] Image data augmentation is performed using a combination of mosaic and mixup methods. The formula for determining whether to use a data augmentation method in each epoch is shown in Equation 23:

[0025]

[0026] Here, boll() represents a Boolean operation, & represents an AND operation, θ1 and θ2 have values ​​of 0.5, and z has a value of 0.7.

[0027] As a further aspect of the present invention, the network model structure design includes Backbone, Attention, FPN, and Head.

[0028] As a further aspect of the present invention: In the backbone network, Yolov7-FC first uses the ELAN module for feature extraction, and then uses the transition module for downsampling, thereby obtaining three effective feature layers for the next step of network construction.

[0029] In the ELAN module, the network divides the input features into five branches for computation: branches with 1, 1, 3, 5, and 7 convolutional blocks respectively. After concatenation, these branches are processed through a final convolutional block for output. This dense residual structure allows for the fusion of features from five different depths, and the use of skip connections in the residual blocks mitigates the vanishing gradient problem caused by increased network depth.

[0030] In the MP module, the network divides the input features into two branches for computation. The first branch is a max pooling plus convolution block, and the second branch is a convolution block with two convolution kernels and different strides. The subsequent output result is obtained by concatenating the two branches.

[0031] As a further aspect of the present invention: after the Attention network extracts features from the input image, the network will use an attention mechanism to increase attention to effective features, generate attention information in both channel and spatial dimensions through a Convolutional Attention Module (CBAM), and combine them to generate a new feature map.

[0032] For the network input feature map F, the process is divided into two parts, which generate channel attention feature maps M respectively. c Spatial attention feature map M s As shown in Equations 2-4:

[0033]

[0034]

[0035]

[0036] Where C, H, and W represent the number of channels, height, and width of the feature map, respectively.

[0037] The process of performing operations on feature map F involves first generating feature map F' using the channel attention algorithm, and then generating feature map F'' using the spatial attention algorithm, as shown in Equations 6 and 7:

[0038]

[0039]

[0040] in This indicates that corresponding elements in the same position are multiplied.

[0041] In the channel attention module, the network compresses the spatial dimension to focus on the discriminative features of the image in the channel dimension. It extracts features using both average pooling and max pooling methods. The average pooling F... c avg Obtain the overall features of the feature region and combine them with max pooling F c max The salient features of the feature regions are extracted, and then a multilayer perceptron (MLP) network with weight sharing is used for feature fusion to obtain the final channel attention feature map M. c The calculation process is shown in Equation 8:

[0042]

[0043] Where σ represents the sigmoid activation function, W0 and W1 are the weights of the MLP, and r is the dimensionality reduction coefficient used in the MLP.

[0044] In the spatial attention module, the network compresses the channel dimension to focus on the spatial orientation features of the image. It also utilizes both average pooling and max pooling methods for feature extraction. The average pooling F... s avg Obtain the overall features of the feature region and combine them with max pooling F s max The salient features of the feature regions are extracted, and then feature fusion is performed using convolutional layers to obtain the final spatial attention feature map M. s The calculation process is shown in Equation 9:

[0045]

[0046] Among them, f 7×7 This indicates that a convolution operation with a kernel size of 7×7 is being performed.

[0047] As a further aspect of this invention: after the FPN performs feature enhancement through an attention mechanism network, the feature map F” enters the feature pyramid stage for processing, where feature enhancement is performed through the Fast Spatial Pyramid Pooling Optimization (SPPFCSPC) module. Compared to the Spatial Pyramid Pooling Optimization (SPPCSPC) module, the improved SPPFCSPC module uses convolutional kernels of the same size for the convolutional modules, replacing three different scale convolutional kernels, and reuses the same convolutional module structure in a concatenated manner, making the network structure more efficient.

[0048] In the remaining part of the network structure of the Feature Pyramid (FPN), after downsampling and upsampling operations of multiple convolutional blocks, feature layers of different scales are fused together, thereby mixing shallow and deep network features and extracting better feature values.

[0049] As a further aspect of this invention: after passing through the feature pyramid module, the Head can obtain three enhanced feature maps. Each feature layer has width, height, and number of channels. The network treats the feature map as a set of each feature point and uses three prior boxes of different sizes to judge these feature points. Then, based on the judgment feedback, it adjusts the prior boxes it contains and uses non-maximum suppression to identify and detect targets of different sizes in the original image, thereby improving the overall detection capability of the neural network for multi-scale targets.

[0050] As a further aspect of this invention: Meanwhile, the exact intersection-union ratio (CIoU) loss is used in the calculation of the location loss function, instead of the binary cross-entropy loss used in regression and classification losses. This allows for a more accurate description of location information. The calculation method for the exact intersection-union ratio loss is shown in Equations 15-19:

[0051] loss CIoU =1-IoU+R CIoU (B,B gt (15)

[0052]

[0053]

[0054]

[0055]

[0056] Where IoU is the intersection-over-union ratio, the predicted bounding box B = (x, y, w, h), and the ground truth bounding box B0 = (x, y, w, h). gt =(x gt ,y gt ,w gt ,h gtThese coordinates (x, y) represent the center point's position, while their width, height, and length (w, h) represent the center point's position. CIoU (B,B gt () represents the predicted bounding box B and the ground truth bounding box B. gt The penalty items, b and b gt Representing B and B gt The center point, ρ(·) represents the Euclidean distance, and c is the diagonal distance of the smallest box that can simultaneously contain both the predicted and ground truth boxes. α is a positive trade-off parameter, and v is a parameter that measures aspect ratio consistency. This gives factors in the overlapping region of the predicted and ground truth boxes higher priority than those in the non-overlapping region during regression calculations.

[0057] The simOTA algorithm is used to determine whether a predicted bounding box is a positive or negative sample, combining the IOU loss between the predicted and ground truth bounding boxes. reg Class loss compared to predicted bounding boxes and ground truth bounding boxes cls The cost matrix Cost is shown in equation (20):

[0058]

[0059] Among them, the balance coefficient Set it to 3 to balance the difficulty of identifying the two types of loss.

[0060] As can be seen from the Cost matrix, the higher the overlap between the ground truth bounding box and the predicted bounding box, the lower the cost; the more accurate the classification, the lower the cost, thus adaptively finding the few predicted bounding boxes that best fit the ground truth bounding box.

[0061] Next, based on the Cost value, N candidate boxes with the largest IOU are selected, and an appropriate number of M positive samples are assigned to different targets to be identified, as shown in Equation 21:

[0062]

[0063] When the same candidate box matches multiple ground truth boxes, the ground truth box with the lower cost is selected as the sole matching target.

[0064] In the early stages of training, a large learning rate can help the network converge quickly, while in the later stages, a small learning rate is more conducive to the network converging to the optimal value. Therefore, an exponential decay strategy of the learning rate is used for training, and the learning rate γ is calculated as shown in Equation 22:

[0065] γ=ε τ γ0 (22)

[0066] Where γ0 represents the initial learning rate, ε is the decay rate, and τ is the number of iterations for training the network.

[0067] The beneficial effects of this invention are:

[0068] This invention proposes a water surface target recognition algorithm that integrates an attention mechanism. By compressing the spatial dimension, the algorithm focuses on the discriminative features of the image in the channel dimension, and by utilizing fast spatial pyramid pooling, the computation speed of the model is improved. Attached Figure Description

[0069] Figure 1 This is a network structure diagram of the Yolov7-FC algorithm for a method of identifying floating objects on the water surface in complex scenarios according to the present invention.

[0070] Figure 2 This is a flowchart of the CBAM attention mechanism in a method for identifying floating objects on the water surface in complex scenarios according to the present invention.

[0071] Figure 3 This is a schematic diagram of the network structure of the SPPFCSPC module before and after the improvement of the method for identifying floating objects on the water surface in complex scenarios according to the present invention.

[0072] Figure 4 This is a distribution map of targets of different sizes in the dataset of the water surface floating object identification method in complex scenarios of the present invention.

[0073] Figure 5 This is a schematic diagram illustrating the integration of the attention mechanism into the water surface floating object recognition method in a complex scenario according to the present invention.

[0074] Figure 6 This is a comparison chart of the test results of the Yolov7 algorithm under different optimizers for the method of identifying floating objects on the water surface in complex scenarios according to the present invention.

[0075] Figure 7 This is a comparison chart of test results of the Yolov7-FC algorithm under different optimizers for the method of identifying floating objects on the water surface in complex scenarios according to the present invention.

[0076] Figure 8 This is a comparison chart of the detection effects of different algorithms in a test video of a method for recognizing floating objects on the water surface in complex scenarios according to the present invention. Detailed Implementation

[0077] The present invention will be further described below.

[0078] A method for identifying floating objects on the water surface in complex scenarios includes the following specific steps:

[0079] S1: Network Model Structure Design: The YOLOv7 algorithm, proposed in 2022, is an object detection algorithm. Test results on the COCO dataset show that it outperforms various object detectors such as YOLOR, YOLOX, Scaled-YOLOv4, and YOLOv5 in both speed and accuracy. Further analysis was conducted on the problem of detecting floating objects on the water surface, and a YOLOv7-FC object detection algorithm incorporating an attention mechanism was proposed. Its network structure diagram is shown below. Figure 1 As shown.

[0080] Figure 1 The upper part is a diagram of the overall framework structure, which includes the Backbone, Attention, FPN, and Head layers, representing the different computational stages of the network structure from shallow to deep. Figure 1 The lower half shows the composition of different structural blocks. CBS represents a convolutional block, which consists of a convolutional layer (Conv), a batch normalization layer (BN), and an activation function (silu). Different colors in CBS indicate different kernel sizes (k) and stride (s) of the convolutional layer. CBM also represents a convolutional block, but unlike CBS, it uses the sigmoid activation function. The calculation methods for the two activation functions are shown in Equations 1 and 2.

[0081]

[0082]

[0083] UPS represents the upsampling layer, which uses the nearest neighbor interpolation method for calculation; ELAN represents the multi-branch stacking module, where concat represents concat connection calculation, o=i means the number of output channels equals the number of input channels, and o=i / 2 means the number of output channels equals half the number of input channels; MP / MP* represents the downsampling transition module, MaxPool represents max pooling, and * means that the CBS* block uses the o=i / 2 method for output calculation.

[0084] The network model structure design includes Backbone, Attention, FPN, and Head.

[0085] In the backbone network, Yolov7-FC first uses the ELAN module for feature extraction, and then uses the transition module for downsampling, thus obtaining three effective feature layers for the next step of network construction.

[0086] In the ELAN module, the network divides the input features into five branches for computation: branches with 1, 1, 3, 5, and 7 convolutional blocks respectively. After concatenation, these branches are processed through a final convolutional block for output. This dense residual structure allows for the fusion of features from five different depths, and the use of skip connections in the residual blocks mitigates the vanishing gradient problem caused by increased network depth.

[0087] In the MP module, the network divides the input features into two branches for computation. The first branch is a max pooling plus convolution block, and the second branch is a convolution block with two convolution kernels and different strides. The subsequent output result is obtained by concatenating the two branches.

[0088] After the backbone network extracts features from the input image, the network uses an attention mechanism to increase attention to effective features. This is achieved through a Convolutional Attention Module (CBAM), which generates attention information in both channel and spatial dimensions. These attention information are then combined to produce a new feature map. The CBAM network structure diagram is shown below. Figure 2 As shown, (a) the generation method of the channel attention feature map and the spatial attention feature map, and (b) the operation process of the input feature map F.

[0089] from Figure 2 As can be seen in (a), the network input feature map F is divided into two processes, which generate channel attention feature maps M respectively. c Spatial attention feature map M s As shown in Equations 2-4:

[0090]

[0091]

[0092]

[0093] Where C, H, and W represent the number of channels, height, and width of the feature map, respectively.

[0094] Figure 2 (b) The process of operating on feature map F is given. First, feature map F' is generated using the channel attention algorithm, and then feature map F'' is generated using the spatial attention algorithm, as shown in Equations 6 and 7:

[0095]

[0096]

[0097] in This indicates that corresponding elements in the same position are multiplied.

[0098] In the channel attention module, the network compresses the spatial dimension to focus on the discriminative features of the image in the channel dimension. It extracts features using both average pooling and max pooling methods. The average pooling F... c avg Obtain the overall features of the feature region and combine them with max pooling F c max The salient features of the feature regions are extracted, and then a multilayer perceptron (MLP) network with weight sharing is used for feature fusion to obtain the final channel attention feature map M. c The calculation process is shown in Equation 8:

[0099]

[0100] Where σ represents the sigmoid activation function, W0 and W1 are the weights of the MLP, and r is the dimensionality reduction coefficient used in the MLP.

[0101] In the spatial attention module, the network compresses the channel dimension to focus on the spatial orientation features of the image. It also utilizes both average pooling and max pooling methods for feature extraction. The average pooling F... s avg Obtain the overall features of the feature region and combine them with max pooling F s max The salient features of the feature regions are extracted, and then feature fusion is performed using convolutional layers to obtain the final spatial attention feature map M. s The calculation process is shown in Equation 9:

[0102]

[0103] Among them, f 7×7 This indicates that a convolution operation with a kernel size of 7×7 is being performed.

[0104] After feature enhancement via the attention mechanism network, the feature map F” enters the feature pyramid stage for processing, where it undergoes feature enhancement through the Fast Spatial Pyramid Pooling Optimization (SPPFCSPC) module. Compared to the Spatial Pyramid Pooling Optimization (SPPCSPC) module, the improved SPPFCSPC module uses convolutional kernels of the same size for all convolutional modules, replacing three different scale convolutional kernels, and reuses the same convolutional module structure through concatenation, resulting in higher network efficiency. The network structures before and after the SPPFCSPC module improvement are shown below. Figure 3 As shown, (a) is the module before improvement, and (b) is the module after improvement.

[0105] The remaining part of the network structure in the Feature Pyramid (FPN) is as follows: Figure 1As shown, after multiple convolutional blocks undergo downsampling and upsampling operations, feature layers of different scales are fused, thereby mixing shallow and deep network features and extracting better feature values.

[0106] After passing through the feature pyramid module, the Head layer produces three enhanced feature maps. Each feature map has width, height, and number of channels. The network treats each feature map as a set of feature points and uses three prior boxes of different sizes to judge these feature points. Then, based on the judgment feedback, it adjusts its own prior boxes and uses non-maximum suppression to identify and detect targets of different sizes in the original image, improving the neural network's overall ability to detect multi-scale targets.

[0107] S2: Network optimization methods include loss functions and image preprocessing;

[0108] The loss function is as follows: The loss function of the Yolov7-FC algorithm consists of three parts: the regression loss of the target object (Loss) reg ), classification loss cls ) and location loss (Loss) loc As shown in equations 10-14:

[0109] Loss=λ reg Loss reg +λ cls Loss cls +λ loc Loss loc (10)

[0110]

[0111]

[0112]

[0113]

[0114] Where, λ reg , λ cls and λ loc These represent the weights of the three different categories of loss in the loss function. The Yolov7-FC network first divides each input image into K×K cells, generating M anchor boxes in each cell. After the network performs preprocessing on each anchor box, it obtains an adjusted bounding box, with a total of K×K×M. ij obj and I ij noobjThis is used to determine whether the center coordinates of the target object are in the j-th anchor frame of the i-th grid. If they are, the former is equal to 1 and the latter is equal to 0; otherwise, they are not. C i Let be the confidence score of the true frame in the i-th cell. p represents the confidence level of the prediction box within the i-th cell. i (k) represents the conditional probability that the true bounding box in the i-th cell contains an object of the k-th type. This represents the conditional probability that the prediction box in the i-th cell contains an object of the k-th type.

[0115] Meanwhile, the Cross-Intersection Over Union (CIoU) loss is used in the location loss function calculation instead of the binary cross-entropy loss used in regression and classification losses, which can more accurately describe the location information. The calculation method of CIoU is shown in Equations 15-19:

[0116] loss CIoU =1-IoU+R CIoU (B,B gt (15)

[0117]

[0118]

[0119]

[0120]

[0121] Where IoU is the intersection-over-union ratio, the predicted bounding box B = (x, y, w, h), and the ground truth bounding box B0 = (x, y, w, h). gt =(x gt ,y gt ,w gt ,h gt These coordinates (x, y) represent the center point's position, while their width, height, and length (w, h) represent the center point's position. CIoU (B,B gt () represents the predicted bounding box B and the ground truth bounding box B. gt The penalty items, b and b gt Representing B and B gt The center point, ρ(·) represents the Euclidean distance, and c is the diagonal distance of the smallest box that can simultaneously contain both the predicted and ground truth boxes. α is a positive trade-off parameter, and v is a parameter that measures aspect ratio consistency. This gives factors in the overlapping region of the predicted and ground truth boxes higher priority than those in the non-overlapping region during regression calculations.

[0122] The simOTA algorithm is used to determine whether a predicted bounding box is a positive or negative sample, combining the IOU loss between the predicted and ground truth bounding boxes. reg Class loss compared to predicted bounding boxes and ground truth bounding boxes cls The cost matrix Cost is shown in equation (20):

[0123]

[0124] Among them, the balance coefficient Set it to 3 to balance the difficulty of identifying the two types of loss.

[0125] As can be seen from the Cost matrix, the higher the overlap between the ground truth bounding box and the predicted bounding box, the lower the cost; the more accurate the classification, the lower the cost, thus adaptively finding the few predicted bounding boxes that best fit the ground truth bounding box.

[0126] Next, based on the Cost value, N candidate boxes with the largest IOU are selected, and an appropriate number of M positive samples are assigned to different targets to be identified, as shown in Equation 21:

[0127]

[0128] When the same candidate box matches multiple ground truth boxes, the ground truth box with the lower cost is selected as the sole matching target.

[0129] In the early stages of training, a large learning rate can help the network converge quickly, while in the later stages, a small learning rate is more conducive to the network converging to the optimal value. Therefore, an exponential decay strategy of the learning rate is used for training, and the learning rate γ is calculated as shown in Equation 22:

[0130] γ=ε τ γ0 (22)

[0131] Where γ0 represents the initial learning rate, ε is the decay rate, and τ is the number of iterations for training the network.

[0132] The image preprocessing specifically involves the mosaic image enhancement method, a new data augmentation algorithm that extends the mixup method. Unlike the two-image overlay fusion method, it uses four images for cropping and stitching to form a new image. This method can better enrich the background of the target object and prevent the network's generalization ability from being reduced due to similar backgrounds in the training set. However, since the distribution of the generated training images differs significantly from that of natural images, it needs to be disabled after z iterations to allow the network to deepen its understanding of natural images.

[0133] Image data augmentation is performed using a combination of mosaic and mixup methods. The formula for determining whether to use a data augmentation method in each epoch is shown in Equation 23:

[0134]

[0135] Here, boll() represents a Boolean operation, & represents an AND operation, θ1 and θ2 have values ​​of 0.5, and z has a value of 0.7.

[0136] Based on the above, the following experiment was conducted:

[0137] 1. Dataset Planning: The dataset used for training and testing in this invention is the FloW dataset, which contains a total of 2000 images with a resolution of 1280×720 and 5271 floating objects. The training and testing sets are divided in a 3:2 ratio, with 1200 images for training and 800 images for testing. Small targets (pixels smaller than 32×32 pixels) account for 2996 images in the dataset, exceeding 50% of all targets. Medium targets (pixels between 96×96 pixels and 32×32 pixels) also account for 1974 images, and together with small targets, they exceed 90% of the total. Even medium targets account for less than 1% of the total number of pixels in the images, posing a challenge to the object detection algorithm. The distribution of large and small targets in the data is shown in the figure below. Figure 4 As shown.

[0138] 2. Training and Testing Results: The algorithm of this invention was implemented on the open-source neural network framework PyTorch (3.8.5). The computing workstation configuration included one GPU (GeForce RTX 3090ti), a CPU (AMD Ryzen 9 3950x16Core / 3.5GHz / 72M), and 128GB of RAM. The small mobile test platform was built based on an NVIDIA Jetson Agx Orin (275TOPS, 60W) development board.

[0139] First, by incorporating an attention enhancement algorithm into the network structure of the Yolov7 algorithm, the attention to floating targets is improved. The test results of the Yolov7 algorithm before and after the improvement are shown in Table 1. Table 1 shows the integration of the CBAM and ECA algorithms into the Yolov7 algorithm structure, and compares the applicability of the attention mechanism at shallow layer A and deep layer B positions in the neural network. The positions are shown in the table below. Figure 5 As shown. Furthermore, the MAP and MAR improvement rates in Table 1 are based on test results of the Yolov7 algorithm optimized on the SGD optimizer without the attention mechanism.

[0140] Table 1. Comparison of Yolov7 algorithm test results before and after incorporating attention enhancement methods.

[0141]

[0142] As shown in Table 1, the network architecture with the best detection performance is the one that incorporates the CBAM attention mechanism at position A and trains it using the ADAM optimizer. The next best architecture is the one that incorporates the ECA attention mechanism at position B and trains it using the SGD optimizer. The test results are compared in the following figure. Figure 6 As shown. By Figure 6 It can be seen that among several different combinations, incorporating the attention mechanism at position A in the algorithm results in a significant performance improvement, and the average test results of the network trained using the ADAM optimizer are also better, as reflected in both precision and recall metrics. However, for the original algorithm without the attention mechanism, the ADAM optimizer performs far worse than the SGD optimizer.

[0143] To further improve the algorithm's performance, the SPPFCSPC structure was used to optimize the FPN layer while incorporating the attention mechanism. The test results of the Yolov7-F algorithm before and after the improvement are shown in Table 2. Similarly, the CBAM and ECA algorithms were incorporated into the Yolov7-F algorithm structure, and the applicability of this structure to the shallow layer A position and the deep layer B position of the neural network was compared. Unlike Table 1, the MAP and MAR improvement rates in Table 2 are based on the test results of the Yolov7 algorithm optimized on the SGD optimizer in Table 1, thus verifying the advantages of the SPPFCSPC structure.

[0144] Table 2 Comparison of Yolov7-F algorithm accuracy before and after incorporating attention enhancement methods

[0145]

[0146] As shown in Table 2, after using the SPPFCSPC structure, the previous optimal combination achieved additional improvements in precision and recall. A comparison of the test results for all combinations is shown in the following figure. Figure 7 As shown. By Figure 7As can be seen, networks trained using the ADAM optimizer all showed improvement. Conversely, networks trained using the SGD optimizer, while showing improved recall, experienced varying degrees of decrease in precision. Therefore, the combination of the SPPFCSPC structure and the ADAM optimizer yielded better results, a point also evident in networks that did not incorporate the attention mechanism. Furthermore, regardless of whether the attention mechanism was incorporated at position A or B, the test results remained relatively stable. Thus, the SPPFCSPC structure did not have a decisive impact on the placement of the attention mechanism.

[0147] Table 3 compares the test results of the YOLOv7-FC algorithm with other algorithms. It can be seen that the YOLOv7-FC algorithm has certain advantages in precision and recall, compared to the Cascade R-CNN algorithm, which performs excellently on this dataset.

[17] The accuracy was still improved by 0.9%. Table 4 shows a comparison of the algorithm's running speed on small mobile devices. It can be seen that the improved algorithm has increased overall structural complexity compared to the original algorithm, resulting in a 2.8% reduction in computation speed, but it still meets the requirements for offline real-time detection on shipborne small mobile devices. The Cascade R-CNN algorithm, which showed better accuracy,...

[17] With a detection speed of only 23.8% of the Yolov7-FC algorithm, it is basically unable to meet the needs of real-time computing. For the DSSD and Yolov3 algorithms, which have faster detection speeds, their accuracy and recall are low, failing to adequately meet the requirements for floating object detection.

[0148] Table 3. Comparison of test results between the Yolov7-FC algorithm and other algorithms.

[0149]

[0150] Table 4: Speed ​​Comparison of Algorithms Running on Small Mobile Devices

[0151]

[0152]

[0153] Figure 8 The paper presents the actual test results of different algorithms, including scenarios under different weather conditions. Figure 8 (a) shows the target detection performance when interference is present. In this case, the target is relatively far away. Although all algorithms have some degree of missed detection, Yolov7, Yolov3, and YoloX algorithms have more missed detections. Furthermore, Yolov7 misidentifies the radar in the lower left corner of the bounding box as a floating object. The improved algorithm, by incorporating an attention mechanism, performs better in this target recognition task. Figure 8 In (b), not only are there interfering objects, but there are also green floating objects that are similar in color to the water surface. In this case, the Yolov3 and Yolov5 algorithms cannot accurately identify them. Figure 8 (c) In the very difficult case of glare, all algorithms performed poorly. The YOLOv7-FC and YOLOX algorithms identified a relatively large number of targets, but YOLOX misidentified the glare spot in the lower right corner. Figure 8 (d) is the case of dense multi-targets. All algorithms performed well, with YOLOv7-FC and YOLOX algorithms performing relatively better. This is likely because the weather conditions were better in the afternoon, with higher visibility and clearer floating objects. Figure 8 (e) is a dark background. In this case, the YOLOv3 algorithm misidentifies the white ship in the distance as a floating object, and the YOLOv5 algorithm also misidentifies the radar in the lower left corner as a floating object. Figure 8 (f) represents weak targets at a distance. Most algorithms cannot identify green floating objects that are close to the background at a distance. Yolov7-FC and Yolov5 perform better, but Yolov5 still misidentifies them.

[0154] Therefore, by comparing the recognition performance under various background conditions, it can be seen that glare has the most severe impact on the recognition algorithm. Good detection results can be achieved against interfering objects and in mildly dark backgrounds. While small, distant targets do affect the algorithm to some extent, they can still be recognized relatively accurately when there is high contrast with the background. However, if the target's color is similar to the background, the recognition difficulty increases significantly, reducing both precision and recall.

[0155] In the test experiment, the improved algorithm improved the recognition accuracy by 1.9% and the recall rate by 1.6%. It also showed excellent recognition performance under different backgrounds such as light spot reflection, insufficient light and the presence of interference, and met the real-time detection requirements on small mobile devices.

[0156] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying floating objects on water surface in complex scenes, characterized in that: The specific steps include: S1: network model structure design: the Yolov7 algorithm is a target detection algorithm proposed in 2022, and the test results on the COCO dataset show that its performance in speed and accuracy is better than that of YOLOR, YOLOX, Scaled-YOLOv4 and YOLOv5 multiple target detectors; further analysis is made on the detection problem of water surface floating objects, and a target detection algorithm Yolov7-FC is proposed, which contains Backbone, Attention, FPN and Head layers in its network structure, representing the framework structure of different stages of network structure from shallow to deep, and also contains CBS representing convolution block, which is composed of convolution layer Conv, batch normalization layer BN and activation function silu, and the different colors of CBS represent the different sizes of convolution kernel k and stride s; CBM also represents convolution block, and the difference between CBM and CBS is that the activation function in CBM adopts sigmoid activation function, and the calculation methods of the two activation functions are shown in formula 1 and formula 2: UPS represents the up-sampling layer, which is calculated by the nearest neighbor interpolation method nearest; ELAN represents the multi-branch stacked module, wherein concat represents the merging connection count concat, o = i represents that the output channel number is equal to the input channel number, and o = i / 2 represents that the output channel number is equal to half of the input channel number; MP / MP* represents the down-sampling transition module, MaxPool represents the maximum pooling, and * represents that the CBS* block in it adopts the o = i / 2 way to output calculation; S2: network optimization method includes loss function and image preprocessing; The loss function is specifically defined as follows: The loss function of the Yolov7-FC algorithm comprises three parts: the regression loss of the target object (Loss). reg Classification loss cls and positional loss loc As shown in equations 10-14: Loss = λ reg • Loss reg + λ cls • Loss cls + λ loc • Loss loc (10) where λ reg , λ cls and λ loc represent the weights of three different types of loss in the loss function, respectively. The Yolov7-FC network divides each input picture into KxK cells first, and M anchors are generated in each grid. After the network performs the previous calculation, the adjusted bounding box is obtained, and the total number is KxKxM; and are used to determine whether the center coordinates of the target object are in the jth anchor box in the ith grid. If yes, the former is equal to 1 and the latter is equal to 0. If not, it is the opposite; C i is the confidence of the real box in the ith cell, is the confidence of the predicted box in the ith cell; p i (k) represents the conditional probability that the real box in the ith cell contains the kth type of target, represents the conditional probability that the predicted box in the ith cell contains the kth type of target. The image preprocessing is specifically: the image data enhancement is performed in the combination of mosaic and mixup, and the judgment formula of whether to use data enhancement method in each iteration epoch is shown in formula 23: Wherein, boll() represents Boolean operation, & represents AND operation, the values of θ1 and θ2 are 0.5, and the value of z is 0.

7.

2. The method of claim 1, wherein: The network model structure design includes Backbone, Attention, FPN and Head.

3. The method of claim 2, wherein: The Backbone is in the backbone network, Yolov7-FC first uses the ELAN module for feature extraction, and then uses the transition module for down-sampling, so as to obtain three effective feature layers for the next network construction; In the ELAN module, the network divides the input features into 5 branches for calculation, which are 1 convolution block, 1 convolution block, 3 convolution blocks, 5 convolution blocks and 7 convolution blocks, respectively, and after concat calculation, 1 convolution block is used for output; through the dense residual structure, 5 features of different depths can be fused, and through the use of skip connection residual block, the influence of gradient disappearance problem caused by the increase of network depth is weakened; In the MP module, the network divides the input features into two branches for calculation, the first branch is a max-pooling plus convolution block, and the second branch is a convolution block with two different convolution kernels and strides, and the output result is obtained by concatenating the two branches.

4. The water surface floating object identification method in a complex scene according to claim 3, characterized in that: After the input picture is extracted by the backbone network, the network uses the attention mechanism to improve the attention to the effective features, generates attention information in the channel and spatial dimensions through the convolution attention module CBAM, and combines the attention information to generate a new feature map. For the network input feature map F, it is divided into upper and lower two processes to generate channel attention feature map M c and spatial attention feature map M s As shown in formulas 2-4: Wherein, C, H and W represent the channel number, height and width of the feature map respectively. The process of feature map F operation first generates feature map F' using channel attention algorithm, and then generates feature F'' using spatial attention algorithm, as shown in formula 6 and formula 7: wherein represents the same element corresponds to multiplication; In the channel attention module, the network focuses on the discriminative features of the image in the channel dimension by compressing the spatial dimension, and uses average pooling and max pooling methods for feature extraction. The average pooling F c avg The overall features of the feature region are obtained and combined with the max pooling F c max The prominent features of the feature region are obtained, and then the weight-shared multi-layer perception (MLP) network is used for feature fusion to obtain the final channel attention feature map M c The calculation process is shown in equation 8: Wherein, sigma represents the sigmoid activation function, W0 and W1 are the weights of MLP, and r is the dimension reduction coefficient used in MLP. In the spatial attention module, the network focuses on the orientation features of the image in the spatial dimension by compressing the channel dimension. Both average pooling and max pooling are used for feature extraction. The average pooling F s avg The overall features of the feature region are obtained and combined with the max pooling F s max The prominent features of the feature region are obtained, and then the convolution layer is used for feature fusion to obtain the final spatial attention feature map M s The calculation process is shown in equation 9: where f 7×7 represents performing a convolution operation with a kernel size of 7x7.

5. The method of claim 4, wherein: After the feature enhancement of the FPN through the attention mechanism network, the feature map F'' enters the feature pyramid stage for processing, and the feature is enhanced through the fast spatial pyramid pooling optimization (SPPFCSPC) module. Compared with the spatial pyramid pooling optimization (SPPCSPC) module, the improved SPPFCSPC module uses the same size of convolution kernel for the convolution module, instead of three different scale convolution kernels, and repeatedly uses the same convolution module structure in series, so that the efficiency of the network structure is higher. In the network structure of the remaining part of the feature pyramid FPN, after the down-sampling and up-sampling operations of multiple convolution blocks, the features of different scales are fused, so that the shallow and deep network features are mixed, and better feature values are extracted.

6. The water surface floating object identification method in a complex scene according to claim 5, characterized in that: After the Head passes through the feature pyramid module, three enhanced feature maps can be obtained, each feature layer has width, height and channel number, and the network regards the feature map as a set of feature points, and uses three different size prior boxes to judge these feature points. Then, according to the judgment feedback, the prior boxes contained in the network are adjusted, and the non-maximum suppression method is used to identify and detect the targets of different sizes in the original image, so as to improve the overall detection ability of the neural network for multi-scale targets.

7. The method of claim 1, wherein: At the same time, the complete intersection over union loss CIoU is used in the calculation of the position loss function, instead of the binary cross entropy loss used in the regression loss and classification loss, which can more accurately describe the position information. The calculation method of the complete intersection over union loss is shown in formula 15-19. Where IoU is the intersection-over-union ratio, the predicted bounding box B = (x, y, w, h), and the ground truth bounding box B0 = (x, y, w, h). gt =(x gt ,y gt ,w gt ,h gt These coordinates consist of x, y coordinates representing the center point's position and w, h coordinates representing its width, height, and length; R CIoU (B,B gt () represents the predicted bounding box B and the ground truth bounding box B. gt The penalty items, b and b gt Represents B and B gt The center point, ρ(·) represents the Euclidean distance, c is the diagonal distance of the smallest box that can simultaneously contain both the predicted box and the ground truth box; α is the positive trade-off parameter, and v is the parameter that measures the consistency of aspect ratio. This makes the factors in the overlapping area between the predicted box and the ground truth box have a higher priority than the non-overlapping parts when performing regression calculations. In judging whether the prediction box is a positive sample or a negative sample, the simOTA algorithm is used to make a decision, and the IOU loss loss between the prediction box and the real box is combined reg The category loss loss of the prediction box and the real box cls As the cost matrix Cost, as shown in equation (20): wherein the balance coefficient is set to 3 to balance the identification difficulty of the two losses; As can be seen from the Cost cost matrix, the higher the overlap between the real box and the predicted box, the lower the cost, and the more accurate the classification, so as to adaptively find several predicted boxes that best fit the real box; Then, according to the value of Cost, select N candidate boxes with the largest IOU, and divide a certain number M of positive samples for different to-be-identified targets, as shown in formula 21: When the same candidate box matches multiple real boxes, the real box with smaller cost is selected as the only matching target. In the early stage of training, a large learning rate can make the network converge quickly, while in the later stage of training, a small learning rate is also conducive to the network converging to the optimal value; therefore, the learning rate is trained by using the exponential decay strategy, and the calculation method of the learning rate γ is shown in formula 22: γ = ε τ γ0 (22) wherein γ0 represents the initial learning rate, ε is the decay rate, and τ is the iteration number of the trained network.

Images