Water surface floating object detection method and system based on machine learning

Abstract

The invention discloses a water surface floating object detection method and system based on machine learning, belongs to the technical field of machine learning, and aims to solve the technical problem of how to quickly and accurately detect water surface floating objects. Comprising the following steps: taking a YOLO5vs network as a basic network, introducing a down-sampling convolutional layer and CARAFE up-sampling to construct a floating object detection model, taking a water surface image as input, performing feature extraction on the water surface image to obtain four feature maps of different scales including newly added feature maps, and performing feature extraction on the feature maps to obtain a floating object detection result; performing type identification and position identification of the floating object based on the four feature maps to obtain a detection result; deploying a trained floating object detection model at an edge end, performing floating object type identification and position identification through the trained floating object detection model, and sending a water area identifier, a water surface image, a predicted detection result and a quantity corresponding to the target water area as an edge detection result to a server end; and pushing the abnormal information to related supervisors at the server side.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a method and system for detecting floating objects on a water surface based on machine learning. Background Art

[0002] Water management has always been a top priority for environmental protection. In recent years, especially in rainy weather, garbage from the upper reaches of the river often impacts the lower reaches, forming floating objects on the water surface. At the beginning of governance, it can only be achieved through a certain amount of manpower, and there will be a situation where "there is pollution cleanup, but the discovery is not timely and accurate."

[0003] As a specific application field of target detection, surface floating object detection can be mainly divided into methods based on traditional image processing and methods based on deep learning. It mainly includes two subtasks: floating object positioning and floating object classification. The goal of floating object positioning is to accurately locate the position of the target object in the image, usually represented by a rectangular box, each of which is represented by four coordinate values, namely the coordinates of the upper left corner and the lower right corner. The goal of floating object classification is to classify the detected objects to determine what categories they belong to, such as plastic bags, bottles, etc.

[0004] How to quickly and accurately detect floating objects on the water surface is a technical problem that needs to be solved. Summary of the invention

[0005] The technical task of the present invention is to address the above shortcomings and provide a method and system for detecting floating objects on the water surface based on machine learning to solve the technical problem of how to quickly and accurately detect floating objects on the water surface.

[0006] In a first aspect, the present invention provides a method for detecting floating objects on a water surface based on machine learning, which is applied to an edge terminal and a service terminal, and comprises the following steps:

[0007] A floating object detection model is constructed by using the YOLO5vs network as a basic network, introducing a downsampling convolution layer and CARAFE upsampling, and the introduced downsampling convolution layer is added to the backbone network of the original YOLO5vs network as a newly added downsampling convolution layer. The newly added downsampling convolution layer is used to perform deep feature extraction on the input image, and the output feature map is used as a newly added vital sign map. The CARAFE upsampling is used to replace the nearest neighbor interpolation upsampling of the neck in the original YOLO5vs network. The floating object detection model takes a water surface image as an input, and is used to obtain four feature maps of different scales including the newly added feature map by extracting features from the water surface image, and performs type and position recognition of floating objects based on the four feature maps to obtain a detection result, and the detection result includes a position, a type and a confidence level.

[0008] Deploy the trained floating object detection model on the edge. For the target water area, collect the water surface image of the target water area through the camera, use the water surface image as input, and use the trained floating object detection model to identify the type and location of floating objects, output the predicted detection results, and perform statistical analysis on the type and number of floating objects in the target water area based on the detection results. Send the water area identification, water surface image, predicted detection result and number corresponding to the target water area to the server as the edge detection result;

[0009] The edge detection results are received and stored on the server, abnormal information of the target waters is generated based on the edge detection results, and the abnormal information is pushed to relevant supervisors.

[0010] Preferably, the floating object detection model comprises an input layer, a backbone network, a neck and a head;

[0011] The input layer uses the water surface image as the input image, and is used to perform image preprocessing on the input image, perform Mosaic data enhancement, adaptive aiming frame calculation and adaptive image scaling through the image preprocessing, and output the preprocessed water surface image;

[0012] The backbone network includes a Focus structure, a newly added downsampling convolution layer, a CSP structure composed of a plurality of downsampling convolution layers and a plurality of C3 modules stacked, and an SPP module. The Focus structure takes the processed water surface image as an input image, is used for slicing the input image and outputting a feature map. The newly added downsampling convolution layer takes the feature map output by the Focus structure as an input image, is used for extracting features from the input image, and outputs the feature map as a P2 feature map. The CSP structure takes the feature map output by the newly added downsampling convolution layer as an input image, is used for extracting multi-scale features from the input image, and outputs feature maps of three different scales, namely, a P3 feature map, a P4 feature map, and a P5 feature map. The SPP module takes the P2 feature map, the P3 feature map, the P4 feature map, and the P5 feature map as input images, is used for downsampling the input features and performing global maximum pooling operations, and outputting a merged feature map.

[0013] The neck includes an FPN module and a PAN module, wherein the FPN module uses the feature map output by the backbone network as an input image, and is used to perform an upsampling operation on the input image through CARAFE upsampling, and performs feature fusion and outputs a predicted feature map through the upsampling operation, and the PAN module uses the feature map output by the FPN module as an input image, and is used to perform a downsampling operation on the input image, and performs feature fusion and outputs a predicted feature map through the downsampling operation;

[0014] The head includes multiple output layers, each of which is used to identify the type and position of floating objects on feature maps of different scales, and generate bounding box coordinates, type probability, and confidence of the existence of floating objects of the prediction box.

[0015] Preferably, the CARAFE upsampling is used as a lightweight upsampling operator to perform upsampling kernel prediction and feature reorganization on the input image;

[0016] Among them, the upsampling kernel prediction includes the following operations:

[0017] For an input image of size H×W×C, a channel compression operation is performed through a 1×1 convolution to reduce the number of channels of the input image to C. m ;

[0018] For the compressed input image, the size is K encoder ×K encoder The convolutional layer encodes the content and expands it in the spatial dimension to obtain a size of The upsampling kernel is normalized by the Softmax function, so that the sum of the convolution kernel weights is equal to l;

[0019] Correspondingly, feature recombination includes the following steps:

[0020] Perform a dot product operation on the central k×k sub-region of the input image and the upsampled kernel of each pixel obtained by prediction to obtain an output feature map of size σH×σW×C;

[0021] For a target position l ′ and the corresponding square area (χ l ,k up ), the recombination process formula is:

[0022]

[0023] r = k up / 2,

[0024] Among them, σ represents the upsampling ratio, χ l It means that k up represents, r represents, χ l ″ indicates that w l′(n,m) express.

[0025] Preferably, the water surface image and the predicted detection results are displayed on the edge through an interactive display screen, and the parameters of the edge device and the deployed floating object detection model are configured and adjusted. The edge device includes a processor and a camera, a display screen and a communication module for data interaction with the processor, wherein the camera and the display screen interact with the processor in a wired communication manner, and the processor interacts with the server in a wireless communication manner through the communication module.

[0026] Preferably, the wireless communication method between the edge end and the server end includes 4G and Ethernet.

[0027] In a second aspect, the present invention provides a system for detecting floating objects on a water surface based on machine learning, including an edge end and a service end;

[0028] The edge end is deployed with a camera and a trained floating object detection model. For the target water area, the edge end is used to perform the following: collect the water surface image of the target water area through the camera, use the water surface image as input, and use the trained floating object detection model to identify the type and position of floating objects, output the predicted detection results, and perform statistical analysis on the type and number of floating objects in the target water area based on the detection results, and send the water area identification, water surface image, predicted detection result and number corresponding to the target water area as the edge detection result to the server;

[0029] The server is used to receive and store edge detection results, generate abnormal information of the target water area based on the edge detection results, and push the abnormal information to relevant supervisors;

[0030] Among them, the floating object detection model is a deep learning model constructed by taking the YOLO5vs network as the basic network, introducing downsampling convolution layers and CARAFE upsampling. The introduced downsampling convolution layers are added as new downsampling convolution layers to the backbone network of the original YOLO5vs network. The new downsampling convolution layers are used to perform deep feature extraction on the input image and output feature maps as new vital sign maps. The CARAFE upsampling is used to replace the nearest neighbor interpolation upsampling of the neck in the original YOLO5vs network. The floating object detection model takes the water surface image as input, and is used to obtain feature maps of four different scales including the new feature map by extracting features from the water surface image, and performs type and position recognition of floating objects based on the four feature maps to obtain detection results, and the detection results include position, type and confidence.

[0031] Preferably, the floating object detection model comprises an input layer, a backbone network, a neck and a head;

[0032] The input layer uses the water surface image as the input image, and is used to perform image preprocessing on the input image, perform Mosaic data enhancement, adaptive aiming frame calculation and adaptive image scaling through the image preprocessing, and output the preprocessed water surface image;

[0033] The backbone network includes a Focus structure, a newly added downsampling convolution layer, a CSP structure composed of a plurality of downsampling convolution layers and a plurality of C3 modules stacked, and an SPP module. The Focus structure takes the processed water surface image as an input image, is used for slicing the input image and outputting a feature map. The newly added downsampling convolution layer takes the feature map output by the Focus structure as an input image, is used for extracting features from the input image, and outputs the feature map as a P2 feature map. The CSP structure takes the feature map output by the newly added downsampling convolution layer as an input image, is used for extracting multi-scale features from the input image, and outputs feature maps of three different scales, namely, a P3 feature map, a P4 feature map, and a P5 feature map. The SPP module takes the P2 feature map, the P3 feature map, the P4 feature map, and the P5 feature map as input images, is used for downsampling the input features and performing global maximum pooling operations, and outputting a merged feature map.

[0034] The neck includes an FPN module and a PAN module, wherein the FPN module uses the feature map output by the backbone network as an input image, and is used to perform an upsampling operation on the input image through CARAFE upsampling, and performs feature fusion and outputs a predicted feature map through the upsampling operation, and the PAN module uses the feature map output by the FPN module as an input image, and is used to perform a downsampling operation on the input image, and performs feature fusion and outputs a predicted feature map through the downsampling operation;

[0035] The head includes multiple output layers, each of which is used to identify the type and position of floating objects on feature maps of different scales, and generate bounding box coordinates, type probability, and confidence of the existence of floating objects of the prediction box.

[0036] Preferably, the CARAFE upsampling is used as a lightweight upsampling operator to perform upsampling kernel prediction and feature reorganization on the input image;

[0037] Among them, the upsampling kernel prediction includes the following operations:

[0038] For an input image of size H×W×C, a channel compression operation is performed through a 1×1 convolution to reduce the number of channels of the input image to C. m ;

[0039] For the compressed input image, the size is K encoder ×K encoder The convolutional layer encodes the content and expands it in the spatial dimension to obtain a size of The upsampling kernel is normalized by the Softmax function, so that the sum of the convolution kernel weights is equal to l;

[0040] Correspondingly, feature recombination includes the following steps:

[0041] Perform a dot product operation on the central k×k sub-region of the input image and the upsampled kernel of each pixel obtained by prediction to obtain an output feature map of size σH×σW×C;

[0042] For a target position l ′ and the corresponding square area (χ l ,k up ), the recombination process formula is:

[0043]

[0044] r = k up / 2,

[0045] Among them, σ represents the upsampling ratio, χ l It means that k up represents, r represents, χ l ″ indicates that w l′(n,m) express.

[0046] Preferably, an interactive display screen is deployed at the edge, which is used to display the water surface image and the predicted detection results, and to configure and adjust the parameters of the edge device and the deployed floating object detection model. The edge device includes a processor and a camera, a display screen and a communication module for data interaction with the processor, wherein the camera and the display screen interact with the processor in a wired communication manner, and the processor interacts with the server in a wireless communication manner through the communication module.

[0047] Preferably, the wireless communication method between the edge end and the server end includes 4G and Ethernet.

[0048] The method and system for detecting floating objects on water surface based on machine learning of the present invention have the following advantages:

[0049] 1. For the floating object detection model, based on the YOLO5vs network, a downsampling convolution layer is introduced into the backbone network of the original YOLO5vs network. The deep feature extraction is performed through the newly added downsampling convolution layer. The deep and shallow features can be extracted through the improved backbone network, which avoids the missed detection of small targets and improves the accuracy of the model for small target detection. The improved floating detection model can realize multi-scale feature extraction, and the type and position prediction of floating objects based on the multi-scale feature map can be performed, which can improve the prediction accuracy;

[0050] 2. For the floating object detection model, based on the YOLO5vs network, CARAFE upsampling is introduced to replace the nearest neighbor interpolation upsampling of the neck in the original YOLO5vs network. As a lightweight upsampling operator, CARAFE upsampling enables the floating object detection model to have a larger receptive field during sampling, and the feature map contains richer semantic information;

[0051] 3. Collect water surface images at the edge, identify the type and location of floating objects on the water surface based on the water surface images, and generate detection results. The edge pushes the detection results to the server. The server analyzes the detection results at the edge to generate abnormal information and pushes the abnormal information to relevant supervisors. Floating object detection and notification are performed through cloud-edge collaboration, reducing the computing pressure on the server. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] The present invention is further described below in conjunction with the accompanying drawings.

[0054] Figure 1 This is a flowchart of a method for detecting floating objects on a water surface based on machine learning in Example 1. DETAILED DESCRIPTION

[0055] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0056] The embodiments of the present invention provide a method and system for detecting floating objects on the water surface based on machine learning, which are used to solve the technical problem of how to quickly and accurately detect floating objects on the water surface.

[0057] Embodiment 1:

[0058] The present invention provides a method for detecting floating objects on a water surface based on machine learning, such as Figure 1 The diagram shows three steps: model building, edge computing, and server computing.

[0059] Step S100 model construction: using the YOLO5vs network as the basic network, introducing downsampling convolution layers and CARAFE upsampling to construct a floating object detection model, the introduced downsampling convolution layer is added as a new downsampling convolution layer to the backbone network of the original YOLO5vs network, the new downsampling convolution layer is used to perform deep feature extraction on the input image, and the output feature map is used as a new sign map, the CARAFE upsampling is used to replace the nearest neighbor interpolation upsampling of the neck in the original YOLO5vs network, the floating object detection model takes the water surface image as input, is used to obtain four feature maps of different scales including the new feature map by extracting features from the water surface image, and performs type recognition and position recognition of floating objects based on the four feature maps to obtain detection results, and the detection results include position, type and confidence.

[0060] As a specific implementation of the floating object detection model, the model includes an input layer, a backbone network, a neck, and a head.

[0061] The input layer uses the water surface image as the input image and is used to preprocess the input image. Mosaic data enhancement, adaptive aiming frame calculation and adaptive image scaling are performed through image preprocessing, and the preprocessed water surface image is output.

[0062] The backbone network includes a Focus structure, a newly added downsampling convolution layer, a CSP structure composed of multiple downsampling convolution layers and multiple C3 modules stacked, and an SPP module. The Focus structure uses the processed water surface image as an input image, is used to slice the input image and output a feature map. The newly added downsampling convolution layer uses the feature map output by the Focus structure as an input image, is used to extract features from the input image, and outputs the feature map as a P2 feature map. The CSP structure uses the feature map output by the newly added downsampling convolution layer as an input image, is used to extract multi-scale features from the input image, and outputs feature maps of three different scales, which are P3 feature maps, P4 feature maps, and P5 feature maps. The SPP module uses the P2 feature map, the P3 feature map, the P4 feature map, and the P5 feature map as input images, is used to downsample the input features and perform global maximum pooling operations, and output a merged feature map.

[0063] The neck includes an FPN module and a PAN module. The FPN module uses the feature map output by the backbone network as an input image, and is used to upsample the input image through CARAFE upsampling, perform feature fusion through the upsampling operation, and output a predicted feature map. The PAN module uses the feature map output by the FPN module as an input image, and is used to downsample the input image, perform feature fusion through the downsampling operation, and output a predicted feature map.

[0064] Among them, CARAFE upsampling is used as a lightweight upsampling operator to perform upsampling kernel prediction and feature reorganization on the input image.

[0065] Among them, the upsampling kernel prediction includes the following operations:

[0066] (1) For an input image of size H×W×C, a channel compression operation is performed through a 1×1 convolution to reduce the number of channels of the input image to C. m ;

[0067] (2) For the compressed input image, the size is K encoder ×K encoder The convolutional layer encodes the content and expands it in the spatial dimension to obtain a size of The upsampling kernel is normalized by the Softmax function, so that the sum of the convolution kernel weights is equal to l.

[0068] Correspondingly, feature recombination includes the following steps:

[0069] (1) Perform a dot product operation on the central k×k subregion of the input image and the upsampled kernel of each pixel obtained by prediction to obtain an output feature map of size σH×σW×C;

[0070] (2) For a target position l ′ and the corresponding square area (χ l ,k up ), the recombination process formula is:

[0071]

[0072] r = k up / 2,

[0073] Among them, σ represents the upsampling ratio, χ l It means that k up represents, r represents, χ l ″ indicates that w l′(n,m) express.

[0074] The head includes multiple output layers, each of which is used to identify the type and location of floating objects on feature maps of different scales, and generate the bounding box coordinates, type probability and confidence of the existence of floating objects of the prediction box.

[0075] Step S200 edge computing: deploy the trained floating object detection model on the edge, and for the target water area, collect the water surface image of the target water area through the camera, use the water surface image as input, and use the trained floating object detection model to identify the type and position of the floating objects, output the predicted detection results, and perform statistical analysis on the type and number of floating objects in the target water area based on the detection results, and send the water area identification, water surface image, predicted detection result and number corresponding to the target water area as the edge detection result to the server.

[0076] As a specific implementation of the edge end, the water surface image and the predicted detection results are displayed on the edge end through an interactive display screen, and the parameters of the edge end equipment and the deployed floating object detection model are configured and adjusted. The edge end equipment includes a processor and a camera, a display screen and a communication module for data interaction with the processor, wherein the camera and the display screen interact with the processor in a wired communication manner, and the processor interacts with the server in a wireless communication manner through the communication module.

[0077] In this embodiment, the wireless communication methods between the edge end and the server end include 4G and Ethernet.

[0078] Step S300 server-side calculation: receiving and storing edge detection results on the server, generating abnormal information of the target water area based on the edge detection results, and pushing the abnormal information to relevant supervisors.

[0079] In the specific case of this embodiment, the edge end uses Jetson nano as the system core board, and cooperates with a high-definition camera to collect water surface images. The high-definition camera interacts with the processor at the edge end in the form of USB, and the detection results can be visualized on the LCD screen at the edge end in a wired manner.

[0080] When using convolutional neural networks to extract image features, as the number of convolutional layers increases, the resolution of the feature map gradually decreases, while the range of the local receptive field continues to expand. At the top level of the network, the feature map tends to capture the global information of the image, which makes deep features less suitable for small target detection. Although the shallow feature map has a smaller local receptive field and can contain more position and detail information, if only shallow features are relied upon for target detection, serious false detection and missed detection problems will occur due to the lack of guidance from high-level semantic information. This embodiment fuses shallow and deep features, and predicts the position of objects on feature maps of different scales after fusion. This multi-scale detection method can significantly improve the performance of the algorithm.

[0081] In YOLOv5s, if the size of the input image is 640×640, after being downsampled by 8 times, 16 times, and 32 times, the feature maps of the corresponding sizes are 80×80, 40×40, and 20×20, and the network finally performs detection on the feature maps of these three scales. Among them, the feature map obtained by 8 times downsampling has the smallest local receptive field, that is, when the feature map is mapped to the original input image, each grid corresponds to the 8×8 area of ​​the original image. For the detection of small targets, the receptive field of the feature map obtained by 8 times downsampling is still relatively large, which easily leads to the loss of the location and detail information of some small targets. In order to improve the problem of missed detection of small targets, the method of this embodiment improves the backbone network of YOLOv5s and introduces a new downsampling convolution layer in the backbone network. The new downsampling convolution layer samples the input image by 4 times, so that the floating object detection model of this embodiment can predict targets of different scales on the feature maps obtained by downsampling by 4 times, 8 times, 16 times and 32 times, effectively improving the detection performance of small targets and improving the effect of the model in multi-scale target detection.

[0082] Upsampling is a basic operation in deep neural networks. It is usually used to upsample low-resolution features to high resolution. It is also often used to achieve the fusion of high-resolution and low-resolution features. In convolutional neural networks, this operation plays a vital role.

[0083] At present, the commonly used upsampling methods mainly include nearest neighbor interpolation, bilinear interpolation and deconvolution. The first two mainly focus on sub-pixel space, with a small receptive field (1×1 for nearest neighbor interpolation and 2×2 for bilinear interpolation), and cannot obtain sufficient semantic information. The deconvolution upsampling method is not sensitive to local changes, and has many parameters and large calculation amount. This embodiment uses CARAFE (Content-aware reassembly of features) upsampling to replace the nearest neighbor interpolation upsampling in the original YOLOv5s network. CARAFE is a lightweight upsampling operator that enables the network to have a larger receptive field when sampling, and can make full use of the semantic information of the feature map.

[0084] Embodiment 2:

[0085] The present invention discloses a water surface floating object detection system based on machine learning, comprising an edge end and a service end.

[0086] The edge end is deployed with a camera and a trained floating object detection model. For the target water area, the edge end is used to perform the following: collect the water surface image of the target water area through the camera, use the water surface image as input, and use the trained floating object detection model to identify the type and position of floating objects, output the predicted detection results, and perform statistical analysis on the type and number of floating objects in the target water area based on the detection results, and send the water area identification, water surface image, predicted detection result and number corresponding to the target water area as the edge detection result to the server.

[0087] Among them, the floating object detection model is a deep learning model constructed by taking the YOLO5vs network as the basic network, introducing downsampling convolution layers and CARAFE upsampling. The introduced downsampling convolution layers are added as new downsampling convolution layers to the backbone network of the original YOLO5vs network. The new downsampling convolution layers are used to perform deep feature extraction on the input image and output feature maps as new vital sign maps. The CARAFE upsampling is used to replace the nearest neighbor interpolation upsampling of the neck in the original YOLO5vs network. The floating object detection model takes the water surface image as input, and is used to obtain feature maps of four different scales including the new feature map by extracting features from the water surface image, and performs type and position recognition of floating objects based on the four feature maps to obtain detection results, and the detection results include position, type and confidence.

[0088] As a specific implementation of the floating object detection model, the model includes an input layer, a backbone network, a neck, and a head.

[0089] The input layer uses the water surface image as the input image and is used to preprocess the input image. Mosaic data enhancement, adaptive aiming frame calculation and adaptive image scaling are performed through image preprocessing, and the preprocessed water surface image is output.

[0090] The backbone network includes a Focus structure, a newly added downsampling convolution layer, a CSP structure composed of multiple downsampling convolution layers and multiple C3 modules stacked, and an SPP module. The Focus structure uses the processed water surface image as an input image, is used to slice the input image and output a feature map. The newly added downsampling convolution layer uses the feature map output by the Focus structure as an input image, is used to extract features from the input image, and outputs the feature map as a P2 feature map. The CSP structure uses the feature map output by the newly added downsampling convolution layer as an input image, is used to extract multi-scale features from the input image, and outputs feature maps of three different scales, which are P3 feature maps, P4 feature maps, and P5 feature maps. The SPP module uses the P2 feature map, the P3 feature map, the P4 feature map, and the P5 feature map as input images, is used to downsample the input features and perform global maximum pooling operations, and output a merged feature map.

[0091] The neck includes an FPN module and a PAN module. The FPN module uses the feature map output by the backbone network as an input image, and is used to upsample the input image through CARAFE upsampling, perform feature fusion through the upsampling operation, and output a predicted feature map. The PAN module uses the feature map output by the FPN module as an input image, and is used to downsample the input image, perform feature fusion through the downsampling operation, and output a predicted feature map.

[0092] Among them, CARAFE upsampling is used as a lightweight upsampling operator to perform upsampling kernel prediction and feature reorganization on the input image.

[0093] Among them, the upsampling kernel prediction includes the following operations:

[0094] (1) For an input image of size H×W×C, a channel compression operation is performed through a 1×1 convolution to reduce the number of channels of the input image to C. m ;

[0095] (2) For the compressed input image, the size is K encoder ×K encoder The convolutional layer encodes the content and expands it in the spatial dimension to obtain a size of The upsampling kernel is normalized by the Softmax function, so that the sum of the convolution kernel weights is equal to l.

[0096] Correspondingly, feature recombination includes the following steps:

[0097] (1) Perform a dot product operation on the central k×k subregion of the input image and the upsampled kernel of each pixel obtained by prediction to obtain an output feature map of size σH×σW×C;

[0098] (2) For a target position l ′ and the corresponding square area (χ l ,k up ), the recombination process formula is:

[0099]

[0100] r = k up / 2,

[0101] Among them, σ represents the upsampling ratio, X l It means that k up represents, r represents, X l ″ indicates that w l′(n,m) express.

[0102] The head includes multiple output layers, each of which is used to identify the type and location of floating objects on feature maps of different scales, and generate the bounding box coordinates, type probability and confidence of the existence of floating objects of the prediction box.

[0103] As a specific implementation of the edge end, the water surface image and the predicted detection results are displayed on the edge end through an interactive display screen, and the parameters of the edge end equipment and the deployed floating object detection model are configured and adjusted. The edge end equipment includes a processor and a camera, a display screen and a communication module for data interaction with the processor, wherein the camera and the display screen interact with the processor in a wired communication manner, and the processor interacts with the server in a wireless communication manner through the communication module.

[0104] In this embodiment, the wireless communication methods between the edge end and the server end include 4G and Ethernet.

[0105] The server is used to receive and store edge detection results, generate abnormal information of the target waters based on the edge detection results, and push the abnormal information to relevant supervisors.

[0106] In the specific case of this embodiment, the edge end uses Jetson nano as the system core board, and cooperates with a high-definition camera to collect water surface images. The high-definition camera interacts with the processor at the edge end in the form of USB, and the detection results can be visualized on the LCD screen at the edge end in a wired manner.

[0107] The system of this embodiment can execute the method disclosed in Example 1 to detect floating objects on the water surface.

[0108] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A method for detecting floating objects on a water surface based on machine learning, characterized in that: Applied to the edge and server, it includes the following steps: A floating object detection model is constructed by using the YOLO5vs network as a basic network, introducing a downsampling convolution layer and CARAFE upsampling, and the introduced downsampling convolution layer is added to the backbone network of the original YOLO5vs network as a newly added downsampling convolution layer. The newly added downsampling convolution layer is used to perform deep feature extraction on the input image, and the output feature map is used as a newly added vital sign map. The CARAFE upsampling is used to replace the nearest neighbor interpolation upsampling of the neck in the original YOLO5vs network. The floating object detection model takes a water surface image as an input, and is used to obtain four feature maps of different scales including the newly added feature map by extracting features from the water surface image, and performs type and position recognition of floating objects based on the four feature maps to obtain a detection result, and the detection result includes a position, a type and a confidence level. Deploy the trained floating object detection model on the edge. For the target water area, collect the water surface image of the target water area through the camera, use the water surface image as input, and use the trained floating object detection model to identify the type and location of floating objects, output the predicted detection results, and perform statistical analysis on the type and number of floating objects in the target water area based on the detection results. Send the water area identification, water surface image, predicted detection result and number corresponding to the target water area to the server as the edge detection result; The edge detection results are received and stored on the server, abnormal information of the target waters is generated based on the edge detection results, and the abnormal information is pushed to relevant supervisors.

2. The method for detecting floating objects on the water surface based on machine learning according to claim 1, characterized in that: The floating object detection model includes an input layer, a backbone network, a neck and a head; The input layer uses the water surface image as the input image, and is used to perform image preprocessing on the input image, perform Mosaic data enhancement, adaptive aiming frame calculation and adaptive image scaling through the image preprocessing, and output the preprocessed water surface image; The backbone network includes a Focus structure, a newly added downsampling convolution layer, a CSP structure composed of a plurality of downsampling convolution layers and a plurality of C3 modules stacked, and an SPP module. The Focus structure takes the processed water surface image as an input image, is used for slicing the input image and outputting a feature map. The newly added downsampling convolution layer takes the feature map output by the Focus structure as an input image, is used for extracting features from the input image, and outputs the feature map as a P2 feature map. The CSP structure takes the feature map output by the newly added downsampling convolution layer as an input image, is used for extracting multi-scale features from the input image, and outputs feature maps of three different scales, namely, a P3 feature map, a P4 feature map, and a P5 feature map. The SPP module takes the P2 feature map, the P3 feature map, the P4 feature map, and the P5 feature map as input images, is used for downsampling the input features and performing global maximum pooling operations, and outputting a merged feature map. The neck includes an FPN module and a PAN module, wherein the FPN module uses the feature map output by the backbone network as an input image, and is used to perform an upsampling operation on the input image through CARAFE upsampling, and performs feature fusion and outputs a predicted feature map through the upsampling operation, and the PAN module uses the feature map output by the FPN module as an input image, and is used to perform a downsampling operation on the input image, and performs feature fusion and outputs a predicted feature map through the downsampling operation; The head includes multiple output layers, each of which is used to identify the type and position of floating objects on feature maps of different scales, and generate bounding box coordinates, type probability, and confidence of the existence of floating objects of the prediction box.

3. The method for detecting floating objects on the water surface based on machine learning according to claim 1 or 2, characterized in that: The CARAFE upsampling is used as a lightweight upsampling operator to perform upsampling kernel prediction and feature reorganization on the input image; Among them, the upsampling kernel prediction includes the following operations: For an input image of size H×W×C, a channel compression operation is performed through a 1×1 convolution to reduce the number of channels of the input image to C. m ; For the compressed input image, the size is K encoder ×K encoder The convolutional layer encodes the content and expands it in the spatial dimension to obtain a size of The upsampling kernel is normalized by the Softmax function, so that the sum of the convolution kernel weights is equal to l; Correspondingly, feature recombination includes the following steps: Perform a dot product operation on the central k×k sub-region of the input image and the upsampled kernel of each pixel obtained by prediction to obtain an output feature map of size σH×σW×C; For a target location l′ and a corresponding square area (χ l ,k up ), the recombination process formula is: r＝k up / 2， Among them, σ represents the upsampling ratio, χ l It means that k up represents, r represents, χ′ l′ It means that w l′(n,m) express.

4. The method for detecting floating objects on the water surface based on machine learning according to claim 1 or 2, characterized in that: At the edge, the water surface image and the predicted detection results are displayed through an interactive display screen, and the parameters of the edge equipment and the deployed floating object detection model are configured and adjusted. The edge equipment includes a processor and a camera, a display screen and a communication module for data interaction with the processor. The camera and the display screen interact with the processor in a wired communication manner, and the processor interacts with the server in a wireless communication manner through the communication module.

5. The method for detecting floating objects on the water surface based on machine learning according to claim 1 or 2, characterized in that: The wireless communication methods between the edge and the server include 4G and Ethernet.

6. A water surface floating object detection system based on machine learning, characterized in that: Including edge side and service side; The edge end is deployed with a camera and a trained floating object detection model. For the target water area, the edge end is used to perform the following: collect the water surface image of the target water area through the camera, use the water surface image as input, and use the trained floating object detection model to identify the type and position of floating objects, output the predicted detection results, and perform statistical analysis on the type and number of floating objects in the target water area based on the detection results, and send the water area identification, water surface image, predicted detection result and number corresponding to the target water area as the edge detection result to the server; The server is used to receive and store edge detection results, generate abnormal information of the target water area based on the edge detection results, and push the abnormal information to relevant supervisors; Among them, the floating object detection model is a deep learning model constructed by taking the YOLO5vs network as the basic network, introducing downsampling convolution layers and CARAFE upsampling. The introduced downsampling convolution layers are added as new downsampling convolution layers to the backbone network of the original YOLO5vs network. The new downsampling convolution layers are used to perform deep feature extraction on the input image and output feature maps as new vital sign maps. The CARAFE upsampling is used to replace the nearest neighbor interpolation upsampling of the neck in the original YOLO5vs network. The floating object detection model takes the water surface image as input, and is used to obtain feature maps of four different scales including the new feature map by extracting features from the water surface image, and performs type and position recognition of floating objects based on the four feature maps to obtain detection results, and the detection results include position, type and confidence.

7. The water surface floating object detection system based on machine learning according to claim 6 is characterized in that: The floating object detection model includes an input layer, a backbone network, a neck and a head; The input layer uses the water surface image as the input image, and is used to perform image preprocessing on the input image, perform Mosaic data enhancement, adaptive aiming frame calculation and adaptive image scaling through the image preprocessing, and output the preprocessed water surface image; The backbone network includes a Focus structure, a newly added downsampling convolution layer, a CSP structure composed of a plurality of downsampling convolution layers and a plurality of C3 modules stacked, and an SPP module. The Focus structure takes the processed water surface image as an input image, is used for slicing the input image and outputting a feature map. The newly added downsampling convolution layer takes the feature map output by the Focus structure as an input image, is used for extracting features from the input image, and outputs the feature map as a P2 feature map. The CSP structure takes the feature map output by the newly added downsampling convolution layer as an input image, is used for extracting multi-scale features from the input image, and outputs feature maps of three different scales, namely, a P3 feature map, a P4 feature map, and a P5 feature map. The SPP module takes the P2 feature map, the P3 feature map, the P4 feature map, and the P5 feature map as input images, is used for downsampling the input features and performing global maximum pooling operations, and outputting a merged feature map. The neck includes an FPN module and a PAN module, wherein the FPN module uses the feature map output by the backbone network as an input image, and is used to perform an upsampling operation on the input image through CARAFE upsampling, and performs feature fusion and outputs a predicted feature map through the upsampling operation, and the PAN module uses the feature map output by the FPN module as an input image, and is used to perform a downsampling operation on the input image, and performs feature fusion and outputs a predicted feature map through the downsampling operation; The head includes multiple output layers, each of which is used to identify the type and position of floating objects on feature maps of different scales, and generate bounding box coordinates, type probability, and confidence of the existence of floating objects of the prediction box.

8. The water surface floating object detection system based on machine learning according to claim 6 or 7, characterized in that: The CARAFE upsampling is used as a lightweight upsampling operator to perform upsampling kernel prediction and feature reorganization on the input image; Among them, the upsampling kernel prediction includes the following operations: For an input image of size H×W×C, a channel compression operation is performed through a 1×1 convolution to reduce the number of channels of the input image to C. m ; For the compressed input image, the size is K encoder ×K encoder The convolutional layer encodes the content and expands it in the spatial dimension to obtain a size of The upsampling kernel is normalized by the Softmax function, so that the sum of the convolution kernel weights is equal to l; Correspondingly, feature recombination includes the following steps: Perform a dot product operation on the central k×k sub-region of the input image and the upsampled kernel of each pixel obtained by prediction to obtain an output feature map of size σH×σW×C; For a target location l′ and a corresponding square area (χ l ,k up ), the recombination process formula is: r＝k up / 2， Among them, σ represents the upsampling ratio, X l It means that k up represents, r represents, χ′ l′ It means that w l′(n,m) express.

9. The water surface floating object detection system based on machine learning according to claim 6 or 7, characterized in that: An interactive display screen is deployed at the edge, which is used to display water surface images and predicted detection results, and to configure and adjust parameters of edge devices and deployed floating object detection models. The edge devices include a processor and a camera, a display screen and a communication module that interact with the processor for data. The camera and the display screen interact with the processor for data in the form of wired communication, and the processor interacts with the server for data in the form of wireless communication through the communication module.

10. The water surface floating object detection system based on machine learning according to claim 6 or 7, characterized in that: The wireless communication methods between the edge and the server include 4G and Ethernet.

Images