Water surface floating object detection method, medium and electronic equipment

Abstract

The invention relates to a water surface floating object detection method, a medium and electronic equipment. The method comprises the following steps: acquiring a continuous image which needs to be subjected to water surface floating object identification; for each frame of image in the continuous images, identifying and dividing a water area range in the image based on a pre-constructed first-stage water area division model; based on a pre-constructed target detection model of a second stage, performing target detection of a preset category of objects on each frame of image according to a time sequence; when the position of the target obtained in the second stage is included in the water area range obtained in the first stage, starting timing of pre-warning; and when the timing time is not lower than a preset threshold value, determining that the detected target is the water surface floating object.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, medium and electronic equipment for detecting floating objects on a water surface. Background Art

[0002] With the development of economy and technology, the management and protection of water resources has become particularly important. The identification and cleaning of floating objects on the water surface has become a part of water resource protection.

[0003] The traditional technology of identifying and cleaning floating objects on the water surface mainly relies on manual inspections. Obviously, this method is inefficient and has high labor costs.

[0004] With the continuous advancement of machine learning technology, many intelligent models, such as neural network models, are used in scenarios of detecting floating objects on the water surface.

[0005] The study found that the existing models for detecting floating objects on the water surface are mainly single-stage detection models, which have the overall problems of high data set acquisition costs, limited feature extraction and generalization capabilities of the trained models, and relatively low final recognition accuracy.

[0006] For example, the defects of the existing model can be reflected in, but not limited to, the following aspects:

[0007] (1) Floating objects on the water surface are widely distributed and difficult to detect and deal with in a timely manner. Traditional manual or mechanical inspection and cleaning methods are inefficient and costly.

[0008] (2) Water surface positioning is difficult: Under normal circumstances, existing water surface floating object detection requires the camera equipment to be aimed at the water surface area and the water surface area to be marked. However, when the camera is rotating, the water surface area is difficult to correct in time through artificial area division, which is very likely to interfere with the target detection results.

[0009] (3) Floating objects are difficult to define: In actual application scenarios, it is observed that since target detection cannot accurately determine the spatial position difference between objects on the water surface and objects above the water surface, some misjudgments may occur.

[0010] (4) Unbalanced dataset samples: There is currently no open source multi-classification dataset of floating objects on the water surface, and the classification definitions are vague.

[0011] (5) Floating object detection requires high target generalization ability: Since there are many types of floating objects on the water surface, including but not limited to fallen leaves, plastic products, foam, domestic garbage, etc., these floating objects vary in shape, size, color, material, etc. Large-scale floating objects coexist with small-target floating objects, which are ever-changing. This makes their feature extraction particularly important, and at the same time, the generalization ability of target detection is extremely high.

[0012] (6) There are many confusing factors: The shadows, reflections, and even ripples of floating objects and their surroundings can interfere with the detection algorithm, leading to false detection or missed detection. The presence of these factors can change the color, texture, and other characteristics of floating objects, making it difficult for the algorithm to accurately identify them. Summary of the invention

[0013] In view of the above problems, the purpose of the present invention is to provide a method for detecting floating objects on the water surface, which can improve the feature extraction and generalization capabilities of the trained model and improve the recognition accuracy of floating objects on the water surface.

[0014] To achieve the above object, the present invention adopts the following technical solutions:

[0015] In a first aspect, the present application provides a method for detecting floating objects on a water surface, the method comprising:

[0016] Acquire continuous images required for identification of floating objects on the water surface;

[0017] For each frame of the continuous image, the water area in the image is identified and divided based on the pre-built first-stage water area division model;

[0018] Based on the pre-built second-stage target detection model, target detection of preset category objects is performed on each frame of image in time sequence;

[0019] When the position of the target obtained in the second stage is included in the water area obtained in the first stage, the timing of the pre-alarm is started;

[0020] When the timing time is not less than the preset threshold, it is determined that the detected target is a floating object on the water surface.

[0021] In one implementation, the continuous images are acquired by a fixed camera or a mobile image acquisition device;

[0022] The fixed camera comprises a spherical camera fixed at a preset position;

[0023] The mobile image acquisition device is realized by a drone equipped with a camera.

[0024] In one implementation, the method further includes a step of constructing a water area division model, specifically including:

[0025] Collect data sets and mark the boundaries of the water area of ​​each sample image in the data sets;

[0026] Based on the data set, a water area division model with a preset network architecture is trained to obtain a water area division model with preset training indicators.

[0027] In one implementation, the network architecture of the water area division model adopts YOLOv8.

[0028] In one implementation, the method further includes a step of constructing a target detection model.

[0029] In one implementation, the network architecture of the target detection model optimizes the YOLO network architecture based on the HGNet network.

[0030] In one implementation, the YOLO network architecture is optimized based on the HGNet network, specifically including:

[0031] Use HGBlock in HGNet to perform convolution processing in the YOLOv5 backbone network; and

[0032] The AIFI module is used to replace the SPPF module in YOLOv5.

[0033] In one implementation, the method of determining that the location of the target obtained in the second stage is included in the water area obtained in the first stage specifically includes:

[0034] When the center point coordinates of the target obtained in the second stage are within the boundary range of the water area obtained in the first stage, it is determined that the position of the target obtained in the second stage is included in the water area obtained in the first stage.

[0035] In a second aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for detecting floating objects on the water surface described in the first aspect.

[0036] In a third aspect, an electronic device is provided, comprising a processor and a memory; the memory stores a computer program, and the processor runs the computer program to control the electronic device to execute the method for detecting floating objects on the water surface described in the first aspect.

[0037] The present invention adopts the above technical solution, which has the following advantages:

[0038] 1. By combining image segmentation with target detection instead of manual water surface area labeling, the changes in water surface area caused by frequent changes in the image can be avoided, thereby increasing the accuracy of algorithm recognition.

[0039] 2. By improving the target detection algorithm of YOLOv5, adding the HGBlock module and the attention-based internal scale feature interaction module, the ability of the network structure to process complex image data is improved.

[0040] 3. By performing IOU comparison and target warning interval judgment based on the previous and next frame images, some non-floating moving objects above the water surface can be effectively filtered out to improve the recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic flow chart of a method for detecting floating objects on a water surface provided by an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of the structure of the network architecture of the target detection model provided by an embodiment of the present invention;

[0043] Figure 3 It is a schematic diagram of the structure of HGBlock in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0045] In view of the problems existing in the prior art of detecting floating objects on the water surface, the technical solution of the present invention provides a method for detecting floating objects on the water surface. The method comprises:

[0046] Acquire continuous images required for identification of floating objects on the water surface;

[0047] For each frame of the continuous image, the water area in the image is identified and divided based on the pre-built first-stage water area division model;

[0048] Based on the pre-built second-stage target detection model, target detection of preset category objects is performed on each frame of image in time sequence;

[0049] When the position of the target obtained in the second stage is included in the water area obtained in the first stage, the timing of the pre-alarm is started;

[0050] When the timing time is not less than the preset threshold, it is determined that the detected target is a floating object on the water surface.

[0051] Please refer to more drawings of the embodiments of the present invention, in which the method and medium provided by the present invention are further explained in more detailed embodiments of the present invention.

[0052] like Figure 1 In one aspect of the embodiments of the present application, a method for detecting floating objects on a water surface is provided, which specifically includes:

[0053] (1) Obtaining continuous images for identifying floating objects on the water surface;

[0054] Specifically, continuous images can be obtained by shooting the water surface with a drone or a spherical camera.

[0055] (2) For each frame of the continuous image, the water area in the image is identified and divided based on the pre-built first-stage water area division model;

[0056] Specifically, before this step, the steps of constructing a watershed division model include:

[0057] Collect data sets and mark the boundaries of the water area of ​​each sample image in the data sets;

[0058] Based on the data set, a water area division model with a preset network architecture is trained to obtain a water area division model with preset training indicators.

[0059] In this step, the YOLOv8 model is used with the self-built data set to perform water surface segmentation training to obtain a segmentation model. The segmentation model detects each frame of the continuous image and outputs the boundary of the water area. The boundary can be a straight line.

[0060] (3) Based on the pre-built second-stage target detection model, target detection of objects of preset categories is performed on each frame of the image in time sequence;

[0061] Specifically, the network architecture used in this step optimizes the YOLOv5 backbone network based on HGNetv2.

[0062] HGNet is composed of multiple HGBlocks and uses a lot of convolution processing. Therefore, its backbone network is used to replace the convolution work of the original backbone structure in YOLOv5, such as Figure 2 The backbone part is shown in the figure.

[0063] HGBlock module such as Figure 3 As shown in Figure 1, it consists of a Conv convolutional layer and an ESE module. The main feature of this structure is that it uses a hierarchical approach to extract features, learns at different scales and abstract levels, and improves the network's ability to process complex image data.

[0064] Then, the AIFI (Attention-based Intra-scale Feature Interaction) module is used to replace the SPPF module in YOLOv5. The AIFI module is an attention-based intra-scale feature interaction module. It first pulls the two-dimensional features into vectors and then passes them to the AIFI module for processing. Its mathematical process is multi-head self-attention and FFN. Then, the output is reconstructed into two dimensions to complete the subsequent "cross-scale feature fusion".

[0065] As a key component, the AIFI module, together with the CNN-based cross-scale feature fusion module (CCFM), constitutes the encoder part of the model. Its features are:

[0066] ① Attention-based feature processing: The AIFI module uses the self-attention mechanism to process high-level features in the image. Self-attention is a mechanism that allows the model to process a specific part of the data while taking into account other relevant parts of the data. This method is particularly suitable for processing high-level image features with rich semantic information.

[0067] ② Selective feature interaction: The AIFI module focuses on internal scale interactions at high-level feature layers. This is based on the recognition that high-level feature layers contain richer semantic concepts and can more effectively capture the connections between conceptual entities in the image. At the same time, avoid the same interactions at low-level feature layers, because low-level features lack the necessary semantic depth and may lead to duplication and confusion in data processing.

[0068] This step performs target detection on each frame of the continuous image based on the trained target detection model, and outputs the semantic category and location information of the target, where the location information can be the coordinates of the center point of the target.

[0069] (4) When the target position obtained in the second stage is within the water area obtained in the first stage, the timer of the pre-alarm is started;

[0070] (5) When the timing time is not less than the preset threshold, the detected target is determined to be a floating object on the water surface.

[0071] Specifically, for example, the IOU accuracy comparison of the previous and next frame targets and the alarm interval judgment can be performed to eliminate the possible non-floating surface area targets (such as unidentified objects moving at high speed above the water surface under the camera, etc.). If it meets the requirements, it will be identified as a floating object on the water surface and an alarm will be output.

[0072] In another aspect of the embodiments of the present application, a computer storage medium is also provided accordingly.

[0073] The embodiment of the present application also provides a computer-readable storage medium, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above method. The specific implementation process will not be repeated here.

[0074] The present application also provides an electronic device in an embodiment. The electronic device in this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned method in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program is executed by the processor, the functions of each model / unit in the device in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0075] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, a server, and a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the electronic device may include more or fewer components than shown in the figure, or may combine certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0076] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0077] The memory may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory may also include both an internal storage unit of the electronic device and an external storage device. The memory is used to store computer programs and other programs and data required by the electronic device. The memory may also be used to temporarily store data that has been output or is to be output.

[0078] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0079] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0080] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (Processor) to perform some steps of the above-mentioned method of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0081] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting floating objects on a water surface, characterized in that: The method comprises: Acquire continuous images required for identification of floating objects on the water surface; For each frame of the continuous image, the water area in the image is identified and divided based on the pre-built first-stage water area division model; Based on the pre-built second-stage target detection model, target detection of preset category objects is performed on each frame of image in time sequence; When the position of the target obtained in the second stage is included in the water area obtained in the first stage, the timing of the pre-alarm is started; When the timing time is not less than the preset threshold, it is determined that the detected target is a floating object on the water surface.

2. The method for detecting floating objects on the water surface according to claim 1, characterized in that: The continuous images are acquired by a fixed camera or a mobile image acquisition device; The fixed camera comprises a spherical camera fixed at a preset position; The mobile image acquisition device is realized by a drone equipped with a camera.

3. The method for detecting floating objects on the water surface according to claim 1, characterized in that: The method also includes a step of constructing a water area division model, specifically including: Collect data sets and mark the boundaries of the water area of ​​each sample image in the data sets; Based on the data set, a water area division model with a preset network architecture is trained to obtain a water area division model with preset training indicators.

4. The method for detecting floating objects on the water surface according to claim 3, characterized in that: The network architecture of the water area division model adopts YOLOv8.

5. The method for detecting floating objects on the water surface according to claim 1, characterized in that: The method also includes a step of constructing a target detection model.

6. The method for detecting floating objects on the water surface according to claim 5, characterized in that: The network architecture of the target detection model optimizes the YOLO network architecture based on the HGNet network.

7. The method for detecting floating objects on the water surface according to claim 6, characterized in that: The YOLO network architecture is optimized based on the HGNet network, including: Use HGBlock in HGNet to perform convolution processing in the YOLOv5 backbone network; and The AIFI module is used to replace the SPPF module in YOLOv5.

8. The method for detecting floating objects on the water surface according to claim 1, characterized in that: The method of determining that the location of the target obtained in the second phase is contained within the water area obtained in the first phase includes: When the center point coordinates of the target obtained in the second stage are within the boundary range of the water area obtained in the first stage, it is determined that the position of the target obtained in the second stage is included in the water area obtained in the first stage.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed, controls the device where the computer-readable storage medium is located to execute the method for detecting floating objects on the water surface according to any one of claims 1 to 8.

10. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores a computer program, and the processor runs the computer program to control the electronic device to execute the method for detecting floating objects on the water surface according to any one of claims 1 to 8.

Images