Water surface floating object detection method and device based on multispectral image and storage medium

Abstract

The invention discloses a water surface floating object detection method and device based on a multispectral image and a storage medium, and the method comprises the steps: 1, carrying out the routing inspection of a target water surface through employing an unmanned plane, and carrying out the preprocessing of an obtained multispectral video; 2, processing the image preprocessed in the step 1 by using a semantic segmentation network based on an OfficientViT architecture to obtain a semantic segmentation result, obtaining floating object pixels, and completing the detection of the water surface floating object; 3, the floating object masks of the semantic segmentation result are converted into a minimum enclosing rectangle, the floating objects are tracked, and the unique number of each floating object is obtained; 4, calculating the position information of each floating object according to the unique number of each floating object, the attitude information of the unmanned aerial vehicle holder and the GPS information; and 5, if the real area of the floating object is larger than the threshold value, salvage processing is conducted according to the position information, obtained in the step 4, of the floating object.

Description

Technical Field

[0001] The present invention belongs to the technical field of resource and environment detection, and in particular relates to a method, a device and a storage medium for detecting floating objects on a water surface based on multispectral images. Background Art

[0002] Protecting the aquatic environment is essential to maintaining ecological balance and human health. In my country, the protection of the aquatic environment has always been a top priority for environmental protection. Floating branches, fallen leaves, plastic and other garbage on the water surface will not only cause negative visual effects, but also may cause the deterioration of the water body. A large amount of floating objects may even interfere with the normal operation of the reservoir, especially for power stations. Floating objects may affect the monitoring of reservoir water levels, the structure of dams and the operation of hydraulic equipment. Therefore, real-time monitoring of floating objects can help improve the operating efficiency of reservoirs and reduce potential damage. However, due to equipment and technical limitations, the current inspection of floating objects mainly relies on manpower, which not only has low inspection efficiency but also long inspection cycles.

[0003] At present, with the rapid development of unmanned intelligent technology, drones and unmanned boats have begun to be used in the inspection of floating objects on the water surface. Drones can switch seamlessly between land and water, while unmanned boats are mainly suitable for water areas. This makes drones more suitable for performing diverse tasks between different terrains. Drones can carry a variety of equipment. At present, in the inspection of floating objects on the water surface, the main equipment is a visible light camera. The CN115019122A patent is based on ordinary visible light cameras to detect floating objects on the water surface.

[0004] However, the information that visible light cameras can capture is limited, which leads to false detections in areas such as ripples on the water surface and reflections on the water surface when performing detection tasks. Multispectral cameras use the multi-band imaging principle in engineering technology to accurately capture the spectral characteristics of objects such as color, humidity, and temperature by recording spectral information of specific wavelengths. Compared with traditional cameras, multispectral cameras have higher spectral resolution and richer color reproduction capabilities. It can capture subtle changes that are imperceptible to the human eye, and can provide richer and more accurate image information, laying the foundation for improving the accuracy of floating object detection on the water surface.

[0005] However, there is no patent or journal disclosure on how to use multispectral cameras and drones in combination for detecting floating objects on the water surface; therefore, in order to effectively improve the efficiency and accuracy of detecting floating objects on the water surface, the present invention provides a method for detecting floating objects on the water surface based on multispectral images, and uses a drone equipped with a multispectral camera to detect and track floating objects on the water surface. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and provide a method, device and storage medium for detecting floating objects on the water surface based on multispectral images, so as to improve the accuracy of detecting floating objects on the water surface and realize the tracking and positioning of floating objects.

[0007] In order to solve the technical problem, the technical method of the present invention is: a method for detecting floating objects on the water surface based on multispectral images, comprising the following steps:

[0008] Step 1: Use a drone equipped with multispectral equipment to inspect the target water surface, align the acquired multispectral video, and perform normalization preprocessing;

[0009] Step 2: Use the semantic segmentation network based on the EfficientViT architecture to process the image preprocessed in step 1 to obtain the semantic segmentation result. The semantic segmentation result includes floating object pixels. Multiple adjacent floating object pixels constitute a floating object mask to complete the detection of floating objects on the water surface.

[0010] Step 3: Convert the floating object mask of the semantic segmentation result into the minimum bounding rectangle, and then track the floating objects based on Kalman filtering and Hungarian algorithm to obtain the unique number of each floating object;

[0011] Step 4: Calculate the location information of each floating object based on the unique number of each floating object, the attitude information of the drone gimbal, and the GPS information;

[0012] Step 5: Convert the floating object pixels in step 2 into the real area of ​​the floating object, and issue an early warning according to the real area of ​​the floating object. If the real area of ​​the floating object is greater than the threshold, salvage the floating object based on the location information of the floating object obtained in step 4.

[0013] Preferably, the step 1 is specifically as follows: the multispectral video includes a visible light image, a red edge image, and a near infrared image, and the collected visible light image, red edge image, and near infrared image are aligned and normalized for preprocessing;

[0014] The alignment method is: using bilinear interpolation to scale the three image sizes to be consistent with the data format and type accepted by the semantic segmentation network based on the EfficientViT architecture in step 2;

[0015] The normalized calculation method is as follows:

[0016]

[0017] Where:

[0018] x′ is the normalized data;

[0019] x is the original data;

[0020] x max is the maximum value;

[0021] x min is the minimum value.

[0022] Preferably, the semantic segmentation network based on the EfficientViT architecture in step 2 is composed of an input trunk, four stages and a segmentation head in sequence. The image data preprocessed in step 1 enters the input trunk through the input, and then passes through stages 1, 2, 3 and 4 in sequence. The output of stage 2 is P2, the output of stage 3 is P3, and the output of stage 4 is P4. The upsampling operation is completed using bilinear interpolation, and P2, P3 and P4 are fused via addition. Finally, the detection of floating objects on the water surface is completed based on the segmentation head to obtain the semantic segmentation result and the floating object mask. At the same time, the EfficientViT module is inserted in stages 3 and 4, and the ReLU linear attention is used to realize the global receptive field and multi-scale learning.

[0023] Preferably, the EfficientViT module is specifically as follows: data is input into linear unit 1, the output of linear unit 1 is then input into QKV, the output of QKV is respectively input into depth separable convolution and ReLU linear attention 1, the output of depth separable convolution is input into 1×1 grouped convolution, the output of 1×1 grouped convolution is input into ReLU linear attention 2, the outputs of ReLU linear attention 1 and ReLU linear attention 2 are additively fused, the fused data is input into linear unit 2, the output of linear unit 2 enters FFN layer + deep convolution network for output, and the EfficientViT module inserts a deep convolution network in each FFN layer, wherein ReLU linear attention captures contextual information, and FFN and deep convolution network capture local information.

[0024] Preferably, the step 3 is specifically as follows: according to the mask of the floating object output in step 2, the mask image is traversed to determine whether the 8 surrounding pixels are still mask pixels, to determine all connected domains in the image, and to find the minimum value x of all points in the connected domain on the x-axis. min and the maximum value x max and the minimum value y on the y-axis min and the maximum value y max , then the coordinates of the upper left corner of the minimum bounding rectangle of the floating object area are (x min ,y min ), the coordinates of the lower right corner are (x max ,y max ), get the floating object pixels, and then complete the tracking of floating objects based on Kalman filtering and Hungarian algorithm, and assign a unique target number to each target.

[0025] Preferably, the step 4 is specifically as follows: assigning a unique target number to each target, and calculating the specific latitude and longitude coordinates of the floating object based on the attitude information, position information, altitude information, gimbal attitude information of the aircraft when the target is found, and the position of the floating object in the image. The calculation process is as follows:

[0026] Step 4-1: First, convert the diagonal field of view provided by the multispectral device into horizontal and vertical field of view.

[0027]

[0028]

[0029] Where:

[0030] W and H represent the width and height of the image, pixels;

[0031] FOV D is the diagonal field of view, degrees;

[0032] FOV H is the horizontal field of view, degree;

[0033] FOV V is the vertical field of view, degree;

[0034] Step 4-2: Calculate the deviation between the center point of the minimum circumscribed rectangle of the floating object and the center point of the image. The specific calculation formula is as follows:

[0035] Δx pix =x obj -(W / 2)

[0036] Δy pix =y obj -(H / 2)

[0037] Where:

[0038] x obj ,y obj The coordinates of the center point of the minimum circumscribed rectangle of the floating object on the water surface in the pixel coordinate system, where the pixel coordinates are the upper left corner of the image as the origin of the coordinate system, and downward and rightward are positive directions;

[0039] Δx pix , Δy pix is the pixel deviation of the floating object on the water surface relative to the center point of the image;

[0040] Step 4-3: Calculate the object distance. The specific formula is as follows:

[0041] dis=height·cos(90-α-β)

[0042] Where:

[0043] dis is the object distance, meters;

[0044] height is the vertical height of the drone from the water surface, meters;

[0045] α is the pitch angle of the drone, degrees;

[0046] β is the pitch angle of the pod, degrees;

[0047] Step 4-4: Calculate the actual deviation of the floating object on the water surface relative to the intersection of the optical axis and the water surface;

[0048] Δx=2Δx pix / W·dis·tan(FOV H / 2)

[0049] Δy=2Δy pix / H·dis·tan(FOV V / 2)

[0050] Step 4-5: Calculate the deviation of the floating object on the water surface relative to the vertical projection point of the drone on the water surface, with due north as 0 degrees;

[0051]

[0052] θ=arctan(Δx / (dis·cos(α+β)+Δy)+yaw uav +yaw pod

[0053] Where:

[0054] yaw uav is the yaw angle of the UAV, degrees;

[0055] yaw pod is the yaw angle of the pod on the drone, degree;

[0056] dis obj is the deviation of the floating object relative to the vertical projection point of the UAV on the water surface, meter;

[0057] θ is the azimuth of the floating object on the water surface relative to the true north direction, degrees;

[0058] Step 4-6: Convert the coordinates of floating objects on the water surface into longitude and latitude;

[0059] The latitude and longitude position of the UAV can be directly obtained through GPS. In step 4-5, the deviation of the floating object relative to the vertical projection point of the UAV on the water surface has been calculated. obj And the azimuth angle θ, the longitude and latitude coordinates of the floating object on the water surface are calculated by the following formula;

[0060] Lonobj =Lon uav +arctan(dis obj sinθ / ARC)

[0061] Lat obj =Lat uav +arctan(dis obj ·cosθ / ARC)

[0062] Where:

[0063] Lon obj and Lat obj is the latitude and longitude of floating objects on the water surface, degrees;

[0064] Lon uav and Lat uav is the latitude and longitude of the drone, degrees;

[0065] dis obj is the deviation of the floating object relative to the vertical projection point of the UAV on the water surface, meter;

[0066] θ is the azimuth of the floating object on the water surface relative to the true north direction, degrees;

[0067] ARC is the radius of the earth, which is 6371.393*1000 meters.

[0068] Preferably, the step 5 specifically comprises: counting each pixel of the floating object tracked in the step 2, and converting it into the real area of ​​the floating object according to the camera parameters, giving an early warning according to the calculated real area of ​​the floating object, and if the real area of ​​the floating object is greater than a threshold, performing salvage processing based on the position information of the floating object obtained in the step 4.

[0069] Preferably, a water surface floating object detection device based on multispectral imaging comprises:

[0070] Acquisition module: Use drones equipped with multispectral equipment to acquire multispectral videos;

[0071] Preprocessing module: align and normalize the acquired multispectral video;

[0072] Processing module: Using the semantic segmentation network based on the EfficientViT architecture, we can obtain floating object pixels and detect floating objects on the water surface. We can then convert the floating object mask into a minimum bounding rectangle, and then track the floating objects based on the Kalman filter and Hungarian algorithm to obtain a unique number for each floating object.

[0073] Positioning module: Calculate the location information of each floating object based on the unique number of each floating object, the attitude information of the drone gimbal, and the GPS information;

[0074] Early warning module: converts the floating object pixels obtained by the processing module into the actual area of ​​the floating object, and issues early warnings according to the actual area of ​​the floating object. If the actual area of ​​the floating object is greater than the threshold, the location information of the floating object obtained by the positioning module is used for salvage.

[0075] Preferably, a storage medium stores computer executable instructions, wherein the computer executable instructions are used to enable a computer to execute the steps included in the method for detecting floating objects on the water surface based on multispectral images.

[0076] Compared with the prior art, the advantages of the present invention are:

[0077] (1) The present invention discloses a method for detecting floating objects on the water surface based on multispectral images. The method realizes the functions of detecting, tracking, locating, and warning floating objects in the water area by performing feature extraction and analysis on the multispectral images. The present invention can effectively improve the difficulty of detecting floating objects on the water surface, which is easily disturbed by ripples, reflections, and reflections of trees and buildings, and improves the accuracy and robustness of detecting floating objects on the water surface.

[0078] (2) The present invention adopts a semantic segmentation network based on the EfficientViT architecture. The ReLU linear attention realizes the global receptive field and multi-scale learning, captures contextual information, and the FFN and deep convolution capture local information, which improves the segmentation ability and is more suitable for extracting floating objects on the water surface.

[0079] (3) The method for detecting floating objects on the water surface based on multi-spectral video of the present invention can effectively improve the inspection efficiency of floating objects on the water surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 , a schematic diagram of the use of a method for detecting floating objects on a water surface based on multispectral images according to the present invention;

[0081] Figure 2 , a schematic diagram of a flow chart of a method for detecting floating objects on a water surface based on multispectral images of the present invention;

[0082] Figure 3 , a schematic diagram of the semantic segmentation network structure of the EfficientViT architecture of the present invention;

[0083] Figure 4 , a structural block diagram of a water surface floating object detection device based on multispectral images of the present invention.

[0084] Description of reference numerals:

[0085] 1. Drone, 2. Multispectral equipment, 3. Floating objects, 4. Water surface. DETAILED DESCRIPTION

[0086] The specific implementation mode of the present invention is described below in conjunction with embodiments:

[0087] It should be noted that the contents shown in the drawings of this specification are only used to cooperate with the principles, features and content descriptions disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions under which the present invention can be implemented. Without affecting the effects and purposes that can be achieved by the present invention, they should still fall within the scope of the technical contents disclosed in the present invention.

[0088] like Figures 1 to 4 As shown, the present invention discloses a method for detecting floating objects on a water surface based on multispectral images, comprising the following steps:

[0089] Step 1: Use a drone equipped with multispectral equipment to inspect the target water surface, align the acquired multispectral video, and perform normalization preprocessing;

[0090] Step 2: Use the semantic segmentation network based on the EfficientViT architecture to process the image preprocessed in step 1 to obtain the semantic segmentation result. The semantic segmentation result includes floating object pixels. Multiple adjacent floating object pixels constitute a floating object mask to complete the detection of floating objects on the water surface.

[0091] Semantic segmentation is essentially pixel-level classification of images, where each pixel is assigned to a specific category. For the detection of floating objects on the water surface, multiple adjacent pixels belonging to the floating object category form a connected domain, which represents a floating object mask.

[0092] Step 3: Convert the floating object mask of the semantic segmentation result into the minimum bounding rectangle, and then track the floating objects based on Kalman filtering and Hungarian algorithm to obtain the unique number of each floating object;

[0093] Step 4: Calculate the location information of each floating object based on the unique number of each floating object, the attitude information of the drone gimbal, and the GPS information;

[0094] Step 5: Convert the floating object pixels in step 2 into the real area of ​​the floating object, and issue an early warning according to the real area of ​​the floating object. If the real area of ​​the floating object is greater than the threshold, salvage the floating object based on the location information of the floating object obtained in step 4.

[0095] Preferably, the step 1 is specifically as follows: the multispectral video includes a visible light image, a red edge image, and a near infrared image, and the collected visible light image, red edge image, and near infrared image are aligned and normalized for preprocessing;

[0096] The red edge image refers to the area between the red light band and the near infrared band in the image spectrum. The alignment method is: using bilinear interpolation to scale the three image sizes to be consistent with the data format and type accepted by the semantic segmentation network based on the EfficientViT architecture in step 2;

[0097] The normalized calculation method is as follows:

[0098]

[0099] Where:

[0100] x′ is the normalized data;

[0101] x is the original data;

[0102] x max is the maximum value;

[0103] x min is the minimum value.

[0104] The purpose of normalization is to eliminate the dimensional differences between different features so that different features are comparable, thereby improving the performance of the semantic segmentation model.

[0105] The purpose of the present invention is to detect, track, locate and warn floating objects on the water surface based on multispectral data obtained by drones. The shapes and sizes of floating objects on the water surface can be very diverse, including but not limited to debris, leaves, phytoplankton, plastic products, etc., which makes it complicated to accurately distinguish floating objects in images. After comprehensive consideration and analysis, the present invention believes that semantic segmentation is more suitable for extracting floating objects on the water surface. The semantic segmentation network is implemented in the EfficientViT architecture, as follows:

[0106] like Figure 3 As shown in the figure, the semantic segmentation network based on the EfficientViT architecture is composed of an input backbone, four stages and a segmentation head. The image data preprocessed in step 1 is input into the input backbone, and then goes through stages 1, 2, 3 and 4 in sequence. The output of stage 2 is P2, the output of stage 3 is P3, and the output of stage 4 is P4. Bilinear interpolation is used to complete the upsampling operation, and P2, P3 and P4 are fused by addition. Finally, the detection of floating objects on the water surface is completed based on the segmentation head to obtain the semantic segmentation result and the floating object mask. At the same time, the EfficientViT module is inserted in stages 3 and 4, and ReLU linear attention is used to achieve global receptive field and multi-scale learning.

[0107] Preferably, the EfficientViT module can improve the segmentation capability. Specifically, the EfficientViT module is as follows: data is input into linear unit 1, the output of linear unit 1 is then input into QKV, the output of QKV is respectively input into depth separable convolution and ReLU linear attention 1, the output of depth separable convolution is input into 1×1 grouped convolution, the output of 1×1 grouped convolution is input into ReLU linear attention 2, the outputs of ReLU linear attention 1 and ReLU linear attention 2 are additively fused, the fused data is input into linear unit 2, the output of linear unit 2 enters FFN layer + deep convolution network for output, and the EfficientViT module inserts a deep convolution network in each FFN layer, wherein ReLU linear attention captures contextual information, and FFN and deep convolution network capture local information.

[0108] Preferably, the step 3 is specifically as follows: according to the mask of the floating object output in step 2, the mask image is traversed to determine whether the 8 surrounding pixels are still mask pixels, to determine all connected domains in the image, and to find the minimum value x of all points in the connected domain on the x-axis. min and the maximum value x max and the minimum value y on the y-axis min and the maximum value y max , then the coordinates of the upper left corner of the minimum bounding rectangle of the floating object area are (x min，ymin ), the coordinates of the lower right corner are (x max ,y max ), get the floating object pixels, and then complete the tracking of floating objects based on Kalman filtering and Hungarian algorithm, and assign a unique target number to each target.

[0109] The main goal of target tracking is to assign a unique number to the same target in different views from the time the target enters the field of view to the time the target leaves the field of view. The floating objects on the water surface use the result output by the semantic segmentation model (the mask of the floating objects). It is more complicated to directly track the corresponding features. The present invention completes the tracking of floating objects by obtaining the minimum circumscribed rectangle of the floating object area and then based on Kalman filtering and Hungarian algorithm.

[0110] The Kalman filter and Hungarian algorithm are conventional algorithms and will not be described in detail in this application.

[0111] Preferably, the step 4 is specifically as follows: assigning a unique target number to each target, and calculating the specific longitude and latitude coordinates of the floating object based on the aircraft's attitude information, position information (latitude and longitude coordinates), altitude information (vertical height from the water surface), gimbal attitude information, and the position of the floating object in the image when the target is discovered. The calculation process is as follows:

[0112] Step 4-1: First, convert the diagonal field of view provided by the multispectral device into horizontal and vertical field of view.

[0113]

[0114] Where:

[0115] W and H represent the width and height of the image, pixels;

[0116] FOV D is the diagonal field of view, degrees;

[0117] FOV H is the horizontal field of view, degree;

[0118] FOV V is the vertical field of view, degree;

[0119] Step 4-2: Calculate the deviation between the center point of the minimum circumscribed rectangle of the floating object and the center point of the image. The specific calculation formula is as follows:

[0120] Δx pix =x obj -(W / 2)

[0121] Δy pix =y obj -(H / 2)

[0122] Where:

[0123] x obj ,y obj The coordinates of the center point of the minimum circumscribed rectangle of the floating object on the water surface in the pixel coordinate system, where the pixel coordinates are the upper left corner of the image as the origin of the coordinate system, and downward and rightward are positive directions;

[0124] Δx pix , Δy pix is the pixel deviation of the floating object on the water surface relative to the center point of the image;

[0125] Step 4-3: Calculate the object distance. The specific formula is as follows:

[0126] dis=height·cos(90-α-β)

[0127] Where:

[0128] dis is the object distance, meters;

[0129] height is the vertical height of the drone from the water surface, meters;

[0130] α is the pitch angle of the drone, degrees;

[0131] β is the pitch angle of the pod (pitch angle coordinate system: the aircraft's forward direction is the positive direction, the horizontal direction is 0 degrees, and the downward direction is positive), degrees;

[0132] Step 4-4: Calculate the actual deviation of the floating object on the water surface relative to the intersection of the optical axis and the water surface;

[0133] Δx=2Δx pix / W·dis·tan(FOV H / 2)

[0134] Δy=2Δy pix / H·dis·tan(FOV V / 2)

[0135] Step 4-5: Calculate the deviation of the floating object on the water surface relative to the vertical projection point of the drone on the water surface, with due north as 0 degrees;

[0136]

[0137] θ=arctan(Δx / (dis·cos(α+β)+Δy)+yaw uav +yaw pod

[0138] Where:

[0139] yaw uav is the yaw angle of the UAV, degrees;

[0140] yaw pod is the yaw angle of the pod on the drone, degree;

[0141] dis obj is the deviation of the floating object relative to the vertical projection point of the UAV on the water surface, meter;

[0142] θ is the azimuth of the floating object on the water surface relative to the true north direction, degrees;

[0143] Step 4-6: Convert the coordinates of floating objects on the water surface into longitude and latitude;

[0144] The latitude and longitude position of the UAV can be directly obtained through GPS. In step 4-5, the deviation of the floating object relative to the vertical projection point of the UAV on the water surface has been calculated. obj And the azimuth angle θ, the longitude and latitude coordinates of the floating object on the water surface are calculated by the following formula;

[0145] Lon obj =Lon uav +arctan(dis obj sinθ / ARC)

[0146] Latobj =Lat uav +arctan(dis obj ·cosθ / ARC)

[0147] Where:

[0148] Lon obj and Lat obj is the latitude and longitude of floating objects on the water surface, degrees;

[0149] Lon uav and Lat uav is the latitude and longitude of the drone, degrees;

[0150] dis obj is the deviation of the floating object relative to the vertical projection point of the UAV on the water surface, meter;

[0151] θ is the azimuth of the floating object on the water surface relative to the true north direction, degrees;

[0152] ARC is the radius of the earth, which is 6371.393*1000 meters.

[0153] Preferably, the step 5 specifically comprises: counting each pixel of the floating object tracked in the step 2, and converting it into the real area of ​​the floating object according to the camera parameters, giving an early warning according to the calculated real area of ​​the floating object, and if the real area of ​​the floating object is greater than a threshold, performing salvage processing based on the position information of the floating object obtained in the step 4.

[0154] like Figure 4 As shown, preferably, a water surface floating object detection device based on multispectral images includes: an acquisition module: using a drone equipped with a multispectral device to acquire a multispectral video;

[0155] Preprocessing module: align and normalize the acquired multispectral video;

[0156] Processing module: Using the semantic segmentation network based on the EfficientViT architecture, we can obtain floating object pixels and detect floating objects on the water surface. We can then convert the floating object mask into a minimum bounding rectangle, and then track the floating objects based on the Kalman filter and Hungarian algorithm to obtain a unique number for each floating object.

[0157] Positioning module: Calculate the location information of each floating object based on the unique number of each floating object, the attitude information of the drone gimbal, and the GPS information;

[0158] Early warning module: converts the floating object pixels obtained by the processing module into the actual area of ​​the floating object, and issues early warnings according to the actual area of ​​the floating object. If the actual area of ​​the floating object is greater than the threshold, the location information of the floating object obtained by the positioning module is used for salvage.

[0159] Preferably, a storage medium stores computer executable instructions, wherein the computer executable instructions are used to enable a computer to execute the steps included in the method for detecting floating objects on the water surface based on multispectral images.

[0160] Example 1

[0161] Using drones for water surface trash detection and monitoring

[0162] a) Equipment preparation

[0163] like Figure 1 As shown, the multispectral device 2 is installed on the drone 1, and the drone 1 flies on the water surface 4 to inspect the floating objects 3.

[0164] The multispectral device 2 is selected from a multispectral camera: a multispectral camera suitable for the drone 1 is selected, which has the ability to capture multiple spectral bands (such as visible light, near infrared, red edge, etc.).

[0165] Drone: Equipped with a stable flight platform, it is capable of carrying a multispectral camera and flying steadily over the target waters.

[0166] b) Data collection

[0167] Flight planning: Plan the flight path of the drone based on the scope and terrain characteristics of the target waters to ensure coverage of the entire water area.

[0168] Flight execution: At the scheduled time and weather conditions, the drone is operated to fly along the planned path and collect data using a multispectral camera. The collected multispectral images should include the water surface and its surrounding environment.

[0169] c) Data preprocessing

[0170] like Figure 2 As shown, image correction: align the spliced ​​visible light image and near infrared and red edge images, and perform normalization preprocessing;

[0171] Alternatively, geometric and radiometric corrections can be performed to ensure the spatial and spectral accuracy of the image.

[0172] d) Model training

[0173] Use the semantic segmentation network based on the EfficientViT architecture to detect floating objects on the water surface;

[0174] Training dataset: Based on historical data and on-site annotations, a training dataset containing various types of floating objects on the water surface (such as plastic waste, wood, algae, etc.) is constructed.

[0175] Machine learning model: Select a suitable machine learning algorithm (such as support vector machine, random forest, deep learning, etc.) and use the training data set to train the model to achieve automatic recognition and classification of floating objects on the water surface.

[0176] e) Floating object identification and tracking

[0177] like Figure 2 As shown, recognition: the preprocessed multispectral image is input into the trained model for real-time detection and recognition of floating objects on the water surface.

[0178] Result output: Generate detection results including the location, type, area and quantity of floating objects.

[0179] The floating object mask of the semantic segmentation result is converted into a minimum bounding rectangle, and then the floating objects are tracked based on the Kalman filter and Hungarian algorithm to obtain a unique number for each floating object.

[0180] f) Positioning of floating objects

[0181] like Figure 2 As shown, the position information of the floating object is calculated based on the attitude information of the drone gimbal and the GPS information, as shown in the above formula.

[0182] g) Early warning

[0183] like Figure 2 As shown, whether an early warning is needed is determined based on whether the area of ​​the floating object is greater than the alarm threshold.

[0184] h) Application and feedback

[0185] Monitoring system: Integrate the detection results into the water area monitoring system to realize automatic monitoring and early warning of floating objects on the water surface.

[0186] Clean-up operations: Based on the test results, a water surface clean-up plan is developed, and clean-up vessels and personnel are dispatched to clean up floating objects.

[0187] Effect evaluation: Regularly evaluate the monitoring and cleaning effects, optimize the drone flight path and detection model, and improve detection accuracy and efficiency.

[0188] The principle of the present invention is as follows:

[0189] like Figures 1 to 4As shown, the present invention discloses a method, device and storage medium for detecting floating objects on the water surface based on multispectral images. The method for detecting floating objects on the water surface based on multispectral video is mainly divided into the following four parts: detection, tracking, positioning and early warning of floating objects on the water surface. Detection of floating objects on the water surface refers to using a deep learning method to find floating objects from water images collected by drones. However, due to problems such as reflection, ripples and reflections, it is often easy to have false detection problems when detecting ordinary RGB images. Multispectral cameras have higher spectral resolution and richer color restoration capabilities. It can provide richer and more accurate image information, thereby effectively reducing false detection of floating objects on the water surface in these complex scenes; tracking of floating objects on the water surface is achieved by converting the detected floating object mask into a minimum circumscribed rectangle, and then based on Kalman filtering and Hungarian algorithm; positioning of floating objects on the water surface is to calculate the longitude and latitude position of the floating object through information such as the current position, attitude, gimbal attitude and flight altitude of the drone, and complete the positioning of the detected floating objects on the water surface; finally, whether to issue an early warning is determined by whether the area of ​​the floating object is greater than the set threshold.

[0190] The present invention uses multispectral data, which is richer in information than visible light data, especially the water surface itself is highly sensitive to near-infrared and red edge data. Therefore, it is easier and more accurate to extract floating objects from the water surface by using multispectral data. In addition, the present invention combines deep learning to segment floating objects, which is more adaptable and accurate than traditional threshold segmentation and manual feature extraction.

[0191] The present invention discloses a method for detecting floating objects on the water surface based on multispectral images. The method realizes functions such as detection, tracking, positioning and early warning of floating objects in water areas by performing feature extraction and analysis on the multispectral images. The present invention can effectively improve the difficult problem that the detection of floating objects on the water surface is easily disturbed by water ripples, reflections, and reflections of trees and buildings, thereby improving the accuracy and robustness of the detection of floating objects on the water surface.

[0192] The present invention adopts a semantic segmentation network based on the EfficientViT architecture. ReLU linear attention realizes global receptive field and multi-scale learning, captures contextual information, FFN and deep convolution capture local information, improves segmentation capability, and is more suitable for extracting floating objects on the water surface.

[0193] The method for detecting floating objects on the water surface based on multispectral video of the present invention can effectively improve the inspection efficiency of floating objects on the water surface. Of course, there are also areas that need to be improved for this method. For example, in order to improve the stability of tracking floating objects on the water surface, it is necessary to limit the time of image acquisition and the flight route of the drone, which will put forward additional requirements for image acquisition personnel. It can be tried to optimize it at the algorithm level to reduce the acquisition requirements.

[0194] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary geologists in this field without departing from the purpose of the present invention.

[0195] Many other changes and modifications may be made without departing from the concept and scope of the present invention.It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A method for detecting floating objects on a water surface based on multispectral images, characterized in that: The following steps are involved: Step 1: Use a drone equipped with multispectral equipment to inspect the target water surface, align the acquired multispectral video, and perform normalization preprocessing; Step 2: Use the semantic segmentation network based on the EfficientViT architecture to process the image preprocessed in step 1 to obtain the semantic segmentation result. The semantic segmentation result includes floating object pixels. Multiple adjacent floating object pixels constitute a floating object mask to complete the detection of floating objects on the water surface. Step 3: Convert the floating object mask of the semantic segmentation result into the minimum bounding rectangle, and then track the floating objects based on Kalman filtering and Hungarian algorithm to obtain the unique number of each floating object; Step 4: Calculate the location information of each floating object based on the unique number of each floating object, the attitude information of the drone gimbal, and the GPS information; Step 5: Convert the floating object pixels in step 2 into the real area of ​​the floating object, and issue an early warning according to the real area of ​​the floating object. If the real area of ​​the floating object is greater than the threshold, salvage the floating object based on the location information of the floating object obtained in step 4.

2. The method for detecting floating objects on a water surface based on multispectral images according to claim 1, characterized in that: The step 1 is specifically as follows: the multispectral video includes visible light images, red edge images, and near infrared images, and the collected visible light images, red edge images, and near infrared images are aligned and normalized for preprocessing; the alignment method is as follows: bilinear interpolation is used to scale the sizes of the three images to be consistent with the data format and type accepted by the semantic segmentation network based on the EfficientViT architecture in step 2; The normalized calculation method is as follows: Where: x′ is the normalized data; x is the original data; x max is the maximum value; x min is the minimum value.

3. The method for detecting floating objects on a water surface based on multispectral images according to claim 1, characterized in that: The semantic segmentation network based on the EfficientViT architecture in step 2 is composed of an input backbone, four stages and a segmentation head in sequence. The image data preprocessed in step 1 is input into the input backbone, and then passes through stages 1, 2, 3 and 4 in sequence. The output of stage 2 is P2, the output of stage 3 is P3, and the output of stage 4 is P4. The upsampling operation is completed using bilinear interpolation, and P2, P3 and P4 are fused through addition. Finally, the detection of floating objects on the water surface is completed based on the segmentation head to obtain the semantic segmentation result and the mask of the floating objects. At the same time, the EfficientViT module is inserted in stages 3 and 4, and the ReLU linear attention is used to realize the global receptive field and multi-scale learning.

4. The method for detecting floating objects on the water surface based on multispectral images according to claim 3 is characterized in that: The EfficientViT module is specifically as follows: data is input into linear unit 1, the output of linear unit 1 is then input into QKV, the output of QKV is respectively input into depthwise separable convolution and ReLU linear attention 1, the output of depthwise separable convolution is input into 1×1 grouped convolution, the output of 1×1 grouped convolution is input into ReLU linear attention 2, the outputs of ReLU linear attention 1 and ReLU linear attention 2 are additively fused, the fused data is input into linear unit 2, the output of linear unit 2 enters FFN layer + deep convolution network for output, and the EfficientViT module inserts a deep convolution network in each FFN layer, where ReLU linear attention captures contextual information, and FFN and deep convolution network capture local information.

5. The method for detecting floating objects on the water surface based on multispectral images according to claim 4 is characterized in that: The step 3 is specifically as follows: according to the mask of the floating object output in step 2, the mask image is traversed to determine whether the 8 surrounding pixels are still mask pixels, to determine all connected domains in the image, and to find the minimum value x of all points in the connected domain on the x-axis. min and the maximum value x max and the minimum value y on the y-axis min and the maximum value y max , then the coordinates of the upper left corner of the minimum bounding rectangle of the floating object area are (x min ,y min ), the coordinates of the lower right corner are (x max ,y max ), get the floating object pixels, and then complete the tracking of floating objects based on Kalman filtering and Hungarian algorithm, and assign a unique target number to each target.

6. The method for detecting floating objects on the water surface based on multispectral images according to claim 5, characterized in that: The step 4 is specifically as follows: assigning a unique target number to each target, and calculating the specific latitude and longitude coordinates of the floating object based on the aircraft's attitude information, position information, altitude information, gimbal attitude information, and the position of the floating object in the image when the target is discovered. The calculation process is as follows: Step 4-1: First, convert the diagonal field of view provided by the multispectral device into horizontal and vertical field of view. Where: W and H represent the width and height of the image, pixels; FOV D is the diagonal field of view, degrees; FOV H is the horizontal field of view, degree; FOV V is the vertical field of view, degree; Step 4-2: Calculate the deviation between the center point of the minimum circumscribed rectangle of the floating object and the center point of the image. The specific calculation formula is as follows: Δx pix =x obj -(W / 2) Δy pix =y obj -(H / 2) Where: x obj ,y obj The coordinates of the center point of the minimum circumscribed rectangle of the floating object on the water surface in the pixel coordinate system, where the pixel coordinates are the upper left corner of the image as the origin of the coordinate system, and downward and rightward are positive directions; Δx pix , Δy pix is the pixel deviation of the floating object on the water surface relative to the center point of the image; Step 4-3: Calculate the object distance. The specific formula is as follows: dis=height·cos(90-α-β) Where: dis is the object distance, meters; height is the vertical height of the drone from the water surface, meters; α is the pitch angle of the drone, degrees; β is the pitch angle of the pod, degrees; Step 4-4: Calculate the actual deviation of the floating object on the water surface relative to the intersection of the optical axis and the water surface; <h2 style=";text-align:left;direction:ltr">Δx = 2△x<h2 style=";text-align:left;direction:ltr"> pix <h2 style=";text-align:left;direction:ltr"> / W·dis·tan(FOV<h2 style=";text-align:left;direction:ltr"> H <h2 style=";text-align:left;direction:ltr"> / 2) Δy=2Δy pix / H·dis·tan(FOV V / 2) Step 4-5: Calculate the deviation of the floating object on the water surface relative to the vertical projection point of the drone on the water surface, with due north as 0 degrees; θ=arctan(Δx / (dis·cos(α+β)+Δy)+yaw uav +wow pod Where: yaw uav is the yaw angle of the UAV, degrees; yaw pod is the yaw angle of the pod on the drone, degree; dis obj is the deviation of the floating object relative to the vertical projection point of the UAV on the water surface, meter; θ is the azimuth of the floating object on the water surface relative to the true north direction, degrees; Step 4-6: Convert the coordinates of floating objects on the water surface into longitude and latitude; The latitude and longitude position of the UAV can be directly obtained through GPS. In step 4-5, the deviation of the floating object relative to the vertical projection point of the UAV on the water surface has been calculated. obj And the azimuth angle θ, the longitude and latitude coordinates of the floating object on the water surface are calculated by the following formula; Lon obj =Lon uav +arctan(dis obj ·sinθ / ARC) Lat obj =Lat uav +arctan(dis obj ·cosθ / ARC) Where: Lon obj and Lat obj is the latitude and longitude of floating objects on the water surface, degrees; Lon uav and Lat uav is the latitude and longitude of the drone, degrees; dis obj is the deviation of the floating object relative to the vertical projection point of the UAV on the water surface, meter; θ is the azimuth of the floating object on the water surface relative to the true north direction, degrees; ARC is the radius of the earth, which is 6371.393*1000 meters.

7. The method for detecting floating objects on the water surface based on multispectral images according to claim 1, characterized in that: The step 5 specifically includes: counting the pixels of each floating object tracked in step 2, and converting them into the real area of ​​the floating object according to the camera parameters, and issuing warnings according to the calculated real area of ​​the floating object. If the real area of ​​the floating object is greater than the threshold, the floating object is salvaged according to the position information of the floating object obtained in step 4.

8. A water surface floating object detection device based on multispectral images, characterized in that: include: Acquisition module: Use drones equipped with multispectral equipment to acquire multispectral videos; Preprocessing module: align and normalize the acquired multispectral video; Processing module: Using the semantic segmentation network based on the EfficientViT architecture, we can obtain floating object pixels and detect floating objects on the water surface. We can then convert the floating object mask into a minimum bounding rectangle, and then track the floating objects based on the Kalman filter and Hungarian algorithm to obtain a unique number for each floating object. Positioning module: Calculate the location information of each floating object based on the unique number of each floating object, the attitude information of the drone gimbal, and the GPS information; Early warning module: converts the floating object pixels obtained by the processing module into the actual area of ​​the floating object, and issues early warnings according to the actual area of ​​the floating object. If the actual area of ​​the floating object is greater than the threshold, the location information of the floating object obtained by the positioning module is used for salvage.

9. A storage medium, characterized in that: The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the steps included in the method for detecting floating objects on the water surface based on multispectral images as described in any one of claims 1 to 7.

Images