Water surface floating object detection method and device based on multispectral image and storage medium

Abstract

The application discloses a water surface floating object detection method and device based on a multispectral image and a storage medium, and comprises the following steps: 1, using a UAV to carry a multispectral device to patrol a target water surface, and pre-processing the obtained multispectral video; 2, using a semantic segmentation network based on an EfficientViT architecture to process the image pre-processed in step 1 to obtain a semantic segmentation result, obtain floating object pixels, and complete the detection of the water surface floating object; 3, converting the floating object mask of the semantic segmentation result into a minimum circumscribed rectangle, tracking the floating object, and obtaining a unique number of each floating object; 4, according to the unique number of each floating object, the attitude information of the UAV gimbal and the GPS information, calculating the position information of each floating object; and 5, if the real area of the floating object is greater than a threshold value, performing salvage processing through the position information of the floating object obtained in step 4.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of resource and environment detection, and particularly relates to a water surface floating object detection method and device based on a multispectral image and a storage medium. BACKGROUND

[0002] Protecting water environment is of great importance to maintaining ecological balance and human health. In China, the protection of water environment has always been the top priority of environmental protection. The branches, leaves and plastics floating on the water surface not only cause negative visual effects, but also may cause water quality deterioration. A large amount of floating objects may even interfere with the normal operation of the reservoir, especially for power stations. Floating objects may affect the monitoring of the water level of the reservoir, the structure of the dam and the operation of the hydraulic equipment. Therefore, real-time monitoring of floating objects helps to improve the operation efficiency of the reservoir and reduce potential damage. However, due to the limitations of equipment and technology, the current floating object inspection mainly relies on manpower, which is not only low in efficiency, but also long in inspection period.

[0003] At present, with the rapid development of unmanned intelligent technology, unmanned aerial vehicles and unmanned ships have begun to be applied in water surface floating object inspection. Unmanned aerial vehicles can seamlessly switch between land and water, while unmanned ships are mainly suitable for water areas. This makes unmanned aerial vehicles more suitable for performing diversified tasks between different terrains. Various devices can be carried by unmanned aerial vehicles. At present, in the task of water surface floating object inspection, the main device carried is a visible light camera. The patent CN115019122A detects water surface floating objects based on an ordinary visible light camera.

[0004] However, the information that can be captured by the visible light camera is limited, which leads to false detection in areas such as water ripples and water reflections when performing detection tasks. The multispectral camera adopts the multi-band imaging principle in engineering technology, which records the spectral information of specific wavelengths to accurately capture the spectral characteristics of objects such as color, humidity and temperature. Compared with traditional cameras, multispectral cameras have higher spectral resolution and richer color restoration capabilities. It can capture subtle changes that the human eye cannot perceive, and can provide richer and more accurate image information to lay the foundation for improving the accuracy of water surface floating object detection.

[0005] However, there is no patent or journal to disclose how to apply multispectral cameras and unmanned aerial vehicles to water surface floating object detection. Therefore, in order to effectively improve the efficiency and accuracy of water surface floating object detection, the present application provides a water surface floating object detection method based on a multispectral image, which uses an unmanned aerial vehicle to carry a multispectral camera to realize the detection and tracking of water surface floating objects. SUMMARY

[0006] The present application aims to overcome the deficiencies of the prior art, and provides a water surface floating object detection method, device and storage medium based on multispectral images, which improves the accuracy of water surface floating object detection and realizes tracking and positioning of floating objects.

[0007] To solve the technical problem, the technical method of the present application is: a water surface floating object detection method based on multispectral images, comprising the following steps:

[0008] Step 1: using a drone to carry a multispectral device to patrol the target water surface, aligning and normalizing the obtained multispectral video for pretreatment;

[0009] Step 2: using a semantic segmentation network based on EfficientViT architecture to process the image pretreated in step 1 to obtain a semantic segmentation result, the semantic segmentation result including floating object pixels, a plurality of adjacent floating object pixels constituting a floating object mask, and completing detection of the water surface floating object;

[0010] Step 3: converting the floating object mask of the semantic segmentation result into a minimum bounding rectangle, and then completing tracking of the floating object based on Kalman filtering and the Hungarian algorithm to obtain a unique number of each floating object;

[0011] Step 4: calculating the position information of each floating object according to the unique number of each floating object, the attitude information of the drone gimbal and the GPS information;

[0012] Step 5: converting the floating object pixels of step 2 into a real area of the floating object, and prewarning according to the classification of the real area of the floating object, and if the real area of the floating object is greater than a threshold, performing salvage processing according to the position information of the floating object obtained in step 4.

[0013] Preferably, step 1 is specifically: the multispectral video includes visible light images, red edge images and near-infrared images, and the collected visible light images, red edge images and near-infrared images are aligned and normalized for pretreatment;

[0014] The alignment method is: using bilinear interpolation to scale the sizes of the three kinds of images to be consistent with the data format and type accepted by the semantic segmentation network based on EfficientViT architecture in step 2;

[0015] The normalization calculation method is as follows:

[0016]

[0017] In the formula:

[0018] x' is the normalized data;

[0019] x is the original data;

[0020] x max is the maximum value;

[0021] x min is the minimum value.

[0022] Preferably, the semantic segmentation network based on the EfficientViT architecture in step 2 is composed of an input stem, four stages, and a segmentation head in sequence. The image data preprocessed in step 1 is input into the input stem, and then sequentially passes through stage 1, stage 2, stage 3, and stage 4. The output of stage 2 is P2, the output of stage 3 is P3, and the output of stage 4 is P4. Bilinear interpolation is used to complete the upsampling operation, P2, P3, and P4 are fused by addition, and finally the detection of the water surface floating object is completed based on the segmentation head to obtain the semantic segmentation result and the floating object mask. Meanwhile, the EfficientViT module is inserted in stages 3 and 4 to realize global receptive field and multi-scale learning using ReLU linear attention.

[0023] Preferably, the EfficientViT module specifically comprises: a data input linear unit 1, which is followed by inputting QKV. The output of QKV is input into a depth separable convolution and a ReLU linear attention 1, respectively. The output of the depth separable convolution is input into a 1x1 grouped convolution. The output of the 1x1 grouped convolution is input into a ReLU linear attention 2. The outputs of the ReLU linear attention 1 and the ReLU linear attention 2 are fused by addition. The fused data is input into a linear unit 2, and the output of the linear unit 2 is input into an FFN layer + a depth convolution network for output. The EfficientViT module inserts a depth convolution network in each FFN layer, wherein the ReLU linear attention captures context information, and the FFN and the depth convolution network capture local information.

[0024] Preferably, step 3 specifically comprises: according to the mask of the floating object output by step 2, traversing the mask image to determine whether the surrounding 8 pixels are still mask pixels, to judge all connected domains in the image, and find the minimum value x min and the maximum value x max of all points in the connected domain on the x-axis, and the minimum value y min and the maximum value y max of all points on the y-axis. If the above conditions are met, the top-left corner coordinates of the minimum bounding rectangle of the floating object region are (x min , y min ), and the bottom-right corner coordinates are (x max , y max ). The floating object pixels are obtained, and then the Kalman filter and the Hungarian algorithm are used to complete the tracking of the floating object. A unique target number is assigned to each target.

[0025] Preferably, the step 4 is specifically: assigning a unique target number to each target, and calculating the specific latitude and longitude coordinates of the floating object based on the attitude information, position information, height information, gimbal attitude information of the aircraft when the target is found, and the position of the floating object in the image. The calculation process is as follows:

[0026] Step 4-1: First, convert the diagonal field of view angle provided by the multispectral device into horizontal and vertical field of view angles,

[0027]

[0028]

[0029] In the formula:

[0030] W and H represent the width and height of the image, in pixels;

[0031] FOV D is the diagonal field of view angle, in degrees;

[0032] FOV H is the horizontal field of view angle, in degrees;

[0033] FOV V is the vertical field of view angle, in degrees;

[0034] Step 4-2: Calculate the deviation between the center point of the smallest circumscribed rectangle of the floating object and the center point of the image, and the specific calculation formula is as follows:

[0035] Δx pix = x obj -(W / 2)

[0036] Δy pix =y obj -(H / 2)

[0037] In the formula:

[0038] x obj , y obj is the center point of the smallest circumscribed rectangle of the floating object on the pixel coordinate system, where the upper left corner of the image is the origin of the coordinate system, and the downward and rightward directions are positive directions;

[0039] Δx pix , Δy pix is the pixel deviation of the floating object relative to the center point of the image;

[0040] Step 4-3: Calculate the object distance, and the specific formula is as follows:

[0041] dis = height · cos (90-α-β)

[0042] In the formula:

[0043] dis is the distance from the UAV to the water surface, in meters;

[0044] height is the vertical height of the UAV from the water surface, in meters;

[0045] α is the pitch angle of the UAV, in degrees;

[0046] β is the pitch angle of the pod, in degrees;

[0047] Step 4-4: Calculate the actual deviation of the water surface floating object relative to the intersection of the optical axis and the water surface;

[0048] Δx = 2Δx pix / W·dis·tan(FOV H / 2)

[0049] Δy = 2Δy pix / H·dis·tan(FOV V / 2)

[0050] Step 4-5: Calculate the deviation of the water surface floating object relative to the vertical projection point of the UAV on the water surface, with 0 degrees as the north;

[0051]

[0052] θ = arctan(Δx / (dis·cos(α+β)+Δy)+yaw uav +yaw pod

[0053] In the formula:

[0054] yaw uav is the yaw angle of the UAV, in degrees;

[0055] yaw pod is the yaw angle of the pod on the UAV, in degrees;

[0056] dis obj is the deviation of the floating object relative to the vertical projection point of the UAV on the water surface, in meters;

[0057] θ is the azimuth angle of the water surface floating object relative to the north, in degrees;

[0058] Step 4-6: Convert the water surface floating object coordinates to latitude and longitude;

[0059] The latitude and longitude position of the UAV can be directly obtained by GPS, and the deviation dis obj of the floating object relative to the vertical projection point of the UAV on the water surface and the azimuth angle θ have been calculated in step 4-5. The latitude and longitude coordinates of the water surface floating object are calculated by the following formula:

[0060] Lonobj = Lon uav + arctan(dis obj · sinθ / ARC)

[0061] Lat obj = Lat uav + arctan(dis obj · cosθ / ARC)

[0062] In the formula:

[0063] Lon obj and Lat obj are the longitude and latitude of the water surface floating object, degrees;

[0064] Lon uav and Lat uav are the longitude and latitude of the unmanned aerial vehicle, degrees;

[0065] dis obj is the deviation of the floating object relative to the vertical projection point of the unmanned aerial vehicle on the water surface, meters;

[0066] θ is the azimuth angle of the water surface floating object relative to the north direction, degrees;

[0067] ARC is the radius of the earth, which is 6371.393*1000 meters.

[0068] Preferably, step 5 is specifically: statistics are performed on each floating object pixel tracked in step 2, and the floating object real area is converted according to the camera parameters, and a warning is given according to the calculated floating object real area by level, if the floating object real area is greater than a threshold, the position information of the floating object obtained through step 4 is used for salvage processing.

[0069] Preferably, a water surface floating object detection device based on multi-spectral images comprises:

[0070] An acquisition module: a multi-spectral device is carried by an unmanned aerial vehicle to acquire a multi-spectral video;

[0071] A preprocessing module: the acquired multi-spectral video is aligned and normalized for preprocessing;

[0072] A processing module: a semantic segmentation network based on an EfficientViT architecture is used to obtain floating object pixels and complete the detection of water surface floating objects; the floating object mask is converted into a minimum circumscribed rectangle, and then the Kalman filtering and Hungarian algorithm are used to complete the tracking of the floating object to obtain a unique number of each floating object;

[0073] A positioning module: according to the unique number of each floating object, the attitude information of the unmanned aerial vehicle gimbal, and the GPS information, the position information of each floating object is calculated.

[0074] The early warning module converts the floating object pixels obtained by the processing module into a floating object real area, and performs early warning according to the floating object real area level, and if the floating object real area is greater than a threshold value, the position information of the floating object obtained by the positioning module is used for salvage processing.

[0075] Preferably, a storage medium stores computer executable instructions for causing a computer to execute the steps of the water surface floating object detection method based on a multispectral image.

[0076] Compared with the prior art, the advantages of the present application are that:

[0077] (1) The present application discloses a water surface floating object detection method based on a multispectral image, which realizes the functions of floating object detection, tracking, positioning and early warning in water areas by performing feature extraction, analysis and other processing on the multispectral image.

[0078] (2) The present application adopts a semantic segmentation network based on the EfficientViT architecture, and realizes global receptive field and multi-scale learning through ReLU linear attention, captures context information, and improves segmentation capability through FFN and deep convolution to capture local information, which is more suitable for extracting water surface floating objects.

[0079] (3) The water surface floating object detection method based on a multispectral video of the present application can effectively improve the inspection efficiency of water surface floating objects. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 The use schematic diagram of the water surface floating object detection method based on a multispectral image of the present application;

[0081] Figure 2 The flowchart of the water surface floating object detection method based on a multispectral image of the present application;

[0082] Figure 3 The semantic segmentation network structure schematic diagram of the EfficientViT architecture of the present application;

[0083] Figure 4 The structural block diagram of the water surface floating object detection device based on a multispectral image of the present application.

[0084] Explanation of reference signs:

[0085] 1, unmanned aerial vehicle, 2, multispectral equipment, 3, floating object, 4, water surface. DETAILED DESCRIPTION

[0086] The specific embodiments of the present application will be described below in conjunction with examples:

[0087] It should be noted that the content shown in the drawings of the present application is only used to cooperate with the principles, features and content disclosed in the specification for understanding and reading by those skilled in the art, and is not used to limit the conditions that can be implemented by the present application. Without affecting the effects and purposes that can be achieved by the present application, it should still fall within the scope of the technology disclosed by the present application.

[0088] As shown in Figures 1-4 The present application discloses a water surface floating object detection method based on multispectral images, comprising the following steps:

[0089] Step 1: using a UAV to carry a multispectral device to patrol the target water surface, aligning and normalizing the preprocessing of the obtained multispectral video;

[0090] Step 2: using a semantic segmentation network based on EfficientViT architecture to process the image after preprocessing in step 1 to obtain a semantic segmentation result, the semantic segmentation result including floating object pixels, a plurality of adjacent floating object pixels constituting a floating object mask, and completing the detection of water surface floating objects;

[0091] Semantic segmentation is essentially a pixel-level classification of images, in which each pixel is assigned to a specific class; for the detection of water surface floating objects, a plurality of adjacent pixels belonging to the floating object class constitute a connected domain, which represents a floating object mask;

[0092] Step 3: converting the floating object mask of the semantic segmentation result into a minimum bounding rectangle, and then completing the tracking of the floating object based on Kalman filtering and the Hungarian algorithm to obtain a unique number for each floating object;

[0093] Step 4: calculating the position information of each floating object according to the unique number of each floating object, the attitude information of the UAV gimbal, and the GPS information;

[0094] Step 5: converting the floating object pixels of step 2 into real floating object areas, and warning according to the real floating object areas by grade, if the real floating object area is greater than a threshold, performing salvage processing according to the position information of the floating object obtained in step 4.

[0095] Preferably, step 1 is specifically: the multispectral video includes visible light images, red edge images, and near-infrared images; the collected visible light images, red edge images, and near-infrared images are aligned and normalized for preprocessing.

[0096] The red edge image refers to a region between a red light wave band and a near-infrared wave band in an image spectrum. The alignment is that the sizes of the three images are scaled to be consistent with a data format and type accepted by the semantic segmentation network based on the EfficientViT architecture in step 2 by using bilinear interpolation;

[0097] The normalization calculation method is as follows:

[0098]

[0099] In the formula:

[0100] x' is the normalized data;

[0101] x is the original data;

[0102] x max is the maximum value;

[0103] x min is the minimum value.

[0104] The purpose of normalization is to eliminate the dimensional difference between different features, so that different features have comparability, thereby improving the performance of the semantic segmentation model.

[0105] The purpose of the application is to detect, track, locate and warn water floating objects based on multispectral data obtained by a UAV. The shape and size of water floating objects can be very diverse, including but not limited to debris, leaves, phytoplankton, plastic products, etc., which makes it complex to accurately distinguish floating objects in images. After comprehensive consideration and analysis, the application believes that semantic segmentation is more suitable for extracting water floating objects, and the semantic segmentation network is implemented based on the EfficientViT architecture, as follows:

[0106] As shown in Figure 3 , the semantic segmentation network based on the EfficientViT architecture is composed of an input backbone, four stages and a segmentation head in sequence, the image data after preprocessing in step 1 is input into the input backbone, then sequentially passes through stage 1, stage 2, stage 3 and stage 4, the output of stage 2 is P2, the output of stage 3 is P3, and the output of stage 4 is P4. Bilinear interpolation is used to complete the upsampling operation, P2, P3 and P4 are fused by addition, and finally the detection of water floating objects is completed based on the segmentation head to obtain the semantic segmentation result and obtain the floating object mask; meanwhile, the EfficientViT module is inserted in stage 3 and stage 4, and ReLU linear attention is used to realize global receptive field and multi-scale learning.

[0107] Preferably, the EfficientViT module can improve segmentation capability. Specifically, the EfficientViT module is as follows: data is input to linear unit 1, the output of linear unit 1 is then input to QKV, the output of QKV is then input to depthwise separable convolution and ReLU linear attention 1 respectively, the output of depthwise separable convolution is then input to 1×1 grouped convolution, the output of 1×1 grouped convolution is then input to ReLU linear attention 2, the outputs of ReLU linear attention 1 and ReLU linear attention 2 are added and fused, the fused data is then input to linear unit 2, the output of linear unit 2 is then fed into an FFN layer + deep convolutional network for output. The EfficientViT module inserts a deep convolutional network in each FFN layer, where ReLU linear attention captures contextual information, and FFN and deep convolutional networks capture local information.

[0108] Preferably, step 3 specifically involves: based on the mask of the floating object output in step 2, traversing the mask image to determine whether its surrounding 8 pixels are still mask pixels, thereby identifying all connected components in the image, and finding the minimum value x of all points within the connected component on the x-axis. min and maximum value x max and the minimum value y on the y-axis min and maximum value y max Then the coordinates of the top-left corner of the smallest bounding rectangle of the floating object region are (x... min，ymin The coordinates of the lower right corner are (x max y max The floating object pixels are obtained, and then the floating object is tracked based on Kalman filtering and Hungarian algorithm, and a unique target number is assigned to each target.

[0109] The main objective of target tracking is to assign a unique number to the same target in different views during the period from when the target enters the field of view to when it leaves the field of view. For floating objects on the water surface, the result of a semantic segmentation model (the mask of the floating object) is used, and direct tracking requires extracting the corresponding features, which is quite complex. This invention calculates the minimum bounding rectangle of the floating object region and then tracks the floating object based on Kalman filtering and the Hungarian algorithm.

[0110] The Kalman filter and Hungarian algorithm mentioned are conventional algorithms, and will not be described in detail in this application.

[0111] Preferably, step 4 specifically involves: assigning a unique target number to each target, and calculating the specific latitude and longitude coordinates of the floating object based on the aircraft's attitude information, position information (latitude and longitude coordinates), altitude information (vertical height above the water surface), gimbal attitude information, and the floating object's position in the image when the target is detected. The calculation process is as follows:

[0112] Step 4-1: First, convert the diagonal field of view angle provided by the multispectral device into the horizontal field of view angle and the vertical field of view angle,

[0113]

[0114] In the formula:

[0115] W and H represent the width and height of the image, pixels;

[0116] FOV D is the diagonal field of view angle, degrees;

[0117] FOV H is the horizontal field of view angle, degrees;

[0118] FOV V is the vertical field of view angle, degrees;

[0119] Step 4-2: Calculate the deviation between the center point of the floating object's minimum bounding rectangle and the center point of the image, the specific calculation formula is as follows:

[0120] Δx pix = x obj - (W / 2)

[0121] Δy pix = y obj - (H / 2)

[0122] In the formula:

[0123] x obj , y obj is the center point of the water surface floating object's minimum bounding rectangle in the pixel coordinate system, where the pixel coordinate system has the upper left corner of the image as the origin, and the downward and rightward directions are positive directions;

[0124] Δx pix , Δy pix is the pixel deviation of the water surface floating object relative to the center point of the image;

[0125] Step 4-3: Calculate the object distance, the specific formula is as follows:

[0126] dis = height · cos (90 - α - β)

[0127] In the formula:

[0128] dis is the object distance, meters;

[0129] height is the vertical height of the unmanned aerial vehicle from the water surface, meters;

[0130] α is the pitch angle of the unmanned aerial vehicle, degrees;

[0131] β is the pitch angle of the pod (pitch angle coordinate system: the forward direction of the aircraft is the positive direction, the horizontal is 0 degrees, and downward is positive), degrees;

[0132] Step 4-4: Calculate the actual deviation of the water surface floating object relative to the intersection point of the optical axis and the water surface;

[0133] Δx = 2Δx pix / W·dis·tan(FOV H / 2)

[0134] Δy = 2Δy pix / H·dis·tan(FOV V / 2)

[0135] Step 4-5: Calculate the deviation of the water surface floating object relative to the vertical projection point of the unmanned aerial vehicle on the water surface, with 0 degrees as the north;

[0136]

[0137] θ = arctan(Δx / (dis·cos(α+β)+Δy)+yaw uav +yaw pod

[0138] In the formula:

[0139] yaw uav is the yaw angle of the unmanned aerial vehicle, degrees;

[0140] yaw pod is the yaw angle of the pod on the unmanned aerial vehicle, degrees;

[0141] dis obj is the deviation of the floating object relative to the vertical projection point of the unmanned aerial vehicle on the water surface, meters;

[0142] θ is the azimuth angle of the water surface floating object relative to the north, degrees;

[0143] Step 4-6: Convert the water surface floating object coordinates to latitude and longitude;

[0144] The latitude and longitude position of the unmanned aerial vehicle can be directly obtained by GPS, and the deviation dis obj of the floating object relative to the vertical projection point of the unmanned aerial vehicle on the water surface and the azimuth angle θ have been calculated in step 4-5, and the latitude and longitude coordinates of the water surface floating object are calculated by the following formula;

[0145] Lon obj = Lon uav + arctan(dis obj · sin θ / ARC)

[0146] Lat obj = Latuav + arctan(dis obj * cosθ / ARC)

[0147] In the formula:

[0148] Lon obj and Lat obj are the longitude and latitude of the water surface floating object, degrees;

[0149] Lon uav and Lat uav are the longitude and latitude of the unmanned aerial vehicle, degrees;

[0150] dis obj is the deviation of the floating object relative to the vertical projection point of the unmanned aerial vehicle on the water surface, meters;

[0151] θ is the azimuth angle of the water surface floating object relative to the north direction, degrees;

[0152] ARC is the radius of the earth, taken as 6371.393*1000 meters.

[0153] Preferably, step 5 is specifically: statistics are performed on each floating object pixel tracked in step 2, and the floating object real area is converted according to the camera parameters, a warning is given according to the calculated floating object real area, and if the floating object real area is greater than a threshold, the position information of the floating object obtained through step 4 is used for salvage processing.

[0154] As Figure 4 shown, preferably, a water surface floating object detection device based on multispectral images, comprising: an acquisition module: using a multispectral device carried by an unmanned aerial vehicle to acquire a multispectral video;

[0155] a preprocessing module: aligning and normalizing the acquired multispectral video for preprocessing;

[0156] a processing module: using a semantic segmentation network based on the EfficientViT architecture to obtain floating object pixels and complete the detection of water surface floating objects; converting the floating object mask into a minimum circumscribed rectangle, and then completing the tracking of the floating object based on Kalman filtering and the Hungarian algorithm to obtain a unique number of each floating object;

[0157] a positioning module: calculating the position information of each floating object according to the unique number of each floating object, the attitude information of the unmanned aerial vehicle gimbal, and the GPS information;

[0158] a warning module: converting the floating object pixels obtained by the processing module into a floating object real area, giving a warning according to the classification of the floating object real area, and if the floating object real area is greater than a threshold, using the position information of the floating object obtained by the positioning module for salvage processing.

[0159] Preferably, a storage medium storing computer executable instructions for causing a computer to perform the steps of the water surface floating object detection method based on multispectral images.

[0160] Embodiment 1

[0161] Water surface garbage detection and monitoring using unmanned aerial vehicles

[0162] a) Equipment preparation

[0163] As shown, a multispectral device 2 is installed on an unmanned aerial vehicle 1, which flies over a water surface 4 to conduct a patrol inspection of floating objects 3. Figure 1

[0164] The multispectral device 2 is selected from a multispectral camera: a multispectral camera suitable for the unmanned aerial vehicle 1 is selected, which has the ability to capture multiple spectral bands (such as visible light, near-infrared, red edge, etc.).

[0165] Unmanned aerial vehicle: equipped with a stable flight platform, capable of carrying a multispectral camera and flying stably over the target water area.

[0166] b) Data collection

[0167] Flight planning: according to the range and terrain characteristics of the target water area, plan the flight path of the unmanned aerial vehicle to ensure coverage of the entire water area.

[0168] Flight execution: under the predetermined time and weather conditions, operate the unmanned aerial vehicle to fly along the planned path and use the multispectral camera to collect data. The collected multispectral images should include the water surface and its surrounding environment.

[0169] c) Data preprocessing

[0170] As shown, image correction: align the stitched visible light image and near-infrared, red edge image, and perform normalization preprocessing; Figure 2 Or perform geometric correction and radiometric correction to ensure the spatial and spectral accuracy of the image.

[0171] d) Model training

[0172] Use a semantic segmentation network based on the EfficientViT architecture to complete the detection of water surface floating objects;

[0173] Training dataset: according to historical data and field labeling, construct a training dataset containing various types of water surface floating objects (such as plastic garbage, wood, algae, etc.).

[0174]

[0175] ​​Machine Learning Model: Select a suitable machine learning algorithm (such as support vector machine, random forest, deep learning, etc.) and train the model using the training data set to achieve automatic recognition and classification of water surface floating objects.

[0176] e) Floating object recognition and tracking

[0177] As shown in Figure 2 , recognition: input the pre-processed multispectral image into the trained model for real-time detection and recognition of water surface floating objects.

[0178] Result output: generate detection results containing floating object position, type, area and number.

[0179] Convert the floating object mask of the semantic segmentation result into the minimum bounding rectangle, and then complete the tracking of the floating object based on Kalman filtering and Hungarian algorithm to obtain the unique number of each floating object.

[0180] f) Floating object positioning

[0181] As shown in Figure 2 , according to the attitude information of the unmanned aerial vehicle gimbal and the GPS information, the position information of the floating object is calculated, as described in the formula above.

[0182] g) Early warning

[0183] As shown in Figure 2 , according to the judgment of whether the floating object area is greater than the alarm threshold, it is judged whether early warning is needed.

[0184] h) Application and feedback

[0185] Monitoring system: integrate the detection results into the water area monitoring system to realize automatic monitoring and early warning of water surface floating objects.

[0186] Cleaning action: according to the detection results, make water surface cleaning plan, dispatch cleaning ships and personnel to clean floating objects.

[0187] Effect evaluation: regularly evaluate the monitoring and cleaning effect, optimize the unmanned aerial vehicle flight path and detection model, and improve the detection accuracy and efficiency.

[0188] The principle of the present application is as follows:

[0189] As shown in Figures 1-4As shown, the application discloses a water surface floating object detection method, device and storage medium based on multispectral images, and the water surface floating object detection method based on multispectral video mainly includes the following four parts: water surface floating object detection, tracking, positioning and early warning, the water surface floating object detection refers to using a deep learning method to find a floating object target from water image collected by a UAV, however, due to the problems of reflection, ripples, reflection, etc., it is easy to cause false detection when detecting in ordinary RGB images, and the multispectral camera has higher spectral resolution and richer color restoration capability, which can provide richer and more accurate image information, thereby effectively reducing the false detection of water surface floating objects in these complex scenes; the water surface floating object tracking is realized by converting the detected floating object mask into a minimum bounding rectangle, and then based on Kalman filtering and Hungarian algorithm; the water surface floating object positioning is realized by calculating the latitude and longitude position of the floating object through the position, attitude, gimbal attitude and flight height of the current UAV, and completing the positioning of the detected water surface floating object; finally, whether to perform early warning is determined according to whether the area of the floating object is greater than the set threshold.

[0190] Compared with visible light data, the multispectral data is more informative, especially the water surface itself has high sensitivity to near-infrared and red edge data. Therefore, it is easier and more accurate to extract floating objects from the water surface by using multispectral data. In addition, the application combines deep learning to segment the floating object, which is more adaptable and more accurate than traditional threshold segmentation and manual feature extraction segmentation.

[0191] The application discloses a water surface floating object detection method based on multispectral images, which realizes the functions of floating object detection, tracking, positioning and early warning in water area by performing feature extraction, analysis and other processing on multispectral images, and can effectively improve the difficulty of water surface floating object detection which is easily disturbed by water ripples, reflection, tree and building reflection, etc., and improve the accuracy and robustness of water surface floating object detection.

[0192] The application adopts a semantic segmentation network based on the EfficientViT architecture, and realizes global receptive field and multi-scale learning through ReLU linear attention, captures context information, and FFN and deep convolution capture local information, thereby improving the segmentation capability and being more suitable for extracting water surface floating objects.

[0193] Through the water surface floating object detection method based on multispectral video, the inspection efficiency of the water surface floating object can be effectively improved. Of course, there are still some improvements to be made for this method, for example, in order to improve the stability of water surface floating object tracking, the time of image acquisition and the route of UAV flight need to be limited, which will put additional requirements on the image acquisition personnel, and optimization can be tried at the algorithm level to reduce the acquisition requirements.

[0194] The preferred embodiments of the present application have been described in detail above, but the present application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

[0195] Many other changes and modifications can be made within the scope of the present application without departing from the spirit thereof. It must be understood that all matter connected with the essence of the present application is intended to be embraced by the following claims.

Claims

1. A method for detecting floating objects on the water surface based on multispectral images, characterized in that, Includes the following steps: Step 1: Use a drone equipped with a multispectral device to inspect the target water surface, align the acquired multispectral videos, and perform normalization preprocessing; Step 2: The image preprocessed in Step 1 is processed using a semantic segmentation network based on the EfficientViT architecture to obtain semantic segmentation results. The semantic segmentation results include floating object pixels. Multiple adjacent floating object pixels constitute a floating object mask to complete the detection of floating objects on the water surface. Step 3: Convert the floating object mask of the semantic segmentation result into the minimum bounding rectangle, and then track the floating objects based on Kalman filtering and Hungarian algorithm to obtain the unique number of each floating object; Based on the floating object mask output in step 2, traverse the mask image to determine if its surrounding 8 pixels are still mask pixels, thereby identifying all connected components in the image and finding the minimum x-value of all points within each connected component on the x-axis. min and maximum value x max and the minimum value y on the y-axis min and maximum value y max Then the coordinates of the top-left corner of the smallest bounding rectangle of the floating object region are (x... min , ymin The coordinates of the lower right corner are (x max ,y max The floating object pixels are obtained, and then the floating object is tracked based on Kalman filtering and Hungarian algorithm. Each target is assigned a unique target number. Step 4: Calculate the position information of each floating object based on its unique number, the attitude information of the drone gimbal, and the GPS information. Step 4-1: First, convert the diagonal field of view provided by the multispectral equipment into a horizontal and a vertical field of view. In the formula: W and H Represents the width and height of the image in pixels; FOV D The diagonal field of view is in degrees. FOV H The horizontal field of view is in degrees. FOV V The vertical field of view is in degrees; Step 4-2: Calculate the deviation between the center point of the smallest bounding rectangle of the floating object and the center point of the image. The specific calculation formula is as follows: In the formula: The coordinates of the center point of the smallest bounding rectangle of the floating object on the water surface are in the pixel coordinate system, where the top left corner of the image is the origin of the coordinate system, and downward and rightward are the positive directions respectively; This represents the pixel deviation of floating objects on the water surface relative to the image center point. Step 4-3: Calculate the object distance. The specific formula is as follows: In the formula: dis is the object distance, in meters; The vertical height of the drone above the water surface, in meters; The pitch angle of the drone is in degrees; The pitch angle of the pod, in degrees; Step 4-4: Calculate the actual deviation of the floating object relative to the intersection of the optical axis and the water surface; Steps 4-5: Calculate the deviation of floating objects on the water surface relative to the vertical projection point of the UAV on the water surface, with true north as 0 degrees; In the formula: The yaw angle of the drone is in degrees; The yaw angle of the drone's pod, in degrees; The deviation of the floating object from the vertical projection point of the UAV on the water surface is expressed in meters. The azimuth angle of a floating object relative to true north, in degrees; Steps 4-6: Convert the coordinates of floating objects on the water surface into latitude and longitude; The latitude and longitude of the drone were obtained directly via GPS. In steps 4-5, the deviation of the floating object relative to the vertical projection point of the drone on the water surface was calculated. and azimuth The latitude and longitude coordinates of floating objects on the water surface can be calculated using the following formula; In the formula: The coordinates of the floating object on the water surface are latitude and longitude, in degrees; The latitude and longitude of the drone, in degrees; The deviation of the floating object from the vertical projection point of the UAV on the water surface is expressed in meters. The azimuth angle of a floating object relative to true north, in degrees; Let be the Earth's radius, and take . rice; Step 5: Convert the floating object pixels from Step 2 into the actual floating object area, and issue warnings based on the actual floating object area. If the actual floating object area is greater than the threshold, retrieve the floating object using the location information obtained in Step 4.

2. The method for detecting floating objects on the water surface based on multispectral images according to claim 1, characterized in that, Step 1 specifically involves: the multispectral video includes visible light images, red-edge images, and near-infrared images; the acquired visible light images, red-edge images, and near-infrared images are aligned and normalized in a preprocessing manner. The alignment method is as follows: using bilinear interpolation to scale the three image sizes to be consistent with the data format and type accepted by the semantic segmentation network based on the EfficientViT architecture in step 2; The normalization calculation method is as follows: In the formula: The data is after normalization; This is the original data; It is the maximum value; It is the minimum value.

3. The method for detecting floating objects on the water surface based on multispectral images according to claim 1, characterized in that, In step 2, the semantic segmentation network based on the EfficientViT architecture consists of an input backbone, four stages, and a segmentation head. The image data preprocessed in step 1 is input into the input backbone, and then passes through stages 1, 2, 3, and 4 in sequence. The output of stage 2 is P2, the output of stage 3 is P3, and the output of stage 4 is P4. Bilinear interpolation is used to perform upsampling. P2, P3, and P4 are fused by addition. Finally, the segmentation head is used to detect floating objects on the water surface to obtain the semantic segmentation result and the mask of the floating objects. At the same time, EfficientViT modules are inserted in stages 3 and 4, and ReLU linear attention is used to achieve global receptive field and multi-scale learning.

4. The method for detecting floating objects on the water surface based on multispectral images according to claim 3, characterized in that, The EfficientViT module is specifically as follows: data is input to linear unit 1, the output of linear unit 1 is then input to QKV, the output of QKV is then input to depthwise separable convolution and ReLU linear attention 1 respectively, the output of depthwise separable convolution is then input to 1×1 grouped convolution, the output of 1×1 grouped convolution is then input to ReLU linear attention 2, the outputs of ReLU linear attention 1 and ReLU linear attention 2 are added and fused, the fused data is then input to linear unit 2, the output of linear unit 2 is then fed into an FFN layer + a deep convolutional network for output, the EfficientViT module inserts a deep convolutional network into each FFN layer, where ReLU linear attention captures contextual information, and FFN and deep convolutional networks capture local information.

5. The method for detecting floating objects on the water surface based on multispectral images according to claim 1, characterized in that, Step 5 specifically involves: statistically analyzing each floating object pixel tracked in step 2, converting it into the actual area of ​​the floating object based on camera parameters, issuing warnings based on the calculated actual area of ​​the floating object at different levels, and retrieving the floating object if its actual area is greater than a threshold, using the location information of the floating object obtained in step 4.

6. A device for detecting floating objects on the water surface based on multispectral images, characterized in that, For implementation of the multispectral image-based method for detecting floating objects on the water surface according to any one of claims 1 to 5, the floating object detection device comprises: Acquisition module: Utilizes a drone equipped with a multispectral device to acquire multispectral video; Preprocessing module: Aligns and normalizes the acquired multispectral video. Processing module: Utilizes a semantic segmentation network based on the EfficientViT architecture to obtain floating object pixels and complete the detection of floating objects on the water surface; transforms the floating object mask into the minimum bounding rectangle, and then completes the tracking of floating objects based on Kalman filtering and Hungarian algorithm to obtain the unique number of each floating object; Positioning module: Calculates the position information of each floating object based on its unique number, the attitude information of the drone gimbal, and GPS information; Early warning module: Converts the floating object pixels obtained by the processing module into the actual area of ​​the floating object, and issues early warnings based on the actual area of ​​the floating object. If the actual area of ​​the floating object is greater than the threshold, it uses the location information of the floating object obtained by the positioning module to carry out salvage processing.

7. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which are used to cause a computer to perform the steps included in the method for detecting floating objects on the water surface based on multispectral images as described in any one of claims 1 to 5.

Images