A method for drone water monitoring and cruising based on machine vision

Abstract

The present invention discloses a method for drone water area monitoring and cruising based on machine vision, comprising the following steps: 1. Collecting water sample images using a drone; 2. Preprocessing water sample images; 3. Detecting the edge of images; 4. Sorting and screening target areas to obtain edge images of water areas; 5. Obtaining edge convex hull points and performing coordinate system conversion; 6. Setting water area cruising routes. The method of the present invention has low cost, high accuracy, and the function of arbitrarily adding target cruising nodes, providing a new drone cruising monitoring solution for water body monitoring in large water areas. The cruising method of the present invention, in conjunction with a water body detection model, can well achieve the task of real-time monitoring of a certain water area.

Description

Technical Field

[0001] The present invention relates to the application of machine vision in the field of unmanned aerial vehicle (UAV) water cruise, and in particular to a method for unmanned aerial vehicle (UAV) water cruise monitoring based on machine vision. Background Art

[0002] With the rapid advancement of modernization, a large number of water bodies have been polluted. In order to better monitor the changes in water quality in large areas of water in real time, it is necessary to conduct real-time monitoring of water quality in large areas of water.

[0003] Traditional water body detection mainly includes dilution multiple method, platinum cobalt colorimetry (i.e. visual colorimetry), etc. Both methods are visually inspected by human eyes, which are highly subjective, with large errors and cumbersome operations. Moreover, the judgment of color is different under different physiological conditions. The spectrophotometry method that has emerged in recent years mainly uses the absorption peak area of ​​the absorption spectrum of the characteristic wavelength as the measurement standard of chromaticity to detect the chromaticity of water bodies. Compared with traditional methods, it has significantly improved accuracy, is time-saving, and has high efficiency, which can avoid the influence of visual differences on the results. However, it also requires the collection of water samples and the configuration of standard chromaticity solutions, which requires human participation and the procedure is relatively complicated. And it cannot monitor the water quality of large areas of water in real time. With the development of deep learning, some people have proposed a water quality detection method based on drone images. This method puts higher requirements on the automated cruising of drones. Summary of the invention

[0004] In response to the demand for drones to cruise in large areas of water, the present invention provides a drone cruising method based on machine vision. First, a drone is used to collect images of the edge of the water area to obtain water area information, and a machine vision method is used to perform edge detection on the acquired images. The convex hull nodes of the edge of the water area are obtained by convex hull calculation in machine vision, and the longitude and latitude information of each convex hull node in the actual world coordinate system is obtained by coordinate system conversion. The cruise route is designed through the convex hull nodes on the map, thereby realizing the drone's water area monitoring cruise function. The method of the present invention is suitable for line patrol navigation of drones in the process of water quality monitoring in large areas of water.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] Step 1: Use drones to collect water area images

[0007] In the target waters, set the drone height and use the drone to collect photos of the monitored waters, try to capture images of the edge of the waters, and ensure that the drone's height is consistent. The captured images need to ensure that the water area is the main content of the image. The collected images are transmitted to the computer terminal in real time.

[0008] Step 2: Water area image preprocessing

[0009] The collected water sample image is converted from a color image to a grayscale image, and the converted grayscale image is processed by histogram averaging to obtain a color-adjusted image, thereby avoiding problems such as unclear key pixels of the image due to various factors such as external light intensity.

[0010] Step 3: Perform edge detection on the image

[0011] Use the Canny operator to perform edge detection on the color-adjusted image obtained in step 2, and obtain the water edge of the corresponding water area through edge detection. This method can remove interference pixels such as trees and lanes in the image that interfere with mapping. The water edge image obtained at this time is not obvious, and the image after edge detection needs to be further processed by morphological dilation and Gaussian filtering to connect the edge boundaries in the image to form a closed edge and increase the width and smoothness of the edge boundaries;

[0012] Step 4: Sorting and filtering target areas

[0013] Since there are many interfering objects and some interfering edges in the captured image, the water area must be the largest area in the captured image, so all the acquired closed image edges are sorted by edge length. The perimeter of all closed edges in the image processed in step 3 is calculated by opencv to obtain the perimeter length results corresponding to each closed edge, and then the perimeter of all closed edges is sorted from large to small by the sort() function provided by opencv. The closed edge with the largest perimeter is the closed edge of the water area.

[0014] For the closed edge with the maximum perimeter, the convex hull algorithm is used to calculate the convex hull of the closed edge image, and only the closed edge area within the convex hull is retained as white, and other areas are all filled with black, so as to obtain the edge image of the water area.

[0015] Step 5: Get the edge convex hull points and perform coordinate system conversion

[0016] Get the convex hull nodes of the entire water area edge, calculate the corresponding coordinate position of the convex hull node in the image, perform coordinate system conversion, and convert all convex hull nodes to latitude and longitude coordinates in the actual space;

[0017] Step 6: Setting the water cruise route

[0018] All convex hull nodes that have undergone coordinate system conversion are numbered, and convex hull nodes with different numbers are selected to set the cruise route according to different cruise requirements, so that the drone can perform the designated water cruise function in sequence between the designated numbers and shoot the water surface during the cruise.

[0019] The method for monitoring and cruising waters with a drone based on machine vision is characterized in that in step 2, during the process of converting the collected water sample image from a color image to a grayscale image, the grayscale gradient has a value range of 0 to 255, and the conversion formula is as follows:

[0020] f=0.31R+0.58G+0.11B

[0021] Where f represents the gray value at a pixel coordinate of the gray image;

[0022] R, G, and B represent the channel parameter values ​​of the red channel, green channel, and blue channel in the RGB image at a certain pixel coordinate.

[0023] The method for monitoring and cruising waters by a drone based on machine vision is characterized in that in step 3, when the Canny operator is used to perform edge detection on the color-adjusted image, the sobel difference operator is used to calculate the pixel gradient values ​​of the image at the x-axis and the y-axis, and the pixel points with obvious pixel gradient changes in the image are taken as the edges of the image. The specific process of edge detection is as follows:

[0024] 1) The lower and upper gradient thresholds in the Canny algorithm are set to 10 and 30 respectively. In the process of edge detection using the Sobel operator, s x With s y are the Sobel operators in the x-axis and y-axis directions respectively, K represents the 3*3 grayscale matrix centered on the pixel [i, j] in the target image, and the specific matrix expression is as follows:

[0025]

[0026] 2) Through the convolution calculation of the Sobel operator and the pixel points of the target grayscale matrix, the gradient grayscale values ​​in the x-axis direction and y-axis direction at the [i, j] pixel point of the image can be obtained. The values ​​are respectively given by G x With G y The specific convolution calculation process in the x-axis and y-axis directions is as follows:

[0027] G x =s x *K

[0028] G y =s y *K;

[0029] 3) Approximately using G x With G y The sum of the absolute values ​​of represents the total gray gradient value G of the pixel, and the calculation formula is as follows:

[0030]

[0031] When the gray value near the coordinate point changes significantly, G is larger, otherwise G is smaller. The total gray gradient value G obtained by the above calculation is used as the criterion for edge detection for effective edge detection.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The method of the present invention has low cost, high accuracy, and the function of adding target cruise nodes at will, providing a new drone cruise monitoring solution for water body monitoring in large waters. The method of the present invention can be used in conjunction with Chinese patent CN201911135262.8, and the drone cruise monitoring solution is determined by the method of the present invention, and then the water quality chromaticity monitoring work is completed by Chinese patent CN201911135262.8. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Flowchart of the drone water monitoring cruise method;

[0035] Figure 2 Grayscale image of water area;

[0036] Figure 3 Water edge detection map;

[0037] Figure 4 Convex hull node graph of water area edge. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with specific embodiments, but the protection scope of the present invention is not limited thereto.

[0039] Example:

[0040] The present invention provides a method for monitoring and cruising waters with a drone based on machine vision, which specifically includes the following steps (the method flow chart is shown in FIG. Figure 1 shown):

[0041] Step 1: Use drones to collect water sample images;

[0042] Step 2: water sample image preprocessing;

[0043] Step 3: Perform edge detection on the image;

[0044] Step 4: Sorting and screening of target areas;

[0045] Step 5: Get the edge convex hull points and perform coordinate system conversion;

[0046] Step 6: Setting the water cruise route.

[0047] 1. Use drones to collect water sample images:

[0048] In the selected waters, the drone takes vertical image data of the target waters at an altitude of 30m above the target waters. The exposure time is set to 1 / 500 seconds, the ISO speed is 200, and the aperture value is f / 2.2. The longitude and latitude coordinates corresponding to the photos are obtained, and the relevant information of the photos is transmitted to the computer terminal.

[0049] 2. Water sample image preprocessing

[0050] In the image preprocessing stage, the RGB format image is converted to a grayscale image. In the process of converting to a grayscale image, the grayscale gradient value range is 0 to 255. The traditional grayscale image conversion method uses the more common weighted average method to calculate the grayscale gradient, where f represents the grayscale value at a certain pixel coordinate of the grayscale image, and R, G, and B represent the channel parameter values ​​of the red channel, green channel, and blue channel in the RGB image at a certain pixel coordinate. The conversion formula is as follows:

[0051] f=0.31R+0.58G+0.11B

[0052] After the grayscale image conversion, the image histogram is averaged using the equalizeHist() function in opencv. Figure 2 shown.

[0053] 3. Perform edge detection on the image:

[0054] Edge detection uses the current mainstream Canny edge detection algorithm. Since edge detection is mainly for objects with large areas and high differentiation, such as roads and vegetation, the lower and upper thresholds of the gradient in the Canny algorithm are set to 10 and 30. The Canny edge detection algorithm includes Gaussian smoothing, finite difference calculation of first-order partial derivatives, non-maximum suppression, dual threshold algorithm detection, and edge connection.

[0055] When using the Canny operator to perform edge detection on a color-adjusted image, the Sobel difference operator is used to calculate the pixel gradient values ​​of the image at the x-axis and y-axis, and the pixel points with obvious pixel gradient changes in the image are taken as the edges of the image.

[0056] The specific process of edge detection is as follows:

[0057] 1) The lower and upper gradient thresholds in the Canny algorithm are set to 10 and 30 respectively. In the process of edge detection using the Sobel operator, s x With s y are the Sobel operators in the x-axis and y-axis directions respectively, K represents the 3*3 grayscale matrix centered on the pixel [i, j] in the target image, and the specific matrix expression is as follows:

[0058]

[0059] 2) Through the convolution calculation of the Sobel operator and the pixel points of the target grayscale matrix, the gradient grayscale values ​​in the x-axis direction and y-axis direction at the [i, j] pixel point of the image can be obtained. The values ​​are respectively given by G x With G y The specific convolution calculation process in the x-axis and y-axis directions is as follows:

[0060] G x =s x *K

[0061] G y =s y *K;

[0062] 3) Approximately using G x With G y The sum of the absolute values ​​of represents the total gray gradient value G of the pixel, and the calculation formula is as follows:

[0063]

[0064] When the gray value near the coordinate point changes significantly, G is larger, otherwise G is smaller. The total gray gradient value G obtained by the above calculation is used as the criterion for edge detection for effective edge detection.

[0065] After canny edge detection, the image is subjected to morphological dilation and Gaussian filtering. The combination of the two can effectively increase the width and smoothness of the edge curve, and avoid the error extraction of the target area due to too many target areas. The specific edge detection effect is shown in the figure below. Figure 3 shown.

[0066] 4. Target area sorting and screening:

[0067] Opencv is used to calculate the perimeter of all closed edges in the image processed in step 3, so as to obtain the perimeter length results corresponding to each closed edge. Then, the perimeters of all closed edges are sorted from large to small using the sort() function provided by Opencv. The closed edge with the largest perimeter is the closed edge of the water area.

[0068] For the closed edge with the maximum perimeter, the convex hull algorithm is used to calculate the convex hull of the closed edge image. Only the closed edge area in the convex hull is kept as white, and the other areas are all filled with black. At this time, the edge image of the water area is obtained. The specific convex hull and the corresponding convex hull nodes are shown in Figure 4 Displayed in.

[0069] 5. Get the edge convex hull nodes and perform coordinate system conversion

[0070] Since the drone shooting position is at the center of the image, based on the drone's altitude, yaw angle, and the horizontal angle of the camera relative to the ground, the coordinates of each convex hull node in the image can be calculated to have a deviation of ΔJ and ΔW from the longitude and latitude of the drone in the world coordinate system. Assume that the longitude and latitude coordinates of each convex hull node i are expressed as (J i , W i ), the longitude and latitude of the drone are expressed as (J0, W0), and the radius of the earth is 6371 kilometers. The longitude and latitude calculation formulas of the corresponding convex hull node i are as follows:

[0071] J i =J0+ΔJ×180×(6371×1000×π)

[0072] W i =W0+ΔW×180×(6371×1000×π).

[0073] 6. Setting of water cruise route

[0074] All the convex hull nodes that have been carried out are numbered. If other convex hull nodes of the water area need to be cruised, the calculated convex hull node coordinates can be converted according to the map scale to obtain new convex hull node coordinates. According to different water body monitoring tasks, the target convex hull node is selected as the cruise node to plan the cruise route. This cruise method combined with the water body detection model can well achieve the real-time monitoring task of a certain water area.

[0075] The contents described in this specification are merely an enumeration of implementation forms of the inventive concept, and the protection scope of the present invention should not be regarded as being limited to the specific forms described in the embodiments.

Claims

1. A method for monitoring and cruising waters with a drone based on machine vision, characterized in that The process includes the following steps: Step 1: Use drones to collect water area images In the target water area, the drone height is set, and the drone is used to collect and monitor water sample images of the target water area, and the collected water sample images are transmitted to the computer terminal in real time; Step 2: Water area image preprocessing The collected water sample image is converted from a color image to a grayscale image, and the converted grayscale image is processed by histogram averaging to obtain a color-adjusted image; Step 3: Perform edge detection on the image Use the Canny operator to perform edge detection on the color-adjusted image obtained in step 2 to remove the interference of the debris pixels in the image. Then, perform morphological dilation processing and Gaussian filtering on the image after edge detection to connect the edge boundaries in the image to form a closed edge and increase the width and smoothness of the edge boundaries. Step 4: Sorting and filtering target areas The perimeters of all closed edges in the image processed in step 3 are calculated by OpenCV to obtain the perimeter length results corresponding to each closed edge. Then, the perimeters of all closed edges are sorted from large to small. The closed edge with the largest perimeter is the closed edge of the water area. For the closed edge with the maximum perimeter, the convex hull algorithm is used to calculate the convex hull of the closed edge image, and only the closed edge area in the convex hull is retained as white, and the other areas are all filled with black, so as to obtain the edge image of the water area; Step 5: Get the edge convex hull points and perform coordinate system conversion Get the convex hull nodes of the entire water area edge, calculate the corresponding coordinate position of the convex hull node in the image, perform coordinate system conversion, and convert all convex hull nodes to latitude and longitude coordinates in the actual space; The drone shooting position is at the center of the image. According to the altitude, yaw angle and horizontal angle of the camera relative to the ground, the coordinates of each convex hull node in the image can be calculated to have a deviation of ΔJ and ΔW from the longitude and latitude of the drone in the world coordinate system. The longitude and latitude coordinates of each convex hull node i are expressed as (J i , W i ), the longitude and latitude of the drone are expressed as (J0, W0), and the radius of the earth is 6371 kilometers. The longitude and latitude calculation formulas of the corresponding convex hull node i are as follows: J i =J0+ΔJ×180×(6371×1000×π) IN i =W0+ΔW×180×(6371×1000×π) Step 6: Setting the water cruise route All convex hull nodes that have undergone coordinate system conversion are numbered, and convex hull nodes with different numbers are selected to set the cruise route according to different cruise requirements, so that the drone can perform the designated water cruise function in sequence between the designated numbers and take pictures of the water surface during the cruise; In step 3, when using the Canny operator to perform edge detection on the color-adjusted image, the Sobel difference operator is used to calculate the pixel gradient values ​​of the image at the x-axis and y-axis, and the pixel points with obvious pixel gradient changes in the image are taken as the edges of the image. The specific process of edge detection is as follows: 1) The lower and upper gradient thresholds in the Canny algorithm are set to 10 and 30 respectively. In the process of edge detection using the Sobel operator, s x With s y are the Sobel operators in the x-axis and y-axis directions respectively, K represents the 3*3 grayscale matrix centered on the pixel [i, j] in the target image, and the specific matrix expression is as follows: 2) Through the convolution calculation of the Sobel operator and the pixel points of the target grayscale matrix, the gradient grayscale values ​​in the x-axis direction and y-axis direction at the [i, j] pixel point of the image can be obtained, and their values ​​are respectively given by G x With G y The specific convolution calculation process in the x-axis and y-axis directions is as follows: G x =s x *K G y =s y *K; 3) Approximately using G x With G y The sum of the absolute values ​​of represents the total gray gradient value G of the pixel, and the calculation formula is as follows: When the gray value near the coordinate point changes significantly, G is larger, otherwise G is smaller. The total gray gradient value G obtained by the above calculation is used as the criterion for edge detection to perform effective edge detection. In step 2, when converting the collected water sample image from a color image to a grayscale image, the grayscale gradient value range is 0 to 255, and the conversion formula is as follows: f=0.31R+0.58G+0.11B Where f represents the gray value at a pixel coordinate of the gray image; R, G, and B represent the channel parameter values ​​of the red channel, green channel, and blue channel in the RGB image at a certain pixel coordinate.

Images