Method for identifying and judging water surface width of canal by low-altitude photography
By using neural network to identify the slope and vertical distance characteristics of the straight lines on both sides of the canal surface in low-altitude photography of drone, the flexibility and accuracy of the canal surface width detection are solved, and automatic identification and monitoring of the canal surface width is realized. under different hydrological conditions.
Patent Information
- Application Number
- CN202411124036.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art relies on fixed angle and proportional relationships when detecting the water surface width of the river canal in low-altitude photography, which limits the flexibility and accuracy of the detection, especially when the relationship between the canal direction and the heading of the drone is unpredictable. .
Using a neural network-based method, the slope of the straight line on both sides of the canal surface and the vertical distance between the image center point and the straight line as geometric features are used to establish a canal surface width prediction model, and online identification and prediction are carried out to realize automatic identification and monitoring of the canal surface width. .
Accurate prediction of the water surface width of the canal under uncertain heading is achieved, it is suitable for monitoring during dry and abundant water periods, and the monitoring ability of the water surface width of most rivers is improved.
Smart Images

Figure CN119027841A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned aerial vehicle detection application, and specifically relates to a method for identifying the width of a river or canal surface through low-altitude photography and a flight control method. Background Art
[0002] In recent years, drone technology has been widely used in many fields, especially in various detection and measurement aspects, and has shown significant advantages and potential.
[0003] Drones can fly along high-voltage lines and use high-definition cameras or thermal imaging equipment to detect damage, corrosion and other potential problems. Drones can quickly obtain image data of roads and railways, identify risks such as cracks and landslides, and perform regular maintenance.
[0004] Drones are also widely used in terrain inspection and hydrological detection. By carrying high-resolution cameras or lidar, drones can obtain high-precision terrain data and generate digital terrain models and digital elevation models. The application of drone low-altitude photography in terrain detection, especially in rivers and hydrology, provides efficient, flexible and accurate technical means, providing important support for environmental protection, water resources management and natural disaster prevention and control.
[0005] Low-altitude drone photography can easily obtain high-resolution images of rivers and their surrounding areas, helping scientists and engineers monitor changes in river morphology in real time, such as river siltation, erosion, riverbed uplift or downcutting. These data are crucial to understanding the dynamic process of rivers and their impact on the environment. Using low-altitude drone photography technology, current images of flood areas can be quickly obtained to help assess the impact of floods, changes in flood water levels, and threats to surrounding infrastructure. By comparing topographic data before and after floods, the long-term impact of floods on river channels and terrain can also be analyzed.
[0006] Among hydrological data, river surface width is the most basic data. Currently, river width is either measured on the spot using instruments or directly calculated based on pixel ratio in images.
[0007] For example, the Chinese invention patent application with application number 202310722121.6 discloses a floating river width measuring instrument, which obtains a first distance and a second distance of the measured distance from a set of radars to the river bank, and calculates the first vertical distance from the measuring instrument to the river bank according to the first angle between the two known radars; obtains a third distance and a fourth distance of the measured distance from another set of radars to the river bank, and calculates the second vertical distance from the measuring instrument to the river bank according to the second angle between the two known radars; finally, the first distance and the second distance are summed to obtain the river width distance. Similarly, the Chinese invention patent application with application number 202111392791.3 discloses a river width extraction method using a river meter, which first places the river meter on the ground on one side of the river bank, detects the river surface and the ground on the opposite bank of the river through an antenna, and obtains the position of the dividing point; then, the distance element at the dividing point is determined according to the amplitude difference between the river echo and the ground echo, as the distance between the river meter and the opposite bank; then, the distance between the river meter and the river bank on the side close to the river meter is measured, and the river width is calculated by combining these two distances.
[0008] In addition to using radar waves for detection, there are also calculation schemes for river width detection through images. For example, the Chinese invention patent application with application number 202311290498.5 discloses a flood warning method and system for wild rivers in arid areas. A water gauge is arranged near a fixed reference object in a wild river, and a fixed camera is used to capture the surface image of the wild river in real time. The image is identified and the water depth and width of the wild river are obtained with the help of the water gauge information in the image. For the river width, the scheme uses a priori proportional data to binarize the extracted image and directly read the river width from the river width binary image. However, the Chinese invention patent application with application number 202311564794.X discloses a real-time monitoring method for the ecological flow of natural rivers based on tower base video; it combines video images, searches for monitoring sections in digital orthophoto data, obtains the actual distance of the monitoring section in the digital orthophoto data, and calibrates the conversion coefficient f=W / N between the water surface width of the monitoring section and the number of water body pixels, wherein W is the actual river width of the monitoring section obtained based on the digital orthophoto data, and N is the number of water body pixels in the monitoring section; based on the conversion coefficient, the river width can be measured according to the image.
[0009] This type of image-based measurement scheme relies on an accurate proportional relationship, that is, the pixel width value of the measured line segment corresponds to a priori determined ratio. Obviously, this scheme requires the video screen to be fixed at the same angle to be comparable, and for drone low-altitude photography, this undoubtedly greatly limits its flexibility in river and canal measurement.
[0010] Therefore, there is a need for a method and system for automatic identification and width detection of rivers and channels based on low-altitude photography with strong generalization performance and wide applicability. Summary of the invention
[0011] As the river flows, it scours the riverbank, thus forming a shoreline. It has been observed that in many sections of a river, there are two nearly parallel shorelines. In particular, in rivers, there is a type of artificial or natural waterway that is built or naturally formed for specific functions, namely canals. At present, the canal system around the world is now very developed. Canals are widely distributed in plains and hilly areas, used to guide river water for large-scale farmland irrigation and discharge excess water. The cross-section of a canal is usually trapezoidal, because the trapezoidal section has good slope stability, which reduces the risk of collapse of the slopes on both sides of the canal. At the same time, the selection of the bottom width and slope angle of the trapezoidal section also helps to control the flow rate and reduce sediment deposition, and the shape of the trapezoidal section is easier to dig and shape by mechanized equipment.
[0012] In order to solve the above technical problems and realize automatic identification of river and canal water surface width based on low-altitude photography, the present invention provides a method for identifying river and canal water surface width based on low-altitude photography and a flight controller method. The features of the river and canal are identified and their water surface width is calculated based on the images taken by the drone at low altitude photography, and there is no need to limit the drone to maintain a certain precise specific orientation during sampling.
[0013] The present invention provides a method for identifying the width of a river or canal water surface by low-altitude photography, which comprises the following steps:
[0014] S1. Initialization, establishing a river channel water surface width prediction model based on a neural network, wherein the input of the neural network is the geometric features of two parallel straight lines relative to the center of the image, and the output is the actual distance between the two parallel straight lines;
[0015] S2, constructing a thin sheet-shaped river channel water surface model with two parallel edges, collecting images of the river channel water surface models with different widths by low-altitude oblique photography in an offline state to form a sample set, and using the sample set to train the neural network;
[0016] S3, collecting river and canal images online at a certain height and pitch angle by low-altitude photography, and determining a ROI region including the river and canal in the river and canal images;
[0017] S4, according to the color space characteristics of the central part of the ROI area, taking the pixel points with similar color space characteristics in the river channel image as foreground objects, and taking the non-foreground object area as background objects;
[0018] S5. Among the foreground objects, according to pixel connectivity, find the largest connected domain as the canal water body area, take the canal water body area as the foreground object, and set all non-foreground objects to zero, so as to form a canal target image;
[0019] S6, obtaining a river channel edge image by edge detection on the river channel target image;
[0020] S7, after performing Hough line transformation on the canal edge image, obtaining straight lines on both sides of the canal water surface along the canal wall through heuristic search;
[0021] S8, obtaining geometric features of the two straight lines on the water surface of the canal relative to the center of the image;
[0022] S9, inputting the geometric features into the trained river channel water surface width prediction model to obtain a river channel water surface width prediction value;
[0023] S10, outputting the width prediction value;
[0024] S11, check whether there is an exit instruction, if yes, go to step S13, otherwise go to step S12;
[0025] S12, after the width identification instruction is triggered, go to step S3;
[0026] S13, ending the identification of the width of the river channel.
[0027] Preferably, when forming the sample, the step S2 further comprises: filtering and graying the collected image of the river and canal water surface model, binarizing it to form a river and canal target image, and then performing edge detection to obtain a river and canal edge image;
[0028] The canal edge image is subjected to Hough line transform, and an edge image containing two straight edges of the canal water surface model is obtained through heuristic search; the geometric feature quantities of the two edge lines relative to the image center are obtained from the edge image, and the width of the current canal water surface model is recorded, and the obtained geometric feature quantities and the recorded model width are formed into a training sample.
[0029] Preferably, the geometric features in steps S1 and S8 further include two input quantities: flight altitude and bird's-eye view angle during collection.
[0030] Preferably, the geometric features in step S1 include four input quantities, namely, the slopes k1 and k2 of two straight lines and the perpendicular distances D1 and D2 from the center point of the image to the two straight lines; the geometric features in step S8 include four input quantities, namely, the slopes k1 and k2 of the two side straight lines of the river channel water surface and the perpendicular distances D1 and D2 from the center point of the image to the two side straight lines.
[0031] Preferably, after the Hough line transformation in step S7, a set of straight lines at the edge of a canal is obtained; in the heuristic search, two straight line segments with close direction angles, maximum mutual distance and longest length are selected from the obtained set of straight lines at the edge of the canal according to heuristic rules as the straight lines on both sides of the canal water surface; the mutual distance is the minimum value of the actual distance between each point on any one of the two straight line segments and each point on the other line segment; wherein, the actual distance d between a point and each point on a target line segment is calculated as follows: if the line connecting the point to any end point of the target line segment forms an obtuse angle with the target line segment, the actual distance d is taken as the distance from the point to the end point; otherwise, the actual distance d is taken as the vertical distance from the point to the target line segment, that is, the perpendicular distance.
[0032] Preferably, the step S1 also includes: establishing a target classification model based on the YOLOv5s network, using the bird's-eye view camera in the image acquisition unit to capture the river and canal scene images by low-altitude photography, and then offline training the network with an image sample set consisting of the river and canal scene images with the river and canal areas marked to obtain a river and canal scene target classification model; in the step S3, the river and canal scene target classification model is used to perform recognition processing on the captured river and canal images to obtain a river and canal area anchor frame, and the area within the anchor frame is used as the river and canal ROI area.
[0033] Preferably, in step S4, a color threshold is determined according to the respective value intervals of the three color channels extracted from the central part of the ROI area, pixels whose colors meet the color threshold range are taken as foreground objects, and non-foreground object areas are taken as background objects, and all background objects in the canal image are set to zero to form a canal binary image; in step S5, according to the connectivity of the foreground object pixels, all connected domains are detected and the largest connected domain is screened out as the canal water body area, and the canal target image is formed with only the canal water body area as the foreground object.
[0034] Preferably, in step S4, the color thresholds are determined to correspond to the threshold intervals of the three primary colors according to the value intervals of the three color channels extracted from the central part of the ROI area. After the upper and lower limits of the grayscale values of each color channel of all pixels in the central seed area are calculated, the upper and lower limits can be extended by a certain distance, such as increasing or decreasing a certain value of 5 to 20, or increasing or decreasing 5 to 10% of the median of the grayscale value interval of the channel.
[0035] Preferably, in step S4, a seed point is set at the intersection of the two diagonals corresponding to the ROI area rectangle, and the river and canal water area is obtained as the foreground object through flood filling processing. After the pixels outside the obtained river and canal water area are set to zero, a river and canal binary image is formed.
[0036] Preferably, in step S5, the binary river and canal water body image, i.e., the river and canal binary image, is further subjected to a morphological closing operation of first dilation and then erosion to remove interference points in the connected domain of the water body. In step S7, any of the two straight lines on the river and canal water surface is described by the coordinates of the two end points of the straight line; and in step S8, the slope of the straight line and the perpendicular distance from the center point of the image to the straight line are respectively obtained based on the two end points; the slope of the straight lines on both sides of the river and canal water surface and the perpendicular distance from the center point of the image to the straight line are used as the geometric features of the river and canal in the river and canal image.
[0037] Preferably, the BP neural network used in the canal water surface width prediction model is a three-layer structure including a hidden layer, wherein:
[0038] The output of the jth node in the hidden layer is
[0039] The output of the output layer node is
[0040] Among them, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.
[0041] Preferably, the geometric features of the straight line are calculated as follows: assuming that the coordinates of the two end points of the straight line are (x1, y1) and (x2, y2), then its slope is k=y1-y2 / x1-x2;
[0042] The perpendicular distance from the center of the image to the straight line is
[0043] Where (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is a constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1(x2-x1);
[0044] According to the calculation formulas of k and d, the slopes k1 and k2 of the two straight lines and the perpendicular distances D1 and D2 from the center point of the image to the two straight lines are calculated respectively, a total of four quantities are obtained as geometric features.
[0045] Preferably, in step S2, training the neural network as a river channel water surface width prediction model includes: T1, initializing the network weights and biases; T2, reading the network parameters and the training sample set recorded in the document; T3, normalizing the sample data; T4, calculating the error between the predicted output value and the expected value of each sample; T5, correcting the network weights and biases by reverse calculation based on the error; T6, if the conditions for ending the network training are met, ending the training, otherwise going to T4 to continue the training; T7, analyzing and outputting the results.
[0046] Preferably, step S2 also includes: arranging the river and canal water surface models in different areas of the ground under the oblique photography field of view, so that the river and canal water surface models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane.
[0047] Preferably, the neural network used as the canal water surface width prediction model is multiple, wherein each canal water surface width prediction model corresponds to a flight altitude and a bird's-eye view angle when the canal image is acquired by low-altitude photography; the canal water surface width prediction model applied in step S9 corresponds to the canal water surface width prediction model neural network corresponding to the flight altitude and the bird's-eye view angle when the canal image is acquired by low-altitude photography in step S3.
[0048] Preferably, in step S10, the coordinate information and orientation information of the current canal segment location are also obtained, and after the water surface width of the canal is identified, the width prediction value and the corresponding coordinate and orientation information are output. After the canal width is detected in sections, the width information of each section of the canal is also marked at the corresponding position of the canal image, and the arrow line is marked between the two water bank straight lines of the section of the canal; the obtained coordinate and position information, as well as the water surface width information of the canal segment, are also stored in the list of the storage module or the cloud server.
[0049] Preferably, in step S10, it is also determined whether the water surface width of the current segment exceeds a preset river channel water surface width threshold range, and if so, an over-limit alarm is issued.
[0050] In another embodiment of the present invention, a flight control method for identifying the width of a river or canal by low-altitude photography is provided, comprising the following steps:
[0051] R1. Initialization, establishing a canal water surface width prediction neural network to map the geometric features of the canal water surface edge straight line in the low-altitude photography image to the canal water surface width, and training the canal water surface width prediction neural network after collecting training samples to obtain a trained canal water surface width prediction model;
[0052] R2. In response to the command of river and canal water surface width identification during flight, low-altitude oblique photography of the field river and canal scene is performed to obtain river and canal images;
[0053] R3, determining a ROI region containing a river channel in the river channel image, determining a color threshold from the value intervals of the three color channels extracted from the central part of the ROI region, taking pixels whose colors meet the color threshold range as foreground objects, and setting all non-foreground object areas in the river channel image to zero to form a river channel binary image;
[0054] R4, according to the connectivity of the foreground object pixels, all connected domains are detected and the largest connected domain is selected as the canal water body area, and only the canal water body area is used as the foreground object to form a canal target image;
[0055] R5. For the canal target image, after edge detection is performed to obtain the canal edge, Hough line transform is performed, and a canal water edge line image containing two straight edges of the canal water surface, i.e., the edge line, is obtained through heuristic search;
[0056] R6. Calculate the geometric feature quantity of each of the two edge lines relative to the image center according to the two edge lines obtained from the canal water edge line image;
[0057] R7, inputting the obtained geometric feature quantity into a river channel water surface width prediction model to obtain a river channel water surface width prediction value, and outputting the width prediction value;
[0058] R8, determine whether the current water surface width exceeds the preset river channel water surface width threshold range of the segment, if so, issue an over-limit alarm;
[0059] R9, determine whether to exit the river channel water surface width identification task, if yes, go to step R11, otherwise go to step R10;
[0060] R10, flying along the direction of the two edge lines of the river channel water surface, when the overlap range between the image in the current field of view and the river channel image in the previous frame is less than a certain ratio, issuing a river channel water surface width recognition instruction, and going to step R2;
[0061] R11. End the flight control for judging the width of river and canal water surface through low-altitude photography.
[0062] Preferably, in step R2, an inverse perspective transformation matrix is obtained according to the height and pitch angle sampled when low-altitude oblique photography is performed on the canal scene; in step R4, the binarized canal target image is converted into a real binary image of the canal based on the inverse perspective transformation matrix; in step R5, edge detection and Hough line transformation are performed on the real binary image of the canal, and a canal water edge line image containing two straight edges, i.e., edge lines, of the canal water surface is obtained through heuristic search; then, it is calculated whether the angles or slopes of the two edge lines of the canal water surface are close, if so, step R6 is performed, otherwise, the canal water surface edge lines are marked as not parallel and step R10 is performed; in step R10, it is first determined whether the canal water surface edge lines are parallel, if not, the flight continues along the canal in the direction parallel to the angular bisectors of the two obtained edge lines of the canal water surface, otherwise, the flight is performed in the direction of the two edge lines of the canal water surface.
[0063] Preferably, in step R2, multiple images may be stitched together by aligning feature points, and the stitched image may be used as the current river channel image.
[0064] Preferably, in step R8, the ratio of the water surface width of the river channel in the current segment to a preset width threshold at that location is calculated, and when the ratio value is greater than a set high ratio threshold, an alarm is issued to indicate that the river width at that location exceeds the warning level and flood prevention is required; and when the ratio value is less than a set low ratio threshold, an alarm is issued to indicate that the water surface width at that location is too small and drought prevention or possible water leakage is required.
[0065] Compared with the prior art, the scheme of the present invention has the following advantages: the present invention analyzes the imaging characteristics of rivers and canals under low-altitude photography by unmanned aerial vehicles, analyzes the characteristic representation of the width of the river and canal water surface in the image space of the river and canal image in view of the unpredictable relative orientation relationship between the posture of the unmanned aerial vehicle and the direction of the river and canal when collecting images, and proposes to use the slopes of the straight lines on both sides of the river and canal water surface and the vertical distances from the center point of the image to the straight lines on both sides of the river and canal water surface as four geometric features as constraints, and establishes a neural network to predict the width of the river and canal water surface. The nonlinear mapping relationship between the width of the river and canal water surface and the four proposed geometric features is identified and trained by collecting samples. Based on the river and canal images photographed from low altitude, the target classification model is used to locate the ROI area of the river and canal target with an anchor frame, and the river and canal water body target is obtained according to the color value interval in the ROI area, and the river and canal water body object is obtained by screening the maximum connected domain. Edge detection is performed on the canal water area, and the straight lines on both sides of the canal water surface are obtained through Hough line transform screening. The four geometric features are extracted from the two extracted edge lines, and the results are input into the trained canal water surface width prediction model to perform online prediction of the canal width.
[0066] The present invention realizes the automatic extraction of river and canal water body objects and water surface shorelines. The established river and canal water surface width prediction model based on neural network has good generalization performance and can accurately predict the water surface width of the river and canal in different periods of dry and flood water according to the river and canal images collected from uncertain navigation directions; at the same time, it can also be applied to the water surface width monitoring of the sections with approximately parallel banks that most rivers have.
[0067] It should be understood that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (provided that such concepts are not mutually inconsistent) can be considered as part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing in the disclosed solution can be considered as part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0068] Figure 1 This is a schematic diagram of the processing flow of the method for identifying the width of river and canal water surface using low-altitude photography;
[0069] Figure 2 It is a schematic diagram of the structure of the device and system for judging the width of the river and canal water surface by low-altitude photography;
[0070] Figure 3A This is the structure diagram of the host unit / controller in the low-altitude photography river and channel water surface width identification device. Figure 3B This is the structure diagram of the water area extraction module. Figure 3C This is the structural diagram of the geometric feature extraction module;
[0071] Figure 4A This is a schematic diagram of the neural network model structure of the river and canal water surface width prediction module. Figure 4B It is a schematic diagram of the partial structure of the host unit / controller in the low-altitude photography river and channel water surface width identification device;
[0072] Figure 5 Schematic diagram of the drone’s shooting posture when collecting samples for low-altitude photography;
[0073] Fig. 6A This is a schematic diagram of the comparison of water surface models of rivers and canals with different widths. Figure 6B Schematic diagram of the river channel water surface model taken from different angles during sample collection;
[0074] Figure 6C This is a schematic diagram of the contour processing of the river and canal water surface model. Fig.6D It is a schematic diagram of the detection result after Hough line transformation;
[0075] Figure 7 The diagram is a schematic diagram of the width constraint conditions and geometric features of the river and canal water surface model proposed;
[0076] Fig. 8A , Figure 8BIt is a schematic diagram of the screening of straight lines on the water surface edges of rivers and canals in different directions and the extraction of geometric features;
[0077] Fig. 9A , Fig. 9B Fitting curve between actual and predicted values of the water surface model width of the test set of rivers and canals;
[0078] Fig.10 Schematic diagram of anchor frame for river and canal target identification;
[0079] Fig.11A , Fig. 11B They are schematic diagrams of direct binarization and contour extraction respectively;
[0080] Fig. 12A , Fig. 12B They are respectively based on the binarization of each color grayscale value interval and the schematic diagram of contour extraction after binarization. Fig. 12C This is a schematic diagram of the morphological processing results. Fig.12D Schematic diagram for screening the maximum connected domain;
[0081] Fig.13A , Fig. 13B , Fig. 13C They are the binary image after the maximum connected domain screening, morphological processing, and the corresponding schematic diagram of water area extraction;
[0082] Fig.14A is the edge image of the river water body, Fig. 14B Schematic diagram of the straight line extracted by Hough line transform;
[0083] Fig.15A This is a comparison chart of the straight line detection results of the edge of the river surface. Fig. 15B This is a schematic diagram of the width of the river and canal water surface;
[0084] Fig.16 This is a schematic diagram of the segmentation of river and canal water surface width identification;
[0085] Fig.17A , Fig. 17B This is a schematic diagram of a river canal scene taken through low-altitude photography.
[0086] Among them: 1000 low-altitude photography river and canal water surface width identification systems, 900 low-altitude photography river and canal water surface width identification devices, 100 host units / controllers, 200 user interface units, 300 handheld terminals, 400 servers, 500 image acquisition units, 600 flight control platforms, 700 positioning units, 800 field environments / river and canal scenes,
[0087] 110 processor, 120 storage module, 130 input and output module,
[0088] 111 general processing module, 112 water area extraction module, 113 geometric feature extraction module, 114 iterative learning module, 115 river channel water surface width prediction module,
[0089] 1121 image acquisition unit, 1122 target detection unit, 1123 water area extraction unit,
[0090] 1131 edge detection unit, 1132 straight line extraction (screening) unit, 1133 water edge extraction unit, 1134 slope feature extraction unit, 1135 distance feature extraction unit,
[0091] 1141 first connection array, 1142 second connection array,
[0092] 1101 neural network model, 131 input unit, 132 output unit
[0093] 210 display screen, 220 operation panel. DETAILED DESCRIPTION
[0094] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, but the present invention is not limited to these embodiments. The present invention covers any substitution, modification, equivalent method and scheme made within the spirit and scope of the present invention.
[0095] In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but users skilled in the art can fully understand the present invention without these detailed descriptions.
[0096] The present invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. It should be noted that the accompanying drawings are all simplified and not in exact proportion, and are only used for the purpose of conveniently and clearly assisting in explaining the embodiments of the present invention.
[0097] Embodiment 1:
[0098] In my country, rivers and canals are widely distributed. In addition to collecting terrain images such as rivers and canals through traditional satellite and other high-altitude photography; Fig.17A , Fig. 17B As shown, with the development of drone technology, it is now very convenient to use low-altitude drones to perform oblique photography sampling of rivers and canals. It is of great significance to identify the width of rivers and canals. To this end, the present invention discloses a method for identifying the water surface width of rivers and canals by low-altitude photography, which does not rely on prior width measurement and does not require a preset specific heading when collecting images of rivers and canals in real time.
[0099] like Figure 1 As shown, the present invention provides a method for identifying the width of a river or canal using low-altitude photography, which comprises the following steps:
[0100] S1. Initialization, establishing a river channel water surface width prediction model based on a neural network, wherein the input of the neural network is the geometric features of two parallel straight lines relative to the center of the image, and the output is the actual distance between the two parallel straight lines;
[0101] S2, constructing a thin sheet-shaped river channel water surface model with two parallel edges, collecting images of the river channel water surface models with different widths by low-altitude oblique photography in an offline state to form a sample set, and using the sample set to train the neural network;
[0102] S3, collecting river and canal images online at a certain height and pitch angle by low-altitude photography, and determining a ROI region including the river and canal in the river and canal images;
[0103] S4, according to the color space characteristics of the central part of the ROI area, taking the pixel points with similar color space characteristics in the river channel image as foreground objects, and taking the non-foreground object area as background objects;
[0104] S5. Among the foreground objects, according to pixel connectivity, find the largest connected domain as the canal water body area, take the canal water body area as the foreground object, and set all non-foreground objects to zero, so as to form a canal target image;
[0105] S6, obtaining a river channel edge image by edge detection on the river channel target image;
[0106] S7, after performing Hough line transformation on the canal edge image, a canal water surface, i.e., two straight lines along the canal wall, is obtained through heuristic search, and a canal water edge line image is formed;
[0107] S8, obtaining geometric features of the two straight lines on the canal water surface relative to the center of the image from the canal water edge image;
[0108] S9, inputting the geometric features into the trained river channel water surface width prediction model to obtain a river channel water surface width prediction value;
[0109] S10, outputting the width prediction value;
[0110] S11, check whether there is an exit instruction, if yes, go to step S13, otherwise go to step S12;
[0111] S12, after the width identification instruction is triggered, go to step S3;
[0112] S13, ending the identification of the width of the river channel.
[0113] like Figure 2As shown, a low-altitude photography river and canal water surface width identification device 900 using the method of the present invention includes a host unit 100 and a user interface unit 200; wherein the host unit 100 includes a processor 110, a storage module 120 and an input / output module 130, and the user interface unit 200 is provided with a display screen 210 and an operation panel 220. The host unit 110 is configured to: obtain a low-altitude photography river and canal image, determine an ROI region containing the river and canal in the river and canal image, determine a color threshold according to the respective value intervals of the three color channels extracted from the central part of the ROI region, take pixels whose colors meet the color threshold range as foreground objects, take non-foreground object regions as background objects, and set all background objects in the river and canal image to zero to form a river and canal binary image; according to the connectivity of the foreground object pixels, detect all connected domains and screen out the largest connected domain as the river and canal water body region, and only take the river and canal as the background object region. The water body area is used as the foreground object to form a river channel target image; the river channel target image is subjected to edge detection to obtain a river channel edge image; the river channel edge image is subjected to Hough line transform, and then a canal water edge line image containing two straight edges of canal water is obtained through heuristic search; the geometric features of the straight lines on both sides of the river channel water surface relative to the image center are obtained from the canal water edge line image, and the geometric features are input into a trained neural network-based river channel water surface width prediction model to obtain a river channel water surface width prediction value, and the width prediction value is output by the output unit 132 in the input-output module 130.
[0114] like Fig.17A As shown, the cross-section of the canal is usually trapezoidal. The water flow in the canal varies greatly during the flood season and the dry season. Low-altitude photography images show that the water bank in the canal moves due to changes in flow, and the distance between the water bank on both sides, that is, the width of the water surface, also changes accordingly.
[0115] like Fig. 17B As shown in the figure, the water flow in the canal is relatively evenly distributed. Although its direction is sometimes curved, it has two nearly parallel water banks in most areas. Fig.17A , Fig. 17B As shown in the figure, in the canal image collected in the wild scene, the canal water surface appears to be wide at one end and narrow at the other end, and in the image, the parallel shorelines on both sides of the actual canal water surface will approximately intersect at a vanishing point at infinity. In order to obtain the river width, on the one hand, it is obvious that when the distance is measured from the pixel points on both sides of the canal in the image and then converted back to the actual width according to the proportional relationship, the proportional relationship is difficult to accurately determine, and it is directly related to the sampling position of the side pixel points. On the other hand, in actual applications, it is not possible to force the drone to face a fixed direction of the canal, such as facing the center of the canal, to collect images in order to perform proportional measurement, otherwise the applicability of recognition will be greatly limited.
[0116] Although the width of the canal water surface in the field of view is certain, when it is sampled from the perspective of the drone, its pixel width and direction in the image space are uncertain. This certain actual width and the uncertain image features are in a contradictory relationship that is difficult to reconcile. Therefore, this contradiction needs to be decoupled and modeled. The heading of the drone is not fixed during flight, and the direction of the canal varies greatly; therefore, the relative relationship between the heading of the drone and the direction of the canal is unpredictable and cannot be preset. In order to improve the versatility and applicability of canal water surface width identification, it is necessary to study how to characterize the relationship between the two in various relative orientations in the image space, that is, what kind of features in the image can determine the width of the canal water surface?
[0117] After observing the water surfaces of rivers and canals in different directions in various low-altitude photography images of rivers and canals, it was found that the key lies in the two edge lines of the water surface, namely the shoreline; the direction of the edge of the river and canal water surface itself is not the only way to determine the width of the river and canal water surface. The angle of the straight line itself can only reflect the relative angle difference between it and the heading. After a large number of experiments and tests, it was also found that the width of the river and canal water surface will affect the imaging effect of the two parallel sides in the image. How to characterize the characteristics of the two sides? Through further research, hypothesis and reasoning, it is proposed to use the slopes k1 and k2 of two parallel lines at different intervals and the perpendicular distances D1 and D2 from the center point of the image to the straight lines on both sides of the straight line as the constraints on the width of different river and canal water surfaces in the image space. Figure 7 As shown. Further, how to map this constraint condition to the width of the canal water surface? Since there is a nonlinear, coupled mapping relationship between the two, inspired by the fitting application of neural networks in system nonlinear modeling, the present invention proposes to use neural networks to establish a canal water surface width prediction model, referring to Figure 4A shown.
[0118] Neural network is a machine learning model that imitates biological neural systems and is particularly good at nonlinear mapping modeling. It learns and represents complex functional relationships through a multi-layer artificial neuron structure. Neural network consists of input layer, hidden layer and output layer; each layer contains several neurons, and each neuron is connected to the neurons in the previous layer through weights. Among them, the input layer receives input data, the hidden layer performs nonlinear transformation and feature extraction, which can have multiple layers, and the output layer outputs the final prediction result.
[0119] like Figure 4A As shown, the canal water surface width prediction model of the present invention adopts a three-layer BP neural network with a hidden layer, and its input layer receives four input quantities, namely the slopes k1 and k2 of the straight lines on both sides of the canal water surface and the vertical distances D1 and D2 from the center point of the image to the straight lines on both sides of the canal water surface, from the general processing module respectively, and the output quantity of the output layer is the canal water surface width.
[0120] To extract the above four input quantities or geometric features of the canal in the image space, it is necessary to first find the canal water area from the image and then extract the straight lines on both sides of the water surface.
[0121] by Fig. 17B Taking the canal scene in the middle left image as an example, the acquired color image is directly grayed out and binarized according to the Otsu threshold method. The processing results are as follows: Fig.11A As shown. Fig.11A After extracting the contour of the binary image obtained in the image, mark it in the original image and get the following Fig. 11B The outline diagram is shown. Fig. 11B In the displayed results, the contour lines enclose the sky and the scenery between the river and the land bank; among them, the river and the water body is submerged in the whole river. Therefore, it is difficult to obtain effective information for finding the river and the water body through ordinary binarization processing.
[0122] After observing a large number of river and canal images from low-altitude photography, it is found that the water body of the river and canal has color space characteristics and position characteristics that are obviously different from those of the land and slopes. Therefore, different modules are set in the processor to extract the water body of the river and canal and the geometric characteristics in sequence.
[0123] See also Figure 2 , Figure 3A As shown, in the processor 110 of the host unit or controller 100, a plurality of processing modules are provided, namely, a general processing module 111, a water body region extraction module 112, a geometric feature extraction module 113, an iterative learning module 114 and a river channel water surface width prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the recognition of the river channel in the collected river channel image, schedules the recognition processing flow, inputs the river channel image collected in real time into the water body region extraction module 112, and outputs the width data recognized and predicted by the river channel water surface width prediction module 115.
[0124] like Figure 3B As shown, in the water area extraction module 112, an image acquisition unit 1121, a target detection unit 1122, and a water area extraction unit 1123 are also provided. The image acquisition unit 1121 acquires the low-altitude photography image taken by the image acquisition unit 500, or acquires the pre-stored river channel image from the storage module 120 in an offline state; removes the high-frequency noise through filtering; then, the target detection unit 1122 identifies the position of the river in the image as the ROI area according to the deep learning feature by a trained target classification model; the water area extraction unit 1123 selects a seed area from the middle of the ROI area to obtain a color threshold, and extracts the river channel water area from the inclined low-altitude shadow image according to the obtained color threshold to form a binary river channel water image.
[0125] In the target detection unit 1122, a target classification model based on a deep network such as a YOLOv5s network is established. After using the bird's-eye view camera in the image acquisition unit to collect outdoor river and canal scene images, the network is trained offline using an image sample set consisting of the scene images with various target areas such as the river and canal labeled to obtain a scene target classification model. When running online, refer to Fig.10 As shown, the water body area extraction unit 1123 uses the scene target classification model to perform recognition processing on the scene image including the river channel, i.e., the river channel image, to obtain a river channel area anchor frame, and uses the area within the anchor frame as the ROI area of the river channel.
[0126] By observing the canal image, it is found that the pixel value color components of the canal water area are very close. Therefore, the river area extraction unit 1123 set in the present invention determines the threshold interval of the color space such as the three primary colors from the central part of the ROI area. Fig.10 As shown, a seed region is set in a rectangular block or a circular block near the intersection of the two diagonal lines corresponding to the ROI region rectangle, and the value intervals of the three color channels are extracted in the seed region, and the color thresholds are determined to correspond to the threshold intervals of the three primary colors. After the upper and lower limits of the grayscale values of all pixels in the central seed region are calculated, the upper and lower limits can be extended by a certain distance, such as increasing or decreasing a certain value of 5 to 20, or increasing or decreasing 5 to 10% of the median of the grayscale value interval of the channel.
[0127] In the collected low-altitude photography image, the pixels that meet the determined three-primary color threshold interval are taken as foreground objects, and the non-foreground object area is taken as the background object. All background objects in the oblique photography image are set to zero, and all foreground objects are set to 1, that is, a binary 255, to form a binary river channel image. The result is as follows Fig. 12A As shown. Fig. 12A After extracting the contour of the binary image obtained in the image, mark it in the original image and get the following Fig. 12B Compare Fig. 12B and Fig. 11B It can be seen that among the contours obtained by the former, in addition to the complete river and canal water body contours in the river and canal part, there are also many small contours formed in the slope and land areas because the grayscale values of individual pixels are close to the water body.
[0128] To this end, the features in the image are analyzed and it is found that in the binary canal image, the water part appears as a large continuous area. Therefore, for the obtained canal image, all connected domains are detected according to the connectivity of the foreground object pixels and the largest connected domain is selected as the canal water area; at the same time, Fig. 12CAs shown in , we can also remove the fragmented target-like regions by performing morphological opening and closing operations on the binary image to reduce the operation of the connected domain search. The obtained maximum connected domain is marked in the original river channel image in the form of contour lines. The result is shown in Fig.12D shown.
[0129] Based on the binary canal image, only the foreground pixels in the canal water area are used as foreground objects to form a binary canal water image. The result is as follows: Fig.13A shown.
[0130] exist Fig.13A In the binary river and canal water image shown in the figure, the boundary of the water area is affected by shadows or water penetration, resulting in some non-foreground pixels in the water body and some broken edges on the right bank that penetrate into the slope area. For this type of image, morphological operations can be used for further processing. Fig.13A , perform a morphological closing operation of dilation followed by erosion, and obtain Fig. 13B The binary image shown in Figure 2 has some interference points eliminated. Fig. 13B After the binary image is contour extracted, it is marked in the original image, and the result is as follows: Fig. 13C As shown in the contour map. Thus, a binary image with the river water body as a single target is obtained, forming a river target image.
[0131] See also Figure 3C As shown, in the geometric feature extraction module 113, an edge detection unit 1131, a straight line extraction and screening unit 1132, a water edge extraction unit 1133, a slope feature extraction unit 1134, and a distance feature extraction unit 1135 are respectively set.
[0132] After the image content is concentrated on the river and canal water area, the edge detection unit 1131 performs edge extraction on the binary river and canal target image. The edge information in the image is often contained in the area where the grayscale values of adjacent pixels in the image change dramatically. In digital image processing technology, there are many edge detection methods, such as using edge detection operators such as Sobel, Canny, and LOG. The edge can be directly extracted from the binary river and canal target image. For the color or grayscale river and canal target image, the color image is first converted into a single-channel image, and then the edge is detected using an edge detection operator.
[0133] The Canny operator is a second-order derivative edge detection operator. In a series of operation combinations, it uses Gaussian smoothing filtering, gradient amplitude and direction calculation, non-maximum suppression, and hysteresis threshold processing. Compared with the first-order derivative edge detection operator, the Canny operator can find an optimal edge and filter out most of the noise edges. The accuracy of edge detection and edge positioning are greatly improved. The present invention uses edge detection operators such as Canny to detect the edge of the water body and form a river channel edge image. The result is as follows: Fig.14A shown.
[0134] In the canal edge image, the most noteworthy thing is the straight line representing the boundaries on both sides of the canal. Hough line transform is to map the points in the image space to the curves in the parameter space, and then find the intersection of the curves in the parameter space to detect the straight line. In the image space, the equation of the straight line can be expressed as: y = kx + b; using polar coordinate representation, the equation of the straight line can be expressed as: ρ = xcosθ + ysinθ, where ρ is the vertical distance from the straight line to the origin of the coordinates, and θ is the angle between the perpendicular line of the straight line and the positive direction of the x-axis. The essence of Hough line transform is to establish a mapping relationship between the image space and the parameter space. On the basis of making full use of the global characteristics of the image, the point-line duality characteristics are used to map the pixel coordinates to the straight line represented by the elements in the parameter space, and then analyze it in the parameter space. The detection task is completed by cumulative statistics in the parameter space.
[0135] The straight line extraction and screening unit 1132 extracts the straight line boundary of the canal water surface from the canal edge image, and performs Hough line transformation on the canal edge image to obtain a canal edge straight line set. From the obtained canal edge straight line set, the water edge extraction unit 1133 screens out two straight line segments with close direction angles, the largest mutual distance and the longest length as the two sides of the canal water surface according to the heuristic rule. The result is as follows Fig. 14B shown.
[0136] See also Fig. 14B As shown, the mutual distance between the two straight line segments is the minimum value of the actual distance between each point on any one of the two straight line segments and each point on the other line segment. The mutual distance can be obtained by traversing and searching the distances between the points on the two straight line segments; and through analysis, it is found that the search process can be simplified. Specifically, the mutual distance is further defined as the minimum value of the four actual distances composed of the actual distances between the endpoints of any line segment and each point on the other line segment. Furthermore, the actual distance d from a point to a target line segment is calculated as follows: assuming that the line connecting the point to any endpoint of the target line segment forms an obtuse angle with the target line segment, the actual distance d is taken as the distance from the point to the endpoint; otherwise, the actual distance d is taken as the vertical distance from the point to the target line segment.
[0137] like Fig. 14B As shown in FIG. 1 , the water edge extraction unit 1133 finds two straight lines marked as 2 and 1 respectively on the edge of the river water body. The two straight line segments found are used as the straight lines on the two sides of the river water surface and marked in the river image. The result is as follows: Fig.15A As shown; it can be seen that through the above processing, the water surface edges on both sides of the river and canal water bodies, namely the water bank lines, can be accurately found.
[0138] For the two extracted river surface edges, the slope feature extraction unit 1134 and the distance feature extraction unit 1135 respectively calculate the slope of the straight line and the vertical distance from the center point of the image to the straight line. Figure 7 Schematic diagram of the width constraint conditions of the canal water surface model, in which the two sides of a canal water surface are shown on the left and right sides of the image plane. Among them, the angle and slope of the straight line on the two sides of the left canal water surface are α(k1) and β(k2), respectively, and the perpendicular distances from the center point of the image to the two sides of the canal water surface on this side are D1 and D2 respectively; correspondingly, the angle and slope of the straight line on the two sides of the right canal water surface are α′(k1′) and β′(k2′), respectively, and the perpendicular distances from the center point of the image to the two sides of the canal water surface on this side are D1′ and D2′ respectively. In the oblique photography image plane, the two parallel lines will intersect at a point O at the far end, and point O is called the vanishing point.
[0139] The algebraic equation of a line can be expressed in three common forms, namely the general form, the two-point form, and the slope-intercept form. The general form is: Ax+By+C=0, where A and B are not zero at the same time, and the slope of the line k=-A / B. In the Cartesian rectangular coordinate system, the intercept of the line on the yy axis b=-C / B, and the general form applies to all line equations.
[0140] The two-point formula is: The two-point equation is the equation of the straight line passing through the two points when the coordinates of the two points are known. The slope of the straight line is k = (y2-y1) / (x2-x1). It can be seen from the two-point equation that this equation is suitable for describing straight lines that are not perpendicular to the x-axis and y-axis.
[0141] The slope-intercept form is: y=kx+b. The slope-intercept form explicitly expresses the slope k and intercept b of the straight line, and is suitable for representing straight lines that are not perpendicular to the x-axis.
[0142] The straight line detected by Hough line transform in the image is described by the coordinates of the two end points of the straight line. The coordinates of the two end points are marked as (x1, y1) and (x2, y2). Substituting the coordinates of the two points into the two-point equation, and then transforming the two-point equation into the slope-intercept equation form, we get: The corresponding slope is k=y1-y2 / x1-x2. The only difference is that the calculation of the slope of the straight line is completed in the image coordinate system (the y-axis is downward in the positive direction), so the slope of the straight line in the image is equal to that in the Cartesian coordinate system, but has the opposite sign.
[0143] Combined with the straight line equation, the vertical distance from the center of the image to the edge of the canal water surface, i.e., the canal water surface edge or water bank line, can be calculated as: Wherein, (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is a constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1(x2-x1). According to the calculation formulas of k and d, the slopes k1 and k2 of the straight lines on both sides of the canal water surface and the vertical distances D1 and D2 from the center point of the image to the two straight lines are calculated respectively, and a total of four quantities are obtained as the geometric features of the canal water surface in the canal image.
[0144] like Fig. 8A , Figure 8B The schematic diagram of the canal edge line screening and geometric feature extraction in the canal water surface model image is given. The intersection of the diagonal lines indicates the center point of the image. In the set of detected straight lines, the lengths of two straight line segments are much longer than the other straight lines, so they are screened as the two edge lines of the canal. It can also be seen that the angles of the two canal water surface edge lines are close. In 8A, Figure 8B In the above example, after excluding the short straight lines, for the two selected edge lines, a perpendicular line is drawn from the center point of the image, represented by a dotted line. Figure 8B In the figure, since the foot of the perpendicular is outside the range of the side lines, extend the side lines to the foot of the perpendicular.
[0145] In order to reveal the nonlinear mapping structure of the canal water surface width prediction model of the present invention, a canal water surface scale model is established and then the model is obliquely photographed to obtain training image samples. Figure 5 As shown in the figure, when the drone takes a low-altitude photo of the ground, the angle between the optical axis of the onboard lens and the horizontal plane is α, which can be selected from a commonly used working angle such as 45 degrees. Since the width of actual rivers and canals varies greatly, it is difficult to obtain uniformly spaced standardized template river and canal sections. To facilitate modeling, the present invention uses rectangular sheets to make river and canal water surface models of various specifications, such as Fig. 6A shown.
[0146] In an offline state, a drone is used to collect images of the produced river channel water surface model and form a sample set, and the sample set is used to train the river channel water surface width prediction model. The river channel water surface model uses a rectangular thin sheet, the width of the rectangular thin sheet is a first preset multiple S1 of the actual river channel water surface width, and the length of the rectangular thin sheet is a second preset multiple S2 of its width, so as to form a model that conforms to a segment in the actual river channel. The multiples S1 and S2 are set according to the cruising flight altitude of the drone and the camera resolution; the different rectangular thin sheets used in the river channel water surface model have a width range corresponding to S1 times the common width range of the river channel, and the width difference between different rectangular thin sheets, that is, the spacing, corresponds to S1 times the preset river channel water surface width difference SD. For example, when the flight altitude is 100 meters, the corresponding sampling height of the model is 2 meters, so as to ensure that the river channel water surface model can occupy a significant image space during imaging, so as to coordinate the image resolution and the recognition accuracy of the river channel water surface width prediction model, and make it unnecessary for the drone to specially lower the altitude for predicting the river channel water surface width, thereby improving the applicable scope of the prediction.
[0147] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 30, and the minimum value of the second preset multiple S2 is between 3 and 7. Preferably, the preset river channel water surface width difference SD is 0.2 to 0.7 meters, and the width of each rectangular sheet used as the river channel model can form an arithmetic progression with S1 times SD as the tolerance.
[0148] A river surface model was built under laboratory conditions, and the actual aerial photography height h real and the actual water surface width w real Scale proportionally to obtain the aerial photography height h under laboratory conditions model and water surface width W model In order to find the relationship between the water surface features of the river channel in the image and the width of the river channel, this embodiment selects the actual aerial photography height h of the drone real When they are 30m, 40m, and 50m respectively, the corresponding aerial photography height h under laboratory conditions model The corresponding sizes are 667mm, 889mm, and 1111mm, and the scaling ratios are:
[0149]
[0150] When building the canal water surface model in the laboratory, we considered the flat riverbed in most cases, that is, the natural slope of the canal is within 3 degrees. model With h real The scaling ratio is 1:45. Because it is proportional scaling, the width of the river surface model is w model The actual width of the canal w realThe ratio is also 1:45, and then the river and canal water surface models of different widths are made with a step size of 10.5mm. The step size of the model corresponding to the actual river and canal water surface width is 0.5m. Fig. 6A As shown. Among them, the width of the river channel water surface model w model The 105mm and 210mm river and canal water surface models are marked.
[0151] Since the image sampling of the canal water surface model is low-altitude oblique image, in order to ensure the diversity of the features representing the canal water surface model in the image as much as possible and improve the generalization ability of the canal water surface width prediction model network, the canal water surface model is photographed from 0 degrees to 180 degrees. At the same time, in order to highlight the distance relationship between the drone and the actual position of the canal, the canal water surface model is arranged in different areas of the ground field of view, so that the canal water surface model in the sample set is distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane. model = 136.5 mm as an example, some of the captured data sets are as follows Figure 6B As shown, in order to improve the accuracy of the model, the photo was taken with a black ground as the background, and for the convenience of display, the picture was inverted.
[0152] The present invention subdivides the size of the river channel water surface model to form a width sequence, and shoots from various relative azimuth angles during the sampling process of the river channel water surface model image, thereby ensuring the sufficiency of the sample, thereby improving the generalization ability of the neural network and reducing the prediction error of the river channel water surface width.
[0153] For the canal image taken based on the canal water surface model, the collected image is filtered and grayed, then binarized to form a canal target image, and then the canal edge image is obtained by edge detection. Taking the canal water surface model image at a certain location taken at the laboratory sampling height h = 667mm as an example, the preprocessing results of each stage are as follows Figure 6C As shown in the figure, the left and right correspond to the processing results after binarization and edge detection. The Hough line transform is performed on the canal edge image after Canny edge detection. The transformed image is shown in the figure. Fig.6D As shown in the figure, the left and right figures are the Hough line transform results of the canal water surface model with a width of 105mm and 136.5mm respectively. After that, a heuristic search is performed on the straight lines obtained after the transformation, and the two straight line segments with the longest length are selected as the two straight line edges of the canal, that is, the straight lines on both sides of the canal water surface. Fig.6D In the figure, there are two long straight lines after excluding the short line close to the horizontal direction.
[0154] See also Figure 7As shown, for the two side lines obtained in each model image, the slopes k1 and k2 and the vertical distances D1 and D2 from the center of the image to the two side lines are calculated, and the corresponding actual model canal water surface width is recorded as the true value of the canal water surface width. The geometric feature quantities and the true value of the canal water surface width are formed into samples. After the samples are collected and recorded to form a sample set, they are input into the canal water surface width prediction model based on the neural network for offline training. Preferably, the sample set can be stored in the form of a data table composed of input and output true values as a record.
[0155] The BP neural network used in the river channel water surface width prediction model in the present invention is as follows: Figure 4A As shown, the model is a three-layer structure including a hidden layer, in which
[0156] The output of the jth node in the hidden layer is
[0157] The output of the output layer node is
[0158] In the formula, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.
[0159] When predicting the width of the canal water surface online, the width of the canal water surface is predicted based on the four geometric features obtained from the canal water surface model image. The prediction model then maps it back to the actual canal water surface width as the actual output value based on the ratio of the model to the actual shooting height.
[0160] Preferably, a plurality of the neural network-based river channel water surface width prediction models are created in the host unit, each of which corresponds to a flight altitude and a bird's-eye view angle of the image acquisition unit; when predicting the river channel water surface width online, the geometric features obtained after the current acquisition process are input into the river channel water surface width prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the river channel water surface width prediction value. In this embodiment, the shooting heights h of the river channel water surface model are selected to be 667mm, 889mm and 1111mm respectively, and the shooting angle is 45 degrees.
[0161] The sample data sets with h of 667mm, 889mm and 1111mm were randomly divided into training sets and test sets. Both training sets and test sets covered 17 different river and canal water surface models. The geometric features of the test set were input into the trained network model as input to obtain the predicted value. The fitting comparison curves of the predicted value and the true value of the river and canal water surface model width at different heights are shown in Figure 1. Fig. 9A , Fig. 9B As shown, their h are 889mm and 1111mm respectively. The test set performance is shown in Table 1.
[0162] Table 1 Test performance of the canal water surface width prediction model
[0163]
[0164] Analyze the regression results at three different laboratory heights and the coefficient of determination R at three different heights 2 The values of the root mean square error (RMSE) at three different heights are 2.2156, 2.0322, and 1.8352, respectively, indicating that the predicted value is close to the true value and the network model has good generalization ability.
[0165] Example 2
[0166] Based on the previous embodiment, Figure 2 As shown, this embodiment provides a low-altitude photography river and canal water surface width identification system 1000 using the method of the present invention, which includes: an image acquisition unit 500 for performing image sensing acquisition of river and canal scenes, a user interface unit 200 for parameter input and operation and display of river and canal water surface width information, a server 400 for data management and information transmission, a handheld terminal 300 for remote operation and interaction of river and canal width information, and a host unit 100 connected to the user interface unit 200, the server 400, and the image acquisition unit 500.
[0167] The host unit 100 includes an input and output module 130, a processor 110 and a storage module 120, and is configured to: obtain a low-altitude photography river channel image, determine a ROI region containing the river channel in the river channel image, determine a color threshold according to the value intervals of the three color channels extracted from the central part of the ROI region, use pixels whose colors meet the color threshold range as foreground objects, use non-foreground object areas as background objects, and set all background objects in the river channel image to zero to form a river channel binary image; detect all connected domains according to the connectivity of the foreground object pixels The largest connected domain is screened out as the canal water body area, and only the canal water body area is used as the foreground object to form a canal target image; the canal target image is subjected to edge detection to obtain a canal edge image; the canal edge image is subjected to Hough line transformation, and a canal water edgeline image containing two straight edges of canal water is obtained through heuristic search; the geometric features of the straight lines on both sides of the canal water surface relative to the image center are obtained from the canal water edgeline image, and the geometric features are input into a trained neural network-based canal water surface width prediction model to obtain a canal water surface width prediction value, and the width prediction value is output.
[0168] Preferably, the low-altitude photography river channel water surface width identification system 1000 may not include the server 400 and the handheld terminal 300, but the low-altitude photography river channel water surface width identification system is connected to the server 400 and the handheld terminal 300 through a network when in use. The low-altitude photography river channel water surface width identification system 1000 may also be provided with a positioning unit 700 for acquiring coordinates.
[0169] Preferably, the low-altitude photography river channel water surface width identification system 1000 also includes a flight control platform 600. In response to the user's operation on the operation panel 220 or the instruction of the host unit 100, the flight control platform 600 executes the instruction to fly and turn toward the specified direction or river channel point.
[0170] The low-altitude photography of the river scene is continuously performed by the drone flight, and the low-altitude photography river channel water surface width identification system 1000 iteratively continuously collects images, and detects the river width based on the collected images. Preferably, according to the flight speed and frame rate, the overlapping range of the river channel scene images processed successively is not less than the set ratio. Preferably, the splicing of multiple images can also be achieved by aligning feature points, and the river channel water surface width is identified based on the spliced images.
[0171] Preferably, after collecting oblique photographic images of river scenes in sections, the ratio of the width in each section to a preset width threshold at that location is calculated; when the ratio value is greater than a set high ratio threshold, an alarm is issued to indicate that the river width at that location exceeds a warning level and flood prevention is required; and when the ratio value is less than a set low ratio threshold, an alarm is issued to indicate that the water surface width at that location is too small and drought prevention or possible water leakage is required.
[0172] Preferably, after detecting the width of the river channel in sections, the width information of each section of the river channel is also marked at the corresponding position of the river channel image, such as by using an arrow line to mark between the two straight lines of the water bank of the section of the river channel; preferably, the acquired coordinate and position information, as well as the water surface width information of the river channel section, are also stored in a list of the storage module, and the list information can be stored in a cloud server.
[0173] Preferably, when the drone is used in a group system of multiple machines and multiple terminals, the system may have multiple handheld terminals and drone platforms accordingly. Through the server, the host unit and the remote client deployed at the handheld terminal, etc. can send messages about the width of the water surface of the canal. Preferably, an MQTT server can be deployed, and the server pushes messages to the clients that subscribe to the topic. Multiple devices can subscribe to multiple topics at the same time, and when publishing messages, those who subscribe to the same topic can receive the same message. The publish-subscribe model is a messaging model that decouples the client that sends the message, i.e., the publisher, from the client that receives the message, i.e., the subscriber, so that the two do not need to establish a direct connection or know each other's existence.
[0174] The server provides subscription and publishing services, and the host unit is also configured to: subscribe to the topic of obtaining river and channel water surface information of the handheld terminal, and in response to the topic message, transmit the detected river and channel water surface width and the corresponding coordinates and position information to the handheld terminal through the server; or when it is detected that the river and channel water surface width reaches a set width threshold, transmit the river and channel water surface width and the corresponding coordinates and position information to the server in the form of a width event topic message.
[0175] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the results of river channel water surface width identification in the form of a list; the host unit also displays the identified river channel water surface width in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the identified width on the curve.
[0176] Preferably, the positioning unit in the system also obtains heading information, and also records and identifies in the output position information the direction identified based on the heading, such as the north direction, and the direction corresponding to the center line of the canal between the two side lines of the canal.
[0177] Preferably, the host unit can also communicate directly with the handheld terminal. The host unit can be a local control unit deployed separately on the ground, or a control unit deployed on the drone platform. When deployed on the drone platform, the host unit on the drone platform performs edge computing for online river and canal water surface width prediction.
[0178] In the first embodiment, in order to find the actual river water body part from the ROI rectangular block area, this embodiment also uses the flood filling method for extraction. The water body area extraction unit of the water body area extraction module selects a seed point from the middle of the obtained ROI area, and extracts the river water body area from the oblique photography image through the flood filling process.
[0179] The flood filling method is to find the same area based on the difference between the gray values of pixels to achieve segmentation. The gray value of the image is understood as the height of the pixel point, so that an image can be seen as a rugged ground or mountain. A certain amount of water is poured into a low place on the ground, and the water will cover the area below a certain height. The flood filling method uses this principle and is an image processing method based on seed points. It starts from the seed point and spreads to the adjacent pixels, and marks the pixels that meet the conditions as the specified color or value.
[0180] By observing the canal image, we found that the pixel values of the canal water area are very close. Therefore, we set seed points in the canal area and select the canal area by flood filling. Fig.10 As shown, the target classification model based on the deep learning network locates the canal ROI area with the anchor frame, i.e., the rectangular frame therein. Then, a seed point is set at the intersection of the two diagonals corresponding to the rectangular frame, and the canal water body area is obtained by flood filling processing. After the pixels outside the obtained canal water body area are set to zero, a binary canal water body image is formed.
[0181] In the area of Example 1, this embodiment also establishes another canal water surface width prediction model. In the BP neural network, the input layer receives six input quantities, namely, the slopes k1 and k2 of the straight lines on both sides of the canal water surface, the vertical distances D1 and D2 from the center point of the image to the straight lines on both sides of the canal water surface, and the flight altitude and overlooking angle during acquisition, from the general processing module. The output of the output layer is the canal water surface width. After establishing a neural network with this structure, the input quantity of each sample in the training set samples is supplemented, and after supplementing enough samples at different heights and overlooking angles, it is trained with a new training sample set. When applied online, the new model can further simplify the image sampling requirements for canal prediction; it uses the generalization performance of the neural network to reduce the number of models required for canal water surface width prediction, and also simplifies the system structure, so that one model can cover different flight sampling scenarios, further improving the applicability of the system.
[0182] Example 3
[0183] Based on the above embodiments, this embodiment provides a flight control method for identifying the width of a river or canal by low-altitude photography, comprising the following steps:
[0184] R1. Initialization, establishing a canal water surface width prediction neural network to map the geometric features of the canal water surface edge straight line in the low-altitude photography image to the canal water surface width, and training the canal water surface width prediction neural network after collecting training samples to obtain a trained canal water surface width prediction model;
[0185] R2. In response to the command of river and canal water surface width identification during flight, low-altitude oblique photography of the field river and canal scene is performed to obtain river and canal images;
[0186] R3, determining a ROI region containing a river channel in the river channel image, determining a color threshold from the value intervals of the three color channels extracted from the central part of the ROI region, taking pixels whose colors meet the color threshold range as foreground objects, and setting all non-foreground object areas in the river channel image to zero to form a river channel binary image;
[0187] R4, according to the connectivity of the foreground object pixels, all connected domains are detected and the largest connected domain is selected as the canal water body area, and only the canal water body area is used as the foreground object to form a canal target image;
[0188] R5. For the canal target image, after edge detection is performed to obtain the canal edge, Hough line transform is performed, and a canal water edge line image containing two straight edges of the canal water surface, i.e., the edge line, is obtained through heuristic search;
[0189] R6. Calculate the geometric feature quantity of each of the two edge lines relative to the image center according to the two edge lines obtained from the canal water edge line image;
[0190] R7, inputting the obtained geometric feature quantity into a river channel water surface width prediction model to obtain a river channel water surface width prediction value, and outputting the width prediction value;
[0191] R8, determine whether the current water surface width exceeds the preset river channel water surface width threshold range of the segment, if so, issue an over-limit alarm;
[0192] R9, determine whether to exit the river channel water surface width identification task, if yes, go to step R11, otherwise go to step R10;
[0193] R10, flying along the direction of the two edge lines of the river channel water surface, when the overlap range between the image in the current field of view and the river channel image in the previous frame is less than a certain ratio, issuing a river channel water surface width identification instruction, and going to step R2;
[0194] R11. End the flight control for judging the width of river and canal water surface through low-altitude photography.
[0195] Combination Figure 2 , Figure 3A , Figure 4B As shown, a low-altitude photography river channel water surface width identification flight controller 100 using a low-altitude photography river channel water surface width identification flight control method of the present invention includes a processor 110, a storage module 120 and an input and output module 130. The processor 110 includes a general processing module 111, a water body area extraction module 112, a geometric feature extraction module 113, an iterative learning module 114 and a river channel water surface width prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the identification of the river channel in the collected river channel image, schedules the identification processing flow, inputs the real-time collected river channel image into the water body area extraction module 112, and outputs the width data identified and predicted by the river channel water surface width prediction module 115, and sends the flight direction instruction to the flight control platform through the output unit.
[0196] Combination Figure 3A , Figure 4B As shown, the river channel image is input into the controller by the input unit 131 in the input-output module 130; the river channel water surface width prediction model is established in the river channel water surface width prediction module 115, and its input layer receives the geometric features of the river channel in the river channel image extracted by the geometric feature extraction module 113 from the general processing module 111, and the output layer is transmitted to the iterative learning module 114 and the general processing module 111 through the first connection matrix 1141 and the second connection matrix 1142 respectively.
[0197] When the river channel water surface width prediction model is trained offline, the iterative learning module 114 adjusts the connection weights of the network according to the actual value of the river channel water surface width and the output value of the neural network in the model respectively input by the general processing module 111 and the river channel water surface width prediction model through the first connection matrix 1141; in the field environment, the first connection matrix 1141 is disconnected, and the neural network of the river channel water surface width prediction model predicts the water surface width in the current river channel image and outputs it to the general processing module 111 through the second connection matrix 1142, and is processed and analyzed by the general processing module 111 and output through the output unit 132 in the input and output module 130.
[0198] The steps of iterative training include: STEP1, initialize the weights and biases of the network; STEP2, read the network parameters and the training sample set written in the document above; STEP3, normalize the sample data; STEP4, calculate the error between the predicted output value and the expected value of each sample; STEP5, reversely calculate and correct the network weights and biases; STEP6: if the conditions for the end of network training are met, end the training, otherwise go to STEP4 to continue training; STEP7, analyze and output the results.
[0199] The general processing module 111 in the controller also obtains the coordinate information and orientation information of the current location of the river channel through the positioning unit, and after determining the water surface width of the river channel, outputs the water surface width of the river channel and the corresponding coordinate and orientation information through the output unit 132.
[0200] like Fig. 15B As shown, after the water surface width of the canal is identified, the water surface width of the canal is also marked in the corresponding area segment of the canal image, and the width direction line segment is marked with an arrow line. Preferably, the acquired coordinates and position information, as well as the water surface width of the canal, are also stored in a list of the storage module 120, and the list information can be stored in the cloud server 400.
[0201] When the water surface width exceeds the upper and lower limits of the preset river channel water surface width threshold, an alarm is issued to prompt that the river channel water surface width exceeds the limit. During the flight of the drone, low-altitude image acquisition is iteratively performed continuously, and width prediction processing is performed based on the acquired images in each preset section. When the width of any preset section is greater than its corresponding preset width threshold, a water surface width exceeds the limit prompt is issued. Preferably, according to the flight speed and frame rate, the overlapping range of the river channel images processed successively is not less than the set ratio.
[0202] The flight control method for identifying the width of the river and canal water surface by low-altitude photography of the present invention is also applied to the water surface width monitoring flight of the approximately parallel two-bank sections that most rivers have. To this end, the step R2 also obtains the inverse perspective transformation matrix according to the height and pitch angle sampled when the river and canal scene is photographed at low altitude; in the step R4, the binary river and canal target image is converted into a real binary image of the river and canal based on the inverse perspective transformation matrix; in the step R5, the real binary image of the river and canal is subjected to edge detection and Hough line transformation, and a canal water edge line image containing two straight edges, i.e., edge lines of the river and canal water surface is obtained through heuristic search; then, it is calculated whether the angles or slopes of the two edge lines of the river and canal water surface are close, if so, it is transferred to step R6, otherwise it is marked that the river and canal water surface edge lines are not parallel and it is transferred to step R10; in the step R10, it is first determined whether the river and canal water surface edge lines are parallel, if not, it continues to fly along the river and canal in the parallel direction of the angle bisectors of the two edge lines of the river and canal water surface, otherwise it flies along the direction of the two edge lines of the river and canal water surface.
[0203] Preferably, in step R2, multiple images may be stitched together by aligning feature points, and the stitched image may be used as the current river channel image.
[0204] Preferably, in step R8, the ratio of the water surface width of the river channel in the current segment to a preset width threshold at that location is calculated, and when the ratio value is greater than a set high ratio threshold, an alarm is issued to indicate that the river width at that location exceeds the warning level and flood prevention is required; and when the ratio value is less than a set low ratio threshold, an alarm is issued to indicate that the water surface width at that location is too small and drought prevention or possible water leakage is required.
[0205] The present invention is applied to identify the width of river and canal water surface in the space of UAV low-altitude photography images. Through automatic identification of river and canal targets, water target seed points or areas are extracted to automatically extract river and canal water bodies. After regional segmentation, river and canal edge detection and water bank line extraction, the slopes of the straight lines on both sides of the river and canal water surface and the vertical distances from the image center point to the straight lines on both sides of the river and canal water surface are used as geometric features and input into the established neural network-based river and canal water surface width prediction model. The model can accurately predict the actual width of the river and canal water surface as basic hydrological data, providing a basis for river and canal flow monitoring and drought and flood warning.
[0206] Several embodiments of the present invention are described above, but these embodiments are presented as examples and do not limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These embodiments or their modifications are included in the scope or main purpose of the invention, and are also included in the invention described in the claims and their equivalents.
Claims
1. A method for identifying the width of a river or canal using low-altitude photography, comprising the following steps: S1. Initialization, establishing a river channel water surface width prediction model based on a neural network, wherein the input of the neural network is the geometric features of two parallel straight lines relative to the center of the image, and the output is the actual distance between the two parallel straight lines; S2, constructing a thin sheet-shaped river channel water surface model with two parallel edges, collecting images of the river channel water surface models with different widths by low-altitude oblique photography in an offline state to form a sample set, and using the sample set to train the neural network; S3, collecting river and canal images online at a certain height and pitch angle by low-altitude photography, and determining a ROI region including the river and canal in the river and canal images; S4, according to the color space characteristics of the central part of the ROI area, taking the pixel points with similar color space characteristics in the river channel image as foreground objects, and taking the non-foreground object area as background objects; S5. Among the foreground objects, according to pixel connectivity, find the largest connected domain as the canal water body area, take the canal water body area as the foreground object, and set all non-foreground objects to zero, so as to form a canal target image; S6, obtaining a river channel edge image by edge detection on the river channel target image; S7, after performing Hough line transformation on the river channel edge image, obtaining straight lines on both sides of the river channel water body surface through heuristic search; S8, obtaining geometric features of the two straight lines on the water surface of the canal relative to the center of the image; S9, inputting the geometric features into the trained river channel water surface width prediction model to obtain a river channel water surface width prediction value; S10, outputting the width prediction value; S11, check whether there is an exit instruction, if yes, go to step S13, otherwise go to step S12; S12, after the width identification instruction is triggered, go to step S3; S13, ending the identification of the width of the river channel.
2. The method for identifying the width of a river or canal using low-altitude photography according to claim 1, characterized in that: The step S2 also includes: The collected image of the river channel water surface model is filtered and grayed out, then binarized to form a river channel target image, and then edge detection is performed to obtain a river channel edge image; Perform Hough line transform on the canal edge image, and obtain the edge image containing two straight edges of the canal water surface model through heuristic search; The geometrical feature quantities of the two edge lines relative to the image center are obtained from the edge line image, the width of the current river channel water surface model is recorded, and the obtained geometrical feature quantities and the recorded model width are formed into a training sample.
3. The method for identifying the width of a river or canal using low-altitude photography according to claim 1, characterized in that: The geometric features in step S1 include four input quantities, namely, the slopes k1 and k2 of two straight lines and the vertical distances D1 and D2 from the center point of the image to the two straight lines; The geometric features in step S8 include four input quantities, namely, the slopes k1 and k2 of the two straight lines on the water surface of the canal and the perpendicular distances D1 and D2 from the center point of the image to the two straight lines.
4. The method for identifying the width of a river or canal using low-altitude photography according to claim 1, characterized in that: After the Hough line transformation in step S7, a canal edge straight line set is obtained; in the heuristic search, two straight line segments with close direction angles, maximum mutual distance and longest length are selected from the obtained canal edge straight line set according to the heuristic rule as the straight lines on both sides of the canal water surface. The mutual distance is the minimum value of the actual distance between each point on any one of the two straight line segments and each point on the other line segment; wherein, the calculation method of the actual distance d between a point and each point on a target line segment is: if the line connecting the point to any end point of the target line segment forms an obtuse angle with the target line segment, then the actual distance d is taken as the distance from the point to the end point; otherwise, the actual distance d is taken as the perpendicular distance from the point to the target line segment, that is, the vertical distance.
5. The method for identifying the width of a river or canal using low-altitude photography according to claim 1, characterized in that: The step S1 further includes: establishing a target classification model based on a YOLOv5s network, using a bird's-eye view camera in an image acquisition unit to capture river and canal scene images through low-altitude photography, and then training the network offline using an image sample set consisting of the river and canal scene images with the river and canal areas marked, to obtain a river and canal scene target classification model; In the step S3, the collected canal image is identified and processed by the canal scene target classification model to obtain a canal region anchor frame, and the area within the anchor frame is used as the canal ROI area.
6. The method for identifying the width of a river or canal using low-altitude photography according to claim 1, characterized in that: The BP neural network used in the river channel water surface width prediction model has a three-layer structure including a hidden layer, wherein: The output of the jth node in the hidden layer is The output of the output layer node is Among them, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.
7. The method for identifying the width of a river or canal using low-altitude photography according to claim 1, characterized in that: The geometric characteristics of the straight line are calculated as follows: Assuming that the coordinates of the two endpoints of the straight line are (x1, y1) and (x2, y2), then its slope is k = y1y2 / x1-x2; The perpendicular distance from the center of the image to the straight line is Where (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is a constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1(x2-x1); According to the calculation formulas of k and d, the slopes k1 and k2 of the two straight lines and the perpendicular distances D1 and D2 from the center point of the image to the two straight lines are calculated respectively, a total of four quantities are obtained as geometric features.
8. The method for identifying the width of a river or canal using low-altitude photography according to claim 1, characterized in that: In the step S2, training the neural network as a river channel water surface width prediction model includes: T1, initializing the network weights and biases; T2, reading the network parameters and the training sample set recorded in the document; T3, normalizing the sample data; T4, calculating the error between the predicted output value and the expected value of each sample; T5, correcting the network weights and biases by reverse calculation based on the error; T6, if the conditions for the end of network training are met, then end the training, otherwise go to T4 to continue training; T7, result analysis and output.
9. The method for identifying the width of a river or canal using low-altitude photography according to claim 1, characterized in that: The step S2 also includes: arranging the river and canal water surface models in different areas of the ground under the oblique photography field of view, so that the river and canal water surface models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane.
10. The method for identifying rural road width using a drone according to claim 1, characterized in that: The neural network used as the canal water surface width prediction model is multiple, wherein each canal water surface width prediction model corresponds to a flight altitude and a bird's-eye view angle when the canal image is acquired by low-altitude photography; The river channel water surface width prediction model used in step S9 corresponds to the river channel water surface width prediction model neural network corresponding to the flight altitude and the bird's-eye view angle when collecting the river channel image in low-altitude photography in step S3.
