Low-altitude photography river channel water surface width identification device and system
By using neural network prediction models in low-altitude photography of drone, combining Hough line transformation and edge detection, automatically extracting the canal water body area and predicting the width, the flexibility and accuracy of canal water surface width detection in the prior art are solved. Automatic identification and monitoring of drones on the water surface width of river canals is realized.
Patent Information
- Application Number
- CN202411124005.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art in the detection of the water surface width of the canal in low-altitude photography of drones, relying on the fixed angle and proportional relationship, limiting the flexibility and accuracy of the detection. Especially when the cross section of the canal is trapezoid, it is difficult to realize automatic identification and Width measurement.
The canal surface width prediction model based on neural network is adopted. The slope of the straight line on both sides of the canal surface and the vertical distance between the image center point to the straight line in the low-altitude photography image is used as geometric features. Combined with Hough line transformation and edge detection, the canal water area is automatically extracted and Conduct width prediction to achieve identification of the width of the canal surface without a fixed course.
Automatic extraction and width prediction of canal water objects and water surface shorelines is realized, with good generalization performance, and can accurately predict the canal water surface width during dry and abundant water periods, and is suitable for monitoring of approximately parallel cross-strait sections of most rivers.
Smart Images

Figure CN119027840A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned aerial vehicle detection application, and in particular relates to a device and system for identifying the width of a river or canal through low-altitude photography. Background Art
[0002] In recent years, drone technology has been widely used in many fields, especially in various detection and measurement aspects, and has shown significant advantages and potential.
[0003] Drones can fly along high-voltage lines and use high-definition cameras or thermal imaging equipment to detect damage, corrosion and other potential problems. Drones can quickly obtain image data of roads and railways, identify risks such as cracks and landslides, and perform regular maintenance.
[0004] Drones are also widely used in terrain inspection and hydrological detection. By carrying high-resolution cameras or lidar, drones can obtain high-precision terrain data and generate digital terrain models and digital elevation models. The application of drone low-altitude photography in terrain detection, especially in rivers and hydrology, provides efficient, flexible and accurate technical means, providing important support for environmental protection, water resources management and natural disaster prevention and control.
[0005] Low-altitude drone photography can easily obtain high-resolution images of rivers and their surrounding areas, helping scientists and engineers monitor changes in river morphology in real time, such as river siltation, erosion, riverbed uplift or downcutting. These data are crucial to understanding the dynamic process of rivers and their impact on the environment. Using low-altitude drone photography technology, current images of flood areas can be quickly obtained to help assess the impact of floods, changes in flood water levels, and threats to surrounding infrastructure. By comparing topographic data before and after floods, the long-term impact of floods on river channels and terrain can also be analyzed.
[0006] Among hydrological data, river surface width is the most basic data. Currently, river width is either measured on the spot using instruments or directly calculated based on pixel ratio in images.
[0007] For example, the Chinese invention patent application with application number 202310722121.6 discloses a floating river width measuring instrument, which obtains a first distance and a second distance of the measured distance from a set of radars to the river bank, and calculates the first vertical distance from the measuring instrument to the river bank according to the first angle between the two known radars; obtains a third distance and a fourth distance of the measured distance from another set of radars to the river bank, and calculates the second vertical distance from the measuring instrument to the river bank according to the second angle between the two known radars; finally, the first distance and the second distance are summed to obtain the river width distance. Similarly, the Chinese invention patent application with application number 202111392791.3 discloses a river width extraction method using a river meter, which first places the river meter on the ground on one side of the river bank, detects the river surface and the ground on the opposite bank of the river through an antenna, and obtains the position of the dividing point; then, the distance element at the dividing point is determined according to the amplitude difference between the river echo and the ground echo, as the distance between the river meter and the opposite bank; then, the distance between the river meter and the river bank on the side close to the river meter is measured, and the river width is calculated by combining these two distances.
[0008] In addition to using radar waves for detection, there are also calculation schemes for river width detection through images. For example, the Chinese invention patent application with application number 202311290498.5 discloses a flood warning method and system for wild rivers in arid areas. A water gauge is arranged near a fixed reference object in a wild river, and a fixed camera is used to capture the surface image of the wild river in real time. The image is identified and the water depth and width of the wild river are obtained with the help of the water gauge information in the image. For the river width, the scheme uses a priori proportional data to binarize the extracted image and directly read the river width from the river width binary image. However, the Chinese invention patent application with application number 202311564794.X discloses a real-time monitoring method for the ecological flow of natural rivers based on tower base video; it combines video images, searches for monitoring sections in digital orthophoto data, obtains the actual distance of the monitoring section in the digital orthophoto data, and calibrates the conversion coefficient f=W / N between the water surface width of the monitoring section and the number of water body pixels, wherein W is the actual river width of the monitoring section obtained based on the digital orthophoto data, and N is the number of water body pixels in the monitoring section; based on the conversion coefficient, the river width can be measured according to the image.
[0009] This type of image-based measurement scheme relies on an accurate proportional relationship, that is, the pixel width value of the measured line segment corresponds to a priori determined ratio. Obviously, this scheme requires the video screen to be fixed at the same angle to be comparable, and for drone low-altitude photography, this undoubtedly greatly limits its flexibility in river and canal measurement.
[0010] Therefore, there is a need for a device and system for automatic identification and width detection of rivers and canals based on low-altitude photography with strong generalization performance and wide applicability. Summary of the invention
[0011] As the river flows, it scours the riverbank, thus forming a shoreline. It has been observed that in many sections of a river, there are two nearly parallel shorelines. In particular, in rivers, there is a type of artificial or natural waterway that is built or naturally formed for specific functions, namely canals. At present, the canal system around the world is now very developed. Canals are widely distributed in plains and hilly areas, used to guide river water for large-scale farmland irrigation and discharge excess water. The cross-section of a canal is usually trapezoidal, because the trapezoidal section has good slope stability, which reduces the risk of collapse of the slopes on both sides of the canal. At the same time, the selection of the bottom width and slope angle of the trapezoidal section also helps to control the flow rate and reduce sediment deposition, and the shape of the trapezoidal section is easier to dig and shape by mechanized equipment.
[0012] In order to solve the above technical problems and realize automatic identification of river and canal water surface width based on low-altitude photography, the present invention provides a low-altitude photography river and canal water surface width identification device, controller and system. The features of the river and canal are identified and their water surface width is calculated based on the images taken by the drone at low altitude photography, and there is no need to limit the drone to maintain a certain precise orientation during sampling.
[0013] The present invention provides a device for identifying the width of a river or canal using low-altitude photography, which includes a host unit and a user interface unit. The user interface unit is used for parameter input and human-computer interaction. The host unit includes an input / output module, a processor, and a storage module and is configured as follows:
[0014] Acquire a river channel image obtained by low-altitude photography, determine a ROI region containing the river channel in the river channel image, determine a color threshold according to respective value intervals of three color channels extracted from the central part of the ROI region, take pixels whose colors meet the color threshold range as foreground objects, take non-foreground object regions as background objects, and set all background objects in the river channel image to zero to form a river channel binary image;
[0015] According to the connectivity of the foreground object pixels, all connected domains are detected and the largest connected domain is selected as the canal water body area, and only the canal water body area is used as the foreground object to form a canal target image;
[0016] For the canal target image, edge detection is performed to obtain a canal edge image; after Hough line transformation is performed on the canal edge image, a canal water edge image containing two straight line edges of canal water is obtained through heuristic search;
[0017] The geometric features of the straight lines on both sides of the canal water surface relative to the image center are obtained from the canal water edge line image, and the geometric features are input into a trained canal water surface width prediction model based on a neural network to obtain a canal water surface width prediction value, and the width prediction value is output.
[0018] Preferably, the processor includes a general processing module, a water body area extraction module, a geometric feature extraction module, an iterative learning module and a river channel water surface width prediction module; the water body area extraction module obtains the river channel image taken by a drone at low altitude from an external image acquisition unit, and after filtering, identifies the position of the river channel in the image as an ROI area according to the deep learning features by a trained target classification model; a seed area is selected from the middle of the ROI area to obtain a color threshold, and the river channel water body area is extracted from the low-altitude photography image according to the obtained color threshold to form a binary river channel water body image; the general processing module responds to the message, and starts the extraction of the river channel water body area in the collected river channel image, schedules the river channel water surface width recognition process, and outputs the width data predicted by the river channel water surface width prediction model.
[0019] Preferably, a canal water surface width prediction model is established in the canal water surface width prediction module, whose input layer receives the geometric features of the canal water surface in the canal image from the general processing module, and the output layer transmits them to the iterative learning module and the general processing module through the first connection array and the second connection array respectively; when training the canal water surface width prediction model offline, the iterative learning module adjusts the connection weights of the network according to the actual values of the canal water surface width input by the general processing module and the canal water surface width prediction model through the first connection array and the output value of the neural network in the model; in the field environment, the first connection array is disconnected, and the neural network of the canal water surface width prediction model predicts the width of the canal water surface in the current canal image and outputs it to the general processing module through the second connection array, which is processed and analyzed by the general processing module and then output through the output part in the input and output module.
[0020] Preferably, the host unit is further configured as follows: a target detection unit is set in the water body area extraction module, a target classification model based on the YOLOv5s network is established in the target detection unit, and after collecting the river and canal scene image by low-altitude photography of the bird's-eye view camera in the image acquisition unit, the network is offline trained with an image sample set consisting of the river and canal scene images with the river and canal areas marked, so as to obtain a river and canal scene target classification model; when running online, the water body area extraction module uses the river and canal scene target classification model to identify and process the current scene image including the river and canal to be tested, i.e., the low-altitude photography river and canal image, to obtain a river and canal area anchor frame, and the area within the anchor frame is used as the river and canal ROI area.
[0021] Preferably, the water body region extraction module determines the color thresholds corresponding to the threshold intervals of the three primary colors according to the value intervals of the three color channels extracted from the central part of the ROI region. After the upper and lower limits of the grayscale values of each color channel of all pixels in the central seed region are calculated, the upper and lower limits can be extended by a certain distance, such as increasing or decreasing a certain value of 5 to 20, or increasing or decreasing 5 to 10% of the median of the grayscale value interval of the channel.
[0022] Preferably, the water area extraction module sets a seed point at the intersection of the two diagonals corresponding to the ROI area rectangle, obtains the river and canal water area as the foreground object through flood filling processing, sets the pixels outside the obtained river and canal water area to zero, and forms a river and canal binary image; after the maximum connected domain search is performed to form the river and canal target image, a binary river and canal edge image is formed through edge detection; after the binary river and canal edge image is subjected to Hough line transformation, the two longest straight line segments are screened out as the straight lines on both sides of the river and canal water surface.
[0023] Preferably, the binary river and canal water body image is subjected to a morphological closing operation of first dilation and then erosion to remove interference points in the connected domain of the water body. For any straight line on both sides of the river and canal water surface, the geometric feature extraction module describes it with the coordinates of the two end points of the straight line, and obtains the slope of the straight line and the perpendicular distance from the center point of the image to the straight line based on the two end points; the slope of the straight line on both sides of the river and canal water surface and the perpendicular distance from the center point of the image to the straight line are used as the geometric features of the river and canal in the river and canal image.
[0024] Preferably, the canal water surface width prediction model adopts a BP neural network, whose input layer receives four input quantities, namely the slopes k1 and k2 of the straight lines on both sides of the canal water surface and the vertical distances D1 and D2 from the center point of the image to the straight lines on both sides of the canal water surface, from the general processing module respectively, and the output of the output layer is the canal water surface width.
[0025] Preferably, the BP neural network used in the canal water surface width prediction model is a three-layer structure including a hidden layer, wherein:
[0026] The output of the jth node in the hidden layer is
[0027] The output of the output layer node is
[0028] Among them, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. jand b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.
[0029] Preferably, the host unit is further configured as follows: before online application, firstly, in an offline state, a drone is used to collect images of the produced canal water surface model and form a sample set, and the canal water surface width prediction model is trained with the sample set; the canal water surface model adopts a rectangular thin sheet, the width of the rectangular thin sheet is a first preset multiple S1 of the actual canal water surface width, and the length of the rectangular thin sheet is a second preset multiple S2 of its width, and the multiples S1 and S2 are set according to the cruising flight altitude of the drone and the camera resolution; the different rectangular thin sheets adopted in the canal water surface model have a width range corresponding to S1 times the common width range of the canal, and the width difference between different rectangular thin sheets corresponds to S1 times the preset canal water surface width difference SD.
[0030] Preferably, during the collection of the sample set, the river and channel models are arranged in different areas of the ground field of view, so that the river and channel models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane.
[0031] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 30, and the minimum value of the second preset multiple S2 is between 3 and 7. Preferably, the preset river channel water surface width difference SD is 0.2 to 0.7 meters, and the width of each rectangular sheet used as the river channel model can form an arithmetic progression with S1 times SD as the tolerance.
[0032] Preferably, for the river channel water surface image based on the river channel water surface model, the collected image is filtered and grayed, and then binarized to form a river channel target image. The binary river channel target image can be directly edge extracted, and for the color or grayscale river channel image, the color image is first converted into a single channel image, and then the edge detection operator such as Sobel, Canny, LOG, etc. is used to detect the edge.
[0033] Preferably, the geometric feature extraction module calculates and extracts geometric features of the straight lines on both sides of the canal water surface. Assuming that the coordinates of the two end points of the canal edge straight line are (x1, y1) and (x2, y2), the slope is k=y1-y2 / x1-x2; the vertical distance from the center of the image to the canal edge straight line is
[0034] Where (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is a constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1(x2-x1);
[0035] According to the calculation formulas of k and d, the slopes k1 and k2 of the straight lines on both sides of the canal water surface and the vertical distances D1 and D2 from the center point of the image to the two straight lines are calculated respectively, a total of four quantities as the geometric features of the canal water surface in the canal image.
[0036] Preferably, a plurality of the neural network-based river channel water surface width prediction models are created in the host unit, and each river channel water surface width prediction model corresponds to a flight altitude and a bird's-eye view angle of the image acquisition unit; when performing online river channel water surface width prediction, the geometric features obtained after the current acquisition process are input into the river channel water surface width prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the river channel water surface width prediction value.
[0037] Preferably, in the BP neural network of the canal water surface width prediction model, the input layer receives six input quantities, namely, the slopes k1 and k2 of the straight lines on both sides of the canal water surface, the vertical distances D1 and D2 from the center point of the image to the straight lines on both sides of the canal water surface, and the flight altitude and overlooking angle during collection, from the general processing module respectively, and the output of the output layer is the canal water surface width.
[0038] Preferably, the host unit is also configured to: obtain the coordinate information and orientation information of the current location of the river channel through the positioning unit, and after identifying the water surface width of the river channel, output the water surface width of the river channel and the corresponding coordinate and orientation information through the input and output unit.
[0039] Preferably, after identifying the width of the water surface of the river channel, the water surface width of the river channel is also marked in the corresponding area segment of the river channel image, such as by using an arrow to mark the width direction line segment; preferably, the acquired coordinates and position information, as well as the water surface width of the river channel, are also stored in a list of the storage module, and the list information can be stored in a cloud server.
[0040] When the water surface width exceeds the preset river channel water surface width threshold range, an alarm is issued. During the flight of the drone, low-altitude image acquisition is performed iteratively, and width prediction processing is performed based on the acquired images in each preset section. When the width of any preset section is greater than its corresponding preset width threshold, a water surface width exceeding limit prompt is issued. Preferably, according to the flight speed and frame rate, the overlapping range of the river channel images processed successively is not less than the set ratio.
[0041] In another embodiment of the present invention, a low-altitude photography river channel water surface width identification system is also provided, which includes: an image acquisition unit for performing image sensing and acquisition of river channel scenes, a user interface unit for parameter input and operation and display of river channel water surface width information, a server for data management and information transmission, a handheld terminal for remote operation and river channel width information interaction, and a host unit connected to the user interface unit, the server, and the image acquisition unit; the host unit includes an input and output module, a processor, and a storage module, and is configured as follows;
[0042] A low-altitude photographic river and canal image is obtained, and a ROI region containing the river and canal is determined in the river and canal image. A color threshold is determined according to the respective value intervals of the three color channels extracted from the central part of the ROI region, and pixels whose colors meet the color threshold range are used as foreground objects, and non-foreground object regions are used as background objects. All background objects in the river and canal image are set to zero to form a river and canal binary image; according to the connectivity of the foreground object pixels, all connected domains are detected and the largest connected domain is screened out as the river and canal water body region, and a river and canal target image is formed using only the river and canal water body region as the foreground object; for the river and canal target image, a river and canal edge image is obtained by edge detection; after performing Hough line transformation on the river and canal edge image, a canal water edge line image containing two straight line edges of canal water is obtained by heuristic search; geometric features of the straight lines on both sides of the river and canal water surface relative to the image center are obtained from the canal water edge line image, and the geometric features are input into a trained river and canal water surface width prediction model based on a neural network to obtain a river and canal water surface width prediction value, and the width prediction value is output.
[0043] Preferably, the low-altitude photography river channel water surface width identification system may not include the server and the handheld terminal, but the low-altitude photography river channel water surface width identification system is connected to the server and the handheld terminal through a network when in use. A positioning unit for obtaining coordinates may also be provided in the low-altitude photography river channel water surface width identification system.
[0044] Preferably, the low-altitude photography river and channel water surface width identification system also includes a flight control platform. In response to the user's operation on the operation panel or the instruction of the host unit, the flight control platform executes the instruction to fly and turn towards the specified direction or river and channel point.
[0045] Preferably, the drone is used to continuously perform low-altitude photography sampling of the river scene, during which images are continuously collected iteratively, and the river width is detected based on the collected images. Preferably, the overlapping range of the river and canal scene images processed successively is not less than a set ratio according to the flight speed and frame rate. Preferably, multiple images can also be spliced by aligning feature points, and the width of the river and canal water surface can be identified based on the spliced images.
[0046] Preferably, after collecting oblique photographic images of river scenes in sections, the ratio of the width in each section to a preset width threshold at that location is calculated. When the ratio value is greater than a set ratio threshold, an alarm is issued to indicate that the river width at that location exceeds a warning level and flood prevention is needed.
[0047] Preferably, after detecting the width of the river channel in sections, the width information of each section of the river channel is also marked at the corresponding position of the river channel image, such as by using an arrow line to mark between the two straight lines of the water bank of the section of the river channel; preferably, the acquired coordinate and position information, as well as the water surface width information of the river channel section, are also stored in a list of the storage module, and the list information can be stored in a cloud server.
[0048] Preferably, when the UAV is used in a group system of multiple machines and multiple terminals, the host unit is also configured to: subscribe to the acquisition information topic of the handheld terminal, and in response to the topic message, transmit the detected river channel water surface width and the corresponding coordinates and position information to the handheld terminal through the server; or when the river channel water surface width is detected to reach a set width threshold, the river channel water surface width and the corresponding coordinates and position information are transmitted to the server in the form of a width event topic message.
[0049] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the results of river channel water surface width identification in the form of a list; the host unit also displays the identified river channel water surface width in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the identified width on the curve.
[0050] Preferably, the positioning unit in the system also obtains heading information, and also records and identifies in the output position information the direction identified based on the heading, such as the north direction, and the direction corresponding to the center line of the canal between the two side lines of the canal.
[0051] Compared with the prior art, the scheme of the present invention has the following advantages: the present invention analyzes the characteristics of the river channel water bank line under the low-altitude photography of the UAV, analyzes the characteristic representation of the river channel water surface width in the river channel image in the image space in view of the unpredictable relative orientation relationship between the posture of the UAV and the direction of the river channel when collecting images, and proposes to use the slopes of the straight lines on both sides of the river channel water surface and the vertical distances from the center point of the image to the straight lines on both sides of the river channel water surface as constraints to establish a neural network to predict the river channel water surface width. The collected samples are used to identify and train the nonlinear mapping relationship between the river channel water surface width and the four proposed geometric features. Based on the low-altitude photography river channel image, the river channel target is located in the ROI area with an anchor frame using a target classification model, and the river channel water body target is obtained according to the color value range in the ROI area, and the river channel water body object is obtained through the maximum connected domain screening. Edge detection is performed on the canal water area, and the straight lines on both sides of the canal water surface are obtained through Hough line transform screening. The four geometric features are extracted from the two extracted edge lines, and the results are input into the trained canal water surface width prediction model to perform online prediction of the canal width.
[0052] The present invention realizes the automatic extraction of river and canal water body objects and water surface shorelines. The established river and canal water surface width prediction model based on neural network has good generalization performance and can accurately predict the water surface width of the river and canal in different periods of dry and flood water according to the river and canal images collected from uncertain navigation directions; at the same time, it can also be applied to the water surface width monitoring of the sections with approximately parallel banks that most rivers have.
[0053] It should be understood that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (provided that such concepts are not mutually inconsistent) can be considered as part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing in the disclosed solution can be considered as part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1A , Figure 1B This is a schematic diagram of a river channel scene taken from low altitude photography;
[0055] Figure 2 It is a schematic diagram of the structure of the device and system for judging the width of the river and canal water surface by low-altitude photography;
[0056] Figure 3A This is the structure diagram of the host unit / controller in the low-altitude photography river and channel water surface width identification device. Figure 3B This is the structure diagram of the water area extraction module. Figure 3C This is the structural diagram of the geometric feature extraction module;
[0057] Figure 4A This is a schematic diagram of the neural network model structure of the river and canal water surface width prediction module. Figure 4B It is a schematic diagram of the partial structure of the host unit / controller in the low-altitude photography river and channel water surface width identification device;
[0058] Figure 5 Schematic diagram of the drone’s shooting posture when collecting samples for low-altitude photography;
[0059] Fig. 6A This is a schematic diagram of the comparison of water surface models of rivers and canals with different widths. Figure 6B Schematic diagram of the river channel water surface model taken from different angles during sample collection;
[0060] Figure 6C This is a schematic diagram of the contour processing of the river and canal water surface model. Fig.6D It is a schematic diagram of the detection result after Hough line transformation;
[0061] Figure 7 The diagram is a schematic diagram of the width constraint conditions and geometric features of the river and canal water surface model proposed;
[0062] Fig. 8A , Figure 8B It is a schematic diagram of the screening of straight lines on the water surface edges of rivers and canals in different directions and the extraction of geometric features;
[0063] Fig.9A , Fig. 9B Fitting curve between actual and predicted values of the water surface model width of the test set of rivers and canals;
[0064] Fig.10 Schematic diagram of anchor frame for river and canal target identification;
[0065] Fig.11A , Fig. 11B They are schematic diagrams of direct binarization and contour extraction respectively;
[0066] Fig. 12A , Fig. 12B They are respectively based on the binarization of each color grayscale value interval and the schematic diagram of contour extraction after binarization. Fig. 12C This is a schematic diagram of the morphological processing results. Fig.12D Schematic diagram for screening the maximum connected domain;
[0067] Fig.13A , Fig. 13B , Fig. 13C They are the binary image after the maximum connected domain screening, morphological processing, and the corresponding schematic diagram of water area extraction;
[0068] Fig.14A is the edge image of the river water body, Fig. 14B Schematic diagram of the straight line extracted by Hough line transform;
[0069] Fig.15AThis is a comparison chart of the straight line detection results of the edge of the river surface. Fig. 15B This is a schematic diagram of the width of the river and canal water surface;
[0070] Fig.16 This is a schematic diagram of the segmentation of river and channel water surface width identification.
[0071] Among them: 1000 low-altitude photography river and canal water surface width identification systems, 900 low-altitude photography river and canal water surface width identification devices, 100 host units / controllers, 200 user interface units, 300 handheld terminals, 400 servers, 500 image acquisition units, 600 flight control platforms, 700 positioning units, 800 field environments / river and canal scenes,
[0072] 110 processor, 120 storage module, 130 input and output module,
[0073] 111 general processing module, 112 water area extraction module, 113 geometric feature extraction module, 114 iterative learning module, 115 river channel water surface width prediction module,
[0074] 1121 image acquisition unit, 1122 target detection unit, 1123 water area extraction unit,
[0075] 1131 edge detection unit, 1132 straight line extraction (screening) unit, 1133 water edge extraction unit, 1134 slope feature extraction unit, 1135 distance feature extraction unit,
[0076] 1141 first connection array, 1142 second connection array,
[0077] 1101 neural network model, 131 input unit, 132 output unit
[0078] 210 display screen, 220 operation panel. DETAILED DESCRIPTION
[0079] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, but the present invention is not limited to these embodiments. The present invention covers any substitution, modification, equivalent method and scheme made within the spirit and scope of the present invention.
[0080] In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but users skilled in the art can fully understand the present invention without these detailed descriptions.
[0081] The present invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. It should be noted that the accompanying drawings are all simplified and not in exact proportion, and are only used for the purpose of conveniently and clearly assisting in explaining the embodiments of the present invention.
[0082] Embodiment 1:
[0083] In my country, rivers and canals are widely distributed. In addition to collecting terrain images such as rivers and canals through traditional satellite and other high-altitude photography; Figure 1A , Figure 1B As shown, with the development of drone technology, it is now very convenient to use low-altitude drones to perform oblique photography sampling of rivers and canals. It is of great significance to identify the width of rivers and canals. To this end, the present invention discloses a low-altitude photography river and canal water surface width identification device that does not rely on prior width measurement and can estimate the width of the river and canal by real-time image collection of the river and canal under a non-fixed heading during flight.
[0084] like Figure 2 As shown, the present invention provides a low-altitude photography river and canal water surface width identification device 900, which includes a host unit 100 and a user interface unit 200; wherein the host unit 100 includes a processor 110, a storage module 120 and an input-output module 130, and the user interface unit 200 is provided with a display screen 210 and an operation panel 220. The host unit 110 is configured to: obtain a low-altitude photography river and canal image, determine an ROI area containing the river and canal in the river and canal image, determine a color threshold according to the respective value intervals of the three color channels extracted from the central part of the ROI area, take pixels whose colors meet the color threshold range as foreground objects, take non-foreground object areas as background objects, and set all background objects in the river and canal image to zero to form a river and canal binary image; according to the connectivity of the foreground object pixels, detect all connected domains and screen out the largest connected domain as the river and canal water body area, and only take the river and canal as the background area. The water body area is used as the foreground object to form a river channel target image; the river channel target image is subjected to edge detection to obtain a river channel edge image; the river channel edge image is subjected to Hough line transform, and then a canal water edge line image containing two straight edges of canal water is obtained through heuristic search; the geometric features of the straight lines on both sides of the river channel water surface relative to the image center are obtained from the canal water edge line image, and the geometric features are input into a trained neural network-based river channel water surface width prediction model to obtain a river channel water surface width prediction value, and the width prediction value is output by the output unit 132 in the input-output module 130.
[0085] like Figure 1A As shown, the cross-section of the canal is usually trapezoidal. The water flow in the canal varies greatly during the flood season and the dry season. Low-altitude photography images show that the water bank in the canal moves due to changes in flow, and the distance between the water bank on both sides, that is, the width of the water surface, also changes accordingly.
[0086] like Figure 1B As shown in the figure, the water flow in the canal is relatively evenly distributed. Although its direction is sometimes curved, it has two nearly parallel water banks in most areas. Figure 1A , Figure 1B As shown in the figure, in the canal image collected in the wild scene, the canal water surface appears to be wide at one end and narrow at the other end, and in the image, the parallel shorelines on both sides of the actual canal water surface will approximately intersect at a vanishing point at infinity. In order to obtain the river width, on the one hand, it is obvious that when the distance is measured from the pixel points on both sides of the canal in the image and then converted back to the actual width according to the proportional relationship, the proportional relationship is difficult to accurately determine, and it is directly related to the sampling position of the side pixel points. On the other hand, in actual applications, it is not possible to force the drone to face a fixed direction of the canal, such as facing the center of the canal, to collect images in order to perform proportional measurement, otherwise the applicability of recognition will be greatly limited.
[0087] Although the width of the canal water surface in the field of view is certain, when it is photographed and sampled from different headings, its pixel width and direction in the image space are uncertain. This certain actual width and uncertain image features show a contradictory relationship that is difficult to unify. Therefore, this contradiction needs to be decoupled and modeled. The heading of the drone is not fixed during flight, and the direction of the canal varies greatly; therefore, the relative relationship between the heading of the drone and the direction of the canal is unpredictable. In order to improve the versatility and applicability of canal water surface width identification, it is necessary to study how to characterize the relationship between the two in various relative orientations in the image space, that is, what kind of features in the image can determine the width of the canal water surface?
[0088] After observing the water surfaces of rivers and canals in different directions in various low-altitude photography images of rivers and canals, it was found that the key lies in the two edge lines of the water surface, namely the shoreline; the direction of the edge of the river and canal water surface itself is not the only way to determine the width of the river and canal water surface. The angle of the straight line itself can only reflect the relative angle difference between it and the heading. After a large number of experiments and tests, it was also found that the width of the river and canal water surface will affect the imaging effect of the two parallel sides in the image. How to characterize the characteristics of the two sides? Through further research, hypothesis and reasoning, it is proposed to use the slopes k1 and k2 of two parallel lines at different intervals and the perpendicular distances D1 and D2 from the center point of the image to the straight lines on both sides of the straight line as the constraints on the width of different river and canal water surfaces in the image space. Figure 7 As shown. Further, how to map this constraint condition to the width of the canal water surface? Since there is a nonlinear, coupled mapping relationship between the two, inspired by the fitting application of neural networks in system nonlinear modeling, the present invention proposes to use neural networks to establish a canal water surface width prediction model, referring to Figure 4A shown.
[0089] Neural network is a machine learning model that imitates biological neural systems and is particularly good at nonlinear mapping modeling. It learns and represents complex functional relationships through a multi-layer artificial neuron structure. Neural network consists of input layer, hidden layer and output layer; each layer contains several neurons, and each neuron is connected to the neurons in the previous layer through weights. Among them, the input layer receives input data, the hidden layer performs nonlinear transformation and feature extraction, which can have multiple layers, and the output layer outputs the final prediction result.
[0090] like Figure 4A As shown, the canal water surface width prediction model of the present invention adopts a three-layer BP neural network with a hidden layer, and its input layer receives four input quantities, namely the slopes k1 and k2 of the straight lines on both sides of the canal water surface and the vertical distances D1 and D2 from the center point of the image to the straight lines on both sides of the canal water surface, from the general processing module respectively, and the output quantity of the output layer is the canal water surface width.
[0091] To extract the above four input quantities or geometric features of the canal in the image space, it is necessary to first find the canal water area from the image and then extract the straight lines on both sides of the water surface.
[0092] by Figure 1B Taking the canal scene in the middle left image as an example, the acquired color image is directly grayed out and binarized according to the Otsu threshold method. The processing results are as follows: Fig.11A As shown. Fig.11A After extracting the contour of the binary image obtained in the image, mark it in the original image and get the following Fig. 11B The outline diagram is shown. Fig. 11B In the displayed results, the contour lines enclose the sky and the scenery between the river and the land bank; among them, the river and the water body is submerged in the whole river. Therefore, it is difficult to obtain effective information for finding the river and the water body through ordinary binarization processing.
[0093] After observing a large number of river and canal images from low-altitude photography, it is found that the water body of the river and canal has color space characteristics and position characteristics that are obviously different from those of the land and slopes. Therefore, different modules are set in the processor to extract the water body of the river and canal and the geometric characteristics in sequence.
[0094] See also Figure 2 , Figure 3AAs shown, in the processor 110 of the host unit or controller 100, a plurality of processing modules are provided, namely, a general processing module 111, a water body region extraction module 112, a geometric feature extraction module 113, an iterative learning module 114 and a river channel water surface width prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the recognition of the river channel in the collected river channel image, schedules the recognition processing flow, inputs the river channel image collected in real time into the water body region extraction module 112, and outputs the width data recognized and predicted by the river channel water surface width prediction module 115.
[0095] like Figure 3B As shown, in the water area extraction module 112, an image acquisition unit 1121, a target detection unit 1122, and a water area extraction unit 1123 are also provided. The image acquisition unit 1121 acquires the low-altitude photography image taken by the image acquisition unit 500, or acquires the pre-stored river channel image from the storage module 120 in an offline state; removes the high-frequency noise through filtering; then, the target detection unit 1122 identifies the position of the river in the image as the ROI area according to the deep learning feature by a trained target classification model; the water area extraction unit 1123 selects a seed area from the middle of the ROI area to obtain a color threshold, and extracts the river channel water area from the inclined low-altitude shadow image according to the obtained color threshold to form a binary river channel water image.
[0096] In the target detection unit 1122, a target classification model based on a deep network such as a YOLOv5s network is established. After using the bird's-eye view camera in the image acquisition unit to collect outdoor river and canal scene images, the network is trained offline using an image sample set consisting of the scene images with various target areas such as the river and canal labeled to obtain a scene target classification model. When running online, refer to Fig.10 As shown, the water body area extraction unit 1123 uses the scene target classification model to perform recognition processing on the scene image including the river channel, i.e., the river channel image, to obtain a river channel area anchor frame, and uses the area within the anchor frame as the ROI area of the river channel.
[0097] By observing the canal image, it is found that the pixel value color components of the canal water area are very close. Therefore, the river area extraction unit 1123 set in the present invention determines the threshold interval of the color space such as the three primary colors from the central part of the ROI area. Fig.10As shown, a seed region is set in a rectangular block or a circular block near the intersection of the two diagonal lines corresponding to the ROI region rectangle, and the value intervals of the three color channels are extracted in the seed region, and the color thresholds are determined to correspond to the threshold intervals of the three primary colors. After the upper and lower limits of the grayscale values of all pixels in the central seed region are calculated, the upper and lower limits can be extended by a certain distance, such as increasing or decreasing a certain value of 5 to 20, or increasing or decreasing 5 to 10% of the median of the grayscale value interval of the channel.
[0098] In the collected low-altitude photography image, the pixels that meet the determined three-primary color threshold interval are taken as foreground objects, and the non-foreground object area is taken as the background object. All background objects in the oblique photography image are set to zero, and all foreground objects are set to 1, that is, a binary 255, to form a binary river channel image. The result is as follows Fig. 12A As shown. Fig. 12A After extracting the contour of the binary image obtained in the image, mark it in the original image and get the following Fig. 12B Compare Fig. 12B and Fig. 11B It can be seen that among the contours obtained by the former, in addition to the complete river and canal water body contours in the river and canal part, there are also many small contours formed in the slope and land areas because the grayscale values of individual pixels are close to the water body.
[0099] To this end, the features in the image are analyzed and it is found that in the binary canal image, the water part appears as a large continuous area. Therefore, for the obtained canal image, all connected domains are detected according to the connectivity of the foreground object pixels and the largest connected domain is selected as the canal water area; at the same time, Fig. 12C As shown in , we can also remove the fragmented target-like regions by performing morphological opening and closing operations on the binary image to reduce the operation of the connected domain search. The obtained maximum connected domain is marked in the original river channel image in the form of contour lines. The result is shown in Fig.12D shown.
[0100] Based on the binary canal image, only the foreground pixels in the canal water area are used as foreground objects to form a binary canal water image. The result is as follows: Fig.13A shown.
[0101] exist Fig.13A In the binary river and canal water image shown in the figure, the boundary of the water area is affected by shadows or water penetration, resulting in some non-foreground pixels in the water body and some broken edges on the right bank that penetrate into the slope area. For this type of image, morphological operations can be used for further processing. Fig.13A , perform a morphological closing operation of dilation followed by erosion, and obtain Fig. 13BThe binary image shown in Figure 2 has some interference points eliminated. Fig. 13B After the binary image is contour extracted, it is marked in the original image, and the result is as follows: Fig. 13C As shown in the contour map. Thus, a binary image with the river water body as a single target is obtained, forming a river target image.
[0102] See also Figure 3C As shown, in the geometric feature extraction module 113, an edge detection unit 1131, a straight line extraction and screening unit 1132, a water edge extraction unit 1133, a slope feature extraction unit 1134, and a distance feature extraction unit 1135 are respectively set.
[0103] After the image content is concentrated on the river and canal water area, the edge detection unit 1131 performs edge extraction on the binary river and canal target image. The edge information in the image is often contained in the area where the grayscale values of adjacent pixels in the image change dramatically. In digital image processing technology, there are many edge detection methods, such as using edge detection operators such as Sobel, Canny, and LOG. The edge can be directly extracted from the binary river and canal target image. For the color or grayscale river and canal target image, the color image is first converted into a single-channel image, and then the edge is detected using an edge detection operator.
[0104] The Canny operator is a second-order derivative edge detection operator. In a series of operation combinations, it uses Gaussian smoothing filtering, gradient amplitude and direction calculation, non-maximum suppression, and hysteresis threshold processing. Compared with the first-order derivative edge detection operator, the Canny operator can find an optimal edge and filter out most of the noise edges. The accuracy of edge detection and edge positioning are greatly improved. The present invention uses edge detection operators such as Canny to detect the edge of the water body and form a river channel edge image. The result is as follows: Fig.14A shown.
[0105] In the canal edge image, the most noteworthy thing is the straight line representing the boundaries on both sides of the canal. Hough line transform is to map the points in the image space to the curves in the parameter space, and then find the intersection of the curves in the parameter space to detect the straight line. In the image space, the equation of the straight line can be expressed as: y = kx + b; using polar coordinate representation, the equation of the straight line can be expressed as: ρ = xcosθ + ysinθ, where ρ is the vertical distance from the straight line to the origin of the coordinates, and θ is the angle between the perpendicular line of the straight line and the positive direction of the x-axis. The essence of Hough line transform is to establish a mapping relationship between the image space and the parameter space. On the basis of making full use of the global characteristics of the image, the point-line duality characteristics are used to map the pixel coordinates to the straight line represented by the elements in the parameter space, and then analyze it in the parameter space. The detection task is completed by cumulative statistics in the parameter space.
[0106] The straight line extraction and screening unit 1132 extracts the straight line boundary of the canal water surface from the canal edge image, and performs Hough line transformation on the canal edge image to obtain a canal edge straight line set. From the obtained canal edge straight line set, the water edge extraction unit 1133 screens out two straight line segments with close direction angles, maximum mutual distance and longest length as the two sides of the canal water surface according to the heuristic rule. The result is as follows Fig. 14B shown.
[0107] See also Fig. 14B As shown, the mutual distance between the two straight line segments is the minimum value of the actual distance between each point on any one of the two straight line segments and each point on the other line segment. The mutual distance can be obtained by traversing and searching the distances between the points on the two straight line segments; and through analysis, it is found that the search process can be simplified. Specifically, the mutual distance is further defined as the minimum value of the four actual distances composed of the actual distances between the endpoints of any line segment and each point on the other line segment. Furthermore, the actual distance d from a point to a target line segment is calculated as follows: assuming that the line connecting the point to any endpoint of the target line segment forms an obtuse angle with the target line segment, the actual distance d is taken as the distance from the point to the endpoint; otherwise, the actual distance d is taken as the vertical distance from the point to the target line segment.
[0108] like Fig. 14B As shown in FIG. 1 , the water edge extraction unit 1133 finds two straight lines marked as 2 and 1 respectively on the edge of the river water body. The two straight line segments found are used as the straight lines on the two sides of the river water surface and marked in the river image. The result is as follows: Fig.15A As shown; it can be seen that through the above processing, the water surface edges on both sides of the river and canal water bodies, namely the water bank lines, can be accurately found.
[0109] For the two extracted river surface edges, the slope feature extraction unit 1134 and the distance feature extraction unit 1135 respectively calculate the slope of the straight line and the vertical distance from the center point of the image to the straight line. Figure 7 Schematic diagram of the width constraint conditions of the canal water surface model, in which the two sides of a canal water surface are shown on the left and right sides of the image plane. Among them, the angle and slope of the straight line on the two sides of the left canal water surface are α(k1) and β(k2), respectively, and the perpendicular distances from the center point of the image to the two sides of the canal water surface on this side are D1 and D2 respectively; correspondingly, the angle and slope of the straight line on the two sides of the right canal water surface are α′(k1′) and β′(k2'), respectively, and the perpendicular distances from the center point of the image to the two sides of the canal water surface on this side are D1′ and D2′ respectively. In the oblique photography image plane, the two parallel lines will intersect at a point O at the far end, and point O is called the vanishing point.
[0110] The algebraic equation of a line can be expressed in three common forms, namely the general form, the two-point form, and the slope-intercept form. The general form is: Ax+By+C=0, where A and B are not zero at the same time, and the slope of the line k=-A / B. In the Cartesian rectangular coordinate system, the intercept of the line on the y-axis b=-C / B, and the general form applies to all line equations.
[0111] The two-point formula is: The two-point equation is the equation of the straight line passing through the two points when the coordinates of the two points are known. The slope of the straight line is k = (y2-y1) / (x2-x1). It can be seen from the two-point equation that this equation is suitable for describing straight lines that are not perpendicular to the x-axis and y-axis.
[0112] The slope-intercept form is: y=kx+b. The slope-intercept form explicitly expresses the slope k and intercept b of the straight line, and is suitable for representing straight lines that are not perpendicular to the x-axis.
[0113] The straight line detected by Hough line transform in the image is described by the coordinates of the two end points of the straight line. The coordinates of the two end points are marked as (x1, y1) and (x2, y2). Substituting the coordinates of the two points into the two-point equation, and then transforming the two-point equation into the slope-intercept equation form, we get: The corresponding slope is k=y1-y2 / x1-x2. The only difference is that the calculation of the slope of the straight line is completed in the image coordinate system (the y-axis is downward in the positive direction), so the slope of the straight line in the image is equal to that in the Cartesian coordinate system, but has the opposite sign.
[0114] Combined with the straight line equation, the vertical distance from the center of the image to the edge of the canal water surface, i.e., the canal water surface edge or water bank line, can be calculated as: Wherein, (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is a constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1(x2-x1). According to the calculation formulas of k and d, the slopes k1 and k2 of the straight lines on both sides of the canal water surface and the vertical distances D1 and D2 from the center point of the image to the two straight lines are calculated respectively, and a total of four quantities are obtained as the geometric features of the canal water surface in the canal image.
[0115] like Fig. 8A , Figure 8B The schematic diagram of the canal edge line screening and geometric feature extraction in the canal water surface model image is given. The intersection of the diagonal lines indicates the center point of the image. In the set of detected straight lines, the lengths of two straight line segments are much longer than the other straight lines, so they are screened as the two edge lines of the canal. It can also be seen that the angles of the two canal water surface edge lines are close. In 8A, Figure 8BIn , after excluding the short straight lines, for the two selected edge lines, draw a perpendicular line from the center point of the image, represented by a dotted line. Figure 8B In the figure, since the foot of the perpendicular is outside the range of the side lines, extend the side lines to the foot of the perpendicular.
[0116] In order to reveal the nonlinear mapping structure of the canal water surface width prediction model of the present invention, a canal water surface scale model is established and then the model is obliquely photographed to obtain training image samples. Figure 5 As shown in the figure, when the drone takes a low-altitude photo of the ground, the angle between the optical axis of the onboard lens and the horizontal plane is α, which can be selected from a commonly used working angle such as 45 degrees. Since the width of actual rivers and canals varies greatly, it is difficult to obtain uniformly spaced standardized template river and canal sections. To facilitate modeling, the present invention uses rectangular sheets to make river and canal water surface models of various specifications, such as Fig. 6A shown.
[0117] In an offline state, a drone is used to collect images of the produced river channel water surface model and form a sample set, and the sample set is used to train the river channel water surface width prediction model. The river channel water surface model uses a rectangular thin sheet, the width of the rectangular thin sheet is a first preset multiple S1 of the actual river channel water surface width, and the length of the rectangular thin sheet is a second preset multiple S2 of its width, so as to form a model that conforms to a segment in the actual river channel. The multiples S1 and S2 are set according to the cruising flight altitude of the drone and the camera resolution; the different rectangular thin sheets used in the river channel water surface model have a width range corresponding to S1 times the common width range of the river channel, and the width difference between different rectangular thin sheets, that is, the spacing, corresponds to S1 times the preset river channel water surface width difference SD. For example, when the flight altitude is 100 meters, the corresponding sampling height of the model is 2 meters, so as to ensure that the river channel water surface model can occupy a significant image space during imaging, so as to coordinate the image resolution and the recognition accuracy of the river channel water surface width prediction model, and make it unnecessary for the drone to specially lower the altitude for predicting the river channel water surface width, thereby improving the applicable scope of the prediction.
[0118] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 30, and the minimum value of the second preset multiple S2 is between 3 and 7. Preferably, the preset river channel water surface width difference SD is 0.2 to 0.7 meters, and the width of each rectangular sheet used as the river channel model can form an arithmetic progression with S1 times SD as the tolerance.
[0119] A river surface model was built under laboratory conditions, and the actual aerial photography height h real and the actual water surface width w real Scale proportionally to obtain the aerial photography height h under laboratory conditions model and the water surface width w modelIn order to find the relationship between the water surface features of the river channel in the image and the width of the river channel, this embodiment selects the actual aerial photography height h of the drone real When they are 30m, 40m, and 50m respectively, the corresponding aerial photography height h under laboratory conditions model The corresponding sizes are 667mm, 889mm, and 1111mm, and the scaling ratios are:
[0120]
[0121] When building the canal water surface model in the laboratory, we considered the flat riverbed in most cases, that is, the natural slope of the canal is within 3 degrees. model With h real The scaling ratio is 1:45. Because it is proportional scaling, the width of the river surface model is w model The actual width of the canal w real The ratio is also 1:45, and then the river and canal water surface models of different widths are made with a step size of 10.5mm. The step size of the model corresponding to the actual river and canal water surface width is 0.5m. Fig. 6A As shown. Among them, the width of the river channel water surface model w model The 105mm and 210mm river and canal water surface models are marked.
[0122] Since the image sampling of the canal water surface model is low-altitude oblique image, in order to ensure the diversity of the features representing the canal water surface model in the image as much as possible and improve the generalization ability of the canal water surface width prediction model network, the canal water surface model is photographed from 0 degrees to 180 degrees. At the same time, in order to highlight the distance relationship between the drone and the actual position of the canal, the canal water surface model is arranged in different areas of the ground field of view, so that the canal water surface model in the sample set is distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle, and right in the image plane. model = 136.5 mm as an example, some of the captured data sets are as follows Figure 6B As shown, in order to improve the accuracy of the model, the photo was taken with a black ground as the background, and for the convenience of display, the picture was inverted.
[0123] The present invention subdivides the size of the river channel water surface model to form a width sequence, and photographs the river channel water surface model images from various relative azimuth angles during sampling, thereby ensuring the sufficiency of samples, thereby improving the generalization ability of the neural network and reducing the prediction error of the river channel water surface width.
[0124] For the canal image taken based on the canal water surface model, the collected image is filtered and grayed, then binarized to form a canal target image, and then the canal edge image is obtained by edge detection. Taking the canal water surface model image at a certain location taken at the laboratory sampling height h = 667mm as an example, the preprocessing results of each stage are as follows Figure 6C As shown in the figure, the left and right correspond to the processing results after binarization and edge detection. The Hough line transform is performed on the canal edge image after Canny edge detection. The transformed image is shown in the figure. Fig.6D As shown in the figure, the left and right figures are the Hough line transform results of the canal water surface model with a width of 105mm and 136.5mm respectively. After that, a heuristic search is performed on the straight lines obtained after the transformation, and the two straight line segments with the longest length are selected as the two straight line edges of the canal, that is, the straight lines on both sides of the canal water surface. Fig.6D In the figure, there are two long straight lines after excluding the short line close to the horizontal direction.
[0125] See also Figure 7 As shown, for the two side lines obtained in each model image, the slopes k1 and k2 and the vertical distances D1 and D2 from the center of the image to the two side lines are calculated, and the corresponding actual model canal water surface width is recorded as the true value of the canal water surface width. The geometric feature quantities and the true value of the canal water surface width are formed into samples. After the samples are collected and recorded to form a sample set, they are input into the canal water surface width prediction model based on the neural network for offline training. Preferably, the sample set can be stored in the form of a data table composed of input and output true values as a record.
[0126] The BP neural network used in the river channel water surface width prediction model in the present invention is as follows: Figure 4A As shown, the model is a three-layer structure including a hidden layer, in which
[0127] The output of the jth node in the hidden layer is
[0128] The output of the output layer node is
[0129] In the formula, the f() function is taken as the tansig function, and w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.
[0130] When predicting the width of the canal water surface online, the width of the canal water surface is predicted based on the four geometric features obtained from the canal water surface model image. The prediction model then maps it back to the actual canal water surface width as the actual output value based on the ratio of the model to the actual shooting height.
[0131] Preferably, a plurality of the neural network-based river channel water surface width prediction models are created in the host unit, each of which corresponds to a flight altitude and a bird's-eye view angle of the image acquisition unit; when predicting the river channel water surface width online, the geometric features obtained after the current acquisition process are input into the river channel water surface width prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the river channel water surface width prediction value. In this embodiment, the shooting heights h of the river channel water surface model are selected to be 667mm, 889mm and 1111mm respectively, and the shooting angle is 45 degrees.
[0132] The sample data sets with h of 667mm, 889mm and 1111mm were randomly divided into training sets and test sets. Both training sets and test sets covered 17 different river and canal water surface models. The geometric features of the test set were input into the trained network model as input to obtain the predicted value. The fitting comparison curves of the predicted value and the true value of the river and canal water surface model width at different heights are shown in Figure 1. Fig.9A , Fig. 9B As shown, their h are 889mm and 1111mm respectively. The test set performance is shown in Table 1.
[0133] Table 1 Test performance of the canal water surface width prediction model
[0134]
[0135] The regression results at three different laboratory heights were analyzed. The coefficients of determination R2 at three different heights were 0.99709, 0.99821, and 0.99912, respectively. The closer the coefficient of determination is to 1, the better the regression effect is, indicating that the model width prediction models at three heights have high reliability. The root mean square error RMSE at three different heights was 2.2156, 2.0322, and 1.8352, respectively, indicating that the predicted value is close to the true value, and the network model has good generalization ability.
[0136] Example 2
[0137] Based on the previous embodiment, Figure 2As shown, this embodiment provides a low-altitude photography river and canal water surface width identification system 1000, which includes: an image acquisition unit 500 for performing image sensing and acquisition of river and canal scenes, a user interface unit 200 for parameter input and operation and display of river and canal water surface width information, a server 400 for data management and information transmission, a handheld terminal 300 for remote operation and interaction of river and canal width information, and a host unit 100 connected to the user interface unit 200, the server 400, and the image acquisition unit 500.
[0138] The host unit 100 includes an input and output module 130, a processor 110 and a storage module 120, and is configured to: obtain a low-altitude photography river channel image, determine a ROI region containing the river channel in the river channel image, determine a color threshold according to the value intervals of the three color channels extracted from the central part of the ROI region, use pixels whose colors meet the color threshold range as foreground objects, use non-foreground object areas as background objects, and set all background objects in the river channel image to zero to form a river channel binary image; detect all connected domains according to the connectivity of the foreground object pixels The largest connected domain is screened out as the canal water body area, and only the canal water body area is used as the foreground object to form a canal target image; the canal target image is subjected to edge detection to obtain a canal edge image; the canal edge image is subjected to Hough line transformation, and a canal water edgeline image containing two straight edges of canal water is obtained through heuristic search; the geometric features of the straight lines on both sides of the canal water surface relative to the image center are obtained from the canal water edgeline image, and the geometric features are input into a trained neural network-based canal water surface width prediction model to obtain a canal water surface width prediction value, and the width prediction value is output.
[0139] Preferably, the low-altitude photography river channel water surface width identification system 1000 may not include the server 400 and the handheld terminal 300, but the low-altitude photography river channel water surface width identification system is connected to the server 400 and the handheld terminal 300 through a network when in use. The low-altitude photography river channel water surface width identification system 1000 may also be provided with a positioning unit 700 for acquiring coordinates.
[0140] Preferably, the low-altitude photography river channel water surface width identification system 1000 also includes a flight control platform 600. In response to the user's operation on the operation panel 220 or the instruction of the host unit 100, the flight control platform 600 executes the instruction to fly and turn toward the specified direction or river channel point.
[0141] The low-altitude photography of the river scene is continuously performed by the drone flight, and the low-altitude photography river channel water surface width identification system 1000 iteratively continuously collects images, and detects the river width based on the collected images. Preferably, according to the flight speed and frame rate, the overlapping range of the river channel scene images processed successively is not less than the set ratio. Preferably, the splicing of multiple images can also be achieved by aligning feature points, and the river channel water surface width is identified based on the spliced images.
[0142] Preferably, after collecting oblique photographic images of river scenes in sections, the ratio of the width in each section to a preset width threshold at that location is calculated; when the ratio value is greater than a set high ratio threshold, an alarm is issued to indicate that the river width at that location exceeds a warning level and flood prevention is required; and when the ratio value is less than a set low ratio threshold, an alarm is issued to indicate that the water surface width at that location is too small and drought prevention or possible water leakage is required.
[0143] Preferably, after detecting the width of the river channel in sections, the width information of each section of the river channel is also marked at the corresponding position of the river channel image, such as by using an arrow line to mark between the two straight lines of the water bank of the section of the river channel; preferably, the acquired coordinate and position information, as well as the water surface width information of the river channel section, are also stored in a list of the storage module, and the list information can be stored in a cloud server.
[0144] Preferably, when the drone is used in a group system of multiple machines and multiple terminals, the system may have multiple handheld terminals and drone platforms accordingly. Through the server, the host unit and the remote client deployed at the handheld terminal, etc. can send messages about the width of the water surface of the canal. Preferably, an MQTT server can be deployed, and the server pushes messages to the clients that subscribe to the topic. Multiple devices can subscribe to multiple topics at the same time, and when publishing messages, those who subscribe to the same topic can receive the same message. The publish-subscribe model is a messaging model that decouples the client that sends the message, i.e., the publisher, from the client that receives the message, i.e., the subscriber, so that the two do not need to establish a direct connection or know each other's existence.
[0145] The server provides subscription and publishing services, and the host unit is also configured to: subscribe to the topic of obtaining river and channel water surface information of the handheld terminal, and in response to the topic message, transmit the detected river and channel water surface width and the corresponding coordinates and position information to the handheld terminal through the server; or when it is detected that the river and channel water surface width reaches a set width threshold, transmit the river and channel water surface width and the corresponding coordinates and position information to the server in the form of a width event topic message.
[0146] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the results of river channel water surface width identification in the form of a list; the host unit also displays the identified river channel water surface width in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the identified width on the curve.
[0147] Preferably, the positioning unit in the system also obtains heading information, and also records and identifies in the output position information the direction identified based on the heading, such as the north direction, and the direction corresponding to the center line of the canal between the two side lines of the canal.
[0148] Preferably, the host unit can also communicate directly with the handheld terminal. The host unit can be a local control unit deployed separately on the ground, or a control unit deployed on the drone platform. When deployed on the drone platform, the host unit on the drone platform performs edge computing for online river and canal water surface width prediction.
[0149] In the first embodiment, in order to find the actual river water body part from the ROI rectangular block area, this embodiment also uses the flood filling method for extraction. The water body area extraction unit of the water body area extraction module selects a seed point from the middle of the obtained ROI area, and extracts the river water body area from the oblique photography image through the flood filling process.
[0150] The flood filling method is to find the same area based on the difference between the gray values of pixels to achieve segmentation. The gray value of the image is understood as the height of the pixel point, so that an image can be seen as a rugged ground or mountain. A certain amount of water is poured into a low place on the ground, and the water will cover the area below a certain height. The flood filling method uses this principle and is an image processing method based on seed points. It starts from the seed point and spreads to the adjacent pixels, and marks the pixels that meet the conditions as the specified color or value.
[0151] By observing the canal image, we found that the pixel values of the canal water area are very close. Therefore, we set seed points in the canal area and select the canal area by flood filling. Fig.10 As shown, the target classification model based on the deep learning network locates the canal ROI area with the anchor frame, i.e., the rectangular frame therein. Then, a seed point is set at the intersection of the two diagonals corresponding to the rectangular frame, and the canal water body area is obtained by flood filling processing. After the pixels outside the obtained canal water body area are set to zero, a binary canal water body image is formed.
[0152] In the area of Example 1, this embodiment also establishes another canal water surface width prediction model. In the BP neural network, the input layer receives six input quantities, namely, the slopes k1 and k2 of the straight lines on both sides of the canal water surface, the vertical distances D1 and D2 from the center point of the image to the straight lines on both sides of the canal water surface, and the flight altitude and overlooking angle during acquisition, from the general processing module. The output of the output layer is the canal water surface width. After establishing a neural network with this structure, the input quantity of each sample in the training set samples is supplemented, and after supplementing enough samples at different heights and overlooking angles, it is trained with a new training sample set. When applied online, the new model can further simplify the image sampling requirements for canal prediction; it uses the generalization performance of the neural network to reduce the number of models required for canal water surface width prediction, and also simplifies the system structure, so that one model can cover different flight sampling scenarios, further improving the applicability of the system.
[0153] Example 3
[0154] Based on the above embodiments, this embodiment provides a low-altitude photography river channel water surface width identification controller or flight controller. Figure 2 , Figure 3A , Figure 4B As shown, the low-altitude photography river channel water surface width identification controller or flight controller 100 includes a processor 110, a storage module 120 and an input and output module 130. The processor 110 includes a general processing module 111, a water body area extraction module 112, a geometric feature extraction module 113, an iterative learning module 114 and a river channel water surface width prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the identification of the river channel in the collected river channel image, schedules the identification processing flow, inputs the real-time collected river channel image into the water body area extraction module 112, and outputs the width data identified and predicted by the river channel water surface width prediction module 115.
[0155] Combination Figure 3A , Figure 4B As shown, the river channel image is input into the controller by the input unit 131 in the input-output module 130; the river channel water surface width prediction model is established in the river channel water surface width prediction module 115, and its input layer receives the geometric features of the river channel in the river channel image extracted by the geometric feature extraction module 113 from the general processing module 111, and the output layer is transmitted to the iterative learning module 114 and the general processing module 111 through the first connection matrix 1141 and the second connection matrix 1142 respectively.
[0156] When the river channel water surface width prediction model is trained offline, the iterative learning module 114 adjusts the connection weights of the network according to the actual value of the river channel water surface width and the output value of the neural network in the model respectively input by the general processing module 111 and the river channel water surface width prediction model through the first connection matrix 1141; in the field environment, the first connection matrix 1141 is disconnected, and the neural network of the river channel water surface width prediction model predicts the water surface width in the current river channel image and outputs it to the general processing module 111 through the second connection matrix 1142, and is processed and analyzed by the general processing module 111 and output through the output unit 132 in the input and output module 130.
[0157] The steps of iterative training include: STEP1, initialize the weights and biases of the network; STEP2, read the network parameters and the training sample set written in the document above; STEP3, normalize the sample data; STEP4, calculate the error between the predicted output value and the expected value of each sample; STEP5, reversely calculate and correct the network weights and biases; STEP6: if the conditions for the end of network training are met, end the training, otherwise go to STEP4 to continue training; STEP7, analyze and output the results.
[0158] The general processing module 111 in the controller also obtains the coordinate information and orientation information of the current location of the river channel through the positioning unit, and after determining the water surface width of the river channel, outputs the water surface width of the river channel and the corresponding coordinate and orientation information through the output unit 132.
[0159] like Fig. 15B As shown, after the water surface width of the canal is identified, the water surface width of the canal is also marked in the corresponding area segment of the canal image, and the width direction line segment is marked with an arrow line. Preferably, the acquired coordinates and position information, as well as the water surface width of the canal, are also stored in a list of the storage module 120, and the list information can be stored in the cloud server 400.
[0160] When the water surface width exceeds the upper and lower limits of the preset river channel water surface width threshold, an alarm is issued to prompt that the river channel water surface width exceeds the limit. During the flight of the drone, low-altitude image acquisition is iteratively performed continuously, and width prediction processing is performed based on the acquired images in each preset section. When the width of any preset section is greater than its corresponding preset width threshold, a water surface width exceeds the limit prompt is issued. Preferably, according to the flight speed and frame rate, the overlapping range of the river channel images processed successively is not less than the set ratio.
[0161] The low-altitude photography river channel water surface width identification controller or flight controller of the present invention is also used for monitoring the water surface width of the approximately parallel bank sections of most rivers.
[0162] Firstly, the basic inverse perspective transformation matrix from the image coordinate system corresponding to the image collected by the image acquisition unit on the ground to the world coordinate system is determined according to the ground calibration experiment; then it is multiplied by the transformation matrix from the ground to the current low-altitude oblique photography sampling posture to obtain the inverse perspective transformation matrix from the current image coordinate system to the world coordinate system. Then, low-altitude field image acquisition is iteratively performed continuously. During the image processing, the binary river channel target image is transformed based on the inverse perspective transformation matrix, and then edge detection is performed; a river channel edge straight line set is obtained through Hough line transformation; from the obtained river channel edge straight line set, the water edge extraction unit 1133 selects two straight line segments with close direction angles, maximum mutual distance and longest length as the straight lines on both sides of the river channel water surface according to heuristic rules; the slope or angle of the obtained straight lines on both sides of the river channel water surface is calculated to determine whether they are approximately parallel. If so, the geometric features of the straight lines on both sides of the river channel water surface relative to the image center are obtained, and the geometric features are input into a trained river channel water surface width prediction model based on a neural network to obtain a river channel water surface width prediction value, and the width prediction value is output and the flight continues along the direction of the obtained river channel water surface straight lines; otherwise, the flight continues along the river channel in the direction of the angle bisector of the obtained river channel water surface straight lines.
[0163] The present invention is applied to identify the width of river and canal water surface in the space of UAV low-altitude photography images. Through automatic identification of river and canal targets, water body target seed points or areas are extracted to automatically extract river and canal water bodies. After regional segmentation, river and canal edge detection and water bank line extraction, the slopes of the straight lines on both sides of the river and canal water surface and the vertical distances from the image center point to the straight lines on both sides of the river and canal water surface are used as geometric features and input into the established neural network-based river and canal water surface width prediction model. The model can accurately predict the actual width of the river and canal water surface as basic hydrological data, providing a basis for river and canal flow monitoring and early waterlogging warning.
[0164] Several embodiments of the present invention are described above, but these embodiments are presented as examples and do not limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These embodiments or their modifications are included in the scope or main purpose of the invention, and are also included in the invention described in the claims and their equivalents.
Claims
1. A device for identifying the width of a river or canal using low-altitude photography, comprising a host unit and a user interface unit, wherein the user interface unit is used for parameter input and human-computer interaction, and the host unit comprises an input / output module, a processor and a storage module and is configured as follows: Acquire a river channel image obtained by low-altitude photography, determine a ROI region containing the river channel in the river channel image, determine a color threshold according to respective value intervals of three color channels extracted from the central part of the ROI region, take pixels whose colors meet the color threshold range as foreground objects, take non-foreground object regions as background objects, and set all background objects in the river channel image to zero to form a river channel binary image; According to the connectivity of the foreground object pixels, all connected domains are detected and the largest connected domain is selected as the canal water body area, and only the canal water body area is used as the foreground object to form a canal target image; For the canal target image, edge detection is performed to obtain a canal edge image; after Hough line transformation is performed on the canal edge image, a canal water edge image containing two straight line edges of canal water is obtained through heuristic search; The geometric features of the straight lines on both sides of the canal water surface relative to the image center are obtained from the canal water edge line image, and the geometric features are input into a trained canal water surface width prediction model based on a neural network to obtain a canal water surface width prediction value, and the width prediction value is output.
2. The low-altitude photography river and canal water surface width identification device according to claim 1, It is characterized in that The processor includes a general processing module, a water area extraction module, a geometric feature extraction module, an iterative learning module and a river channel water surface width prediction module. The water area extraction module obtains the river channel image taken by the drone at low altitude from the external image acquisition unit, and after filtering, identifies the position of the river channel in the image as the ROI area by a trained target classification model according to the deep learning features; selects a seed area from the middle of the ROI area to obtain a color threshold, and extracts the river channel water area from the low altitude photography image according to the obtained color threshold to form a binary river channel water image. The general processing module responds to the message, starts the extraction of the river channel water body area in the collected river channel image, schedules the river channel water surface width identification process, and outputs the width data predicted by the river channel water surface width prediction model.
3. The low-altitude photography river and canal water surface width identification device according to claim 1, It is characterized in that A canal water surface width prediction model is established in the canal water surface width prediction module, whose input layer receives the geometric features of the canal water surface in the canal image from the general processing module, and the output layer is transmitted to the iterative learning module and the general processing module through the first connection array and the second connection array respectively; when the canal water surface width prediction model is trained offline, the iterative learning module adjusts the connection weights of the network according to the actual values of the canal water surface width input by the general processing module and the canal water surface width prediction model through the first connection array and the output value of the neural network in the model; in the field environment, the first connection array is disconnected, and the neural network of the canal water surface width prediction model predicts the width of the canal water surface in the current canal image and outputs it to the general processing module through the second connection array, and the general processing module processes and analyzes it and then outputs it through the output part in the input and output module.
4. The low-altitude photography river and canal water surface width identification device according to claim 1, It is characterized in that The host unit is also configured as follows: a target detection unit is set in the water body area extraction module, a target classification model based on the YOLOv5s network is established in the target detection unit, and after collecting the river and canal scene image by low-altitude photography using the bird's-eye view camera in the image acquisition unit, the network is offline trained with an image sample set consisting of the river and canal scene images with the river and canal areas marked, so as to obtain a river and canal scene target classification model; when running online, the water body area extraction module uses the river and canal scene target classification model to identify and process the current scene image including the river and canal to be tested, i.e., the low-altitude photography river and canal image, to obtain a river and canal area anchor frame, and the area within the anchor frame is used as the river and canal ROI area.
5. The low-altitude photography river and canal water surface width identification device according to claim 1, It is characterized in that The canal water surface width prediction model adopts BP neural network, and its input layer receives the slopes k of the straight lines on both sides of the canal water surface from the general processing module. 1 , k 2 And the perpendicular distance D from the center point of the image to the straight lines on both sides of the water surface of the canal 1 , D 2 There are 4 input quantities in total, and the output of the output layer is the width of the river surface.
6. The low-altitude photography river and canal water surface width identification device according to claim 1, It is characterized in that The BP neural network used in the river channel water surface width prediction model has a three-layer structure including a hidden layer, wherein: The output of the jth node in the hidden layer is The output of the output layer node is Among them, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.
7. The low-altitude photography river and canal water surface width identification device according to claim 1, It is characterized in that The host unit is further configured to: before online application, firstly use a drone to collect images of the prepared river and canal water surface model offline to form a sample set, and use the sample set to train the river and canal water surface width prediction model, The canal water surface model adopts a rectangular thin sheet, the width of the rectangular thin sheet is a first preset multiple S1 of the actual canal water surface width, and the length of the rectangular thin sheet is a second preset multiple S2 of its width, and the multiples S1 and S2 are set according to the cruising flight altitude of the UAV and the camera resolution; the different rectangular thin sheets adopted in the canal water surface model have a width range corresponding to S1 times the common width range of the canal, and the width difference between different rectangular thin sheets corresponds to S1 times the preset canal water surface width difference SD.
8. The low-altitude photography river and canal water surface width identification device according to claim 1, It is characterized in that The geometric feature extraction module calculates and extracts geometric features of the straight lines on both sides of the water surface of the river channel. Assume that the coordinates of the two endpoints of the canal edge line are (x 1 ,y 1 )、(x 2 ,y 2 ), then its slope is k = y 1 -y 2 / x 1 -x 2 ; The perpendicular distance from the center of the image to the edge of the canal is In the formula, (x 0 ,y 0 ) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is the constant term and: A = y 2 -y 1 , B = x 1 -x 2 , C = -x 1 (y 2 -y 1 )+y 1 (x 2 -x 1 ); According to the calculation formula of k and d, the slope k of the straight lines on both sides of the river water surface is calculated respectively. 1 , k 2 And the perpendicular distance D from the center point of the image to the two straight lines 1 , D 2 A total of 4 quantities serve as the geometric features of the river channel water surface in the river channel image.
9. The low-altitude photography river and canal water surface width identification device according to claim 1, It is characterized in that A plurality of the neural network-based river and canal water surface width prediction models are created in the host unit, each of which corresponds to a flight altitude and a bird's-eye view angle of an image acquisition unit. When predicting the river and canal water surface width online, the geometric features obtained after the current acquisition process are input into the river and canal water surface width prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the river and canal water surface width prediction value.
10. Low-altitude photography river and canal water surface width identification system, include: An image acquisition unit for performing image sensing acquisition of a river channel scene, a user interface unit for parameter input and operation and display of river channel water surface width information, a server for data management and information transmission, a handheld terminal for remote operation and interaction of river channel width information, and a host unit connected to the user interface unit, the server, and the image acquisition unit; the host unit includes an input and output module, a processor, and a storage module, and is configured as follows: Acquire a river channel image obtained by low-altitude photography, determine a ROI region containing the river channel in the river channel image, determine a color threshold according to respective value intervals of three color channels extracted from the central part of the ROI region, take pixels whose colors meet the color threshold range as foreground objects, take non-foreground object regions as background objects, and set all background objects in the river channel image to zero to form a river channel binary image; According to the connectivity of the foreground object pixels, all connected domains are detected and the largest connected domain is selected as the canal water body area, and only the canal water body area is used as the foreground object to form a canal target image; For the canal target image, edge detection is performed to obtain a canal edge image; after Hough line transformation is performed on the canal edge image, a canal water edge image containing two straight line edges of canal water is obtained through heuristic search; The geometric features of the straight lines on both sides of the canal water surface relative to the image center are obtained from the canal water edge line image, and the geometric features are input into a trained canal water surface width prediction model based on a neural network to obtain a canal water surface width prediction value, and the width prediction value is output.
