River width detection method based on oblique photography

Through the river width and narrow detection method based on tilt photography, the river target recognition network and edge detection technology are used to automatically identify the narrowest part of the river, solving the problem of automatic detection of drones in river areas, and improving the accuracy of water flow monitoring and water conservancy projects and efficiency.

CN119022889APending Publication Date: 2024-11-26CHINA JILIANG UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411123964.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art is difficult to automatically detect and identify the narrowest parts of rivers during drone flight, especially in mountainous and rural areas, which affects the accuracy and efficiency of water flow monitoring and water conservancy engineering construction.

Method used

The river width and narrow detection method based on tilt photography is adopted. By establishing a river target recognition network, tilt photography images are used to extract river water bodies and edge detection, find straight lines on both sides of the river, and calculate the minimum distance between the banks to determine the narrowest part of the river. , and flight control is performed to achieve automatic identification and identification.

Benefits of technology

It realizes that the drone automatically extracts river width and narrow information in river scenes, and identifies the narrowest position of each section of the river. It is suitable for detection under unfamiliar terrain, reduces battery consumption and flight time, and improves the accuracy of hydrological monitoring and water conservancy engineering. Sex and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119022889A_ABST
    Figure CN119022889A_ABST
Patent Text Reader

Abstract

The invention discloses a river width detection method based on oblique photography, and the method comprises the steps: building a river target recognition network, collecting a river basin image at a certain height and pitch angle through oblique photography, and selecting a river ROI region based on the river target recognition network. The method comprises the following steps: automatically extracting a river water body area according to color space characteristics of an ROI area center and connected domain search, heuristically searching broken edges on two sides of a river from river water body edges after inverse perspective transformation, segmenting two river bank edge lines according to the broken edges, and obtaining a distance and change between two banks through traversal calculation, and outputting the minimum width of the river segment and the corresponding bank point position. According to the method, the river shoreline does not need to be appointed manually, oblique photography image acquisition of the transverse axis approximately parallel to the river trend and automatic extraction of the river shoreline are realized, the river width can be accurately detected, and the width change of each section of the river can be identified, so that a decision basis is provided for unmanned aerial vehicle flight, hydrological detection, river-crossing facility construction and the like.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of unmanned aerial vehicle detection application, and in particular relates to a river width detection method based on oblique photography. Background Art

[0002] Rivers are widely distributed in our living environment. Large rivers are important shipping channels. my country has also actively used the river drop to build many hydropower stations. Rivers have important ecological value in maintaining biodiversity, regulating climate and purifying water quality. As a source of water supply for cities, trunk rivers play an irreplaceable role.

[0003] Compared with large rivers, small rivers are smaller in size, but they also have unique value in ecology, economy and society. In most rural areas of my country, a small river is often an important source of water for villagers' lives and production. They rely on the water from small rivers to irrigate farmland and feed livestock. Small rivers are also habitats for many small aquatic organisms and amphibians. The wetlands and vegetation around small rivers also provide places for birds to forage and live.

[0004] As an important natural resource and landform, it is very important to systematically monitor and evaluate rivers and their surrounding environment. Water quality monitoring is used to detect changes in river water quality, including pollutant content, dissolved oxygen level, pH value, etc. In hydrological parameter measurement, hydrological parameters such as river water level, flow rate, and flow are obtained to understand the hydrodynamic characteristics of the river; these parameters are of great significance for flood warning, water resource allocation, and hydropower station operation and management. In topographic and geomorphic mapping, the terrain model of the river is established by measuring the width, depth, and terrain characteristics of the river channel; providing a basis for river channel management, flood control design, and land use planning. At the same time, we also understand the ecosystem around the river through ecosystem monitoring; by monitoring natural disasters such as floods, droughts, and geological disasters in the river, we can timely assess their impact on humans and the ecosystem; by collecting and analyzing river monitoring data, we can formulate sustainable water resource management strategies to ensure the effective use and supply of water resources.

[0005] On the one hand, river width is the most basic data in river hydrological data. For example, the Chinese invention patent with application number 202010662956.3 uses relevant instruments to first measure the width of the river, and then uses a river flow prediction method based on the relationship between the river's mid-surface velocity and the river width to measure the flow of the river. The Chinese invention patent with application number 201810429322.6 measures the width of the river in the sample river grid unit in the digital map, and then calculates the increased water surface width of the reservoir river grid unit due to the construction of the dam in combination with the slope gradient and increased water depth of the reservoir river grid unit.

[0006] In addition to obtaining the river width through prior data such as digital maps, in the prior art, the river width is often detected through the application of remote sensing and geographic information systems, such as the Chinese invention patent with application number 202310573885.3, which discloses a river complex section water level observation method based on remote sensing data. It is necessary to obtain a remote sensing image of the observed river section, identify the water body according to the remote sensing reflectivity of the green band and the mid-infrared band in the remote sensing image, and obtain the river width according to a specified number of river surface pixels through a certain proportional relationship.

[0007] Important rivers around cities can often collect remote sensing images taken by satellites. However, it is difficult to conduct comprehensive monitoring of all sections of the most widely distributed small rivers in the upstream rural and mountainous areas through satellite remote sensing. Compared with satellite remote sensing, drones have many advantages in river monitoring. The operating costs of drones are usually much lower, and they can complete complex tasks in a shorter time, thereby reducing the overall cost of data collection and analysis. Drones can be equipped with high-resolution cameras and sensors to capture detailed aerial images and videos. Drones can be deployed quickly and complete aerial photography tasks in a short period of time, with higher flexibility and responsiveness, which enables rapid acquisition of critical geographic information in emergency situations such as flood monitoring. In addition, drones can easily cover a large geographical area, and even difficult-to-reach or complex terrain areas can be effectively monitored and data collected.

[0008] On the other hand, surveys have shown that river width is one of the key factors affecting river morphology. my country is a mountainous country. The interaction between the wide and narrow river morphology and the flood process with steep water level rise often leads to flooding or breaching of riverbeds, causing serious casualties and economic losses. Chinese patent application number 201910834310.6 discloses a method for identifying areas prone to flash flooding disasters in wide and narrow river sections based on changes in mountain river morphology and flood levels. It explains that the gradual change in the width of a wide and narrow river will cause changes in water depth and flow rate, with the water depth at the narrowest point being the lowest and the flow rate being the highest. Due to the high flow rate, the scouring force on the river bank is also the strongest at the narrowest part of the river, and it is most likely to cause a breach.

[0009] In the most widely distributed rural areas of my country, especially in mountainous areas, the changes in the width of rivers serve as a basic basis for water flow monitoring, identification of flash flood-prone areas, and construction of water conservancy infrastructure. In the planning of water conservancy projects, determining the location of the smallest river width is of great significance for the construction of bridges, dams and other facilities. The engineering design can be optimized according to the width of the river, reducing construction costs and construction difficulties. For river regulation and flood control projects, finding the narrowest part of the river can more accurately plan the layout and reinforcement measures of the embankment.

[0010] As a low-cost, easy-to-deploy platform, drones are very suitable for scene sampling of small rivers. Even for drone flights themselves, when flying in river areas, the narrowest part of the river usually means that the drone needs to fly a shorter distance, thereby reducing battery consumption and flight time, and reducing the risk of crashing due to low battery power. Therefore, identifying the narrowest part of the river is also of great reference significance for drones flying over rivers.

[0011] To this end, a method and system is needed that is easy to deploy, based on drone oblique photography, and can automatically detect and identify the width of a river. Summary of the invention

[0012] The purpose of the present invention is to provide a river width detection method, a flight control method and a system based on oblique photography, which are used to extract and identify the characteristics of the river according to the image in the field of view of the unmanned aerial vehicle, and to find and mark the narrowest position of the river, and to perform flight control and image sampling by identifying the direction of the river.

[0013] In order to solve the above technical problems, the present invention provides a river width detection method based on oblique photography, which comprises the following steps:

[0014] S1, initialization, establishing a river target recognition network;

[0015] S2, using oblique photography to collect river basin images with a horizontal axis approximately parallel to the direction of the river at a certain height and pitch angle, i.e., oblique photography images with a horizontal axis approximately parallel to the direction of the river;

[0016] S3, using the river target recognition network to identify the collected river basin images, and using the anchor frame to select the ROI area containing the river;

[0017] S4, according to the color space characteristics of the central part of the ROI area, taking the pixel points with similar color space characteristics in the river basin image as foreground objects, and taking the non-foreground object area as background objects;

[0018] S5. Among the foreground objects, according to pixel connectivity, find the largest connected domain as the river water area, take the river water area as the foreground object, and set all non-foreground objects to zero, so as to form a binary river water image;

[0019] S6, obtaining an inverse perspective transformation matrix according to the height and pitch angle when the river basin image is collected, and converting the binary river water body image into a real binary image of the river based on the inverse perspective transformation matrix;

[0020] S7, performing edge detection on the river real binary image to obtain a river edge image, and obtaining the broken edge straight lines on both sides of the river in the river edge image based on heuristic search;

[0021] S8, in the river edge image, using the endpoints of the two broken edge straight lines as boundaries to segment two river bank edge lines;

[0022] S9, traversing the distance between any two points belonging to different river bank edge lines, searching for the minimum distance between the two banks;

[0023] S10, taking the obtained minimum distance as the minimum width of the river, taking the two bank points corresponding to the minimum distance as the narrowest part of the river, and outputting the minimum width and the positions of the two bank points;

[0024] S11, check whether there is an exit instruction, if yes, go to step S13, otherwise go to step S12;

[0025] S12, after the width and narrowness detection instruction is triggered, go to step S2;

[0026] S13, ending the river width detection.

[0027] Preferably, in step S2, an image of a river basin including a river is collected at a certain height and pitch angle by oblique photography; step S5 further comprises: determining whether the current heading has been adjusted, if so, proceeding to step S6, otherwise proceeding to step T1;

[0028] T1, performing edge detection on the binary river water body image to obtain a water body edge image;

[0029] T2, performing Hough line transform on the edge image to obtain several longest straight line segments, forming an initial candidate riverbank straight line set;

[0030] T3, based on heuristic search, searching for two straight line segments with close direction angles and the largest mutual distance from the initial candidate river bank straight line set as the two bank line segments;

[0031] T4, taking two reference points on each of the straight line segments on the two banks, connecting two reference points belonging to different straight line segments to form a first reference line segment and a second reference line segment, and making the intersection of the two reference line segments outside the maximum connected domain;

[0032] T5. respectively find the midpoints of the first reference line segment and the second reference line segment, and form a river direction straight line by connecting the two midpoints;

[0033] T6. Calculate the direction of the river straight line as the river direction, and go to step S2.

[0034] Preferably, the step T6 further comprises: obtaining angle data between the horizontal axis of the image and the straight line of the river direction, and after re-adjusting the orientation of the image acquisition unit according to the angle, proceeding to step S2 to obtain an oblique photographic image with the horizontal axis approximately parallel to the river direction.

[0035] Preferably, the step T3 further includes: in the initial candidate river bank straight line set, if there are two or more straight line segments whose direction angles of two river bank straight line segments are less than a very small fitting angle threshold and whose adjacent endpoints are also close, they can be approximately fitted into one river bank straight line segment.

[0036] Preferably, in step T3, the mutual distance is the minimum value of the actual distance between each point on any one of the two straight line segments and each point on the other line segment; wherein, the actual distance d between a point and each point on a target line segment is calculated as follows: if the line connecting the point to any end point of the target line segment forms an obtuse angle with the target line segment, then the actual distance d is taken as the distance from the point to the end point; otherwise, the actual distance d is taken as the perpendicular distance from the point to the target line segment, i.e., the vertical distance.

[0037] Preferably, in step T4, the reference point is taken as the endpoint of the two bank line segments; and when forming the first reference line segment and the second reference line segment, any reference point is connected to a closer reference point in different straight line segments.

[0038] Preferably, the step S4 also includes: setting a sub-region in a small rectangular block near the intersection of two diagonal lines corresponding to the ROI area rectangle, extracting the value intervals of each of the three color channels in the seed region, and determining that the color thresholds correspond to the threshold intervals of the three primary colors respectively; and taking pixels whose grayscale values ​​meet the determined three primary color threshold intervals as foreground objects.

[0039] Preferably, the step S4 further comprises: selecting a seed point around the intersection of two diagonals corresponding to the ROI area rectangle, and extracting the river water area from the river basin image, i.e. the oblique photography image, as a foreground object through flood filling processing.

[0040] Preferably, the step S1 also includes: establishing a river target recognition network based on the YOLOv5s network, using a bird's-eye view camera to capture river basin images, i.e., river scene images, and then offline training the network with an image sample set consisting of the river scene images with river areas labeled to obtain a river target classification model.

[0041] Preferably, the step S7 further comprises: performing Hough line transform on the river edge image to extract a straight line set; and selecting two straight lines with the largest angle with the horizontal axis and the longest length from the straight line set as the broken edge straight lines on both sides of the river.

[0042] Preferably, the step S8 also includes: representing the edge points in the river edge image with a circular linked list; according to the endpoints of the two river broken edge straight lines, splitting two independent linked lists outside the river broken edge straight lines from the circular linked list to form two river bank edge lines.

[0043] Preferably, step S9 also includes: taking one of the two river bank edge lines as a reference river bank and the other as an opposite river bank; for each point in the reference river bank, calculating its Euclidean distance with each point in the object river bank, traversing and searching for the minimum distance as the minimum width of the river and identifying the corresponding two river bank points located on the reference river bank and the opposite river bank respectively.

[0044] Preferably, the step S5 further comprises: performing a morphological closing operation of first dilation and then erosion on the binary river water body image to remove interference points in the connected domain of the water body.

[0045] Preferably, the step S8 further comprises: when the edge points in the river edge image are represented by a circular linked list, the edge points are arranged according to the connectivity of the edge points and the continuous unidirectional change of the horizontal axis coordinate is used as the priority to form a linked list.

[0046] Preferably, step S9 further includes: for each current bank point in the reference river bank, the bank distance measurement module uses the horizontal coordinate of the current bank point as an index to find the opposite bank reference point with the same horizontal coordinate in the opposite bank linked list, and uses the k points before and after the opposite bank reference point as candidate points; for the current bank point, only the distance between it and the candidate point is calculated, and the minimum distance is updated. Preferably, when the water body flows smoothly and the distance between the two banks of the river does not change drastically, the river width distance is much larger than the single pixel distance, so the distance between the two bank points can be represented by the vertical coordinate difference between the bank points with the same horizontal coordinate.

[0047] Preferably, the step S10 further comprises: obtaining the coordinate information and orientation information of the current river location through the positioning unit, and outputting the river width and the corresponding coordinate and orientation information after detecting the narrowest position of the river. Preferably, after perspective transformation, it is marked in the original image; and the orientation and geographical location points are marked in the image, and the narrowest part of the river information is output.

[0048] Preferably, step S10 further includes: after collecting oblique photographic images of the river scene in sections, calculating the ratio of the narrowest width in each section to the average river width, and when the ratio value is less than a set ratio threshold, issuing an alarm to indicate that the river width at that location has been significantly narrowed.

[0049] Preferably, the step S10 also includes: after detecting the width of the river in segments, marking the narrowest information of each river segment at the corresponding position of the river image, such as using straight lines to mark the two shore points at the narrowest point and the narrowest river crossing line; preferably, storing the acquired coordinates and position information, as well as the narrowest information of the river segment, in a list of the storage module, and the list information can be stored in a cloud server.

[0050] In another embodiment of the present invention, a flight control method for river width detection based on oblique photography is provided, comprising the following steps:

[0051] R1, initialization, establishing the river target recognition network, assigning the heading adjusted variable to negative;

[0052] R2, using oblique photography to collect river basin images at a certain height and elevation angle;

[0053] R3, using the river target recognition network to identify the collected river basin images, if a river is identified, use the anchor frame to select the ROI area containing the river and go to step R4; otherwise, continue flying and wait until the next cycle to go to step R2;

[0054] R4, according to the color space characteristics of the central part of the ROI area, taking the pixel points with similar color space characteristics in the river basin image as foreground objects, and taking the non-foreground object area as background objects;

[0055] R5. Among the foreground objects, according to pixel connectivity, find the largest connected domain as the river water area, take the river water area as the foreground object, and set all non-foreground objects to zero, so as to form a binary river water image;

[0056] Determine whether the heading adjusted variable is true, if so, go to step R6, otherwise go to step TR;

[0057] TR, based on the binary river water body image, extract the water body edge and obtain several longest straight line segments through Hough line transformation, and search for two straight line segments with close direction angles and the largest mutual distance as the two bank segments; take two reference points on any of the two bank segments, connect the reference points belonging to different straight line segments in pairs to form a first reference line segment and a second reference line segment, and make the intersection of the two reference line segments outside the maximum connected domain; respectively find the midpoints of the first reference line segment and the second reference line segment, and form a river direction straight line with the line connecting the two midpoints;

[0058] Adjust the heading of the drone until it is consistent with the normal direction of the straight line of the river, and make the projection of the optical axis collected by the oblique photography image on the horizontal plane consistent with the heading of the drone;

[0059] Assign the heading adjusted variable to yes and go to step R2;

[0060] R6. Obtain an inverse perspective transformation matrix according to the height and pitch angle when the river basin image is collected, and convert the binary river water body image into a real binary image of the river based on the inverse perspective transformation matrix;

[0061] R7, performing edge detection on the river binary image to obtain a river edge image, and obtaining the broken edge straight lines on both sides of the river in the river edge image based on heuristic search;

[0062] R8. In the river edge image, two river bank edge lines are segmented using the endpoints of the two broken edge straight lines as boundaries;

[0063] R9, traversing the distance between any two points belonging to different river bank edge lines, searching for the minimum distance between the two banks;

[0064] R10, taking the obtained minimum distance as the minimum width of the river, taking the two bank points corresponding to the minimum distance as the narrowest part of the river, and outputting the minimum width and the positions of the two bank points;

[0065] R11, detect whether there is a command to fly over the river. If there is, take the nearest one of the two bank points at the narrowest point as the target point and fly towards the target point; then, deflect the heading of the UAV so that the target point points to the direction of the other bank point, and fly over the river along the narrowest river crossing line segment;

[0066] R12, check whether there is an exit instruction, if yes, go to step R14, otherwise go to step R13;

[0067] R13, after the width and narrowness detection instruction is triggered, go to step R2;

[0068] R14. End the river width detection flight control.

[0069] Preferably, the step TR comprises: TR1, performing edge detection on the binary river water body image to obtain a water body edge image;

[0070] TR2, performing Hough line transform on the edge image to obtain several longest straight line segments, forming an initial candidate riverbank straight line set;

[0071] TR3, based on heuristic search, searching for two straight line segments with close direction angles and the largest mutual distance from the initial candidate river bank straight line set as the two bank line segments;

[0072] TR4. Take two reference points on each of the straight line segments on the two banks, connect the reference points belonging to different straight line segments in pairs to form a first reference line segment and a second reference line segment, and make the intersection of the two reference line segments outside the maximum connected domain;

[0073] TR5. Calculate the midpoints of the first reference line segment and the second reference line segment respectively, and form a river direction straight line by connecting the two midpoints;

[0074] TR6. Calculate the direction of the river direction straight line as the river direction;

[0075] TR7. Adjust the heading of the drone and the optical axis of the oblique photography image acquisition unit so that the projection of the optical axis on the horizontal plane is in the same direction as the normal line of the river direction straight line;

[0076] TR8. Assign the heading adjusted variable to yes and go to step R2.

[0077] Wherein, in the step TR7, the optical axis faces the side where the river is located. Preferably, in the step TR7, the normal line, i.e., the vertical line, of the direction line is obtained, the angle between the normal line and the current heading, i.e., the longitudinal axis of the image is calculated, and the angle data is sent to the flight control platform, which controls the UAV to turn in a direction less than 90 degrees so that the heading of the UAV is consistent with the normal line. During this period, the angle data between the horizontal axis of the image and the current direction line of the river can be iteratively obtained, and the heading can be readjusted according to the angle to obtain an oblique photography image in which the horizontal axis is approximately parallel to the direction of the river.

[0078] Preferably, the direction of the river is detected periodically, and the heading of the drone is controlled so that the drone continues to fly along the direction of the river on one side of the river. Then, if the optical axis of the oblique photography image acquisition unit and the yaw angle of the nose are in the same horizontal direction, the heading of the drone is adjusted until it is consistent with the normal direction of the straight line of the river direction; otherwise, if the gimbal of the oblique photography image acquisition unit is independently controlled, the rotation of the gimbal is controlled; in both cases, the projection of the optical axis of the oblique photography image acquisition on the horizontal plane is consistent with the normal direction of the straight line of the river direction.

[0079] Compared with the prior art, the scheme of the present invention has the following advantages: the present invention analyzes the characteristics of river scene images taken by oblique photography using a drone, proposes to use an oblique photography image with a horizontal axis approximately parallel to the direction of the river as a basis, and uses a target classification model based on a deep learning network to locate the ROI area of ​​the river target with an anchor frame, thereby obtaining the river water body target according to the color value interval in the ROI area, and obtaining the river water body object through the maximum connected domain screening, thereby realizing the automatic extraction of the river water body object. After converting the river water body image into a real binary image of the river through inverse perspective transformation, the edge of the river water body is obtained, and the heuristic rules of the river features in the image are used to find the broken edges on both sides of the river, and two river bank edge lines are segmented from the river edge represented by a circular list, thereby realizing the automatic extraction of the shorelines on both sides of the river. Furthermore, by calculating the distance between each point on the shoreline on both sides of the river, the width information of the river is obtained, thereby providing a basis for flights across the river or water conservancy project construction, as well as hydrological monitoring. At the same time, the present invention also uses heuristic rules of river features in the image to find straight lines on both sides of the river at the river edge, and extracts the straight lines of the river direction based on geometric features, so as to automatically obtain an oblique photographic image with a horizontal axis approximately parallel to the river direction according to the river direction.

[0080] The present invention does not require artificial designation of river shorelines, and realizes the automatic acquisition of oblique photographic images with a horizontal axis approximately parallel to the direction of the river and the automatic extraction of the river shoreline; for river areas without prior digital maps, the width of both banks of the river can be automatically detected, and the narrowest position of each section of the river can be identified; it is suitable for detecting the width of rivers in unfamiliar terrain.

[0081] It should be understood that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (provided that such concepts are not mutually inconsistent) can be considered as part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing in the disclosed solution can be considered as part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is a schematic diagram of the processing flow of the river width detection method based on oblique photography;

[0083] Figure 2 It is a schematic diagram of the structure of a river width detection device and system based on oblique photography;

[0084] Figure 3 This is a schematic diagram of the optical axis of the drone's tilt photography;

[0085] Figure 4A This is a structural diagram of the host unit / controller in the river width detection method based on oblique photography. Figure 4B Extract the module structure diagram for the river area. Figure 4C It is the structural diagram of the geometric feature extraction module;

[0086] Figure 5 This is a river scene diagram for the following situation;

[0087] Fig. 6A , Figure 6B They are schematic diagrams of direct binarization and contour extraction in case one;

[0088] Figure 7 This is a schematic diagram of the river anchor frame and color space feature extraction in case one;

[0089] Fig. 8A , Figure 8B They are respectively the binarization based on grayscale interval and the schematic diagram of contour extraction after binarization;

[0090] Fig. 9A , Fig. 9B , Fig. 9C They are the schematic diagrams of maximum connected domain screening, morphological processing, and corresponding river area extraction;

[0091] Fig. 10A , Fig. 10B They are schematic diagrams of the comparison of oblique photography images and image features after inverse perspective transformation;

[0092] Fig.11A This is the binary image of the river area in the world coordinate system in case 1. Fig. 11B is the corresponding edge image, Fig. 11C is the straight line extracted by Hough line transform, Fig.11D Obtain river edge diagram for heuristic search;

[0093] Fig. 12A , Fig. 12B , Fig. 12C They are schematic diagrams of the upper side, lower side and banks of the river obtained by heuristic search;

[0094] Fig.13 This is a schematic diagram of the results obtained from searching the narrowest part of the river;

[0095] Fig.14 This is the river scene diagram under situation 2;

[0096] Fig.15A This is a schematic diagram of river color space feature extraction for situation 2. Fig. 15B This is a schematic diagram of extracting the edge of the river connected domain and the boundary straight line in the edge in case 2. Fig. 15C This is a schematic diagram of the distance detection between two straight line segments. Fig.15D Obtain a schematic diagram for the course of the river;

[0097] Fig.16This is a schematic diagram of a river scene in oblique photography;

[0098] Fig.17 This is a schematic diagram of the shore distance measurement process.

[0099] Among them: 1000 river width detection system based on oblique photography, 900 river width detection device based on oblique photography, 100 host unit / controller, 200 user interface unit, 300 handheld terminal, 400 server, 500 image acquisition unit, 600 flight control platform, 700 positioning unit, 800 river environment / scene,

[0100] 110 processor, 120 memory, 130 input and output modules,

[0101] 111 general processing module, 112 river area extraction module, 113 geometric feature extraction module, 114 river bank extraction module, 115 bank distance measurement module, 116 river direction acquisition module,

[0102] 1121 image acquisition unit, 1122 target detection unit, 1123 river area extraction unit,

[0103] 1131 coordinate transformation unit, 1132 edge detection unit, 1133 straight line extraction and screening unit, 1134 broken edge extraction unit,

[0104] 210 display screen, 220 operation panel. DETAILED DESCRIPTION

[0105] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, but the present invention is not limited to these embodiments. The present invention covers any substitution, modification, equivalent method and scheme made within the spirit and scope of the present invention.

[0106] In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but users skilled in the art can fully understand the present invention without these detailed descriptions.

[0107] The present invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. It should be noted that the accompanying drawings are all simplified and not in exact proportion, and are only used for the purpose of conveniently and clearly assisting in explaining the embodiments of the present invention.

[0108] In my country, rivers are widely distributed. In addition to collecting river and other terrain images through traditional satellite and other high-altitude photography; Fig.16As shown, with the development of drone technology, it is now very convenient to use low-altitude drones to perform oblique photography sampling of rivers. It is of great significance to detect the width of a river and identify the narrowest part of a river. To this end, the present invention discloses a river width detection method, controller and system based on oblique photography, which are used to extract and identify the characteristics of a river according to the image in the field of view of the drone, and find and mark the narrowest position of the river.

[0109] like Figure 1 As shown, the present invention provides a river width detection method based on oblique photography, which includes the following steps:

[0110] S1, initialization, establishing a river target recognition network;

[0111] S2, using oblique photography to collect river basin images with a horizontal axis approximately parallel to the direction of the river at a certain height and pitch angle, i.e., oblique photography images with a horizontal axis approximately parallel to the direction of the river;

[0112] S3, using the river target recognition network to identify the collected river basin images, and using the anchor frame to select the ROI area containing the river;

[0113] S4, according to the color space characteristics of the central part of the ROI area, taking the pixel points with similar color space characteristics in the river basin image as foreground objects, and taking the non-foreground object area as background objects;

[0114] S5. Among the foreground objects, according to pixel connectivity, find the largest connected domain as the river water area, take the river water area as the foreground object, and set all non-foreground objects to zero, so as to form a binary river water image;

[0115] S6, obtaining an inverse perspective transformation matrix according to the height and pitch angle when the river basin image is collected, and converting the binary river water body image into a real binary image of the river based on the inverse perspective transformation matrix;

[0116] S7, performing edge detection on the river real binary image to obtain a river edge image, and obtaining the broken edge straight lines on both sides of the river in the river edge image based on heuristic search;

[0117] S8, in the river edge image, using the endpoints of the two broken edge straight lines as boundaries to segment two river bank edge lines;

[0118] S9, traversing the distance between any two points belonging to different river bank edge lines, searching for the minimum distance between the two banks;

[0119] S10, taking the obtained minimum distance as the minimum width of the river, taking the two bank points corresponding to the minimum distance as the narrowest part of the river, and outputting the minimum width and the positions of the two bank points;

[0120] S11, check whether there is an exit instruction, if yes, go to step S13, otherwise go to step S12;

[0121] S12, after the width and narrowness detection instruction is triggered, go to step S2;

[0122] S13, ending the river width detection.

[0123] like Figure 2 As shown, a river width detection method device 900 based on oblique photography using the method of the present invention includes a host unit 100 and a user interface unit 200; wherein the host unit 100 includes a processor 110, a memory 120 and an input and output module 130, and the user interface unit 200 is provided with a display screen 210 and an operation panel 220. The host unit 110 is configured to: obtain an oblique photography image whose horizontal axis is approximately parallel to the direction of the river, determine an ROI region containing the river in the oblique photography image, and determine a color threshold according to the value intervals of the three color channels extracted from the central part of the ROI region; use pixels in the oblique photography image whose colors are within the color threshold range as foreground objects, and use non-foreground object regions as background objects, and set all background objects in the oblique photography image to zero to form a river target image; in the river target image, detect all connected domains according to the connectivity of the foreground object pixels and select the largest connected domain as the river water body region, and form a binary river water body image using only the river water body region as the foreground object; according to The height and pitch angle of the oblique photography image are obtained to obtain an inverse perspective transformation matrix, and the binary river water body image is converted into a real binary image of the river based on the inverse perspective transformation matrix; the river edge image is obtained after edge detection of the real binary image of the river, and based on heuristic search, the river edge straight lines on both sides are obtained in the river edge image, and two river bank edge lines are divided with the endpoints of the straight lines as dividing points; a bank point is taken on each of the two river bank edge lines, and the Euclidean distance between the two bank points is calculated, the distance is calculated cyclically and the minimum distance is used as the minimum width of the river, and the two bank points corresponding to the minimum distance are the narrowest parts of the river, and the output unit 132 in the input and output module 130 outputs the minimum width and the positions of the two bank points.

[0124] See also Fig.16 As shown in Figure 1, in order to detect the width of a river in an oblique photographic image, it is necessary to know the location of the river bank image points and distinguish the two banks. Existing technologies often do this by manually specifying the river bank. If you want to automatically extract the river bank, how do you do it? At the same time, see Fig.16 , Fig.14 As shown in the figure, the direction of the river is changing. In random natural images, the river may appear to be close to horizontal direction or close to vertical direction. In different cases, how to achieve automatic extraction of river banks?

[0125] For this purpose, see Figure 5 As shown, the present invention detects the width of a river based on an acquired oblique photographic image whose horizontal axis is approximately parallel to the direction of the river.

[0126] To extract the relative geometric features of the river bank and its two sides, it is necessary to first obtain the river outline from the image. Fig. 6A As shown, the acquired color image is directly grayed out and binarized according to the Otsu threshold method; Fig. 6A After extracting the contour of the binary image obtained in the image, mark it in the original image and get the following Figure 6B The contour diagram shown. Fig. 6A In the results shown, the upper edge of the river is an open discontinuous dotted line, while the lower edge has almost no formed edge. It can be seen that it is difficult to obtain effective information for finding the river bank through ordinary binarization processing alone.

[0127] After observing a large number of river images in oblique photography, it is found that the river water body has color space characteristics that are obviously different from the land. Therefore, different modules are set in the processor to extract the river water body and geometric features in sequence.

[0128] See also Figure 2 , Figure 4A As shown, in the processor 110 of the host unit or controller 100, multiple processing modules are set, namely, a general processing module 111, a river area extraction module 112, a geometric feature extraction module 113, a river bank extraction module 114 and a bank distance calculation module 115. Among them, the general processing module 111 responds to events and messages, and starts the extraction of the river water area in the collected oblique photography image, schedules the river width detection processing flow, and outputs the data detected by the bank distance calculation module 115.

[0129] like Figure 4BAs shown, in the acquisition and preprocessing module 112, an image acquisition unit 1121, a target detection unit 1122, and a river region extraction unit 1123 are also provided. The image acquisition unit 1121 acquires the oblique photography image taken by the image acquisition unit 500, and removes high-frequency noise through filtering; then, the target detection unit 1122 identifies the position of the river in the image as the ROI region according to the deep learning feature by a trained target classification model; the river region extraction unit 1123 selects a seed region from the middle of the ROI region to obtain a color threshold, and extracts the river water region from the oblique photography image according to the obtained color threshold to form a binary river water image.

[0130] In the target detection unit 1122, a target classification model based on the YOLOv5s network is established. After the river scene image is collected by the bird's-eye view camera in the image acquisition unit, the network is trained offline with an image sample set consisting of the river scene image with the river area marked, and a river scene target classification model is obtained. During online operation, the target detection unit in the river area extraction module uses the river scene target classification model to perform recognition processing on the current scene image to be tested, i.e., the oblique photography image, to obtain a river area anchor frame, and the area within the anchor frame is used as the river ROI area.

[0131] The river region extraction unit 1123 determines the threshold interval of the color space such as the three primary colors from the central part of the ROI region. Figure 7 As shown, the target classification model based on the deep learning network locates the river target with an anchor frame, i.e., the large rectangular frame, to locate the river ROI area. Then, a sub-region is set in the small rectangular block near the intersection of the two diagonals corresponding to the ROI area rectangle, and the value intervals of the three color channels are extracted in the seed area, and the color thresholds are determined to correspond to the threshold intervals of the three primary colors. Among them, after the upper and lower limits of the grayscale values ​​of each color channel of all pixels in the central seed area are calculated, the upper and lower limits can be extended by a certain distance, such as increasing or decreasing a certain value of 5 to 20, or increasing or decreasing 5 to 10% of the median of the grayscale value interval of this channel.

[0132] In the acquired oblique photography image, the pixels whose grayscale values ​​meet the determined three-primary color threshold interval are taken as foreground objects, and the non-foreground object area is taken as the background object. All background objects in the oblique photography image are set to zero, and all foreground objects are set to 1, that is, binary 255, to form a binary river target image. The result is as follows: Fig. 8A As shown. Fig. 8A After extracting the contour of the binary image obtained in the image, mark it in the original image and get the following Figure 8B Compare Figure 8B and Figure 6B,It can be seen that among the newly acquired contours, in addition to the ,complete river water body contours in the river part, there are also ,many small contours in the land area.

[0133] To this end, the features in the image are analyzed and it is found that in the binary river target image, the water part appears as a large continuous area. Therefore, for the obtained river target image, all connected domains are detected according to the connectivity of the foreground object pixels and the largest connected domain is selected as the river water area; based on the binary river target image, only the foreground pixels in the river water area are used as the foreground objects to form a binary river water image. The result is as follows: Fig. 9A shown.

[0134] exist Fig. 9A In the binary river water image shown in the figure, some non-foreground pixels are formed in the water area due to the cross-river power lines. For such images, morphological operations can be used for further processing. Fig. 9A , perform a morphological closing operation of dilation followed by erosion, and obtain Fig. 9B The binary map shown in Figure 1 removes interference points such as power lines across the river in the water body connectivity domain. Fig. 9B After the binary image is contour extracted, it is marked in the original image, and the result is as follows: Fig. 9C The contour map shown. Thus, a binary map with the river water body as the single target is obtained.

[0135] like Figure 3 As shown in the figure, when the drone is shooting the ground in the air, the angle between the optical axis of the onboard lens and the horizontal plane is α, such as 45 degrees. This oblique photography will cause the perspective transformation effect of near objects appearing larger and far objects appearing smaller in the image space. This effect can be seen in Fig. 10A As shown in the figure, the two parallel sides of the truck roof in reality appear as an eight-shaped straight line in the image. In the image space, the two parallel lines will converge to a vanishing point. For this kind of deformation in the graphic space, the geometric proportion in the world coordinate system can be restored by inverse perspective transformation. Fig. 10B As shown, through the inverse perspective transformation, the parallel relationship between the two sides of the ceiling is restored.

[0136] Perspective transformation uses the condition that the perspective center, image point, and target point are collinear, and according to the law of perspective rotation, the perspective plane is rotated around the perspective axis by a certain angle, destroying the original projection light beam, while still maintaining the projection geometry on the receiving surface unchanged. Inverse perspective transformation is the inverse process of perspective transformation, which combines the intrinsic and extrinsic parameters of the camera to map the image from the image coordinate system to the world coordinate system, thereby eliminating the interference and error caused by perspective effect on the river width detection and identification task.

[0137] The inverse perspective transformation involves transforming the image from the image plane back to the world coordinate system. It is necessary to know the camera's intrinsic parameter matrix, i.e., focal length and optical center, as well as the extrinsic parameter matrix, i.e., rotation and translation, and the corresponding points of the target image and the source image. Generally, based on the capture of multiple images of a chessboard, the camera can be calibrated to obtain the inverse perspective transformation matrix from the image coordinate system to the world coordinate system. For drone oblique photography, there are also transformations based on flight altitude and pitch angle relative to the camera or image coordinate system in the ground calibration experiment. Generally, the drone roll angle is 0 degrees, i.e., horizontal shooting, and the yaw angle is not considered, and the current orientation is taken as the longitudinal direction in the world coordinate system. According to the basic inverse perspective transformation matrix in the ground calibration experiment, it is multiplied by the transformation matrix from the ground to the current oblique photography sampling posture to obtain the inverse perspective transformation matrix from the current image coordinate system to the world coordinate system.

[0138] See also Figure 4C As shown, in the geometric feature extraction module 113, a coordinate transformation unit 1131 and an edge detection unit 1132 are respectively set. Among them, the coordinate transformation unit 1131 determines the inverse perspective transformation matrix from the current image coordinate system to the world coordinate system; and based on the inverse perspective transformation matrix, the obtained binary river water body image is converted into a river real binary image, and the result is as shown in FIG. Fig.11A As shown. For the convenience of display, the black area of ​​the upper part that is not a river is cut off and only the partial image including all non-zero pixels is intercepted. Then, the edge detection unit 1132 uses the edge detection operator such as Canny to detect the edge of the water body on the obtained river real binary image to form a river edge image. The result is shown in Fig. 11B shown.

[0139] Combination Figure 5 , Fig. 11B As shown, although the river edge is obtained, the present invention focuses on the river width. If, based on the obtained river edge, the pixel distance calculation is performed for each edge point in the river edge image to search for the narrowest river width; then, since there is no prior judgment of the river bank of the pixel point, it will be impossible to judge whether the line between pixels is the distance between the two banks.

[0140] To this end, in the geometric feature extraction module 113, a straight line extraction unit 1133 and a broken edge line extraction unit 1134 are also provided. The straight line extraction unit 1133 performs Hough line transformation on the river edge image to obtain a river edge straight line set. The result is as follows: Fig. 11C The broken edge extraction unit 1134 selects the two longest straight line segments in the vertical direction from the obtained river edge straight line set according to the heuristic rule as the river broken edge straight lines on both sides. The result is as follows Fig.11D As shown in Figure 2, the heuristic rules include two aspects: the direction and length of the straight line.

[0141] like Fig. 11C As shown in the figure, there are 9 straight lines in the river edge straight line set. Since the horizontal axis of the oblique photography image is approximately parallel to the river direction, most of the straight lines are located in the river banks on both sides. Figure 5 As shown in the figure, in the oblique photography image, the river's broken edge corresponds to the vertical edge lines on the left and right sides of the image; after the inverse perspective transformation, it appears as two straight lines in an inverted figure eight shape. Therefore, the heuristic rule is defined as finding two straight lines with the largest angle with the horizontal axis and the longest length from the straight line set. Fig.11D As shown in the figure, through heuristic search, the broken edge lines LL and LR on the left and right sides of the river are accurately found.

[0142] After finding the river edge straight lines, the river bank edge lines can be extracted based on the feature of their endpoints. The river bank extraction module 114 set in the processor 110 first represents the edge points in the obtained river edge image in a circular linked list, and then, based on the endpoints of the two river edge straight line segments, separates two independent linked lists outside the river edge straight line segments from the circular linked list to form two river bank edge lines respectively. The upper and lower edge river banks found are respectively as follows: Fig. 12A , Fig. 12B As shown, Fig. 12C The recognition results of two river bank edge lines in the river edge image are illustrated in bold.

[0143] After finding the river bank, the width between the two banks is calculated and the width is identified through the traversal method. The bank distance measurement module 115 set in the processor 110 takes one of the two river bank edge lines as the reference river bank and the other as the opposite river bank. For each point in the reference river bank, the Euclidean distance between it and each point in the target river bank is calculated, and the minimum distance is traversed and searched as the minimum width of the river and the corresponding two river bank points located on the reference river bank and the opposite river bank are marked. The traversal search results are as follows: Fig.13 As shown in FIG. 1 , a straight line is used to mark the two river bank points with the smallest distance. The distance between different points of the river can be the Euclidean distance in the image, or it can be converted into the length in the world coordinate system according to the calibration scale.

[0144] Preferably, the host unit is further configured to obtain the coordinate information and orientation information of the current river bankline through the external positioning unit 700, and after detecting the narrowest position of the river, output the river width information and the corresponding coordinate and orientation information through the input and output unit. Furthermore, after perspective transformation, the river bankline orientation and geographical location points can be marked in the original image, and the geographical information of the narrowest part of the river can be output.

[0145] Example 2

[0146] Based on Example 1, see Figure 4A As shown, this embodiment further provides a river direction acquisition module 116 in the host unit / controller, and the river direction acquisition module 116 is used to acquire the direction of the river and provide river orientation information for acquiring an oblique photographic image with a horizontal axis approximately parallel to the river direction.

[0147] Specifically, in step S2, an image of a river basin including a river is collected at a certain height and pitch angle by oblique photography; step S5 further comprises: determining whether the current heading has been adjusted, if so, proceeding to step S6, otherwise proceeding to step T1;

[0148] T1, performing edge detection on the binary river water body image to obtain a water body edge image;

[0149] T2, performing Hough line transform on the edge image to obtain several longest straight line segments, forming an initial candidate riverbank straight line set;

[0150] T3, based on heuristic search, searching for two straight line segments with close direction angles and the largest mutual distance from the initial candidate river bank straight line set as the two bank line segments;

[0151] T4, taking two reference points on each of the straight line segments on the two banks, connecting two reference points belonging to different straight line segments to form a first reference line segment and a second reference line segment, and making the intersection of the two reference line segments outside the maximum connected domain;

[0152] T5. respectively find the midpoints of the first reference line segment and the second reference line segment, and form a river direction straight line by connecting the two midpoints;

[0153] T6. Calculate the direction of the straight line of the river as the river direction, and go to step S2.

[0154] Preferably, the step T6 further comprises: obtaining angle data between the horizontal axis of the image and the straight line of the river direction, and after re-adjusting the orientation of the image acquisition unit according to the angle, proceeding to step S2 to obtain an oblique photographic image with the horizontal axis approximately parallel to the river direction.

[0155] like Fig.14 As shown in Figure 1, in the images collected randomly from nature, the direction of the river is not necessarily close to the horizontal. Therefore, it is necessary to obtain the direction of the river. Fig.15A After obtaining the oblique photography image, the trained target classification model is used to determine the ROI area containing the river, such as Fig.15AAs shown in the large rectangular block in . The color threshold is determined according to the value intervals of the three color channels extracted from the central part of the ROI area; the pixels in the oblique photography image whose colors are within the color threshold range are taken as foreground objects, and a binary image is formed with only the foreground objects as white pixels; according to the connectivity of the foreground object pixels, the maximum connected domain in the binary image is detected and the edge detection is performed on the maximum connected domain to obtain the water body edge image, and the Hough line transform is performed on the edge image to obtain several longest straight line segments to form an initial candidate river bank straight line set, and the result is shown in FIG. Fig. 15B Shown in the middle left picture.

[0156] In the obtained initial candidate river bank line set, based on the heuristic rule, search for two straight line segments with close direction angles and the largest mutual distance as the two bank line segments. Fig. 15C As shown in the middle left figure, the mutual distance between two straight line segments is the minimum value of the actual distance between each point on any one of the two straight line segments and each point on the other line segment.

[0157] The mutual distance can be obtained by searching the distances between points on two straight line segments; however, through analysis, it is found that the search process can be simplified. Specifically, the mutual distance is defined as the minimum value of the four actual distances composed of the actual distances between the endpoints of any line segment and the points on the other line segment.

[0158] Further, see Fig. 15C As shown in the middle right figure, the actual distance d from a point to a target line segment is calculated as follows: assuming that the line connecting the point to any endpoint of the target line segment forms an obtuse angle with the target line segment, the actual distance d is taken as the distance from the point to the endpoint; otherwise, the actual distance d is taken as the perpendicular distance from the point to the target line segment.

[0159] Fig. 15C In the right figure, in order to find the distance from point A to the line segment Distance First determine whether ∠ABC≥π / 2 is true. If true, then Similarly, determine whether ∠ACB≥π / 2 holds. If so, then As shown in the figure If none of the above equations hold, then make A to The vertical line, It is the distance from A to the foot of the perpendicular, as shown in the figure

[0160] By simplifying the traversal of the distances between points in straight line segments into the calculation of the actual distances from the endpoints to the opposite line segments, and further simplifying the actual distances from the endpoints to the opposite line segments into the calculation of the distances from the endpoints to the endpoints or the feet of perpendiculars of the opposite line segments, the amount of calculation is greatly reduced.

[0161] exist Fig. 15B In the left figure, the two straight line segments with the largest mutual distance found in the edge image are marked with bold lines. Among them, there is one straight line segment on the left bank, and the straight line segment on the right side with the largest distance is the right bank straight line segment. Fig. 15B The right figure shows the identification of the straight line segments of the river bank found in the original oblique photography image. It can be seen that through the above processing, the trend lines of the river banks on both sides of the river can be accurately found.

[0162] Preferably, the step T3 further comprises: if there are two or more straight line segments in the initial candidate river bank straight line set whose direction angles are less than a very small fitting angle threshold and whose adjacent endpoints are also close, they can be approximately fitted into one river bank straight line segment. Fig.15D As shown in FIG. 1 , since there are two straight line segments on the right bank with similar directions and positions, they are merged into one straight line segment as the right bank segment. The left bank segment is extended to the white dotted line AE, and the right bank segment is extended to the white dotted line BF.

[0163] Furthermore, the direction of the river is determined based on the obtained line segments on both banks. Two reference points are taken on any straight line segment in the line segments on both banks, and the reference points belonging to different straight line segments are connected in pairs to form a first reference line segment and a second reference line segment, and the intersection of the two reference line segments is located outside the maximum connected domain. When forming the first reference line segment and the second reference line segment, any reference point is connected to the closer reference point in the different straight line segments. Then, the midpoints of the first reference line segment and the second reference line segment are calculated respectively, and the straight line of the river direction is formed by the line connecting the two midpoints.

[0164] Specifically, see Fig.15D As shown, in the left bank segment, the endpoint E and a point C in the middle are taken, and in the right bank segment, the endpoint F and a point D in the middle are taken. When forming a reference segment, any reference point is connected to the closer reference point in the different straight line segments, so EF and CD are connected to form the first reference segment and the second reference segment. The midpoints W and V of EF and CD are obtained respectively, and the straight line WV connecting the two midpoints is the river direction straight line, and the direction of the obtained river direction straight line is taken as the river direction.

[0165] Preferably, the reference point is taken as the endpoint of the two bank segments. The first reference line segment can be selected by trying to start from the bottom endpoints of the two bank segments, such as Fig.15D AB line segment in the figure; then, find a second reference line segment, such as the EF line segment formed by the top endpoints of the two side segments.

[0166] By orderly pairing the endpoints, the two reference line segments are made far enough apart, and the midpoints of the reference line segments are also far enough apart, thereby effectively improving the fitting accuracy of the river direction.

[0167] Example 3

[0168] Based on the above embodiments, Figure 2 As shown, this embodiment provides a river width detection system 1000 based on oblique photography using the method of the present invention, which includes: an image acquisition unit 500 for image sensing and acquisition of river scenes, a user interface unit 200 for parameter input and operation and river width information display, a server 400 for data management and information transmission, a handheld terminal 300 for remote operation and river width information interaction, and a host unit 100 connected to the user interface unit 200, the server 400, and the image acquisition unit 500.

[0169] The host unit 100 includes an input and output module 130, a processor 110 and a memory 120, and is configured to: obtain an oblique photographic image whose horizontal axis is approximately parallel to the direction of the river, determine a ROI region containing the river in the oblique photographic image, and determine a color threshold according to the value intervals of the three color channels extracted from the central part of the ROI region; use pixels in the oblique photographic image whose colors are within the color threshold range as foreground objects, use non-foreground object areas as background objects, and set all background objects in the oblique photographic image to zero to form a river target image; in the river target image, detect all connected domains according to the connectivity of foreground object pixels and select the largest connected domain as the river water area, and only use the river water area as the background area; A binary river water body image is formed by taking the body area as the foreground object; an inverse perspective transformation matrix is ​​obtained according to the height and pitch angle when the oblique photography image is obtained, and the binary river water body image is converted into a real binary image of the river based on the inverse perspective transformation matrix; a river edge image is obtained after edge detection is performed on the real binary image of the river, and based on heuristic search, the river broken edge straight lines on both sides are obtained in the river edge image, and two river bank edge lines are divided with the endpoints of the broken edge straight lines as dividing points; a bank point is taken on each of the two river bank edge lines, and the Euclidean distance between the two bank points is calculated, the distance is calculated cyclically, and the minimum distance is taken as the minimum width of the river, and the two bank points corresponding to the minimum distance are taken as the narrowest part of the river, and the minimum width and the positions of the two bank points are output.

[0170] Preferably, the river width detection system 1000 based on oblique photography may not include the server 400 and the handheld terminal 300, but the river width detection system 1000 based on oblique photography is connected to the server 400 and the handheld terminal 300 through a network during application. A positioning unit 700 for acquiring coordinates may also be provided in the river width detection system 1000 based on oblique photography.

[0171] Preferably, the river width detection system 1000 based on oblique photography also includes a flight control platform 600. In response to the user's operation on the operation panel 220 or the instruction of the host unit 100, the flight control platform 600 executes the instruction to fly and turn in the specified direction.

[0172] The river width detection system 1000 based on oblique photography continuously processes multiple frames of oblique photography images to detect the river width information. After determining the narrowest position of the river, the direction corresponding to the line connecting the shore points at the narrowest point is used as the target heading, and the target heading information is output to the flight control platform 600 through the output unit, and the flight control platform controls the power unit to make the drone fly over the narrowest point.

[0173] After detecting the width of the river in sections, the narrowest information of each river section is also marked at the corresponding position of the river image, such as using straight lines to mark the two bank points at the narrowest point and the narrowest river crossing line. In addition, the acquired coordinates and position information, as well as the narrowest information of the river section, are stored in a list of the storage module, and the list information can be stored in the cloud server.

[0174] Preferably, when used in a comprehensive application scenario of multiple points and multiple terminals, the system may have multiple handheld terminals and drone platforms accordingly, and for this purpose, subscription and publishing services may be provided. Through the server, the host unit and the remote client deployed at the handheld terminal and the like send messages about the width and narrowness of the river. Preferably, an MQTT server may be deployed, and the server pushes messages to the clients subscribed to the topic. Multiple devices can subscribe to multiple topics at the same time, and when publishing messages, those subscribing to the same topic can receive the same message. The publish-subscribe mode is a messaging mode that decouples the client that sends the message, i.e., the publisher, from the client that receives the message, i.e., the subscriber, so that the two do not need to establish a direct connection or know each other's existence.

[0175] The server provides subscription and publishing services, and the host unit is further configured to: subscribe to the acquisition information topic of the handheld terminal, and transmit the detected river width data and corresponding coordinates and location information to the handheld terminal through the server in response to the topic message; or transmit the width information and corresponding coordinates and location information to the server in the form of width event topic message publishing when the river width narrowing degree is detected to reach a set ratio threshold. The river width narrowing degree is defined as the ratio of the width at the narrowest part of the river to the widest or average width.

[0176] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the river width detection results in the form of a list; the host unit also displays the identified river width in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the narrowest part of each segment on the curve.

[0177] Preferably, the positioning unit in the system also obtains heading information, and also records and identifies the direction identified based on the heading, such as the north direction, and the direction corresponding to the straight line of the river in the output position information.

[0178] Preferably, the host unit can also communicate directly with the handheld terminal. The host unit can be a local control unit deployed separately on the ground, or a control unit deployed on the drone platform. When deployed on the drone platform, the host unit on the drone platform performs edge computing for online river width detection.

[0179] In the embodiment 1, this embodiment also adopts another processing method for extracting river water bodies. The river region extraction unit of the river region extraction module selects a seed point from the middle of the obtained ROI region, and extracts the river water body region from the oblique photography image through flood filling processing.

[0180] The flood filling method is to find the same area based on the difference between the gray values ​​of pixels to achieve segmentation. The gray value of the image is understood as the height of the pixel point, so that an image can be seen as a rugged ground or mountain. A certain amount of water is poured into a low place on the ground, and the water will cover the area below a certain height. The flood filling method uses this principle and is an image processing method based on seed points. It starts from the seed point and spreads to the adjacent pixels, and marks the pixels that meet the conditions as the specified color or value.

[0181] By observing the oblique photography images, it is found that the pixel values ​​of the river water area are very close. Therefore, the present invention sets seed points in the area representing the river and selects the river area by the flood filling method. Figure 7 As shown in the figure, the target classification model based on the deep learning network locates the river target with the anchor frame, that is, the large rectangular frame in the river ROI area. Then, a seed point is set at the intersection of the two diagonals corresponding to the ROI area rectangle, and the river water area is obtained by flood filling. After the pixels outside the obtained river water area are set to zero, a binary river water image is formed.

[0182] Preferably, when the edge points in the river edge image are represented by a circular linked list in step S8, the edge points are arranged and a linked list is formed based on the connectivity of the edge points and the horizontal axis coordinates are continuously changed in one direction as a priority. Thus, the horizontal axis coordinates of the node elements in the linked list of the two river bank edge lines remain unchanged or continue to increase from the front to the back, and remain unchanged or decrease after the turning point. This ordered feature can facilitate the subsequent calculation of the river width.

[0183] Since the river bank is approximately horizontal, we can search for only a few points on the opposite bank that are close to the horizontal axis coordinates of each reference point in the river width calculation. Fig.17 As shown, preferably, step S9 includes:

[0184] S91, assign an initial value to the minimum distance to the river bank, i.e., the minimum width of the river, dmin=inf;

[0185] S92, taking the head node from the reference riverbank linked list as the current reference riverbank point;

[0186] S93, using the x-coordinate of the current reference shore point as an index, finding the opposite shore reference point with the same horizontal coordinate;

[0187] S94, in the opposite bank linked list, taking the k points before and after the opposite bank reference point as a candidate point set, and selecting the first opposite bank point from the candidate point set;

[0188] S95, calculating the distance d between the current reference shore point and the current opposite shore point;

[0189] S96, determine whether d<dmin is established, if so, update dmin, that is, execute dmin=d; otherwise, go to step S97;

[0190] S97, determining whether the current opposite bank point is the last element in the candidate point set, if so, proceeding to step S98; otherwise, taking the next opposite bank point from the candidate point set and proceeding to step S96;

[0191] S98, determining whether the current reference bank point reaches the end of the reference river bank linked list, if so, proceeding to step S99; otherwise, taking the next node from the reference river bank linked list as the current reference bank point, and proceeding to step S93;

[0192] S99. Output the minimum river width dmin.

[0193] The parameter k is optimized according to the flatness of the river bank and the image resolution, and can be set between 5 and 15. By sorting the elements of the linked list and aligning the horizontal axis, it is not necessary to calculate and search the distances to all points on the opposite river bank, which saves the amount of calculation and speeds up the processing.

[0194] As a preferred method, when the water body flows smoothly and the distance between the two banks of the river does not change drastically, the river width distance is much larger than the single pixel distance, so the distance between the two bank points can be represented by the vertical coordinate difference between the bank points with the same horizontal coordinate, thereby avoiding the square root operation and further saving the amount of calculation.

[0195] Example 4

[0196] Based on the above embodiments, this embodiment provides a flight control method for river width detection based on oblique photography, comprising the following steps:

[0197] R1, initialization, establishing the river target recognition network, assigning the heading adjusted variable to negative;

[0198] R2, using oblique photography to collect river basin images at a certain height and elevation angle;

[0199] R3, using the river target recognition network to identify the collected river basin images, if a river is identified, use the anchor frame to select the ROI area containing the river and go to step R4; otherwise, continue flying and wait until the next cycle to go to step R2;

[0200] R4, according to the color space characteristics of the central part of the ROI area, taking the pixel points with similar color space characteristics in the river basin image as foreground objects, and taking the non-foreground object area as background objects;

[0201] R5. Among the foreground objects, according to pixel connectivity, find the largest connected domain as the river water area, take the river water area as the foreground object, and set all non-foreground objects to zero, so as to form a binary river water image;

[0202] Determine whether the heading adjusted variable is true, if so, go to step R6, otherwise go to step TR;

[0203] TR, based on the binary river water body image, extract the water body edge and obtain several longest straight line segments through Hough line transformation, and search for two straight line segments with close direction angles and the largest mutual distance as the two bank segments; take two reference points on any of the two bank segments, connect the reference points belonging to different straight line segments in pairs to form a first reference line segment and a second reference line segment, and make the intersection of the two reference line segments outside the maximum connected domain; respectively find the midpoints of the first reference line segment and the second reference line segment, and form a river direction straight line with the line connecting the two midpoints;

[0204] Adjust the heading of the drone until it is consistent with the normal direction of the straight line of the river, and make the projection of the optical axis collected by the oblique photography image on the horizontal plane consistent with the heading of the drone;

[0205] Assign the heading adjusted variable to yes and go to step R2;

[0206] R6. Obtain an inverse perspective transformation matrix according to the height and pitch angle when the river basin image is collected, and convert the binary river water body image into a real binary image of the river based on the inverse perspective transformation matrix;

[0207] R7, performing edge detection on the river binary image to obtain a river edge image, and obtaining the broken edge straight lines on both sides of the river in the river edge image based on heuristic search;

[0208] R8. In the river edge image, two river bank edge lines are segmented using the endpoints of the two broken edge straight lines as boundaries;

[0209] R9, traversing the distance between any two points belonging to different river bank edge lines, searching for the minimum distance between the two banks;

[0210] R10, taking the obtained minimum distance as the minimum width of the river, taking the two bank points corresponding to the minimum distance as the narrowest part of the river, and outputting the minimum width and the positions of the two bank points;

[0211] R11, detect whether there is a command to fly over the river. If there is, take the nearest one of the two bank points at the narrowest point as the target point and fly towards the target point; then, deflect the heading of the UAV so that the target point points to the direction of the other bank point, and fly over the river along the narrowest river crossing line segment;

[0212] R12, check whether there is an exit instruction, if yes, go to step R14, otherwise go to step R13;

[0213] R13, after the width and narrowness detection instruction is triggered, go to step R2;

[0214] R14. End the river width detection flight control.

[0215] Combination Figure 2 , Figure 4AAs shown, a flight controller 100 for detecting the width of a river based on oblique photography using a flight control method for detecting the width of a river based on oblique photography of the present invention includes a processor 110, a memory 120 and an input / output module 130. The processor 110 includes a general processing module 111, a river region extraction module 112, a geometric feature extraction module 113, a river bank extraction module 114 and a bank distance calculation module 115. The general processing module 111 responds to the message and the operation of the user interface unit, and starts the extraction of the river water area in the acquired oblique photography image, schedules the river width detection processing flow, inputs the real-time acquired oblique photography image into the river region extraction module 112, and outputs the data detected by the bank distance calculation module 115. The general processing module 111 in the controller also obtains the coordinate information and orientation information of the current river location through the positioning unit, and outputs the river information through the output unit after detecting the direction and width information of the river.

[0216] This embodiment acquires an oblique photographic image whose horizontal axis is approximately parallel to the river direction by detecting the river direction and controlling the heading of the drone. Specifically, the step TR includes:

[0217] TR1. Perform edge detection on the binary river water body image to obtain a water body edge image;

[0218] TR2, performing Hough line transform on the edge image to obtain several longest straight line segments, forming an initial candidate riverbank straight line set;

[0219] TR3, based on heuristic search, searching for two straight line segments with close direction angles and the largest mutual distance from the initial candidate river bank straight line set as the two bank line segments;

[0220] TR4. Take two reference points on each of the straight line segments on the two banks, connect the reference points belonging to different straight line segments in pairs to form a first reference line segment and a second reference line segment, and make the intersection of the two reference line segments outside the maximum connected domain;

[0221] TR5. Calculate the midpoints of the first reference line segment and the second reference line segment respectively, and form a river direction straight line by connecting the two midpoints;

[0222] TR6. Calculate the direction of the river direction straight line as the river direction;

[0223] TR7. Adjust the heading of the drone and the optical axis of the tilt photography image acquisition unit so that the projection of the optical axis on the horizontal plane is in the same direction as the normal line of the river direction straight line;

[0224] TR8. Assign the heading adjusted variable to yes and go to step R2.

[0225] Wherein, in the step TR7, the optical axis is directed toward the side where the river is located. Preferably, in the step TR7, the normal line, i.e., the vertical line, of the direction straight line is obtained, the angle between the normal line and the current heading, i.e., the longitudinal axis of the image is calculated, and the angle data is sent to the flight control platform, which controls the UAV to turn in a direction less than 90 degrees so that the heading of the UAV is consistent with the normal line. During this period, the angle data between the horizontal axis of the image and the current direction straight line of the river can be iteratively obtained, and the heading can be readjusted according to the angle to obtain an oblique photographic image in which the horizontal axis is approximately parallel to the direction of the river. Wherein, when the flight exceeds the current image range, in the new scene, the heading adjusted variable should be reassigned to be negative.

[0226] Preferably, the direction of the river is periodically detected, and the heading of the UAV is controlled so that the UAV continues to fly along the direction of the river on one side of the river; then, if the optical axis of the oblique photography image acquisition unit and the yaw angle of the nose are in the same horizontal direction, the heading of the UAV is adjusted until it is consistent with the normal of the straight line of the river direction; otherwise, if the gimbal of the oblique photography image acquisition unit is independently controlled, the rotation of the gimbal is controlled; in both cases, the projection of the optical axis of the oblique photography image acquisition on the horizontal plane is consistent with the normal of the straight line of the river direction.

[0227] Preferably, when it is detected that the lower edge of the image is the edge of the river area, the drone is controlled to fly backward so that the field of view includes two river banks with approximately horizontal directions.

[0228] Combination Fig.16 , Figure 5 , Fig. 11B As shown in the figure, in the process of detecting the river width, generally, if the straight lines where the river broken edges on the left and right sides of the oblique photography image are located include land pixels, the lengths of the two broken edge lines found are relatively close; otherwise, if the lengths of the broken edge lines on both sides are quite different, it can be judged that the direction of the river in the image has changed. By controlling the drone to move left and right and then resampling, an oblique photography image can be obtained in which the directions of the rivers on the left and right sides are approximately parallel to the horizontal axis of the image.

[0229] In order to continuously sample the river scene by oblique photography through the flight of a drone, image acquisition is performed iteratively and continuously, and width detection is performed based on the acquired images. Preferably, the overlapping range of the river scene images processed successively is not less than a set ratio according to the flight speed and frame rate. Preferably, multiple images can also be spliced ​​by aligning feature points, and the river width detection is performed based on the spliced ​​images.

[0230] Preferably, when detecting the width of a river in segments, the distances to several preset points near the broken edge position are not taken into account, and the ranges of adjacent segments are overlapped to achieve full coverage of the river width detection and avoid missed detection of broken edge points due to incomplete search.

[0231] Preferably, after collecting oblique photographic images of river scenes in sections, the ratio of the narrowest width in each section to the average river width is calculated. When the ratio value is less than a set ratio threshold, an alarm is issued to indicate that the river width at that location has been significantly narrowed.

[0232] The present invention is applied to detect the width of rivers in the image space sampled by drones. After detecting the direction of the river, an oblique photographic image with a horizontal axis approximately parallel to the direction of the river is obtained. By automatically identifying river targets, regional seed points or regions are extracted to automatically extract the river water body. After inverse perspective transformation, a binary image and edge line of the water body are obtained. The river bank lines on both sides are found by identifying the broken edges of the river in the edge, so that the river width and its changes can be detected and identified. The present invention can automatically extract the river bank line, accurately detect the river width, and identify the width changes of each section of the river, thereby providing a decision-making basis for drone flight, hydrological detection, etc.

[0233] Several embodiments of the present invention are described above, but these embodiments are presented as examples and do not limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These embodiments or their modifications are included in the scope or main purpose of the invention, and are also included in the invention described in the claims and their equivalents.

Claims

1. A method for detecting river width based on oblique photography, comprising the following steps: S1. Initialization, establishment of river target recognition network: S2, using oblique photography to collect river basin images at a certain height and pitch angle with the horizontal axis approximately parallel to the river direction; S3, using the river target recognition network to identify the collected river basin images, and using the anchor frame to select the ROI area containing the river; S4, according to the color space characteristics of the central part of the ROI area, taking the pixel points with similar color space characteristics in the river basin image as foreground objects, and taking the non-foreground object area as background objects; S5. Among the foreground objects, according to pixel connectivity, find the largest connected domain as the river water area, take the river water area as the foreground object, and set all non-foreground objects to zero, so as to form a binary river water image; S6, obtaining an inverse perspective transformation matrix according to the height and pitch angle when the river basin image is collected, and converting the binary river water body image into a real binary image of the river based on the inverse perspective transformation matrix; S7, performing edge detection on the river real binary image to obtain a river edge image, and obtaining the broken edge straight lines on both sides of the river in the river edge image based on heuristic search; S8, in the river edge image, using the endpoints of the two broken edge straight lines as boundaries to segment two river bank edge lines; S9, traversing the distance between any two points belonging to different river bank edge lines, searching for the minimum distance between the two banks; S10, taking the obtained minimum distance as the minimum width of the river, taking the two bank points corresponding to the minimum distance as the narrowest part of the river, and outputting the minimum width and the positions of the two bank points; S11, check whether there is an exit instruction, if yes, go to step S13, otherwise go to step S12; S12, after the width and narrowness detection instruction is triggered, go to step S2; S13, ending the river width detection.

2. The method for detecting river width based on oblique photography according to claim 1, characterized in that: The step S5 also includes: determining whether the current heading has been adjusted, if so, proceeding to step S6, otherwise proceeding to step T1; T1, performing edge detection on the binary river water body image to obtain a water body edge image; T2, performing Hough line transform on the edge image to obtain several longest straight line segments, forming an initial candidate riverbank straight line set; T3, based on heuristic search, searching for two straight line segments with close direction angles and the largest mutual distance from the initial candidate river bank straight line set as the two bank line segments; T4, taking two reference points on each of the straight line segments on the two banks, connecting two reference points belonging to different straight line segments to form a first reference line segment and a second reference line segment, and making the intersection of the two reference line segments outside the maximum connected domain; T5. respectively find the midpoints of the first reference line segment and the second reference line segment, and form a river direction straight line by connecting the two midpoints; T6. Calculate the direction of the straight line of the river as the river direction, and go to step S2.

3. The method for detecting river width based on oblique photography according to claim 2, characterized in that: The step T3 further includes: in the initial candidate river bank straight line set, if there are two or more straight line segments whose direction angles are less than a very small fitting angle threshold and whose adjacent endpoints are also close, they can be approximately fitted into one river bank straight line segment.

4. The method for detecting river width based on oblique photography according to claim 2, characterized in that: In step T3, the mutual distance is the minimum value of the actual distance between each point on any one of the two straight line segments and each point on the other line segment. The calculation method of the actual distance d between a point and each point on a target line segment is as follows: if the line connecting the point to any endpoint of the target line segment forms an obtuse angle with the target line segment, the actual distance d is the distance from the point to the endpoint; otherwise, the actual distance d is the perpendicular distance from the point to the target line segment, that is, the vertical distance.

5. The method for detecting river width based on oblique photography according to claim 2, characterized in that: In step T4, the reference points are taken as the endpoints of the two bank line segments; and when forming the first reference line segment and the second reference line segment, any reference point is connected to the closer reference point in different straight line segments.

6. The method for detecting river width based on oblique photography according to claim 1, characterized in that: The step S4 also includes: setting a seed region in a small rectangular block near the intersection of two diagonal lines corresponding to the ROI area rectangle, extracting the value intervals of the three color channels in the seed region, and determining that the color thresholds correspond to the threshold intervals of the three primary colors respectively; and taking pixels whose grayscale values ​​meet the determined three primary color threshold intervals as foreground objects.

7. The method for detecting river width based on oblique photography according to claim 1, characterized in that: The step S1 also includes: establishing a river target recognition network based on the YOLOv5s network, using a bird's-eye view camera to collect river basin images, i.e., river scene images, and then offline training the network with an image sample set consisting of the river scene images with river areas marked to obtain a river target classification model.

8. The method for detecting river width based on oblique photography according to claim 1, characterized in that: The step S7 also includes: performing Hough line transform on the river edge image to extract a straight line set; and selecting two straight lines with the largest angle with the horizontal axis and the longest length from the straight line set as the broken edge straight lines on both sides of the river.

9. The method for detecting river width based on oblique photography according to claim 1, characterized in that: The step S8 also includes: representing the edge points in the river edge image with a circular linked list; and according to the endpoints of the two river edge straight lines, splitting two independent linked lists outside the river edge straight lines from the circular linked list to form two river bank edge lines.

10. The method for detecting river width based on oblique photography according to claim 1, characterized in that: The step S9 also includes: taking one of the two river bank edge lines as a reference river bank and the other as an opposite river bank; for each point in the reference river bank, calculating its Euclidean distance with each point in the object river bank, traversing and searching for the minimum distance as the minimum width of the river and marking the corresponding two river bank points located on the reference river bank and the opposite river bank respectively.

Citation Information

Cited By

  • Road image processing method based on inverse perspective transformation and electronic equipment

    CN122492431A