Unmanned aerial vehicle (UAV) rural road width identification method
Through the neural network recognition method, the road images collected by tilt photography are used to calculate the slope and offset of straight lines on both sides of the road, and a width prediction model is established, which solves the problem of road width recognition in rural environments and improves the safety and application of drone landings. sex.
Patent Information
- Application Number
- CN202410947935.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-08
- Publication Date
- 2024-11-08
AI Technical Summary
The prior art is difficult to effectively identify and measure road widths in rural environments, especially when drones land, resulting in limited landing safety, and existing methods rely on prior data and fixed-angle image acquisition, with limited applicability.
Using neural network recognition method, road images are collected through tilt photography, grayscale and binarization processing, edge extraction and Hough line transformation, calculate the slope and oblique distance of straight lines on both sides of the road, and establish a road width prediction model to achieve different Accurate identification and prediction of width roads.
The road width recognition without relying on prior data and fixed angles is realized, which improves the safety and applicability of drones in rural environments, and enhances the real-time and generalization performance of road detection.
Smart Images

Figure CN118918462A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned aerial vehicle (UAV) rural transportation application, and specifically relates to a method for identifying the width of a rural road of an unmanned aerial vehicle (UAV), and a rural flight control method based on the road width. Background Art
[0002] In recent years, the application of drones in rural transportation has gradually increased, and has shown significant advantages and potential. Drones are widely used in many fields, including medical supplies distribution, agricultural monitoring and management, express package delivery, post-disaster rescue and reconstruction, environmental monitoring and protection, infrastructure inspection, education and publicity. In the medical field, drones can quickly deliver medicines, vaccines and blood samples, especially in emergency situations. In agriculture, drones help farmers monitor farmland, detect crop health and pests and diseases, and improve production efficiency through high-resolution cameras and sensors.
[0003] Drones also demonstrate their efficiency in rural express delivery and package delivery. They can quickly deliver goods in areas with inconvenient transportation and improve logistics efficiency. In addition, after a natural disaster, drones can quickly assess the disaster, deliver relief supplies, and assist in rescue work. In terms of environmental monitoring, drones can monitor water quality and air quality and provide forest fire warnings to help protect the ecological environment. Infrastructure inspection is another major application of drones. By regularly inspecting power lines, bridges, and roads, drones improve the efficiency and safety of maintenance work. At the same time, drones can also be used for rural education and publicity, record and spread rural culture, and promote tourism and economic development. During the drone mission, the images captured are a key source of information for autonomous decision-making or intervention.
[0004] For example, the Chinese invention patent with publication number CN116596428B discloses a rural logistics intelligent distribution system based on drones. The system constructs image blocks for each pixel in the smoothed real-life grayscale image, and obtains a uniformity sequence by sorting the degree of change of the grayscale values of the image blocks; the similarity between the image blocks is obtained according to the grayscale fluctuation difference between the image blocks, and finally the noise intensity in the real-life grayscale image is accurately evaluated and the Wiener filter is optimized to obtain a clear real-life picture with a better filtering effect, so as to carry out distribution planning based on the clear real-life picture.
[0005] With the continuous advancement of technology and the reduction of operating costs, the application prospects of drones in rural transportation are becoming more and more broad. In the future, the development trend of intelligence, automation and multi-functionality will further enhance the application effect of drones and promote economic development and improvement of quality of life in rural areas.
[0006] The application of drones has the advantages of being fast, efficient, low-cost and flexible, but it also faces some challenges. For example, when a drone encounters an abnormal situation during transportation, such as insufficient power or weather changes, it needs to land and dock. On the one hand, the ground where the drone lands is usually required to be flat to ensure landing safety; in rural scenes, most of the ground areas are uneven areas such as crops, soil, and greenhouses, and the areas available for drones to land and dock are limited. On the other hand, there are roads of different widths in rural areas, and roads of different widths have different guarantees for the safety of drone landing. The wider the road width, the safer the drone landing.
[0007] At present, the road width is either based on prior data measured on the spot or directly calculated based on the pixel ratio. The former is a Chinese patent application with application number 2020104520176, which discloses a vehicle speed measurement method based on road width and drones. The width information of all roads in the measured area is collected, the ratio of the road width on the image to the actual width is calculated, and the vehicle speed is calculated after obtaining the displacement of the vehicle on the image within a certain period of time. This scheme measures the vehicle speed by the ratio of the actual road width and the road width on the image. Obviously, the width of more rural roads cannot be known a priori, and the rural environment may change constantly. Therefore, this method of collecting information is difficult to have universal applicability.
[0008] The Chinese patent application with application number 2022104047694 adopts the latter method; it discloses a method and system for warning insufficient road width based on a vehicle-mounted drone, using a gimbal camera carried by the drone to measure the actual width of the road to be measured according to the AR mapping information received by the central control screen, and to perform alarm processing through the audio-visual entertainment domain controller according to the measured actual width value of the road; wherein the AR mapping information includes a road measurement line of the road to be measured drawn on the orthophoto of the road to be measured displayed on the central control screen, and the starting point of the road measurement line is one edge of the road to be measured, and the end point of the road measurement line is the other edge of the road to be measured. In this scheme, it relies on an accurate proportional relationship, that is, the pixel width value of the measured line is required to correspond to a priori determined ratio. Obviously, this scheme requires that the drone video screen is fixed at the same angle to be comparable, which undoubtedly greatly limits the measurability of the road.
[0009] Therefore, a method and system for rural road detection and width recognition with strong generalization performance and wide applicability is needed. Summary of the invention
[0010] The purpose of the present invention is to provide a method, controller and system for identifying the width of rural roads using a drone, which are used to identify the characteristics of rural roads based on images collected by the drone and to measure the width of the road surface, without requiring the drone to maintain a certain precise orientation during sampling.
[0011] In order to solve the above technical problems, the present invention provides a method for identifying the width of rural roads by using a drone, which comprises the following steps:
[0012] S1. Initialization, establishing a rural road width recognition neural network, and building a thin-sheet rural road model with two parallel edges;
[0013] S2, in offline state, using oblique photography to collect images of rural road models of different widths under the drone’s field of view at a certain height;
[0014] S3, graying and binarizing the rural road model image to obtain a road target image, and extracting edges from the binarized road target image to obtain a road edge image;
[0015] S4, performing Hough line transform on the road edge image, and obtaining a road edge image containing two straight line edges of the road through heuristic search;
[0016] S5. Calculate the geometric feature quantity of each of the two sidelines relative to the image center according to the two sidelines obtained in the road sideline image;
[0017] S6, judging whether the current stage is offline collection or online recognition, if it is the latter, going to step S12, otherwise, recording the road width of the current road model, and forming the obtained geometric feature quantity and the recorded road width into a training sample;
[0018] S7, determine whether the current training sample needs to be supplemented, if so, go to step S2, otherwise go to step S8;
[0019] S8. Based on the obtained training sample set, a rural road width recognition neural network is trained to obtain a trained road width recognition neural network as a road width prediction model;
[0020] S9, in an online state, perform road width recognition in an actual rural scene; detect whether there is an exit instruction, if so, proceed to step S13, otherwise, proceed to the next step S10 after the road width recognition instruction is triggered;
[0021] S10, acquiring a rural road image, determining a ROI region containing the road, extracting the road region from the ROI region as a foreground object, and setting all non-foreground object regions in the rural road image to zero to form a road target image;
[0022] S11, for the obtained road target image, obtain a road edge image by edge detection, and then go to step S4;
[0023] S12, inputting the geometric feature quantities of the two edge lines relative to the image center into the road width prediction model to obtain a road width prediction value, and outputting the width prediction value; and then going to step S10;
[0024] S13, ending the rural road width recognition.
[0025] Preferably, in step S5, the slope k of the two edge lines obtained from the road edge line image is calculated. 1 , k 2 And the perpendicular distance D from the center of the image to the two sides 1 , D 2 There are 4 geometric features in total;
[0026] In the step S6, it is determined whether the current phase is offline collection or online recognition. If it is the latter, the process goes to step S12. Otherwise, the road width of the current road model is recorded, and the four geometric feature quantities and one road width are formed into a training sample.
[0027] In the step S12, the four obtained geometric feature quantities are input into a road width prediction model to obtain a road width prediction value.
[0028] Preferably, in step S5, after the binary road edge image is subjected to Hough line transform, two straight line segments with the longest lengths are selected as straight lines on both sides of the road; any one of the straight lines on both sides of the road is described by the coordinates of its two end points, and the slope of the straight line and the perpendicular distance from the center point of the image to the straight line are respectively calculated based on the two end points.
[0029] Preferably, in step S5, the method for calculating and extracting geometric features of two road sidelines includes:
[0030] Assume that the coordinates of the two endpoints of the road edge line are (x 1 ,y 1 )、(x 2 ,y 2 ), then its slope is k = y 1 -y 2 / x1 / x 1 -x 2 ;
[0031] The perpendicular distance from the center of the image to the straight line at the edge of the road is
[0032] In the formula, (x 0 ,y 0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is the constant term and: A = y 2 -y 1 , B = x 1 -x 2 , C = -x 1 (y 2 -y 1 )+y 1 (x 2 -x 1 );
[0033] According to the calculation formula of k and d, the slope k of the straight lines on both sides of the road are calculated respectively 1 , k 2 And the perpendicular distance D from the center point of the image to the two straight lines 1 , D 2 There are 4 geometric characteristics in total.
[0034] Preferably, in step S8, training the rural road width recognition neural network includes: T1, initializing the network weights and biases; T2, reading the network parameters and the training sample set recorded in the document; T3, normalizing the sample data; T4, calculating the error between the predicted output value and the expected value of each sample; T5, correcting the network weights and biases by reverse calculation based on the error; T6, if the conditions for the end of network training are met, then end the training, otherwise go to T4 to continue training; T7, analyze and output the results.
[0035] Preferably, in the heuristic search of step S4, when the angle is represented by the slope, when the absolute value of the difference between the slopes of the two edge lines is less than a very small first slope threshold, the two edge lines are merged; at the same time, if the angle tangent value or the slope is greater than a very large second slope threshold, the edge line is determined to be a vertical line perpendicular to the horizontal axis.
[0036] Preferably, step S2 further includes: arranging the road models in different areas of the ground under the field of view of the drone, so that the road models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane.
[0037] Preferably, the sheet-shaped rural road model in step S1 adopts a rectangular sheet, the width of the rectangular sheet is a first preset multiple S1 of the actual road width, and the length of the rectangular sheet is a second preset multiple S2 of its width, and the multiples S1 and S2 are set according to the rural cruising flight altitude and camera resolution of the UAV; the different rectangular sheets used in the road model have a width range corresponding to S1 times the common width range of rural roads, and the width difference between different rectangular sheets, i.e. the spacing, corresponds to S1 times the actual road width difference SD.
[0038] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 30, and the minimum value of the second preset multiple S2 is between 3 and 5. Preferably, the actual road width difference SD is between 0.2 and 0.5 meters, and the width of each rectangular slice used as the road model can form an arithmetic progression with S1 times SD as the tolerance.
[0039] Preferably, the rural road width recognition neural network and the corresponding road width prediction model are multiple, wherein each road width prediction model corresponds to a flight altitude and a bird's-eye view angle when the rural road image is acquired by oblique photography;
[0040] The road width prediction model used in step S12 corresponds to the rural road width recognition neural network corresponding to the flight altitude and the bird's-eye view angle when acquiring the rural road image in step S10.
[0041] Preferably, the rural road width recognition neural network and the corresponding road width prediction model have an input layer that receives the slopes k of the straight lines on both sides of the road from the general processing module. 1 and k 2 , the perpendicular distance D from the center point of the image to the straight lines on both sides of the road 1 and D 2 , as well as the flight altitude and the overlooking angle during collection, a total of 6 input quantities, and the output of the output layer is the road width.
[0042] Preferably, the initialization further includes: pre-establishing a target classification model based on a YOLOv5s network, and after collecting rural scene images using oblique photography, offline training the network with an image sample set consisting of the rural scene images labeled with various targets such as rural road areas to obtain a rural scene target classification model. The step S10 further includes: using the rural scene target classification model to perform recognition processing on the rural scene image to be tested, i.e., the rural road image, to obtain a rural road area anchor frame, and using the area within the anchor frame as the ROI area of the rural road.
[0043] Preferably, the step S10 further includes: selecting a seed point from the middle of the ROI area, obtaining a road area as a foreground object through flood filling processing; and removing the portion outside the foreground object area in the original rural road image to form a road target image.
[0044] Preferably, the step S11 further comprises: for the obtained road target image, converting the color image into a single-channel image, and then using edge detection operators such as Sobel, Canny, LOG, etc. to detect edges to obtain a road edge image.
[0045] Preferably, S4 comprises: when performing the Hough transform on each foreground pixel point in the road edge image, accumulating and obtaining a ρ-θ plane matrix corresponding to the image, wherein ρ represents the distance of the vertical segment of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; in the accumulating, the foreground pixel point (u, v) in the image and a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is:
[0046] Take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where i and j are the row and column labels in the ρ-θ plane matrix, and ε is a preset small positive number.
[0047] Preferably, the step S12 further includes: acquiring coordinate information and orientation information of the current road location, and after detecting the width of the road, outputting the road width and corresponding coordinate and orientation information.
[0048] Preferably, the step S12 further includes: after identifying the road width, marking the road width in the corresponding extracted road area segment of the rural road image, such as using an arrow line to mark the width direction line segment; preferably, storing the acquired coordinates and position information, as well as the road width, in a list of a storage module, and storing the list information in a cloud server.
[0049] In another embodiment of the present invention, a rural flight control method based on road width is provided, comprising the following steps:
[0050] R1. Initialization, establishing a rural road width recognition neural network to map the geometric features of the road edge in the oblique photography image to the road width, collecting training samples and training the rural road width recognition neural network to obtain a road width prediction model;
[0051] R2, in response to the road width recognition command during flight, take oblique photos of the rural scene to obtain rural road images;
[0052] R3, determining a ROI region containing a road in the rural road image, extracting a road region from the ROI region as a foreground object, and setting all non-foreground object regions in the rural road image to zero to form a road target image;
[0053] R4, obtaining a road edge image by edge detection on the obtained road target image; performing Hough line transform on the road edge image, and obtaining a road edge image containing two straight line edges of the road by heuristic search;
[0054] R5. Calculate the geometric feature quantity of each of the two sidelines relative to the image center according to the two sidelines obtained in the road sideline image;
[0055] R6. Input the obtained geometric feature quantity into a road width prediction model to obtain a road width prediction value, and output the width prediction value;
[0056] R7, when the predicted width value is greater than the preset width threshold, the coordinate position value of a point on the central line of the road is given as the target stop point information; otherwise, an alarm is issued to indicate that the road width is insufficient, and the process goes to step R9;
[0057] R8, sending the target stop point information to the flight control platform, and after the flight control platform controls the UAV to land, go to step R11;
[0058] R9, determine whether the road width recognition state is currently exited, if yes, go to step R11, otherwise go to step R10;
[0059] R10, flying along the rural road, when the overlap range between the image in the current field of view and the image of the rural road in the previous frame is less than a certain ratio, issuing a road width recognition instruction, and going to step R2;
[0060] R11. End of rural flight control based on road width.
[0061] Preferably, the identified road width is also displayed in segments on the display screen in the form of a curve, and the coordinates and positions corresponding to the identified width are marked on the curve. Preferably, the heading information is also obtained through the positioning unit, and the direction identified based on the heading, such as the due north direction, and the direction corresponding to the road centerline between the two side lines of the road are also recorded and marked in the output position information.
[0062] Preferably, the step R7 further includes: displaying the road width recognition result in the form of a list on a display device, and the operator selects one of the roads as a target stop point.
[0063] Compared with the prior art, the scheme of the present invention has the following advantages: the present invention analyzes the characteristic representation of road width in the image space of rural road images in view of the unpredictable relative orientation relationship between the posture and the road orientation when the drone collects images, and proposes to use the slope of the straight lines on both sides of the rural road and the vertical distance from the center point of the image to the straight lines on both sides of the road as four geometric features as constraints to establish a neural network to predict the road width; collect samples to identify and train the nonlinear mapping relationship between the width of the rural road and the four proposed geometric features. Among them, each sample can select the rural road by classifying and identifying the target in the aerial image, and obtain the straight lines on both sides of the road by extracting and detecting the edge of the road, and then extracting the four geometric features of the two extracted edges, and the results are input into the trained road width prediction model to make an online prediction of the width of the rural road. The road width prediction model based on neural network established by the present invention has good generalization performance, can accurately predict the width of roads of various specifications according to rural road images collected from uncertain navigation, and has a wide range of applications; at the same time, due to the strong robustness of the established model, when extracting road edges, the step size in the accumulation matrix of the Hough line transform does not need to be set too small, and the correspondence between edge points and straight line elements in the matrix is simplified, the amount of calculation is reduced, and the search speed is accelerated, thereby further improving the real-time performance of rural road detection and width recognition.
[0064] It should be understood that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (provided that such concepts are not mutually inconsistent) can be considered as part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing in the disclosed solution can be considered as part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 This is a schematic diagram of the processing flow of the method for identifying the width of rural roads using drones;
[0066] Figure 2 The structure diagram of the rural road width identification device and system of a drone using the method of the present invention is shown in FIG.
[0067] Figure 3A This is the structure diagram of the host unit / controller in the UAV rural road width identification method. Figure 3B To obtain and preprocess the module structure diagram, Figure 3C This is the structural diagram of the feature extraction module;
[0068] Figure 4A This is a schematic diagram of the neural network model structure of the road width prediction module. Figure 4B This is a schematic diagram of the local structure of the host unit / controller in the method for identifying the width of rural roads using drones;
[0069] Figure 5 This is a schematic diagram of the drone's posture during sample collection;
[0070] Fig. 6A This is a schematic diagram for comparing road models with different widths. Figure 6B Schematic diagram of the road model taken from different angles during sample collection;
[0071] Figure 6C This is a schematic diagram of road contour processing. Fig.6D The road model image after Hough line transformation straight line detection;
[0072] Figure 7 Schematic diagram of the proposed road model width constraint and road geometric features;
[0073] Fig. 8A , Figure 8B Schematic diagram of road edge straight line screening and geometric feature extraction;
[0074] Fig.9A , Fig. 9B Fitting curve between actual and predicted values of road model width in test set;
[0075] Fig. 10A , Fig. 10B Two drone images of rural roads were taken when the measured road width was 4.25m;
[0076] Fig.11A , Fig. 11B Extract schematic diagrams for road areas;
[0077] Fig. 12A , Fig. 12B It is the straight line detection result diagram;
[0078] Fig.13A , Fig. 13B It is a schematic diagram of the road target anchor frame and width identification;
[0079] Fig.14A , Fig. 14B These are two road images of different widths taken by a drone at a flight altitude of 30m;
[0080] Fig.15A , Fig. 15B The straight line detection result diagram of the road edge with different widths;
[0081] Fig.16 This is a schematic diagram of a rural road scene.
[0082] Among them: 1000 UAV rural road width recognition system, 900 UAV rural road width recognition method, 100 host unit / controller, 200 user interface unit, 300 handheld terminal, 400 server, 500 image acquisition unit, 600 flight control platform, 700 positioning unit, 800 rural road environment / scene,
[0083] 110 processor, 120 storage module, 130 input and output module,
[0084] 111 general processing module, 112 acquisition and preprocessing module, 113 feature extraction module, 114 iterative learning module, 115 road width prediction module,
[0085] 1121 image acquisition unit, 1122 target detection unit, 1123 road area extraction unit,
[0086] 1131 edge detection unit, 1132 straight line extraction (screening) unit, 1133 slope feature extraction unit, 1134 distance feature extraction unit,
[0087] 1141 first connection array, 1142 second connection array,
[0088] 1101 neural network model, 131 input unit, 132 output unit
[0089] 210 display screen, 220 operation panel. DETAILED DESCRIPTION
[0090] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, but the present invention is not limited to these embodiments. The present invention covers any substitution, modification, equivalent method and scheme made within the spirit and scope of the present invention.
[0091] In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but users skilled in the art can fully understand the present invention without these detailed descriptions.
[0092] The present invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. It should be noted that the accompanying drawings are all simplified and not in exact proportion, and are only used for the purpose of conveniently and clearly assisting in explaining the embodiments of the present invention.
[0093] Embodiment 1:
[0094] When a drone is performing a mission in a rural area, it needs to land and stop when encountering an abnormal situation. Fig.16As shown in the figure, in rural scenes, there are mostly farmland and vegetable fields; however, the rural road network is relatively developed, and flat roads can be used to provide emergency landing sites. In order to improve the safety of drone landing, it is necessary to understand the width of rural roads to make decisions on landing points.
[0095] Since rural roads are often not standard roads, their width specifications vary; and the property rights may also belong to different entities, width data is difficult to collect a priori. To this end, the present invention provides a method for identifying the width of rural roads by using a drone, which does not rely on a priori width measurement and can estimate the road width by collecting images of rural roads in real time under a non-fixed heading during flight.
[0096] like Figure 1 As shown, the present invention provides a method for identifying the width of a rural road by using a drone, comprising the following steps:
[0097] S1. Initialization, establishing a rural road width recognition neural network, and building a thin-sheet rural road model with two parallel edges;
[0098] S2, in offline state, using oblique photography to collect images of rural road models of different widths under the drone’s field of view at a certain height;
[0099] S3, graying and binarizing the rural road model image to obtain a road target image, and extracting edges from the binarized road target image to obtain a road edge image;
[0100] S4, performing Hough line transform on the road edge image, and obtaining a road edge image containing two straight line edges of the road through heuristic search;
[0101] S5. Calculate the geometric feature quantity of each of the two sidelines relative to the image center according to the two sidelines obtained in the road sideline image;
[0102] S6, judging whether the current stage is offline collection or online recognition, if it is the latter, going to step S12, otherwise, recording the road width of the current road model, and forming the obtained geometric feature quantity and the recorded road width into a training sample;
[0103] S7, determine whether the current training sample needs to be supplemented, if so, go to step S2, otherwise go to step S8;
[0104] S8. Based on the obtained training sample set, a rural road width recognition neural network is trained to obtain a trained road width recognition neural network as a road width prediction model;
[0105] S9, in an online state, perform road width recognition in an actual rural scene; detect whether there is an exit instruction, if so, proceed to step S13, otherwise, proceed to the next step S10 after the road width recognition instruction is triggered;
[0106] S10, acquiring a rural road image, determining a ROI region containing the road, extracting the road region from the ROI region as a foreground object, and setting all non-foreground object regions in the rural road image to zero to form a road target image;
[0107] S11, for the obtained road target image, obtain a road edge image by edge detection, and then go to step S4;
[0108] S12, inputting the geometric feature quantities of the two edge lines relative to the image center into the road width prediction model to obtain a road width prediction value, and outputting the width prediction value; and then going to step S10;
[0109] S13, ending the rural road width recognition.
[0110] like Figure 2 As shown, a drone rural road width identification device 900 using the method of the present invention includes a host unit 100 and a user interface unit 200; wherein the host unit 100 includes a processor 110, a storage module 120 and an input-output module 130, and the user interface unit 200 is provided with a display screen 210 and an operation panel 220. The host unit 110 is configured to: acquire a rural road image, determine a ROI area containing the road in the rural road image, extract the road area from the ROI area of the road as a foreground object, use the non-foreground object area in the rural road image as a background object, and set all background objects in the rural road image to zero to form a road target image; obtain a road edge image by edge detection on the road target image; perform Hough line transformation on the road edge image, and obtain a road sideline image containing two straight edges of the road by heuristic search; obtain geometric features of the straight lines on both sides of the road relative to the image center from the road sideline image, input the geometric features into a trained road width prediction model based on a neural network, obtain a road width prediction value, and output the width prediction value by the output unit 132 in the input-output module 130.
[0111] See also Figure 1 As shown in the figure, in the image captured in the rural scene, the road appears to be wide at one end and narrow at the other end, and in the image, the two parallel sides of the real road will approximately intersect at a vanishing point at infinity. On the one hand, it is obvious that when measuring the distance from the pixels on both sides of the road in the image and then converting it back to the actual width according to the proportional relationship, the proportional relationship is difficult to accurately determine, and it is directly related to the sampling position of the side pixel points. On the other hand, referring to Fig. 10A, Fig. 10B As shown, in practical applications, it is not mandatory to require that the drone must face the same direction as the road, that is, shoot directly at the road to collect images in order to perform proportional measurement, otherwise the applicability of the inspection will be greatly limited.
[0112] Although the width of the rural road in the field of view is certain, when it is sampled from different headings, its pixel width and direction in the image space are uncertain. This certain width and uncertain image features show a contradictory relationship that is difficult to unify. Therefore, this contradiction needs to be decoupled and modeled. The heading of the drone is not fixed during flight, and the road orientation varies greatly; therefore, the relative relationship between the heading of the drone and the road orientation is unpredictable. In order to improve the versatility and applicability of road width detection, it is necessary to study how to characterize the relationship between the two in various relative orientations in the image space, that is, what kind of features in the image can determine the road width?
[0113] After observing the roads in various directions in the image, it is found that the key lies in the two road edges; however, the road edge itself is not the only way to determine the road width. The angle of the straight line itself can only reflect the relative angle difference between it and the heading. After a lot of experiments and tests, it is also found that the road width will affect the imaging effect of the two parallel sides in the image. How to characterize the characteristics of the two sides? Through further research, hypothesis and reasoning, it is proposed to use the slope k of the two parallel lines at different spacings 1 , k 2 And the perpendicular distance D from the center point of the image to the straight lines on both sides of the straight line 1 , D 2 There are 4 input quantities in total, which serve as the constraints on the width of different roads in the image space. Furthermore, how to correspond this constraint to the road width? Since there is a nonlinear and coupled mapping relationship between the two, inspired by the fitting application of neural networks in system nonlinear modeling, the present invention proposes to establish a road width prediction model using neural networks.
[0114] Neural network is a machine learning model that imitates biological neural systems and is particularly good at nonlinear mapping modeling. It learns and represents complex functional relationships through a multi-layer artificial neuron structure. Neural network consists of input layer, hidden layer and output layer; each layer contains several neurons, and each neuron is connected to the neurons in the previous layer through weights. Among them, the input layer receives input data, the hidden layer performs nonlinear transformation and feature extraction, which can have multiple layers, and the output layer outputs the final prediction result.
[0115] like Figure 4AAs shown, the road width prediction model of the present invention adopts a three-layer BP neural network with one hidden layer, and its input layer receives the slopes k of the straight lines on both sides of the road from the general processing module respectively. 1 , k 2 And the perpendicular distance D from the center point of the image to the straight lines on both sides of the road 1 , D 2 There are 4 input quantities in total, and the output of the output layer is the road width.
[0116] To extract the four input quantities or geometric features of the rural road in the image space, it is necessary to first obtain the straight lines on both sides of the road from the image. Figure 3A As shown, in the processor 110 of the host unit or controller 100, multiple processing modules are set, namely, a general processing module 111, an acquisition and preprocessing module 112, a feature extraction module 113, an iterative learning module 114 and a road width prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the recognition of the road width in the collected image, schedules the recognition processing flow, inputs the rural road image collected in real time into the acquisition and preprocessing module 112, and outputs the width data recognized and predicted by the road width prediction module 115.
[0117] The acquisition and preprocessing module 112 acquires the rural road image taken by the image acquisition unit 500, removes high-frequency noise through filtering, and then determines the ROI area including the road according to the deep learning feature or texture feature; and selects a seed point from the middle of the ROI area, and obtains the road area as the foreground object through flood filling processing; in the original rural road image, the background parts corresponding to the fields, trees, etc. outside the foreground object area are removed to form a road target image; the road target image is subjected to edge detection to form a binary road edge image. After the binary road edge image is subjected to Hough line transformation, the two straight line segments with the longest length are selected as the straight lines on both sides of the road; for any of the straight lines on both sides of the road, the feature extraction module 113 describes it with the coordinates of the two end points of the straight line, and respectively obtains the slope of the straight line and the perpendicular distance from the center of the image to the straight line based on the two end points; the slope of the straight lines on both sides of the road and the perpendicular distance from the center of the image to the straight line are used as the geometric features of the road in the rural road image.
[0118] Specifically, Figure 3B As shown, in the acquisition and preprocessing module 112, it is further divided into an image acquisition unit 1121, a target detection unit 1122, and a road area extraction unit 1123.
[0119] The image acquisition unit 1121 acquires the rural road image acquired in real time by the external image acquisition unit 500 through the general processing module 111, or acquires the pre-stored road image from the storage module 120 in an offline state. The road area extraction unit 1123 preliminarily determines the ROI area including the road in the image. Preferably, a YOLO-based method can be established in the target detection unit 1122. v The target classification model of the 5s network is obtained by using the bird's-eye view camera in the image acquisition unit to collect rural scene images, and then using the image sample set composed of the rural scene images with various target areas such as rural roads annotated to train the network offline to obtain the rural scene target classification model; when running online, refer to Fig.13A , Fig. 13B As shown, the road area extraction unit 1123 uses the rural scene target classification model to identify and process the rural scene image including the road, i.e., the rural road image, to obtain a rural road area anchor frame, and uses the area within the anchor frame as the ROI area of the rural road.
[0120] In order to find the actual road part from the ROI rectangular block area, the flood filling method is used for extraction. The flood filling method is to find the same area based on the difference between the grayscale values of pixels to achieve segmentation. The grayscale value of the image is understood as the height of the pixel point, so that an image can be seen as a rugged ground or mountain. A certain amount of water is poured into a low place on the ground, and the water will cover the area below a certain height. The flood filling method uses this principle and is an image processing method based on seed points. It starts from the seed point and spreads to the adjacent pixels, and marks the pixels that meet the conditions as the specified color or value.
[0121] By observing the rural road image, it is found that the pixel values of the area where the road is located are very close. Therefore, the present invention sets up seed points in the area representing the road, and selects the road area by the flood filling method. Specifically, after the target classification model based on the deep learning network locates the ROI area of the road with an anchor frame, a seed point is set at the intersection of the two diagonals corresponding to the rectangle of the ROI area, and the road area is obtained by flood filling. At the same time, the background outside the road area is inverted in the original image, and the background such as fields and trees is eliminated, and the road image we need is highlighted to form a road target image, which is convenient for the subsequent extraction of the straight lines on the edge of the road. Fig.11A , Fig. 11B Corresponding to Fig. 10A , Fig. 10B The original rural road image is the image after the background is removed after the flood filled road image is inverted. For the convenience of display, Fig.11A , Fig. 11B Inverted color processing.
[0122] like Figure 3C As shown, in the feature extraction module 113, an edge detection unit 1131, a straight line extraction and screening unit 1132, a slope feature extraction unit 1133, and a distance feature extraction unit 1134 are also provided.
[0123] After the image content is concentrated on the road area, the edge detection unit 1131 performs edge extraction on the road target image. The edge information in the image is often contained in the area where the grayscale values of adjacent pixels in the image change dramatically. In digital image processing technology, there are many edge detection methods, such as using edge detection operators such as Sobel, Canny, and LOG. Edge extraction can be performed directly on the binary road target image. For the color or grayscale road target image, the color image is first converted into a single-channel image, and then the edge detection operator is used to detect the edge.
[0124] The Canny operator is a second-order derivative edge detection operator, which uses Gaussian smoothing filtering, calculation of gradient amplitude and direction, non-maximum suppression, and hysteresis threshold processing in a series of operation combinations. Compared with the first-order derivative edge detection operator, the Canny operator can find an optimal edge, filter out most of the noise edges, and greatly improve the accuracy of edge detection and edge positioning. The present invention uses the Canny edge detection operator to extract the edge of the road target image. Fig. 12A , Fig. 12B Corresponding to Fig.11A , Fig. 11B The result of edge extraction of the road target image shown is a binary road edge image, which is also subjected to color inversion processing for the convenience of display.
[0125] The straight line extraction and screening unit 1132 extracts two straight line boundaries of the road from the binary road edge image. In the binary road edge image, the most noteworthy are the straight lines representing the boundaries on both sides of the road. First, we need to analyze how to extract these two straight lines, i.e., the edge lines. The present invention uses the Hough line transform. The Hough line transform maps points in the image space to curves in the parameter space, and then finds the intersection of the curves in the parameter space to detect the straight lines. In the image space, the straight line equation can be expressed as: y=kx+b; using polar coordinate notation, the straight line equation can be expressed as: ρ=xcosθ+ysinθ, where, ρ is the vertical distance from the straight line to the origin of the coordinate system, and θ is the angle between the perpendicular line of the straight line and the positive direction of the x-axis.
[0126] The essence of Hough line transform is to establish a mapping relationship between image space and parameter space. On the basis of making full use of the global characteristics of the image, the point-line duality characteristic is used to map the pixel coordinates to the straight lines represented by the elements in the parameter space, and then the analysis is performed in the parameter space. The detection task is completed by cumulative statistics in the parameter space.
[0127] In order to detect straight lines, a plane matrix, namely an accumulator array (ρ, θ), is created in the ρ-θ plane to divide the parameter space into discrete grids. In general, for each edge point (x, y) in the image space, the ρ value of ρ = xcosθ + ysinθ under different θ is calculated, and the corresponding position in the accumulator array is voted. After voting, peak detection is performed, and the element with a larger value in the accumulator array is the detected straight line parameter (ρ, θ). In the heuristic search for obtaining the road edge line, the two straight line segments with the longest length are selected as the two straight edges of the road, that is, the straight lines on both sides of the road.
[0128] In the Hough line transform, the ρ distance resolution is usually 1 pixel, while the θ angle resolution is usually π / 180. Since it is found that the boundary straight line features in rural roads have relatively obvious characteristics, the Hough line transform processing process can be optimized. As a preference, when traversing and accumulating the element values in the ρ-θ plane matrix, the spacing between elements in the value interval can take a larger value according to a priori testing, that is, the spacing between distances and angles is increased accordingly, and the number of rows and columns of the matrix is reduced, thereby greatly reducing the amount of calculation in the calculation judgment and shortening the calculation time.
[0129] Preferably, before peak detection, the elements in the accumulator array (ρ, θ) are merged. When the values of several adjacent elements in the array are above a preset pixel point threshold, i.e., the minimum length of a straight line, the centroid of the polygon enclosed by the coordinates of the several adjacent elements is obtained, and the (ρ, θ) corresponding to the obtained centroid is used as the road edge line.
[0130] Before performing the Hough line transform, a dilation operation can also be used to perform a continuous processing of the binary road edge image A. The specific execution order of the continuous processing is: set an N×N structure element matrix B to generate an all-zero matrix C of the same size as the image A; use the structure element B to traverse the image A to be processed, that is, gradually move the center point of the structure element B along each non-zero grid in the image A, extract the intersection of the non-zero elements of the structure element B and the image A, and if the intersection is not empty, then set the elements in the matrix C, that is, the pixels, to 1 or the pixel value 255 at the same position in the grid of all non-zero elements of the structure element B in A; update the matrix C during the traversal process, and output the matrix C after the traversal is completed, so as to form the final road edge image with the output matrix C. In the continuous processing, as a preferred embodiment, a right-angle L-shaped, T-shaped or cross-shaped structure element can be used. Through the continuous processing, some fine edges caused by vegetation or roadside damage are connected into lines, thereby improving the detection effect of the road edge.
[0131] For the two road edge lines searched, the slope feature extraction unit 1133 and the distance feature extraction unit 1134 respectively calculate the slope of the straight line and the vertical distance from the center point of the image to the straight line. Figure 7 The diagram of the road model width constraint condition shows two sides of a road on the left and right sides of the image plane. The angle and slope of the straight line on the two sides of the left road are α(k1) and β(k2), and the vertical distance from the center of the image to the two sides of the road are D1 and D2 respectively; correspondingly, the angle and slope of the straight line on the two sides of the right road are α′(k1′) and β′(k2′), and the vertical distance from the center of the image to the two sides of the road are D1′ and D2′ respectively. In the oblique photography image plane, the two parallel lines will intersect at a point O at the far end, and point O is called the vanishing point.
[0132] The algebraic equation of a line can be expressed in three common forms, namely the general form, the two-point form, and the slope-intercept form. The general form is: Ax+By+C=0, where A and B are not zero at the same time, and the slope of the line k=-A / B. In the Cartesian rectangular coordinate system, the intercept of the line on the y-axis b=-C / / B, and the general form applies to all line equations.
[0133] The two-point formula is: The two-point equation is the equation of a straight line passing through the two points when the coordinates of the two points are known. The slope of the straight line is k = (y 2 -y 1 ) / (x 2 -x 1 ), it can be seen from the two-point equation that this equation is suitable for describing straight lines that are not perpendicular to the x-axis and y-axis.
[0134] The slope-intercept form is: y=kx+b. The slope-intercept form explicitly expresses the slope k and intercept b of the straight line, and is suitable for representing straight lines that are not perpendicular to the x-axis.
[0135] The straight line detected by Hough line transform in the image is described by the coordinates of the two end points of the straight line, and the two end point coordinates are marked as (x 1 ,y 1 )、(x 2 ,y 2 ), substitute the coordinates of the two points into the two-point equation, and then transform the two-point equation into the slope-intercept equation form to obtain: Its corresponding slope k = y 1 -y 2 / x 1 -x 2 The only difference is that the calculation of the slope of the line is completed in the image coordinate system (the y-axis is downward in the positive direction), so the slope of the line in the image is equal to that in the Cartesian rectangular coordinate system, but has the opposite sign.
[0136] Combined with (x 1 ,y 1 )、(x 2 ,y 2 ) equation, the perpendicular distance from the center of the image to the edge of the road can be calculated as: In the formula, (x 0 ,y 0 ) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is the constant term and: A = y 2 -y 1 , B = x 1 -x 2 , C = -x 1 (y 2 -y 1 )+y 1 (x 2 -x 1 ). According to the calculation formula of k and d, the slope k of the straight lines on both sides of the road is calculated respectively. 1 , k 2 And the perpendicular distance D from the center point of the image to the two straight lines 1 , D 2 A total of 4 quantities serve as the geometric features of the road in the rural road image.
[0137] like Fig. 8A , Figure 8B A schematic diagram of road edge straight line screening and geometric feature extraction is given. Among them, the intersection of diagonal lines indicates the center point of the image. In the set of detected straight lines, there are two straight line segments whose lengths far exceed other straight lines, and thus they are screened as two edge lines of the road. It can also be seen that the angles of the two road edge lines are close. Therefore, in the heuristic search, the straight line can be accepted or excluded as a road boundary line based on whether the length of the straight line segment is a preset ratio of the maximum straight line segment length, such as 0.85 to 0.95 times; and a heuristic rule for the direction of a straight line can also be set, that is, if there are two straight line segments with similar lengths and angles, it is determined that the two straight lines may be road edge lines. In 8A, Figure 8B After excluding the lines whose length is less than the preset ratio of the longest straight line length, for the two selected edge lines, a perpendicular line is drawn from the center point of the image, represented by a dotted line. Figure 8B In the figure, since the foot of the perpendicular is outside the range of the side lines, extend the side lines to the foot of the perpendicular.
[0138] Preferably, a human-computer interaction interface is provided in the user interface unit, and the requirement for the minimum length of a straight line can be reduced for a road section with a larger turning radius in the image, so that the tangent straight lines of a road section with a certain curvature can be marked with straight lines after the Hough transform; and the straight lines on both sides of the road are marked through human-computer interaction, and the geometric features of the marked road edge lines are obtained to predict the road width.
[0139] In order to reveal the nonlinear mapping structure of the road width prediction model of the present invention, a rural road scale model is established and then the model is tilted to obtain training image samples. Figure 5 As shown, when the UAV is shooting the ground in the air, the angle between the optical axis of the onboard lens and the horizontal plane is α, which can be selected from commonly used working angles such as 45 degrees.
[0140] Since the width of actual rural roads varies greatly, it is difficult to obtain standardized template road sections with uniform spacing. To facilitate modeling, the present invention uses rectangular sheets to make road models of various specifications, such as Fig. 6A shown.
[0141] In an offline state, a drone is used to collect images of the prepared road model and form a sample set, and the road width prediction model is trained with the sample set. The road model uses a rectangular sheet, the width of which is a first preset multiple S1 of the actual road width, and the length of which is a second preset multiple S2 of its width, so as to form a model that conforms to a section of the actual road. The multiples S1 and S2 are set according to the rural cruising flight altitude of the drone and the camera resolution; the different rectangular sheets used in the road model have a width range corresponding to S1 times the common width range of rural roads, and the width difference between different rectangular sheets, i.e., the spacing, corresponds to S1 times the actual road width difference SD. For example, when the flight altitude is 100 meters, the sampling height of the corresponding model is 2 meters, so as to ensure that the road model can occupy a significant image space during imaging, so as to coordinate the image resolution and the recognition accuracy of the road width prediction model, and make it unnecessary for the drone to specially lower its altitude to predict the road width, thereby improving the applicable scope of the prediction.
[0142] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 30, and the minimum value of the second preset multiple S2 is between 3 and 5. Preferably, the actual road width difference SD is between 0.2 and 0.5 meters, and the width of each rectangular slice used as the road model can form an arithmetic progression with S1 times SD as the tolerance.
[0143] A road model was built under laboratory conditions, and the actual aerial photography height h real and the actual road width W real Scale proportionally to obtain the aerial photography height h under laboratory conditionsmodel and road width w model In order to find the relationship between the road features and the road width in the image, this embodiment selects the actual aerial photography height h of the drone real When they are 30m, 40m, and 50m respectively, the corresponding aerial photography height h under laboratory conditions model The corresponding sizes are 667mm, 889mm, and 1111mm, and the scaling ratios are:
[0144]
[0145] According to the "Guiding Opinions on Rural Highway Construction Standards", the road width of rural roads is generally not less than 4.5m. However, for sections that are difficult to implement due to economic, topographical, geological and other natural conditions, it can be adapted to local conditions and, after approval, it is not necessary to strictly abide by the "Highway Engineering Technical Standards". Sometimes it may be increased or decreased according to the terrain. When building a road model in the laboratory, most of the plain straight roads are considered, that is, the natural slope of the road is within 3° and the road curvature is close to 0. model With h real The scale ratio is 1:45. Because it is proportional scaling, the width of the road model W model The actual width of the road W real The ratio is also 1:45, and then road models of different widths are made with a step size of 10.5 mm. The step size of the model corresponding to the actual road width is 0.5 m.
[0146] Through field research, it was found that the width of many rural roads does not fully meet the condition that the road width is not less than 4.5m. Therefore, the road width below 4.5m is also taken into account and a road model that meets the corresponding road width is produced. The produced road model is as follows: Fig. 6A As shown. Among them, the road model width W model The 105mm and 210mm road models are marked.
[0147] Since the image sampling of the road model is an oblique image, in order to ensure the diversity of the features representing the road model in the image as much as possible and improve the generalization ability of the network model for predicting road width, the road model is photographed from 0°-180°. The reason for non-360° all-round shooting is that the effect of shooting the road model at 180°-360° is the same as that at 0°-180°, which saves space for model construction. At the same time, in order to highlight the distance relationship between the drone and the actual position of the road, the road model is arranged in different areas of the ground under the field of view of the drone, so that the road models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane. With w model = 136.5 mm as an example, some of the captured data sets are as follows Figure 6BAs shown, in order to improve the accuracy of the model, the photo was taken with a black ground as the background, and for the convenience of display, the picture was inverted.
[0148] The present invention subdivides the road model size to form a width sequence, and shoots from various relative azimuth angles during the sampling of road images, thereby ensuring the sufficiency of samples, thereby improving the generalization ability of the neural network and reducing the prediction error of the road width.
[0149] For the rural road image based on the road model, the collected image is filtered and grayed, then binarized to form a road target image, and then the road edge image is obtained by edge detection. Taking the road model image of a certain location taken at the laboratory sampling height h = 667mm as an example, the preprocessing results of each stage are as follows Figure 6C As shown in the figure, from left to right they correspond to the processing results after grayscale, binarization, and edge detection. The road edge image after Canny edge detection is subjected to Hough line transform, and the transformed image is as follows: Fig.6D As shown in the figure, the left and right figures are the Hough line transform results of the road model with a width of 105mm and 136.5mm respectively. After that, a heuristic search is performed on the transformed straight lines to select the two straight line segments with the longest length as the two straight line edges of the road, i.e., the straight lines on both sides of the road. Fig.6D In the figure, there are two long straight lines after excluding the short line close to the horizontal direction.
[0150] To ensure the correct detection and identification of road edges, a manual confirmation step can be added to the edge search. Specifically, in the process of collecting straight line features representing roads and establishing a training sample library, the interference of non-road boundary straight lines is eliminated through the human-computer interface. With the help of the HighGUI module provided in openCV, a graphical user interface is established and the mouse callback function setMouseCallback() is used. In the callback function, a left mouse button click event and a right mouse button click event are created. Each time the left mouse button is clicked, a straight line detected by Hough line transform is drawn in the image, and it is confirmed after being compared with the original road image.
[0151] Refer to the above Figure 7 For each model road image, the slope k of the two edge lines is calculated. 1 , k 2 And the perpendicular distance D from the center of the image to the two sides 1 , D 2There are 4 geometric features in total, and the corresponding actual model road width is recorded as the true value of the road width. The geometric features and the true value of the road width are formed into samples. After the samples are collected and recorded to form a sample set, they are input into the road width prediction model based on the neural network for offline training. Preferably, the sample set can be stored in the form of a data table composed of input and output true values as a record.
[0152] As a preference, the BP neural network used in the road width prediction model in this embodiment is as follows: Figure 4A As shown, the model is a three-layer structure including a hidden layer, in which
[0153] The output of the jth node in the hidden layer is
[0154] The output of the output layer node is
[0155] In the formula, the f() function is taken as the tansig function, and w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.
[0156] When predicting the road width online, the width of the road model is predicted based on the four geometric features obtained from the road model image. The prediction model then maps it back to the actual road width as the actual output value based on the ratio of the model to the actual value.
[0157] Preferably, multiple neural network-based road width prediction models are created in the host unit, each road width prediction model corresponds to a flight altitude and a bird's-eye view angle of the image acquisition unit; when predicting the road width online, the geometric features obtained after the current acquisition process are input into the road width prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the road width prediction value. In this embodiment, the shooting heights h of the road models are selected to be 667mm, 889mm and 1111mm, respectively, and the shooting angle is 45°.
[0158] The sample data sets with h of 667mm, 889mm and 1111mm were randomly divided into training sets and test sets, and both training sets and test sets covered 17 different road models. The geometric features of the test set were input into the trained network model as input to obtain the predicted value. The fitting comparison curves of the predicted value and the true value of the road model width at different heights are shown in the figure. Fig.9A , Fig. 9BAs shown, their h are 889mm and 1111mm respectively. The test set performance is shown in Table 1.
[0159] Table 1 Road width prediction model test performance
[0160]
[0161] The regression results at three different laboratory heights were analyzed. The coefficients of determination R2 at three different heights were 0.99709, 0.99821, and 0.99912, respectively. The closer the coefficient of determination is to 1, the better the regression effect is, indicating that the model width prediction models at three heights have high reliability. The root mean square error RMSE at three different heights was 2.2156, 2.0322, and 1.8352, respectively, indicating that the predicted value is close to the true value, and the network model has good generalization ability.
[0162] To further verify the feasibility of the model and algorithm, two groups of control experiments were designed by taking actual flight photos of rural roads. In the first group, the same road width was predicted when the actual flight altitudes were 30m, 40m, and 50m, and the prediction error between the predicted value and the true value was calculated to verify the reliability of the model at several common cruising altitudes. In the second group, taking the flight altitude of 30m as an example, the prediction error between the predicted value and the true value of roads of different widths was calculated to verify the generalization ability of the model for predicting different road widths.
[0163] The pictures collected when the actual flight altitude is 30m and 50m and the measured road width is 4.25m are as follows Fig. 10A , Fig. 10B As shown in the figure; the two images after the road image is filled with water and the background is removed by reverse selection are respectively Fig.11A , Fig. 11B As shown, the road edge line is obtained through Hough line transformation and screening as shown in Fig. 12A , Fig. 12B shown.
[0164] After detecting the road straight line, the geometric features are calculated and substituted into the constructed road width prediction models at three different heights. The prediction results are shown in Table 2, where w mp Represents the predicted value of the road model width, W rp Indicates the road prediction value according to the scale ratio (1:45) between the model and the actual road, w real is the actual value of the road.
[0165] Table 2 Prediction results of roads of the same width at three cruising altitudes
[0166]
[0167] It can be seen from Table 2 that when the actual road width is 4.25 m, the relative error between the predicted value and the actual value of the road width is within 2%, indicating that the model prediction accuracy is high.
[0168] In the second set of experiments, taking the 30m height as an example, four roads with different widths were selected. Fig.14A , Fig. 14B Two scenarios with road widths of 5.22m and 6m are given. Fig.15A , Fig. 15B The corresponding road edge straight line detection result diagram is shown in Table 3. The prediction results of the road width prediction model at a height of 30m for roads of different widths are shown in Table 3.
[0169] Table 3 Prediction results of different road widths at h = 30m
[0170]
[0171] It can be seen from Table 3 that under the condition of h = 30m, the prediction model has relatively accurate prediction results for different road widths, with a relative error within 4%. In particular, the road width in the offline model does not correspond to the actual road width, indicating that the model has sufficient generalization performance.
[0172] Example 2
[0173] Based on the previous embodiment, Figure 2 As shown, this embodiment provides a UAV rural road width identification system 1000 using the method of the present invention, which includes: an image acquisition unit 500 for image sensing and acquisition of rural scenes, a user interface unit 200 for parameter input and operation and rural road information display, a server 400 for data management and information transmission, a handheld terminal 300 for remote operation and rural road information interaction, and a host unit 100 connected to the user interface unit 200, the server 400, and the image acquisition unit 500.
[0174] The host unit 100 includes an input / output module 130, a processor 110 and a storage module 120, and is configured to: acquire a rural road image, determine a ROI region containing a road in the rural road image, extract a road region from the ROI region of the road as a foreground object, use a non-foreground object region in the rural road image as a background object, and set all background objects in the rural road image to zero to form a road target image; obtain a road edge image by edge detection on the road target image; perform Hough line transformation on the road edge image, and obtain a road sideline image containing two straight edges of the road by heuristic search; obtain geometric features of the straight lines on both sides of the road relative to the image center from the road sideline image, input the geometric features into a trained road width prediction model based on a neural network, obtain a road width prediction value, and output the width prediction value.
[0175] Preferably, the drone rural road width identification system 1000 may not include the server 400 and the handheld terminal 300, but the drone rural road width identification system is connected to the server 400 and the handheld terminal 300 through a communication network during application. A positioning unit 700 for obtaining coordinates may also be provided in the drone rural road width identification system 1000.
[0176] Preferably, the UAV rural road width identification system 1000 also includes a flight control platform 600. In response to the user's operation on the operation panel 220 or the instruction of the host unit 100, the flight control platform 600 executes the instruction and flies toward the specified road point.
[0177] The UAV rural road width recognition system 1000 continuously detects multiple frames of rural road images and predicts the road width. When the road width meets the requirements, it compares the widths of multiple road sections, determines the section with the largest width as the target landing point, and outputs the coordinate position information of the target landing point to the flight control platform 600 through the output unit. The flight control platform controls the power unit to make the UAV fly to the target landing point.
[0178] Preferably, when drones are used in group systems such as logistics and transportation, the system may have multiple handheld terminals and drone platforms accordingly. Through the server, the host unit and the remote client deployed at the handheld terminal can send messages about the road width. Preferably, an MQTT server can be deployed, and the server pushes messages to the clients that subscribe to the topic. Multiple devices can subscribe to multiple topics at the same time, and when publishing messages, those who subscribe to the same topic can receive the same message. The publish-subscribe mode is a messaging mode that decouples the client that sends the message, i.e., the publisher, from the client that receives the message, i.e., the subscriber, so that the two do not need to establish a direct connection or know each other's existence.
[0179] The server provides subscription and publishing services, and the host unit is also configured to: subscribe to the information acquisition topic of the handheld terminal, and transmit the detected road width and corresponding coordinates and location information to the handheld terminal through the server in response to the topic message; or when it is detected that the road width reaches a set width threshold, transmit the road width and corresponding coordinates and location information to the server in the form of a width event topic message.
[0180] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the road width recognition result in the form of a list; the host unit also displays the recognized road width in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the recognized width on the curve.
[0181] Preferably, the host unit can also communicate directly with the handheld terminal.
[0182] The host unit can be a local control unit deployed separately on the ground, or a control unit deployed on the UAV platform. When deployed on the UAV platform, the host unit on the UAV platform performs edge computing for online road width prediction. Due to the limited resources of the edge computing unit, the processing calculation of the host unit is optimized.
[0183] In the general Hough line transform, for each edge point (x, y) in the image space, the ρ value of ρ = xcosθ + ysinθ under different θ is calculated, and the corresponding position in the accumulator array is voted. This process of calculating ρ requires two floating-point multiplication operations.
[0184] To this end, based on the obvious boundary straight line features in rural roads, the pixel check accumulation process of Hough line transform is optimized. Specifically, when the Hough transform is performed on each foreground pixel point (u, v) in the road edge image, in the cumulative statistics, it is determined whether the pixel point is adjacent to a straight line element l i,j (ρi,j ,θ i,j ) The corresponding judgment method is:
[0185] Take a point H(ρ i,j ·cosθ i ,j,ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where i and j are the row and column labels in the ρ-θ plane matrix, and ε is a preset small positive number.
[0186] By setting the fulcrum, a tangent operation is performed once before traversal judgment, so that each time the foreground pixel is judged, only two subtraction operations and one multiplication operation are required to perform judgment accumulation, thereby avoiding two time-consuming floating-point operations for each pixel, further saving computing costs, and thus suitable for edge computing deployed on drone platforms.
[0187] Preferably, in the heuristic search for obtaining road edge lines, when the angle is expressed as a slope, when the absolute value of the difference in the edge line slopes is less than a very small first slope threshold, the two edge lines are merged; at the same time, if the angle tangent value or the slope is greater than a very large second slope threshold, the edge line is determined to be a vertical line perpendicular to the horizontal axis, thereby preventing the slope from being judged as different straight lines due to slight differences when the slope is very large, and avoiding the false detection interference of short edge lines on the road edge straight lines.
[0188] In the embodiment 1, in the BP neural network of the road width prediction model, the input layer receives the slopes k of the straight lines on both sides of the road from the general processing module. 1 and k 2 , the perpendicular distance D from the center point of the image to the straight lines on both sides of the road 1 and D 2 , and the flight altitude and overlooking angle during acquisition, a total of 6 input quantities, and the output of the output layer is the road width. After establishing a neural network with this structure, the input quantity of each sample in the training set is supplemented, and after supplementing enough samples at different heights and overlooking angles, it is trained with a new training sample set. When applied online, the new model can further simplify the image sampling requirements for rural road prediction; it uses the generalization performance of neural networks to reduce the number of models required for rural road width prediction, and also simplifies the system structure, so that one model can cover different flight sampling scenarios, further improving the applicability of the system.
[0189] Example 3
[0190] Based on the above embodiments, this embodiment provides a rural flight control method based on road width, comprising the following steps:
[0191] R1. Initialization, establishing a rural road width recognition neural network to map the geometric features of the road edge in the oblique photography image to the road width, collecting training samples and training the rural road width recognition neural network to obtain a road width prediction model;
[0192] R2, in response to the road width recognition command during flight, take oblique photos of the rural scene to obtain rural road images;
[0193] R3, determining a ROI region containing a road in the rural road image, extracting a road region from the ROI region as a foreground object, and setting all non-foreground object regions in the rural road image to zero to form a road target image;
[0194] R4. Obtain a road edge image by edge detection on the obtained road target image: perform Hough line transform on the road edge image, and obtain a road edge image containing two straight line edges of the road by heuristic search;
[0195] R5. Calculate the geometric feature quantity of each of the two sidelines relative to the image center according to the two sidelines obtained in the road sideline image;
[0196] R6. Input the obtained geometric feature quantity into a road width prediction model to obtain a road width prediction value, and output the width prediction value;
[0197] R7, when the predicted width value is greater than the preset width threshold, the coordinate position value of a point on the central line of the road is given as the target stop point information; otherwise, an alarm is issued to indicate that the road width is insufficient, and the process goes to step R9;
[0198] R8, sending the target stop point information to the flight control platform, and after the flight control platform controls the UAV to land, go to step R11;
[0199] R9, determine whether the road width recognition state is currently exited, if yes, go to step R11, otherwise go to step R10;
[0200] R10, flying along the rural road, when the overlap range between the image in the current field of view and the image of the rural road in the previous frame is less than a certain ratio, issuing a road width recognition instruction, and going to step R2;
[0201] R11. End of rural flight control based on road width.
[0202] Combination Figure 2 , Figure 3A , Figure 4BAs shown, the UAV rural road width recognition controller 100 using the rural flight control method based on road width of the present invention includes a processor 110, a storage module 120 and an input and output module 130. The processor 110 includes a general processing module 111, an acquisition and preprocessing module 112, a feature extraction module 113, an iterative learning module 114 and a road width prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the road recognition in the collected image, schedules the recognition processing flow, inputs the rural road image collected in real time into the acquisition and preprocessing module 112, and outputs the width data recognized and predicted by the road width prediction module 115.
[0203] Combination Figure 3A , Figure 4B As shown, the rural road image is input into the controller by the input unit 131 in the input-output module 130; the road width prediction model is established in the road width prediction module 115, and its input layer receives the geometric features of the road in the rural road image extracted by the feature extraction module 113 from the general processing module 111, and the output layer is transmitted to the iterative learning module 114 and the general processing module 111 through the first connection matrix 1141 and the second connection matrix 1142 respectively.
[0204] When the road width prediction model is trained offline, the iterative learning module 114 adjusts the connection weights of the network according to the actual road width value input by the general processing module 111 and the road width prediction model through the first connection matrix 1141 and the output value of the neural network in the model; in the field environment, the first connection matrix 1141 is disconnected, and the road width prediction model neural network predicts the width of the road in the current rural road image and outputs it to the general processing module 111 through the second connection matrix 1142, and is processed and analyzed by the general processing module 111 and output through the output unit 132 in the input and output module 130.
[0205] The steps of iterative training include: T1, initializing the weights and biases of the network; T2, reading the network parameters and the training sample set written in the document; T3, normalizing the sample data; T4, calculating the error between the predicted output value and the expected value for each sample; T5, reverse calculation to correct the network weights and biases; T6: if the conditions for the end of network training are met, the training ends, otherwise go to T4 to continue training; T7, result analysis and output. Through multiple experiments, the number of nodes in the hidden layer is determined to be 13, the learning rate is 0.01, and the number of iterations is 2000.
[0206] The general processing module 111 in the controller also obtains the coordinate information and orientation information of the current road location through the positioning unit, and after identifying the width of the road, outputs the road width and the corresponding coordinate and orientation information through the output unit 132.
[0207] like Fig.13A , Fig. 13B As shown, after the road width is identified, the road width is also marked in the road area segment corresponding to the extracted road in the rural road image, such as by using an arrow to mark the width direction line segment. Preferably, the acquired coordinates and position information, as well as the road width, are also stored in a list of the storage module 120, and the list information can be stored in the cloud server 400.
[0208] When the width is greater than the preset width threshold, the coordinate azimuth value of a point on the center line of the road is given; otherwise, an alarm is issued to indicate that the road width is insufficient. During the flight of the drone, image acquisition is performed iteratively and continuously, and the drone rural road width recognition controller 100 performs width prediction processing based on the acquired images. When a width greater than the preset width threshold appears within the field of view, a prompt is issued that a road segment with sufficient width has appeared. Preferably, according to the flight speed and frame rate, the overlapping range of the rural road images processed successively is not less than the set ratio.
[0209] Preferably, the step R7 further includes: displaying the road width recognition result in the form of a list on a display device, and the operator selects one of the roads as a target stop point.
[0210] The present invention is used to identify the width of rural roads in the UAV sampling image space. After road area segmentation, road edge extraction and sideline determination, the slope k of the straight lines on both sides of the road is used to identify the width of rural roads in the UAV sampling image space. 1 , k 2 And the perpendicular distance D from the center point of the image to the straight lines on both sides of the road 1 , D 2 A total of four geometric features were input into the established road width prediction model based on neural network. The model can accurately predict the actual width of rural roads, thus providing a decision-making basis for the flight control and docking of UAVs, and improving the safety and convenience of UAV operations.
[0211] Several embodiments of the present invention are described above, but these embodiments are presented as examples and do not limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These embodiments or their modifications are included in the scope or main purpose of the invention, and are also included in the invention described in the claims and their equivalents.
Claims
1. UAV rural road width recognition method, The following steps are involved: S1. Initialization, establishing a rural road width recognition neural network, and building a thin-sheet rural road model with two parallel edges; S2, in offline state, using oblique photography to collect images of rural road models of different widths under the drone’s field of view at a certain height; S3, graying and binarizing the rural road model image to obtain a road target image, and extracting edges from the binarized road target image to obtain a road edge image; S4, performing Hough line transform on the road edge image, and obtaining a road edge image containing two straight line edges of the road through heuristic search; S5. Calculate the geometric feature quantity of each of the two sidelines relative to the image center according to the two sidelines obtained in the road sideline image; S6, judging whether the current stage is offline collection or online recognition, if it is the latter, going to step S12, otherwise, recording the road width of the current road model, and forming the obtained geometric feature quantity and the recorded road width into a training sample; S7, determine whether the current training sample needs to be supplemented, if so, go to step S2, otherwise go to step S8; S8. Based on the obtained training sample set, a rural road width recognition neural network is trained to obtain a trained road width recognition neural network as a road width prediction model; S9, in an online state, perform road width recognition in an actual rural scene; detect whether there is an exit instruction, if so, proceed to step S13, otherwise, proceed to the next step S10 after the road width recognition instruction is triggered; S10, acquiring a rural road image, determining a ROI region containing the road, extracting the road region from the ROI region as a foreground object, and setting all non-foreground object regions in the rural road image to zero to form a road target image; S11, for the obtained road target image, obtain a road edge image by edge detection, and then go to step S4; S12, inputting the obtained geometric feature quantities of the two edge lines relative to the image center into a road width prediction model to obtain a road width prediction value, and outputting the width prediction value; And go to step S10; S13, ending the rural road width recognition.
2. The method for identifying the width of rural roads by using a drone according to claim 1, It is characterized in that In step S5, the slope k of the two edge lines obtained from the road edge line image is calculated. 1 , k 2 And the perpendicular distance D from the center of the image to the two sides 1 , D 2 There are 4 geometric features in total; In the step S6, it is determined whether the current phase is offline collection or online recognition. If it is the latter, the process goes to step S12. Otherwise, the road width of the current road model is recorded, and the four geometric feature quantities and one road width are formed into a training sample. In the step S12, the four obtained geometric feature quantities are input into a road width prediction model to obtain a road width prediction value.
3. The method for identifying the width of rural roads by using a drone according to claim 1, It is characterized in that In step S5, the method for calculating and extracting geometric features of two road sidelines includes: Assume that the coordinates of the two endpoints of the road edge line are (x 1 ,y 1 )、(x 2 ,y 2 ), then its slope is k = y 1 -y 2 / x 1 -x 2 ; The perpendicular distance from the center of the image to the edge of the road is In the formula, (x 0 ,y 0 ) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is the constant term and: A = y 2 -y 1 , B = x 1 -x 2 , C = -x 1 (y 2 -y 1 )+y 1 (x 2 -x 1 ); According to the calculation formula of k and d, the slope k of the straight lines on both sides of the road are calculated respectively 1 , k 2 And the perpendicular distance D from the center point of the image to the two straight lines 1 , D 2 There are 4 geometric characteristics in total.
4. The method for identifying the width of rural roads by using a drone according to claim 1, It is characterized in that In step S8, training the rural road width recognition neural network includes: T1, initializing the network weights and biases; T2, reading the network parameters and the training sample set recorded in the document; T3, normalizing the sample data; T4, calculating the error between the predicted output value and the expected value of each sample; T5, correcting the network weights and biases based on the error reverse calculation; T6, if the conditions for the end of network training are met, then end the training, otherwise go to T4 to continue training; T7, result analysis and output.
5. The method for identifying the width of rural roads by using a drone according to claim 1, It is characterized in that In the heuristic search of step S4, when the angle is represented by the slope, when the absolute value of the difference between the slopes of the two edge lines is less than a very small first slope threshold, the two edge lines are merged; at the same time, if the angle tangent value or the slope is greater than a very large second slope threshold, the edge line is determined to be a vertical line perpendicular to the horizontal axis.
6. The method for identifying the width of rural roads by using a drone according to claim 1, It is characterized in that The step S2 also includes: arranging the road models in different areas of the ground under the field of view of the drone, so that the road models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane.
7. The method for identifying the width of rural roads by using a drone according to claim 1, It is characterized in that The sheet-shaped rural road model in step S1 adopts a rectangular sheet, the width of the rectangular sheet is a first preset multiple S1 of the actual road width, and the length of the rectangular sheet is a second preset multiple S2 of its width, and the multiples S1 and S2 are set according to the rural cruising flight altitude and camera resolution of the UAV; the different rectangular sheets used in the road model have a width range corresponding to S1 times the common width range of rural roads, and the width difference between different rectangular sheets, i.e. the spacing, corresponds to S1 times the actual road width difference SD.
8. The method for identifying the width of rural roads by using a drone according to claim 1, It is characterized in that The rural road width recognition neural network and the corresponding road width prediction model are multiple, wherein each road width prediction model corresponds to a flight altitude and a bird's-eye view angle when the rural road image is acquired by oblique photography; The road width prediction model used in step S12 corresponds to the rural road width recognition neural network corresponding to the flight altitude and the bird's-eye view angle when acquiring the rural road image in step S10.
9. The method for identifying the width of rural roads by using a drone according to claim 1, It is characterized in that The step S10 further includes: selecting a seed point from the middle of the ROI area, obtaining a road area as a foreground object through flood filling processing; and removing the part outside the foreground object area in the original rural road image to form a road target image.
10. The method for identifying the width of rural roads by using a drone according to claim 1, It is characterized in that The S4 comprises: when performing the Hough transform on each foreground pixel point in the road edge image, accumulating and obtaining a ρ-θ plane matrix corresponding to the image, wherein ρ represents the distance of the vertical segment of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; in the accumulating, the foreground pixel point (u, v) in the image and a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is: Take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where i and j are the row and column labels in the ρ-θ plane matrix, and ε is a preset small positive number.
Citation Information
Patent Citations
Vehicle speed detection method based on road width and unmanned aerial vehicle
CN113723143A
Road width insufficiency early warning method and system based on vehicle-mounted unmanned aerial vehicle
CN114941989A
A rural logistics intelligent distribution system based on drones
CN116596428B
Cited By
Method and device for collecting road image by unmanned aerial vehicle, electronic equipment and medium
CN120416663A
