Unmanned aerial vehicle country road detection and judgment device and system

By using neural network models in drones combined with Hough line transformation and water filling treatment technology, the problem of drones identifying road widths in rural areas is solved, and accurate prediction of uneven ground road widths is achieved, which improves the improvement of Safety and applicability of drone landing.

CN118918465APending Publication Date: 2024-11-08CHINA JILIANG UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410948079.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The prior art is difficult to effectively identify and calculate the width of roads on uneven grounds in rural drone transportation, especially when the course and road orientation are unpredictable when the drone is flying, resulting in limited landing safety.

Method used

The road width prediction model based on neural network is adopted, and the slope of the straight line on both sides of the road in the image and the oblique distance from the center point of the image to the straight line as input features are established. The BP neural network model is established to predict the road width online, combining Hough line transformation and diff. Water filling treatment extracts road edges and ROI areas to achieve accurate identification of road widths of various specifications.

Benefits of technology

It realizes accurate prediction of rural road widths without fixed heading and road orientation, improves the safety and applicability of drone landing, and enhances the generalization performance and real-time nature of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918465A_ABST
    Figure CN118918465A_ABST
Patent Text Reader

Abstract

The invention discloses an unmanned aerial vehicle country road detection and judgment device and system, and the device comprises a user interface unit and a host unit, the host unit obtains a country road image through a processor, extracts a road region from the country road image to form a road target image, and obtains a road edge image through edge detection. Two side lines of a road in an image are obtained through Hough line transformation and heuristic search, the slope of the two side lines and the vertical distance between the center of the image and the side lines are obtained based on geometric calculation and serve as geometric features of the rural road, the geometric features serve as input, and a road width prediction model based on a neural network is established in a processor. After a sample is collected to train the model, the rural road width is estimated on line according to a real shot image, and the rural road width is output from a user interface unit or a handheld terminal. The road width prediction model established by the device is good in generalization performance, can accurately predict the widths of roads of various specifications according to the rural road images collected from different directions, and is high in applicability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle (UAV) rural transportation application, and specifically relates to an unmanned aerial vehicle (UAV) rural road inspection device and system. Background Art

[0002] In recent years, the application of drones in rural transportation has gradually increased, and has shown significant advantages and potential. Drones are widely used in many fields, including medical supplies distribution, agricultural monitoring and management, express package delivery, post-disaster rescue and reconstruction, environmental monitoring and protection, infrastructure inspection, education and publicity. In the medical field, drones can quickly deliver medicines, vaccines and blood samples, especially in emergency situations. In agriculture, drones help farmers monitor farmland, detect crop health and pests and diseases, and improve production efficiency through high-resolution cameras and sensors.

[0003] Drones also demonstrate their efficiency in rural express delivery and package delivery. They can quickly deliver goods in areas with inconvenient transportation and improve logistics efficiency. In addition, after a natural disaster, drones can quickly assess the disaster, deliver relief supplies, and assist in rescue work. In terms of environmental monitoring, drones can monitor water quality and air quality and provide forest fire warnings to help protect the ecological environment. Infrastructure inspection is another major application of drones. By regularly inspecting power lines, bridges, and roads, drones improve the efficiency and safety of maintenance work. At the same time, drones can also be used for rural education and publicity, record and spread rural culture, and promote tourism and economic development. During the drone mission, the images captured are a key source of information for autonomous decision-making or intervention.

[0004] For example, the Chinese invention patent with publication number CN116596428B discloses a rural logistics intelligent distribution system based on drones. The system constructs image blocks for each pixel in the smoothed real-life grayscale image, and obtains a uniformity sequence by sorting the degree of change of the grayscale values ​​of the image blocks; the similarity between the image blocks is obtained according to the grayscale fluctuation difference between the image blocks, and finally the noise intensity in the real-life grayscale image is accurately evaluated and the Wiener filter is optimized to obtain a clear real-life picture with a better filtering effect, so as to carry out distribution planning based on the clear real-life picture.

[0005] With the continuous advancement of technology and the reduction of operating costs, the application prospects of drones in rural transportation are becoming more and more broad. In the future, the development trend of intelligence, automation and multi-functionality will further enhance the application effect of drones and promote economic development and improvement of quality of life in rural areas.

[0006] The application of drones has the advantages of being fast, efficient, low-cost and flexible, but it also faces some challenges. For example, when a drone encounters an abnormal situation during transportation, such as insufficient power or weather changes, it needs to land and dock. On the one hand, the ground where the drone lands is usually required to be flat to ensure landing safety; in rural scenes, most of the ground areas are uneven areas such as crops, soil, and greenhouses, and the areas available for drones to land and dock are limited. On the other hand, there are roads of different widths in rural areas, and roads of different widths have different guarantees for the safety of drone landing. The wider the road width, the safer the drone landing.

[0007] At present, the road width is either based on prior data measured on the spot or directly calculated based on the pixel ratio. The former is a Chinese patent application with application number 2020104520176, which discloses a vehicle speed measurement method based on road width and drones. The width information of all roads in the measured area is collected, the ratio of the road width on the image to the actual width is calculated, and the vehicle speed is calculated after obtaining the displacement of the vehicle on the image within a certain period of time. This scheme measures the vehicle speed by the ratio of the actual road width and the road width on the image. Obviously, the width of more rural roads cannot be known a priori, and the rural environment may change constantly. Therefore, this method of collecting information is difficult to have universal applicability.

[0008] The Chinese patent application with application number 2022104047694 adopts the latter method; it discloses a method and system for warning insufficient road width based on a vehicle-mounted drone, using a gimbal camera carried by the drone to measure the actual width of the road to be measured according to the AR mapping information received by the central control screen, and to perform alarm processing through the audio-visual entertainment domain controller according to the measured actual width value of the road; wherein the AR mapping information includes a road measurement line of the road to be measured drawn on the orthophoto of the road to be measured displayed on the central control screen, and the starting point of the road measurement line is one edge of the road to be measured, and the end point of the road measurement line is the other edge of the road to be measured. In this scheme, it relies on an accurate proportional relationship, that is, the pixel width value of the measured line is required to correspond to a priori determined ratio. Obviously, this scheme requires that the drone video screen is fixed at the same angle to be comparable, which undoubtedly greatly limits the measurability of the road.

[0009] Therefore, there is a need for a device and system for rural road detection and width recognition with strong generalization performance and wide applicability. Summary of the invention

[0010] The purpose of the present invention is to provide a drone rural road detection device, controller and system, which are used to identify the characteristics of rural roads based on images in the drone's field of view and measure their road width, without requiring the drone to maintain a certain precise orientation during sampling.

[0011] In order to solve the above technical problems, the present invention provides a drone rural road detection device, which includes a host unit and a user interface unit, the host unit includes an input and output module, a processor and a storage module, and the user interface unit is provided with an operation panel; the host unit is configured as follows:

[0012] Acquire a rural road image, determine a ROI region containing a road in the rural road image, extract a road region from the ROI region of the road as a foreground object, use a non-foreground object region in the rural road image as a background object, and set all background objects in the rural road image to zero to form a road target image;

[0013] For the road target image, edge detection is performed to obtain a road edge image; after Hough line transformation is performed on the road edge image, a road edge line image containing two straight line edges of the road is obtained through heuristic search;

[0014] The geometric features of the straight lines on both sides of the road relative to the center of the image are obtained from the road sideline image, the geometric features are input into a trained road width prediction model based on a neural network, a road width prediction value is obtained, and the width prediction value is output.

[0015] Preferably, the road width prediction model adopts a BP neural network, whose input layer receives the slopes k of the straight lines on both sides of the road from the general processing module respectively. 1 , k 2 And the perpendicular distance D from the center point of the image to the straight lines on both sides of the road 1 , D 2 There are 4 input quantities in total, and the output quantity of the output layer is the road width.

[0016] Preferably, the host unit is further configured as follows: before online application, firstly, in an offline state, a drone is used to collect images of the produced road model and form a sample set, and the road width prediction model is trained with the sample set; the road model uses a rectangular sheet, the width of the rectangular sheet is a first preset multiple S1 of the actual road width, and the length of the rectangular sheet is a second preset multiple S2 of its width, and the multiples S1 and S2 are set according to the rural cruising flight altitude of the drone and the camera resolution; the different rectangular sheets used in the road model have a width range corresponding to S1 times the common width range of rural roads, and the width difference between different rectangular sheets, i.e., the spacing, corresponds to S1 times the actual road width difference SD. In the process of collecting the sample set, the road models are arranged in different areas in the ground field of view, so that the road models in the sample set are distributed in each square of the nine-square grid formed by the upper, middle, lower, left, middle, and right in the image plane.

[0017] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 30, and the minimum value of the second preset multiple S2 is between 3 and 5. Preferably, the actual road width difference SD is between 0.2 and 0.5 meters, and the width of each rectangular slice used as the road model can form an arithmetic progression with S1 times SD as the tolerance.

[0018] Preferably, the processor includes a general processing module, an acquisition and preprocessing module, a feature extraction module, an iterative learning module and a road width prediction module; the acquisition and preprocessing module acquires the rural road image from the image acquisition unit, determines the ROI area including the road according to the deep learning feature or the texture feature after filtering, selects a seed point from the middle of the ROI area, and obtains the road area as the foreground object through flood filling processing; removes the part outside the foreground object area in the original rural road image to form a road target image, and then forms a binary road edge image through edge detection; after performing Hough line transformation on the binary road edge image, the two straight line segments with the longest length are selected as the straight lines on both sides of the road; for any of the straight lines on both sides of the road, the feature extraction module describes it with the coordinates of the two end points of the straight line, and respectively calculates the slope of the straight line and the perpendicular distance from the center of the image to the straight line based on the two end points; the slope of the straight lines on both sides of the road and the perpendicular distance from the center of the image to the straight line are used as the geometric features of the road in the rural road image.

[0019] Preferably, the host unit is further configured as follows: a target detection unit is set in the acquisition and preprocessing module, a target classification model based on the YOLOv5s network is established in the target detection unit, and after the rural scene image is acquired by the bird's-eye view camera in the image acquisition unit, the network is offline trained with an image sample set consisting of the rural scene images with the rural road areas marked, so as to obtain a rural scene target classification model; when running online, the acquisition and preprocessing module uses the rural scene target classification model to perform recognition processing on the rural scene image including the road to be tested, i.e., the rural road image, to obtain a rural road area anchor frame, and the area within the anchor frame is used as the ROI area of ​​the rural road.

[0020] As a preferred method, for the road image based on the road model, the collected image is filtered and grayed, and then binarized to form a road target image; for the rural road image collected online, the target classification model based on the deep learning network uses an anchor frame to locate the ROI area of ​​the road, and a seed point is set at the intersection of the two diagonals corresponding to the ROI area rectangle, and the road area is obtained by flood filling. After the pixels outside the obtained road area are set to zero, the road target image is formed. The edge extraction can be directly performed on the binary road target image. For the color or grayscale road target image, the color image is first converted into a single-channel image, and then the edge detection operator such as Sobel, Canny, LOG, etc. is used to detect the edge.

[0021] Preferably, a road width prediction model is established in the road width prediction module, and its input layer receives the geometric features of the road in the rural road image from the general processing module, and the output layer transmits them to the iterative learning module and the general processing module through the first connection matrix and the second connection matrix respectively; when training the road width prediction model offline, the iterative learning module adjusts the connection weights of the network according to the actual road width values ​​input by the general processing module and the road width prediction model through the first connection matrix and the output values ​​of the neural network in the model respectively; in the field environment, the first connection matrix is ​​disconnected, and the road width prediction model neural network predicts the width of the road in the current rural road image and outputs it to the general processing module through the second connection matrix, and the general processing module processes and analyzes it and then outputs it through the output part in the input and output module.

[0022] Preferably, the host unit is further configured to: obtain the coordinate information and orientation information of the current road location through the positioning unit, and after detecting the width of the road, output the road width and the corresponding coordinate and orientation information through the input and output unit.

[0023] Preferably, after identifying the road width, the road width is also marked in the corresponding extracted road area segment of the rural road image, such as by using an arrow to mark the width direction line segment; preferably, the acquired coordinates and position information, as well as the road width, are also stored in a list of a storage module, and the list information can be stored in a cloud server.

[0024] When the width is greater than the preset width threshold, the coordinate azimuth value of a point on the center line of the road is given; otherwise, an alarm is issued to indicate that the road width is insufficient. During the flight of the drone, image acquisition is performed iteratively and width prediction processing is performed based on the acquired images. When a width greater than the preset width threshold appears within the field of view, a prompt is issued that a road segment with sufficient width has appeared. Preferably, according to the flight speed and frame rate, the overlapping range of the successively processed rural road images is not less than the set ratio.

[0025] Preferably, the feature extraction module calculates and extracts geometric features of the straight lines on both sides of the road. Assume that the coordinates of the two end points of the straight line on the edge of the road are (x 1 ,y 1 )、(x 2 ,y 2 ), then its slope is k = y 1 -y 2 / x 1 -x 2 ;

[0026] The perpendicular distance from the center of the image to the edge of the road is

[0027] In the formula, (x 0 ,y 0 ) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is the constant term and: A = y 2 -y 1 , B = x 1 -x 2 , C = -x 1 (y 2 -y 1 )+y 1 (x 2 -x 1 );

[0028] According to the calculation formula of k and d, the slope k of the straight lines on both sides of the road are calculated respectively 1 , k 2 And the perpendicular distance D from the center point of the image to the two straight lines 1 , D 2 A total of 4 quantities serve as the geometric features of the road in the rural road image.

[0029] Preferably, the BP neural network used in the road width prediction model is a three-layer structure including a hidden layer, wherein:

[0030] The output of the jth node in the hidden layer is

[0031] The output of the output layer node is

[0032] Among them, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.

[0033] Preferably, a plurality of the neural network-based road width prediction models are created in the host unit, each road width prediction model corresponds to a flight altitude and a bird's-eye view angle of the image acquisition unit; when performing road width prediction online, the geometric features obtained after the current acquisition process are input into the road width prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the road width prediction value.

[0034] Preferably, in the BP neural network of the road width prediction model, the input layer receives the slopes k of the straight lines on both sides of the road from the general processing module respectively. 1 and k 2 , the perpendicular distance D from the center point of the image to the straight lines on both sides of the road 1 and D 2 , as well as the flight altitude and the overlooking angle during collection, a total of 6 input quantities, and the output of the output layer is the road width.

[0035] Preferably, the host unit is further configured to: when performing the Hough transform on each foreground pixel point in the road edge image, accumulate and obtain the ρ-θ plane matrix corresponding to the image, wherein ρ represents the vertical segment distance of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; in the accumulation, the foreground pixel point (u, v) in the image and a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is:

[0036] Take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθi,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where i and j are the row and column labels in the ρ-θ plane matrix, and ε is a preset small positive number.

[0037] In another embodiment of the present invention, a rural road inspection system for a drone is provided, which includes: an image acquisition unit for performing image sensing and acquisition of rural scenes, a user interface unit for parameter input and operation and rural road information display, a server for data management and information transmission, a handheld terminal for remote operation and rural road information interaction, and a host unit connected to the user interface unit, the server, and the image acquisition unit;

[0038] The host unit includes an input and output module, a processor and a storage module, and is configured to: obtain a rural road image, determine a ROI area containing a road in the rural road image, extract a road area from the ROI area of ​​the road as a foreground object, use a non-foreground object area in the rural road image as a background object, and set all background objects in the rural road image to zero to form a road target image; obtain a road edge image for the road target image through edge detection; obtain a road sideline image containing two straight edges of the road through heuristic search after performing Hough line transformation on the road edge image; obtain geometric features of the straight lines on both sides of the road relative to the image center from the road sideline image, input the geometric features into a trained road width prediction model based on a neural network, obtain a road width prediction value, and output the width prediction value.

[0039] Preferably, the drone rural road inspection system may not include the server and the handheld terminal, but the drone rural road inspection system is connected to the server and the handheld terminal through a communication network during application. A positioning unit for obtaining coordinates may also be provided in the drone rural road inspection system.

[0040] Preferably, the UAV rural road inspection system also includes a flight control platform. In response to the user's operation on the operation panel or the instruction of the host unit, the flight control platform executes the instruction and flies towards the specified road point.

[0041] The UAV rural road detection system continuously detects multiple frames of rural road images and predicts the road width. When the road width meets the requirements, the widths of multiple sections of the road are compared, and the section with the largest width is determined as the target landing point. The coordinate position information of the target landing point is output to the flight control platform through the output unit, and the flight control platform controls the power unit to make the UAV fly to the target landing point.

[0042] Preferably, when drones are used in group systems such as logistics and transportation, the system may have multiple handheld terminals and drone platforms, and subscription and publishing services may be provided for this purpose; the host unit is also configured to: subscribe to the acquisition information topic of the handheld terminal, and in response to the topic message, transmit the detected road width and corresponding coordinates and location information to the handheld terminal through the server; or when it is detected that the road width reaches a set width threshold, transmit the road width and corresponding coordinates and location information to the server in the form of a width event topic message.

[0043] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the road width recognition result in the form of a list; the host unit also displays the recognized road width in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the recognized width on the curve.

[0044] Preferably, the positioning unit in the system also obtains heading information, and also records and identifies in the output position information the direction identified based on the heading, such as the north direction, and the direction corresponding to the center line of the road between the two side lines of the road.

[0045] Compared with the prior art, the scheme of the present invention has the following advantages: the present invention aims at the unpredictable relative orientation relationship between the posture and the road orientation when the drone collects images, analyzes the characteristic representation of the road width in the image space of the rural road image, proposes to use the slopes of the straight lines on both sides of the rural road and the vertical distances from the center point of the image to the straight lines on both sides of the road as four geometric features as constraints, and establishes a neural network to predict the road width; collects samples to identify and train the nonlinear mapping relationship between the rural road width and the four proposed geometric features; wherein each sample can select the rural road by classifying and identifying the target in the aerial image, extracts and detects the road edge, and obtains the straight lines on both sides of the road through Hough line transformation screening, extracts the four geometric features of the two extracted edge lines, and the result is input into the trained road width prediction model to perform online prediction of the width of the rural road. The road width prediction model based on neural network established by the present invention has good generalization performance, can accurately predict the width of roads of various specifications according to the rural road images collected from the uncertain navigation direction, and has a wide range of applications; at the same time, due to the strong robustness of the established model, when the road edge is extracted, the step size in the accumulation matrix of the Hough line transform does not need to be set too small, and the correspondence between the edge points and the straight line elements in the matrix is ​​simplified, the calculation amount is reduced, and the search speed is accelerated, thereby further improving the real-time performance of rural road detection.

[0046] It should be understood that all combinations of the aforementioned concepts and the additional concepts discussed in more detail below (provided that such concepts are not mutually inconsistent) can be considered as part of the inventive subject matter disclosed herein. In particular, all combinations of the claimed subject matter appearing in the disclosed solution can be considered as part of the inventive subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of a rural road scene;

[0048] Figure 2 This is a schematic diagram of the structure of the UAV rural road inspection device and system;

[0049] Figure 3A This is the structure diagram of the host unit / controller in the UAV rural road inspection device. Figure 3B To obtain and preprocess the module structure diagram, Figure 3C This is the structural diagram of the feature extraction module;

[0050] Figure 4A This is a schematic diagram of the neural network model structure of the road width prediction module. Figure 4B This is a schematic diagram of the partial structure of the host unit / controller in the UAV rural road inspection device;

[0051] Figure 5 This is a schematic diagram of the drone's posture during sample collection;

[0052] Fig. 6A This is a schematic diagram for comparing road models with different widths. Figure 6B Schematic diagram of the road model taken from different angles during sample collection;

[0053] Figure 6C This is a schematic diagram of road contour processing. Fig.6D The road model image after Hough line transformation straight line detection;

[0054] Figure 7 Schematic diagram of the proposed road model width constraint and road geometric features;

[0055] Fig. 8A , Figure 8B Schematic diagram of road edge straight line screening and geometric feature extraction;

[0056] Fig.9A , Fig. 9B Fitting curve between actual and predicted values ​​of road model width in test set;

[0057] Fig. 10A , Fig. 10B Two drone images of rural roads were taken when the measured road width was 4.25m;

[0058] Fig.11A , Fig. 11B Extract schematic diagrams for road areas;

[0059] Fig. 12A , Fig. 12B It is the straight line detection result diagram;

[0060] Fig.13A , Fig. 13B It is a schematic diagram of the road target anchor frame and width identification;

[0061] Fig.14A , Fig. 14B These are two road images of different widths taken by a drone at a flight altitude of 30m;

[0062] Fig.15A , Fig. 15B The following are the straight line detection results for road edges of different widths.

[0063] Among them: 1000 UAV rural road inspection system, 900 UAV rural road inspection device, 100 host unit / controller, 200 user interface unit, 300 handheld terminal, 400 server, 500 image acquisition unit, 600 flight control platform, 700 positioning unit, 800 rural road environment / scene,

[0064] 110 processor, 120 storage module, 130 input and output module,

[0065] 111 general processing module, 112 acquisition and preprocessing module, 113 feature extraction module, 114 iterative learning module, 115 road width prediction module,

[0066] 1121 image acquisition unit, 1122 target detection unit, 1123 road area extraction unit,

[0067] 1131 edge detection unit, 1132 straight line extraction (screening) unit, 1133 slope feature extraction unit, 1134 distance feature extraction unit,

[0068] 1141 first connection array, 1142 second connection array,

[0069] 1101 neural network model, 131 input unit, 132 output unit

[0070] 210 display screen, 220 operation panel. DETAILED DESCRIPTION

[0071] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, but the present invention is not limited to these embodiments. The present invention covers any substitution, modification, equivalent method and scheme made within the spirit and scope of the present invention.

[0072] In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but users skilled in the art can fully understand the present invention without these detailed descriptions.

[0073] The present invention is described in more detail in the following paragraphs by way of example with reference to the accompanying drawings. It should be noted that the accompanying drawings are all simplified and not in exact proportion, and are only used for the purpose of conveniently and clearly assisting in explaining the embodiments of the present invention.

[0074] Embodiment 1:

[0075] When a drone is performing a mission in a rural area, it needs to land and stop when encountering an abnormal situation. Figure 1 As shown in the figure, in rural scenes, there are mostly farmland and vegetable fields; however, the rural road network is relatively developed, and flat roads can be used to provide emergency landing sites. In order to improve the safety of drone landing, it is necessary to understand the width of rural roads to make decisions on landing points.

[0076] Since rural roads are often not standard roads, their width specifications vary; and the property rights may also belong to different entities, width data is difficult to collect a priori. To this end, the present invention provides a drone rural road inspection device that does not rely on a priori width measurement and can estimate the road width by collecting images of rural roads in real time under a non-fixed heading during flight.

[0077] like Figure 2 As shown, the present invention provides a drone rural road inspection device 900, which includes a host unit 100 and a user interface unit 200; wherein the host unit 100 includes a processor 110, a storage module 120 and an input and output module 130, and the user interface unit 200 is provided with a display screen 210 and an operation panel 220. The host unit 110 is configured to: acquire a rural road image, determine a ROI area containing the road in the rural road image, extract the road area from the ROI area of ​​the road as a foreground object, use the non-foreground object area in the rural road image as a background object, and set all background objects in the rural road image to zero to form a road target image; obtain a road edge image by edge detection on the road target image; perform Hough line transformation on the road edge image, and obtain a road sideline image containing two straight edges of the road by heuristic search; obtain geometric features of the straight lines on both sides of the road relative to the image center from the road sideline image, input the geometric features into a trained road width prediction model based on a neural network, obtain a road width prediction value, and output the width prediction value by the output unit 132 in the input-output module 130.

[0078] See also Figure 1As shown in the figure, in the image captured in the rural scene, the road appears to be wide at one end and narrow at the other end, and in the image, the two parallel sides of the real road will approximately intersect at a vanishing point at infinity. On the one hand, it is obvious that when measuring the distance from the pixels on both sides of the road in the image and then converting it back to the actual width according to the proportional relationship, the proportional relationship is difficult to accurately determine, and it is directly related to the sampling position of the side pixel points. On the other hand, referring to Fig. 10A , Fig. 10B As shown, in practical applications, it is not mandatory to require that the drone must face the same direction as the road, that is, shoot directly at the road to collect images in order to perform proportional measurement, otherwise the applicability of the inspection will be greatly limited.

[0079] Although the width of the rural road in the field of view is certain, when it is sampled from different headings, its pixel width and direction in the image space are uncertain. This certain width and uncertain image features show a contradictory relationship that is difficult to unify. Therefore, this contradiction needs to be decoupled and modeled. The heading of the drone is not fixed during flight, and the road orientation varies greatly; therefore, the relative relationship between the heading of the drone and the road orientation is unpredictable. In order to improve the versatility and applicability of road width detection, it is necessary to study how to characterize the relationship between the two in various relative orientations in the image space, that is, what kind of features in the image can determine the road width?

[0080] After observing the roads in various directions in the image, it is found that the key lies in the two road edges; however, the road edge itself is not the only way to determine the road width. The angle of the straight line itself can only reflect the relative angle difference between it and the heading. After a lot of experiments and tests, it is also found that the road width will affect the imaging effect of the two parallel sides in the image. How to characterize the characteristics of the two sides? Through further research, hypothesis and reasoning, it is proposed to use the slope k of the two parallel lines at different spacings 1 , k 2 And the perpendicular distance D from the center point of the image to the straight lines on both sides of the straight line 1 , D 2 There are 4 input quantities in total, which serve as the constraints on the width of different roads in the image space. Furthermore, how to correspond this constraint to the road width? Since there is a nonlinear and coupled mapping relationship between the two, inspired by the fitting application of neural networks in system nonlinear modeling, the present invention proposes to establish a road width prediction model using neural networks.

[0081] Neural network is a machine learning model that imitates biological neural systems and is particularly good at nonlinear mapping modeling. It learns and represents complex functional relationships through a multi-layer artificial neuron structure. Neural network consists of input layer, hidden layer and output layer; each layer contains several neurons, and each neuron is connected to the neurons in the previous layer through weights. Among them, the input layer receives input data, the hidden layer performs nonlinear transformation and feature extraction, which can have multiple layers, and the output layer outputs the final prediction result.

[0082] like Figure 4A As shown, the road width prediction model of the present invention adopts a three-layer BP neural network with one hidden layer, and its input layer receives the slopes k of the straight lines on both sides of the road from the general processing module respectively. 1 , k 2 And the perpendicular distance D from the center point of the image to the straight lines on both sides of the road 1 , D 2 There are 4 input quantities in total, and the output quantity of the output layer is the road width.

[0083] To extract the four input quantities or geometric features of the rural road in the image space, it is necessary to first obtain the straight lines on both sides of the road from the image. Figure 3A As shown, in the processor 110 of the host unit or controller 100, multiple processing modules are set, namely, a general processing module 111, an acquisition and preprocessing module 112, a feature extraction module 113, an iterative learning module 114 and a road width prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the road detection in the collected image, schedules the detection processing flow, inputs the rural road image collected in real time into the acquisition and preprocessing module 112, and outputs the width data identified and predicted by the road width prediction module 115.

[0084] The acquisition and preprocessing module 112 acquires the rural road image taken by the image acquisition unit 500, removes high-frequency noise through filtering, and then determines the ROI area including the road according to the deep learning feature or texture feature; and selects a seed point from the middle of the ROI area, and obtains the road area as the foreground object through flood filling processing; in the original rural road image, the background parts corresponding to the fields, trees, etc. outside the foreground object area are removed to form a road target image; the road target image is subjected to edge detection to form a binary road edge image. After the binary road edge image is subjected to Hough line transformation, the two straight line segments with the longest length are selected as the straight lines on both sides of the road; for any of the straight lines on both sides of the road, the feature extraction module 113 describes it with the coordinates of the two end points of the straight line, and respectively obtains the slope of the straight line and the perpendicular distance from the center of the image to the straight line based on the two end points; the slope of the straight lines on both sides of the road and the perpendicular distance from the center of the image to the straight line are used as the geometric features of the road in the rural road image.

[0085] Specifically, Figure 3B As shown, in the acquisition and preprocessing module 112, it is further divided into an image acquisition unit 1121, a target detection unit 1122, and a road area extraction unit 1123.

[0086] The image acquisition unit 1121 acquires the rural road image acquired in real time by the external image acquisition unit 500 through the general processing module 111, or acquires the pre-stored road image from the storage module 120 in an offline state. The road area extraction unit 1123 preliminarily determines the ROI area including the road in the image. Preferably, a target classification model based on the YOLOv5s network can be established in the target detection unit 1122. After the rural scene image is acquired by the bird's-eye view camera in the image acquisition unit, the network is trained offline with the image sample set composed of the rural scene images marked with various target areas such as rural roads, so as to obtain the rural scene target classification model; when running online, refer to Fig.13A , Fig. 13B As shown, the road area extraction unit 1123 uses the rural scene target classification model to identify and process the rural scene image including the road, i.e., the rural road image, to obtain a rural road area anchor frame, and uses the area within the anchor frame as the ROI area of ​​the rural road.

[0087] In order to find the actual road part from the ROI rectangular block area, the flood filling method is used for extraction. The flood filling method is to find the same area based on the difference between the grayscale values ​​of pixels to achieve segmentation. The grayscale value of the image is understood as the height of the pixel point, so that an image can be seen as a rugged ground or mountain. A certain amount of water is poured into a low place on the ground, and the water will cover the area below a certain height. The flood filling method uses this principle and is an image processing method based on seed points. It starts from the seed point and spreads to the adjacent pixels, and marks the pixels that meet the conditions as the specified color or value.

[0088] By observing the rural road image, it is found that the pixel values ​​of the area where the road is located are very close. Therefore, the present invention sets up seed points in the area representing the road, and selects the road area by the flood filling method. Specifically, after the target classification model based on the deep learning network locates the ROI area of ​​the road with an anchor frame, a seed point is set at the intersection of the two diagonals corresponding to the rectangle of the ROI area, and the road area is obtained by flood filling. At the same time, the background outside the road area is inverted in the original image, and the background such as fields and trees is eliminated, and the road image we need is highlighted to form a road target image, which is convenient for the subsequent extraction of the straight lines on the edge of the road. Fig.11A , Fig. 11B Corresponding to Fig. 10A , Fig. 10B The original rural road image is the image after the background is removed after the flood filled road image is inverted. For the convenience of display, Fig.11A , Fig. 11B Inverted color processing.

[0089] like Figure 3C As shown, in the feature extraction module 113, an edge detection unit 1131, a straight line extraction and screening unit 1132, a slope feature extraction unit 1133, and a distance feature extraction unit 1134 are also provided.

[0090] After the image content is concentrated on the road area, the edge detection unit 1131 performs edge extraction on the road target image. The edge information in the image is often contained in the area where the grayscale values ​​of adjacent pixels in the image change dramatically. In digital image processing technology, there are many edge detection methods, such as using edge detection operators such as Sobel, Canny, and LOG. Edge extraction can be performed directly on the binary road target image. For the color or grayscale road target image, the color image is first converted into a single-channel image, and then the edge detection operator is used to detect the edge.

[0091] The Canny operator is a second-order derivative edge detection operator, which uses Gaussian smoothing filtering, calculation of gradient amplitude and direction, non-maximum suppression, and hysteresis threshold processing in a series of operation combinations. Compared with the first-order derivative edge detection operator, the Canny operator can find an optimal edge, filter out most of the noise edges, and greatly improve the accuracy of edge detection and edge positioning. The present invention uses the Canny edge detection operator to extract the edge of the road target image. Fig. 12A , Fig. 12B Corresponding to Fig.11A , Fig. 11B The result of edge extraction of the road target image shown is a binary road edge image, which is also subjected to color inversion processing for the convenience of display.

[0092] The straight line extraction and screening unit 1132 extracts two straight line boundaries of the road from the binary road edge image. In the binary road edge image, the most noteworthy are the straight lines representing the boundaries on both sides of the road. First, we need to analyze how to extract these two straight lines, i.e., the edge lines. The present invention uses the Hough line transform. The Hough line transform maps the points in the image space to the curves in the parameter space, and then finds the intersection of the curves in the parameter space to detect the straight line. In the image space, the equation of the straight line can be expressed as: y=kx+b: Using polar coordinate notation, the equation of the straight line can be expressed as: ρ=xcosθ+ysinθ, where ρ is the vertical distance from the straight line to the origin of the coordinate system, and θ is the angle between the perpendicular line of the straight line and the positive direction of the x-axis.

[0093] The essence of Hough line transform is to establish a mapping relationship between image space and parameter space. On the basis of making full use of the global characteristics of the image, the point-line duality characteristic is used to map the pixel coordinates to the straight lines represented by the elements in the parameter space, and then the analysis is performed in the parameter space. The detection task is completed by cumulative statistics in the parameter space.

[0094] In order to detect straight lines, a plane matrix, namely an accumulator array (ρ, θ), is created in the ρ-θ plane to divide the parameter space into discrete grids. In general, for each edge point (x, y) in the image space, the ρ value of ρ = xcosθ + ysinθ under different θ is calculated, and the corresponding position in the accumulator array is voted. After voting, peak detection is performed, and the element with a larger value in the accumulator array is the detected straight line parameter (ρ, θ). In the heuristic search for obtaining the road edge line, the two straight line segments with the longest length are selected as the two straight edges of the road, that is, the straight lines on both sides of the road.

[0095] In the Hough line transform, the ρ distance resolution is usually 1 pixel, while the θ angle resolution is usually π / 180. Since it is found that the boundary straight line features in rural roads have relatively obvious characteristics, the Hough line transform processing process can be optimized. As a preference, when traversing and accumulating the element values ​​in the ρ-θ plane matrix, the spacing between elements in the value interval can take a larger value according to a priori testing, that is, the spacing between distances and angles is increased accordingly, and the number of rows and columns of the matrix is ​​reduced, thereby greatly reducing the amount of calculation in the calculation judgment and shortening the calculation time.

[0096] Preferably, before peak detection, the elements in the accumulator array (ρ, θ) are merged. When the values ​​of several adjacent elements in the array are above a preset pixel point threshold, i.e., the minimum length of a straight line, the centroid of the polygon enclosed by the coordinates of the several adjacent elements is obtained, and the (ρ, θ) corresponding to the obtained centroid is used as the road edge line.

[0097] Before performing the Hough line transform, a dilation operation can also be used to perform a continuous processing of the binary road edge image A. The specific execution order of the continuous processing is: set an N×N structure element matrix B to generate an all-zero matrix C of the same size as the image A; use the structure element B to traverse the image A to be processed, that is, gradually move the center point of the structure element B along each non-zero grid in the image A, extract the intersection of the non-zero elements of the structure element B and the image A, and if the intersection is not empty, then set the elements in the matrix C, that is, the pixels, to 1 or the pixel value 255 at the same position in the grid of all non-zero elements of the structure element B in A; update the matrix C during the traversal process, and output the matrix C after the traversal is completed, so as to form the final road edge image with the output matrix C. In the continuous processing, as a preferred embodiment, a right-angle L-shaped, T-shaped or cross-shaped structure element can be used. Through the continuous processing, some fine edges caused by vegetation or roadside damage are connected into lines, thereby improving the detection effect of the road edge.

[0098] For the two road edge lines searched, the slope feature extraction unit 1133 and the distance feature extraction unit 1134 respectively calculate the slope of the straight line and the vertical distance from the center point of the image to the straight line. Figure 7 The diagram of the road model width constraint condition shows two sides of a road on the left and right sides of the image plane. The angle and slope of the straight line on the two sides of the left road are α(k1) and β(k2), and the vertical distance from the center of the image to the two sides of the road are D1 and D2 respectively; correspondingly, the angle and slope of the straight line on the two sides of the right road are α′(k1′) and β′(k2′), and the vertical distance from the center of the image to the two sides of the road are D1′ and D2′ respectively. In the oblique photography image plane, the two parallel lines will intersect at a point O at the far end, and point O is called the vanishing point.

[0099] The algebraic equation of a line can be expressed in three common forms, namely the general form, the two-point form, and the slope-intercept form. The general form is: Ax+By+C=0, where A and B are not zero at the same time, and the slope of the line k=-A / B. In the Cartesian rectangular coordinate system, the intercept of the line on the y-axis b=-C / B, and the general form applies to all line equations.

[0100] The two-point formula is: The two-point equation is the equation of a straight line passing through the two points when the coordinates of the two points are known. The slope of the straight line is k = (y 2 -y 1 ) / (x 2 -x 1 ), it can be seen from the two-point equation that this equation is suitable for describing straight lines that are not perpendicular to the x-axis and y-axis.

[0101] The slope-intercept form is: y=kx+b. The slope-intercept form explicitly expresses the slope k and intercept b of the straight line, and is suitable for representing straight lines that are not perpendicular to the x-axis.

[0102] The straight line detected by Hough line transform in the image is described by the coordinates of the two end points of the straight line, and the two end point coordinates are marked as (x 1 ,y 1 )、(x 2 ,y 2 ), substitute the coordinates of the two points into the two-point equation, and then transform the two-point equation into the slope-intercept equation form to obtain: Its corresponding slope k = y 1 -y 2 / x 1 -x 2 The only difference is that the calculation of the slope of the line is completed in the image coordinate system (the y-axis is downward in the positive direction), so the slope of the line in the image is equal to that in the Cartesian rectangular coordinate system, but has the opposite sign.

[0103] Combined with (x 1 ,y 1 )、(x 2 ,y 2 ) equation, the perpendicular distance from the center of the image to the edge of the road can be calculated as: In the formula, (x 0 ,y 0 ) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is the constant term and: A = y 2 -y 1 , B = x 1 -x 2 , C = -x 1 (y 2 -y1 )+y 1 (x 2 -x 1 ). According to the calculation formula of k and d, the slope k of the straight lines on both sides of the road is calculated respectively. 1 , k 2 And the perpendicular distance D from the center point of the image to the two straight lines 1 , D 2 A total of 4 quantities serve as the geometric features of the road in the rural road image.

[0104] like Fig. 8A , Figure 8B A schematic diagram of road edge straight line screening and geometric feature extraction is given. Among them, the intersection of diagonal lines indicates the center point of the image. In the set of detected straight lines, there are two straight line segments whose lengths far exceed other straight lines, and thus they are screened as two edge lines of the road. It can also be seen that the angles of the two road edge lines are close. Therefore, in the heuristic search, the straight line can be accepted or excluded as a road boundary line based on whether the length of the straight line segment is a preset ratio of the maximum straight line segment length, such as 0.85 to 0.95 times; and a heuristic rule for the direction of a straight line can also be set, that is, if there are two straight line segments with similar lengths and angles, it is determined that the two straight lines may be road edge lines. In 8A, Figure 8B After excluding the lines whose length is less than the preset ratio of the longest straight line length, for the two selected edge lines, a perpendicular line is drawn from the center point of the image, represented by a dotted line. Figure 8B In the figure, since the foot of the perpendicular is outside the range of the side lines, extend the side lines to the foot of the perpendicular.

[0105] Preferably, a human-computer interaction interface is provided in the user interface unit, and the requirement for the minimum length of a straight line can be reduced for a road section with a larger turning radius in the image, so that the tangent straight lines of a road section with a certain curvature can be marked with straight lines after the Hough transform; and the straight lines on both sides of the road are marked through human-computer interaction, and the geometric features of the marked road edge lines are obtained to predict the road width.

[0106] In order to reveal the nonlinear mapping structure of the road width prediction model of the present invention, a rural road scale model is established and then the model is tilted to obtain training image samples. Figure 5 As shown, when the UAV is shooting the ground in the air, the angle between the optical axis of the onboard lens and the horizontal plane is α, which can be selected from commonly used working angles such as 45 degrees.

[0107] Since the width of actual rural roads varies greatly, it is difficult to obtain standardized template road sections with uniform spacing. To facilitate modeling, the present invention uses rectangular sheets to make road models of various specifications, such as Fig. 6A shown.

[0108] In an offline state, a drone is used to collect images of the prepared road model and form a sample set, and the road width prediction model is trained with the sample set. The road model uses a rectangular sheet, the width of which is a first preset multiple S1 of the actual road width, and the length of which is a second preset multiple S2 of its width, so as to form a model that conforms to a section of the actual road. The multiples S1 and S2 are set according to the rural cruising flight altitude of the drone and the camera resolution; the different rectangular sheets used in the road model have a width range corresponding to S1 times the common width range of rural roads, and the width difference between different rectangular sheets, i.e., the spacing, corresponds to S1 times the actual road width difference SD. For example, when the flight altitude is 100 meters, the sampling height of the corresponding model is 2 meters, so as to ensure that the road model can occupy a significant image space during imaging, so as to coordinate the image resolution and the recognition accuracy of the road width prediction model, and make it unnecessary for the drone to specially lower its altitude to predict the road width, thereby improving the applicable scope of the prediction.

[0109] Preferably, the first preset multiple S1 is between 1 / 50 and 1 / 30, and the minimum value of the second preset multiple S2 is between 3 and 5. Preferably, the actual road width difference SD is between 0.2 and 0.5 meters, and the width of each rectangular slice used as the road model can form an arithmetic progression with S1 times SD as the tolerance.

[0110] A road model was built under laboratory conditions, and the actual aerial photography height h real and the actual road width w real Scale proportionally to obtain the aerial photography height h under laboratory conditions model and road width w model In order to find the relationship between the road features and the road width in the image, this embodiment selects the actual aerial photography height h of the drone real When they are 30m, 40m, and 50m respectively, the corresponding aerial photography height h under laboratory conditions model The corresponding sizes are 667mm, 889mm, and 1111mm, and the scaling ratios are:

[0111]

[0112] According to the "Guiding Opinions on Rural Highway Construction Standards", the road width of rural roads is generally not less than 4.5m. However, for sections that are difficult to implement due to economic, topographical, geological and other natural conditions, it can be adapted to local conditions and, after approval, it is not necessary to strictly abide by the "Highway Engineering Technical Standards". Sometimes it may be increased or decreased according to the terrain. When building a road model in the laboratory, most of the plain straight roads are considered, that is, the natural slope of the road is within 3° and the road curvature is close to 0. model With hreal The scale ratio is 1:45. Because it is proportional scaling, the width of the road model is w model The actual width of the road w real The ratio is also 1:45, and then road models of different widths are made with a step size of 10.5 mm. The step size of the model corresponding to the actual road width is 0.5 m.

[0113] Through field research, it was found that the width of many rural roads does not fully meet the condition that the road width is not less than 4.5m. Therefore, the road width below 4.5m is also taken into account and a road model that meets the corresponding road width is produced. The produced road model is as follows: Fig. 6A As shown. Among them, the road model width w model The 105mm and 210mm road models are marked.

[0114] Since the image sampling of the road model is an oblique image, in order to ensure the diversity of the features representing the road model in the image as much as possible and improve the generalization ability of the network model for predicting road width, the road model is photographed from 0°-180°. The reason for non-360° all-round shooting is that the effect of shooting the road model at 180°-360° is the same as that at 0°-180°. At the same time, in order to highlight the distance relationship between the drone and the actual position of the road, the road model is arranged in different areas of the ground field of view, so that the road models in the sample set are distributed in each square of the nine-square grid formed by the top, middle, bottom, left, middle and right in the image plane. With w model = 136.5 mm as an example, some of the captured data sets are as follows Figure 6B As shown, in order to improve the accuracy of the model, the photo was taken with a black ground as the background, and for the convenience of display, the picture was inverted.

[0115] The present invention subdivides the road model size to form a width sequence, and shoots from various relative azimuth angles during the sampling of road images, thereby ensuring the sufficiency of samples, thereby improving the generalization ability of the neural network and reducing the prediction error of the road width.

[0116] For the rural road image based on the road model, the collected image is filtered and grayed, then binarized to form a road target image, and then the road edge image is obtained by edge detection. Taking the road model image of a certain location taken at the laboratory sampling height h = 667mm as an example, the preprocessing results of each stage are as follows Figure 6C As shown in the figure, from left to right they correspond to the processing results after grayscale, binarization, and edge detection. The road edge image after Canny edge detection is subjected to Hough line transform, and the transformed image is as follows: Fig.6DAs shown in the figure, the left and right figures are the Hough line transform results of the road model with a width of 105mm and 136.5mm respectively. After that, a heuristic search is performed on the transformed straight lines to select the two straight line segments with the longest length as the two straight line edges of the road, i.e., the straight lines on both sides of the road. Fig.6D In the figure, there are two long straight lines after excluding the short line close to the horizontal direction.

[0117] To ensure the correct detection and identification of road edges, a manual confirmation step can be added to the edge search. Specifically, in the process of collecting straight line features representing roads and establishing a training sample library, the interference of non-road boundary straight lines is eliminated through the human-computer interface. With the help of the HighGUI module provided in openCV, a graphical user interface is established and the mouse callback function setMouseCallback() is used. In the callback function, a left mouse button click event and a right mouse button click event are created. Each time the left mouse button is clicked, a straight line detected by Hough line transform is drawn in the image, and it is confirmed after being compared with the original road image.

[0118] Refer to the above Figure 7 For each model road image, the slope k of the two edge lines is calculated. 1 , k 2 And the perpendicular distance D from the center of the image to the two sides 1 , D 2 There are 4 geometric features in total, and the corresponding actual model road width is recorded as the true value of the road width. The geometric features and the true value of the road width are formed into samples. After the samples are collected and recorded to form a sample set, they are input into the road width prediction model based on the neural network for offline training. Preferably, the sample set can be stored in the form of a data table composed of input and output true values ​​as a record.

[0119] The BP neural network used in the road width prediction model of the present invention is as follows: Figure 4A As shown, the model is a three-layer structure including a hidden layer, in which

[0120] The output of the jth node in the hidden layer is

[0121] The output of the output layer node is

[0122] In the formula, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.

[0123] When predicting the road width online, the width of the road model is predicted based on the four set feature quantities obtained from the road model image. The prediction model then maps it back to the actual road width as the actual output value based on the ratio of the model to the actual value.

[0124] Preferably, multiple neural network-based road width prediction models are created in the host unit, each road width prediction model corresponds to a flight altitude and a bird's-eye view angle of the image acquisition unit; when predicting the road width online, the geometric features obtained after the current acquisition process are input into the road width prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the road width prediction value. In this embodiment, the shooting heights h of the road models are selected to be 667mm, 889mm and 1111mm, respectively, and the shooting angle is 45°.

[0125] The sample data sets with h of 667mm, 889mm and 1111mm were randomly divided into training sets and test sets, and both training sets and test sets covered 17 different road models. The geometric features of the test set were input into the trained network model as input to obtain the predicted value. The fitting comparison curves of the predicted value and the true value of the road model width at different heights are shown in the figure. Fig.9A , Fig. 9B As shown, their h are 889mm and 1111mm respectively. The test set performance is shown in Table 1.

[0126] Table 1 Road width prediction model test performance

[0127]

[0128] Analyze the regression results at three different laboratory heights and the coefficient of determination R at three different heights 2 The values ​​of the root mean square error (RMSE) at three different heights are 2.2156, 2.0322, and 1.8352, respectively, indicating that the predicted value is close to the true value and the network model has good generalization ability.

[0129] To further verify the feasibility of the model and algorithm, two groups of control experiments were designed by taking actual flight photos of rural roads. In the first group, the same road width was predicted when the actual flight altitudes were 30m, 40m, and 50m, and the prediction error between the predicted value and the true value was calculated to verify the reliability of the model at several common cruising altitudes. In the second group, taking the flight altitude of 30m as an example, the prediction error between the predicted value and the true value of roads of different widths was calculated to verify the generalization ability of the model for predicting different road widths.

[0130] The pictures collected when the actual flight altitude is 30m and 50m and the measured road width is 4.25m are as follows Fig. 10A , Fig. 10B As shown in the figure; the two images after the road image is filled with water and the background is removed by reverse selection are respectively Fig.11A , Fig. 11B As shown, the road edge line is obtained through Hough line transformation and screening as shown in Fig. 12A , Fig. 12B shown.

[0131] After detecting the road straight line, the geometric features are calculated and substituted into the constructed road width prediction models at three different heights. The prediction results are shown in Table 2, where w mp Represents the predicted value of the road model width, w rp Indicates the road prediction value according to the scale ratio (1:45) between the model and the actual road, w real is the actual value of the road.

[0132] Table 2 Prediction results of roads of the same width at three cruising altitudes

[0133]

[0134] It can be seen from Table 2 that when the actual road width is 4.25 m, the relative error between the predicted value and the actual value of the road width is within 2%, indicating that the model prediction accuracy is high.

[0135] In the second set of experiments, taking the 30m height as an example, four roads with different widths were selected. Fig.14A , Fig. 14B Two scenarios with road widths of 5.22m and 6m are given. Fig.15A , Fig. 15B The corresponding road edge straight line detection result diagram is shown in Table 3. The prediction results of the road width prediction model at a height of 30m for roads of different widths are shown in Table 3.

[0136] Table 3 Prediction results of different road widths at h = 30m

[0137]

[0138] It can be seen from Table 3 that under the condition of h = 30m, the prediction model has relatively accurate prediction results for different road widths, with a relative error within 4%. In particular, the road width in the offline model does not correspond to the actual road width, indicating that the model has sufficient generalization performance.

[0139] Example 2

[0140] Based on the previous embodiment, Figure 2As shown, this embodiment provides a UAV rural road inspection system 1000, which includes: an image acquisition unit 500 for image sensing and acquisition of rural scenes, a user interface unit 200 for parameter input and operation and rural road information display, a server 400 for data management and information transmission, a handheld terminal 300 for remote operation and rural road information interaction, and a host unit 100 connected to the user interface unit 200, the server 400, and the image acquisition unit 500.

[0141] The host unit 100 includes an input / output module 130, a processor 110 and a storage module 120, and is configured to: acquire a rural road image, determine a ROI region containing a road in the rural road image, extract a road region from the ROI region of the road as a foreground object, use a non-foreground object region in the rural road image as a background object, and set all background objects in the rural road image to zero to form a road target image; obtain a road edge image by edge detection on the road target image; perform Hough line transformation on the road edge image, and obtain a road sideline image containing two straight edges of the road by heuristic search; obtain geometric features of the straight lines on both sides of the road relative to the image center from the road sideline image, input the geometric features into a trained road width prediction model based on a neural network, obtain a road width prediction value, and output the width prediction value.

[0142] Preferably, the drone rural road inspection system 1000 may not include the server 400 and the handheld terminal 300, but connect the drone rural road inspection system with the server 400 and the handheld terminal 300 through a communication network when used. A positioning unit 700 for obtaining coordinates may also be provided in the drone rural road inspection system 1000.

[0143] Preferably, the UAV rural road inspection system 1000 also includes a flight control platform 600. In response to the user's operation on the operation panel 220 or the instruction of the host unit 100, the flight control platform 600 executes the instruction and flies toward the specified road point.

[0144] The UAV rural road detection system 1000 continuously detects multiple frames of rural road images and predicts the road width. When the road width meets the requirements, it compares the widths of multiple road sections, determines the section with the largest width as the target landing point, and outputs the coordinate position information of the target landing point to the flight control platform 600 through the output unit. The flight control platform controls the power unit to make the UAV fly to the target landing point.

[0145] Preferably, when drones are used in group systems such as logistics and transportation, the system may have multiple handheld terminals and drone platforms accordingly. Through the server, the host unit and the remote client deployed at the handheld terminal can send messages about the road width. Preferably, an MQTT server can be deployed, and the server pushes messages to the clients that subscribe to the topic. Multiple devices can subscribe to multiple topics at the same time, and when publishing messages, those who subscribe to the same topic can receive the same message. The publish-subscribe mode is a messaging mode that decouples the client that sends the message, i.e., the publisher, from the client that receives the message, i.e., the subscriber, so that the two do not need to establish a direct connection or know each other's existence.

[0146] The server provides subscription and publishing services, and the host unit is also configured to: subscribe to the information acquisition topic of the handheld terminal, and transmit the detected road width and corresponding coordinates and location information to the handheld terminal through the server in response to the topic message; or when it is detected that the road width reaches a set width threshold, transmit the road width and corresponding coordinates and location information to the server in the form of a width event topic message.

[0147] Preferably, the user interface unit is provided with an operation panel for parameter input, and is also provided with a display screen for assisting parameter input and displaying the road width recognition result in the form of a list; the host unit also displays the recognized road width in segments in the form of a curve on the display screen, and marks the coordinates and positions corresponding to the recognized width on the curve.

[0148] Preferably, the positioning unit in the system also obtains heading information, and also records and identifies in the output position information the direction identified based on the heading, such as the north direction, and the direction corresponding to the center line of the road between the two side lines of the road.

[0149] Preferably, the host unit can also communicate directly with the handheld terminal.

[0150] The host unit can be a local control unit deployed separately on the ground, or a control unit deployed on the UAV platform. When deployed on the UAV platform, the host unit on the UAV platform performs edge computing for online road width prediction. Due to the limited resources of the edge computing unit, the processing calculation of the host unit is optimized.

[0151] In the general Hough line transform, for each edge point (x, y) in the image space, the ρ value of ρ = xcosθ + ysinθ under different θ is calculated, and the corresponding position in the accumulator array is voted. This process of calculating ρ requires two floating-point multiplication operations.

[0152] To this end, based on the obvious boundary straight line features in rural roads, the pixel check accumulation process of Hough line transform is optimized. Specifically, when the Hough transform is performed on each foreground pixel point (u, v) in the road edge image, in the cumulative statistics, it is determined whether the pixel point is adjacent to a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is:

[0153] Take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where i and j are the row and column labels in the ρ-θ plane matrix, and ε is a preset small positive number.

[0154] By setting the fulcrum, a tangent operation is performed once before traversal judgment, so that each time the foreground pixel is judged, only two subtraction operations and one multiplication operation are required to perform judgment accumulation, thereby avoiding two time-consuming floating-point operations for each pixel, further saving computing costs, and thus suitable for edge computing deployed on drone platforms.

[0155] Preferably, in the heuristic search for obtaining the road edge line, when the angle is expressed as a slope in a rectangular coordinate system, when the absolute value of the difference in the edge line slopes is less than a very small first slope threshold, the two edge lines are merged; at the same time, if the angle tangent value or the slope is greater than a very large second slope threshold, the edge line is determined to be a vertical line perpendicular to the horizontal axis, thereby preventing the slope from being judged as different straight lines due to slight differences when it is very large, and avoiding the false detection interference of short edge lines on the road edge straight lines.

[0156] In the embodiment 1, in the BP neural network of the road width prediction model, the input layer receives the slopes k of the straight lines on both sides of the road from the general processing module respectively. 1 and k 2 , the perpendicular distance D from the center point of the image to the straight lines on both sides of the road 1 and D 2, and the flight altitude and overlooking angle during acquisition, a total of 6 input quantities, and the output of the output layer is the road width. After establishing a neural network with this structure, the input quantity of each sample in the training set is supplemented, and after supplementing enough samples at different heights and overlooking angles, it is trained with a new training sample set. When applied online, the new model can further simplify the image sampling requirements for rural road prediction; it uses the generalization performance of neural networks to reduce the number of models required for rural road width prediction, and also simplifies the system structure, so that one model can cover different flight sampling scenarios, further improving the applicability of the system.

[0157] Example 3

[0158] Based on the above embodiments, this embodiment provides a drone rural road detection controller. Figure 2 , Figure 3A , Figure 4B As shown, the UAV rural road detection controller 100 includes a processor 110, a storage module 120 and an input and output module 130. The processor 110 includes a general processing module 111, an acquisition and preprocessing module 112, a feature extraction module 113, an iterative learning module 114 and a road width prediction module 115. Among them, the general processing module 111 responds to the message and the operation of the user interface unit, and starts the road detection in the collected image, schedules the detection processing flow, inputs the rural road image collected in real time into the acquisition and preprocessing module 112, and outputs the width data identified and predicted by the road width prediction module 115.

[0159] Combination Figure 3A , Figure 4B As shown, the rural road image is input into the controller by the input unit 131 in the input-output module 130; the road width prediction model is established in the road width prediction module 115, and its input layer receives the geometric features of the road in the rural road image extracted by the feature extraction module 113 from the general processing module 111, and the output layer is transmitted to the iterative learning module 114 and the general processing module 111 through the first connection matrix 1141 and the second connection matrix 1142 respectively.

[0160] When the road width prediction model is trained offline, the iterative learning module 114 adjusts the connection weights of the network according to the actual road width value input by the general processing module 111 and the road width prediction model through the first connection matrix 1141 and the output value of the neural network in the model; in the field environment, the first connection matrix 1141 is disconnected, and the road width prediction model neural network predicts the width of the road in the current rural road image and outputs it to the general processing module 111 through the second connection matrix 1142, and is processed and analyzed by the general processing module 111 and output through the output unit 132 in the input and output module 130.

[0161] The steps of iterative training include: STEP 1, initialize the weights and biases of the network; STEP 2, read the network parameters and the training sample set written in the document; STEP 3, normalize the sample data; STEP 4, calculate the error between the predicted output value and the expected value for each sample; STEP 5, reversely calculate and correct the network weights and biases; STEP 6: if the conditions for the end of network training are met, end the training, otherwise go to STEP 4 to continue training; STEP 7, analyze and output the results. Through multiple experiments, the number of nodes in the hidden layer is determined to be 13, the learning rate is 0.01, and the number of iterations is 2000.

[0162] The general processing module 111 in the controller also obtains the coordinate information and orientation information of the current road location through the positioning unit, and after detecting the width of the road, outputs the road width and the corresponding coordinate and orientation information through the output unit 132.

[0163] like Fig.13A , Fig. 13B As shown, after the road width is identified, the road width is also marked in the road area segment corresponding to the extracted road in the rural road image, such as by using an arrow to mark the width direction line segment. Preferably, the acquired coordinates and position information, as well as the road width, are also stored in a list of the storage module 120, and the list information can be stored in the cloud server 400.

[0164] When the width is greater than the preset width threshold, the coordinate azimuth value of a point on the center line of the road is given; otherwise, an alarm is issued to indicate that the road width is insufficient. During the flight of the drone, image acquisition is performed iteratively and continuously, and the drone rural road detection controller 100 performs width prediction processing based on the acquired images. When a width greater than the preset width threshold appears within the field of view, a prompt is issued that a road segment with sufficient width has appeared. Preferably, according to the flight speed and frame rate, the overlapping range of the rural road images processed successively is not less than the set ratio.

[0165] The present invention is used to detect the width of rural roads in drone sampling image space. After road area segmentation, road edge extraction and sideline determination, the slope k of the straight lines on both sides of the road is used to determine the width of rural roads. 1 , k 2 And the perpendicular distance D from the center point of the image to the straight lines on both sides of the road 1 , D 2 A total of four geometric features were input into the established road width prediction model based on neural network. The model can accurately predict the actual width of rural roads, thus providing a decision-making basis for the docking of drones and improving the safety and convenience of drone operations.

[0166] Several embodiments of the present invention are described above, but these embodiments are presented as examples and do not limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the main purpose of the invention. These embodiments or their modifications are included in the scope or main purpose of the invention, and are also included in the invention described in the claims and their equivalents.

Claims

1. A drone rural road inspection device, comprising a host unit and a user interface unit, wherein the host unit comprises an input and output module, a processor and a storage module, and the user interface unit is provided with an operation panel; The host unit is configured as: Acquire a rural road image, determine a ROI region containing a road in the rural road image, extract a road region from the ROI region of the road as a foreground object, use a non-foreground object region in the rural road image as a background object, and set all background objects in the rural road image to zero to form a road target image; For the road target image, edge detection is performed to obtain a road edge image; after Hough line transformation is performed on the road edge image, a road edge line image containing two straight line edges of the road is obtained through heuristic search; The geometric features of the straight lines on both sides of the road relative to the center of the image are obtained from the road sideline image, the geometric features are input into a trained road width prediction model based on a neural network, a road width prediction value is obtained, and the width prediction value is output.

2. The drone rural road inspection device according to claim 1 is characterized in that: The road width prediction model adopts a BP neural network, whose input layer receives four input quantities, namely the slopes k1 and k2 of the straight lines on both sides of the road and the vertical distances D1 and D2 from the center point of the image to the straight lines on both sides of the road, from the general processing module, and the output quantity of the output layer is the road width.

3. The drone rural road detection device according to claim 1 is characterized in that: The host unit is further configured to: before online application, firstly use a drone to collect images of the prepared road model offline to form a sample set, and use the sample set to train the road width prediction model. The road model adopts a rectangular thin sheet, the width of the rectangular thin sheet is a first preset multiple S1 of the actual road width, and the length of the rectangular thin sheet is a second preset multiple S2 of its width, and the multiples S1 and S2 are set according to the rural cruising flight altitude and camera resolution of the UAV; the different rectangular thin sheets used in the road model have a width range corresponding to S1 times the common width range of rural roads, and the width difference between different rectangular thin sheets corresponds to S1 times the actual road width difference SD.

4. The drone rural road detection device according to claim 1 is characterized in that: The processor includes a general processing module, an acquisition and preprocessing module, a feature extraction module, an iterative learning module and a road width prediction module. The acquisition and preprocessing module acquires the rural road image from the image acquisition unit, selects a seed point from the middle of the ROI area according to the deep learning feature after filtering, and obtains the road area as the foreground object through flood filling processing; removes the part outside the foreground object area from the original rural road image to form a road target image, and forms a binary road edge image through edge detection; After the binary road edge image is subjected to Hough line transformation, the two straight line segments with the longest length are selected as the straight lines on both sides of the road. For any of the straight lines on both sides of the road, the feature extraction module describes it using the coordinates of the two end points of the straight line, and calculates the slope of the straight line and the perpendicular distance from the center point of the image to the straight line based on the two end points; the slope of the straight lines on both sides of the road and the perpendicular distance from the center point of the image to the straight line are used as the geometric features of the road in the rural road image.

5. The drone rural road inspection device according to claim 1, characterized in that: The road width prediction model established in the road width prediction module has an input layer receiving the geometric features of the road in the rural road image from the general processing module, and an output layer transmitting the features to the iterative learning module and the general processing module through the first connection matrix and the second connection matrix respectively; when training the road width prediction model offline, the iterative learning module adjusts the connection weights of the network according to the actual road width values ​​respectively input by the general processing module and the road width prediction model through the first connection matrix and the output values ​​of the neural network in the model; in the field environment, the first connection matrix is ​​disconnected, and the road width prediction model neural network predicts the width of the road in the current rural road image and outputs it to the general processing module through the second connection matrix, and the general processing module processes and analyzes it and then outputs it through the output part in the input and output module.

6. The drone rural road detection device according to claim 1, characterized in that: The feature extraction module calculates and extracts geometric features of the straight lines on both sides of the road. Assuming that the coordinates of the two endpoints of the road edge line are (x1, y1) and (x2, y2), its slope is k = y1-y2 / x1-x2; The perpendicular distance from the center of the image to the edge of the road is Where (x0, y0) represents the coordinates of the center of the image, A and B are the coefficients of the general formula of the straight line equation, C is a constant term and: A = y2-y1, B = x1-x2, C = -x1(y2-y1)+y1(x2-x1); According to the calculation formulas of k and d, the slopes k1 and k2 of the straight lines on both sides of the road and the vertical distances D1 and D2 from the center point of the image to the two straight lines are calculated respectively, a total of four quantities as the geometric features of the road in the rural road image.

7. The drone rural road detection device according to claim 1, characterized in that: The BP neural network used in the road width prediction model has a three-layer structure including a hidden layer, wherein: The output of the jth node in the hidden layer is The output of the output layer node is Among them, the f() function is taken as the tansig function, w ij and v j are the connection weights from the input layer to the hidden layer and from the hidden layer to the output layer, respectively. j and b are the hidden layer and output layer thresholds respectively, m=4 is the number of input variables, k is the number of hidden layer nodes, and the gradient descent method is used for network training.

8. The drone rural road inspection device according to claim 1, characterized in that: A plurality of the neural network-based road width prediction models are created in the host unit, each road width prediction model corresponds to a flight altitude and a bird's-eye view angle of an image acquisition unit; when performing road width prediction online, the geometric features obtained after the current acquisition process are input into the road width prediction model corresponding to the flight altitude and the bird's-eye view angle during acquisition, and the model output value is obtained as the road width prediction value.

9. The drone rural road detection device according to claim 1, characterized in that: The host unit is further configured to: when performing the Hough transform on each foreground pixel point in the road edge image, accumulate and obtain the ρ-θ plane matrix corresponding to the image, where ρ represents the vertical segment distance of the straight line from the origin, and θ is the angle of the vertical line to the horizontal axis; in the accumulation, the foreground pixel point (u, v) in the image and a straight line element l i,j (ρ i,j ,θ i,j ) The corresponding judgment method is: Take a point H(ρ i,j ·cosθ i,j , ρ i,j sinθ i,j ), recorded as H(u′, v′), calculate J = |vv′-(uu′)·tanθ i,j |, if |J|≤ε, then the pixel point is determined to be on the straight line and accumulated, where ε is a preset small positive number.

10. UAV rural road inspection system, which includes: An image acquisition unit for performing image sensing and acquisition of rural scenes, a user interface unit for parameter input and operation and rural road information display, a server for data management and information transmission, a handheld terminal for remote operation and rural road information interaction, and a host unit connected to the user interface unit, the server, and the image acquisition unit, The host unit includes an input / output module, a processor and a storage module, and is configured as follows: Acquire a rural road image, determine a ROI region containing a road in the rural road image, extract a road region from the ROI region of the road as a foreground object, use a non-foreground object region in the rural road image as a background object, and set all background objects in the rural road image to zero to form a road target image; For the road target image, edge detection is performed to obtain a road edge image; after Hough line transformation is performed on the road edge image, a road edge line image containing two straight line edges of the road is obtained through heuristic search; The geometric features of the straight lines on both sides of the road relative to the center of the image are obtained from the road sideline image, the geometric features are input into a trained road width prediction model based on a neural network, a road width prediction value is obtained, and the width prediction value is output.

Citation Information

Cited By

  • Method and device for positioning reference point in measurement area

    CN119515984A

  • A method and device for locating reference points in a measurement area

    CN119515984B

  • Air detection drone

    TWI915115B