Disease acquisition and presentation method based on three-dimensional urban road

Abstract

The invention discloses a disease collection and presentation method based on a three-dimensional urban road. The method comprises the following steps: S1, building the three-dimensional urban road; s2, building a material library; road diseases are divided into expansion joints, linear cracks, blocky cracks, pits and fuzzy road marking lines according to the road disease types; step S3, a data acquisition step; s4, a road disease data analysis step; step S5, a data presentation step; reading road disease data in the database; and according to latitude and longitude data of the road diseases in the data, generating the road diseases of the corresponding contours at specified positions of the scene. Through the method provided by the invention, the urban road disease collected by the collection equipment is presented on the three-dimensional urban road, so that the appearance of the urban road is more truly restored, and road maintenance personnel can visually observe the type, the outline and the hazard degree of the road disease.

Description

Technical Field

[0001] The present invention belongs to the technical field of highway road disease detection, and specifically is a disease collection and presentation method based on three-dimensional urban roads. Background Art

[0002] Today, as smart cities are being vigorously promoted, many companies have begun to build three-dimensional city bases, including urban road networks, buildings, greening, etc. Many related businesses have been derived from the three-dimensional city base. Displaying various businesses in a three-dimensional way allows people to understand the business content simply and clearly, which is convenient for management.

[0003] With the popularization of transportation and the acceleration of urbanization, road construction and maintenance have become indispensable public services in modern cities. As a basic link in road maintenance work, road inspection has the main task of discovering and recording road diseases so as to carry out maintenance and repair in a timely manner. Ordinary road disease collection is done by manual field photography and recording, which is inefficient. At the same time, the traditional road inspection system displays road diseases by displaying pictures. Due to the limitations of pictures, it cannot directly reflect the location of road diseases and the impact of road diseases around them, nor can it directly reflect the overall urban road conditions. Summary of the invention

[0004] The purpose of the present invention is to provide a method for collecting and presenting defects based on three-dimensional urban roads, so as to solve the problem that in the prior art mentioned in the background technology, the road defects are collected by manual field shooting and recording, which is inefficient. At the same time, the traditional road inspection system displays road defects by displaying pictures, because of the limitation of pictures, it cannot intuitively reflect the location of road defects and the impact of road defects around them.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] The disease collection and presentation method based on three-dimensional urban roads includes the following steps:

[0007] Step S1, building a three-dimensional city road; using modeling software to build the road that needs to be managed in the three-dimensional city scene;

[0008] Step S2, building a material library step; classifying road damage into expansion joints, linear cracks, block cracks, potholes, and blurred road markings according to the road damage type, placing corresponding material images in a folder and naming them according to the type;

[0009] Step S3, data collection step: install a camera and a GPS positioning device on the collection vehicle, adjust the time of the camera and the GPS device, turn on the device, drive the collection vehicle to travel along the road where road damage is required, collect video data and positioning data, and record the data collection time;

[0010] Step S4, road damage data analysis step;

[0011] Step S5, data presentation step; reading the road damage data in the database in the three-dimensional city scene; displaying the corresponding points at the corresponding positions in the three-dimensional scene and connecting them to form the outline of the road damage according to the type of road damage data and the longitude and latitude point set of the road damage contour; selecting the corresponding material in the road damage material library for filling according to the road damage type.

[0012] According to the above technical solution, in step S1, building the roads that need to be managed in the modeling software is specifically as follows: building the roads that need to be managed in the modeling software according to the latitude, longitude and elevation information of the actual roads, and restoring various attributes of the roads in the modeling software according to the actual road information, including lane width, number of lanes and lane markings.

[0013] According to the above technical solution, step S4, the road damage data analysis step includes the following steps:

[0014] Step S401, extracting a frame image from the video data, with a resolution of 1920×1080, performing road disease recognition analysis on the image according to an existing road disease recognition model, extracting a pixel point set of the road disease contour in the image and identifying the road disease type, and recording the frame image and the time of the frame image as well as the pixel contour point set and type of the road disease;

[0015] Step S402: Analyze the frame image using the SDK and corresponding API method provided by the ZED camera based on the pixel coordinate point set of the identified contour points of the road damage on the image. The SDK uses the data from the depth module to calculate the 3D position of each pixel point of the bounding box of each road damage, and obtain the 3D coordinates of each pixel point.

[0016] Step S403, obtaining GPS data of the corresponding time according to the time on the frame image, and calculating the longitude and latitude point set corresponding to the 3D contour points of the road disease;

[0017] Step S404, storing the calculated data into a database, including road damage type, road damage longitude and latitude, road damage outline, road damage collection time, and road damage image data.

[0018] According to the above technical solution, in step S403, the latitude and longitude points of the road damage points are calculated as follows:

[0019] Step S4031, when the road damage is detected, the longitude and latitude of the GPS device is pLatLon (l at, l on), and the heading angle is θ, and pLatLon (l at, l on) is converted into p0 (x0, y0, z0) by Cartesian transformation;

[0020] Step S4032: The relative position coordinates of the road damage point in the camera are p1(x1, y1, z1), and the quaternion is obtained from these coordinates

[0021] Step S4033, obtaining the heading angle quaternion θ(cos(θ / 2),0,0,sin(θ / 2)) according to the four-element calculation formula;

[0022] Step S4034, obtain the offset coordinates p3 (x3, y3, z3) of point p1 (x1, y1, z1) relative to point p0 (x0, y0, z0) according to the cross product of the four elements of the coordinates of point p1 and the four elements of the heading angle;

[0023] Step S4035, obtain the world coordinates p2 (x2, y2, z2) of point p2 by adding point p0 to point p3, and perform Cartesian transformation on p2 to obtain the longitude and latitude point pLatLon2 of the road defect.

[0024] According to the above technical solution, in step S4034, the offset coordinate p3 (x3, y3, z3) is obtained by cross-producting the four elements of the coordinates of point p1 and the four elements of the heading angle, which is specifically:

[0025]

[0026] Where p3(x3,y3,z3) is the offset coordinate.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] Through the method of the present invention, the urban road diseases collected by the collection equipment are presented on the three-dimensional urban road, which not only restores the appearance of the urban road more realistically, but also allows road maintenance personnel to intuitively observe the type, outline and degree of harm of the road diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a system architecture diagram of the present invention;

[0030] Figure 2 It is a schematic diagram of the collection vehicle of the present invention;

[0031] Figure 3 This is the conversion from Euler angle to quaternion of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Embodiment 1

[0034] like Figure 1 As shown, the disease collection and presentation method based on three-dimensional urban roads includes the following steps:

[0035] Step S1, building a three-dimensional city road; using modeling software to build the road that needs to be managed in the three-dimensional city scene;

[0036] Step S2, building a material library step; classifying road damage into expansion joints, linear cracks, block cracks, potholes, and blurred road markings according to the road damage type, placing corresponding material images in a folder and naming them according to the type;

[0037] Step S3, data collection step: install a camera and a GPS positioning device on the collection vehicle, adjust the time of the camera and the GPS device, turn on the device, drive the collection vehicle to travel along the road where road damage is required, collect video data and positioning data, and record the data collection time;

[0038] Step S4, road damage data analysis step;

[0039] Step S5, data presentation step; reading the road damage data in the database in the three-dimensional city scene; displaying the corresponding points at the corresponding positions in the three-dimensional scene and connecting them to form the outline of the road damage according to the type of road damage data and the longitude and latitude point set of the road damage contour; selecting the corresponding material in the road damage material library for filling according to the road damage type.

[0040] Through the method of the present invention, the urban road diseases collected by the collection equipment are presented on the three-dimensional urban road, which not only restores the appearance of the urban road more realistically, but also allows road maintenance personnel to intuitively observe the type, outline and degree of harm of the road diseases.

[0041] Embodiment 2

[0042] This embodiment is a further refinement of the first embodiment.

[0043] In step S1, building the road that needs to be managed in the modeling software is specifically as follows: building the road that needs to be managed in the modeling software according to the latitude, longitude and elevation information of the actual road, and restoring various attributes of the road in the modeling software according to the actual road information, including lane width, number of lanes and lane markings.

[0044] Step S4, the road damage data analysis step includes the following steps:

[0045] Step S401, extracting a frame image from the video data, with a resolution of 1920×1080, performing road disease recognition analysis on the image according to an existing road disease recognition model, extracting a pixel point set of the road disease contour in the image and identifying the road disease type, and recording the frame image and the time of the frame image as well as the pixel contour point set and type of the road disease;

[0046] The road damage recognition model uses existing technologies, such as the street view image-based road damage detection network YOLOv5s-M: This model is proposed for road damage detection in street view images and can accurately detect a variety of road damage in street view images, such as longitudinal cracks, transverse cracks, cracks, potholes, etc. The model was published in the Journal of Applied Earth Observation and Geoinformation and was trained using a dataset called SVRDD, which contains 8,000 street view images obtained from Baidu Maps and annotates more than 20,000 road damages, covering multiple areas and multiple urban road types in Beijing.

[0047] Drone road surface disease (crack) identification system: This system uses drones to inspect roads, and uses AI recognition systems to identify cracks and other diseases through road photos or videos collected by drones. The system can automatically identify road cracks larger than 5mm, with a recognition rate of more than 85%. The system is based on deep learning algorithms to train road crack disease images, and is efficient, accurate, and stable, and can continuously identify real-time road photo data.

[0048] Deep learning-based road damage detection model: This model uses deep learning technology to detect road damage, including different types of damage such as vertical cracks, horizontal cracks, alligator cracks, potholes, etc. The model uses CNN (convolutional neural network) as a basis and is trained with a large amount of road image data to identify and classify different types of road damage. This model is applicable to multiple scenarios and is trained through transfer training mode to adapt to different road environments and damage types.

[0049] Step S402: Analyze the frame image using the SDK and corresponding API method provided by the ZED camera based on the pixel coordinate point set of the identified contour points of the road damage on the image. The SDK uses the data from the depth module to calculate the 3D position of each pixel point of the bounding box of each road damage, and obtain the 3D coordinates of each pixel point.

[0050] Step S403, obtain the GPS data of the corresponding time according to the time on the frame image, and calculate the longitude and latitude point set corresponding to the 3D contour points of the road damage; Step S404, store the calculated data into the database, including the road damage type, longitude and latitude of the road damage, road damage contour, road damage collection time and road damage image data.

[0051] Step S403, calculating the latitude and longitude points of the road damage points is specifically:

[0052] Step S4031, when the road damage is detected, the longitude and latitude of the GPS device is pLatLon (l at, l on), and the heading angle is θ, and pLatLon (l at, l on) is converted into p0 (x0, y0, z0) by Cartesian transformation;

[0053] Step S4032: The relative position coordinates of the road damage point in the camera are p1(x1, y1, z1), and the quaternion is obtained from these coordinates

[0054] Step S4033, obtain the heading angle quaternion according to the four-element calculation formula

[0055] Step S4034, obtain the offset coordinates p3 (x3, y3, z3) of point p1 (x1, y1, z1) relative to point p0 (x0, y0, z0) according to the cross product of the four elements of the coordinates of point p1 and the four elements of the heading angle;

[0056] Step S4035, obtain the world coordinates p2 (x2, y2, z2) of point p2 by adding point p0 to point p3, and perform Cartesian transformation on p2 to obtain the longitude and latitude point pLatLon2 of the road defect.

[0057] Step S4034, the offset coordinate p3 (x3, y3, z3) is obtained by cross-producting the four elements of the coordinates of point p1 and the four elements of the heading angle, which is specifically:

[0058]

[0059] Where p3(x3,y3,z3) is the offset coordinate.

[0060] Embodiment 3

[0061] The inventive concept of the present invention is:

[0062] Build 3D urban roads; use 3D road editing software (such as E IVDSIM) or modeling software to build roads that need to be managed in 3D urban scenes. The latitude, longitude and elevation information of the roads must be roughly the same as the actual ones, with an error of no more than 1m; restore various attributes of the roads, including lane width, number of lanes, and lane markings.

[0063] Steps to build a material library: Classify road defects into expansion joints, linear cracks, block cracks, potholes, and blurred road markings according to the road defect type, put the corresponding material images in the folder and name them according to the type.

[0064] Data collection steps: Install cameras and GPS positioning equipment on the collection vehicle, such as Figure 2 As shown. Adjust the time of the camera and GPS device to be consistent with reality. Turn on the equipment and drive the collection vehicle to drive the road where road damage needs to be presented, collect video data and positioning data, and record the data collection time.

[0065] Steps for analyzing road damage data;

[0066] Step A1, extract frame images from video data, use convolutional neural network (CNN) deep learning algorithm to analyze road image data in real time, automatically extract features related to road diseases, such as cracks, potholes, etc. from road images, and classify and locate them according to these features. Report the corresponding images and road disease categories.

[0067] Step A2, according to the time on the road disease picture, obtain the GPS data of the corresponding time, that is, Figure 2 The latitude, longitude, elevation and heading angle data of point B in the middle. According to the SDK and API provided in the binocular camera, the 3D coordinate point data of the relative position of the road disease in the camera is extracted.

[0068] Step A3, calculate the longitude and latitude point (pLatLon2) of the road damage. When the road damage is detected, the longitude and latitude of the GPS device is pLatLon (l at, l on), the heading angle is θ, and the relative position coordinates of the road damage point in the camera are p1 (x1, y1, z1); the calculation principle is to obtain the three-dimensional world coordinate point p0 (x0, y0, z0) of the pLatLon point by Cartesian coordinate transformation, establish a spatial coordinate system under the camera's perspective, take the p0 point as the origin of the coordinate system, and rotate the p1 point toward the north by an angle of θ to obtain the offset coordinate p3 of the p1 point relative to the p0 point. The result of P3 plus the world coordinates of the p0 point is the world coordinates of the p2 point, and the longitude and latitude points of the p2 point are obtained through Cartesian transformation. The detailed calculation steps are as follows,

[0069] 1. It is known that the longitude and latitude of the GPS device when the road damage is detected is pLatLon (l at, l on). Cartesian transformation of pLatLon (lat, l on) is performed to obtain p0 (x0, y0, z0);

[0070] 2. The relative position coordinates of the road damage point in the camera are known to be p1(x1, y1, z1), from which the quaternion can be obtained

[0071] 3. The heading angle is known to be θ. According to Figure 3 The four-element calculation formula gets the heading angle quaternion Note: Because it is rotating toward due north, the angles φ and Ψ are both 0;

[0072] 4. According to the cross product of the four elements of the coordinates of point p1 and the four elements of the heading angle, the offset coordinates p3 (x3, y3, z3) of point p1 (x1, y1, z1) relative to point p0 (x0, y0, z0) are obtained;

[0073]

[0074] 5. Add point p0 to point p3 to get the world coordinates of point p2 (x2, y2, z2), and perform Cartesian transformation on p2 to get the longitude and latitude point pLatLon2 of the road disease;

[0075] p2=p0+p3

[0076] Step A4, storing the calculated data, including road damage type, road damage longitude and latitude, road damage outline, road damage collection time, and road damage image data, into a database.

[0077] Data presentation steps: reading road damage data from a database in a three-dimensional urban scene; generating road damage with corresponding contours at a specified location in the scene based on the latitude and longitude data of the road damage in the data; and selecting corresponding materials from a road damage material library for filling based on the road damage type.

[0078] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0079] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for collecting and presenting defects on three-dimensional urban roads, characterized by: The following steps are involved: Step S1, building a three-dimensional city road; Use modeling software to build roads that need to be managed in three-dimensional urban scenes; Step S2, building a material library step; classifying road damage into expansion joints, linear cracks, block cracks, potholes, and blurred road markings according to the road damage type, placing corresponding material images in a folder and naming them according to the type; Step S3, data collection step: install a camera and a GPS positioning device on the collection vehicle, adjust the time of the camera and the GPS device, turn on the device, drive the collection vehicle to travel along the road where road damage is required, collect video data and positioning data, and record the data collection time; Step S4, road damage data analysis; Step S5, data presentation step; Read the road damage data in the database in the three-dimensional city scene; display the corresponding points at the corresponding positions in the three-dimensional scene and connect them to form the outline of the road damage according to the type of road damage data and the longitude and latitude point set of the road damage contour; select the corresponding material in the road damage material library for filling according to the road damage type.

2. The method for collecting and presenting defects based on three-dimensional urban roads according to claim 1 is characterized in that: In step S1, building the road that needs to be managed in the modeling software is specifically as follows: building the road that needs to be managed in the modeling software according to the latitude, longitude and elevation information of the actual road, and restoring various attributes of the road in the modeling software according to the actual road information, including lane width, number of lanes and lane markings.

3. The method for collecting and presenting defects based on three-dimensional urban roads according to claim 1 is characterized in that: Step S4, the road damage data analysis step includes the following steps: Step S401, extracting a frame image from the video data, performing road damage recognition on the image according to a road damage recognition model, extracting a pixel point set of the contour of the road damage in the image and identifying the type of the road damage, and recording the frame image and the time of the frame image as well as the pixel contour point set and type of the road damage; Step S402, analyzing the frame image according to the pixel coordinate point set of the identified contour points of the road damage on the image, calculating the 3D position of each pixel point of the boundary box of each road damage, and obtaining the 3D coordinates corresponding to each pixel point; Step S403, obtaining GPS data of the corresponding time according to the time on the frame image, and calculating the longitude and latitude point set corresponding to the 3D contour points of the road disease; Step S404, storing the calculated data into a database, including road damage type, road damage longitude and latitude, road damage outline, road damage collection time, and road damage image data.

4. The method for collecting and presenting defects based on three-dimensional urban roads according to claim 3 is characterized in that: Step S403, calculating the latitude and longitude points of the road damage points is specifically: Step S4031, when the road damage is detected, the longitude and latitude of the GPS device is pLatLon (lat, lon), and the heading angle is θ, and pLatLon (lat, lon) is converted into p0 (x0, y0, z0) by Cartesian transformation; Step S4032: The relative position coordinates of the road damage point in the camera are p1(x1, y1, z1), and the quaternion is obtained from these coordinates Step S4033, obtain the heading angle quaternion according to the four-element calculation formula Step S4034, obtain the offset coordinates p3 (x3, y3, z3) of point p1 (x1, y1, z1) relative to point p0 (x0, y0, z0) according to the cross product of the four elements of the coordinates of point p1 and the four elements of the heading angle; Step S4035, obtain the world coordinates p2 (x2, y2, z2) of point p2 by adding point p0 to point p3, and perform Cartesian transformation on p2 to obtain the longitude and latitude point pLatLon2 of the road defect.

5. The method for collecting and presenting defects based on three-dimensional urban roads according to claim 4 is characterized in that: Step S4034, the offset coordinate p3 (x3, y3, z3) is obtained by cross-producting the four elements of the coordinates of point p1 and the four elements of the heading angle, which is specifically: Where p3(x3,y3,z3) is the offset coordinate.

Images