Lidar-based detection of roadway markings from a bird's eye view

Abstract

The invention relates to a method for detecting roadway markings from a bird's eye view, having the steps of: providing (S1) an irregular monochromatic image with intensity pixels from an aircraft-mounted or spacecraft-mounted lidar system; generating (S2) a monochromatic pixel image from the irregular monochromatic image, wherein the monochromatic pixel image describes the region of the earth's surface by means of adjacent pixels with respective brightness levels; and ascertaining (S5) roadway markings in the monochromatic pixel image by using an algorithmic superimposition of primitives with finite lengths on regions in the monochromatic pixel image which are particularly intensely reflective and by combining the roadway sections in order to form storable courses of the roadway markings.

Description

[0001] LiDAR-based detection of road markings from a bird's eye view

[0002] The invention relates to a computer-implemented method for detecting road markings from a bird's eye view.

[0003] LiDAR is an abbreviation for "Light Detection And Ranging." A LiDAR system typically uses light pulses from a laser emitter to scan an area. Scanning can be achieved, for example, by a rotating emitter; the emitter's speed relative to the area can also be used for this purpose. Observing the laser beam reflected in the area allows a distance measurement z to these locations by measuring the time t of the laser pulse, which may be reflected multiple times, using the speed of light constant c over ct=z. This means that for each scanning step, the LiDAR system may generate several three-dimensional coordinate points corresponding to objects that have (partially) reflected the light pulse. This, in turn, can be used to create spatial maps of the surrounding area.

[0004] LiDAR systems are therefore particularly suitable for environmental and traffic monitoring for highly automated vehicles, but are also used on aircraft and satellites, particularly for mapping and surveying.

[0005] For the typical application of a LiDAR system for spatial detection of the environment, it is not necessary to use polychromatic lasers, since monochromatic laser light is also well suited for distance measurement via the time of flight or indirectly via the time of flight using a sinusoidal laser signal with changing frequency and testing the interference of the transmitted signal and the signal received by reflection.

[0006] As long as a sufficiently high intensity of the reflected laser beam can be received by a sensor in the LiDAR system, the distance between the LiDAR system and the respective reflecting location in the environment can be determined. Depending on the surface of the reflecting location, different intensities are received by the LiDAR system. Using the known position and orientation of the LiDAR system and the associated knowledge of the emission direction of the laser beam, the reflecting location can be referenced in spatial coordinates, so that the information from the reflection can be assigned to a position on the Earth's surface, for example, by specifying the longitude and latitude in the EPSG:4326 system, and thus also to a position in a digital map.

[0007] A variety of methods, such as earth observation from an aircraft or satellite, are suitable for improving the quality of digital maps. High-precision road maps, in particular, are becoming increasingly important for enabling highly automated driving. This also includes lane markings and road quality.

[0008] The object of the invention is to automatically record road markings over a large area in order to be able to store information about them.

[0009] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0010] A first aspect of the invention relates to a computer-implemented method for detecting road markings from a bird's eye view, comprising the steps:

[0011] - Providing an irregular monochromatic image with intensity pixels assigned to a region of the Earth's surface, which were obtained from the sequential emission of a laser signal over a plurality of different locations on the Earth's surface by an aircraft-borne or satellite-borne LiDAR system and by the sensory detection of reflections of the laser signal by means of a sensor unit of the LiDAR system;

[0012] - generating a monochromatic pixel image from the irregular monochromatic image, wherein the monochromatic pixel image describes the area of ​​the Earth's surface by means of adjacent pixels with respective brightness levels; and

[0013] - Determining road markings in the monochromatic pixel image by algorithmically superimposing, in particular, vector-graphic primitives of finite length onto road markings appearing in the monochromatic pixel image as particularly intensely reflecting elongated areas, and by composing the, in particular, vector-based route sections to, in particular, vector-graphic storable courses of the road markings.

[0014] The distance measurement of the LiDAR system, for which such a LiDAR system is typically designed, is not necessary for identifying lane markings, but can be used in combination, for example, to better assign the identified lane markings to a corresponding position on the Earth's surface. In particular, the combination of highly reliably identified lane markings with simultaneous, highly accurate knowledge of their position can be used to add or update digital maps with information about lanes.

[0015] Furthermore, a statistical distribution of the brightness levels of the monochromatic pixel image over a georeferenced area of ​​the monochromatic pixel image of the road markings or the entire road surface can be used to characterize the quality of the road markings or the quality of the entire road surface. The area of ​​the monochromatic pixel image for roads can be localized, for example, from the georeferenced overlay with cadastral city maps.

[0016] Information about road markings can be obtained in high quality in monochromatic images (for example in GeoTiff format) from the LiDAR laser pulse reflection information due to the high reflectivity of road markings compared to their surroundings, especially on the road surface.

[0017] It is an advantageous effect of the invention that otherwise unused intensity data regarding the reflected laser signals of a LiDAR system can be used to identify road markings.

[0018] Since lane markings and other road markings are typically designed to be highly reflective, for example to reflect the light from vehicle headlights or street lamps, they also reflect the laser light emitted by the LiDAR system particularly well. For this purpose, lane markings often contain a binding agent, for example, based on acrylate, as well as special reflective elements such as reflective beads, particularly made of glass, and color pigments. These reflective properties can be ideally utilized when the laser signal from the LiDAR system strikes the surface perpendicularly at the point of reflection, and is thus also reflected essentially perpendicularly from the surface and received by a sensor of the LiDAR system on an aircraft or satellite.

[0019] This method enables the fully automated, large-area detection of road markings from airborne LiDAR measurements. Aircraft-mounted LiDAR systems are preferred over satellite-based LiDAR systems, as the significantly greater distance from satellites to the Earth's surface relative to the distance from aircraft leads to increased noise levels, typically a greater broadening of the laser pulse, and potentially a very significant attenuation of the intensity of the emitted laser pulse as it travels to the Earth's surface and back to the satellite after reflection from the Earth's surface.

[0020] The use of a monochromatic LiDAR system actively scans the environment with a well-defined wavelength. This approach avoids the inaccuracies inherent in passive sensors that rely on reflected sunlight, allowing the generated data to have a better signal-to-noise ratio. Furthermore, the LiDAR system can also be used at night.

[0021] According to an advantageous embodiment, the method further comprises the following step, which is carried out before determining the road markings: converting the monochromatic pixel image into an image with binary information; wherein, when determining road markings, the image with the binary information is the monochromatic pixel image, because an image with binary information is a special case of a monochromatic image.

[0022] Ideally, converting the data into binary information results in the road markings being present as information in the image, while all other information is removed. This corresponds to a form of semantic segmentation. To obtain the image from binary information, a threshold can be defined that divides all information in the pixel image into two groups (often referred to in technical terms as applying a "threshold"). For example, the threshold is defined as the mean plus one standard deviation of the reflectance values ​​of all cells of the regular monochromatic pixel image or any georeferenced part of the monochromatic pixel image. For example, the image with binary information is defined by the value 1 in each cell with reflectance values ​​above the threshold and with the value 0 below the threshold.In addition, the image can be stabilized with binary information, particularly through the application of a binary dilation erosion filter (also known in technical terms as a "binary dilation erosion filter"). The dilation erosion filter allows, for example, reducing noise in a binary image when individual, isolated pixels have a value of 1, but their surroundings have a value of 0.

[0023] According to an advantageous embodiment, the method further comprises the step which is carried out before determining the road markings:

[0024] Graphically enhancing the monochromatic pixel image by at least one of the following methods: increasing a contrast, in particular using a bandpass filter, preferably with a Difference of Gaussians filter; adjusting a brightness; adjusting a gamma value;

[0025] According to an advantageous embodiment, at least one of the following is used for the particularly vector-graphic primitives: curves, in particular Bezier curves, straight track sections;

[0026] According to an advantageous embodiment, the algorithmic superimposition of, in particular, vector-graphic primitives of finite length onto the road markings appearing in the monochromatic pixel image is carried out by means of image-analytical edge detection.

[0027] Edge detection highlights contours and edges in the pixel image data. Various well-known edge detection algorithms can be used, such as the well-known "Canny edge detector."

[0028] According to an advantageous embodiment, in order to determine the road markings in the monochromatic pixel image, only the particularly intensely reflecting areas of the monochromatic pixel image are taken into account in the area of ​​a road existing according to a digital map.

[0029] According to an advantageous embodiment, after the road markings have been identified, the respective condition of the respective road markings is determined using the intensity values ​​of the laser signals of the LiDAR system reflected by the road markings. The intensity of the reflection provides information about the condition of a road marking, since worn, older road markings tend to reflect the laser signals of the LiDAR system significantly less effectively than new, flawless road markings. By identifying the road markings using the image data of the LiDAR system, their positions in the image data are known, thus allowing the intensity values ​​to be analyzed precisely at these positions. This advantageously prevents intensity values ​​from locations on the earth's surface that are not road markings from being erroneously used to determine the quality of the road markings.

[0030] According to an advantageous embodiment, the quality of the road markings can be determined by recording them repeatedly in succession, e.g., monthly, using the laser system. The corresponding time series can provide information about the degradation of the road markings.

[0031] According to an advantageous embodiment, the time series forms the basis for an automated system for the quality control of streets, for example if the laser system is installed on a highly automated drone that systematically flies over streets georeferenced by digital maps.

[0032] According to an advantageous embodiment, a position area of ​​a roadway, associated with the determined road markings, is determined with the aid of a digital map, and a respective condition of the respective roadway is determined by means of the intensity values ​​of the laser signals of the LiDAR system reflected from the roadway.

[0033] According to an advantageous embodiment, the variability of the mean value in the pixel image can also be determined for a road in order to determine the quality or condition of the entire roadway. This road can be georeferenced, for example, using digital maps.

[0034] According to an advantageous embodiment, the monochromatic pixel image is generated from the irregular monochromatic image by superimposing a regular grid with a predetermined cell size over the irregular monochromatic image, wherein respective pixel values ​​are assigned to the cells of the grid depending on intensity pixels obtained by laser reflections in the irregular monochromatic image.

[0035] The irregular monochromatic image generally exhibits a certain variability in the geometric distribution of the obtained laser reflections relative to the Earth's surface. The pixel image is generated using the grid to obtain a regular image. A cell size based on the average spacing of the geocoordinates of the LiDAR points is preferred, for example: For conventional, airborne LiDAR measurements, one square meter typically contains 10 measurement points, which, depending on the terrain, generate 10 to 20 LiDAR points. This means that the measurement points have an average spacing of approximately (1 m). 2 / 10) 5, i.e. approximately 30cm. With a circle diameter of 2m, more than 30 LiDAR points are usually recorded, which corresponds to sufficient statistics for the brightness levels. Preferably, the center of each cell of the pixel image is annotated with an intensity value, whereby particularly preferably statistical parameters relating to the intensity are determined for each of the cells, in particular: minimum, maximum, mean, standard deviation, etc.; this can also be used to filter out outlier values, for example using quantile statistics such as the median. The pixel image thus provisionally obtained can contain cells with empty statistics, i.e. no-data cells. Cells without data can, for example, be filled with a value from a surrounding cell.

[0036] According to an advantageous embodiment, a probabilistic Hough transformation is used to determine the road markings in the monochromatic pixel image in order to superimpose the vector-graphic primitives of finite length, in particular, on the road markings in the pixel image.

[0037] According to an advantageous embodiment, a respective one of the intensity pixels of the irregular monochromatic image is formed from a respective distance profile of elementary reflections of a laser signal for a respective location on the earth's surface by identifying a respective maximum of the elementary reflections for each location-related laser signal and determining an area of ​​the distance profile in the region of the respective maximum as the intensity of the respective intensity pixel.

[0038] It has been found that using the area instead of the amplitude value of the maximum in the distance curve of elementary reflections can lead to higher-quality values ​​for determining the intensity at a respective pixel of the irregular monochromatic image. Integrated variables are less sensitive to noise than individual measured values. A further aspect of the invention relates to a system comprising an aircraft having a LiDAR system configured to generate an irregular monochromatic image with intensity pixels of a region of the Earth's surface, which are obtained from the emission of a laser signal over a plurality of different locations on the Earth's surface by the LiDAR system and by the sensory detection of reflections of the laser signal by means of a sensor unit of the LiDAR system, and having a computing unit configured to carry out the method according to one of the preceding steps.

[0039] A further aspect of the invention relates to a data processing device comprising a processor configured to carry out the method according to one of the preceding steps.

[0040] A further aspect of the invention relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to one of the preceding steps.

[0041] Advantages and preferred developments of the proposed system, the proposed device and the proposed computer program result from an analogous and analogous transfer of the statements made above in connection with the proposed computer-implemented method.

[0042] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0043] They show:

[0044] Fig. 1 : Schematic of a method for detecting road markings from a bird's eye view according to an embodiment of the invention.

[0045] Fig. 2: An example irregular monochromatic image of a LiDAR system.

[0046] Fig. 3: An example monochromatic pixel image. Fig. 4: An example monochromatic pixel image processed by edge detection.

[0047] Fig. 5: An example of a monochromatic pixel image prepared by brightness and contrast adaptation.

[0048] Fig. 6: An example binary image generated by threshold application.

[0049] Fig. 7: The overlay of parameterized curves on the road markings of the LiDAR image according to an embodiment of the invention.

[0050] Fig. 8: An exemplary distance curve of elementary reflections in relation to a laser pulse for a location on the earth's surface.

[0051] Fig. 2 to Fig. 7 do not show any device objects or method sequences within a respective figure. The recognition of details is therefore irrelevant. Fig. 2 to Fig. 7 merely serve to illustrate an exemplary embodiment of the method for recognizing road markings by a LiDAR system of an aircraft or a satellite. All images in Fig. 2 to Fig. 7 relate to the same area of ​​the earth's surface that is searched for road markings, so that the figures represent intermediate results in ascending order or alternative approaches to the goal of Fig. 7 by way of example. In comparison to Fig. 2 to 6, Fig. 7 shows a small partial section of the exemplary earth's surface scene in order to illustrate road details and make individual road markings more visible. Fig. 2 to Fig.7 do not exclude the possibility that deviations from the exemplary procedure and the observation of other areas of the Earth's surface may lead to completely different appearances.

[0052] Fig. 1 shows a method executed on a computer for detecting road markings that can be extracted from an image obtained by an aircraft LiDAR. Data from a LiDAR system is available that can be combined into an irregular monochromatic image with irregularly distributed intensity pixels. The intensity pixels are created by a pulsed laser of the LiDAR system being operated from the aircraft and sequentially scanning raster points on the earth's surface. The echo is recorded on the aircraft by a sensor unit and stored georeferenced as monochromatic intensity pixels. In a first step of the method executed on the computer, this irregular monochromatic image is provided S1.This is done via a corresponding data interface, which receives the irregular monochromatic image of the LiDAR system from the aircraft, for example wirelessly or via a data storage device. An example irregular monochromatic image is shown in Fig. 2. The grayscale pixels represent reflections of the laser signal. White dots represent missing measurements, i.e., points on the earth's surface with missing measurement information, so-called no-data. The laser signals hit several roads that have lane markings. In the second step, a monochromatic pixel image S2 is generated from the irregular monochromatic image, which is shown as an example in Fig. 3. For this purpose, a predefined regular grid is superimposed on the irregular monochromatic image of Fig. 2 in order to define individual pixels for image areas of an image. These pixels lie adjacent to one another and thus provide the entire image area with information.Instead of interpolating the intensity values ​​of the irregular monochromatic image, it is preferred to assign exactly one intensity value to a pixel (also called a cell) according to statistics of the brightness values ​​in the cell, so that each pixel receives a monochromatic brightness value. With the regular monochromatic pixel image, a regular image analogous to a photograph is obtained, which can be manipulated using conventional image processing. If necessary, the graphic enhancement S3 of the monochromatic pixel image is carried out in a further step using at least one of the following methods: increasing a contrast, in particular using a bandpass filter, preferably a Difference of Gaussians filter; adjusting a brightness; adjusting a gamma value. However, this step is optional and is carried out when required. Using a Canny Edge Detector, which is applied to the enhanced pixel image of Fig.3 is applied, contours can be identified. This is shown as an example in Fig. 4. Fig. 5, on the other hand, shows an alternatively enhanced regular pixel image in which an attempt is made to graphically highlight the particularly reflective properties of the road markings by means of contrast and brightness changes. In all cases, as shown in Fig. 6, the monochromatic pixel image obtained using one of the options described above (for example, in this case in grayscale) is converted into an image with binary information. For this purpose, a threshold value is defined, and all brightness values ​​are split along this threshold value. As a result, as indicated in Fig. 6, every pixel with a value above the threshold value is highlighted as an example.Now, the determination S5 of the road markings can be performed by algorithmically superimposing vector-graphic primitives of finite length onto the elongated areas in the monochromatic pixel image that are particularly intensely reflective. This is achieved using a Hough transformation, which superimposes Bezier curves or straight road sections of finite length onto the road markings visible in the binary image. This is indicated in Fig. 7, where the short white, vector-based defined road sections converge onto the luminous road markings using the Hough algorithm. By algorithmically combining these primitives into courses corresponding to the extent of the real road markings, a data set of the position and extent of the real road markings is obtained.These represent continuous, vector-graphic, storable progressions of the road markings. For example, a road marking several hundred meters long can be incorporated into a digital map as a dataset, analogous to a vector graphic. Furthermore, the georeferenced primitives can be used, for example, to document the degradation of the road markings during monthly LiDAR measurements of a geographical area. Typically, the reflectivity of the marking decreases due to abrasion when driving over it with car tires and due to weather conditions. More generally, the structure and texture of the entire road surface can be quantified and documented over time by overlaying the georeferenced pixel image with road maps.

[0053] Fig. 8 schematically outlines an example echo signal of a laser signal emitted by a LiDAR system. Due to very weak reflections from the atmosphere and objects in the atmosphere, an interference pattern occurs, depending on the height s of the respective reflections, which can be interpreted as noise. However, a clear maximum in the amplitude A of reflection is obtained from objects such as power lines, tree branches, roof edges, or on the earth's surface. Road markings, in particular, are highly reflective objects. To determine the overall intensity of an image point based on such an echo signal, the maximum of the elementary reflection is identified, and an area of ​​the distance curve A(s) around the maximum is determined as the intensity (shaded area in Fig. 8). If several such intensities are combined into one value through statistical analysis in a local environment, this value forms the basis for the regular pixel image.

[0054] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

[0055] List of reference symbols

[0056] S1 Provision

[0057] 52 Generate

[0058] 53 Upgrade S4 Transfer

[0059] S5 Determine

Claims

Patent claims 1 . A computer-implemented method for detecting road markings from a bird's-eye view, comprising the steps: - Providing (S1) an irregular monochromatic image with intensity pixels obtained from the sequential emission of a laser signal over a plurality of different locations on the Earth's surface by an aircraft-borne or satellite-borne LiDAR system and by the sensory detection of reflections of the laser signal by means of a sensor unit of the LiDAR system; - generating (S2) a monochromatic pixel image from the irregular monochromatic image, wherein the monochromatic pixel image describes the area of ​​the earth's surface by means of adjacent pixels with respective brightness levels; and - Determining (S5) road markings in the monochromatic pixel image by algorithmically superimposing primitives of finite length onto areas in the monochromatic pixel image with particularly intense brightness levels, and by composing the route sections to storable courses of the road markings.

2. The method according to claim 1, further comprising the step performed before determining the lane markings: - graphically enhancing (S3) the monochromatic pixel image by at least one of the following methods: increasing a contrast; adjusting a brightness; adjusting a gamma value; image texture filter; 3. Method according to one of the preceding claims, further comprising the step which is carried out before determining the lane markings: - converting (S4) the monochromatic pixel image into an image with binary information; wherein, when determining road markings, the image with the binary information is the monochromatic pixel image. 14 REVISED SHEET (RULE 91) ISA / EP 4. Method according to one of the preceding claims, wherein the algorithmic superimposition of primitives of finite length onto the road markings appearing in the monochromatic pixel image is carried out by means of image-analytical edge detection.

5. Method according to one of the preceding claims, wherein, in order to determine the road markings in the monochromatic pixel image, only the particularly intensely reflecting areas of the monochromatic pixel image are taken into account in the area of ​​a road existing according to a digital map.

6. Method according to one of the preceding claims, wherein after determining the road markings, a respective state of the respective road markings is determined by means of the intensity values ​​of the laser signals of the LiDAR system reflected by the road markings.

7. Method according to one of the preceding claims, wherein a position range of a roadway, associated with the determined road markings, is determined with the aid of a digital map, and a respective state of the respective roadway is determined by means of the intensity values ​​and their spatial texture profiles of the laser signals of the LiDAR system reflected from the roadway.

8. Method according to one of the preceding claims, wherein the monochromatic pixel image is generated from the irregular monochromatic image by superimposing a regular grid with a predetermined cell size over the irregular monochromatic image, wherein respective pixel values ​​are assigned to the cells of the grid depending on intensity pixels obtained by laser reflections in the irregular monochromatic image.

9. Method according to one of the preceding claims, wherein a probabilistic Hough transformation is applied to determine the road markings in the monochromatic pixel image in order to superimpose the finite-length primitives on the road markings in the pixel image. 15 REVISED SHEET (RULE 91) ISA / EP 10. Method according to one of the preceding claims, wherein a respective one of the intensity pixels of the irregular monochromatic image is formed from a respective distance profile of elementary reflections of a laser signal for a respective location on the earth's surface by identifying a respective maximum of the elementary reflections for each location-related laser signal and determining an area of ​​the distance profile in the region of the respective maximum as the intensity of the respective intensity pixel. 16 REVISED SHEET (RULE 91) ISA / EP

Images