Procedures for classifying road markings and vehicles

Abstract

Method for classifying road markings (1), wherein a light pattern (3) consisting of at least two differently illuminated polygonal surfaces (4.1, 4.2) is projected at least temporarily onto a road section (6) in front of an ego vehicle (5) by means of at least one projector (2), and at least one camera image generated by at least one vehicle camera (7), which includes at least one road section (6) partially covered by the light pattern (3), is evaluated on a computing unit (8) for the detection of road markings (1), characterized in that at least one road marking (1) detected in a camera image and at least partially covered by the two differently illuminated polygonal surfaces (4.1, 4.2) is classified as a yellow road marking (1) if the road marking (1) is visible in both polygonal surfaces (4.1, 4.2).2) is essentially yellow in color and the road marking (1) is classified as white road marking (1) if the road marking (1) has a different color in the differently illuminated polygonal areas (4.1, 4.2).

Description

[0001] The invention relates to a method for classifying road markings according to the type defined in the preamble of claim 1 and to a vehicle according to the type defined in the preamble of claim 9.

[0002] To increase road safety and driver comfort, modern vehicles are often equipped with a variety of driver assistance systems. For example, adaptive cruise control allows a vehicle to automatically maintain a constant distance from the vehicle ahead. Emergency braking assist enables a vehicle to react quickly to obstacles suddenly appearing in front of it, braking to avoid a collision. Other driver assistance systems include lane keeping assist and traffic sign recognition. These systems typically require environmental monitoring to detect and analyze relevant information.In addition to sensor systems such as distance radar, ultrasonic sensors or laser scanners, optical systems using cameras are particularly employed for this purpose.

[0003] One problem with using camera systems when the sun is low in the sky is the tinting of white road markings by yellow, orange, or red sunlight. The color temperature of daylight shifts with the sun's position. The color temperature emitted by the sun changes from white or light yellow through orange to red. Thus, white road markings appear white during the day. However, if the formerly white road markings reflect the light of a low sun, these lanes can also appear yellow. In this case, there is a risk that a driver assistance system, which uses camera images as input, will incorrectly identify the white road markings as yellow. Yellow road markings are considered temporary in Europe, for example, in construction zones.Typically, driver assistance systems adjust their behavior when a vehicle is operating in an area with temporary lane markings. For example, speeds are reduced and / or adjusted safety distances are maintained. However, such a change in the behavior of a driver assistance system due to a falsely detected yellow lane marking should be avoided.

[0004] For environmental recognition and image processing, various methods are known from the state of the art, e.g. the following: German patent DE 10 2015 008 774 A1 discloses a device for detecting and a method for illuminating the surroundings of a vehicle. The method disclosed in the document uses active triangulation to determine distances and generate depth information. For this purpose, a light pattern is projected into the surroundings of a vehicle using a projector, and the light pattern reflected from the surroundings is captured by a camera positioned at a lateral offset from the projector. Distance information can then be obtained by determining and evaluating the distance traveled along a propagation line by a specific feature of the light pattern until it is reflected from a surface. A vehicle headlight, for example, can be used as the projector. In particular, this is a so-called matrix headlight.To avoid distracting the driver when the light pattern is emitted, it is displayed for such a short time that the driver cannot perceive it. The pattern can be emitted while the vehicle's main lighting, such as dipped or main beam headlights, is active. This allows the pattern to be emitted in addition to the main lighting, or the main lighting can be briefly deactivated or at least dimmed to allow the pattern to be displayed. This maintains a constant average light intensity during the pattern's emission, further reducing its perceptibility to the driver. Various patterns can be used, such as lines, stripes, dots, grids, checkerboards, and / or pseudo-random patterns.

[0005] Furthermore, DE 10 2017 124 956 A1 discloses a method for extracting a light pattern from a camera image. The document also describes a method for distance measurement using active triangulation. In this process, a filter operator applied in image processing improves the perceptibility of the light pattern emitted into the environment within a camera image.

[0006] Furthermore, DE 10 2017 124 955 A1 describes a method for determining the positional relationship between a camera and a vehicle's headlight. Using the method disclosed in the publication, a particularly simple and fast calibration of a system consisting of a camera and a headlight suitable for determining distances using active triangulation is possible. The light pattern can, for example, also include a checkerboard pattern. The vertices of adjacent rectangles in the checkerboard pattern are used as feature points.

[0007] Furthermore, DE 10 2008 031 861 A1 discloses an optical lane sensor that can be used to issue control commands to a vehicle based on detected lane markings. The optical lane sensor detects lane markings by means of a contrast difference between the lane marking and the road surface. The optical lane sensor can include filters of different colors, thus enabling the detection of a contrast difference between the road surface and the lane markings colored according to the filters. Alternatively, active illumination of the scene with colored light can be used as a filter.

[0008] The present invention is based on the objective of providing a method for classifying road markings, by means of which it is possible to distinguish between white and yellow road markings using a camera system, even if white road markings appear yellow due to the reflection of warm white light, such as light emitted by a low sun. A further objective of the present invention is to provide a vehicle that uses the method according to the invention.

[0009] According to the invention, this problem is solved by a method for classifying road markings with the features of claim 1 and a vehicle with the features of claim 9. Advantageous embodiments and further developments are described in the dependent claims.

[0010] In a method for classifying road markings of the type mentioned at the outset, according to the invention, at least one road marking detected in a camera image and at least partially covered by the two differently illuminated polygonal surfaces is classified as a yellow road marking if the road marking has a substantially yellow color in both polygonal surfaces, and the road marking is classified as a white road marking if the road marking has a different color in the differently illuminated polygonal surfaces.

[0011] Using the method according to the invention, an optical environmental perception system employing at least one camera is reliably able to distinguish between yellow and white road markings, even when the sun is low. By illuminating a road marking with varying intensity in certain sections, the actual color of the road marking can be discerned in a camera image. For example, if a white road marking appears yellow due to the reflection of warm white sunlight from a low sun, a corresponding, dimly illuminated section of the road marking will appear yellow, while a section illuminated particularly brightly by the projector will appear white.For this purpose, the projector emits light which, in the brightly illuminated polygonal areas, has a luminosity at least as high as the sunlight reflected by the road marking. This allows the reflected sunlight to be overpowered, revealing the true color of the road marking. If, however, the road marking is yellow, it will appear yellow in both the brightly and dimly lit areas. The polygonal areas are designed so that a road marking with a defined shape and minimum area, at a specified distance from the projector or camera, is illuminated in both strong and dim lighting conditions over at least one section.

[0012] In general, it is also conceivable that a white road marking could appear as a yellow road marking due to the reflection of warm white light emitted by a light source that deviates from a low-lying sun.

[0013] For example, a billboard, street light, neon sign or the like can make a white road marking appear yellow.

[0014] An advantageous further development of the method provides that the projector emits cool white light, in particular light with a color temperature of at least 6500 K. By using cool white light to illuminate the brightly lit polygonal surfaces, yellow and white road markings can be distinguished from one another even more easily. This allows the warm white light emitted by the sun to be overpowered with a particularly clear color contrast. If warm white light, for example with a color temperature of 3000 K, were used, there would still be a risk that even a section of a white road marking illuminated by a bright polygonal surface would appear yellow. According to another advantageous embodiment of the method, the projector emits a light pattern in the form of a checkerboard pattern. In general, the light pattern can be designed as desired.For example, it can consist of bar-shaped polygonal areas extending across a roadway perpendicular to the direction of travel of a vehicle, alternating between light and dark or unlit areas. However, if only polygonal areas running perpendicular to a roadway are used, there is a risk that, for example, when driving around a curve and / or when lane markings running diagonally to the roadway, they will not be sufficiently covered by both a light and a dark polygonal area. For instance, a lane marking could run parallel to a polygonal area, making it impossible to distinguish between a yellow and a white lane marking, as the lane marking would only appear in a single color.However, by using a checkerboard-shaped light pattern, the pattern features at least one brightly and one dimly illuminated polygonal area both in the direction of travel and perpendicular to it. This increases the reliability of illuminating the road markings in such a way as to allow differentiation between yellow and white road markings.

[0015] An advantageous further development of the method provides that at least one vehicle headlight, in particular a matrix headlight, is used as the projector, wherein the vehicle headlight preferably uses LEDs as the light source. By using a vehicle headlight as a projector, the effort required to integrate a corresponding system for carrying out the method according to the invention can be reduced. Thus, a separate projector is no longer necessary for carrying out the method according to the invention. In particular, a light pattern in the form of a checkerboard pattern can be generated very easily when using a matrix headlight. For this purpose, for example, a first number of light sources of the matrix headlight are activated, and the remaining light sources of the matrix headlight are dimmed or deactivated. LEDs are preferably used as the light source because they allow for particularly efficient operation of the matrix headlight.Furthermore, LEDs can be used to produce a particularly cool white light, especially with a color temperature above 6500 K. Generally, however, it is also conceivable that the vehicle headlight could use a laser as its light source. The light pattern can also be generated, for example, by incorporating a stencil, lens, and / or prism arrangement into the light propagation path between the light source and the road surface. This, however, makes the vehicle headlight design more complex. Additionally, the light pattern can be emitted simultaneously by multiple projectors or headlights. This allows for the illumination of a larger area.

[0016] According to a further advantageous embodiment of the method, a multifunction camera is used as the vehicle camera. Often, a multifunction camera is already integrated into a vehicle. The multifunction camera generates camera images, which, or information obtained from an analysis of the camera images, are used as input by a variety of different driver assistance systems. The multifunction camera can be either a mono or a stereo camera. In any case, however, the multifunction camera must be able to recognize different colors, in particular to distinguish between the color temperatures of warm white light and cool white light. It is also conceivable, however, that at least one separate and / or even a further camera is provided on the vehicle to carry out the method according to the invention.A multi-function camera is typically oriented in one direction of travel of the vehicle. It is conceivable, however, that at least one camera is also positioned at the rear of the vehicle to monitor the area behind it. It is also conceivable that the vehicle has cameras oriented laterally, i.e., perpendicular to one direction of travel. This allows for the monitoring of a larger surrounding area, enabling the reliable detection of lane markings that are not directly in front of the vehicle. Accordingly, it is also necessary that a corresponding light pattern is projected into the areas monitored by the other cameras in order to classify the lane markings to the sides.

[0017] A further advantageous embodiment of the method provides that the light pattern is emitted for such a short time that it cannot be perceived by a person, in particular by operating the projector in a pulsed mode, with the light pattern being projected at a frequency on the order of 20 Hz. This ensures that a driver is not disturbed by the emitted light pattern. The duration for which the light pattern is emitted during a pulse can be, for example, 1 / 100 or 1 / 1000 of a second. For this purpose, the camera and projector are synchronized so that a camera image is generated precisely when the light pattern is emitted. By emitting the light pattern several times per second, continuous monitoring of the roadway or road markings is possible.This reduces the risk of a road marking being "overlooked." A frequency of 20 Hz has proven effective. However, it is also possible to emit the light pattern more or less frequently than 20 times per second. The emission duration can also correspond exactly to one-twentieth of a second.

[0018] According to a further advantageous embodiment of the method, camera image analysis is performed using an edge filter to amplify intensity differences between two adjacent white and yellow road marking sections. This allows for even more reliable differentiation between a white and a yellow road marking. The color difference in a camera image is thus amplified, enabling an image recognition algorithm to more reliably distinguish between yellow and white color temperatures when evaluating camera-generated images. Such a filter can, for example, be provided as a physical filter element and arranged in a beam path between the projector and the camera, perhaps integrated into the camera optics. Alternatively, the edge filter can be implemented in software, i.e., by an image recognition algorithm.

[0019] Preferably, the less brightly lit polygonal areas are not illuminated. By not illuminating the dimly lit polygonal areas, a particularly clear and sharply defined brightness and color contrast is created between two adjacent sections of a road marking, one covered by a brightly lit polygonal area and the other by a dimly lit polygonal area. This makes it even easier to distinguish white from yellow sections of a white road marking.

[0020] According to the invention, in a vehicle with at least one projector, in particular in the form of a matrix headlight, a processing unit, and a camera, the processing unit is configured to carry out a method described above. The vehicle can be any type of vehicle, such as a car, truck, van, bus, motorcycle, or the like. The processing unit can be a separate processing unit, a central on-board computer, or a control unit of a vehicle subsystem.

[0021] Preferably, the vehicle is at least partially automated, with lane markings detected by a camera system serving as input for a driver assistance system for the at least partially automated control of the vehicle. Using the method according to the invention, such a vehicle can thus be operated safely and reliably even when the sun is low, since there is no risk of misinterpreting white lane markings as yellow lane markings when the sun is low.

[0022] Further advantageous embodiments of the method and the vehicle according to the invention also result from the exemplary embodiment, which is described in more detail below with reference to the figure.

[0023] The single figure shows a schematic representation of an application of a method according to the invention for distinguishing between yellow and white road markings when the sun is low in the sky.

[0024] The figure shows a top view of a roadway 13, which has several lanes separated by lane markings 1. The figure depicts an ego-vehicle 5 or vehicle 12 driving towards a sunset on roadway 13, indicated by a sun 15 shown above a horizon 14 from the perspective of the driver of the ego-vehicle 5. The sun 15 emits light with a warm white color temperature. Typically, the lane markings 1 are white. However, due to the low angle of the sun 15, they appear yellow. In Europe, yellow lane markings 1.G are considered temporary lane markings 1, used, for example, to warn drivers of a critical road surface. This often results in a reduction of lane width in construction zones. Furthermore, this is usually accompanied by a reduced speed limit.

[0025] The ego-vehicle 5 includes a vehicle camera 7, which captures a section of road 6 ahead of the ego-vehicle 5. Camera images generated by the vehicle camera 7 are evaluated on a processing unit 8 of the ego-vehicle 5. The ego-vehicle 5 can be controlled at least semi-automatically, whereby lane markings detected by the evaluation of the camera images are taken into account for the control of the ego-vehicle 5. If the processing unit 8 detects, for example, white lane markings 1.W, the ego-vehicle 5 is controlled according to a first operating mode. If the processing unit 8 detects yellow lane markings 1.G, the ego-vehicle 5 is controlled according to a second operating mode. However, due to the low angle of the sun 15, the white lane markings 1.W also reflect the white sunlight with a warm white color temperature, which makes the white lane markings 1.W appear yellow.This is indicated by the two lane markings 1 located directly in front of the ego vehicle 5 in the direction of travel.

[0026] However, with the aid of a method according to the invention, it is possible to distinguish between white and yellow road markings 1.W and 1.G even when the sun is low 15. For this purpose, the ego vehicle 5 comprises at least one projector 2, in the example shown in the figure in the form of two vehicle headlights 10, which projects a light pattern 3 onto a road section 6 in front of the ego vehicle 5 onto the roadway 13. The light pattern 3 comprises a plurality of polygonal surfaces 4.1 and 4.2, which have different illuminance levels. The polygonal surfaces 4.1 have a bright luminance, and the polygonal surfaces 4.2 have a dimmer luminance in comparison. Preferably, the polygonal surfaces 4.2 are not illuminated. The light pattern 3 is projected onto the road section 6 in such a way that it at least partially covers the road markings 1. Preferably, the light pattern 3 is designed as a checkerboard pattern 9.The individual polygonal surfaces 4.1 and 4.2 can, for example, be designed as squares or rectangles, as shown.

[0027] To illustrate a functional principle of the method according to the invention, the figure shows yellow road markings 1.G in the left half of the figure and white road markings 1.W in the right half. If the road markings 1 are yellow road markings 1.G, then both the sections of the road marking 1 superimposed by the light polygonal areas 4.1 and those superimposed by the dark polygonal areas 4.2 appear yellow, as indicated by reference numeral 11.2. However, if the road markings 1.W are white, then the sections of the white road markings 1.W covered by the light polygonal areas 4.1 reflect light with a color temperature essentially corresponding to the light emitted by the projectors 2, and the sections of the white road markings 1.W covered by the dark polygonal areas 4.2 reflect yellow light. Thus, the sections indicated by reference numeral 11.2 appear yellow.Sections of the white road markings 1.W marked with 1 and sections of the white road markings 1.W marked with reference sign 11.2 are yellow, which allows yellow 1.G road markings to be distinguished from white 1.W road markings.

[0028] In other words, when the light pattern 3 is projected onto yellow road markings 1.G, they appear yellow when reflected by warm white sunlight, and white road markings 1.W appear alternately yellow and white. Preferably, the vehicle headlights 10, particularly in the form of matrix LED headlights, emit cool white light. Preferably, the color temperature of the light emitted by the vehicle headlights 10 is at least 6500 K. This improves the visibility of the white road markings 1.W. It is also conceivable to use a light pattern 3 that deviates from a checkerboard pattern 9 to illuminate the road markings 1, for example, in the form of bars extending across the roadway 13, which are alternately brightly and dimly illuminated.However, a reliable alternating illumination of the lane markings 1 can be achieved using a checkerboard pattern 9, especially if the lane markings 1 do not extend parallel to the lane 13. Lane markings 1 running perpendicular to the lane 13 can also be illuminated alternately in this way. The ego vehicle 5 can have one or more projectors 2. Preferably, the vehicle's headlights 10 are used for this purpose, although the provision of a separate projector 2 is also conceivable.

Claims

[1] Method for classifying road markings (1), wherein by means of at least one projector (2) a light pattern (3) consisting of at least two polygonal surfaces (4.1, 4.2) illuminated to different degrees is projected at least temporarily onto a road section (6) in front of an ego vehicle (5), and at least one camera image generated by at least one vehicle camera (7), which comprises at least one road section (6) partially covered by the light pattern (3), is evaluated on a computing unit (8) for detecting road markings (1), characterized bythat at least one road marking (1) detected in a camera image and at least partially covered by the two polygonal surfaces (4.1, 4.2) illuminated to different degrees is classified as a yellow road marking (1) if the road marking (1) has a substantially yellow color in both polygonal surfaces (4.1, 4.2) and the road marking (1) is classified as a white road marking (1) if the road marking (1) has a different color in the polygonal surfaces (4.1, 4.2) illuminated to different degrees. [2] Method according to claim 1, characterized by that the projector (2) emits cold white light, in particular light with a color temperature of at least 6500 K. [3] Method according to claim 1 or 2, characterized by that the projector (2) emits a light pattern (3) in the form of a checkerboard pattern (9). [4] Method according to one of claims 1 to 3, characterized by that at least one vehicle headlight (10), in particular a matrix headlight, is used as the projector (2), wherein the vehicle headlight (10) preferably uses LEDs as light sources. [5] Method according to one of claims 1 to 4, characterized by that a multifunctional camera is used as the vehicle camera (7). [6] Method according to one of claims 1 to 5, characterized by that the light pattern (3) is emitted for such a short time that it cannot be perceived by a person, in particular the projector (2) is operated in a pulsed mode, wherein a projection of the light pattern (3) takes place at a frequency in the order of 20 Hz. [7] Method according to one of claims 1 to 6, characterized bythat a camera image evaluation is carried out using a flank filter in order to amplify intensity differences between two adjacent white (11.1) and yellow road marking sections (11.2). [8] Method according to one of claims 1 to 7, characterized by that the comparatively less illuminated polygonal areas (4.2) are not illuminated. [9] Vehicle (12) with at least one projector (2), in particular in the form of a matrix headlight, a computing unit (8) and a camera (7), characterized by that the computing unit (8) is configured to carry out a method according to one of claims 1 to 8. [10] Vehicle (12) according to claim 9, characterized by at least partially automated control.

Images