Recognition of road surface coverings

Abstract

Recognition of road surface coverings The present invention relates to a method, in particular a computer-implemented method, for recognizing road surface coverings (F) on a road using an on-board camera system (1) of a vehicle, characterized in that it comprises the following process steps: providing a first image (I1) of the vehicle's surroundings taken by the on-board camera system and having a first exposure time (b1), providing a second image (I2) of the vehicle's surroundings and having a second exposure time (b2) longer than the first exposure time (b1), and deriving an assessment of the presence or absence of a road surface covering (F) using at least the second image (b2). The present invention also relates to a computer program for implementing the method according to the invention, and to a computer-readable storage medium on which the computer program according to the invention is stored.

Description

[Technical Field]

[0001] The present invention relates to a method for recognizing road surface coverings on roads, particularly preferably a computer-implemented method, a computer program for carrying out the method according to the invention, and a computer-readable storage medium. [Background technology]

[0002] Advanced driver-assistance systems (ADAS) are systems that assist the driver of a vehicle. Primarily, these ADAS functions are used to assist the driver in situations where the driver retains control of the vehicle's driving. However, with a high degree of automation, fully automated driving is also possible.

[0003] Camera-based ADAS systems capture the surroundings of the vehicle using a camera system including at least one camera. In this context, mono cameras, particularly preferably front cameras, stereo cameras or so-called surround view camera systems, with which the entire periphery of the vehicle can be captured, are known.

[0004] When driving a vehicle, whether manually or automatically, it is always necessary to take into account the road conditions, i.e. the coefficient of friction between the tires and the road surface in that condition, and to adapt the driving style accordingly, as the coefficient of friction has a decisive influence on the vehicle's reaction characteristics, for example, during braking.

[0005] For example, DE 10 2004 018 088 A1 discloses a road recognition system that includes a temperature sensor, an ultrasonic sensor, and a camera. Measurement data acquired by the installed sensors are compared with reference data, and based on this comparison, the state of the lane surface is determined by classifying the lane surface (e.g., concrete, asphalt, dirt, glass, sand, or gravel) and its condition (e.g., dry, icy, snowy, wet).

[0006] DE102014214243A1 discloses a method for identifying road conditions, characterized in that road condition data on weather maps and / or road maps are referenced during road condition identification and are re-digitized.

[0007] WO2012 / 110030A2 describes a method for estimating the coefficient of friction using a 3D camera. From the camera image data, a height profile of the road surface is created and a probable local coefficient of friction is estimated. Using the specially determined height profile, a classification of the road surface is performed in each individual case.

[0008] According to WO2013 / 117186A1, image data from a 3D camera is used to measure height profiles of the road surface on multiple lines transverse to the direction of travel of the vehicle, and the nature of the road surface is determined based on these profiles. Optionally, the 2D image data of the mono camera can be additionally analyzed and taken into account in recognizing the road surface condition, for example using texture or pattern analysis.

[0009] EP3069296A1 proposes a method for analyzing image data acquired from a camera system through image processing to identify indirect evidence indicating the presence of road surface coverings as targets. The identified indirect evidence is referenced when identifying road surface coverings and, if necessary, when identifying road surface conditions. Examples of indirect evidence indicating the presence of road surface coverings include the effects of precipitation on the road surface, vehicle, or vehicle glass in the image data, or the effects of at least one tire of a vehicle passing over the road surface coverings. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] DE102004018088A1 [Patent Document 2] DE102014214243A1 [Patent Document 3] WO2012 / 110030A2 [Patent Document 4] WO2013 / 117186A1 [Patent Document 5] EP3069296A1 [Patent Document 6] DE102009041566B4 Summary of the Invention [Problem to be solved by the invention]

[0011] Therefore, the problem to be solved by the present invention is to improve the detection capability for the presence or absence of road surface coverings. [Means for solving the problem]

[0012] This problem is solved by a method according to claim 1, a computer program according to claim 14 and a computer-readable storage medium according to claim 15. Advantageous embodiments are the subject of the dependent patent claims.

[0013] Regarding the method, the problem underlying the present invention is solved by a method, in particular a computer-implemented method, for recognizing road surface coverings on a road using an on-board camera system of a vehicle, comprising the following process steps: providing a first image of the vehicle surroundings captured by an on-board camera system, the first image having a first exposure time; providing a second image having a second exposure time longer than the first exposure time of the vehicle surroundings; and deriving an assessment of the presence or absence of a road surface covering using at least the second image;

[0014] The first image is preferably an image taken during continuous operation of the vehicle-mounted camera system. Typically, the exposure time of the vehicle-mounted camera system is automatically controlled and selected appropriately depending on the lighting conditions. In other words, the "first image" referred to here is essentially an image taken with an optimal exposure time. A longer exposure time, especially compared to the optimal exposure time, selected for the second image typically results in more noticeable motion blur, which should be avoided for subsequent image evaluation. However, such images with a longer exposure time can be advantageously used to recognize road surface coverings when driving over them.

[0015] The vehicle camera system includes one or more cameras, which may be, for example, a so-called surround view camera system, at least one of which may have a fisheye lens.

[0016] The camera system is preferably fixed to the vehicle so that at least one camera of the on-board camera system can capture at least one wheel of the vehicle, and therefore the first image and / or the second image are preferably images that at least partially show the wheel of the vehicle and the surrounding area of ​​the wheel, i.e., the surrounding area of ​​the wheel.

[0017] When driving on a road, any road surface coverings that may be present are thrown away by the tires, particularly in the front and side directions. The thrown-away road surface coverings are captured as images by at least one camera of the vehicle-mounted camera system. The thrown-away road surface coverings cause motion blur in the direction of the thrown-away road surface coverings due to their relative movement with respect to the moving vehicle and the long exposure time of the second image. This can also be used to recognize the presence or absence of road surface coverings.

[0018] In one embodiment, the road surface covering refers to water, snow, ice, fallen leaves, or particles such as sand and dust. However, in a broader sense, the road surface covering can also refer to any medium / object, such as a planar covering (such as a pavement cover or a carpet-like covering) on ​​a road surface (asphalt, tar, concrete, etc.). Such planar coverings can also be described as covers or carpets made of the medium or object. However, the road surface does not necessarily need to be completely covered by the road surface covering. Therefore, various types of road surface coverings are conceivable, all of which are within the scope of the present invention. The evaluation of the presence or absence of a road surface covering may, for example, be an evaluation of the type of road surface covering.

[0019] In one embodiment of the method, an assessment of the coefficient of friction and / or the coefficient of friction class for the road surface on which the vehicle is traveling is determined, in particular based on an assessment of the presence or absence of road coverings, preferably a road covering type. The assessment of the coefficient of friction can be determined using various methods or means, preferably based on an assessment of the presence or absence of certain road coverings. From the coefficient of friction, a driving strategy can then be preferably derived, e.g., with regard to reaction characteristics in an emergency.

[0020] German patent DE 102009041566 B4 describes, for example, a method for determining the road surface friction coefficient for each friction coefficient class on the basis of determined friction coefficient characteristic values.

[0021] In an advantageous embodiment of the method according to the invention, the road surface covering is water and its depth is determined. The water depth is directly related to the water volume and can be determined, for example, from the determined, particularly preferably, water volume displaced by one or more tires per time unit. In addition to information about the friction coefficient, information about the water depth is also important for determining a driving strategy depending on the situation.

[0022] It is advantageous to obtain an assessment of the risk of aquaplaning based on the water depth, the speed of the vehicle and / or the slip behaviour of at least one tire of the vehicle, i.e. the method according to the invention allows to carry out an assessment of the risk of aquaplaning.

[0023] It is advantageous to distinguish between a wet road surface, precipitation and aquaplaning risk based on the size and / or strength of the detected water droplets, the detected amount of water splash and / or the detected water clusters in the first image and / or the second image.

[0024] As the driving speed increases, if there is an aquaplaning risk, i.e. if there is a large amount of water on the road, splashes and plumes of water often occur. This can be used advantageously when making an aquaplaning risk assessment, and in particular when recognizing the current aquaplaning risk. For example, the presence of detected water droplets, detected water droplet volumes, water clusters, splashes or plumes can be classified into predefined classes. In addition to the above, the vehicle speed can and should also be taken into account when making a judgment about the aquaplaning risk.

[0025] According to one embodiment of the method of the present invention, the second exposure time is selected depending on the first exposure time determined by the lighting control / feedback control means, i.e., the vehicle is equipped with lighting control / feedback control means that operates continuously and determines the exposure time of the vehicle-mounted camera system. The exposure time is continuously feedback-controlled or controlled and adapted to the respective lighting conditions around the vehicle. A longer second exposure time is then selected based on the current value of the first exposure time set using the lighting control / feedback control means.

[0026] It is advantageous if the second exposure time is selected depending on the speed of the vehicle, in order to optimally visualize the rejected road surface covering, in particular its lateral movement in the direction of the rejection.

[0027] Furthermore, it is also advantageous if the second exposure time is selected depending on the brightness of the surroundings of the vehicle, i.e. the second exposure time is selected depending on the current lighting conditions.

[0028] In one embodiment of the method according to the invention, the second exposure time is gradually increased, particularly preferably by predefinable intervals or steps or by a predefinable factor, based on the first exposure time. It is preferable to select the steps or intervals so that the step height or the interval length varies, for example, exponentially. However, the second exposure time can also be determined using a weighing method. It is advantageous to increase the second exposure time starting from the first exposure time until a predefinable condition is met. In this context, various conditions can be taken into account, such as the visibility of certain elements in the image captured by the vehicle camera system or similar conditions.

[0029] According to a particularly preferred embodiment of the method according to the invention, the assessment of the presence or absence of a road surface covering and / or in particular the type of road surface covering is determined using methods belonging to the field of machine learning.

[0030] In this sense, it is also advantageous if the assessment of the presence or absence of road surface coverings and / or in particular the type of road surface covering is carried out using at least one neural network, particularly preferably a trained neural network, which is configured to be able to determine and output the presence or absence of road surface coverings and / or in particular the type of road surface covering from at least the second image.

[0031] The assessment of the presence or absence of road surface coverings may be, for example, an assessment of road surface covering types. First, an assessment of the presence or absence of road surface coverings can be performed. However, it is also possible to determine where and how many road surface coverings are present. Alternatively or additionally, it is possible to determine which road surface covering types, i.e., the types of road surface coverings, respectively.

[0032] Preferably, the neural network is a convolutional neural network (CNN), a recurrent neural network (RNN) or a so-called region proposal network (RPN). Furthermore, if the trained neural network is to be used for assessing the presence or absence of road coverings, in particular the type of road covering, it is possible to train the neural network using not only image data from images captured by an on-board camera system, but also, for example, manually labeled image data from various scenarios. However, it is also possible to generate suitable training data at least in part statically. Furthermore, it is also possible to use suitable reference sensors for training the neural network, which are capable of reliably and precisely assessing the presence or absence of road coverings and / or in particular the type of road covering, and thus setting training target values.

[0033] An alternative embodiment involves determining the presence or absence of a road surface covering and / or in particular the type of road surface covering by means of at least one decision tree, particularly preferably a randomized decision forest, i.e. the presence or absence of a road surface covering and / or in particular the type of road surface covering can be determined by evolutionary methods.

[0034] The method according to the present invention, in one of its forms described herein, can be advantageously used to recognize the presence or absence of road surface coverings at night or when there is little or no lighting. That is, the method according to the present invention can be used particularly advantageously in nighttime situations with little or no lighting, for example, during suburban driving where there is no external scattered light. The present invention is based on the finding that when using the method according to the present invention at night or when there is little or no lighting, the afterglow of the vehicle lights in combination with the second exposure time is sufficient to make an assessment of the presence or absence of road surface coverings.

[0035] Furthermore, the problem on which the present invention is based is solved by a system for processing data, which system includes means for implementing the method according to the present invention in one of the above-mentioned forms.

[0036] Furthermore, the problem underlying the present invention is also solved by a computer program comprising instructions for implementing one of the methods of the above aspects when the program is executed by a computer, as well as by a computer-readable storage medium on which the computer program according to the present invention is stored.

[0037] The invention as well as advantageous embodiments are explained in more detail using the following figures: [Brief explanation of the drawings]

[0038] [Figure 1] FIG. 1 is a flow chart illustrating the method according to the present invention; [Figure 2] FIG. 2 is a flow chart for an embodiment of the method according to the present invention using machine learning methods; and [Figure 3] Figure 3 shows photographs of the wheel area of ​​a vehicle in the presence of various road coverings. DETAILED DESCRIPTION OF THE INVENTION

[0039] FIG. 1 illustrates a method according to the present invention. In a first step, a first image I1 is provided by an on-board camera system with a first exposure time b1. Subsequently, a second image I2 is provided with a second exposure time b2. Optionally, the second exposure time b2 can be selected depending on the first exposure time b1. This variation is indicated by the dotted line in FIG. 1. In this case, the second exposure time b2 is an arbitrary function of the first exposure time b1. The second exposure time can be selected depending on, for example, the vehicle speed v and / or the brightness of the vehicle's surroundings, i.e., the illumination conditions at that time.

[0040] In the present invention, the second exposure time b2 is longer than the first exposure time b1. In this way, the second image I2 can be used to determine an assessment of the presence or absence of a road surface covering F. Furthermore, it is also possible to determine an assessment of the coefficient of friction of the vehicle on the road surface, or, if the road surface covering is water, the water depth and / or the aquaplaning risk.

[0041] Further optionally, therefore indicated by a dotted line, the first exposure time b1 can be determined, particularly preferably feedback-controlled or controlled, by the illumination control / feedback control means 2. In such a case, the first exposure time b1 is continuously and automatically selected accordingly, particularly preferably optimized for subsequent image processing during the capture of the image I.

[0042] 2 shows an advantageous embodiment of the method according to the invention, in which a machine learning method is used to determine the assessment of the presence or absence of a road surface covering F. In the embodiment shown, a second image I2 is provided as input to a trained neural network NN, which is configured to use at least the second image I2 to determine and output the presence or absence of a road surface covering F.

[0043] However, other methods of machine learning, such as decision trees, can also be used in the present invention.

[0044] In the case where road surface coverings F are present, they are rejected by the tires, particularly in the forward and lateral directions, as the vehicle travels on the road. The rejected road surface coverings F generate motion blur, which can be applied to recognizing the presence or absence of the rejected road surface coverings F in the direction from which they are rejected, due to their relative movement with respect to the traveling vehicle and the long exposure time b2 for the second image I2. Due to the vehicle's movement on the road, the relative movement of the rejected road surface coverings F, and the reflection of scattered light from the rejected road surface coverings F, characteristic patterns are observed in the second image I2, which has a long exposure time b2, in the direction of travel of the vehicle. For example, it is possible to distinguish various colors, shapes, sizes, and directions relative to a predetermined pattern axis. For example, significant image blurring in the forward or lateral directions in image I2 can only be caused by the rejected road surface coverings F. Therefore, by using the characteristic pattern thus obtained, it is possible to determine the presence or absence of the road surface covering F. This will be explained in detail using FIG.

[0045] 3 shows four different images of the vehicle wheel area, i.e., the tire 3 and its surroundings, for four different road surface coverings F. Such images can be taken, for example, using a surround view camera, although the use of other types of camera systems is also conceivable and possible within the scope of the present invention.

[0046] Figure 3a shows the case of a slightly wet road surface. The tire 3 is surrounded by a first characteristic pattern M1, which is made visible by a long exposure time b2 in the second image I2. Figure 3b, on the other hand, shows the image seen in the case of a significantly wet road surface. The characteristic pattern M2 obtained in this case is significantly different from the first characteristic pattern M1 of Figure 3a. Comparing Figures 3a and 3b, it is clear that different degrees of wetness of the road surface can be reliably distinguished. In the case of high wetness (Fig. 3b), not only the amount of liquid splashed by the tire but also its intensity is significantly greater.

[0047] To determine the aquaplaning risk, different classes can be formed, for example, for different amounts and intensities of water splashing generated by the tires. In this way, it is possible to distinguish between different risk classes for the occurrence of aquaplaning, for example, by additionally taking into account the vehicle speed. However, it is also conceivable to use different types of evaluation methods to determine the water depth and / or aquaplaning risk from the characteristic pattern M in the second image I2 with a long exposure time b2, which is also within the scope of the present invention.

[0048] Finally, Fig. 3c depicts a pattern M3 characteristic of the presence of sand on the road surface, and Fig. 3d depicts a pattern M4 characteristic of the presence of snow on the road surface. Thus, the present invention is not limited to the recognition of a specific road surface covering. Rather, it is a method that can reliably determine the type of road surface covering present. The advantage of the present invention is that it can reliably determine the presence or absence of road coverings, i.e., the road surface condition, and even the risk of aquaplaning, regardless of the external lighting conditions around the vehicle's wheels, i.e., especially at night or when there is little or no lighting, using conventional, especially inexpensive, cameras.

Claims

1. A method, in particular a computer-implemented method, for recognizing road surface coverings (F) on a road using a vehicle's on-board camera system (1), characterized in that it comprises the following process steps: The first exposure time (b １ ) a first image (I １ ) providing First exposure time around the vehicle (b １ ) longer than the second exposure time (b ２ ) with a second image (I ２ ) providing a At least the second image (b ２ ) to derive an assessment of the presence or absence of road coverings (F).

2. The road surface covering (F) is water, snow, ice, fallen leaves, or particles of sand, dust, etc. The method of claim 1.

3. The evaluation of the coefficient of friction and / or the coefficient of friction class for the road surface on which the vehicle is traveling is determined, in particular based on the evaluation of the presence or absence of a road surface covering, preferably based on the evaluation of the type of road surface covering. The method according to claim 1 or 2.

4. The road surface covering (F) is water, and its depth is calculated.

10. A method according to any one of the preceding claims.

5. Obtain an assessment of the risk of aquaplaning based on water depth, vehicle speed, and / or the slip behavior of at least one tire of the vehicle.

5. The method of claim 4.

6. Second exposure time (b ２ ) is the first exposure time (b ｉ ) is selected depending on 10. A method according to any one of the preceding claims.

7. Second exposure time (b ２ ) is selected depending on the vehicle speed (v) 10. A method according to any one of the preceding claims.

8. Second exposure time (b ２ ) is selected depending on the brightness of the surrounding area of ​​the vehicle.

10. A method according to any one of the preceding claims.

9. Second exposure time (b ２ ) is the first exposure time (b １ ) and are extended successively, particularly preferably at predefinable intervals or steps or by a predefinable factor.

10. A method according to any one of the preceding claims.

10. The presence or absence of road coverings (F) and / or in particular the type of road covering is assessed using methods belonging to the field of machine learning.

10. A method according to any one of the preceding claims.

11. the assessment of the presence or absence of a road surface covering (F) and / or in particular the type of road surface covering is carried out using at least one neural network, particularly preferably a trained neural network, The neural network receives at least a second image (I ２ ) and is configured to be able to determine and output the presence or absence of road surface coverings (F) and / or in particular the type of road surface coverings (F). The method of claim 10, wherein:

12. The presence or absence of a road surface covering (F) and / or in particular the type of road surface covering is assessed using at least one decision tree, particularly preferably a random decision forest.

11. The method of claim 10.

13. Use of a method according to any one of the preceding claims for recognizing the presence or absence of road coverings (F) at night or in conditions of low or no lighting.

14. A computer program comprising instructions which, when executed, cause a computer to carry out the method according to claims 1 to 12.

15. A computer-readable storage medium having stored thereon the computer program according to claim 14.

Images