Identification of lane cover on lane

Abstract

The invention relates to a method, in particular a computer-implemented method, for detecting a lane covering (F) on a lane by means of an on-board camera system (1) of a vehicle, comprising the following steps: providing a first image (I1) of the surroundings of the vehicle captured by the vehicle camera system at a first exposure time (b1), providing a second image (I2) of the vehicle surroundings captured with a second exposure time (b2) that is longer than the first exposure time (b1), and determining, at least on the basis of the second image (b2), an indication as to whether the lane covering (F) is present. The invention also relates to a computer program for carrying out the method according to the invention, and to a computer-readable storage medium for storing the computer program according to the invention.

Description

Technical Field

[0001] The present invention relates to a method for identifying a lane covering on a lane, in particular to a computer-implemented method, a computer program for implementing the method according to the present invention, and a computer-readable storage medium. Background Art

[0002] Driver assistance systems (ADAS) are used to support the driver of a vehicle. On the one hand, the corresponding ADAS functions are used to support the driver, while the driver still controls the driving of the vehicle. On the other hand, with a higher degree of automation, fully automated driving can also be achieved.

[0003] Camera-based driver assistance systems (ADAS) detect the vehicle's surroundings with the aid of a camera system comprising at least one camera. In this context, monocular cameras, in particular front cameras, stereo cameras or so-called surround view camera systems are known, which can detect the entire vehicle's surroundings.

[0004] When driving a vehicle, whether manually or automatically, one must always take into account the road conditions and the coefficient of friction between the tires and the road and adapt the respective driving style to the prevailing conditions. The coefficient of friction has a decisive influence on the vehicle's reaction characteristics, for example during braking.

[0005] Thus, DE 10 2004 018 088 A1 discloses a lane recognition system with a temperature sensor, an ultrasonic sensor and a camera device. The measurement data detected by the existing sensors are compared with reference data and, based on the comparison result, the lane surface condition is determined by classifying the lane surface (e.g. concrete, asphalt, dirt, grass, sand or gravel) and its condition (e.g. dry, icy, snowy, wet, etc.).

[0006] DE 10 2014 214 243 A1 discloses a method for determining road conditions, wherein road conditions are determined using road condition data of a weather map and / or a road map and are re-digitized.

[0007] WO 2012 / 110030 A2 describes the possibility of estimating the friction coefficient with the aid of a three-dimensional (3D) camera. Based on the image data of the camera, a height profile of the road surface can be established and the expected local friction coefficient can be estimated. Based on the specially determined height profile, the lane surface can be classified in a single case.

[0008] According to WO2013 / 117186A1, the height curves of the road surface transverse to the vehicle's driving direction along a plurality of lines are determined based on the image data of a three-dimensional (3D) camera device, and the lane condition is identified based on these curves. As an option, the two-dimensional (2D) image data of the monocular camera device can also be additionally evaluated, for example by means of a texture analysis or pattern analysis, and taken into account when identifying the characteristics of the lane surface.

[0009] EP3069296A1 proposes to use image processing to analyze and evaluate image data obtained by means of a camera system, and to specifically determine whether there are signs of a lane covering. The determined signs are then used to determine the lane covering and, if necessary, to detect the lane condition. Signs of the presence of a lane covering include, for example, the impact of rain on the lane or the vehicle or the window in the image data, or the impact of the lane covering when at least one tire of the vehicle passes by. Summary of the invention

[0010] The object underlying the present invention is to improve the detection capabilities of the presence of a roadway covering.

[0011] This object is achieved by a method according to claim 1, a computer program according to claim 14 and a computer-readable storage medium according to claim 15. Advantageous embodiments are the subject matter of the dependent claims.

[0012] In view of the method, the object underlying the present invention is achieved by a method, in particular a computer-implemented method, for detecting a lane covering on a lane by means of an onboard camera system of a vehicle, the method comprising the following method steps:

[0013] Providing a first image of the vehicle surroundings captured by the vehicle-mounted camera system with a first exposure time,

[0014] providing a second image of the vehicle surroundings having a second exposure time longer than the first exposure time, and

[0015] An indication of whether a roadway covering is present is determined at least based on the second image.

[0016] The first image is preferably an image taken during continuous operation of the onboard camera system. Typically, the exposure time of the onboard camera system is automatically controlled and appropriately selected in each case depending on the lighting conditions. The first image is therefore an image with a substantially optimal exposure time. In particular, a longer exposure time than the optimal exposure time selected for the second image leads to increased motion blur, which is generally to be avoided in the subsequent image analysis. However, such images with a longer exposure time can be used in a beneficial manner to detect the lane cover when the vehicle passes by.

[0017] The vehicle-mounted camera system includes one or more camera devices. For example, the camera device may also be a so-called panoramic camera system. At least one camera device may have a fisheye lens.

[0018] The camera system is preferably fixed to the vehicle in such a way that the image of the surroundings of at least one wheel of the vehicle can be captured by means of at least one camera device of the vehicle-mounted camera system. Therefore, the first image and / or the second image preferably at least partially shows the wheel of the vehicle and the area around the wheel, i.e., an image close to the wheel area.

[0019] When driving on a lane, any lane covering that may exist will be displaced by the tires, especially displaced forward and to the side. Such displaced lane covering will be detected in the form of an image by at least one camera device of the onboard camera system. At the same time, the displaced lane covering will be blurred in the scattering direction due to the relative motion with the moving vehicle and the longer exposure time of the second image. This is used to identify whether the lane covering exists.

[0020] According to one design solution, the lane covering is water, snow, ice, leaves or particles, in particular sand or dust. However, the lane covering can generally also be any medium / object (covering layer, covering surface) that is spread out in a planar manner on the road surface (asphalt, asphalt, concrete, etc.). At the same time, the planar spreading can be described as a covering layer or covering surface of a medium or object. The lane is not necessarily completely covered by the lane covering layer. Therefore, it is conceivable that there are different types of lane covering layers that fall within the scope of the present invention. Accordingly, the description of the presence of a lane covering layer can also be, for example, a description of the type of lane covering layer.

[0021] In the first embodiment of the method, information about the friction coefficient and / or the friction coefficient level of the vehicle on the lane covering is determined, in particular, as a function of information about the presence of a lane covering, preferably as a function of the type of lane covering. The information about the friction coefficient can be determined in different ways. The information about the friction coefficient is preferably determined as a function of information about the presence of a lane covering. Based on the friction coefficient, a driving strategy, such as a reaction behavior in an emergency situation, can be derived in an advantageous manner.

[0022] DE 10 2009 041 566 B4, for example, describes a method for determining a roadway friction coefficient according to a friction coefficient class as a function of a determined friction coefficient characteristic parameter.

[0023] In an advantageous embodiment of the method according to the invention, the roadway covering is water, wherein the water depth is determined. The water depth is directly related to the water quantity and can be determined, for example, from the determined water quantity, in particular the water quantity displaced by one or more tires per unit time. In addition to knowing the friction coefficient, knowledge of the water depth is also essential for developing a driving strategy that is adapted to the actual situation.

[0024] It would be very advantageous if a statement about the risk of aquaplaning could be determined based on the water depth, the vehicle speed and / or the hydroplaning behavior of at least one tire of the vehicle. Thus, a statement about the risk of aquaplaning can be made according to the method according to the invention.

[0025] Based on the size and / or intensity of the water droplets detected in the first image and / or the second image, the amount of water splashing detected and / or based on the amount of water clusters detected, a distinction can be made between a wet roadway, rain and a risk of aquaplaning in a useful manner. In the case of a risk of aquaplaning, i.e. a large amount of water on the road, water mist or water spray often forms as the driving speed increases. This can be used to make a statement about the risk of aquaplaning in a useful manner, in particular for identifying a serious risk of aquaplaning. For example, the presence of detected water droplets, the amount of water splashing detected, water clusters or water mist or water spray can be divided into predeterminable categories. In addition, the speed of the vehicle on the roadway should also be taken into account when evaluating the risk of aquaplaning.

[0026] According to a design of the method according to the invention, the second exposure time is selected as a function of the first exposure time determined by the exposure control / regulator. The vehicle thus has an exposure control / regulator, by means of which the exposure time of the onboard camera system can be determined in continuous operation. The exposure time can preferably be adjusted or controlled continuously and adapted to the respective lighting conditions in the vehicle's surroundings. Then, a second, longer exposure time is selected as a function of the current value of a first exposure time set by means of the exposure control / regulator.

[0027] It is advantageous to select the second exposure time as a function of the speed of the vehicle. In this way, a lateral movement of the displaced lane cover, in particular in the scattering direction, can be optimally observed.

[0028] In addition, it is also beneficial to select the second exposure time according to the brightness of the vehicle's surroundings. The second exposure time should be selected according to the current lighting conditions.

[0029] In a design of the method according to the invention, the second exposure time is increased gradually from the first exposure time, in particular in predeterminable intervals or steps, or by means of a predeterminable factor. The steps or intervals are preferably selected in such a way that the step height or the length of the interval varies, for example exponentially. However, the second exposure time can also be determined by a trade-off method. It is advantageous if the second exposure time is increased from the first exposure time until a predeterminable criterion is met. In this context, completely different criteria can be used, such as the visibility of specific elements in the image captured by the vehicle camera system.

[0030] According to a particularly preferred embodiment of the method according to the invention, the information about the presence of a roadway covering and / or in particular about the type of roadway covering is determined with the aid of methods from the field of machine learning.

[0031] In this regard, it is advantageous to use at least one neural network, in particular a trained neural network, to determine information about the presence of a lane covering and / or in particular about the type of lane covering, wherein the neural network is configured to determine and output information about the presence of a lane covering and / or in particular the type of lane covering at least based on the second image.

[0032] The information on the presence or absence of a lane covering can be specified in different ways. On the one hand, it can be specified whether there is any lane covering on the lane. On the other hand, it is also possible to determine the location of the lane covering or how many lane coverings there are. As an alternative or in addition, it is also possible to determine the type of lane covering, i.e. what type of lane covering is the respective one.

[0033] The neural network is preferably a convolutional neural network (CNN), a recurrent neural network (RNN) or a so-called region proposal network (RPN). When using a trained neural network for the indication of the presence of a lane cover and / or in particular the type of a lane cover, the network can be trained using image data of different types of scenes captured with the aid of an onboard camera system, for example manually annotated. However, suitable training data can also be generated at least partially synthetically. Furthermore, the neural network can be trained using suitable reference sensors in order to determine the indication of the presence of a lane cover and / or in particular the type of a lane cover with high reliability and accuracy and to predetermine training target values ​​with the aid of the relevant indications.

[0034] An alternative embodiment includes the use of at least one decision tree, in particular a random forest, to determine the presence of a lane covering and / or the information about the type of lane covering. Thus, the presence of a lane covering on the lane and / or in particular the type of lane covering on the lane can also be determined based on an evolutionary method.

[0035] The method according to the invention according to one of the embodiments described here is used in an advantageous manner to detect whether a roadway covering is present at night or in low or no light conditions. The method according to the invention is therefore particularly suitable in an advantageous manner for nighttime situations with insufficient or no light, for example for long-distance driving without ambient scattered light. The invention is based on the finding that when the method according to the invention is used at night or in low or no light conditions, the residual light from the vehicle headlights in combination with the second exposure time is sufficient to indicate whether a roadway covering is present.

[0036] Furthermore, the object underlying the invention is achieved by a data processing system which comprises means for carrying out the method according to the invention in accordance with one of the configurations described.

[0037] In addition, the task based on the present invention is solved by a computer program, which contains relevant instructions. When a computer executes the relevant program, the instructions enable the computer to implement the method according to the present invention according to one of the design schemes. The task is also solved by a computer-readable storage medium, on which the computer program according to the present invention is stored. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention and its advantageous design solutions are explained in detail with reference to the following drawings, wherein:

[0039] Figure 1 A flow chart illustrating the method according to the present invention is shown;

[0040] Figure 2 A flowchart showing a design scheme of the method according to the present invention using a machine learning method; and

[0041] Figure 3 Shown are recorded images of the wheel area of ​​a vehicle with different roadway coverings. DETAILED DESCRIPTION

[0042] Figure 1 The method according to the invention is shown. In a first step, a first image I1 captured by a vehicle-mounted camera system with a first exposure time b1 is provided. In addition, a second image I2 captured with a second exposure time b2 is also provided. As an option, the second exposure time b2 can be selected based on the first exposure time b1. This variant is Figure 1 The second exposure time b2 is then an arbitrary function of the first exposure time b1. When selecting the second exposure time, for example, the speed v of the vehicle and / or the brightness of the vehicle's surroundings, ie, the respective main lighting conditions, may also be taken into account.

[0043] According to the invention, the second exposure time b2 is longer than the first exposure time b1. In this way, information about whether there is a roadway covering F can be determined from the second image I2. In addition, information about the friction coefficient of the vehicle on the roadway and, if the roadway covering is water, the water depth and / or the risk of aquaplaning can also be determined.

[0044] Alternatively and indicated by dashed lines, the first exposure time can also be determined, in particular regulated or controlled, by an exposure control / regulator 2. In this case, the first exposure time b1 is automatically and continuously selected appropriately, in particular optimized with regard to the downstream image evaluation of the recorded image I.

[0045] Figure 2 is a diagram of an advantageous embodiment of the method according to the invention, in which a machine learning method is used to determine the information about whether a roadway covering F is present. In the embodiment shown, the second image I2 is used as input for a trained neural network NN. The neural network NN is configured to determine and output whether a roadway covering F is present based on at least the second image I2.

[0046] However, other machine learning methods such as decision trees may also be used according to the present invention.

[0047] In the presence of a lane covering F, when a vehicle is traveling on the lane, the lane covering is displaced by the tires, in particular displaced forward and laterally. The displaced lane covering F causes motion blur in the scattering direction, which is caused by the relative movement between the displaced lane covering and the traveling vehicle, and the longer exposure time b2 of the second image I2 used to identify whether the lane covering F is present. Due to the movement of the vehicle on the lane, and due to the relative movement of the displaced lane covering F and the reflection of scattered light on the displaced lane covering F, a characteristic pattern appears in the second image I2 with a longer exposure time b2 relative to the direction of vehicle movement. For example, different tones, shapes, sizes and directions relative to a predetermined axis of the relevant pattern can be distinguished. For example, an image blur that is clearly tilted forward or to the side in image I2 is entirely caused by the displaced lane covering F. Therefore, a description of whether the lane covering F is present can be inferred from the characteristic pattern generated thereby. According to Figure 3 This will be explained in detail.

[0048] Figure 3Shown are examples of four different camera images of a tire 3 and its immediate surroundings with four different roadway coverings F, which capture the vehicle area close to the wheel. The corresponding images can be captured, for example, with the aid of a panoramic camera. However, other types of camera systems are also conceivable and can be used within the scope of the invention.

[0049] Figure 3 a refers to the case where the road is slightly wet. Since the exposure time b2 of the second image I2 is longer, the tire 3 is surrounded by the first characteristic pattern M1. Figure 3 b shows a comparable image when the road is significantly wet. In this case, the resulting feature pattern M2 is similar to Figure 3 The first characteristic pattern M1 in a is obviously different. Figure 3 a and Figure 3 b clearly shows that the different levels of moisture in the lane can be reliably distinguished. Figure 3 b) The amount and intensity of the fluid displaced by the tire are significantly increased.

[0050] For example, to determine the risk of aquaplaning, different levels can be constructed for the respective different amounts of water displaced by the tires and the strength with which the water is displaced. In this way, different risk levels for the occurrence of aquaplaning can be distinguished, for example, by additionally taking into account the vehicle speed. However, it is also conceivable to evaluate the characteristic patterns M in the respective second images I2 with a longer exposure time b2 in other ways to determine the water depth and / or the risk of aquaplaning, which also belongs to the scope of the present invention.

[0051] at last, Figure 3 c shows the characteristic pattern M3 when there is sand on the road. Figure 3 d shows a characteristic pattern M4 when there is snow on the roadway. The invention is therefore not limited to the recognition of any roadway covering on the roadway. The type of roadway covering present in each case can also be reliably determined. One advantage of the invention is that the presence of a roadway covering, i.e. the roadway condition, and also the presence of a hydroplaning risk can be reliably determined using conventional camera devices, in particular economically priced camera devices, independently of external lighting conditions in the vehicle's surroundings, in particular at night or in the case of insufficient or no lighting in the vehicle's surroundings.

Claims

1. A method for detecting a lane cover (F) on a lane by means of a camera system (1) on board a vehicle, in particular a computer-implemented method, the method comprising the following steps: providing a first image (I1) of the vehicle surroundings captured by a vehicle-mounted camera system with a first exposure time (b1), providing a second image (I2) of the vehicle surroundings having a second exposure time (b2) longer than the first exposure time (b1), At least based on the second image (b2), a statement is determined as to whether a roadway covering (F) is present.

2. The method according to claim 1, It is characterized in that The roadway covering (F) is water, snow, ice, leaves or particles, especially sand or dust.

3. The method according to claim 1 or 2, It is characterized in that In particular, information about the coefficient of friction and / or the level of the coefficient of friction for a vehicle located on the roadway is determined based on information about whether a roadway covering is present, preferably based on the type of roadway covering.

4. The method according to at least one of the preceding claims, It is characterized in that The lane cover (F) is water, wherein the water depth is determined.

5. The method according to claim 4, It is characterized in that A statement about the risk of aquaplaning is determined as a function of the water depth, the vehicle speed and / or the hydroplaning behavior of at least one tire of the vehicle.

6. The method according to at least one of the preceding claims, It is characterized in that The second exposure time (b2) is determined according to the first exposure time (b i )choose.

7. The method according to at least one of the preceding claims, It is characterized in that The second exposure time (b2) is selected according to the speed (v) of the vehicle.

8. The method according to at least one of the preceding claims, It is characterized in that The second exposure time (b2) is selected according to the brightness of the vehicle's surroundings.

9. The method according to at least one of the preceding claims, It is characterized in that The second exposure time (b2) is increased stepwise from the first exposure time (b1), in particular in predeterminable intervals or steps, or by a predeterminable factor.

10. The method according to at least one of the preceding claims, It is characterized in that A statement about whether a roadway covering (F) is present and / or in particular a statement about the type of roadway covering is determined using methods from the field of machine learning.

11. The method according to claim 10, It is characterized in that Using at least one neural network, in particular a trained neural network, information about the presence of a lane covering (F) and / or in particular information about the type of lane covering is determined, wherein the neural network is configured to determine and output the presence of a lane covering (F) and / or in particular the type of lane covering at least based on the second image (I2).

12. The method according to claim 10, It is characterized in that A statement about whether a lane covering (F) is present and / or a statement about the type of lane covering is determined using at least one decision tree, in particular using a random forest.

13. Use of the method according to at least one of the preceding claims for detecting a roadway covering (F) at night or in low-light or no-light conditions.

14. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of claims 1 to 12.

15. A computer-readable storage medium on which the computer program according to claim 14 is stored.

Images