Method and device for detecting and evaluating lane conditions and weather-related environmental influences

Abstract

The invention relates to a method for classifying a lane condition based on image data of an on-board camera system and a corresponding on-board camera system. The method comprises the following steps: providing (S12) image data by a vehicle-mounted camera system, the vehicle-mounted camera system being configured to image at least a portion of a vehicle's surroundings, wherein at least a portion of the image portion includes a lane on which the vehicle is traveling; distinguishing (S14) diffuse reflection and specular reflection of the lane by analyzing and evaluating the difference in appearance images of at least one point of the lane in at least two images of the camera system; wherein the images are taken from different shooting angles; determining (S16) whether there is interference caused by a wheel of the vehicle lifting up lane debris when the vehicle runs over the lane in at least one of the images of the camera system; and classifying (S18) the lane condition into one of the following five lane condition levels, taking into account the relevant reflection type and interference intensity results: a) dry lane: diffuse reflection type without interference; b) ordinary wet lane: specular reflection type with interference; c) extremely wet lane with risk of hydroplaning: specular reflection type with a lot of interference; d) snowy lane: diffuse reflection type with interference; e) icy lane (thin ice layer): specular reflection type without interference.

Description

Technical Field

[0001] The invention relates to a method for classifying a lane condition based on image data of an on-board camera system and a corresponding on-board camera system. Background Art

[0002] Technological advances in the field of optical image detection have made it possible to use camera-based driver assistance systems that are installed at various locations in the vehicle and detect the vehicle's surroundings in a way that is visually perceived by the driver. At the same time, the range of functions of such systems ranges from automatic high beam control, recognition and display of speed limits to warnings of lane keeping errors or dangerous collision situations.

[0003] Cameras are used in modern vehicles today for a wide variety of applications, from pure front-end vehicle detection to full 360-degree panoramic views in the case of so-called surround view camera systems, and are widely used in various functions of driver assistance systems. The main tasks of digital image processing are firstly the recognition of objects, then their classification and tracking in the image field, either as a stand-alone function or in combination with radar or lidar sensors. Classic objects usually include different vehicles such as cars, trucks, two-wheelers or pedestrians. In addition, cameras also take on the task of detecting information about traffic signs, lane markings, crash barriers, spaces or other common objects.

[0004] Automatically learning and identifying target categories and their hierarchies is the most important task in digital image processing. Many methods are known in this regard. Based on the current very advanced processing methods, they can almost complete this task as well as humans. Therefore, the current research focus has shifted from the approximate location of the target to the precise location of the target.

[0005] In the field of modern driver assistance, various sensors, including video cameras, are used to detect information about the vehicle's surroundings as accurately and reliably as possible. This surrounding information is combined with information about the vehicle's driving dynamics (e.g. from inertial sensor systems) to provide a good impression of the vehicle's current driving state and the overall driving situation. From this, the severity of the driving state can be inferred and the corresponding driver information / warnings can be initiated, up to and including driving dynamics interventions through braking and steering.

[0006] However, since the available friction coefficients or road conditions for driver assistance systems have not been available or known so far, the warning and intervention times are basically designed based on a dry road with a high friction coefficient between the tire and the road. The friction coefficient, also known as friction value, adhesion coefficient, coefficient of (static) friction or coefficient of friction, indicates the maximum force that can be transmitted between a road surface and a vehicle tire (e.g. in the tangential direction) relative to the wheel load and is therefore an important indicator of driving safety. In addition to the road conditions, the characteristics of the tire are also required to fully determine the friction coefficient.

[0007] This poses a real challenge to driver assistance systems and automated driving. Systems for accident prevention or accident damage reduction warn the driver too late or intervene in the system, which can only avoid an accident or reduce the consequences of an accident to an acceptable level on a truly dry road. If the road conditions are less than ideal, such as wet, snowy or even icy, the system cannot prevent the accident from happening, nor can it achieve a satisfactory effect in reducing the severity of the accident consequences.

[0008] If the system can determine the lane conditions and even the applicable friction coefficient, driver warnings and system interventions can be more targeted accordingly. This will significantly increase the effectiveness of driver assistance systems for accident prevention. Monitoring lane conditions or estimating friction coefficients is a basic prerequisite for automated driving systems.

[0009] For example, WO 2014 / 127777 A2 shows the determination of lane conditions based on images using an onboard camera device. Here, an image region including a lane surface image is fed to a classification device. Summary of the invention

[0010] It is an object of the present invention to provide an improved classification of the road situation based on a camera device and, if applicable, to account for weather-related environmental influences that occur.

[0011] The starting point of the present solution is the recognition that the characteristics of a specific lane situation can be distinguished based on diffuse and specular reflections on the lane surface in the camera image data. Such a method is described in WO 2016 / 177371A1.

[0012] In particular, the combination of different camera devices and camera systems on the vehicle with different viewing angles and viewing directions can firstly use digital image processing and machine learning devices to distinguish diffuse and specular reflections on the road surface in an image or a local area of ​​an image or a sequence of images. For example, due to the movement of the vehicle, the viewing angles adopted by each camera device at a spatial point at different points in time are different, thereby providing different viewing angles. As another example, two camera devices with partially overlapping detection areas already provide different viewing angles at a spatial point in the overlapping area at a time point.

[0013] A second basic feature for the surface condition of a lane is to determine disturbances in an image or a partial area of ​​an image or a sequence of images, which are caused by accumulations in the lane (such as water or snow) being kicked up when the wheels of the own vehicle or the wheels of other vehicles drive over or run over. The accumulations in the lane can be analyzed and evaluated based on their impact in the image. Particles of accumulations in the lane kicked up by the wheels will cause disturbances to varying degrees in the image or a partial area of ​​an image or a sequence of images. Such a method is described in WO 2015 / 070861A1.

[0014] These two characteristics can now be combined and jointly assigned to a precisely defined lane situation. This allows the following five relevant lane situations to be differentiated based solely on the camera image:

[0015] - Dry driveway

[0016] - Wet driveway with a small amount of standing water

[0017] - Wet driveway with a lot of standing water (hydroplaning hazard)

[0018] - Snowy driveway

[0019] -Icy driveway.

[0020] The distinction is made based on the following simple criteria:

[0021] Dry driveway: diffuse reflections without interference from wheels;

[0022] Wet driveway with a little standing water: Specular reflections with a little interference;

[0023] Wet lane with lots of water (hydroplaning hazard): lots of interfering mirror reflections;

[0024] Snowy driveway: diffuse reflections with interference; and

[0025] Icy driveway: undisturbed mirror reflections.

[0026] Disturbances caused by splashing water on a wet road can preferably be detected by the fact that the interfering image or image sub-region or image sequence differs in brightness and / or contrast and / or color and also in structure from a non-disturbing image or non-disturbing image sub-region or non-disturbing image sequence.

[0027] On the other hand, in the case of snow, there are no differences in brightness and / or contrast and / or color. The raised snow vortex only creates differences in structure.

[0028] Dry road surfaces are characterized in particular by an undisturbed diffuse reflection.

[0029] Icy driveways (thin layers of ice / "black ice") typically show an undisturbed specular reflection.

[0030] In principle, the method can be applied to a single camera device, as long as the camera device is suitable for detecting vortices raised by the wheels of the own vehicle or other vehicles. Thus, the camera system can consist of a single camera device, as long as the camera device has a suitable detection area in which interference (if any) can be detected and the type of reflection can be determined.

[0031] This is preferably suitable for a (monocular) vehicle-mounted front camera device, which at least partially detects the lane ahead and can recognize lane accumulation if a vehicle ahead, a vehicle crossing laterally or a vehicle coming from the opposite direction creates, for example, splashes of water or swirls of snow and dust when its tires pass over lane accumulation.

[0032] It is particularly preferred that the image data is generated by a laterally mounted vehicle camera device, for example a camera device facing towards a wheel of the own vehicle, towards a side of the own vehicle or towards the area around a wheel. In other words, at least one wheel or a wheel cover covering the wheel or one side of the vehicle is in the field of view (field of view - FOV for short) of the side camera device. This applies, for example, to a first side camera device of a panoramic surround camera system. The image area to be evaluated (region of interest, ROI for short) preferably relates to the space around the front wheel on the side where the most severe splashing water or blowing snow effects occur. As an alternative or additional measure, the space around the rear wheel can be used as the (second) region of interest (ROI).

[0033] In other words, image disturbances are determined in at least one region of special interest (ROI), which are caused by road debris being raised near the observed wheel due to the wheel running over the road, thereby causing disturbances in the at least one region of special interest (ROI).

[0034] According to another preferred embodiment, a second camera device arranged on the side of the vehicle generates second image data on the opposite side of the vehicle in a comparable manner. Thus, for example, two front wheels and / or two rear wheels can be "monitored".

[0035] The vehicle-mounted camera system advantageously comprises a rear camera device for detecting the rear area of ​​the vehicle.

[0036] The image data of different individual camera devices of an onboard camera system are preferably provided in different viewing directions and / or viewing angles, thereby making it possible to more reliably distinguish between types of reflections of the roadway.

[0037] The interference intensity can also be reliably determined.

[0038] A surround view camera system with at least or exactly four wide-angle (field of view of up to 180 degrees) camera devices can detect the front area, left area, right area and rear area relative to the vehicle and its surroundings, which will provide ideal possibilities for stable detection of reflection types and interferences in a beneficial way.

[0039] The video images are preferably divided into a plurality of different regions of interest (ROIs), which are used in a targeted manner in different ways depending on their content for evaluating reflection types and disturbances.

[0040] Individual camera devices are preferably used in different ways to specifically evaluate the reflection type and interference. For example, a camera device arranged on the side can be used to evaluate interference in the wheel area, while a front camera device or a rear camera device can be used to determine the reflection type.

[0041] A particular advantage of a preferred combination of different camera devices and image regions is that disturbances caused by wheel-generated vortices in the road surface can be searched for in a very targeted manner in different images and / or regions.

[0042] Such a combination makes the overall results significantly more reliable and increases the usability of the information.

[0043] Another advantageous feature is combining one or more front cameras with the cameras of a surround view camera system, which can provide a better resolution (and less distortion) of the forward (further away) surrounding area than a satellite camera of a surround view camera system facing forward.

[0044] Driving dynamics data and / or other sensor data and / or surroundings information and / or weather information from other non-camera-based surroundings sensors of a vehicle can preferably be used as additional decision bases for evaluating or classifying lane conditions and surrounding influences related to weather conditions.

[0045] In order to distinguish lane conditions, a first machine learning method is preferably trained based on a first training sample. The first machine learning method comprises the following steps:

[0046] - Detect and / or classify reflection type; and

[0047] - Detection and / or regression of disturbance strength.

[0048] According to a preferred implementation method, a mapping from reflection type and interference intensity to lane condition level / category is learned based on a second training sample with the aid of a second machine learning method.

[0049] Image features of at least one region are preferably extracted from at least one camera image, optionally converted into an intermediate representation and mapped onto an interference intensity by means of a regression device or onto a reflection type and / or a lane condition by means of a classification device.

[0050] As an alternative, a neural network is preferably trained which learns to map one or more image parts from one or more camera images onto the lane conditions based on a third random training sample. This end-to-end classification can be performed in particular as a deep learning method; the neural network can be a convolutional neural network (CNN).

[0051] The driving dynamics data and / or the sensor data are preferably used as additional input for a machine learning method or for training a neural network.

[0052] Furthermore, the present invention also relates to an on-board camera system comprising at least one camera device and an image data analysis and evaluation unit.

[0053] The camera device is configured (especially arranged and controlled) to image at least a portion of the vehicle's external environment, wherein at least a portion of the portion includes a lane on which the vehicle is traveling, and to provide image data to an image data analysis and evaluation unit. The image data analysis and evaluation unit is configured to distinguish between diffuse reflection and specular reflection of the lane by analyzing and evaluating the difference in appearance images of at least one point of the lane in at least two images taken by the camera system from different shooting angles.

[0054] Furthermore, the image data analysis and evaluation unit is configured to determine whether interference is present in at least one image of the camera system due to a vehicle's wheels lifting up road debris when the vehicle is moving over the road. Furthermore, the image data analysis and evaluation unit is configured to classify the road condition into one of the following five road condition classes, taking into account the reflection type and the interference intensity result:

[0055] a) Dry lane: Non-interfering diffuse reflection

[0056] b) Ordinary wet lane: Specular reflection type with interference

[0057] c) Extremely wet lanes with risk of hydroplaning: Specular type with lots of interference

[0058] d) Snowy driveway: diffuse reflection with interference

[0059] e) Icy lanes (thin ice layer): Specular reflection type without interference.

[0060] The on-board camera system is particularly designed to detect road conditions such as dry, wet, snowy and icy roads, using digital image processing and machine learning algorithms, and to reliably detect dangerous situations such as hydroplaning and distinguish them by cross-comparison of road reflection type and interference intensity.

[0061] The image data analysis and evaluation unit may in particular include a microcontroller or microprocessor, a digital signal processor (DSP), an ASIC (application-specific integrated circuit), an FPGA (field programmable gate array) and many other similar devices, as well as software for executing corresponding method steps.

[0062] Thus, the invention may be implemented in digital electronic circuitry, computer hardware, firmware, or software.

[0063] In addition, the vehicle-mounted camera system may further include an output unit for notifying a driver or controlling an automatically (semi-automatically or fully automatically) driven vehicle.

[0064] The calculation unit for analyzing and evaluating the friction coefficient can be integrated into the vehicle-mounted camera system or the image data analysis and evaluation unit.

[0065] Another subject of the present invention relates to a vehicle with an on-board camera system according to the present invention. The vehicle preferably comprises an actuator for intervening in the vehicle's driving dynamics, which is controlled by the on-board camera system as a function of the determined lane conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The following is a more detailed description of the embodiments and drawings.

[0067] Figure 1 shows different detection areas of a vehicle-mounted camera system with a plurality of individual camera devices;

[0068] Figure 2 Four individual images of a panoramic view camera system are shown, wherein the panoramic view camera system has four cameras with fisheye lenses

[0069] Figure 3 A schematic diagram showing a vehicle tire on a very wet roadway;

[0070] Figure 4 A binary camera image of a wheel of a vehicle traveling on a wet road on a rainy day is schematically shown;

[0071] Figure 5 Three exemplary image portions, captured by a camera device placed on the right side of the vehicle, are shown, respectively for one level of interference intensity in different wet road conditions;

[0072] Figure 6 A schematic diagram of a vehicle-mounted camera device is shown, and

[0073] Figure 7 A schematic diagram of method steps of a method for identifying and evaluating lane conditions is shown. DETAILED DESCRIPTION

[0074] Figure 1 The invention discloses a camera system for detecting areas (1a-1d, 2, 3) arranged in or on a first vehicle (E). A second vehicle (F) is located laterally in front of the first vehicle (E) in the direction of travel. The camera system of the first vehicle (E) comprises three different camera subsystems: a panoramic view system (1) comprising four individual camera sensors with wide-angle detection areas (1a to 1d) which together can detect the vehicle at 360 degrees; a front camera with a forward detection area (2) and a rear camera with a rearward detection area (3).

[0075] Interference in an image or a portion of an image can be detected or determined by a camera sensor having a forward detection area (1a, 2) in the direction of travel. When there is lane accumulation, the interference will be caused by a vehicle ahead (F) or an oncoming vehicle (not shown) passing through the lane, thereby determining whether there is lane accumulation on the lane.

[0076] The image processing for determining a roadway accumulation can advantageously be limited to one or more camera image regions having a tire contact patch.

[0077] By means of a camera sensor having a laterally directed detection area (1b, 1c), disturbances caused by roadway debris in the image can be determined when the tires of the host vehicle (E) pass over the roadway.

[0078] The advantage of this is that the currently relevant lane accumulation is determined independently of other vehicles (F) by detecting and determining the influence caused by the own vehicle (E) based on a sensor facing to the side, without having to rely on foreign vehicles (F) driving ahead or deviating from the lane. Since today's vehicles are increasingly equipped with 360-degree camera sensors that can perform surround view detection, which can, for example, display the surround view to the driver as a "top view" from a bird's eye view, a reliable determination of lane accumulation can be ensured in a beneficial way.

[0079] Figure 2 An example of four separate images of a panoramic surround view camera system with four cameras with fisheye lenses is shown. The images above (20a), left (20b), right (20c) and below (20d) correspond to Figure 1 The detection areas in the front (1a), left side (1b), right side (1c) and rear (1d) are shown. In particular, the images of the camera device arranged on the left and the camera device arranged on the right contain the front wheels and the rear wheels respectively. The corresponding image areas or regions of interest (ROI) should be selected and input to a classification device, which determines an image interference intensity in the area based on the roadway accumulation.

[0080] Figure 3 A schematic diagram shows a situation with a risk of hydroplaning. The contact between a tire 30 and a roadway 35 is decisive. The water 37 on the roadway 35, i.e. the roadway accumulation, is indicated by hatching. At the same time, the contact surface between the tire 30 and the roadway 35 can be divided into three areas 31 to 33. If the vehicle and thus the tire 30 moves in the driving direction indicated by the arrow, an approach area 31 is first formed on the tire, in which there is some wedge-shaped water between the tire 30 and the roadway 35. A transition area 32 is adjacent to the approach area 31. In the contact area 33 adjacent to the transition area 32, the tire 30 is in direct contact with the roadway 35, thereby achieving a corresponding adhesion effect. As the water level increases and / or the driving speed increases, the contact area 33 and the contact area between the tire 30 and the roadway 35 decreases, and if this area of ​​the contact area 33 approaches zero, there is a direct risk of hydroplaning. This situation is therefore called micro-hydroplaning. If there is no longer any contact zone 33, that is to say the entire area between tire 30 and roadway 35 is filled with water, hydroplaning occurs, in which state a vehicle can no longer be steered or braked in a controlled manner.

[0081] Figure 4A black and white (inverted) image of a camera image of a vehicle tire 30 when driving over a wet roadway on a rainy day is shown. The vehicle tire 30 is on a wheel rim (white, round) and the vehicle is driving to the left. When the vehicle drives over the water-covered roadway surface, the accumulated water is displaced and thrown out, whereby water droplets and water lines 40 (white dots and lines) can be seen. When driving over a wet road, splashes of water are formed from the vehicle tires. Due to the high surface pressure, the tires discharge the accumulated water on the roadway from all sides. This phenomenon is particularly evident in trucks, whose tires have a relatively high surface pressure and must discharge more water than cars. When viewed in the direction of travel, splashes of water appear mainly behind and to the sides of the wheels. There, the splashes of water can be detected by an on-board camera device and recognized as splashes by an image processing device, whereby a layer of rainwater can be inferred as accumulation on the roadway.

[0082] If there is slush on the roadway, this forms splashing slush in a comparable manner and can also be detected and identified in the same way.

[0083] Depending on the road surface, for example if the road is damp rather than wet, and depending on the structure of the road surface, no splashing occurs when the vehicle passes over it, but only spray or water spray or both. When driving over roads covered with fresh snow or powder snow, typical snow vortices appear, which spread as tornadoes of snow powder on the sides and behind the vehicle.

[0084] Any lifted road debris causes disturbances in an image region close to the wheels, which can be detected by the intensity of these disturbances.

[0085] Figure 5 Three example image sections from a side-mounted (right) vehicle camera are shown in a lane with different humidity levels.

[0086] Figure 5 a shows a dry road, wherein the road surface is formed in the region 51 adjacent to the vehicle side (underneath), the structure of which is not disturbed in the image and can therefore be seen.

[0087] Figure 5 b shows a normal wet roadway, the image of the roadway surface being disturbed by identifiable splashing water droplets 52 .

[0088] Figure 5 c shows an extremely wet lane (see Figure 3 ), which has a serious risk of hydroplaning. Here, the road surface can hardly be recognized because the strong splashing water 53 causes serious interference in this image area.

[0089] Figure 6A schematic diagram of a vehicle camera system is shown, which includes at least one camera device 10 and an image data analysis and evaluation unit 60. The image data analysis and evaluation unit 60 determines the type of reflection and the interference intensity and can thus classify the lane condition. The lane condition class can be output to a vehicle actuation device 66, a driver warning device or a driver assistance function control unit.

[0090] Figure 7 A schematic diagram showing method steps for identifying and analyzing lane conditions.

[0091] In step S12, a vehicle-mounted camera system provides image data. In step S14, it is distinguished based on the image data whether there is diffuse reflection or specular reflection caused by the lane surface or a lane accumulation.

[0092] In step S16, the interference intensity of an image region is determined. In step S18, the lane conditions are classified according to the results of steps S14 and S16.

[0093] In an optional step S20 , a lane condition level is output.

[0094] Methods of digital image processing and machine learning:

[0095] In an advantageous embodiment, an algorithm of digital image processing and machine learning is used with the goal of reliably detecting lane reflections in conjunction with rolled-up lane deposits such as water or snow to identify lane conditions such as dry, wet, snowy, icy, and hazardous situations such as hydroplaning. The method is partially applicable to monocular cameras, stereo cameras, and panoramic surround cameras and / or combinations thereof.

[0096] The basis is one or more areas in one or more camera images that can be statically defined or automatically determined, for example, by machine learning. It is important that the image area provides information about the type of reflection and whether there is any raised debris on the roadway. Alternatively, a combination of several image areas and / or several camera devices with several image areas provides information about the type of reflection and whether there is any raised debris that interferes with the image area.

[0097] To detect the lane reflections, one or more regions in one or more camera images that represent the lane surface features are used. Depending on the runtime and accuracy requirements, this can be a segmented cross section or, in a particularly advantageous embodiment, a trapezoid that is converted into a rectangular top view (“bird's eye view”) using an estimated homography. As described in WO 2016 / 177371 A1, different types of lane reflections (diffuse and specular) can now be determined, for example, from a temporal context.

[0098] In order to detect the raised road debris, corresponding areas are defined in one or more camera images, which in a particularly advantageous manner are image sections of a panoramic camera device in the vicinity of the vehicle tires and / or the vehicle body, viewed from the side. This allows the occurrence of raised debris to be determined in a targeted manner within a restricted area and without being influenced by interference from the vehicle's surroundings. Another possibility is to detect the wheels and / or wheel sections in the corresponding scene (own vehicle and other vehicles) with the aid of machine learning. In this case, a detection device that provides the required image sections can be trained.

[0099] The classification of a given lane condition class constitutes a central aspect. The detected image parts serve as the basis for possible rolled-up lane debris and information about the current reflection type on the road. These data are then fed into a machine learning system which learns to map image parts, reflection type and additional information to one of the defined lane conditions based on a training sample. This results in a model which, on the one hand, extracts / learns a structure-based feature for describing rolled-up lane debris and, on the other hand, estimates a suitable combination of these features with reflection type and / or additional information.

[0100] Reflection\Interference none ordinary powerful Diffuse dry snow - Specular Reflection Thin ice layer General humidity Water drifting danger

[0101] Table 1: Classification of lane conditions by cross-comparison of reflection type and interference level

[0102] Furthermore, the classification can be supported by contextual knowledge / additional information. In this context, it is particularly suitable to use the integration of vehicle dynamics data and available sensor data (temperature, air humidity, rain sensor, wiper activity, etc.).

[0103] Fisher vector coding with front camera system and surround view camera system:

[0104] The proposed method requires as input one or more camera images that contain the areas of the image that are important for decision making and that exclude unnecessary background. This is important because interference from surrounding infrastructure and other traffic participants can affect the method.

[0105] Due to the large size of the selected image portion, it is possible even with a very small database to make a decision not based on the expected vortex structure of the roadway accumulation, but on background phenomena that occur by chance in the individual case.

[0106] Local features on a dense grid are extracted from the input image regions of all training images, which describe the image parts through an edge histogram (including HOG "Histogram of Oriented Gradients", LBP "Local Binary Pattern"). Then, a Gaussian mixture distribution model is estimated based on the entire feature set. Then, the individual images are described based on high-order statistics from the feature vectors and the mixture distribution model, the so-called Fisher-Vector-Encoding. Finally, with the help of the image representation obtained for the entire training set, a linear classification device is taught, which provides probabilistic class affiliation for a given image at runtime.

[0107] A particularly advantageous approach is to use two image areas from a front camera and image parts from a panoramic surround camera. A first image area from the front camera is used to normalize existing reflections in the lane. In addition, another larger image part also provides global image background information about weather conditions and influences related to the lane condition, which are caused by infrastructure (such as snow-covered sidewalks and green belts, etc.). On the other hand, the image part of the panoramic surround camera provides information about raised lane debris. A combination of these image parts from different cameras can be used as input for Fisher vector coding.

[0108] The encoding not only maps the characteristic structure of the vortex, but also reflects the relationship between the raised vortex and the reflected and additional information. In addition, the structural features in front of and beside the vehicle can be detected, which also helps to distinguish the lane conditions (lane in snow lane).

[0109] If necessary, the classification results can be stabilized by a post-processing by smoothing the estimates of each image over time. Alternatively, a hysteresis thresholding method can also filter out uncertain decisions.

[0110] Deep learning using front camera and surround camera systems:

[0111] Classical machine learning, like the lane estimation method with the help of Fisher vector coding (see above), consists of several steps that are performed independently of each other. First, features are extracted from the image, which can be used directly or converted into an intermediate representation. Then, a classifier is taught, which makes a classification decision based on the features or the intermediate representation.

[0112] In contrast, in deep learning, the individual steps are not performed explicitly, but rather in an implicit manner as the depth of the neural network increases. This is known as so-called end-to-end learning, which leads to the joint optimization of all parameters present in a system. For the present invention, this means that image parts or the entire image from the front camera and the surround view camera are used as input to the neural network. In addition, reflection types and additional information such as sensor data and driving dynamics data can also be integrated as input. The network is trained based on a training sample without feature extraction or explicit modeling of intermediate representations. Only a mapping of the input data to the corresponding lane conditions is learned.

Claims

1. A method for identifying and evaluating a road condition and surrounding environmental influences associated with weather conditions, the method comprising the following method steps: -(S12) providing image data by means of an onboard camera system, the onboard camera system being configured to image at least a portion of the vehicle's surroundings, wherein: The portion at least partially contains the lane in which the vehicle is traveling; -(S14) distinguishing reflection types by analyzing and evaluating the difference in appearance images of at least one point of the lane in at least two images of the camera system, wherein the reflection types include diffuse reflection and specular reflection of the lane; wherein the images are taken from different shooting angles; (S16) determining whether there is interference caused by at least one wheel of the vehicle rolling up road debris when rolling over the road and determining the intensity of the interference in at least one image of the camera system, - (S18) Classifying the lane condition into one of the following five lane condition classes, taking into account the reflection type and the interference intensity results: a) Dry lane: Non-interfering diffuse reflection b) Ordinary wet lane: Specular reflection type with interference c) Extremely wet lanes with risk of hydroplaning: Specular type with lots of interference d) Snowy driveway: diffuse reflection with interference e) Icy lanes: Non-interfering mirror reflection type, The vehicle-mounted camera system is configured to capture at least a portion of the space around the vehicle wheels and determine whether there is a disturbance in at least one area of ​​at least one image caused by the vehicle wheels rolling up road debris when rolling over the road. Image data captured by different individual camera devices of the vehicle-mounted camera system in different viewing directions and / or viewing angles are provided.

2. The method according to claim 1, wherein: The vehicle-mounted camera system includes a first side-mounted camera device.

3. The method according to claim 2, wherein: The vehicle-mounted camera system comprises a second side camera device, which is arranged opposite to the first side camera device of the vehicle.

4. The method according to any one of claims 1 to 3, wherein: The vehicle-mounted camera system includes a rear camera device of the vehicle.

5. The method according to any one of claims 1 to 3, wherein: The vehicle-mounted camera system includes a vehicle-mounted front camera device.

6. The method according to any one of claims 1 to 3, wherein: The camera image is divided into a plurality of different regions of particular interest, which are used in a targeted manner for evaluating reflection types and interferences depending on their content.

7. The method according to any one of claims 1 to 3, wherein: The individual camera devices are used in a targeted manner in different ways for evaluating reflection types and disturbances.

8. The method according to any one of claims 1 to 3, wherein: The vehicle-mounted camera system has a detection area in front of the vehicle (E) in which the vehicle is located, and determines the interference in the image caused by lane debris raised when the vehicle in front, the vehicle crossing laterally or the vehicle coming from the opposite direction passes over the lane debris.

9. The method according to any one of claims 1 to 3, wherein: Driving dynamics data and / or surroundings information and / or weather information from other non-camera-based surroundings sensors of the vehicle are used as additional decision bases for evaluating the lane conditions and surroundings influences associated with weather conditions.

10. The method according to any one of claims 1 to 3, wherein: To distinguish lane conditions, the machine learning system is trained based on training samples, which includes the following steps: - Detect and / or classify reflection types; - Detection and / or regression of disturbance strength.

11. The method according to claim 10, wherein: With the help of a machine learning system, a mapping from reflection type and interference intensity to lane condition level is learned based on training samples.

12. The method according to claim 8, wherein: Image features are extracted from at least one region of at least one camera image and mapped onto interference intensities by means of a regression device or onto reflection types and / or lane conditions by means of a classification device.

13. The method according to any one of claims 1 to 3, characterized in that: A neural network is trained that learns to map one or more image portions from one or more camera images to lane conditions based on training samples.

14. A vehicle-mounted camera system comprising at least one camera device and an image data analysis and evaluation unit, wherein: The camera device is configured to image at least a portion of the vehicle's surroundings, wherein the portion at least partially includes a lane on which the vehicle is traveling, and provide image data to an image data analysis and evaluation unit; and The image data analysis and evaluation device is configured to: Differentiating between types of reflection, including diffuse reflection and specular reflection of the lane, by analyzing and evaluating differences in appearance images of at least one point of the lane in an image sequence of at least two images of the camera system; Determining whether there is a disturbance in at least one image of the camera system caused by a wheel of the vehicle lifting up road debris when running over the road and determining the intensity of the disturbance; and Taking into account the reflection type and interference intensity results, the lane condition is classified into one of the following five lane condition classes: a) Dry lane: Non-interfering diffuse reflection b) Ordinary wet lane: Specular reflection type with interference c) Extremely wet lanes with risk of hydroplaning: Specular type with lots of interference d) Snowy driveway: diffuse reflection with interference e) Icy lanes: Non-interfering mirror reflection type, The vehicle-mounted camera system is configured to capture at least a portion of the space around the vehicle wheels and determine whether there is a disturbance in at least one area of ​​at least one image caused by the vehicle wheels rolling up road debris when rolling over the road. Image data captured by different individual camera devices of the vehicle-mounted camera system in different viewing directions and / or viewing angles are provided.

Images