Method and device for determining local weather conditions and local road surface conditions

Abstract

Method for determining local weather conditions (41), local precipitation (42) and / or local road surface conditions (43) in the immediate vicinity of a vehicle (1) by evaluating signals (23-30) from at least one vehicle environment sensor (2; 3), wherein at least one interference signal (24; 26; 28; 30) of the at least one vehicle environment sensor is taken into account for determining the local weather conditions (41), local precipitation (42) and / or the local road surface conditions (43), wherein the at least one vehicle environment sensor (2; 3) comprises a radar, lidar sensor and / or a laser scanner.

Description

[0001] The invention relates to a method and a device for determining local weather conditions, local precipitation and a local road surface condition, in particular in a vehicle.

[0002] Driver assistance systems support the driver in their driving tasks, thus contributing to making road traffic safer in the future and reducing the number of accidents. This requires equipping the vehicle with sensors to detect its surroundings as accurately as possible. Primarily, radar, lidar, video-based systems, and laser scanners are used to detect other vehicles and vulnerable road users such as pedestrians and cyclists, as well as infrastructure, thereby creating the basis for a precise description of the vehicle's immediate environment. Laser sensors in the short-, medium-, and long-wave infrared ranges are also well-known. These emit light onto the road surface and receive the reflected light back to, for example, infer the road surface's condition and state. The level of automation in motor vehicles will continue to increase in the future.This will lead to a significant increase in the level of environmental sensor technology used in vehicles.

[0003] Depending on the level of automation (partially automated, highly automated, or autonomous driving), sensors with significantly different sensor principles will work together in the future and will need to be designed with redundancy, at least in some areas. Due to economies of scale, environmental sensors are becoming increasingly affordable and will therefore be indispensable in motor vehicles in the future.

[0004] State-of-the-art technology includes the sensors listed above for detecting other vehicles, vulnerable road users, and infrastructure elements such as road markings or traffic signs. Such sensors are now mass-produced and widely used.

[0005] Sensors for detecting precipitation on the windshield and controlling the windshield wipers accordingly, using light-emitting diodes, photodiodes, or image sensors, are also becoming increasingly common in production vehicles.

[0006] A key part of a driver's task is to recognize the road surface conditions and correctly assess the available coefficient of friction between the tires and the road surface in order to adjust their driving style accordingly. In the future, highly and fully automated vehicles will completely take over this driving task, at least in certain aspects. For this reason, it is essential that the system accurately assesses or records the current local and upcoming road conditions and / or the coefficient of friction.

[0007] DE 10 2004 018 088 A1 discloses a road surface detection system comprising a temperature sensor, an ultrasonic sensor, and a camera. The temperature, roughness, and image data (road surface data) obtained from the sensors are filtered and compared with reference data, and a confidence level is generated for this comparison. Based on this comparison of the filtered road surface data with the reference data, the condition of the road surface is determined. The road surface (e.g., concrete, asphalt, dirt, grass, sand, or gravel) and its condition (e.g., dry, icy, snowy, wet) can thus be classified.

[0008] German patent application DE 42 04 165 C1 discloses a method for optronic snowfall detection consisting of the following steps: emitting a modulated light beam, receiving the reflected portion of the emitted light beam, measuring the increased noise component caused by individual reflections from snowflakes at different distances and thus with different reflection coefficients, and comparing this measurement with the known constant intrinsic noise of the receiving arrangement and the background noise. The method is based on the understanding that during snowfall, the high backscattering of the emitted light beam generates a high level of signal noise, caused by individual reflections from snowflakes at different distances and with different reflection coefficients.Since these criteria only occur during snowfall, they can be used for reliable detection of snowfall because they significantly exceed the known and constant intrinsic noise of the device as well as the background noise from other environmental influences.

[0009] One object of the present invention is to make the determination of a local road surface condition better and more robust.

[0010] The problem is solved by a method having the features of claim 1 and by a device having the features of claim 12.

[0011] The invention is based, among other things, on the following considerations: Highly and fully automated vehicles will be equipped with a variety of different sensors that directly or indirectly monitor local weather conditions or... Weather conditions can affect the signal. This usually results in a deterioration of signal quality and / or a reduction in visibility, up to and including sensor failure. Interference signals caused by weather conditions are sometimes suppressed with great effort in order to make the useful signals, for example for object detection, more visible.

[0012] Local weather and precipitation such as rain or snowfall significantly influence the grip of road surfaces. "What matters is what happens on the pitch," a quote from the world of football could be applied to the road: The current local road surface condition, in addition to tire characteristics, is crucial for the maximum force that can be transmitted between the tire and the road surface. This is because the road surface condition and any existing pavement layer have a substantial influence on the actual coefficient of friction. The coefficient of friction, also known as the friction coefficient, friction coefficient, or friction factor, indicates the maximum force, relative to the wheel load, that can be transmitted between a road surface and a vehicle tire (e.g., in the tangential direction) and is therefore a key measure of driving safety. Besides the road surface condition, tire characteristics are also necessary for a complete determination of the coefficient of friction.

[0013] Precipitation creates an intermediate medium between the tire and the road surface, which significantly influences friction. Currently, no method is known for serial production that detects the road surface condition, or more precisely, the grip of road surfaces, while the vehicle is moving and makes this information available, for example, to driver assistance systems. One approach of the present invention is therefore to also consider local weather conditions or precipitation when determining the current condition of the road surface being driven over.

[0014] A method according to the invention for determining local weather conditions, local precipitation, and / or local road conditions in the immediate vicinity of a vehicle is carried out by evaluating signals from at least one vehicle environment sensor. In this process, at least one interference signal from the at least one vehicle environment sensor is taken into account to determine the local weather conditions, local precipitation, or local road conditions. Signals are, in particular, data or measured values ​​from a vehicle environment sensor.

[0015] A disturbance signal from at least one vehicle environment sensor can be taken into account to determine local weather conditions, local precipitation or the local road surface condition, in particular by checking whether the disturbance signal is an effect of local weather conditions, local precipitation or a local road surface condition, i.e. a weather-related disturbance signal.

[0016] Considering interference signals encompasses both the case where only vehicle environment sensor interference signals are used to determine local weather conditions, precipitation, or road surface conditions, and the case where these interference signals are used only to validate previously assumed local weather conditions, precipitation, or road surface conditions. In the first case, the absence of interference signals leads to the determination of good local weather conditions, no precipitation, or an unaffected road surface condition. This implies both good visibility and a high coefficient of friction.

[0017] The system advantageously evaluates signals from multiple vehicle environment sensors. A combination of different sensor types is particularly beneficial, such as an optical sensor (camera, lidar, laser scanner, photon mixing detector) with a radar or ultrasonic sensor (i.e., a non-optical sensor), or the combination of a passive sensor (camera) with an active sensor (radar, lidar, laser scanner, ultrasonic sensor), i.e., a sensor with both a transmitter and receiver. Such complementary combinations offer comprehensive coverage of both the vehicle's surroundings and weather-related interference.

[0018] A method according to the invention offers the advantage that adverse environmental conditions, which lead to a reduction in the vehicle environment sensor's detection quality, can nevertheless be detected and taken into account. A change or deterioration of the signals can indicate an influence from the weather. From this, the local weather and / or local precipitation and / or the local road surface condition or pavement type can be derived.

[0019] According to a preferred embodiment, a reduction in signal quality, signal availability, or detection range of the at least one vehicle environment sensor as a result of local weather conditions, local precipitation, and / or local road surface conditions is considered a disturbance signal. Therefore, to detect local weather conditions, precipitation, or road surface conditions in the immediate vicinity of the vehicle, influences (especially negative ones) on the signal quality, signal availability, or detection range of the at least one vehicle environment sensor as a result of local weather conditions, local precipitation, or local road surface conditions can advantageously be considered or taken into account as a disturbance signal.

[0020] With a camera sensor, it is advantageous to consider the signal quality, signal availability, and visibility range as a function of the current ambient brightness or lighting conditions. Time-of-day-dependent changes in brightness, for example, do not constitute interference in this sense, as they are not caused by precipitation or weather conditions and also have no influence on the road surface.

[0021] Image data from a camera sensor can preferably be used to identify the road surface condition and precipitation in the form of rain, raindrops, snow or snowflakes by means of digital image processing and classification based on characteristic features.

[0022] Similarly, heavy cloud cover or darkening in the surroundings can be advantageously detected from image data of a camera sensor. The brightness of the vehicle's surroundings can be measured using either a daylight or ambient brightness sensor, such as those commonly used in rain sensors, or a camera sensor. This measurement can then be preferably compared with the typical brightness for the same time of day and year, and local weather conditions can be derived from this comparison.

[0023] A beneficial consideration of useful and interference signals from the vehicle's environmental sensors offers very good opportunities to determine the actual local weather, precipitation, or road surface condition.

[0024] To verify the plausibility of the data recorded in the vehicle, it is advantageous to use location-resolved weather information or rain radar data from external sources, e.g. from the internet, for example with the help of redigitized maps.

[0025] Preferably, a local probability of precipitation can initially be derived based on weather and map information. However, this only provides limited information about the actual local weather, precipitation, and road conditions in the immediate vicinity of the vehicle. Therefore, it is preferable to include all information from the vehicle's environmental sensors in order to determine the actual local weather, precipitation, and road conditions, and to derive or estimate, for example, the local coefficient of friction as precisely as possible.

[0026] Sensor-based detection of precipitation in the form of rain, fog, hail and / or snowfall and its effects in the form of road surface conditions when driven over is advantageously conceivable in two ways: a) Based on specific features, a classifier or a neural network recognizes the local precipitation and / or the local road condition and / or the local weather as a useful signal in the image or in the environmental sensor data, based on video images or environmental sensor data. b) Precipitation and / or road surface conditions, for example when driven over by vehicles ahead, significantly influence the availability, signal quality, and range of environmental sensors as interference signals. The local weather and / or precipitation and / or road surface conditions are derived from these detected interference effects. Visibility impairments (especially for camera systems) result, among other things, from precipitation (rain, snow, blizzards, or even fog) or, for example, from spray or water from vehicles ahead on wet roads, even after the rain has stopped.

[0027] A humidity sensor can preferably be used to determine increased humidity in the immediate vicinity of the vehicle due to local precipitation or a watery road surface, which can be used to validate such a result, e.g., from vehicle environment sensor signals.

[0028] Similarly, a rain sensor can be used to advantageously perform a plausibility check if it detects water on the windshield due to local precipitation (rain) or splash water that has hit the windshield when a vehicle driving over it from the side or in front is present on a wet road surface.

[0029] Further advantageous embodiments of the method according to the invention are characterized by the features of further dependent patent claims.

[0030] The invention further relates to a device for determining local weather conditions, local precipitation, or local road surface conditions in the immediate vicinity of a vehicle. The device comprises at least one vehicle environment sensor and at least one evaluation unit. The at least one evaluation unit is configured such that it can detect at least one interference signal from signals of the at least one vehicle environment sensor and takes this interference signal into account when determining the local weather conditions, local precipitation, or local road surface conditions.

[0031] If, advantageously, in addition to the interference signals from at least one vehicle environment sensor, further input variables are considered when determining local weather conditions, local precipitation, or local road surface condition, the evaluation unit can also perform the function of a fusion unit that fuses the signals or further input variables to determine the local weather conditions, local precipitation, or local road surface condition. In an alternative advantageous embodiment, the device can comprise a separate fusion unit. With a plurality of different vehicle environment sensors or vehicle environment sensor types, one evaluation unit can be provided for each vehicle environment sensor or sensor type.

[0032] The invention will be explained in more detail below with reference to figures and exemplary embodiments. Fig. Figure 1 schematically shows a vehicle with environmental sensors driving on a roadway. Fig. Figure 2 schematically shows a fusion of different signal sources to determine local weather conditions, local precipitation, or a local road surface condition.

[0033] In Fig. Figure 1 shows a vehicle (1) equipped with a first (vehicle) environment sensor (2) and a second environment sensor (3). The first environment sensor (2) is located in an area of ​​the windshield inside the vehicle (1), which is particularly suitable for optical environment sensors. The first environment sensor (2) could be a mono or stereo camera sensor, a lidar sensor, a laser scanner, or even a PMD sensor (photonic mixing device, photon mixing detector). The detection area(s) (12) of the first environment sensor (2) are schematically represented by dashed lines. A camera sensor, for example, has corresponding horizontal and vertical opening angles. A laser scanner could, for example, adjust a detection plane accordingly.

[0034] The second environmental sensor (3) is located in the area of ​​the vehicle's radiator grille (1) and has, for example, the detection area (13) shown with dots. The second environmental sensor could be, in particular, a radar sensor (for long-range or medium- to short-range detection) or an ultrasonic sensor. The first environmental sensor (2) and the second environmental sensor always include a radar sensor, a lidar sensor, and / or a laser scanner. Interference signals:

[0035] Camera / optical sensors: Contrast and therefore visibility are significantly affected by fog, precipitation, spray, snow plumes or sea spray; artifacts or local disturbances in the camera image of the vehicle's surroundings due to raindrops, snowflakes, etc. in the field of view of the camera or optical sensor.

[0036] Radar: Precipitation, spray, snow plumes or sea spray generally lead to attenuation; with snow / slush on the radome, the radar signal is sometimes no longer usable, and "ghost objects" may be detected. Use of useful signals:

[0037] Camera / optical sensors: Rain, fog, splashing water, and spray can be detected as such. Clouds or storm fronts within the detection range can be determined from image data, for example.

[0038] Radar: Objects are largely unaffected by fog. This can therefore be used, together with optical sensor signals that cannot detect these objects, as a complementary signal to determine local weather conditions or to determine the maximum visibility range of optical sensors.

[0039] Large raindrops, hailstones, spray, etc. can be resolved as objects.

[0040] From an image from a camera acting as a first environmental sensor (2), the roadway ahead in the direction of travel can be determined, particularly based on lane markings (5) in the image. Roadway boundary objects such as guideposts (6) can be detected, for example, by a radar sensor acting as a second environmental sensor (2) when they appear within its detection range (13).

[0041] In Fig. Figure 2 shows an exemplary fusion (20) of different signal sources or input data (21a-34) to determine local weather conditions (41), local precipitation (42) or local road surface condition (43).

[0042] A possible starting point for the determination can be a location-resolved weather information (21b) (weather map) together with position information, e.g., GPS (21a). This weather information, including precipitation (21b), can be provided, for example, by a weather service and received wirelessly by a vehicle receiver. In this way, for example, local weather information or rain radar data from the internet can be used, for instance, with the help of re-digitized maps. The TMCplus service, for example, provides weather information that can also be included in the fusion (20).

[0043] I can also imagine communication via current local / location-resolved weather information (21b) between vehicles located close to each other (car-to-car or vehicle-to-vehicle).

[0044] A cloud offers the possibility to obtain weather information (21b) that forms the basis of the fusion (20) and, on the other hand, to make the local weather conditions (41) or local precipitation (42) obtained from the fusion available to other road users via the cloud and corresponding back-end servers - as an update of the weather information (21b), which corresponds to vehicle-to-infrastructure or car-to-cloud communication.

[0045] In principle, Car-to-X (22) is suitable for exchanging not only weather and rain information (21b) but also direct information on road conditions or an estimated coefficient of friction. Car-to-X (22) encompasses both car-to-car communication, i.e., direct communication between two closely located vehicles with appropriate transmitting and receiving units or telematics units, and communication with infrastructure facilities (e.g., traffic lights with transmitting and receiving units that can provide information about the current road conditions on site, or a back-end server that can compile individual information from many vehicles or other sources into a corresponding map).

[0046] The fusion (20) can also incorporate the humidity (31) and / or temperature (32) of the vehicle environment and signals from a rain sensor (34) or control signals for a windscreen wiper (33) that provide information on whether there is currently precipitation, e.g. raindrops (4) on the windscreen of the vehicle (1).

[0047] From usable camera images as camera usable signal (23) it can be determined by means of image evaluation that, for example, hailstones or snowflakes are falling onto the roadway (7), that the roadway is already covered with hail or snow, or that spray or mist from other vehicles is splashing up when driving over a rain-wet roadway.

[0048] If a jet of water hits the windshield in the camera's field of view, the actual image of the vehicle's surroundings is temporarily completely disrupted, but this is rectified by the next wiper action. In addition to such temporary camera interference (24), dense fog can even lead to a persistent camera interference signal. Camera interference signals that are based on a malfunction of the camera electronics, on the other hand, provide no information about local weather conditions and should therefore not be considered during fusion (20).

[0049] Similar considerations apply to interference signals (28; 30) and useful signals (27; 29) from other types of optical environmental sensors as for the camera sensor. For example, certain interference signals (28) from a lidar sensor or interference signals (30) from a laser scanner or a PMD sensor can indicate the presence of weather conditions or precipitation in the detectable environment of the vehicle (1). Fog can be detected from lidar sensor signals (27; 28) because distance echoes are received from different depths of the fog, which sometimes leads to a merging of the individual pulse responses; the same applies to spray.

[0050] In addition to optical environmental sensors, radar sensors and / or ultrasonic sensors are also affected by certain types of precipitation or fogged sensor surfaces (especially radomes). Generally, radar signals (25; 26) are more robust against weather influences than optical sensor signals (23; 24; 27-30). However, a radar signal is attenuated, for example, by a wet layer (rain and / or especially snow) on the radome. Large raindrops or hailstones can also be detected as objects in the radar signal, which can simultaneously be considered a radar interference signal (26), since the precise environmental detection, especially of objects directly relevant to traffic such as other road users or obstacles, is thereby impaired as the actual radar signal (25).

[0051] By fusion (20) of the aforementioned input data taking into account environmental sensor interference signals (24; 26; 28; 30), the local condition of the roadway (7) can ultimately be determined more precisely and robustly. Reference symbol list 1 vehicle 2 first vehicle environment sensor 3 Second vehicle environment sensor 4 rain 5. Road markings 6 guideposts 7 lanes 12 Detection range of the first vehicle environment sensor 13 Detection range of the second vehicle environment sensor 20 Fusion 21a GPS / Positioning unit 21b Location-resolved weather information 22 Car-to-X 23 Camera signal or image data 24 camera interference signal 25 Radar useful signal 26 Radar jamming signal 27 Lidar useful signal 28 Lidar jamming signal 29 Laser signal 30 Laser interference signal 31 Humidity / Humidity 32 Temperature 33 Windscreen wiper system 34 Rain sensor 41 Local weather conditions 42 Local precipitation 43 Local road surface condition

Claims

[1] Method for determining local weather conditions (41), local precipitation (42) and / or local road surface conditions (43) in the immediate vicinity of a vehicle (1) by evaluating signals (23-30) from at least one vehicle environment sensor (2; 3), wherein at least one interference signal (24; 26; 28; 30) of the at least one vehicle environment sensor is taken into account to determine the local weather conditions (41), local precipitation (42) and / or the local road surface conditions (43), wherein the at least one vehicle environment sensor (2; 3) comprises a radar, lidar sensor and / or a laser scanner. [2] Method according to claim 1, characterized by, that a reduction in signal quality and / or signal availability and / or the detection range of the at least one vehicle environment sensor (2; 3) as an effect of local weather conditions (41), local precipitation (42) and / or local road surface conditions (43) is considered as a disturbance signal (24; 26; 28; 30). [3] Method according to claim 1 or 2, characterized by , that at least one useful signal (23; 25; 27; 29) of the at least one vehicle environment sensor (2; 3) is taken into account to determine the local weather conditions (41), the local precipitation (42) and / or the local road surface condition (43). [4] Method according to any of the aforementioned claims, characterized by , that location-resolved weather information (22) is taken into account to determine local weather conditions (41), local precipitation (42) and / or local road conditions (43). [5] Method according to claim 3 or 4, characterized by, that the at least one vehicle environment sensor (2; 3) comprises a camera sensor, wherein an evaluation of image data (23) captured with the camera sensor is carried out to determine the weather conditions (41), the local precipitation (42) and / or the local road condition (43) from the captured image data of the vehicle environment. [6] Method according to claim 5, characterized by , that features are determined from image data (23) which are compared with typical features of certain weather situations, precipitation patterns, road surfaces and / or road coverings. [7] Method according to any of the aforementioned claims, characterized by , that at least one measurement from an ambient light sensor of the vehicle (1) is taken into account to determine the local weather conditions (41), local precipitation (42) and / or the local road surface condition (43). [8] Method according to any of the aforementioned claims, characterized by, that at least one measurement of the humidity (31) of the vehicle environment from a humidity sensor is taken into account to determine the local weather conditions (41), local precipitation (42) and / or local road surface condition (43). [9] Method according to any of the aforementioned claims, characterized by , that at least one measurement from a rain sensor (34) of the vehicle (1) is taken into account to determine the local weather conditions (41), the local precipitation (42) and / or the local road surface condition (43). [10] Method according to any of the aforementioned claims, characterized by , that the activity of a windscreen wiper system (33) of the vehicle is taken into account to determine the local weather conditions (41), local precipitation (42) and / or the local road surface condition (43). [11] Method according to any of the aforementioned claims, characterized by, that at least one measurement from a temperature sensor measuring the ambient and / or road surface temperature (32) is taken into account to determine the local weather conditions (41), local precipitation (42) and / or local road surface condition (43). [12] Device for determining local weather conditions (41), local precipitation (42) and / or local road surface conditions (43) in the immediate vicinity of a vehicle (1) comprising at least one vehicle environment sensor (2; 3) and at least one evaluation unit, wherein the at least one evaluation unit is configured to detect at least one interference signal (24; 26; 28; 30) from signals (23-30) of the at least one vehicle environment sensor (2; 3) and to take into account the at least one interference signal (24; 26; 28; 30) when determining the local weather conditions (41), local precipitation (42) and / or the local road surface conditions (43), wherein the at least one vehicle environment sensor (2; 3) comprises a radar, lidar sensor and / or a laser scanner.

Images