Method for detecting dirt on a windshield of a vehicle

Abstract

The invention relates to a method for detecting soiling (V) on a windscreen (2) of a vehicle (3), wherein in an application method - in data of the vehicle environment recorded by an environment detection sensor (5), positions of surrounding objects (O1 to On) and distances (d, d1 to d8) of the surrounding objects (O1 to On) to the vehicle (3) are determined, - in images (B1 to Bz) of the eyes (A1, A2) of the driver (F) captured by a driver observation camera (4), the gaze direction (BR1 to BRm) of the driver (F) and the accommodation (AK) of the eyes (A1, A2) are determined, - a correspondence analysis determines which surrounding object (O1 to On) the driver (F) is looking at, and - using a calibration function (f) determined in a calibration process, a soiling-free windshield (2) is detected if a relationship between the distance (d, d1 to d8) of the respective surrounding object (O1 to On) and the accommodation (AK) of the eyes (A1, A2) of the driver (F) when looking at this surrounding object (O1 to On) lies within a predetermined target range, and soiling (V) of the windshield (2) in the field of vision of the driver (F) is detected if the relationship lies outside the predetermined target range. The invention further relates to a method for operating a cleaning device (1) for a windshield (2) of a vehicle (3).

Description

[0001] The invention relates to a method for detecting dirt on a windshield of a vehicle.

[0002] The invention further relates to a method for operating a cleaning device for a windshield of a vehicle.

[0003] DE 10 2020 001 344 A1 discloses a method for detecting the surroundings of a motor vehicle using a camera system comprising a camera. The camera is positioned in a first position, and a recording of the surroundings is recorded using the camera. The camera system comprises an electronic computing device, by means of which a control signal is generated for the camera system. The camera is adjusted to a second position different from the first position by means of a holding device arranged on the camera system, depending on the control signal. The recording of the surroundings is analyzed using the electronic computing device, and contamination on a window of the motor vehicle is detected by evaluating the recording.

[0004] Furthermore, DE 10 2009 054 194 A1 discloses a vehicle with a device for generating a head-up display. A camera comprising an image sensor and a focusing element is statically or dynamically introduced into the beam path of the head-up display via an optical or mechanical element. A region of the exterior of the windshield is imaged in focus on the image sensor, or a driver's eye area is imaged in focus on the image sensor. The vehicle has assistance functions based on an evaluation of the data from the image sensor, wherein the assistance functions include driver detection, fatigue detection, and rain / dirt detection.

[0005] Furthermore, an interval-based windshield cleaning system based on the interaction of a vehicle driver with a cleaning device is known from the prior art. In this system, the windshield is cyclically cleaned by the cleaning device based on an interval specified by the driver. This cleans both large-area and localized uneven soiling.

[0006] Furthermore, a camera-assisted cleaning of a vehicle windshield is known from the prior art. An algorithm is applied via a front camera. This algorithm detects the degree of soiling in the visible area of ​​the front camera and accordingly initiates cleaning using a cleaning device. This removes uniform soiling, such as water films, from the windshield.

[0007] The invention is based on the object of providing a novel method for detecting soiling on a windshield of a vehicle and a novel method for operating a cleaning device for a windshield of a vehicle.

[0008] The object is achieved according to the invention by - a method for detecting soiling on a windshield of a vehicle, which has the features specified in claim 1, and - a method for operating a cleaning device for a windshield of a vehicle, which has the features specified in claim 9.

[0009] Advantageous embodiments of the invention are the subject of the subclaims.

[0010] The method for detecting dirt on a windscreen of a vehicle is characterized according to the invention in that - in a calibration procedure, a calibration function is determined with a soil-free windscreen, which represents a relationship between a determined real distance of a respective surrounding object to the vehicle and a determined accommodation of the eyes of a driver of the vehicle when looking at this surrounding object, and - in an application process - the positions of surrounding objects and the distances of the surrounding objects to the vehicle are determined in data of the vehicle environment recorded by an environment detection sensor, - in images of the driver's eyes captured by a driver observation camera, the driver's gaze direction and the accommodation of the eyes are determined, - a correspondence analysis is used to determine which surrounding object the driver is looking at, and - using the calibration function, a contamination-free windscreen is detected if a relationship between the distance of the respective surrounding object and the accommodation of the driver's eyes when looking at this surrounding object lies within a specified target range, and contamination of the windscreen in the driver's field of vision is detected if the relationship lies outside the specified target range.

[0011] Using the present method, it is possible, particularly compared to the camera-based methods known from the prior art, to detect contamination outside the detection range of a vehicle's front camera. The detection correlates with the driver's field of vision due to a coupling of the driver's eye accommodation and objects in the vehicle's vicinity. Thus, the method can detect contamination that occurs partially on the windshield, such as salt lines. Such contamination is spatially limited and is generally not present across the entire surface of the windshield.Such contamination cannot be reliably detected using the prior art method using the front camera, as the partial contamination may be located outside the detection range of the front camera, but may still be in the driver's field of vision, restricting their view. Using the present method, such local contamination can be very reliably detected, particularly in the driver's field of vision, so that, for example, when such contamination occurs, a cleaning device for cleaning the windshield can be automatically activated. Cleaning can involve wetting the windshield with a cleaning fluid and activating the vehicle's windshield wipers.

[0012] Thus, the method enables contamination detection based on physical driver parameters.

[0013] The surrounding objects are detected using environmental detection sensors, such as cameras, lidar sensors, radar sensors, etc. Since such environmental detection sensors and a driver monitoring camera are often already present in vehicles for operating driver assistance functions, no additional hardware is required to implement the process. Positions and distances of the surrounding objects from the vehicle, as well as object classification, can be derived, for example, from dynamic object fusion.

[0014] Further advantages of the procedure include - in a possible recurring activation of the same in the vehicle, - in a possible derivation of the calibration function during real operation of the vehicle, - in the use of already known image evaluation methods, - in real-time detection of contamination - independent of the existing lighting situation and - in the possibility of detecting a variety of contaminants, such as salt, mud, sludge, etc.

[0015] A possible design of the procedure provides that in the calibration procedure - the positions of surrounding objects and the distances of the surrounding objects to the vehicle are determined in data of the vehicle environment recorded by the environment detection sensors, - in images of the vehicle driver's eyes captured by the driver observation camera, a direction of gaze of the driver and an accommodation of the eyes are determined, - a correspondence analysis is used to determine which surrounding object the driver is looking at, and - the calibration function is determined, which represents the relationship between the determined real distance of a respective environmental object to the vehicle and the determined accommodation of the driver's eyes when looking at this environmental object.

[0016] Determining the calibration function in this way enables excellent comparability of the accommodation and distance relationships from the calibration function with the relationships determined in the application procedure. This allows for particularly reliable detection of dirt on the vehicle's windshield.

[0017] Another possible embodiment of the method provides for the correspondence analysis to determine a correspondence between the viewing direction and the position of the respective surrounding object by comparing the driver's viewing directions with the trajectory of the detected surrounding object. Such a determination of the correspondence when tracking the trajectory is particularly reliable.

[0018] Another possible embodiment of the method provides for the determination of the driver's eye accommodations during the detection of contamination in the application method for distances of different positions of a respective environmental object along the trajectory and their evaluation using the calibration function. Determining the accommodations in this way while tracking the trajectory enables particularly reliable detection of contamination on the vehicle's windshield.

[0019] Another possible embodiment of the method provides for the application process to be repeated after an automatic windshield cleaning has been performed due to the detection of contamination. This allows it to be determined whether the cleaning was successful and, if necessary, automatically perform another windshield cleaning until the contamination is removed.

[0020] Another possible embodiment of the method provides that, in the calibration process and the application process, the positions and distances of the surrounding objects from the vehicle and / or an object classification are derived from dynamic object fusion. Using data from such an object fusion, the positions, distances, and object classification can be determined very reliably.

[0021] Another possible embodiment of the method provides for the determination of which surrounding object the driver is looking at, performed in the correspondence analysis, to be carried out based on intersection detection of the driver's gaze directions with the surrounding objects. Such intersection detection is particularly easy to implement and enables the correspondence analysis to be carried out precisely.

[0022] Another possible embodiment of the method provides for a threshold value analysis to determine whether the relationship between the distance of the respective surrounding object and the driver's eye accommodation when looking at this surrounding object lies within a specified target range. Such a threshold value analysis is particularly easy to implement and allows for the precise determination of whether the relationship lies within the specified target range. A corresponding threshold value can be realistically determined, for example, in test drives, applications, etc.

[0023] In the method according to the invention for operating a cleaning device for a windshield of a vehicle - the above-mentioned procedure is used to check whether there is any contamination of the windscreen in the driver's field of vision, and - If such contamination is present, the cleaning device is automatically activated, which cleans the windscreen.

[0024] The process enables automatic windshield cleaning without requiring the driver to interact with a cleaning device. This prevents driver distraction from traffic events resulting from manual activation, increasing driving comfort. Contamination of the windshield can also be removed quickly and reliably, improving driver visibility and increasing road safety.

[0025] One possible embodiment of the method provides for the application process to be repeated after the automatic cleaning of the windshield. This allows it to be determined whether the cleaning was successful and, if necessary, to automatically clean the windshield again until the contamination is removed.

[0026] Embodiments of the invention are explained in more detail below with reference to drawings.

[0027] Showing: Fig. 1 schematically shows a sequence of a method for operating a cleaning device of a windshield of a vehicle, Fig. 2 schematically a visualization of a first method step of the method according to Fig. 1, Fig. 3 schematically a visualization of a second method step of the method according to Fig. 1, Fig. 4 schematically a visualization of a third method step of the method according to Fig. 1, Fig. 5 schematically a visualization of a fourth method step of the method according to Fig. 1, Fig. 6 schematically a visualization of a fifth method step of the method according to Fig. 1, Fig. 7 schematically shows a visualization of a sixth method step of the method according to Fig. 1, Fig. 8 schematically a visualization of a seventh method step of the method according to Fig. 1, Fig. 9 schematically a visualization of an eighth method step of the method according to Fig. 1, Fig. 10 schematically shows a visualization of a ninth method step of the method according to Fig. 1 and Fig. 11 schematically shows a visualization of a tenth method step of the method according to Fig. 1.

[0028] Corresponding parts are provided with the same reference numerals in all figures.

[0029] In Fig. 1 is a schematic diagram of a process for operating a cleaning device 1 (shown in Fig. 11) a windshield 2 (shown in Fig. 4) of a vehicle 3 (shown in Fig. 4). The Fig. 2 to 11 show visualizations of process steps VS1 to VS10 of the process.

[0030] The method enables the detection of Fig. 11 shown in more detail, locally distributed dirt V on the windscreen 2 by coupling a Fig. 4 driver observation camera 4 shown in more detail with a detection of in a vehicle environment existing and in Fig. 5 shown in more detail. This assumes that the driver observation camera 4 and a Fig. 4, the environment detection sensors 5 shown in more detail are calibrated and adjusted accordingly extrinsically and intrinsically and a transformation between coordinate systems of the driver observation camera 4 and the environment detection sensors 5 is possible.

[0031] First, in a calibration process, with the windscreen 2 free of dirt in at least one field of vision of the driver F, in a first process step VS1, as in Fig. 2 visualized, images B1 to Bz of a driver F of the vehicle 3, in particular images B1 to Bz of the driver F, are captured by means of the driver observation camera 4.

[0032] In a second process step VS2, as in Fig. 3, by evaluating the images B1 to Bz a movement of eyes A1, A2 of the driver F is detected. Fig. 8, which essentially describes a change in the lens of a respective eye A1, A2, whereby the refractive power of the eye A1, A2 is adapted so that objects in space are focused. The accommodation AK correlates with a distance d to an environmental object O1 to On focused by the driver F. It is possible that the accommodation AK is derived from video images by tracking the eyes A1, A2, also known as eye tracking.

[0033] In a third process step VS3, as in Fig. 4 visualized, gaze directions BR1 to BRm of the driver F, also called gaze, are determined starting from a head position KPOS of the driver F.

[0034] In a fourth process step VS4, as in Fig. 5 visualized, a three-dimensional detection of the vehicle environment is carried out by means of the environmental detection sensor system 5, in which a depth of the vehicle environment is determined in relation to a vehicle origin. For this purpose, the environmental detection sensor system 5 comprises appropriately designed sensors, for example cameras, lidars, radars, etc. Based on data acquired by the environmental detection sensor system 5, positions of environmental objects O1 to On and in Fig. 8, the distances d of the surrounding objects O1 to On to the vehicle 3 are determined. The surrounding objects O1 to On can also be classified and thus, for example, a distinction can be made between passenger cars, trucks, pedestrians, cyclists, etc. Furthermore, a respective trajectory of the surrounding objects O1 to On is determined based on the data acquired by the surrounding detection sensor system 5.

[0035] In a fifth process step VS5, as in Fig. 6 visualized, a transformation of a coordinate system KS1 of the environment detection sensor 5, i.e. an environment model, into a coordinate system KS2 of the driver observation camera 4 is carried out.

[0036] In a sixth process step VS6, as in Fig. 7 visualized, a correspondence analysis is carried out in which it is determined which environmental object O1 to On the driver F is looking at. In this case, a correspondence between a respective viewing direction BR1 to BRm and a position of the respective environmental object O1 to On is determined by a course of viewing directions BR1 to BRm of the driver F and a trajectory of the detected environmental object O1 to On. This means that an analysis of a correspondence between an object trajectory and a viewing direction course of the driver F is carried out. In this case, it is determined which of the detected environmental objects O1 to On the driver F is focusing on. This can be done, for example, using intersection detection of the viewing directions BR1 to BRm with the environmental objects O1 to On.

[0037] In a seventh process step VS7, as in Fig. 8, a calibration function f is determined, which represents a relationship between a determined real distance d of a respective environmental object O1 to On from the vehicle 3 and a determined accommodation AK of the eyes A1, A2 of the driver F when looking at this environmental object O1 to On as a function of a polar angle PW. The calibration function f is a three-dimensional function that represents the relationship between real depth and accommodation AK of the eyes A1, A2 of the driver F.

[0038] The previously described calibration procedure with the method steps VS1 to VS7 can be carried out during real operation of the vehicle 3, for example, in dangerous situations in which it is known that the driver F is focusing on an object O1 to On in front. Such a dangerous situation can be, for example, an emergency braking in a traffic situation at a road intersection.

[0039] In an application method, i.e. for the detection of contamination V during vehicle operation 3, the previously mentioned process steps VS1 to VS6 are also carried out, with the exception of process step VS7. - the positions of surrounding objects O1 to On and distances d of the surrounding objects O1 to On to the vehicle 3 are determined from the vehicle environment data acquired by the environment detection sensor system 5, - the gaze direction BR1 to BRm of the driver F and the accommodation AK of the eyes A1, A2 of the driver F are determined in images B1 to Bz of the eyes A1, A2 of the driver F captured by the driver observation camera 4 and - the correspondence analysis determines which environmental object O1 to On the driver F is looking at.

[0040] In the application method, a comparison is carried out between the detected accommodation AK and the real object depth during real operation of the vehicle 3.

[0041] Subsequently, in an eighth process step VS8, as in Fig. 9 visualized, then, if a correspondence between the gaze direction BR1 to BRm of the driver F and a position of an environmental object O1 to On was determined in the correspondence analysis, accommodations AK1 to AK8 of the eyes A1, A2 of the driver F along the trajectory of the corresponding environmental object O1 to On are compared using the calibration function f. This means that the congruence of the relationship between the accommodations AK1 to AK8 of the eyes A1, A2 of the driver F with a respective distance d1 to d8 of the environmental object O1 to On is checked with the relationship stored in the calibration function f at such a distance d1 to d8.

[0042] It is assumed that if the windshield 2 in the driver's field of vision is free of dirt, the accommodation AK, AK1 to AK8 follows at least essentially the object depth, i.e. the respective distance d, d1 to d8 of the surrounding object O1 to On to the vehicle 3 on its trajectory. However, if dirt V is present on the windshield 2 in the driver's field of vision, it can be assumed that the driver F repeatedly attempts to focus on the surrounding object O1 to On. In doing so, the driver F attempts to view the scenery, but is at least partially limited by the dirt V because he focuses on the dirt V instead of the scenery. This results in a divergence between the distance d, d1 to D8 derived from the accommodation AK and the distance d, d1 to d8 derived from the data of the environment detection sensor system 5.This means that the distance d, d1 to d8, derived from the accommodation AK, is compared with the distance d, d1 to d8, determined from the data acquired by the environment detection sensor 5.

[0043] This means that a difference ΔAK, ΔAK1 to ΔAK8 of the accommodation AK, AK1 to AK8 of the eyes A1, A2 determined in the application procedure and the accommodation AK, AK1 to AK8 originating from the calibration function f at the same distance d, d1 to d8 is determined.

[0044] In a ninth process step VS9, as in Fig. 10, using the calibration function f, a contamination-free windshield 2 is detected if a relationship between the distance d of the respective surrounding object O1 to On and the accommodation AK of the eyes A1, A2 of the driver F when looking at this surrounding object O1 to On lies within a specified target range. Contamination V of the windshield 2 located in the field of vision of the driver F, on the other hand, is detected if the relationship lies outside the specified target range.

[0045] A threshold value analysis is carried out, whereby a threshold value SW of the difference ΔAK between the accommodations AK from test drives, applications, etc. is determined and specified accordingly. If the difference ΔAK exceeds this threshold value SW, for example, in at least three consecutive frames of the images B1 to Bz captured by the driver observation camera 4, a divergence is detected between a scenario depth, i.e. the corresponding distance d, d1 to d8 of the position of the respective environmental object O1 to On, and the accommodation AK of the eyes A1, A2 of the driver F determined from the images B1 to Bz, at which relationship lies outside the specified target range due to contamination V in the field of vision of the driver F.

[0046] If a contamination V in the field of vision of the driver F is detected in this way, in a tenth process step VS10, as in Fig. 11 visualized, automatically activates the cleaning device 1, which cleans the windshield 2.

[0047] As in Fig. 11, the driver F, according to the representations of the first two images Bl1, BI2, looks through a soiling-free area of ​​the windshield 2 at surrounding objects O1, O2, which are located at a distance d1, d2 from the vehicle 3. The difference ΔAK of the accommodations AK does not exceed the threshold value SW, so that a soiling-free area of ​​the windshield 2 is detected in the area of ​​the respective viewing direction BR1 to BRm of the driver F.

[0048] If, as shown in images BI3 to BI5, the driver F looks through an area of ​​the windshield 2 covered with contamination V, for example, through a layer of ice, at surrounding objects O3 to O5 located at a distance d3 to d5 from the vehicle 3, the difference ΔAK of the accommodations AK exceeds the threshold value SW. Thus, the contamination V of the windshield 2 is detected in the area of ​​the respective viewing direction BR1 to BRm of the driver F, and the cleaning device 1 is automatically activated.

[0049] Once the cleaning has been completed, the application process is repeated. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2020 001 344 A1

[0003] DE 10 2009 054 194 A1

[0004]

Claims

[1] Method for detecting soiling (V) on a windscreen (2) of a vehicle (3), characterized by , that - in a calibration process, with a soil-free windscreen (2), a calibration function (f) is determined which represents a relationship between a determined real distance (d, d1 to d8) of a respective surrounding object (O1 to On) to the vehicle (3) and a determined accommodation (AK) of eyes (A1, A2) of a driver (F) of the vehicle (3) when looking at this surrounding object (O1 to On), and - in an application process - in data of the vehicle environment recorded by an environment detection sensor (5), positions of surrounding objects (O1 to On) and distances (d, d1 to d8) of the surrounding objects (O1 to On) to the vehicle (3) are determined, - in images (B1 to Bz) of the eyes (A1, A2) of the driver (F) captured by a driver observation camera (4), the gaze direction (BR1 to BRm) of the driver (F) and the accommodation (AK) of the eyes (A1, A2) are determined, - a correspondence analysis determines which surrounding object (O1 to On) the driver (F) is looking at, and - using the calibration function (f), a contamination-free windshield (2) is detected if a relationship between the distance (d, d1 to d8) of the respective surrounding object (O1 to On) and the accommodation (AK) of the eyes (A1, A2) of the driver (F) when looking at this surrounding object (O1 to On) lies within a predetermined target range, and contamination (V) of the windshield (2) in the field of vision of the driver (F) is detected if the relationship lies outside the predetermined target range. [2] Method according to claim 1, characterized bythat in the calibration procedure - in data of the vehicle environment recorded by the environment detection sensor system (5), positions of surrounding objects (O1 to On) and distances (d, d1 to d8) of the surrounding objects (O1 to On) to the vehicle (3) are determined, - in images (B1 to Bz) of the eyes (A1, A2) of the driver (F) of the vehicle (3) captured by the driver observation camera (4), a direction of gaze (BR1 to BRm) of the driver (F) and an accommodation (AK) of the eyes (A1, A2) are determined, - a correspondence analysis determines which surrounding object (O1 to On) the driver (F) is looking at, and - the calibration function (f) is determined, which represents the relationship between the determined real distance (d, d1 to d8) of a respective environmental object (O1 to On) to the vehicle (3) and the determined accommodation (AK) of eyes (A1, A2) of the driver (F) when looking at this environmental object (O1 to On). [3] Method according to claim 1 or 2, characterized by that in the correspondence analysis a correspondence between the viewing direction (BR1 to BRm) and a position of the respective environmental object (O1 to On) is determined by a course of viewing directions (BR1 to BRm) of the driver (F) and a trajectory of the detected environmental object (O1 to On). [4] Method according to claim 3, characterized bythat relations of accommodations (AK) of the eyes (A1, A2) of the driver (F) during the detection of the contamination (V) in the application method for distances (d, d1 to d8) of different positions of a respective environmental object (O1 to On) along the trajectory are determined and evaluated using the calibration function (f). [5] Method according to one of the preceding claims, characterized by that the application process is carried out again after an automatic cleaning of the windscreen (2) carried out due to a detection of contamination (V). [6] Method according to one of the preceding claims, characterized by that in the calibration method and the application method, positions and distances (d, d1 to d8) of the surrounding objects (O1 to On) to the vehicle (3) and / or an object classification are derived from a dynamic object fusion. [7] Method according to one of the preceding claims, characterized by that the determination carried out in the correspondence analysis as to which environmental object (O1 to On) the driver (F) is looking at is carried out by means of an intersection detection of the viewing directions (BR1 to BRm) of the driver (F) with the environmental objects (O1 to On). [8] Method according to one of the preceding claims, characterized by that the check as to whether the relationship between the distance (d, d1 to d8) of the respective surrounding object (O1 to On) and the accommodation (AK) of the eyes (A1, A2) of the driver (F) when looking at this surrounding object (O1 to On) lies within a predetermined target range is carried out using a threshold value analysis. [9] Method for operating a cleaning device (1) for a windscreen (2) of a vehicle (3), wherein - by means of a method according to one of the preceding claims, it is checked whether there is any contamination (V) of the windscreen (2) in the field of vision of the driver (F), and - in the event of such contamination (V), the cleaning device (1) is automatically activated, which cleans the windscreen (2). [10] Method according to claim 8, characterized by that the application procedure is repeated after the automatic cleaning of the windshield (2).

Images