Method for detecting contaminants on a protective screen of an optical sensor

By dividing the sensor's field of view into a grid and using motion vectors to differentiate between contamination and environmental feedback loss, the method reduces false positives, improving sensor reliability in driver assistance systems.

DE102023002136A1Pending Publication Date: 2024-11-28MERCEDES BENZ GROUP AG
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
DE102023002136
Authority / Receiving Office
DE · DE
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Grid-based, active, optical sensors in driver assistance systems face false-positive contamination detection in environments with limited optical feedback, leading to reduced system availability.

Method used

A method that divides the sensor's field of view into a grid of cells, assigns contamination states based on sensor measurements, updates these states over time, and uses motion vectors to distinguish between actual contamination and environmental feedback loss, employing additional sensors for validation.

Benefits of technology

Significantly reduces false-positive contamination detection, enhancing system reliability without requiring additional hardware and applicable to existing sensors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

The invention relates to a method for detecting contaminants on a protective screen of an optical sensor (2), wherein a sensor field of view (3) of the sensor (2) is or is divided into a grid with a plurality of grid cells (Rij), wherein each grid cell (Rij) is assigned one of the states contaminated (K), potentially contaminated (PK) or uncontaminated (NK) depending on measurements of the sensor (2) that are carried out in a solid angle of the respective grid cell (Rij), wherein the states are updated in successive time steps depending on the respective measurement of the sensor (2), wherein for the update a motion vector (v) is assigned to each grid cell (Rij) to which one of the states contaminated (K) or potentially contaminated (PK) is assigned. Kont ij) of the respective assumed contamination over a given period of time (δt), and with a determined movement vector (v Umgebung) of a static object (4) located in the solid angle of the respective grid cell (Rij) in the surrounding area is compared.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a method for detecting contamination on a protective screen of an optical sensor according to the preamble of claim 1 and to a driver assistance system according to the preamble of claim 7.

[0002] Raster-based active optical sensors are used as part of driver assistance systems. These sensors are designed to detect contamination or other types of sensor blockages for self-monitoring purposes. Such sensors are problematic in environments where the sensor receives no optical feedback from the environment in parts of the sensor's field of view (e.g., areas with few or no landmarks, such as deserts). The absence of this feedback can be misinterpreted as contamination. This false-positive detection of contamination leads to significantly reduced availability of the driver assistance systems.

[0003] DE 10 2020 115 252 A1 discloses a method for detecting contamination on a protective screen of a lidar sensor. The detection area of ​​the lidar sensor is divided into several sectors, and it is determined sector by sector whether contamination is present on the protective screen in the respective sector.

[0004] The invention is based on the object of providing a novel method for detecting contamination on a protective screen of a sensor, a novel driver assistance system and a novel vehicle.

[0005] The object is achieved according to the invention by a method having the features specified in claim 1, by a driver assistance system having the features of claim 7 and by a vehicle having the features of claim 9.

[0006] Advantageous embodiments of the invention are the subject of the subclaims.

[0007] A method for detecting contamination on a protective screen of an optical sensor is proposed, wherein a sensor field of view of the sensor is or will be divided into a grid with a plurality of grid cells.According to the invention, each grid cell is assigned one of the states contaminated, potentially contaminated or non-contaminated depending on measurements of the sensor which are carried out in a solid angle of the respective grid cell, wherein the states are updated in successive time steps depending on the respective measurement of the sensor, wherein for the update for each grid cell to which one of the states contaminated or potentially contaminated is assigned, a movement vector of the respective assumed contamination over a predetermined period of time is determined and compared with a determined movement vector of a static object in the environment which is located in the solid angle of the respective grid cell.

[0008] The proposed invention significantly reduces false positive detection of contamination in the absence or absence of optical feedback from the environment. Furthermore, the invention generally improves false positive behavior. Since no additional hardware is required to implement the invention, it can also be applied to sensors already in use.

[0009] In one embodiment, the respective grid cell is assigned the state contaminated or potentially contaminated if the measured values ​​for internal sensor reflections and / or if there is increased optical feedback from the immediate near field of the sensor compared to grid cells classified as non-contaminated and / or if there is no optical feedback from the environment over several measurement cycles, and otherwise the state non-contaminated.

[0010] In one embodiment, if the motion vector of the respective assumed contamination is not equal to the motion vector of the static object in the environment located in the solid angle of the respective grid cell, the respective grid cell is assigned the state contaminated and otherwise the state uncontaminated.

[0011] In one embodiment, at least one further optical sensor is used, the sensor field of view of which is or will be divided into a grid with a plurality of grid cells, wherein each grid cell of the further sensor is assigned one of the states contaminated, potentially contaminated or non-contaminated depending on measurements of the sensor which are carried out in a solid angle of the respective grid cell, wherein the states are updated in successive time steps depending on the respective measurement of the sensor, wherein the state of the grid cell of the first sensor is changed from contaminated to potentially contaminated if a grid cell of the further sensor which is congruent with this grid cell is classified as non-contaminated according to the criteria specified in claim 2.

[0012] In one embodiment, a grid cell of the first sensor classified as potentially contaminated is classified as contaminated if the previously detected condition is confirmed in the subsequent time step according to the criteria of claim 2 or 3 and / or if a grid cell of the further sensor that is congruent with this grid cell is classified as not contaminated according to the criteria specified in claim 2.

[0013] In one embodiment, a grid cell of the first sensor classified as potentially contaminated is classified as non-contaminated if a grid cell of the further sensor that is congruent with this grid cell is classified as potentially contaminated according to the criteria specified in claim 2 and / or 3.

[0014] According to one aspect of the present invention, a driver assistance system for a vehicle or a driving system for an autonomous vehicle is proposed, comprising at least one optical sensor, wherein the driver assistance system or driving system is configured to carry out the method described above.

[0015] In one embodiment, the optical sensor is designed as a camera or as a lidar sensor.

[0016] According to one aspect of the present invention, a vehicle is proposed comprising a driver assistance system or driving system as described above.

[0017] The vehicle can, for example, be designed as an autonomous or semi-autonomous vehicle, in particular as a passenger car, commercial vehicle or bus.

[0018] The invention relates to a method for detecting contamination on the sensor viewing window of an optical sensor, in particular a lidar sensor.

[0019] Embodiments of the invention are explained in more detail below with reference to drawings.

[0020] Showing: Fig. 1 a schematic view of a vehicle with an optical first sensor with a sensor field of view, Fig. 2 a schematic view of the sensor field of view of the first sensor, in which contamination states of different grid cells are marked, Fig. 3 a schematic view of the sensor field of view of a second sensor, in which contamination states of different grid cells are marked, Fig. 4 a schematic state diagram showing which state transitions are possible for the grid cells of the first sensor, Fig. 5 a schematic state diagram showing which state transitions are possible for the grid cells of the second sensor.

[0021] Corresponding parts are provided with the same reference numerals in all figures.

[0022] The invention relates to a method for detecting contamination on the sensor viewing window of an optical sensor.

[0023] Fig. Figure 1 is a schematic view of a vehicle 1 with an optical first sensor 2 having a predetermined sensor field of view 3. The sensor 2 is, for example, a lidar sensor. The sensor 2 scans the surroundings of the vehicle 1 in its sensor field of view 3 in the horizontal and vertical directions, each with a predetermined horizontal angular resolution αh and a vertical angular resolution αv.

[0024] The sampling is repeated cyclically in specified time steps.

[0025] The sensor field of view 3 of the sensor 2 is divided into a grid with a plurality of grid cells Rij. The dimensions of a grid cell Rij correspond to the horizontal angular resolution αh or the vertical angular resolution αv of the sensor 2 in the horizontal and vertical directions, respectively. Alternatively, the dimensions of the grid cells Rij can be increased for improved robustness of the detection function, typically to integer multiples of the horizontal angular resolution αh or the vertical angular resolution αv. Each grid cell Rij is thus assigned a specific solid angle of the sensor field of view 3.

[0026] In each time step, each grid cell Rij is assigned a measured value mij, which is determined with sensor 2 in the solid angle of the respective grid cell Rij. The measured value mij contains information about one (or more) measuring points M recorded in the respective solid angle. Ion an object 4, where the light detected by sensor 2 is reflected back to sensor 2. The measured value mij consists of the position of the measuring points M I in space (typically as solid angle (αh M , αv M ) and radial distance r), as well as a measure of the intensity of the reflected laser beam. For simplicity, however, only the measured value mij will be referred to below.

[0027] Each grid cell Rij is assigned a contamination state at each time step. The contamination state can take the following three values: i. Condition contaminated K, ii. potentially contaminated PK, and iii. Condition not contaminated NK.

[0028] Fig. Figure 2 is a schematic view of the sensor field of view 3 of the first sensor 2, in which the contamination states of the different grid cells Rij are marked.

[0029] Black stands for contaminated K, hatched stands for potentially contaminated PK and white stands for non-contaminated NK.

[0030] A second optical sensor (not shown), which also scans the environment horizontally and vertically, can optionally be used to verify the contamination levels of the grid cells Rij of the first sensor 2. In such a case, a congruent second grid is created for this second sensor. Analogous to the above case, each grid cell Rij is assigned a measured value and one of the above-mentioned contamination levels.

[0031] Fig. Figure 3 is a schematic view of the sensor field of view 3 of the second sensor, in which the contamination states of the various grid cells Rij are marked. Black represents contaminated K, hatched represents potentially contaminated PK, and white represents uncontaminated NK.

[0032] When the procedure is started, all grid cells Rij are assigned one of the predefined contamination states in an initialization step, for example the state uncontaminated NK.

[0033] After the initialization step, the contamination states of all grid cells Rij are updated in each subsequent time step.

[0034] The contamination status of a grid cell Rij is updated depending on the measured value mij, which was determined in the current time step t1 to t2 = t1 + Δt in the solid angle of the respective grid cell Rij.

[0035] If a grid cell Rij is assigned the state contaminated K in a time step, this is interpreted as a contamination present in the solid angle of this grid cell Rij on a viewing window of the sensor 2. Thus, it is detected in which area of ​​the viewing window is contaminated, and cleaning of the viewing window, in particular a targeted cleaning of the contaminated area, can then be initiated.

[0036] The states of all grid cells Rij are updated in each time step.

[0037] Fig. 4 is a schematic view of a state diagram showing which state transitions are possible for the grid cells Rij of the first sensor 2.

[0038] The respective state transitions occur when the conditions indicated by the arrows are met. This means: - The contaminated state K is maintained if the condition Zaa is met. - The contaminated state K changes to the uncontaminated state NK when the condition Zac is met. - The contaminated state K changes to the potentially contaminated state PK when the condition Zab is met. - The potentially contaminated PK status is maintained if the Zbb condition is met. - The potentially contaminated state PK changes to the contaminated state K when the condition Zba is met. - The potentially contaminated PK state changes to the non-contaminated NK state when the Zbc condition is met. - The non-contaminated NK state is maintained if the condition Zcc is met. - The non-contaminated state NK changes to the potentially contaminated state PK when the condition Zcb is met.

[0039] The state transitions Zaa to Zcb are described below: The state at the beginning of the observation period was determined in the period t0 to t1 = t0 + Δt, while the state transition is determined in the current period t1 to t2 = t1 + Δt.

[0040] The following describes measured values ​​for the state changes according to the measurement methods for optical sensors: (1) Criteria for measured values ​​of potentially contaminated PK and contaminated K states: - increased measured values ​​for internal sensor reflections (light reflections from the interior of sensor 2, caused by light reflection at the viewing window of sensor 2 in the transmission path), - enhanced optical feedback (increased intensity and / or density of measuring points M I compared to a non-contaminated viewing window of sensor 2; also sensor-specific and to be applied and / or calibrated accordingly) from the immediate near field of sensor 2 (i.e. up to about 0.5 m, depending on the depth resolution of sensor 2), - no optical feedback from the environment over several measurement cycles. (2) Criteria for measured values ​​of the non-contaminated state NK (inverse to (1)): - low measured values ​​for internal reflections, - reduced optical feedback from the immediate near field of sensor 2, - continuous visual feedback from the environment.

[0041] In addition to the conditions (1) and (2) imposed on the measured value mij in the cell Rij, a further criterion is introduced to distinguish between contamination of the sensor 2 and missing optical feedback from the environment in a partial area of ​​the sensor field of view 3.

[0042] Missing optical feedback in the sub-area Rij of the sensor field of view 3 can have several causes: i. Contamination of sensor 2, which prevents sensor 2 from measuring points M in grid cell Rij Ifrom the environment. ii. Missing objects 4 in the environment that can reflect the laser beam and thus also missing measuring points M I in the grid cell Rij.

[0043] For reliable contamination detection, a distinction must now be made between cases i. and ii. For this purpose, the potential contaminations in the grid are observed over a period δt > Δt, and their movement within the grid is recorded. For a potential or confirmed contamination at the beginning of the observation period δt in the grid cell Rij (state potentially contaminated PK or contaminated K), a two-dimensional movement vector v Kont ij determines which can assume the following states: iii.v Kont ij = 0 if the contamination remains in the grid cell Rij during the period δt. iv.v Kontij ≠ 0 if the contamination migrates from grid cell Rij to grid cell Rkl during the period δt. The movement vector v Kont ij then contains information about the movement of the contamination in the defined grid.

[0044] In case iv., a distinction is necessary between the cases mentioned in i. and ii., since it is necessary to differentiate whether an actual contamination of sensor 2 changes its position in the grid (i.) or whether it is merely a lack of optical feedback from the observed environment (ii.).

[0045] For this purpose, another motion vector v is used for the direction of movement of the environment. Umgebung which is determined based on information on the vehicle's own movement 1 (externally provided or calculated based on detected static objects 4 in the environment).

[0046] By comparing the motion vectors v Kont ij and v UmgebungAnother criterion for the state transitions arises: (3) Criteria for the movement vectors for the states potentially contaminated PK and contaminated K: Vector v Kont ij ≠ v Umfeld : The directions of movement of the (potential) contamination and the surrounding area are different. Thus, the observed contamination in cell Rij can be identified as actual contamination. (4) Criteria movement vectors for the states potentially contaminated PK and contaminated K: Vector v Kont ij = v Umfeld : The movement directions of the (potential) contamination and the surrounding area are identical. Thus, the observed potential contamination in cell Rij can be identified as a lack of optical feedback from the surrounding area, and contamination can be ruled out.

[0047] Fig.Figure 5 is a schematic view of a state diagram showing which state transitions are possible for the grid cells Rij of the second sensor.

[0048] The respective state transitions also occur in this case if the conditions indicated by the arrows are met. This means that the conditions for state transitions apply analogously to the first sensor 2.

[0049] Here, the potentially contaminated PK state serves as additional confirmation during the state transition Zba for the first sensor 2.

[0050] The technical definitions of these conditions are described in detail below. a) Zaa: For the grid cell Rij, the measured values ​​in the time step from t1 to t2=t1+Δt continue to fulfill the conditions mentioned in (1) and thus confirm the previously detected contamination. b) Zac: For grid cell Rij, the measured values ​​in the time step from t1 to t2=t1+Δt satisfy the conditions stated in (2). Thus, the previously detected contamination is no longer present. c) Zab: The state transition is only possible in conjunction with at least a second optical sensor.

[0051] For grid cell Rij, the measured values ​​of the second (and / or at least one other) sensor cannot confirm the previously detected contamination (conditions (2) are met for the second (and / or at least one other) sensor). d) Zbb: The detection function is still within the observation period Δt. A potentially contaminated PK state previously identified for grid cell Rij thus remains marked as such. After a completed observation period Δt, a potentially contaminated PK state is not intended. e) Zba: For grid cell Rij, the measured values ​​in the time step from t1 to t2 = t1 + Δt satisfy the conditions in (1). The previously detected potentially contaminated state PK is thus confirmed as the actual contaminated state K. The confirmation is based on I. and / or II.: I. Condition (3) is fulfilled (vector v Kont ij ≠ v Umfeld ) II. Condition (2) is met for the second (and / or at least one additional) sensor. The second (and / or at least one additional) sensor no longer confirms potential contamination at the same location in the grid. f) Zbc: For grid cell Rij, the measured values ​​in the time step from t1 to t2 = t1 + Δt satisfy the conditions described in (2). The previously detected potentially contaminated state PK is thus not confirmed as the actual contaminated state K. This state is removed based on I. (necessary) and ii. (sufficient): I. Condition (4) is satisfied. (Vector v Kont = v Umfeld ) II. Conditions (1) and (4) are met for the second (and / or at least one additional) sensor. The second (and / or at least one additional) sensor confirms the potentially contaminated PC status at the same location and with identical movement in the grid. g) Zcc: For grid cell Rij, the measured values ​​in the time step from t1 to t2=t1+Δt satisfy the conditions in (2). This confirms that the uncontaminated NK state still exists. h) Zcb: For the grid cell Rij, previously identified as uncontaminated, the measured values ​​in the time step from t1 to t2=t1+Δt satisfy the conditions in (1). Thus, it can be concluded that the potentially contaminated PK state now exists.

[0052] To avoid false positive detection of contamination, the detection logic can be optimized in three steps.

[0053] Step 1: The grid of the sensor field of view 3 is optimized for the detection of contamination: The grid of the sensor field of view 3 is defined based on the angular resolution of the sensor 2 (e.g., a rectangular grid with a resolution of 0.1°×0.1° if the angular resolution of the sensor is 0.05°×0.05°). It is expected that the contamination will not change its position significantly over time. Therefore, a time-based criterion is used, which assumes that the contamination in the grid remains constant over a certain period of time (e.g., 5 s of observation time). In this way, the spatial grid used for detection can be reduced and / or optimized, as the time-based criterion prevents undesirable false positive behavior.

[0054] Step 2: To account for slight movements of the contamination, a velocity vector is calculated for each assumed contamination to track its movement. Based on predefined thresholds (e.g., 0.2° / s), this vector is interpreted either as contamination or as a raster section with no visual feedback from the environment.

[0055] Step 3: Since most of the sensors considered are operated in the context of a driver assistance system containing multiple such sensors, contamination detection is performed in a second step using fusion. In this step, the velocity vectors of one of the sensors are compared with the corresponding velocity vectors of one, several, or all other sensors (active or passive). QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2020 115 252 A1

[0003]

Claims

[1] Method for detecting contamination on a protective screen of an optical sensor (2), wherein a sensor field of view (3) of the sensor (2) is or will be divided into a grid with a plurality of grid cells (Rij), characterized by that each grid cell (Rij) is assigned one of the states contaminated (K), potentially contaminated (PK) or non-contaminated (NK) depending on measurements of the sensor (2) which are carried out in a solid angle of the respective grid cell (Rij), wherein the states are updated in successive time steps depending on the respective measurement of the sensor (2), wherein for the update for each grid cell (Rij) to which one of the states contaminated (K) or potentially contaminated (PK) is assigned, a movement vector (v Kont ij) of the respective assumed contamination over a given period of time (δt), and with a determined movement vector (v Umgebung) of a static object (4) located in the solid angle of the respective grid cell (Rij) in the environment. [2] Method according to claim 1, characterized by that the respective grid cell (Rij) is assigned the status contaminated (K) or potentially contaminated (PK) in the case of increased measured values ​​for sensor-internal reflections and / or in the case of increased optical feedback from the immediate near field of the sensor (2) in comparison with grid cells (Rij) classified as non-contaminated and / or in the case of lack of optical feedback from the environment over several measuring cycles, and otherwise the status non-contaminated (NK). [3] Method according to claim 1 or 2, characterized by that the respective grid cell (Rij) when the motion vector (v Kont ij) of the respective assumed contamination unequal to the movement vector (v Umgebung) of the static object (4) located in the solid angle of the respective grid cell (Rij) in the environment, the state contaminated (K) is assigned and otherwise the state not contaminated (NK) is assigned. [4] Method according to one of claims 2 or 3, characterized bythat at least one further optical sensor is used, the sensor field of view (3) of which is or will be divided into a grid with a multiplicity of grid cells (Rij), wherein each grid cell (Rij) of the further sensor is assigned one of the states contaminated (K), potentially contaminated (PK) or non-contaminated (NK) depending on measurements by the sensor which are carried out in a solid angle of the respective grid cell (Rij), wherein the states are updated in successive time steps depending on the respective measurement of the sensor (2), wherein the state of the grid cell (Rij) of the first sensor (2) is changed from contaminated (K) to potentially contaminated (PK) if a grid cell (Rij) of the further sensor which is congruent with this grid cell (Rij) is classified as non-contaminated (NK) according to the criteria specified in claim 2. [5] Method according to one of claims 2 to 4, characterized bythat a grid cell (Rij) of the first sensor (2) classified as potentially contaminated (PK) is classified as contaminated (K) if the previously detected state is confirmed according to the criteria of claim 2 or 3 in the subsequent time step and / or if a grid cell (Rij) of the further sensor that is congruent with this grid cell (Rij) is classified as non-contaminated (NK) according to the criteria specified in claim 2. [6] Method according to one of claims 2 to 5, characterized by that a grid cell (Rij) of the first sensor (2) classified as potentially contaminated (PK) is classified as non-contaminated (NK) if a grid cell (Rij) of the further sensor that is congruent with this grid cell (Rij) is classified as potentially contaminated (PK) according to the criteria specified in claim 2 and / or 3. [7] Driver assistance system for a vehicle (1), comprising at least one optical sensor (2), characterized by that the driver assistance system is configured to carry out the method according to one of the preceding claims. [8] Driver assistance system according to claim 7, characterized by that the optical sensor is designed as a camera or as a lidar sensor. [9] Vehicle (1) comprising a driver assistance system according to one of claims 7 or 8. [10] Vehicle (1) according to claim 9, characterized by that the vehicle is designed as an autonomous or semi-autonomous vehicle (1).

Citation Information

Patent Citations

  • Method and apparatus for detecting contaminants on a protective screen of a lidar sensor

    DE102020115252A1

  • Cross-validating sensors of an autonomous vehicle

    US9555740B1