METHOD FOR DETECTING AN OPTICAL BLOCKAGE OF A VEHICLE CAMERA AND METHOD FOR OPERATING A VEHICLE

Description

[0001] The invention relates to a method for detecting an optical blockage of a camera of a vehicle and a method for operating a vehicle.

[0002] DE 10 2019 002 487 A1 discloses a method for testing an environment detection sensor of a vehicle. The vehicle is located on a digital map, and features of stored stationary objects in the vehicle's surroundings that are expected to be detected by the environment detection sensor are identified in the digital map. Furthermore, the vehicle's surroundings are detected using the environment detection sensor, and degradation of the environment detection sensor is inferred if expected features are not detected by the environment detection sensor, or if features actually detected by the environment detection sensor deviate significantly from the expected features. The digital map, along with the stored features of the stationary objects, is retrieved from a central storage unit external to the vehicle for a specified look-ahead horizon.

[0003] From DE 10 2018 122 725 A1 a method for detecting a blockage of an optical image sensor of a vehicle is known, in which a recorded image of a predetermined object is compared with a target image of the predetermined object and image noise is determined and in which, if the image noise exceeds a permissible limit, cleaning of a lens of the image sensor is initiated.

[0004] The invention is based on the object of providing a novel method for detecting an optical blockage of a camera of a vehicle and a novel method for operating a vehicle.

[0005] The object is achieved according to the invention by a method for detecting an optical blockage of a camera of a vehicle, which has the features specified in claim 1, and a method for operating a vehicle, which has the features specified in claim 10.

[0006] Advantageous embodiments of the invention are the subject of the subclaims.

[0007] In a method for detecting an optical blockage of a camera of a vehicle according to claim 1, an image of a static scene from the surroundings of the vehicle is recorded with the camera. Static scene means in particular that there are no dynamic objects in the scene, thus in particular no objects that are moving or capable of movement, in particular no other road users such as other vehicles or pedestrians. In addition, the recording conditions prevailing while the image is being recorded. The recorded image is compared with a reference image, i.e. with an image that can be expected when the static scene is recorded using the camera. This reference image is retrieved from a server external to the vehicle. It is based on images taken by a plurality of vehicles of the same static scene under the same or similar recording conditions.Similar recording conditions are understood to mean recording conditions that deviate from the recording conditions during the image capture only within specified limits. If deviations in image features are detected between the captured image and the reference image and these deviations exceed a specified tolerance, these deviations are considered an indication of optical blockage of the camera.

[0008] For camera-based assistance systems in the vehicle, this detection of an optical blockage of the camera, for example due to dirt, is essential.

[0009] However, image processing algorithms have difficulty detecting this reliably under certain environmental conditions. This can lead to reduced system availability, particularly in the case of an incorrectly detected optical blockage that is not actually present, as well as to critical system behavior due to, for example, other road users not being detected, particularly in the case of an incorrectly undetected optical blockage that is actually present. The method according to the invention solves this problem by enabling improved and more robust detection of an optical blockage of the camera.

[0010] Recording conditions include, for example, a location, a time of day, a lighting situation, weather information, the orientation of the vehicle, and / or the camera's field of view. This allows the appropriate reference image to be determined with a high degree of certainty.

[0011] Image features considered include brightness, the number of detected edges, the sharpness of the detected edges, homogeneity, and / or color information. This allows a comparison of the captured image with the reference image based on a variety of different features, allowing for reliable detection of any optical blockage in the camera.

[0012] The reference image is based in particular on averaged image information from the recordings taken by a majority of vehicles of the same static scene under the same or similar recording conditions. This is based on the knowledge that optical blockage of vehicle cameras, e.g. in the form of dirt, is a phenomenon that only occurs occasionally. If an image content, in particular at a given point in a special scenario, is averaged across several recordings taken at different times, i.e. on different days, but advantageously at the same time of day, with the same recording conditions, then isolated optical blockage situations are averaged out. In this way, a reliable reference image is created for comparison with the currently recorded image.

[0013] For example, the image features taken into account include the brightness in the entire image and in the entire reference image and / or the brightness in individual image regions of the image and the reference image and / or the number of detected edges in the entire image and in the entire reference image and / or the number of detected edges in individual image regions of the image and the reference image and / or the sharpness of the detected edges in the entire image and in the entire reference image and / or the sharpness of the detected edges in individual image regions of the image and the reference image and / or the homogeneity in individual image regions of the image and the reference image and / or averaged color information across individual image regions of the image and the reference image. The image regions can be the same as or different from the image regions.This enables a reliable comparison of the captured image with the reference image in order to reliably detect any optical blockage of the camera.

[0014] For example, a lower number of detected edges in individual image regions of the image than the number of detected edges in the corresponding image regions of the reference image is considered an indication of optical blockage of the camera. Alternatively or additionally, a greater homogeneity of the entire image than the homogeneity of the entire reference image exceeding the specified tolerance is considered an indication of optical blockage of the camera. Alternatively or additionally, this can also be done in a corresponding manner with the other features mentioned above.

[0015] In one possible embodiment, the captured image is transmitted to the off-vehicle server and added to the images captured by the plurality of vehicles of the same static scene under the same or similar recording conditions. This increases the number of images on which the reference image is based, thus improving, in particular, the quality and reliability of the reference image.

[0016] If there is an indication of an optical blockage of the camera, it can be provided, for example, that an alternative method for detecting the optical blockage of the vehicle's camera is subsequently carried out. This alternative method is, in particular, a so-called online blockage detection. For example, the alternative method is already preconditioned with the results determined in the manner described above, in particular in order to then examine a potential optical blockage in more detail in relevant image areas. This alternative method therefore no longer has to examine an entire image, but only one or more relevant image areas in which an indication of an existing optical blockage of the camera has been determined. This is carried out in particular when there is only a predetermined small deviation of the image features between the recorded image and the reference image, i.e.especially if this deviation does not exceed specified limits.

[0017] In a method according to the invention for operating the vehicle according to claim 10, which is designed to carry out automated driving, in particular highly automated or autonomous driving, the above-described method for detecting an optical blockage of the vehicle's camera is carried out. If there is an indication of an optical blockage, the degree of automation of the driving operation is reduced or a driving task for driving the vehicle is transferred to a driver of the vehicle. This is carried out in particular when there is an excessive deviation of the image features between the recorded image and the reference image, iein particular if this deviation exceeds specified limit values, in particular in the case of a large and clear deviation, so that an additional test using an alternative method for detecting the optical blockage of the vehicle's camera is no longer necessary, since the existing deviation can be used to conclude with certainty or with a high degree of probability that there is actually an optical blockage of the camera.

[0018] To ensure availability and safety in automated driving, a certain level of camera performance for capturing the vehicle's surroundings must be ensured, particularly with regard to visibility, accuracy, and error rate, to ensure reliable automated operation of the vehicle. However, a blocked camera, especially one that is blocked too severely, does not meet these performance requirements, resulting in reduced availability of a vehicle function and / or an unsafe system state with a high risk of accident. The method makes it possible to check the camera so that limited performance can be reliably detected. Depending on the performance, the degree of automation of the automated driving operation can be adjusted or the driving task can be completely transferred to the driver.This ensures safe vehicle operation at all times. This means that the process can be used to prevent potentially dangerous situations, such as the failure or delay in detecting dynamic and stationary objects in the vehicle's surroundings, caused by undetected drops in the camera's performance during environment detection. This means that the process can detect an error caused by a blockage in the vehicle's camera, and initiate supportive system responses, such as slowing down the vehicle's speed or even an emergency stop.

[0019] If the driver fails to take over the driving task, the vehicle is, for example, brought to a standstill within a specified period of time, or first guided to the edge of the road or hard shoulder within a specified period of time and then brought to a standstill. This significantly increases the safety of the vehicle, its occupants, and other road users. If the vehicle is first guided to the edge of the road or hard shoulder and only then brought to a standstill, the risk of collision with other road users is further reduced, thereby further increasing safety.

[0020] Embodiments of the invention are explained in more detail below with reference to a drawing.

[0021] It shows: Fig. 1 schematically shows a sequence of a method for detecting an optical blockage of a camera of a vehicle.

[0022] In the only Figure 1 a sequence of a possible embodiment of a method for detecting an optical blockage of a camera of a vehicle 1 is shown.

[0023] In this method, vehicle 1 uses its camera to capture an image B of a static scene from the surroundings of vehicle 1. A static scene means, in particular, that no dynamic objects are present in the scene.

[0024] In addition, the recording conditions prevailing in the vehicle 1 during the recording of image B are determined and transmitted, in particular as a scenario description, to a server 2 external to the vehicle. The image B recorded by the camera is advantageously also transmitted to the server 2 external to the vehicle.

[0025] A reference image is retrieved from the vehicle-external server 2 by the vehicle 1, in particular according to the scenario description, i.e., according to the recording conditions, and thus transmitted to the vehicle 1. The reference image is based on images taken by a plurality of vehicles of the same static scene under the same or similar recording conditions. The reference image is based in particular on averaged images of the same scenarios, in particular on averaged image information from the images taken by the plurality of vehicles of the same static scene under the same or similar recording conditions.

[0026] The connection of the vehicle 1 with the vehicle-external server 2 is in particular a radio connection, in particular a mobile radio connection, for example LTE.

[0027] In a comparison step V of the method, the captured image B is compared with the reference image. The reference image represents the expected image, i.e., an image that can be expected when capturing the static scene using the camera.

[0028] If deviations in image features are detected between the captured image B and the reference image and these deviations exceed a specified tolerance level, these deviations are interpreted as an indication of optical blockage of the camera. Corresponding blockage information BI is then generated, which can subsequently be used by vehicle 1, for example, in the method described here or in another method, as described below.

[0029] A possible procedure is described in detail below.

[0030] Using this method, vehicle 1 attempts to detect at a specific position whether the camera, in particular a camera sensor for capturing the external environment of vehicle 1, is currently impaired by an optical blockage. The current scenario, i.e., the currently present static scene, is defined in particular by the recording conditions.

[0031] These recording conditions include, for example, a current location of the vehicle 1, in particular an exact position of the vehicle 1 determined by means of a global navigation satellite system, a current orientation of the vehicle 1, a current field of view of the camera, a current time, a currently existing lighting situation, for example a status of an exterior light of the vehicle 1, cloud cover and / or sunlight and / or information from a light sensor of the vehicle 1, current weather information, for example whether it is raining or sudden falling or whether it is dry, and of course the information that no dynamic objects, in particular other road users, for example other vehicles or pedestrians, are present in the field of view of the camera, because only then is a static scene present.The timeliness of this information regarding the recording conditions refers to the time at which image B is taken by the camera.

[0032] The corresponding reference image is retrieved from the vehicle-external server 2 and thus in particular from a database located thereon, also referred to as an online database, for comparison with the image B recorded by the camera. As already mentioned, the reference image is based in particular on averaged image information of the images recorded by the plurality of vehicles of the same static scene under the same or similar recording conditions. It is therefore based in particular on averaged image information from the same or similar, in particular exactly the same, scenarios as the static scene of the recorded image B. The database on the vehicle-external server 2 advantageously consists of images uploaded by the plurality of vehicles, i.e. transmitted to the vehicle-external server 2, which are in particular clustered, i.e. in particular classified according to different scenarios, i.e.according to the different static scenes and their respective recording conditions.

[0033] As already mentioned, the image B currently recorded by the vehicle 1 is advantageously also transmitted to the vehicle-external server 2 and thereby included in this database in order to increase the basis for the respective reference image. This advantageously only takes place after the comparison of the recorded image B with the reference image has been carried out. For example, this can avoid including inappropriate images B in the database, in particular images B for which the comparison has provided an indication of the optical blockage of the camera or at least images B for which the comparison has shown that there is an excessively large and clear deviation of the image features between the recorded image B and the reference image, which deviation in particular exceeds predetermined limit values, so that it can be concluded with certainty or with a high probability that an actual optical blockage of the camera is present.This can improve the reliability of the reference image.

[0034] The comparison of image B captured by the camera of vehicle 1 with the reference image retrieved from the vehicle-external server 2 in comparison step V of the method consists in particular of an evaluation of the image characteristics such as brightness, number of detected edges, sharpness of the detected edges, homogeneity, and / or color information. These image characteristics are also referred to as image information.

[0035] For example, the comparison of the image B recorded by the camera of the vehicle 1 with the reference image retrieved from the vehicle-external server 2 in comparison step V of the method consists of the evaluation of the following image features: the brightness in the entire image B and in the entire reference image and the brightness in individual image regions of image B and the reference image, the number and sharpness of the detected edges in the entire image B and in the entire reference image and in individual image regions of image B and the reference image, the homogeneity of image B and the reference image in individual image regions, the averaged color information over individual image regions of image B and the reference image.

[0036] From the comparison described above, a robust conclusion can be drawn as to whether the image B currently captured by the camera is affected by an optical blockage of the camera. As described, deviations in the image features between the captured image B and the reference image, if they exceed the specified tolerance level, are considered an indication of an optical blockage of the camera.

[0037] For example, if the number of detected edges in individual image areas of image B is significantly lower than the averaged value from the database, i.e., than the number of detected edges in the individual image areas of the reference image, this indicates an optical blockage in these areas.

[0038] For example, if the homogeneity of the entire image B is greater than the averaged value from the database, i.e., the homogeneity of the entire reference image, this indicates an optical blockage.

[0039] Thus, for example, a lower number of detected edges in individual image regions of image B that exceeds the specified tolerance level compared to the number of detected edges in the corresponding image regions of the reference image is considered an indication of optical blockage of the camera, and / or a greater homogeneity of the entire image B that exceeds the specified tolerance level compared to the homogeneity of the entire reference image is considered an indication of optical blockage of the camera. Alternatively or additionally, this can also be done in a corresponding manner with the other features mentioned above.

[0040] The result of the comparison step V, ie the blockage information BI, in particular if it contains the indication of the optical blockage of the camera, is used, for example, to subsequently additionally carry out an alternative method for detecting the optical blockage of the camera of the vehicle 1, in particular an online blockage detection, and in particular to precondition this in order to examine a potential optical blockage in more detail in the relevant image areas, in particular in the case of minor deviations in the comparison of the image B recorded by the camera with the reference image.

[0041] Alternatively, the result of the comparison step V, ie the blockage information BI, in particular if it contains the indication of the optical blockage of the camera, is used, for example, to directly conclude that there is an optical blockage situation and to switch off corresponding assistance systems of the vehicle 1, in particular in the case of large, clear deviations in the comparison of the image B recorded by the camera with the reference image.For example, in a method for operating the vehicle 1, which is designed to carry out an automated driving operation, in particular a highly automated or autonomous driving operation, it is provided that the above-described method for detecting an optical blockage of the camera of the vehicle 1 is carried out, wherein, if there is an indication of an optical blockage, a degree of automation of the driving operation is reduced or a driving task for driving the vehicle 1 is transferred to a driver of the vehicle 1. This is carried out in particular when there is an excessive deviation of the image features between the recorded image B and the reference image, iein particular if this deviation exceeds specified limit values, in particular in the case of a large and clear deviation, so that an additional test using an alternative method for detecting the optical blockage of the camera of the vehicle 1 is no longer necessary, since the existing deviation can be used to conclude with certainty or with a high degree of probability that there is actually an optical blockage of the camera.

[0042] If the driver fails to take over the driving task, it is intended, for example, that vehicle 1 is brought to a standstill within a specified period of time or is first driven to the edge of the road or a hard shoulder within a specified period of time and then brought to a standstill.

Claims

1. Computer-implemented method for detecting an optical blockage of a camera of a vehicle (1), characterized in that - the camera is used to capture an image (B) of a static scene from an environment of the vehicle (1), - capturing conditions present during the capturing of the image (B) are determined, - the captured image (B) is compared with a reference image which is retrieved from a vehicle-external server (2) and is based on captures taken by a plurality of vehicles of the same static scene under the same or similar capturing conditions, - deviations in image features between the captured image (B) and the reference image, if they exceed a specified tolerance level, are considered as an indication of the optical blockage of the camera.

2. Method according to claim 1, characterized in that a location, a time, a lighting situation, weather information, an orientation of the vehicle (1) and / or a field of view of the camera are determined as capturing conditions.

3. Method according to any one of the preceding claims, characterized in that a brightness, a number of detected edges, a sharpness of the detected edges, a homogeneity and / or color information are taken into account as image features.

4. Method according to claim 3, characterized in that the following are taken into account as image features: - the brightness in the entire image (B) and in the entire reference image and / or the brightness in individual image regions of the image (B) and the reference image, - the number of detected edges in the entire image (B) and in the entire reference image and / or the number of detected edges in individual image regions of the image (B) and the reference image, - the sharpness of the detected edges in the entire image (B) and in the entire reference image and / or the sharpness of the detected edges in individual image regions of the image (B) and the reference image, - the homogeneity in individual image regions of the image (B) and the reference image, and / or - averaged color information over individual image regions of the image (B) and the reference image.

5. Method according to any one of the preceding claims, characterized in that the reference image is based on averaged image information of the captures taken by the plurality of vehicles of the same static scene under the same or similar capturing conditions.

6. Method according to any one of claims 3 to 5, characterized in that a lower number of detected edges in individual image regions of the image (B), which exceeds the specified tolerance level, compared to the number of detected edges in the corresponding image regions of the reference image is considered as an indication of the optical blockage of the camera.

7. Method according to any one of claims 3 to 6, characterized in that a greater homogeneity of the entire image (B), which exceeds the specified tolerance level, compared to the homogeneity of the entire reference image is considered as an indication of the optical blockage of the camera.

8. Method according to any one of the preceding claims, characterized in that the captured image (B) is transmitted to the vehicle-external server (2) and is added to the captures taken by the plurality of vehicles of the same static scene under the same or similar capturing conditions.

9. Method according to any one of the preceding claims, characterized in that, if there is an indication of the optical blockage of the camera, an alternative method for detecting the optical blockage of the camera of the vehicle (1) is subsequently carried out in addition.

10. Method for operating a vehicle (1) which is designed to carry out an automated driving operation, wherein a method according to any one of the preceding claims is carried out and, if there is an indication of an optical blockage of a camera of the vehicle (1), a degree of automation of the driving operation is reduced or a driving task is handed over to a driver of the vehicle (1).