Method for detecting optical blockage of a vehicle camera and method for operating a vehicle - Patents.com

Abstract

The invention relates to a method for detecting an optical obstruction of a camera of a vehicle (1), according to which an image (B) of a static scene is recorded from the surrounding environment of the vehicle (1) by means of said camera. During the recording of said image (B), current recording conditions are determined. The recorded image (B) is compared with a reference image obtained from an off-vehicle server (2) and based on recordings of the same static scene recorded under the same or similar recording conditions by a number of vehicles. If a deviation of image features between the recorded image (B) and the reference image exceeds a predefined tolerance, said deviation is evaluated as an indication of an optical obstruction of the camera. Furthermore, the invention relates to a method for operating a vehicle (1), which is configured to perform automated driving operations.

Description

[Technical field]

[0001] The present invention relates to a method for detecting an optical blockage of a camera of a vehicle and a method for operating the vehicle. [Background technology]

[0002] From DE 10 2019 002 847 A1 a method for testing a surroundings detection sensor of a vehicle is known. The vehicle is located in a digital map, in which stored static object features of the vehicle's surroundings, which are supposed to be identified by the surroundings detection sensor, are identified. The surroundings of the vehicle are further detected by the surroundings detection sensor, and a degradation of the surroundings detection sensor is estimated if the features supposed to be identified are not identified by the surroundings detection sensor or if the features actually identified by the surroundings detection sensor deviate significantly from the features supposed to be identified. The digital map with the stored static object features is retrieved from a central storage unit outside the vehicle for a given prediction horizon. Summary of the Invention [Problem to be solved by the invention]

[0003] It is an object of the present invention to provide a novel method of detecting optical blockage of a vehicle's camera and a novel method of operating the vehicle. [Means for solving the problem]

[0004] This object is achieved according to the invention by a method for detecting an optical blockage of a camera of a vehicle having the features of claim 1 and by a method for operating a vehicle having the features of claim 10.

[0005] Advantageous embodiments of the invention are the subject matter of the dependent claims.

[0006] According to the invention, in a method for detecting an optical blocking of a camera of a vehicle, an image of a static scene is recorded from the vehicle's surroundings by means of the camera. A static scene means, in particular, that there are no dynamic objects in the scene, thus in particular no moving or movable objects, in particular no other road users, such as other vehicles or pedestrians. Furthermore, during the recording of the images, the current recording conditions are determined. The recorded image is compared with a reference image, i.e. an image that is assumed when recording the static scene by means of the camera. This reference image is obtained (called up) from an off-vehicle server. The reference image is based on recordings of the same static scene recorded by a number of vehicles under the same or similar recording conditions. In particular, similar recording conditions are understood here to be recording conditions that deviate from the (current) recording conditions during the recording of the images only within predefined limits. Deviations of image features are ascertained between the recorded image and the reference image, and if these deviations exceed predefined tolerances, these deviations are evaluated as an indication of an optical blocking of the camera.

[0007] For a camera-based assistance system in a vehicle, this identification of optical obstructions of the camera, e.g. due to dirt, is essential. However, image processing algorithms have problems in robustly identifying optical obstructions in some ambient conditions. This can lead to both reduced availability of the system, especially in cases where a non-existent optical obstruction is mistakenly identified, and critical operation of the system, e.g. due to other road users not being identified, especially in cases where a non-existent optical obstruction is mistakenly not identified. The method according to the invention solves this problem, since it provides an improved and robust identification of the optical obstruction of the camera.

[0008] The recording conditions, for example the location, the time, the light conditions, the weather information, the orientation of the vehicle and / or the field of view of the camera, are determined, so that a suitable reference image can be determined with a high degree of certainty.

[0009] As image features, for example brightness, number of identified edges, sharpness of the identified edges, homogeneity and / or color information are taken into account, which allows the comparison of the recorded image with a reference image using several different features in order to reliably detect potentially existing optical blockages of the camera.

[0010] The reference image is based on the averaged image information of the recordings of the same static scene, recorded by several vehicles under the same or similar recording conditions. This is based on the recognition that optical obstructions of the vehicle camera, for example in the form of dirt, are events that occur only sporadically. If the image content is averaged over several recordings, particularly at a given location of a particular scenario, under the same recording conditions, at different times (hours), i.e. on different days, but preferably at the same time (hour) in each case, sporadic optical obstruction situations are averaged out. In this way, a reliable reference image is generated for comparison with the latest recorded image.

[0011] For example, the image features considered are the brightness in the entire image and in the entire reference image, and / or in the individual image regions of the image and the reference image, and / or the number of edges identified in the entire image and in the entire reference image, and / or the number of edges identified in the individual image regions of the image and the reference image, and / or the sharpness of the edges identified in the entire image and in the entire reference image, and / or the sharpness of the edges identified in the individual image regions of the image and the reference image, the homogeneity in the individual image regions of the image and the reference image, and / or the color information averaged over the individual image regions of the image and the reference image, where the image regions can be the same or different from the image regions. This allows a reliable comparison of the recorded image with the reference image in order to reliably detect any potential optical blockages of the camera.

[0012] For example, a number of edges identified in an individual image area of ​​the image is evaluated as an indication of an optical blockage of the camera if the number of edges identified in the corresponding image area of ​​the reference image is lower by more than a predetermined tolerance. Alternatively or additionally, for example, a homogeneity of the entire image is evaluated as an indication of an optical blockage of the camera if the homogeneity of the entire reference image is higher by more than a predetermined tolerance. Alternatively or additionally, this can also be implemented in a manner corresponding to the other features mentioned above.

[0013] In one possible embodiment, the recorded images are transmitted to an off-vehicle server and added to recordings of the same static scene recorded by multiple vehicles under the same or similar recording conditions, thereby increasing the number of recordings on which the reference image is based and thus improving, among other things, the quality and reliability of the reference image.

[0014] In the case of indications of optical blocking of the camera, an alternative method for detecting optical blocking of the vehicle camera can additionally be carried out subsequently. This alternative method is, in particular, a so-called online blocking identification. For example, this alternative method is already preconditioned with the results determined in the above-mentioned method, in particular in order to check the relevant image area for possible optical blocking in more detail. This alternative method therefore no longer requires checking the entire image, but instead only checks one or more relevant image areas in which indications of a present optical blocking of the camera have been determined. This is carried out in particular when there are only predefined small deviations of the image features between the recorded image and the reference image, i.e. in particular when the deviations do not exceed predefined limit values.

[0015] In the method according to the invention for operating a vehicle, which is configured to perform automated driving operations, in particular highly automated or autonomous driving operations, the above-mentioned method for detecting optical blockage of the camera of the vehicle is implemented. If there are indications of optical blockage, the automation level of the driving operation is reduced or the driving task for driving the vehicle is handed over to the driver of the vehicle. This is carried out in particular when there is an excessively large deviation of the image features between the recorded image and the reference image, i.e. in particular when this deviation exceeds a predefined limit value, in particular when the deviation is large and clear, so that from the existing deviation, the actually existing optical blockage of the camera can be deduced reliably or with a high probability, and therefore an additional check by an alternative method for detecting optical blockage of the camera of the vehicle is no longer necessary.

[0016] In order to guarantee reliable automated driving operation of the vehicle with respect to availability and safety during automated driving, certain performances of the camera for detecting the vehicle's surroundings must be guaranteed, in particular with respect to visibility, accuracy and error rate. However, blocked cameras, in particular cameras that are blocked too much, do not meet these performance requirements, which results in reduced availability of vehicle functions and / or an uncertain system state with a high risk of accidents. Using the method, a camera check can be realized, so that limited camera performance can be identified with high reliability and, depending on the performance, the automation level of the automated driving operation can be adapted or the driving task can be completely handed over to the driver. Thus, a safe driving operation of the vehicle is always provided. That is, using the method, it is possible to avoid potentially risky situations, such as dynamic and static objects not being detected or being detected too late in the vehicle's surroundings because a reduced performance of the camera during the detection of the surroundings is not identified. That is, the method can be used to identify error cases (errors, defects) due to a blocked vehicle camera, after which an assistance system response can be initiated, such as driving slowly until an emergency stop is made.

[0017] If the driver does not take over the driving task, for example, the vehicle is stopped within a predetermined time, or is first guided to the edge of the road or to the shoulder within a predetermined time and then stopped. This significantly increases the safety of the vehicle, the occupants in the vehicle, and other road users. If the vehicle is first guided to the edge of the road or to the shoulder and then stopped, the risk of collision with other road users in particular is further reduced, thereby further increasing safety.

[0018] In the following, embodiments of the invention will be explained in more detail with reference to the drawings. [Brief description of the drawings]

[0019] [Figure 1] 2 illustrates a schematic diagram of a method for detecting an optical blockage of a vehicle camera; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Only in FIG. 1 is the procedure of one possible embodiment of a method for detecting an optical blockage of a camera of a vehicle 1 shown.

[0021] In this method, a static scene image B is recorded from the vehicle 1's surrounding environment using a camera of the vehicle 1. A static scene here means in particular that no dynamic objects are present in the scene.

[0022] Furthermore, during the recording of image B, the current recording conditions are determined by vehicle 1 and transmitted, in particular as a scenario description, to off-vehicle server 2. Advantageously, image B recorded by means of the camera is also transmitted to off-vehicle server 2.

[0023] A reference image, in particular a reference image corresponding to the scenario description, i.e. corresponding to the recording conditions, is retrieved (called) by the vehicle 1 from the off-vehicle server 2 and transmitted to the vehicle 1. The reference image is based on recordings of the same static scene recorded under the same or similar recording conditions by multiple vehicles. The reference image is based in particular on averaged recordings of the same scenario, in particular on averaged image information of recordings of the same static scene recorded under the same or similar recording conditions by multiple vehicles.

[0024] The connection between the vehicle 1 and the off-vehicle server 2 is in particular a wireless connection, in particular a mobile wireless connection, for example LTE.

[0025] In a comparison step V of the method, the recorded image B is compared with a reference image, which represents an expected image, i.e. an image that is expected when recording a static scene with a camera.

[0026] If deviations of image features are ascertained between the recorded image B and the reference image, and if these deviations exceed a predefined tolerance, these deviations are evaluated as an indication of an optical obstruction of the camera. Corresponding obstruction information BI is then generated, which can then be used by the vehicle 1, for example in the manner described herein or in other ways, as will be described below.

[0027] In the following, possible method steps are explained in detail.

[0028] With this method, the vehicle 1 attempts to identify whether at a particular position the camera, in particular the camera sensor system for detecting the external surroundings of the vehicle 1, is currently impaired by an optical blockage or not. The current scenario, i.e. the currently existing static scene, is defined in particular by the recording conditions.

[0029] These recording conditions include, for example, the current location of the vehicle 1, in particular the exact position of the vehicle 1 determined using the Global Navigation Satellite System (GNSS), the current orientation of the vehicle 1, the current field of view of the camera, the current time, the currently existing light situation, for example the state of the exterior lights of the vehicle 1, cloud cover and / or solar radiation (sunlight) and / or information from the light sensors of the vehicle 1, current weather information, for example whether there is rain or snow (sudden rainfall) or whether it is dry or not, and of course the information that there are no dynamic objects, in particular no other road users, for example other vehicles or pedestrians, in the field of view of the camera, since only in such a case is there a static scene. Here, the timeliness of these information regarding the recording conditions relates to the point in time (time, hour) at which the image B is recorded by the camera, respectively.

[0030] From the off-vehicle server 2, and thus in particular from a data bank, also called an online data bank, present on the off-vehicle server 2, a corresponding reference image is obtained for comparison with the image B recorded by the camera. As already mentioned, the reference image is based in particular on averaged image information of recordings of the same static scene recorded by a number of vehicles under the same or similar recording conditions, and therefore in particular on averaged image information from an identical or similar, in particular identical scenario, to the static scene of the recorded image B. For this purpose, the data bank of the off-vehicle server 2 is advantageously made up of recordings uploaded, i.e. transmitted, by a number of vehicles, which are clustered, i.e. in particular divided, in accordance with different scenarios, i.e. in accordance with different static scenes and the respective recording conditions.

[0031] As already mentioned, advantageously, the image B currently recorded by the vehicle 1 is also transmitted to the off-board server 2 and recorded in this data bank in order to increase the basis of the respective reference image. This is advantageously done only after a comparison of the recorded image B with the reference image has been carried out. This makes it possible, for example, to avoid recording in the data bank unsuitable images B, in particular images B in which the comparison reveals indications of an optical blockage of the camera, or at least images B in which the comparison shows excessively large and clear deviations of the image characteristics between the recorded image B and the reference image, in particular deviations above a predefined limit value, so that it is possible to reliably or with a high probability to assume that an optical blockage of the camera actually exists. This makes it possible to increase the reliability of the reference image.

[0032] The comparison of the image B recorded by the camera of the vehicle 1 with the reference image obtained from the off-vehicle server 2 in method step V of the method consists in particular of the evaluation of the brightness of the image features, the number of identified edges, the sharpness of the identified edges, the homogeneity and / or the color information. These image features are also referred to as image information.

[0033] For example, in the comparison step V of the method, the comparison of the image B recorded by the camera of the vehicle 1 with the reference image obtained from the off-vehicle server 2 consists of the evaluation of the following image features: the brightness in the entire image B and in the entire reference image, and in the individual image regions of image B and of the reference image, the number and sharpness of edges identified in the entire image B and in the entire reference image, and in individual image regions of image B and of the reference image, the homogeneity of image B and the reference image in the individual image regions, - Colour information averaged over the individual image areas of image B and the reference image.

[0034] From the above comparison, it is possible to robustly determine whether the image B currently recorded by the camera is corrupted by an optical blockage of the camera. As explained, if deviations of image features between the recorded image B and the reference image exceed a predefined tolerance, these deviations are evaluated as an indication of an optical blockage of the camera.

[0035] For example, if the number of edges identified in individual image areas of image B is significantly lower than the average value from the databank, i.e. compared to the number of edges identified in individual image areas of the reference image, this indicates optical blockage in these areas.

[0036] For example, if the homogeneity of the whole image B is higher than the average value from the databank, ie the homogeneity of the whole reference image, this indicates an optical blockage.

[0037] Thus, for example, a number of edges identified in a particular image area of ​​image B that is less than a predefined tolerance compared to the number of edges identified in the corresponding image area of ​​the reference image is evaluated as an indication of an optical blockage of the camera, and / or, for example, a homogeneity of the whole image B that is more than a predefined tolerance compared to the homogeneity of the whole reference image is evaluated as an indication of an optical blockage of the camera. Alternatively or additionally, this can be implemented in a manner corresponding to the other features mentioned above.

[0038] The result of the comparison step V, i.e. the obstruction information BI, in particular if the obstruction information BI contains indications of an optical obstruction of the camera, is used, for example, to additionally subsequently carry out an alternative method for detecting an optical obstruction of the camera of the vehicle 1, in particular an online obstruction identification, and uses this obstruction information BI as a prerequisite for a more accurate check for potential optical obstructions in the relevant image range, in particular if the deviations when comparing the image B recorded by the camera with the reference image are small.

[0039] Alternatively, the result of the comparison step V, i.e. the blocking information BI, is used, for example, to directly estimate the optical blocking situation, especially if the blocking information BI comprises indications of optical blocking of the camera, and is used to switch off corresponding assistance systems of the vehicle 1, especially if the deviation of the comparison of the image B recorded by the camera with the reference image is large and clear. For example, in a method for operating a vehicle 1 configured to perform automated driving operations, in particular highly automated or autonomous driving operations, the above-mentioned method is implemented to detect optical blocking of the camera of the vehicle 1, providing that in case of indications of optical blocking, the automation level of the driving operations is reduced or the driving task for driving the vehicle 1 is handed over to the driver of the vehicle 1. This is done in particular when there is an excessively large deviation of the image features between the recorded image B and the reference image, i.e. in particular when this deviation exceeds a predetermined limit value, in particular when the deviation is large and clear, so that from the deviation present the actual presence of an optical blockage of the camera can be deduced reliably or with a high degree of probability, so that additional checks with alternative methods for detecting an optical blockage of the camera of vehicle 1 are no longer necessary.

[0040] If the driver does not take over the driving task, for example, the vehicle 1 is stopped within a predetermined time, or is first guided to the edge of the road or to the shoulder within a predetermined time and then stopped. [Prior art documents] [Patent documents]

[0041] [Patent Document 1] DE 102019002847

Claims

1. A method for detecting an optical blockage of a camera of a vehicle (1), comprising: - by means of said camera, an image (B) of a static scene is recorded from the surrounding environment of said vehicle (1), - during recording of said image (B), the current recording conditions are determined, - said recorded image (B) is compared with a reference image obtained from an off-vehicle server (2) and based on recordings of the same static scene recorded by a number of vehicles under the same or similar recording conditions, - if the deviation of image features between the recorded image (B) and the reference image exceeds a predefined tolerance, said deviation is evaluated as an indication of an optical blockage of the camera; A method comprising:

2. As the recording conditions, the location, the time, the light conditions, the weather information, the orientation of the vehicle (1) and / or the field of view of the camera are determined.

2. The method according to claim 1 , characterized in that

3. As image features, brightness, the number of identified edges, the sharpness of the identified edges, homogeneity and / or color information are considered.

2. The method according to claim 1 , characterized in that

4. As the image features, the brightness in the whole image (B), the brightness in the whole reference image and / or the brightness in individual image areas of the image (B) and of the reference image, the number of edges identified in the entire image (B) and in the entire reference image and / or in individual image regions of the image (B) and the reference image, the sharpness of the edges identified in the entire image (B) and in the entire reference image and / or in individual image regions of the image (B) and of the reference image, the homogeneity in the individual image areas of said image (B) and said reference image, and / or - color information averaged over the respective image areas of said image (B) and said reference image, is taken into consideration 4. The method according to claim 3, characterized in that

5. The reference image is based on averaged image information of the recordings of the same static scene recorded by the multiple vehicles under the same or similar recording conditions.

2. The method according to claim 1 , characterized in that

6. A number of edges identified in each image area of ​​the image (B) being less than the number of edges identified in the corresponding image area of ​​the reference image by more than the predetermined tolerance is evaluated as an indication of the optical blockage of the camera.

4. The method according to claim 3, characterized in that

7. A homogeneity across the image (B) greater than the predetermined tolerance compared to the homogeneity across the reference image is evaluated as an indication of an optical blockage in the camera.

4. The method according to claim 3, characterized in that

8. The recorded images (B) are transmitted to an off-vehicle server (2) and added to a recording of the same static scene recorded by the multiple vehicles under the same or similar recording conditions.

2. The method according to claim 1 , characterized in that

9. If there is an indication of optical obstruction of the camera, following said evaluation, an additional method for detecting optical obstruction of the camera of the vehicle (1) is performed, said method being different from said evaluation.

2. The method according to claim 1 , characterized in that

10. A method for operating a vehicle (1) configured to perform automated driving operations, the method comprising: a) implementing a method according to any one of claims 1 to 9; and b) reducing the level of automation of driving operations or handing over the driving task to a driver of the vehicle (1) in case of indications of optical blocking of a camera of the vehicle (1). A method comprising:

Images