Method for operating a motor vehicle camera, as well as motor vehicle camera, motor vehicle and computer program product

Abstract

According to the invention, a sequence of successive images (B1, B2) is captured using a vehicle camera (2). In a first image (B1) at a first time point (t1), an area (12) is detected in which the recognition of an environmental detail (8) is impaired. From a second image (B2), which was captured at a second time point (t2) prior by a predetermined time offset (x), an image section (A2) is extracted such that the environmental detail (8) is contained in at least approximately the same area of ​​the image section (A2) as in the first image (B1). The first image (B1) is compared with the corresponding image section (A2) of the second image (B2). A further comparison is performed analogously at a subsequent time point.This determines whether the detection of the environmental detail (8) is impaired due to permanent contamination of the vehicle camera (2) or environmental obscuration of the environmental detail (8).

Description

[0001] The invention relates to a vehicle camera and a method for operating such a vehicle camera. The invention also relates to a vehicle equipped with such a vehicle camera and a computer program product.

[0002] Besides parking assistance functions, vehicle cameras are also used for other purposes, such as providing support while driving, for example, for sign recognition, safety functions (e.g., pedestrian warning), and the like. It is therefore crucial that the vehicle camera, and especially its functionality, is robust against influences such as contamination. For example, contamination in one area of ​​the image can prevent the surroundings from being captured. If it can be detected that the vehicle camera, particularly its lens, is contaminated, compensation can be applied if necessary.

[0003] DE 10 2012 015 282 B4 describes a method for detecting a concealed state of an image acquisition device of a motor vehicle, a camera system for this purpose, and a motor vehicle with such a camera system. According to the method, successive images are divided into image cells based on a fixed matrix, and for each image cell, it is determined by comparing the corresponding image cells between the two images whether the camera system is concealed.

[0004] EP 2 871 101 A1 describes an environmental monitoring system for a motor vehicle. This system includes a decision device that determines whether there is contamination on a camera lens of the environmental monitoring system.

[0005] The invention is based on the objective of enabling robust operation of a motor vehicle camera.

[0006] This problem is solved according to the invention by a method for operating a motor vehicle camera with the features of claim 1. Likewise, this problem is solved according to the invention by a motor vehicle camera with the features of claim 7, a motor vehicle with the features of claim 8, and a computer program product with the features of claim 9. Further advantageous and partly inventive embodiments and developments of the invention are set forth in the dependent claims and the following description.

[0007] The method according to the invention serves to operate a vehicle camera. According to the method, a sequence of temporally successive images is first captured by the vehicle camera. In a first image, which is captured at a first time, an area is detected in which the recognition of an environmental detail is impaired (or made more difficult). From a second image, which was captured at a second time offset by a predetermined time interval, a section of the image is then extracted such that the environmental detail is contained in at least approximately the same area of ​​the image as in the first image. Subsequently, the first image is compared with the corresponding image section of the second image, at least with respect to the area containing the environmental detail.Furthermore, a similar procedure is carried out with a third image, taken at a different time than the first and second, and a fourth image, captured at a predetermined time interval preceding (and, in particular, following) the third. This means that the area corresponding to the first image in the third image is compared with the corresponding area in the fourth image, specifically a section of the fourth image. The section of the fourth image is selected such that another environmental detail located in the third image (corresponding to the area of ​​the first image) is contained in at least approximately the same area of ​​the image section as in the third image.This allows for a distinction to be made as to whether the recognition of the environmental detail, in particular the first image, is impaired due to permanent contamination of the vehicle camera or an environmental obscuration of the environmental detail.

[0008] Put simply, the potentially disrupted area is identified in the first image. The second image is then chosen, for example, the one taken one second earlier – at a driving speed of 10 m / s, the environmental details in the second image are therefore 10 m "lower" than in the first. The image frame is then selected so that the relevant environmental detail from the first image, such as a traffic sign, is located in the same position in the second image, i.e., its section of the frame. Figuratively speaking, the second image is "zoomed in." If the disruption is caused by contamination of the camera's optics (e.g., a cover glass, lens, or similar component), the disruption in the second image, specifically its section of the frame, is highly unlikely to be located at the same position as the environmental detail.To continue with the example, the traffic sign in the second image should not be obstructed, or at least not in the same way as in the first image (e.g., partially obscured). If this observation is repeated, for example, at two later times (e.g., the third image is taken two seconds after the first and the fourth image one second before the third), it can also be determined with a higher probability whether the obstruction is on the vehicle camera (its lens) or in the surrounding environment. For example, a pedestrian who partially obscures the traffic sign in the first image might no longer have any effect in the third image, especially compared to the fourth image. This information can be used to advantage in interpreting future images.

[0009] The terms "approximately the same area" or "the same location" are used here and in the following to mean, in particular, exactly the same area or, within technically reasonable limits (tolerance ranges), approximately the same area. For example, the aforementioned area of ​​the first image may be described by a certain degree of blurring, so that the surrounding detail in the image section of the second image may not be positioned exactly, but rather shifted by this blurring, e.g., by up to 10 or up to 20 pixels, compared to the first image.

[0010] Preferably, the respective time offset is chosen between 0.5 and 20 seconds, expediently dependent on the vehicle speed, in particular between approximately 1 and 3 seconds. For example, at low vehicle speeds the time offset can be chosen to be larger, since the vehicle camera travels correspondingly shorter distances in the same time and thus, with a short time offset, dirt could still obscure the environmental detail to be viewed.

[0011] In one practical variant of the procedure, the respective image and the corresponding (associated) image section (of the temporally preceding image, i.e., in particular the second and fourth images) are subtracted from each other for each comparison. This allows differences between the respective image and image section to be easily "visible".

[0012] Preferably, the respective comparison is represented in a difference matrix. In particular, this difference matrix contains information about where differences are located. The matrix preferably represents exactly or at least approximately the image area covered by the respective image and the associated image section.

[0013] By means of at least two comparisons, an aging of the comparison results is expediently created. The comparison data is thus aged. This advantageously supports a decision as to whether the vehicle camera is dirty or "only" obstructed by a visual obstruction in the surroundings. A (particularly permanent) dirt would always lead to a malfunction, at least approximately, in the same location and therefore would not "age" or would age only slowly.

[0014] Preferably, all comparisons, in particular those between the first and second images, the third and fourth images, and also subsequent comparisons, are represented in the aforementioned difference matrix, and in particular summed. This consolidates permanent contamination, which consequently always lies in the same image areas, and "clean" areas, whereas variable disturbances are attenuated over time and can also disappear from the difference matrix.

[0015] Preferably, to decide whether permanent contamination (disturbances) is present, a threshold comparison is performed on the difference matrix, so that only "sufficiently" consolidated (summed) differences in the difference matrix, which thus have a correspondingly high value, are also recognized as such disturbances.

[0016] Another advantage is that such a comparison described above between two images, specifically an image and a corresponding section of a temporally prior image, forms a deterministic approach (algorithm) and requires comparatively little computing power, thus conserving resources in terms of available computing capacity.

[0017] The vehicle camera according to the invention has a control unit (also referred to as a "controller") which is configured to carry out the method described above, in particular automatically.

[0018] The motor vehicle according to the invention comprises a vehicle camera and a control unit ("controller") configured to perform the method described above, in particular automatically. Optionally, the vehicle camera described above incorporates the control unit itself. Alternatively, the control unit is integrated into a higher-level control unit of the vehicle.

[0019] In a preferred embodiment, the control unit (the controller) is formed, at least in its core, by a microcontroller with a processor and a data memory, in which the functionality for carrying out the method according to the invention is implemented programmatically in the form of operating software (firmware), so that the method is carried out automatically when the operating software is executed in the microcontroller. Alternatively, within the scope of the invention, the controller can also be formed by a non-programmable electronic component, e.g., an ASIC, in which the functionality for carrying out the method according to the invention is implemented by circuitry.

[0020] The computer program product according to the invention comprises program code which, when executed on a computing unit (in particular the controller), configures (causes) it to carry out the method described above, in particular automatically.

[0021] An embodiment of the invention is explained in more detail below with reference to a drawing. The drawing shows: Fig. 1. A motor vehicle in a schematic side view, and Fig. 2 in four consecutive tiles schematically depicts a process for operating a vehicle camera.

[0022] Fig. Figure 1 schematically shows a motor vehicle 1. This vehicle has a motor vehicle camera (short: camera 2) which is directed forward in the main direction of travel and is equipped to detect obstacles, traffic signs, etc.

[0023] Furthermore, the motor vehicle 1 has a control unit 4 (also: “controller”).

[0024] Using camera 2, a multitude of images of the surroundings 6, located in front of the vehicle 1 in the main direction of travel, are captured during a journey by the vehicle 1. In order to be able to detect as reliably as possible whether the lens of camera 2 is dirty or whether there is an obstacle in the surroundings 6 that prevents the detection of a detail of the surroundings, the control unit 4 is configured to carry out a procedure described in more detail below.

[0025] In a first processing step S1, the control unit 4 of the camera 2 retrieves a first image B1 captured at a first time t1. In this first image B1, the control unit 4 searches for an environmental detail that is potentially obscured by dirt, specifically one that cannot be recognized or cannot be fully recognized. Here, the control unit 4 incorrectly identifies a traffic sign 8 as such an environmental detail due to a potential smudge 10. The control unit 4 marks an (image) area 12 around the smudge 10.

[0026] Subsequently, in a second process step S2, the control unit 4 retrieves a second image B2. This image was taken at a second time point t2 (corresponding to t1-x) earlier by a predetermined time offset x. Assuming a driving speed of 10 m / s, the time offset x is chosen to be, for example, one second. In this case, the vehicle 1 is therefore ten meters further away from the traffic sign 8.

[0027] In a third process step S3, a section of the image A2 is selected and enlarged from this second image B2 in such a way that the surrounding detail, here the traffic sign 8, is arranged in the same place in relation to the enlarged image as the area 12 in the first image B1.

[0028] In a fourth process step S4, the control unit 4 compares the first image B1 with the enlarged image section A2 from the third process step. Specifically, the control unit 4 subtracts image section A2 from the first image B1. Consequently, image differences remain. These image differences—apart from optical shifts—are located, in the present embodiment, in the spot 10 described above. This is because spot 10 is positioned at a different location relative to the image content in the second image B2 than in the first image B1. The result of the comparison, i.e., the subtraction, is represented in a difference matrix D.

[0029] This process is repeated continuously, resulting in several such comparisons occurring sequentially. Their results are then represented in the difference matrix, specifically summed with the preceding results. This causes the existing results to age. If the spot 10 remains in the same position on the camera 2's lens, it is entered in the difference matrix D in each comparison, specifically in area 12, and increases the value entered for that location. Disturbances that occur in only one or a few comparisons are thus attenuated in the difference matrix D over time. This allows for differentiation between whether the disturbance was only temporary or localized in the surrounding area 6, for example, a pedestrian, tree, or similar object that obscures the traffic sign 8 only in the first image B1. This makes it easy to keep a current record of disturbances from camera 2.

[0030] The subject matter of the invention is not limited to the embodiment described above. Rather, further embodiments of the invention can be derived by a person skilled in the art from the above description. REFERENCE MARK LIST: 1 motor vehicle 2 cameras 4 Control unit 6 Environment 8 traffic sign 10 spots 12 area B1 image B2 image t1 Time t2 time x time offset A2 image section D Difference matrix S1-4 Process Step

Claims

[1] Method for operating a motor vehicle camera (2) wherein according to the method - a sequence of temporally successive images (B1, B2) is captured by means of the vehicle camera (2), - in a first image (B1) at a first time point (t1) an area (12) is detected in which the detection of an environmental detail (8) is disturbed by searching for an environmental detail (8) that cannot be detected or cannot be fully detected, - from a second image (B2), which was captured at a second time point (t2) earlier by a given time offset, an image section (A2) is taken such that the surrounding detail (8) is contained in at least approximately the same area of ​​the image section (A2) as in the first image (B1), - the first image (B1) is compared with the corresponding image section (A2) of the second image (B2) at least with regard to the area (12) containing the environmental detail (8), - in a third image, which is taken at a third time point different from the first and second time points (t1, t2), the area corresponding to the area (12) of the first image (B1) is compared with the corresponding area of ​​a fourth image, which is captured at a fourth time point preceding the third time point by a predetermined time offset, and from which an image section is taken such that a further environmental detail, which is arranged in the area in the third image, is contained in at least approximately the same area of ​​the image section as in the third image, and - thereby making a distinction as to whether the recognition of the environmental detail (8) is disturbed due to a permanent contamination of the vehicle camera (2) or an environmental obscuration of the environmental detail (8). [2] Method according to claim 1, wherein for the respective comparison the respective image (B1, B2) and the corresponding image section (A2) are subtracted from each other. [3] Method according to claim 2, wherein the respective comparison is represented in a difference matrix (D). [4] Method according to any one of claims 1 to 3, wherein an aging of the comparison results is formed by means of the at least two comparisons. [5] Method according to claim 4 with reference to claim 3, wherein all comparisons are represented in the difference matrix (D), in particular summed. [6] Method according to any one of claims 1 to 5, wherein the respective time offset is selected, optionally depending on the driving speed, between 0.5 and 20 seconds, in particular between about 1 and 3 seconds. [7] Motor vehicle camera (2) with a control unit (4) configured to perform a method according to any one of claims 1 to 6, in particular automatically. [8] Motor vehicle (1) with a motor vehicle camera (2) and with a control unit (4) which is configured to carry out a method according to one of claims 1 to 6, in particular automatically. [9] Computer program product comprising program code which, when executed on a computing unit, configures it to carry out a method according to any one of claims 1 to 6, in particular automatically.

Images