Method for operating a first lighting device, a second lighting device and an optical sensor, control device, gated camera device and motor vehicle

Abstract

The invention relates to a method for operating a first lighting device (5.1), a second lighting device (5.2) and an optical sensor (7), wherein the control of the first lighting device (5.1), the second lighting device (5.2) and the optical sensor (7) is coordinated in time with one another, this coordinated control corresponds to an assigned visible distance range (15), a first image (19.1) is acquired by the optical sensor (7) by means of the coordinated control under illumination by means of the first lighting device (5.1), a second image (19.2) is acquired by the optical sensor (7) by means of the coordinated control under illumination by means of the second lighting device (5.2), a third image (19.3) is acquired by the optical sensor (7) without illumination by means of one of the lighting devices (5) and a difference image (19.4) is formed on the basis of the first image (19.1), the second image (19.2) and the third image (19.3).

Description

Technical Field

[0001] The present invention relates to a method for operating a first lighting device, a second lighting device and an optical sensor, a control device for performing the method, a gating camera device having such a control device, and a motor vehicle having such a gating camera device. Summary of the Invention

[0002] Methods for operating lighting devices and optical sensors are known. For example, US 5034810 A and US20180203122 A1 have both shown methods for operating lighting devices and gated camera devices. The disadvantage of these known methods is that, on the one hand, they only consider a single lighting device, and on the other hand, they are based on environments with low ambient lighting.

[0003] DE 102017204836 A1 also discloses a method in which two lighting devices are mounted on a motor vehicle in a positionally separate manner. Furthermore, DE 102020003199 A1 also illustrates a method in which the control of the lighting devices and the optical sensor or gated camera device is coordinated in time to create at least two consecutive images by means of the optical sensor.

[0004] However, all of these known methods are not applicable to sunlight conditions and / or conditions of strong solar radiation.

[0005] Additionally, from the book "Gated2Depth: Real-Time Dense Lidar From Gated Images" by Tobias Glubb et al. https: / / arxiv.org / pdf / 1902.04997.pdf I learned about a method for creating images containing real-time distance information. The problem is that this method can only be used within a range of no more than 80 meters. Summary of the Invention

[0007] Therefore, the object of the present invention is to provide a method for operating a first lighting device, a second lighting device and an optical sensor, a control device for performing the method, a gating camera device having such a control device, and a motor vehicle having such a gating camera device, wherein the disadvantages are at least partially eliminated and preferably avoided.

[0008] This task is accomplished by providing the teachings of this technology, especially the teachings of the independent claims and the embodiments disclosed in the dependent claims and the specification.

[0009] Specifically, this task is accomplished by providing a method for operating a first illumination device, a second illumination device, and an optical sensor, wherein the control of the first illumination device, the second illumination device, and the optical sensor is coordinated with each other in time, and this coordinated control is assigned a visible distance range. A first image is acquired by the optical sensor with the aid of the first illumination device under illumination by the coordinated control. A second image is acquired by the optical sensor with the aid of the second illumination device under illumination by the coordinated control. A third image is acquired by the optical sensor without illumination by any of the illumination devices. Furthermore, a difference image is formed based on the first image, the second image, and the third image.

[0010] In particular, the third image corresponds to the ambient image, especially the daylight image. Advantageously, by subtracting the third image, the influence of ambient light, and especially daylight, is eliminated from the first and second images. Therefore, a more stable and reliable evaluation of the difference images can be achieved.

[0011] A method for generating an image by means of time-coordinated control of at least one illumination device and an optical sensor, particularly a method known as a gated imaging method; in particular, the optical sensor is a camera, which is switched to sensitivity only within a specified finite time range, this is called "gated control", and the camera is therefore a gated camera. At least one illumination device is also correspondingly controlled in time only at specified selected time intervals to illuminate the object-side scene.

[0012] In particular, a predetermined number of light pulses are emitted by the first and second illumination devices, each preferably having a duration of 5 to 20 ns. The start and end of the exposure of the optical sensor are correlated with the number and duration of the emitted light pulses. Therefore, a certain visible distance range can be acquired by the optical sensor at a correspondingly defined local location, specifically at a predetermined distance from the optical sensor according to the near and far boundaries of the visible distance range, through time control of the first and second illumination devices on the one hand and the optical sensor on the other.

[0013] The visible distance range here is the object-side region in three-dimensional space, which is mapped onto the image plane of the optical sensor in a two-dimensional image by combining the number and duration of light pulses from the first and / or second illumination devices with the start and end of exposure of the optical sensor.

[0014] The term "object side" as used herein and hereinafter refers to a region in real space. The term "image side" as used herein and hereinafter refers to a region within the image plane of the optical sensor. The visible distance range is obtained here on the object side. This corresponds to the image-side region on the image plane, assigned by the imaging theorem and the time control of the first illumination device, the second illumination device, and the optical sensor.

[0015] Based on the start and end of the exposure of the optical sensor after the illumination begins with the aid of the first and / or second illumination devices, light pulse photons illuminate the optical sensor. The farther the visible distance range is from the first and / or second illumination devices and the optical sensor, the longer the duration required for a single photon reflected within that distance to illuminate the optical sensor. Therefore, the farther the visible distance range is from the first illumination device, the second illumination device, and the optical sensor, the longer the time difference between the end of illumination and the start of exposure.

[0016] Therefore, according to one design of the method, and particularly likely, the location and spatial width of the visible distance range, especially the distance between the near and far boundaries of the visible distance range, are defined by appropriately selecting, on the one hand, a first illumination device and / or a second illumination device and on the other hand, a time control of an optical sensor.

[0017] In a preferred design of the method, a visible distance range is defined, wherein the time coordination of the first illumination device and / or the second illumination device and the optical sensor is determined and set accordingly.

[0018] In a preferred design, the lighting device has at least one surface-emitting element, particularly a so-called VCSE laser. Alternatively or additionally, the optical sensor is preferably a camera.

[0019] In one embodiment of the method, a third image is acquired after the first and second images.

[0020] In another embodiment of this method, a third image is acquired after the first image. A second image is acquired after the third image.

[0021] In another embodiment of the method, the third image is acquired before the first and second images.

[0022] Advantageously, the method can be executed continuously. In one implementation of the method, the first image, the second image, and the third image are acquired at the same frequency within a unit time.

[0023] In another embodiment of the method, which is executed sequentially, an image selected alternately from the first and second images is acquired alongside the third image, i.e., in this order: a first image followed by a third image, then a second image, then another third image, then a first image again, and so on. Therefore, the time difference between individual illuminations using the first lighting device or between individual illuminations using the second lighting device is extended. The longer time difference allows for optimal cooling of the first or second lighting device and thus, illumination with higher output energy.

[0024] In another preferred embodiment of the method, the first image, the second image, and the third image are acquired with a time difference of less than 0.01 seconds, preferably less than 0.001 seconds.

[0025] According to an improved embodiment of the present invention, a first partial difference image is formed as the difference between a first image and a third image. Furthermore, a second partial difference image is formed as the difference between a second image and a third image. The difference image is then formed as the difference between the first partial difference image and the second partial difference image.

[0026] According to an improved embodiment of the invention, an image registration method is applied to the first, second, and third images before forming the difference images. Advantageously, image registration is used to compensate for the motion of the motor vehicle itself.

[0027] In one embodiment of the method, an image registration method is applied to a first image and a third image, wherein the third image is adapted to the first image to form a first partial difference image. Additionally, an image registration method is applied to a second image and a third image, wherein the third image is adapted to the second image to form a second partial difference image. The image registration method is applied to the first partial difference image and the second partial difference image after they have been formed.

[0028] In a preferred embodiment of the method, an image registration method is applied to the first and third images, wherein the first image is adapted to the third image. Furthermore, an image registration method is applied to the second and third images, wherein the second image is adapted to the third image. Further image registration is advantageously eliminated because the first and second images are adapted to the third image. The first partial difference image and the second partial difference image are thus automatically registered.

[0029] According to an improved embodiment of the invention, the search for objects in a difference image is specified. Advantageously, only image information, which is visible only in the first image or only in the second image, can be seen in the difference image. This image information specifically includes shadows generated by the object due to the different illumination of the object using a first lighting device and a second lighting device. The object can be inferred from this image information, particularly the shadows.

[0030] In a preferred embodiment of this method, objects are detected only from a predetermined horizontal image-side shadow width Δu. The predetermined horizontal image-side shadow width Δu allows for stable and reliable object detection in the difference image.

[0031] According to an improved embodiment of the invention, ranging is performed in the difference image.

[0032] In a preferred embodiment of this method, ranging is performed according to a method known from German publication DE 10 2020002994A1. To perform this method, the object-side position of the visible distance range, the image-side position of the visible distance range, and the base image line of the object must be known. The object-side position of the visible distance range is known from the coordinated control of the first illumination device, the second illumination device, and the optical sensor. The image-side position of the visible distance range is known not only from the first image but also from the second image. The base image line of the object is known from the difference image. Therefore, the object distance is estimated particularly based on the object's shadow.

[0033] The baseline image line of a shadow can vary depending on the shape of the shadow. The accuracy of determining the baseline image line depends particularly on the shape of the shadow. Especially in the case of a triangular shadow that widens from bottom to top on the image side, object detection begins from a predetermined horizontal image-side shadow width Δu. This triangular shadow is particularly generated when, on the one hand, the illumination devices are horizontally spaced from each other, and on the other hand, at least one illumination device is horizontally spaced from the optical sensor. Distance measurement based on the predetermined horizontal image-side shadow width Δu is still reliable. This reliability is explained in conjunction with the following considerations.

[0034] Assuming the object is flattened along the direction of travel of the vehicle, especially in the x-direction, the radiation theorem can be used to determine the object-side shadow distance x from any position within the shadow, especially the triangular shadow, to the optical sensor. W The horizontal shadow width y on the object side of the shadow at any location. W The distance f between the image plane of the optical sensor and the image plane of the optical sensor lens and the predetermined horizontal image-side shadow width Δu are thus related as follows:

[0035]

[0036] Similarly, the horizontal illumination distance y from one of the two illumination devices to the optical sensor can be calculated using the ray theorem. B Object-side shadow distance x W and the distance x between the object and the optical sensor O Such interconnections:

[0037]

[0038] From the combination of formulas (1) and (2), the following formula is used to obtain the distance x at any position used to observe the object's side shadow distance. W Distance x from the object O The distance difference Δx between them:

[0039]

[0040] For the illumination distance y B=2m, image plane distance f = 5000px, and predetermined horizontal image side shadow width Δu = 3px, at the current object distance x O When the distance is 200m, a distance difference Δx of approximately 13m is obtained. Therefore, the ranging error is only 7.5%. This error occurs at object distance x. O This is acceptable for a length of 200m.

[0041] This also accomplishes the task of providing a control device configured to perform the method of the present invention or the method according to one or more of the foregoing embodiments. The control device is preferably designed as a computing device, particularly preferably a computer, or a controller, especially a controller for a motor vehicle. With regard to this control device, the advantages already explained with respect to the method are particularly evident.

[0042] This task is also accomplished by providing a gated camera device having a first illumination device, a second illumination device, an optical sensor, and a control device of the present invention or a control device according to one or more of the foregoing embodiments. The control device is preferably effectively connected to the first illumination device, the second illumination device, and the optical sensor and configured for controlling their respective components. In particular, the advantages already explained regarding the method and control device are obtained with regard to the gated camera device.

[0043] According to an improved embodiment of the invention, the first lighting device and the second lighting device are arranged horizontally offset from each other. In particular, the first lighting device and / or the second lighting device are arranged horizontally offset relative to the optical sensor.

[0044] According to an improved embodiment of the invention, the first lighting device and the second lighting device are arranged vertically offset from each other. In particular, the first lighting device and / or the second lighting device are arranged vertically offset relative to the optical sensor.

[0045] In a preferred embodiment, the first lighting device and the second lighting device are arranged staggered from each other both vertically and horizontally. Alternatively or additionally, the first distance between the first lighting device and the optical sensor is smaller than the second distance between the second lighting device and the optical sensor. The first distance is particularly preferably less than 50 cm, more preferably less than 20 cm, and most preferably less than 10 cm. Furthermore, the second distance is particularly preferably greater than 50 cm, more preferably greater than 100 cm, and most preferably greater than 150 cm.

[0046] Finally, this task is accomplished in the same way, namely, by providing a motor vehicle having the gating camera device of the present invention or a gating camera device according to one or more of the foregoing embodiments. Regarding motor vehicles, the advantages already explained with respect to the method, control device, and gating camera device are particularly advantageous.

[0047] In a favorable design, the motor vehicle is designed as a truck. However, it is also possible that the motor vehicle is a car, a commercial vehicle, or another type of motor vehicle.

[0048] In a preferred embodiment, the vehicle is a truck. The optical sensor and the first lighting device are arranged above the windshield and have a mutual distance of less than 50 cm, preferably less than 20 cm, and more preferably less than 10 cm, i.e., a first distance. The second lighting device is preferably arranged within the bumper area. Attached Figure Description

[0049] The invention will be explained in detail below with reference to the figures, wherein:

[0050] Figure 1 A schematic diagram of a motor vehicle embodiment having a gated camera device is shown.

[0051] Figure 2 A schematic diagram illustrating an embodiment of a method for operating a first lighting device, a second lighting device, and an optical sensor is shown.

[0052] Figure 3 A schematic diagram is shown for determining the distance difference Δx. Detailed Implementation

[0053] Figure 1 A schematic diagram of a motor vehicle 1 having an embodiment of a gated camera device 3 is shown. The gated camera device 3 includes a first lighting device 5.1, a second lighting device 5.2, an optical sensor 7 (especially a camera), and a control device 9. The control device 9 is effectively connected / operates with the first lighting device 5.1, the second lighting device 5.2, and the optical sensor 7 in a manner not explicitly shown and is configured for the control of their respective components.

[0054] The first lighting device 5.1 and the second lighting device 5.2 are preferably arranged vertically offset from each other. Alternatively or additionally, the first lighting device 5.1 and the second lighting device 5.2 are preferably arranged horizontally offset from each other.

[0055] The first distance between the first lighting device 5.1 and the optical sensor 7 is preferably less than the second distance between the second lighting device 5.2 and the optical sensor 7. The first distance is particularly preferably less than 50 cm, particularly less than 20 cm, and preferably less than 10 cm. It is also particularly preferred that the second distance is greater than 50 cm, preferably greater than 100 cm, and preferably greater than 150 cm.

[0056] The first lighting device 5.1 and the second lighting device 5.2 preferably have at least one surface light-emitting element, especially a so-called VCSE laser.

[0057] Figure 1Specifically, the first illumination cone 11.1 of the first illumination device 5.1, the second illumination cone 11.2 of the second illumination device 5.2, and the observation area 13 of the optical sensor 7 are shown. Furthermore, the visible distance range 15 is shown in shaded lines, which is formed as a subset of the first illumination cone 11.1 of the first illumination device 5.1, the second illumination cone 11.2 of the second illumination device 5.2, and the observation area 13 of the optical sensor 7. An object 17 is arranged within the visible distance range 15.

[0058] The control device 9 is specifically provided for performing a method for operating the first lighting device 5.1, the second lighting device 5.2, and the optical sensor 7, such as... Figure 2 The detailed implementation methods are described below.

[0059] The control of the first illumination device 5.1, the second illumination device 5.2, and the optical sensor 7 is coordinated with each other in time, and coordinated control is assigned to the visible distance range 15. With the aid of coordinated control, a first image 19.1 is acquired using the optical sensor 7 under the illumination of the first illumination device 5.1. The first image 19.1 is... Figure 2 As shown in a). Furthermore, with the aid of coordinated control, the second image 19.2 is acquired using the optical sensor 7 under illumination by the second illumination device 5.2. The second image 19.2 is... Figure 2 As shown in c). Furthermore, the third image 19.3 was acquired without illumination from either the first illumination device 5.1 or the second illumination device 5.2, using the optical sensor 7. The third image 19.3 in Figure 2 It is shown in b). A difference image 19.4 is then formed based on the first image 19.1, the second image 19.2, and the third image 19.3. The difference image 19.4 is shown in... Figure 2 It is shown in f).

[0060] The image-side object 17′ can be seen in the first image 19.1, the second image 19.2, and the third image 19.3. Ideally, in such a case... Figure 2 In the first image 19.1 shown in a), no shadow is visible. Furthermore, in the second image 19.2, the image-side visible shadow 21' of object 17 is visible. The image-side visible shadow 21' appears because the second illumination device 5.2 and the optical sensor 7 are arranged spaced apart from each other, particularly horizontally offset from each other. Preferably, the first illumination device 5.1 and the second illumination device 5.2 are arranged such that the image-side visible shadows 21' in the first image 19.1 and the second image 19.2 are different from each other.

[0061] Because the image-side visible shadow 21′ in the first image 19.1 and the second image 19.2 are different from each other, therefore in Figure 2 In the difference image 19.4 within f), only the image-side visible shadow 21′ can still be seen.

[0062] The third image 19.3 is preferably acquired between the first image 19.1 and the second image 19.2. Alternatively, the third image 19.3 is acquired before the first image 19.1 and the second image 19.2. Alternatively, the third image 19.3 is acquired after the first image 19.1 and the second image 19.2.

[0063] The first image and the second image are preferably acquired with a time difference of less than 0.1 seconds, and more preferably with a time difference of less than 0.01 seconds.

[0064] Preferably, in step A of the method, a first partial difference image 19.5 is formed as the difference between the first image 19.1 and the third image 19.3. The first partial difference image 19.5 is... Figure 2 As shown in d). In method step B, the second partial difference image 19.6 is formed as the difference between the second image 19.2 and the third image 19.3. The second partial difference image 19.6 is... Figure 2 As shown in e). By subtracting the third image 19.3 from the first image 19.1 and the second image 19.2, background illumination is removed from images 19.1 and 19.2, and unexposed areas, especially image-side shadows 21′, can be clearly seen in the first partial difference image 19.5 and the second partial difference image 19.6. In method step C, difference image 19.4 is formed as the difference between the first partial difference image 19.5 and the second partial difference image 19.6.

[0065] Preferably, image registration is also performed in method steps A and B. Preferably, the first image 19.1 and the second image 19.2 are adapted to the third image 19.3. Image registration may also be performed in method step C. However, if the first image 19.1 and the second image 19.2 have already been adapted to the third image 19.3 in method steps A and B, image registration in method step C is not necessary.

[0066] The object detection method is preferably performed in the difference image 19.4.

[0067] The ranging method is preferably performed in the difference image 19.4 in order to measure, in particular, the distance between the optical sensor 7 and the detected object 17.

[0068] Figure 3 Show Figure 1 A top view of the situation, especially the xy plane. The second illumination device 5.2 and the optical sensor 7 are arranged horizontally offset from each other. The object-side shadow area 21 corresponds to the image-side visible shadow 21′ of the object 17 under illumination by the second illumination device 5.2. Using the ray theorem, the distance x from any position within the object-side shadow area 21 to the object-side shadow of the optical sensor 7 can be calculated.W The horizontal shadow width y on the object side of the shadow at any location W The distance f between the image plane 23 of the optical sensor 7 and the image plane of the lens of the optical sensor 7 is related to the predetermined horizontal image-side shadow width Δu (1). Similarly, the horizontal illumination distance y from the second illumination device 5.2 to the optical sensor 7 can be related by means of the ray theorem. B Object-side shadow distance x W The distance x between object 17 and optical sensor 7 O Interrelated (2). In summary, formula (3) is then used to calculate x at any position used to observe the object-side shadow area 21. W Distance x from the object O The distance difference Δx between them.

Claims

1. A method for operating a first lighting device (5.1), a second lighting device (5.2), and an optical sensor (7), wherein, - The control of the first lighting device (5.1), the second lighting device (5.2), and the optical sensor (7) is coordinated with each other in time. - This coordinated control is assigned a visible distance range (15). - The first image (19.1) is acquired by the optical sensor (7) under the illumination of the first illumination device (5.1) with the aid of the coordinated control. - The second image (19.2) is acquired by the optical sensor (7) under the illumination of the second illumination device (5.2) with the aid of the coordinated control. - The third image (19.3) was acquired by the optical sensor (7) without illumination by one of the illumination devices (5), and - A difference image (19.4) is formed based on the first image (19.1), the second image (19.2), and the third image (19.3). - A first partial difference image (19.5) is formed as the difference between the first image (19.1) and the third image (19.3). - A second partial difference image (19.6) is formed as the difference between the second image (19.2) and the third image (19.3). - The difference image (19.4) is formed by the difference between the first part difference image (19.5) and the second part difference image (19.6).

2. The method according to claim 1, wherein, Before forming the difference image (19.4), the image registration method is applied to the first image (19.1), the second image (19.2), and the third image (19.3).

3. The method according to any one of claims 1-2, wherein, Search for objects (17) in the difference image (19.4).

4. The method according to any one of claims 1-2, wherein, Distance measurements are performed in the difference image (19.4).

5. A control device (9) configured to perform the method according to any one of claims 1 to 4.

6. A gated camera device (3) comprising a first illumination device (5.1), a second illumination device (5.2), an optical sensor (7), and a control device (9) according to claim 5.

7. The gated camera device (3) according to claim 6, wherein, The first lighting device (5.1) and the second lighting device (5.2) are arranged horizontally offset from each other.

8. The gated camera device (3) according to claim 6 or 7, wherein, The first lighting device (5.1) and the second lighting device (5.2) are arranged vertically offset from each other.

9. A motor vehicle (1) having a gating camera device (3) according to any one of claims 6 to 8.

Images