Method for detecting missing image information, control device for carrying out such a method, detection device having such a control device and motor vehicle having such a detection device

Abstract

The present invention relates to a method for detecting lost image information by means of an illumination device (5) and an optical sensor (7), wherein the driving control of the illumination device (5) and the optical sensor (7) is mutually time-coordinated, wherein a visible distance region (15) is obtained in the observation area (13) of the optical sensor (7) by means of the time coordination of the driving control of the illumination device (5) and the optical sensor (7), wherein a photograph (25) of the observation area (13) is acquired by means of the coordinated driving control using the optical sensor (7), wherein image information is searched in the photograph (25) in the area outside the visible distance region (15) on the image side, wherein the found image information is detected and made available for use.

Description

Technical Field

[0001] The present invention relates to a method for detecting lost image information, a control device for performing the method, a detection device having such a control device, and a motor vehicle having such a detection device. Background Technology

[0002] A method is disclosed in international patent application WO 2017 / 009848 A1, in which an illumination device and an optical sensor are driven in a time-coordinated manner to capture a certain visible distance region within the observation area of ​​the optical sensor, wherein the visible distance region is derived from the time coordination of the driving of the illumination device and the optical sensor. Image information is detected in the photograph only within the visible distance region. However, in this region, retroreflective objects are overexposed and cannot be read using software techniques under traffic signs. The evaluation of image regions in the photograph that do not correspond to the visible distance region is unknown. Summary of the Invention

[0003] Therefore, the object of the present invention is to provide a method for detecting lost image information, a control device for performing the method, a detection device having such a control device, and a motor vehicle having such a detection device, wherein the aforementioned disadvantages are at least partially eliminated, and preferably avoided.

[0004] This objective is achieved by providing the teachings of this technology, particularly the teachings of the independent claims and the embodiments disclosed in the dependent claims and the specification.

[0005] In particular, this objective is achieved by providing a method for detecting lost image information using an illumination device and an optical sensor. Here, the illumination device and the optical sensor are driven in a time-coordinated manner. The local location of the visible distance region within the observation area of ​​the optical sensor is obtained by time-coordinating the driving of the illumination device and the optical sensor. A photograph of this observation area is acquired using the optical sensor with coordinated driving. Finally, image information is searched for in the region outside the visible distance region on the image side within the photograph. If image information is found in the region outside the visible distance region on the image side within the photograph, the image information is detected and made available. "Make available" here specifically means that the detected image information is transferred to a controller for further processing and / or for use in autonomous driving or otherwise made available to the controller.

[0006] The method proposed herein advantageously allows for the detection of objects outside the visible range and provides relevant information for use by vehicles performing the method. In particular, it enables the high-precision detection and identification of retroreflective traffic signs with some fluorescent and / or phosphorescent effects. This is especially possible because traffic signs absorb illumination photons due to their photoluminescent properties and then emit these photons again after a period of time, particularly up to several seconds, following the end of illumination. Therefore, retroreflective traffic signs with photoluminescent properties are overexposed due to additional reflection when directly illuminated, i.e., when they are within the visible range, and thus cannot be read by software techniques. Once the traffic sign is no longer within the visible range, the photoluminescence effect begins to take effect, and the photons emitted by this effect make the traffic sign clearly visible in the photograph of the optical sensor. Advantageously, this method is also suitable for identifying bicycles with photoluminescent components and people wearing photoluminescent clothing.

[0007] This method can be advantageously used in autonomous vehicles, especially autonomous trucks. Particularly during nighttime driving, it can be advantageously used to detect traffic signs or other road users. This method allows for timely and appropriate responses to detected traffic signs and other road users. Such appropriate responses could include, for example, deceleration, emergency braking, or driving along a specially designated avoidance trajectory if necessary.

[0008] The method for generating photographs by means of mutual temporal coordination between an illumination device and an optical sensor particularly involves a method known as gated imaging; the optical sensor, in particular a camera, switches to sensitivity only within a specified, limited time range, which is called "gated control," and the camera is thus a gated camera. The illumination device is also correspondingly controlled in time only at certain selected time intervals to illuminate the scene on the object side.

[0009] Specifically, the illumination device emits a predetermined number of light pulses, preferably with a duration of 5ns-20ns. The start and end of the exposure of the optical sensor are related to the number and duration of the output light pulses. Therefore, a certain visible distance range can be determined by the optical sensor according to the corresponding specified local position (i.e., in particular, the specified distance from the starting point of the distance range to the optical sensor and the specified width of the distance range) through temporal control of the illumination device and the optical sensor.

[0010] Here, the visible distance region is the object-side region in three-dimensional space, which is presented on the image plane of the optical sensor in the form of a two-dimensional photograph by means of the number and duration of light pulses from the illumination device combined with the exposure start and end points of the optical sensor.

[0011] In contrast, the observation area, especially the object-side region in three-dimensional space, can be generally and especially maximized in a two-dimensional photograph when fully illuminated and exposed by the optical sensor. In particular, the observation area corresponds to the entire theoretically illuminateable, exposing image area of ​​the optical sensor. The visible distance region is therefore a subset of the observation area in real space.

[0012] The image-side observation region corresponds to all image rows present based on the optical sensor. The visible distance region on the image side, as a local region of the image plane, exists particularly between the starting and ending image rows. The starting image row defines the beginning of the visible distance region in the photograph. Furthermore, the ending image row defines the end of the visible distance region in the photograph.

[0013] When we speak of "object side" here and below, we are referring to the region in real space, that is, the side on which the object is being observed. When we speak of "image side" here and below, we are referring to the region on the image plane of the optical sensor. The observation area and the visible distance area exist here on the object side. Correspondingly, there is the image side region on the image plane, which is associated with imaging theorems and the temporal control of the illumination device and the optical sensor.

[0014] Based on the start and end points of the optical sensor exposure after the illumination begins with the aid of the illumination device, light pulse photons strike the optical sensor. The farther the visible distance region is from the illumination device and the optical sensor, the longer the time required for reflected photons in that region to strike the optical sensor. Therefore, the farther the visible distance region is from the illumination device and the optical sensor, the longer the time interval between the end of illumination and the start of exposure.

[0015] Therefore, one particularly feasible design according to this method is to define the position and spatial width of the visible distance area by appropriately selecting the temporal driving control of both the lighting device and the optical sensor.

[0016] In an alternative design of this method, a visible distance region can be set, thereby determining and accordingly setting the temporal coordination between the lighting device and the optical sensor.

[0017] An image row here specifically refers to the entire set of pixels in a photograph within the image plane of an optical sensor, which are located on the same horizontal line within the image plane.

[0018] In particular, the detection of preferred photoluminescent objects for image information was used on all image rows that were not located between the start and end image rows.

[0019] In the preferred design, the lighting device is a laser. In the preferred design, the optical sensor is a camera.

[0020] According to an improved embodiment of the invention, for photographs of the distance region, a row histogram is created by summing the illumination intensities of each image row from the optical sensor to represent all image rows corresponding to the evaluation region within the observation area based on the optical sensor. The starting and ending image rows are then determined using the row histogram. This determines the image-side region outside the image-side distance region. This advantageously allows for the determination of the image location of the region corresponding to the visible distance region on the object side based on the optical sensor. That is, a clear brightness transition / brightness change at the start and end points of the image-side distance region is thus obtained through temporal control of both the illumination device and the optical sensor. Ultimately, it becomes possible to determine the region outside the visible distance region on the image side, in which image information, especially for photoluminescent objects, is searched.

[0021] In this context, the row histogram specifically refers to the sum of the illumination intensities of all pixels within the evaluation region that correspond to the respective image rows of the optical sensor. In this way, corresponding brightness transitions caused by temporal control can be easily and reliably detected within the image plane of the optical sensor.

[0022] According to a preferred design, the evaluation region and the observation region are identical. This corresponds to a particularly easy-to-implement method implementation. However, it is also possible, according to another preferred design, for the evaluation region to be smaller than the observation region, particularly as a target region or "region of interest" where the object to be detected might reside. This advantageously allows for the rapid and efficient execution of the method. For this purpose, only pixels located within the evaluation region are included in the summation, thus the evaluation region is also particularly likely to be horizontally restricted.

[0023] Preferably, the evaluation area in the photograph is identified by GPS prediction (especially when the road orientation is back-projected onto the image plane) and / or by optical trajectory tracking before the row histogram is calculated.

[0024] According to an improved embodiment of the present invention, the search and detection of the image information is performed using a pattern recognition algorithm.

[0025] According to an improved embodiment of the present invention, the search and detection of the image information is performed using deep learning.

[0026] According to an improved embodiment of the invention, the detected image information is reconstructed and stored. Therefore, it is particularly likely that the detected image information is displayed to the driver in a photographic format. In the case of an autonomous vehicle, the detected image information is preferably stored so that the vehicle's response can be interpreted after driving.

[0027] According to an improved embodiment of the invention, two photographs, namely a first photograph and a second photograph, are acquired using an optical sensor by means of two different time-coordinated drives of the illumination device and the optical sensor. The first photograph, the second photograph, and the detected image information are combined into a single complete photograph. Preferably, the visible distance regions of the first photograph and the second photograph do not overlap. Photoluminescent objects located within the visible distance region of the first photograph are overexposed in the first photograph. Advantageously, the photoluminescent objects are not located within the visible distance region of the second photograph. Therefore, the object can be identified outside the visible distance region due to its afterglow properties. In the complete photograph, when the first and second photographs are combined, it is preferable to replace the overexposed display of the object from the first photograph with the display of the object from the second photograph. In particular, this method is used to reconstruct the detected image information.

[0028] This objective is also achieved by providing a control device configured to perform the method of the present invention or the method according to one of the foregoing embodiments. This control device is preferably designed as a computing device, particularly a computer, or as a controller, and especially a vehicle controller. The control device particularly possesses the advantages already explained in relation to the described method.

[0029] This objective is also achieved by providing a detection device having an illumination device, an optical sensor, and a control device of the present invention or a control device according to one of the foregoing embodiments. The detection device particularly possesses the advantages explained in relation to the method and control device.

[0030] The control device is preferably connected to both the lighting device and the optical sensor and is configured to drive and control them.

[0031] Finally, this objective is also achieved by providing a motor vehicle equipped with the detection device of the present invention or a detection device according to one of the foregoing embodiments. Motor vehicles, in particular, possess the advantages explained in relation to the described method, control device, and detection device.

[0032] In a favorable design, the motor vehicle is designed as a truck. However, it is also possible for the motor vehicle to be a car, a truck, or another type of motor vehicle. Attached Figure Description

[0033] The invention will be explained in detail below with reference to the figures, wherein:

[0034] Figure 1 A schematic diagram of a motor vehicle embodiment showing an embodiment of a detection device;

[0035] Figure 2 A schematic diagram showing a photograph acquired using an optical sensor within the scope of the method implementation;

[0036] Figure 3 A schematic diagram of a row histogram used in an implementation of the method is shown. Detailed Implementation

[0037] Figure 1 A schematic diagram of one embodiment of a motor vehicle 1 having a detection device 3 is shown. The detection device 3 has an illumination device 5 and an optical sensor 7. In addition, the detection device 3 has a control device 9, which is only schematically shown herein and functions connected to the illumination device 5 and the optical sensor 7 in a manner not explicitly shown, for the respective control of them. Figure 1 The illumination cone 11 of the illumination device 5 and the observation area 13 of the optical sensor 7 are shown in particular. In addition, the visible distance area 15, which exists as a subset of the observation area 13 of the optical sensor 7, is shown in shaded lines.

[0038] Traffic signs 17 are installed within the visible distance area 15. Traffic signs 19 are installed outside the visible distance area 15.

[0039] exist Figure 1 The starting point 21 and ending point 23 of the visible distance region 15 are also shown.

[0040] The control device 9 is specifically provided with an embodiment for performing a method for detecting lost image information in a photograph 25 generated by means of the illumination device 5 and the optical sensor 7, as detailed below.

[0041] Here, the illumination device 5 and the optical sensor 7 are driven in a time-coordinated manner, wherein the visible distance region 15 within the observation area 13 is obtained by time-coordinating the driving of the illumination device 5 and the optical sensor 7. An image of the observation area 13 is acquired using the optical sensor 7 under coordinated driving.

[0042] Figure 2 A schematic diagram of this photograph 25 is shown within the image plane of the optical sensor 7. Here, in Figure 2Photograph 25 shows the starting image row 27 for the starting point 21 of the visible distance region 15 and the ending image row 29 for its ending point 23. Additionally, images of traffic signs 17 and 19 are shown in Photograph 25. In Photograph 25, the image of traffic sign 17 is designated as 17', and the image of traffic sign 19 is designated as 19'. Due to the retroreflective properties of traffic signs 17 and 19, the mark is not visible in the image of traffic sign 17'. Traffic sign 19 was positioned within the visible distance region on the object side shortly before Photograph 25 was created. Due to the photoluminescent properties of traffic sign 19, the mark can be clearly identified within the image of traffic sign 19'.

[0043] A method for detecting lost image information is then used to search for image information, preferably objects with photoluminescent properties, in the region above the endpoint image line 29 and below the starting image line 27. If image information 19′ is found, the image information is detected and made available to the control device 9 and, consequently, the vehicle 1 and / or the driver.

[0044] Preferably, the method for searching and detecting lost image information is based on pattern recognition or deep learning.

[0045] In addition, Figure 2 The evaluation area 31 is shown, which can be determined in particular by GPS forecasting and / or methods used for optical trajectory tracking. Evaluation area 31 is here considered as a target area smaller than observation area 13, but it can also overlap with it.

[0046] Figure 3 Showing according to Figure 2 A schematic diagram of the row histogram 33 of photograph 25 or evaluation region 31 of photograph 25. In the row histogram 33, each image row of the optical sensor 7 is plotted on the horizontal axis, and the sum of the illumination intensities of each pixel within the corresponding pixel range of the image row in the evaluation region 31 is plotted on the vertical axis for each image row. The row histogram 33 is created by summing the illumination intensities of each image row of the optical sensor 7 within the range of all image rows corresponding to the evaluation region 31 based on the optical sensor 7. Thus, the starting image row 27 and the ending image row 29 are identified by the row histogram 33, in particular because of the temporal coordination of the illumination device 5 and the optical sensor 7, significant intensity abrupt changes in both the starting image row 27 and the ending image row 29 can be identified.

Claims

1. A method for detecting lost image information using an illumination device (5) and an optical sensor (7), wherein, - The driving and control of the lighting device (5) and the optical sensor (7) are mutually time-coordinated, wherein, - A visible distance region (15) is obtained in the observation region (13) of the optical sensor (7) by time coordination of the driving control of the illumination device (5) and the optical sensor (7), wherein the optical sensor (7) acquires a photograph (25) of the observation region (13) by means of coordinated driving control, wherein, - In the photograph (25), search for image information in the area outside the visible distance region (15) on the image side, the image information being derived from the afterglow of the photoluminescent object after illumination, wherein, - Detect and make usable image information. In order to identify the visible distance region (15) on the image side, a row histogram (33) is created by summing the illumination intensity of each image row of the optical sensor (7) to identify all image rows that correspond to the evaluation region (31) within the observation area (13) based on the optical sensor (7). The starting image row and the ending image row are determined by means of the row histogram (33), and the region outside the visible distance region (15) on the image side of the photograph (25) is identified by means of the starting image row and the ending image row.

2. The method according to claim 1, wherein, Pattern recognition is used to perform the search and detection of image information.

3. The method according to claim 1, wherein, Deep learning is used to perform the search and detection of image information.

4. The method according to claim 1, wherein, Reconstruct and store the detected image information.

5. The method according to claim 1, wherein, Two photographs (25) are acquired using an optical sensor (7) with the aid of two time-coordinated drive mechanisms, wherein the photographs (25) and the detected image information are merged into a single full photograph.

6. A control device (9) configured to perform a method for detecting lost image information according to any one of the preceding claims.

7. A detection device (3) having an illumination device (5), an optical sensor (7) and a control device (9) according to claim 6.

8. A motor vehicle (1) having the detection device (3) according to claim 7.

Images