Object tracking method for at least one object, control device for carrying out such a method, object tracking device having such a control device and motor vehicle having such an object tracking device

Abstract

The present invention relates to a method for object tracking of at least one object (17) in at least two consecutive photographs (19.1; 19.2) using an illumination device (5) and an optical sensor (7), wherein: - the control of the illumination device (5) and the optical sensor (7) is coordinated with each other in time; - a first visible distance region (15.1) is associated with the first coordinated control; - a first photograph (19.1) of the first visible distance region (15.1) is taken using the optical sensor (7) with the first coordinated control under the illumination of the illumination device (5); - when the first photograph (19.1) is taken... When the at least one object (17) is detected in the first visible distance region (15.1) in .1), a distance measurement of the at least one object is performed, and the second coordination control of the illumination device (5) and the optical sensor (7) and the corresponding second visible distance region (15.2) are determined such that the object (17) detected in the first photograph (19.1) is centered in the second visible distance region (15.2), wherein the second photograph (19.2) of the second visible distance region (15.2) is taken by the optical sensor (7) with the aid of the second coordination control under the illumination of the illumination device (5).

Description

Technical Field

[0001] The present invention relates to an object tracking method for at least one object, a control device for performing the method, an object tracking device having such a control device, and a motor vehicle having such an object tracking device. Background Technology

[0002] Methods for object detection and tracking using illumination devices and optical sensors are known. A similar method is disclosed in International Patent Application Publication No. WO 2017 / 009848 A1, in which the illumination device and optical sensor are controlled in a time-coordinated manner to capture an image of a certain visible distance region within the observation area of ​​the optical sensor, wherein this visible distance region depends on the time-coordinated control of the illumination device and optical sensor. A drawback of this method is the lack of feedback between the detected and tracked object and the illumination device and optical sensor.

[0003] From the publicly published book "Gated2Depth: Real-Time Dense Lidar From Gated Images" (Tobias Gruber et al.) (https: / / arxiv.org / pdf / 1902.04997.pdf) I learned about a method for generating photos that include real-time distance information. The problem is that this method can only be used within a range of up to 80 meters. Summary of the Invention

[0004] The object of the present invention is therefore to provide an object tracking method for at least one object, a control device for performing the method, an object tracking device having such a control device, and a motor vehicle having such an object tracking device, wherein the disadvantages are at least partially eliminated, and preferably avoided.

[0005] This task is accomplished by providing the technical teachings, especially the teachings of the independent claims and the embodiments disclosed in the dependent claims and the specification.

[0006] Specifically, this task is accomplished by providing a method for object tracking of at least one object in at least two consecutive photographs taken at different times using an illumination device and an optical sensor, wherein the control of the illumination device and the optical sensor is coordinated temporally. A first coordinated control of the illumination device and the optical sensor corresponds to a first visible distance region, wherein a first photograph of the first visible distance region is taken using the optical sensor with the first coordinated control under illumination by the illumination device. When at least one object is detected in the first photograph within the first visible distance region, a distance measurement is performed to determine the distance between the at least one detected object and the optical sensor. A second coordinated control of the illumination device and the optical sensor and a corresponding second visible distance region are determined such that the object detected in the first photograph is centered within the second visible distance region. Subsequently, a second photograph of the second visible distance region is taken using the optical sensor with the second coordinated control under illumination by the illumination device.

[0007] Advantageously, the method proposed here allows the visible range region to be adapted to the detected object. Therefore, object detection and coordinated control are interconnected through a feedback loop. By centering the object within the visible range region, the detected object is optimally illuminated. This optimal illumination enables better semantic segmentation in subsequent image processing.

[0008] The distance between the detected object and the optical sensor is determined using an appropriate method. Such an appropriate method is disclosed in particular by German publication number DE 10 2020 002 994 A1.

[0009] In a preferred embodiment of the method, the first and second photographs are taken in less than 0.1 seconds, and particularly preferably in less than 0.01 seconds.

[0010] This method can be particularly advantageously used in autonomous vehicles, especially autonomous trucks. Especially when driving at night in the absence of other vehicles, and where high range requirements are necessary, this method can advantageously detect and / or track and / or classify non-run-over objects, particularly those smaller than the vehicle, positioned in the vehicle's lane. This method allows for timely and appropriate responses to the detection and tracking of such objects, and especially to the identification of the vehicle's distance from the object. Such appropriate responses could, for example, be emergency braking or following a potentially temporarily designated avoidance trajectory.

[0011] These small, non-runaway objects are generally referred to as "lost goods." However, such objects can also be people or animals lying on the road, especially those involved in accidents.

[0012] Methods for generating photographs by means of temporally coordinated control of an illumination device and an optical sensor, particularly those known as gated imaging methods; specifically, the optical sensor is a camera that is switched to sensitivity only within a certain finite time range, a process referred to as "gated control," thus making the camera a gated camera. The illumination device is also correspondingly controlled temporally only at certain selected time intervals to illuminate the scene on the object side. Gated imaging methods are particularly advantageous in low-light conditions, especially in rain and / or fog, and at night, because the image signal and sensor noise can be distinguished significantly better than with a standard camera, especially a monocular standard camera.

[0013] In particular, a predetermined number of light pulses, preferably with a duration of 5 ns to 20 ns, are emitted by the illumination device. The start and end of the exposure of the optical sensor are related to the number and duration of the output light pulses. Therefore, a certain visible distance range can be measured by the optical sensor through time control of both the illumination device and the optical sensor at a correspondingly predetermined local position, that is, in particular, a certain distance from the starting point of the distance range to the optical sensor and a certain width of the distance range.

[0014] The visible distance region here is the three-dimensional spatial region on the object side as follows, which is imaged in a two-dimensional photograph on the image plane of the optical sensor by means of the number and duration of light pulses from the illumination device combined with the start and end of the exposure of the optical sensor.

[0015] Here and below, "object side" refers to the region in real space, i.e., on the side of the object to be observed. Here and below, "image side" refers to the region on the image plane of the optical sensor. The visible distance region appears here on the object side. It corresponds to the corresponding image side region on the image plane, determined by the imaging theorem and the time control of the illumination device and the optical sensor.

[0016] Based on the start and end of the exposure of the optical sensor after the illumination device begins, light pulse photons illuminate the optical sensor. The farther the visible distance region is from the illumination device and the optical sensor, the longer the duration required for a single photon reflected within that region to illuminate the optical sensor. Therefore, the farther the visible distance region is from the illumination device and the optical sensor, the longer the time difference between the end of illumination and the start of exposure.

[0017] Therefore, according to one design of this method, and particularly likely, the location and spatial width of the visible distance area are defined by appropriately selecting the timing control of both the lighting device and the optical sensor.

[0018] In an alternative design of this method, a visible distance range can be defined, thereby determining and accordingly setting the time coordination between the lighting device and the optical sensor.

[0019] In the preferred design, the lighting device is a laser. In the preferred design, the optical sensor is a camera.

[0020] According to an improved embodiment of the invention, the object is detected in a second photograph, and the distance between the object and the optical sensor is determined. Therefore, continuous object detection and / or object tracking can be achieved by adjusting the visible distance region.

[0021] According to an improvement of the invention, the object tracking method is performed alternately for at least two different first visible distance regions. Advantageously, this improvement of the method can be used to detect and track at least two different objects. For each of the at least two different objects, the corresponding second visible distance region is adjusted based on distance measurements.

[0022] According to an improved embodiment of the invention, the first control of the optical sensor and the illumination device is coordinated such that the first coordinated control corresponds to a first visible distance region as a first first visible distance region and a second first visible distance region. First photographs of the first and second first visible distance regions are taken using the optical sensor with the aid of the illumination device and the first coordinated control. A first distance measurement is performed when a first object is detected in the first first visible distance region in the first photograph to determine a first distance between the optical sensor and the first object. A second distance measurement is performed, alternatively or additionally, when a second object is detected in the second first visible distance region in the first photograph to determine a second distance between the optical sensor and the second object. Then, the second coordinated control of the illumination device and the optical sensor, and the second visible distance region corresponding to the first second visible distance region and the additional second visible distance region corresponding to the second coordinated control, are determined such that the first object is centered in the first second visible distance region and / or the second object is centered in the second second visible distance region. Then, second photographs of the first and second second visible distance regions are taken using the optical sensor with the aid of the illumination device and the second coordinated control. Advantageously, it is possible to detect and track two different objects simultaneously.

[0023] According to an improved embodiment of the invention, the at least one detected object is classified, in particular, by means of deep learning.

[0024] This task is also accomplished by providing a control device configured to perform the method of the present invention or the method according to one of the foregoing embodiments. The control device is preferably designed as a computing device, especially preferably as a computer, or as a controller, particularly a vehicle controller. In particular, the advantages already explained with respect to the method are obtained in connection with the control device.

[0025] This also accomplishes the task by providing an object tracking device having an illumination device, an optical sensor, and a control device of the present invention or a control device according to one of the foregoing embodiments. In particular, the advantages already explained with respect to the method and the control device are obtained in connection with the object tracking device.

[0026] The control device is preferably connected to both a lighting device and an optical sensor, and is configured to control both. In a preferred embodiment, the control device is further configured to determine the distance between the optical sensor and at least one detected object.

[0027] Ultimately, this task is accomplished by providing a vehicle equipped with the object tracking device of the present invention or an object tracking device according to one of the foregoing embodiments. The advantages explained with respect to the method, control device, and object tracking device are particularly advantageous in relation to motor vehicles.

[0028] In a favorable design, the motor vehicle is designed as a truck. However, it is also possible that the motor vehicle is a car, a truck, or another type of motor vehicle. Attached Figure Description

[0029] The invention will be explained in detail below with reference to the figures, wherein:

[0030] Figure 1 A schematic diagram of a motor vehicle embodiment having an object tracking device and a schematic diagram of a first photograph of a first visible distance area are shown.

[0031] Figure 2 A schematic diagram of a motor vehicle with an object tracking device and a second photograph showing a second visible distance area. Detailed Implementation

[0032] Figure 1 a) A schematic diagram showing an embodiment of a motor vehicle 1, which includes an embodiment of an object tracking device 3. The object tracking device 3 includes an illumination device 5 and an optical sensor 7. Additionally, the object tracking device 3 includes a control device 9, which is shown only schematically herein and is effectively connected to the illumination device 5 and the optical sensor 7 in a manner not explicitly shown for their respective control. Figure 1The illumination cone 11 of the illumination device 5 and the observation area 13 of the optical sensor 7 are shown in particular. Furthermore, the shaded area represents a first visible distance region 15.1, which is a subset of the observation area 13 of the optical sensor 7. The object 17 is partially arranged within the first visible distance region 15.1.

[0033] Figure 1 b) A schematic diagram showing a first photograph 19.1 taken using the optical sensor 7 by means of first coordinated control of the lighting device 5 and the optical sensor 7. In the first photograph 19.1, the image of the first visible distance region 15.1 is marked with 15.1′. Similarly, the image of the object 17 is marked with 17′ in the first photograph 19.1.

[0034] Figure 2 It is immediately following in time Figure 1 The view of the scene following the scene. Figure 1 Scene and Figure 2 The time difference between scenes should ideally be less than 0.1s, and especially less than 0.01s.

[0035] Figure 2 a) A motor vehicle 1 equipped with an object tracking device 3 is shown. The object tracking device 3 includes a lighting device 5 and an optical sensor 7. Additionally, the object tracking device 3 includes a control device 9. Similar to... Figure 1 On the ground, Figure 2 The illumination cone 11 of the illumination device 5 and the observation area 13 of the optical sensor 7 are particularly shown. Furthermore, the shaded area indicates a second visible distance region 15.2, which is a subset of the observation area 13 of the optical sensor 7. The object 17 is centered within the second visible distance region 15.2.

[0036] Figure 2 b) A schematic diagram showing a second photograph 19.2 taken using optical sensor 7 by means of a second coordinated control of illumination device 5 and optical sensor 7. In the second photograph 19.2, the image of the second visible distance region 15.2 is marked with 15.2′. Similar to... Figure 1 In the second photograph 19.2, the image of object 17 is labeled 17′.

[0037] The following method is used to perform object tracking and adapt the second visible distance region 15.2 to the local location of object 17.

[0038] By coordinating the illumination device 5 and the optical sensor 7, a first photograph 19.1 of a first visible distance region 15.1 is captured. In the first photograph 19.1, an object 17 is detected in image 15.1′ of the first visible distance region 15.1. The distance of the object 17 from the optical sensor 7 is determined using a suitable method.

[0039] Based on the distance measurement of object 17, the second coordination control and the corresponding second visible distance region 15.2 are determined such that object 17 is centered within the second visible distance region 15.2. Finally, under the illumination of the illumination device 5, the optical sensor 7 is used with the aid of the second coordination control to capture a second photograph 19.2 of the second visible distance region 15.2.

[0040] As a supplement, the distance between object 17 and optical sensor 7 can be determined again in the second photograph 19.2. Furthermore, the object tracking method can be continuously executed by means of repeated distance determinations.

[0041] Alternatively or additionally, the detected object 17 can be classified, particularly with the aid of deep learning. Thus, it is advantageous to classify the detected object 17 and determine whether object tracking is required.

Claims

1. A method for object tracking of at least one object (17) in at least two consecutive photographs (19.1; 19.2) taken with the aid of an illumination device (5) and an optical sensor (7), wherein, - The control of the lighting device (5) and the optical sensor (7) is coordinated with each other in time. - The first visible distance region (15.1) corresponds to the first coordinated control of the lighting device (5) and the optical sensor (7). - The first photograph (19.1) of the first visible distance area (15.1) was taken using the optical sensor (7) with the aid of the first coordination control under the illumination of the illumination device (5). - When at least one object (17) is detected within the first visible distance region (15.1) in the first photograph (19.1), the distance between the at least one detected object (17) and the optical sensor (7) is determined, and - The second coordinated control of the lighting device (5) and the optical sensor (7) and the corresponding second visible distance region (15.2) are determined such that the object (17) detected in the first photograph (19.1) is centered within the second visible distance region (15.2). - The second photograph (19.2) of the second visible distance area (15.2) was taken using the optical sensor (7) with the aid of the second coordination control under the illumination of the illumination device (5).

2. The method according to claim 1, wherein, The at least one object (17) is detected in the second photograph (19.2), and the distance between the detected object (17) and the optical sensor (7) is determined.

3. The method according to claim 1, wherein, The object tracking method is performed alternately for at least two distinct first visible distance regions.

4. The method according to claim 2, wherein, The object tracking method is performed alternately for at least two distinct first visible distance regions.

5. The method according to any one of claims 1-4, wherein, - The first control of the lighting device (5) and the optical sensor (7) is coordinated such that the first visible distance region (15.1), which is a first first visible distance region and a second first visible distance region, is correspondingly assigned to the first coordinated control. - The first photograph (19.1) of the first first visible distance area and the second first visible distance area was taken using the optical sensor (7) with the aid of the first coordinated control under the illumination of the illumination device (5). - When the first object (17) is detected in the first visible distance area in the first photograph (19.1), the first distance between the first object (17) and the optical sensor (7) is determined, and - When a second object is detected within the second first visible distance area in the first photograph (19.1), a second distance between the second object and the optical sensor (7) is determined, and - The second coordinated control of the lighting device (5) and the optical sensor (7), and the corresponding second visible distance region (15.2) as the first second visible distance region and the corresponding second second visible distance region, are determined such that the first object (17) is centered within the first second visible distance region and / or the second object is centered within the second second visible distance region. - Using the optical sensor (7) under the illumination of the illumination device (5), a second photograph (19.2) of the first second visible distance area and the second second visible distance area is taken with the aid of the second coordinated control.

6. The method according to any one of claims 1-4, wherein, The at least one object being probed (17) is classified.

7. The method according to claim 6, wherein, The at least one object being probed (17) is classified using deep learning.

8. The method according to claim 5, wherein, The at least one object being probed (17) is classified.

9. The method according to claim 8, wherein, The at least one object being probed (17) is classified using deep learning.

10. A control device (9) configured to perform the object tracking method according to any one of claims 1-9.

11. An object tracking device (3) having an illumination device (5), an optical sensor (7) and a control device (9) according to claim 10.

12. A motor vehicle (1) having an object tracking device (3) according to claim 11.

Images