Device for spatial image acquisition of an environment moving relative to the device, and operating methods for this purpose
A stereo camera and time-of-flight system synchronized with flash illumination enhances object distance determination, addressing the challenge of moving environments in assisted or autonomous driving, ensuring precise and reliable image capture.
Patent Information
- Application Number
- DE102023205343
- Authority / Receiving Office
- DE · DE
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-06-07
- Publication Date
- 2024-12-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing spatial image capture devices struggle to accurately determine object distances in environments where objects are moving relative to the device, particularly in applications like assisted or autonomous driving, due to limitations in triangulation and time-of-flight methods.
A device combining a stereo camera with a flash light source and sensor array for time-of-flight processes, synchronized to enhance triangulation and time-of-flight image capture, allowing for precise distance determination by coordinating focal length, baseline, and disparity, with redundant systems for increased reliability.
The solution provides precise and reliable object distance detection, enabling safe and predictive image capture suitable for driver-assisted and autonomous driving, even in challenging conditions like fog, smoke, or dust, with reduced motion blur and improved contrast.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The invention relates to a device for capturing spatial images of an environment that is moving relative to the device. Furthermore, the invention relates to a method for operating such a device.
[0002] Such devices are known from WO 2022 / 069 424 A1, WO 2022 / 069 425 A2, WO 2022 / 179 998 A1 and DE 10 2011 080 102 B3.
[0003] It is an object of the present invention to provide a device for spatial image acquisition which is well adapted to the practice, in particular of image acquisition for ensuring assisted or autonomous driving.
[0004] This object is achieved according to the invention by a method having the features specified in claim 1.
[0005] According to the invention, it was recognized that the functions of at least one stereo camera and an image capture system using a time-of-flight illumination method complement each other advantageously. Both the stereo camera and the components enabling a time-of-flight illumination method, namely the flash source and the sensor array, allow the determination of the distance between objects within the environment.
[0006] The device can be used for spatial image capture of an environment moving relative to the device. The at least one stereo camera can be used for triangulatory object distance determination within the captured environment in the forward direction of a relative movement of the device to the environment. This can be used to improve the reliability of environmental detection, particularly in connection with driver-assisted or autonomous driving. The device can be a component of a vehicle, for example, a car or truck.
[0007] The control device can, in particular, specify a delay period between the start of flash illumination of the at least one flash light source and the start of detection by the respective sensor array of the cameras of the at least one stereo camera. The distance between the cameras of the stereo camera results in a baseline length, i.e., a distance between the centers of the entrance pupils of the cameras of the at least one stereo camera. The device can combine the principles of a TOF (Time of Flight) camera with a triangulatory distance-measuring stereo camera. A corresponding TOF image acquisition is described, for example, in US 2019 / 056 498 A1. The flash light source can be designed in a way that is known in principle from TOF cameras or LIDAR systems. The same applies to the sensor arrays of the cameras. A sensor array used for TOF image acquisition can be constructed as a single-photon avalanche diode (SPAD) array.Such a SPAD array and its application in connection with a TOF measurement is known from a scientific article by T. Swedish et al.: "Beyond the Line of Sight? What's New in Optical Perception", AUVSI XPONENTIAL 2022-SWEDISH (conference handout), pp. 1-8, https: / / www.xponential.org / xponential2022 / Custom / Handout / Speaker5158 4_Session4220_1.pdf and from US 2022 / 0174255 A1.
[0008] For triangulatory object distance determination, the two cameras of the stereo camera are synchronized with each other via a synchronization unit. This synchronization enables, in particular, the exact simultaneous capture of the respective object by the stereo camera cameras. This distinguishes a stereo camera for triangulatory object distance determination from a stereoscopic camera, which exclusively creates a spatial impression of an image capture and is described, for example, in US 2015 / 296 200 A1. The use of a SPAD array can lead to a reduction in motion blur and can be used to improve the contrast of an image capture.
[0009] Synchronization and, in particular, simultaneous object detection in triangulatory object distance determination makes it possible to precisely detect the distance of even externally moving objects, since the influence of an additional relative speed due to the external movement of the object does not affect the distance determination.
[0010] A focal length f of the respective camera of a stereo camera, the baseline b of the stereo camera and a disparity D, i.e. a difference between image coordinates of imaging positions of the same point on the sensor arrays of the cameras of the stereo camera, are coordinated in such a way that, taking into account the relationship d=fb / D For the object distance value d to be recorded, the following applies: 3 m≤d≤300 m
[0011] In addition, the device can be designed such that a corresponding object distance range is detected via the TOF image acquisition functionality of the device.
[0012] The flash light source can be a controlled flash lamp or a time-controlled LED or laser light source. Flash light sources known for use in LIDAR devices, for example, can be used.
[0013] The flash illumination duration of the flash light source can be in the range between 10 ns and 200 ns, for example, in the range of 100 ns. The detection duration of the sensor arrays of the at least one stereo camera is usually specified to be twice the flash illumination duration of the flash light source. The detection duration can be in the range between 20 ns and 400 ns, for example, in the range of 200 ns.
[0014] Both the flash illumination duration and the detection duration can be set via the control device.
[0015] The synchronization unit can be part of the control device. A memory unit, particularly in the form of a flash memory, can be part of the control device. The synchronization unit can be part of a synchronization device for synchronizing the flash light source and the sensor arrays of the cameras of the at least one stereo camera. The synchronization device can, in turn, be part of the control device of the image capture device.
[0016] Synchronization that can be provided by the synchronization device can be better than 5 ns, can be better than 2 ns, can be better than 1 ns, and can be even better.
[0017] The stereo camera's cameras can be spaced apart laterally relative to the forward direction. Depending on the cameras' detection angle, the application, and the stereo camera's primary detection direction, other camera arrangements are also possible.
[0018] The cameras of at least one stereo camera can be equipped with a telephoto lens for capturing objects at distances between 50 m and 300 m. Depending on the application, the stereo camera cameras can also be equipped with lenses of other focal lengths. These camera lenses are designed to cover an entire specified distance range, particularly between 3 m and 300 m. In principle, even shorter distances can be covered.
[0019] The distance ranges of triangulatory object detection on the one hand and TOF distance detection on the other are coordinated and can, for example, be identical. Partial overlap of these distance ranges is also possible, so that both "triangulatory" and "TOF" distance determination principles are available in one distance range, and one of these principles is available in other distance ranges.
[0020] Using the cameras of the at least one stereo camera, a correspondence analysis can be performed to identify objects that differ from the background and are present in a field of view of interest. Methods for such correspondence analysis are known from WO 2022 / 069 424 A1 and WO 2022 / 069 425 A2. In preparation for operation of the device, a stereo calibration of the cameras of the at least one stereo camera can be performed, which is generally known from WO 2022 / 069 424 A1 and DE 10 2011 080 102 B3.
[0021] The at least one stereo camera may comprise more than two cameras spaced apart from each other, which in particular enables switching between different baselines, which is also described, for example, in WO 2022 / 069 424 A1.
[0022] Further object information, in particular object distance information, can be obtained by evaluating an object's shadow, which is cast by an object within the captured environment as a shadow of the flash illumination and is imaged differently by the two spaced-apart cameras of the at least one stereo camera. In particular, the height of the object can be determined because an image of an object's shadow depends on the object's elevation and lateral offset.
[0023] The device can be used to capture spatial images of the environment, whether moving or stationary relative to the device, which, for example, enables a driver-assisting function or even the function of autonomous driving when the device is used as part of a vehicle.
[0024] The flash light source of the device or another light source of the device for ambient illumination can have a measuring light beam that is textured or structured across its beam cross-section. Such texturing / structuring of the measuring light beam can be achieved by constructing it as a plurality of individual beams that cover the environment in the form of a grid. Such a light source for generating a textured / structured measuring light beam can be embodied as a laser. Examples of a detection device that operates using such a textured / structured measuring light beam can be found in DE 10 2018 213 976 A1 and the references cited therein, in particular in WO 2012 / 095 258 A1, US Pat. No. 9,451,237, and WO 2013 / 020 872 A1.Using such textured / structured lighting, optically unstructured, particularly diffuse, scattering media within the encompassed environment can be measured with respect to their distance. Such diffuse scattering media can be, for example, fog, smoke, or dust.
[0025] Using the spatial image acquisition device, shape redundancy and / or functional redundancy can be achieved. Shape redundancy is achieved when different technologies are used to determine the same measurand. Functional redundancy exists when multiple devices of the same technology are used to measure the same measurand. The terms "shape redundancy" and "functional redundancy" are primarily used in the aviation sector. There, a combination of shape and functional redundancy is required to prevent the multiple effects of a fault. Such redundancies may be required by law, depending on the device's application.
[0026] At least one additional flash source and / or at least one additional stereo camera according to claim 2 enable additional functional redundancy of the device. If at least one additional flash source is provided, switching to the additional flash source is possible, for example, after the failure of the initially used flash source, thus avoiding a prolonged downtime of the device. Using at least one additional stereo camera, for example, with parallel triangulation measurements using two stereo cameras, further redundancy in object distance determination can be achieved.
[0027] The cameras of the additional stereo camera can each have a sensor array with a controlled preset detection start and duration, so that the additional stereo camera can essentially replace the original stereo camera in its function. Redundant TOF determination of object distances is also possible using two corresponding stereo cameras with controlled preset detection start and duration, which provides additional reliability in object distance determination and helps detect synchronization problems, for example.
[0028] When using multiple stereo cameras, it is also possible to assign cameras from different stereo cameras to a new "virtual" stereo camera with a correspondingly modified baseline, which provides further redundancy and reliability in object distance determination. For example, the camera pair that proves to be particularly favorable for triangulation determination with respect to a specific surrounding object can be selected as the baseline.
[0029] At least one fisheye stereo camera according to claim 3 enables the use of additional triangulation baselines, which can be used to detect additional surrounding objects and / or to check object distance results from the other stereo cameras and / or to calibrate the other stereo cameras, in particular stereo cameras implemented with telephoto lenses. The main detection direction can be the forward direction of a relative movement of the device to the surroundings. The use of fisheye cameras is known from the above-mentioned references and from WO 2022 / 179 998 A1. A fisheye camera that is attached as rigidly as possible to a telephoto lens camera allows for more precise calibration of the stereo cameras to each other due to the larger aperture angle of the fisheye camera. An inertial measurement unit (IMU) in the cameras of the respective stereo cameras allows for further optimization of the relative calibration.In particular, the IMU can be used to determine a rotation rate, i.e. a relative tilt of the respective camera around a defined tilt axis.
[0030] The device can contain multiple fisheye stereo cameras, which allows for additional redundancy. The cameras of the at least one fisheye stereo camera can also each have sensor arrays with controlled presetting of a detection start and a detection duration, thus enabling a TOF determination of object distances. In principle, the at least one fisheye stereo camera can also be designed without a TOF function. If the device has multiple fisheye stereo cameras, it is also possible for at least one of the fisheye stereo cameras to be designed with a TOF function and at least one other of the fisheye stereo cameras to be designed without a TOF function.
[0031] In principle, the device according to the above claims can be used in such a way that precisely one distance range is detected by means of the TOF components, for example in the distance range between 30 m and 300 m, in particular between 100 m and 200 m, from the device.
[0032] The object mentioned at the outset is also achieved by an operating method having the features specified in claim 5.
[0033] The operating method according to claim 5 enables real-time operation of the device, so that, for example, autonomous driving is possible when the device is used as part of a vehicle.
[0034] In the operating method according to claim 5, a covered total distance range is divided into a plurality of adjacent or partially overlapping distance ranges. For example, three to 100 such distance ranges can be specified using corresponding image acquisition parameters, for example, 50 distance ranges that lie within the total distance range. During operation of the device, the total distance range can be acquired across the specified individual distance ranges several times per second, for example, at a rate of 10 Hertz. During operation, the acquisition can begin with the distance range furthest away within the total distance range.This ensures that when images acquired in a previous acquisition step are overwritten with the most recently acquired image, nearby objects overwrite more distant objects in non-filtered image areas of the most recently acquired image. This ensures that the result of the operating method, namely the most recently updated, output image, always represents the objects closest to the device, which can then be responded to, for example, by intervening in a movement of the device. Buffering of images acquired in previous acquisition steps can then be eliminated, which increases the speed of the process.
[0035] The last updated output image is a 2D image with unfiltered objects in all distance ranges captured within the overall distance range. Information from previously captured distance ranges within the overall distance range that was overwritten during the operating procedure can also be retained for independent spatial (3D) analysis.
[0036] For the non-filtered objects assigned to the respective detection step, the method results in a distance range in which the object is located relative to the device. This resulting object distance information, determined using TOF image acquisition, in conjunction with triangulatory distance detection enabled by the at least one stereo camera, enables form redundancy of the operating method.
[0037] The distance information can be output as the smallest distance of the detected object or as a distance per object pixel of the respective sensor array.
[0038] The overall result is both safe and predictive image capture, which is well adapted to the requirements of driver assistance or autonomous driving in particular.
[0039] When digitally filtering to filter out structureless image areas, a stereo correspondence determination can be used, which is described, for example, in WO 2022 / 069 424 A1.
[0040] A temporal overlap according to claim 6 enables a rapid execution of the operating method. The temporal overlap can be such that at least two distance ranges with simultaneous detection start and the same detection duration are simultaneously detected by the sensor arrays, whereby these detection ranges are specified by different specifications of the respective flash illumination start and / or the respective flash illumination duration with regard to their image acquisition parameters. It is also possible to design the repetition sequence of the operating method such that a detection actually occurs at exactly one time period, i.e. at a predetermined detection start and with a predetermined detection duration, for all predetermined distance ranges. This simultaneous detection of at least two detection ranges can reduce image processing effort and accelerate the entire method.
[0041] Differing distance extensions of the distance ranges according to claim 7 make it possible, in particular, to specify a distance sensitivity depending on the distance, which also helps to shorten the duration of the operating method. With increasing distance, for example, the distance extension of the respective distance range can be increased, in particular progressively increased.
[0042] An additional triangulation determination according to claim 8 enables a comparison of object distance values which were determined on the one hand via the distance-resolved detection using the TOF image acquisition parameters and on the other hand via the triangulation.
[0043] A ROI specification according to claim 9 enables a reduction in the image data to be processed, which can also shorten the method. The specified ROI can be a road area in the forward direction.
[0044] An operating method according to claim 10 can be used alternatively or in addition to the operating method explained above. Such an operating method enables redundant distance determination of a respective object distance. This can, in particular, provide form redundancy, i.e., redundancy provided by using different distance detection technologies.
[0045] The use of the detected TOF distance as an input variable for the triangulatory detection according to claim 11 can facilitate correspondence analysis during triangulatory detection. In particular, the correspondence analysis can be used to narrow down a disparity range to be covered. Conversely, the triangulatory detected distance can also be used as an input variable for TOF detection. The use of a detected object distance can be used in the operation of the device for tracking or tracing detected objects. Exactly one such object can be tracked, or a plurality of such objects can be tracked simultaneously. Object tracking steps can alternate with detection steps of an entire predefined distance range.For object tracking, image acquisition parameters of the TOF distance detection and / or the triangulatory distance detection can be adapted to the object distance expected during tracking.
[0046] A flash intensity / distance range adjustment according to claim 12 makes it possible, in particular, to draw conclusions about diffuse scattering media present in the flashlight and / or camera detection path. Safe device operation is then possible even in foggy, dusty, or smoky environments.
[0047] A quadratic dependence of the flash intensity on the distance to be detected according to claim 13 enables, particularly when the illumination intensity increases quadratically with increasing distance range, a tuning such that, with identical ambient backscatter and negligible scattering along a flash path and along a detection path between source and object, a constant intensity illumination of the at least one sensor array is controlled. This can be used to determine diffuse scattering media in the detected distance range.
[0048] A pixel sensitivity control according to claim 14 can, in particular, be non-linear with the incident light intensity. In particular, a gamma correction (γ) can be performed. The sensitivity of the at least one sensor array can then be specifically adapted to the lighting and / or detection conditions prevailing in the application.
[0049] An adapted distance range / movement mode selection according to claim 15 enables efficient utilization, in particular, of a computing power of the device and can also increase operational reliability.
[0050] When evaluating with triangulatory stereoscopy in binocular vision, two images are compared, so that any brightness changes due to changing density gradients can be neglected to a first approximation. When comparing the images with trained data (neural networks, AI) in monocular vision, a homogeneous brightness distribution over distance increases detection performance, whereas changing density distributions make reliable object recognition more difficult.
[0051] Embodiments of the invention are explained in more detail below with reference to the drawings, in which: Fig. 1 schematically shows a plan view of a device for spatial image capture of an environment moving relative to the device, wherein an environmental object is illustrated by way of example and in perspective in the forward direction of a relative movement of the device to the environment; Fig. 2 a time diagram to illustrate image acquisition parameters during operation of the device according to Fig. 1; Fig. 3 in one to Fig. 1 shows a further embodiment of a device for spatial image capture of an environment moving relative to the device, wherein environmental objects are illustrated by way of example and in perspective in the forward direction of a relative movement of the device to the environment; Fig. 4 in one to Fig. 2 similar representation a temporal sequence of an operating method of the device according to Fig. 3; and Fig. 5 in a side view of main components of an embodiment of the device in the spatial image capture of an environment within three exemplary distance parameters, wherein a cloud-shaped diffuse scattering medium is present within the captured environment.
[0052] Fig. 1 shows a plan view of a device 1 for spatial image capture of an environment moving relative to the device 1, which is illustrated by an environmental object 2 in the form of a cylindrical body.
[0053] The device 1 can be part of a vehicle 3, for example a car or a truck. A forward direction of the relative movement of the device 1 to the environment is shown in the Fig. 1 is illustrated by an arrow 4.
[0054] The device 1 has several stereo cameras for triangulatory object distance measurement of objects within the detected environment in the forward direction 4, i.e. in particular for determining the distance of the object 2.
[0055] A first stereo camera 5 of these stereo cameras has two cameras 51, 52 spaced apart from each other. A distance A between these cameras 51, 52 of the stereo camera 5 results in a baseline length of the stereo camera 5, i.e., a length of a distance between the centers of the entrance pupils of the cameras 51, 52 of the stereo camera. The cameras 51, 52 of the stereo camera 5 each have a sensor array 61, 62 with controlled specification of a detection start of the respective camera 51 and 52 and a detection duration of the respective camera 51, 52. The sensor arrays 61, 62 are designed like corresponding sensors of time-of-flight (TOF) cameras and can realize a detection duration in the range between 1 ns and 500 ns. A typical detection time is in the range between 50 ns and 400 ns, for example between 150 ns and 250 ns.
[0056] The sensor arrays 61 and 62 can each have 100 × 100 pixels or 200 × 200 pixels. A higher number of pixels is also possible, in particular 640 × 480 or 1440 × 1080 pixels.
[0057] The sensor arrays 61, 62 can be designed as CMOS arrays.
[0058] The cameras 51, 52 have a telephoto lens for detecting objects at distances between 50 m and 300 m.
[0059] The device 1 also includes a flash light source 7, which can also be designed like the light source of a TOF camera and allows for controlled setting of a flash illumination start time and flash illumination duration for successive illumination flashes. A flash illumination duration can be in the range between 5 ns and 150 ns, for example, in the range of 100 ns.
[0060] Fig. Figure 2 illustrates a temporal sequence during the operation of the device 1 with the stereo camera 5 and the flash light source 7. At a time t = 0, an illumination flash of the flash light source 7 begins. t = 0 thus represents the illumination flash start of a Fig. 2 schematically shown as a signal box. A flash duration of the lighting flash 8 is shown in the Fig. 2 with Δt flash specified.
[0061] Also at time t = 0 a delay time Δt starts lag , which is also called delay time. This delay time Δt lag is in the range between 300 ns and 2 ms and can, for example, be in the range between 500 ns and 1 ms, for example in the range between 700 ns and 900 ns, for example 800 ns.
[0062] After the delay period Δt lag begins at time t = t Ea detection 9 by the respective sensor array 61 or 62 of the stereo camera 5. t E thus represents the start of detection. The respective sensor array 61, 62 then detects incident light from the flash light source 7 during a detection period Δt exp .
[0063] The detection 9 by the respective sensor array 61, 62 of the stereo camera 5 is in the Fig. 2 is again illustrated by a time box.
[0064] The image acquisition parameters explained above, namely the flash illumination start (t = 0), the flash illumination duration Δt flash , the start of detection (t = t E ) and the detection time Δt exp are specified in a time-synchronized manner by a control device 10 of the device 1. For this purpose, a synchronization unit of a synchronization device of the control device 10 can be used.
[0065] The control device 10 has an image processor for real-time data processing. This can be a processor with a field-programmable gate array (FPGA). The image processor can perform ultrafast image grabbing of the images captured by the sensor arrays 6, triggered by the control device 10. i captured individual images may be trimmed.
[0066] Due to the synchronized specification of the start of the flash illumination and the start of the detection, the control device 10 necessarily also specifies the delay period Δt lag between the start of the illumination flash (t = 0) and the start of detection (t = t E ) of the device 1.
[0067] Correlating with the speed of light, this results in a distance range 11 in the forward direction 4 of the device 1, captured by the stereo camera 5. The object 2 is at a distance from the device 1 along the forward direction 4 that lies within this distance range 11, so that the object 2 is captured by the stereo camera 5. The captured distance range 11 is approximately 100 m to 150 m from the device 1 and has an extension of, for example, 30 m along the forward direction 4.
[0068] This distance is in the Fig. 1 is given at 12 and is not to scale compared to distance range 11.
[0069] Due to the detection via the two cameras 51, 52 of the stereo camera 5, a spatial image of the object 2 is recorded. Triangulation can then be carried out redundantly for the TOF distance measurement over the delay period Δt lagA reference distance determination of the object 2 to the device 1 can also be made. Device 1 is therefore essentially a TOF camera with additional triangulation.
[0070] The stereo camera 5 can also perform a correspondence analysis of individual camera images captured by the two cameras 51, 52 in order to separate stereoscopically captured objects from background noise present only in one of the two cameras 51, 52. Such a correspondence analysis is described, for example, in WO 2022 / 069424 A1 and WO 2022 / 069425 A2.
[0071] Initially, a stereo calibration of the stereo camera 5, in particular of the orientations and positions of the cameras 51, 52 of the stereo camera 5, can be performed using a method also described in WO 2022 / 069424 A1. Regarding the calibration, reference is also made to DE 10 2011 080 102 B3.
[0072] During spatial image acquisition using device 1, a TOF distance of at least one detected object, acquired using the TOF principle, can be compared with a triangulatory distance of the object, acquired using the triangulatory principle by the at least one stereo camera. The TOF distance can be used, for example, as an input variable for the triangulatory acquisition. This can facilitate correspondence analysis during the triangulatory acquisition, for example, by limiting a disparity range to be covered by the correspondence analysis.
[0073] Using the images of the object 2 illuminated by the flash light source 7 captured by the cameras 51, 52 of the stereo camera 5, a shadow cast by the object 2 can also be captured for the illumination light of the flash light source 7. By analyzing the shadow cast of the object 2 in this way, recorded by the two cameras 51, 52 of the stereo camera 5, an additional redundant determination of the distance of the object 2 from the device 1 is possible.
[0074] The device 1 has an additional flashlight source 13, whose function corresponds to that of the flashlight source 7, thus also serving for the controlled specification of a flashlight start and a flashlight duration of successive illumination flashes. The additional flashlight source 13 can be used redundantly to the initially described flashlight source 7, for example, in the event of a failure of the initially described flashlight source 7. During operation of the additional flashlight source 13, the initially described flashlight source 7 can then be replaced, for example, so that uninterrupted operation of the device 1 is possible, for example, during a monitored journey of the car or truck on which the device 1 is mounted.
[0075] The device 1 has a further stereo camera 14 with cameras 141, 142, whose structure and function correspond to what was already explained above with reference to the stereo camera 5. The stereo camera 141 is arranged directly adjacent to the stereo camera 51 and covers practically the same field of view in the forward direction 4 as the camera 51. The camera 142, in turn, is arranged directly adjacent to the camera 52 and covers the same field of view in the forward direction 4 as the camera 52.
[0076] The stereo camera 14 is in turn redundant to the stereo camera 5. The same applies here as described for the flash light sources 7 and 13.
[0077] The device 1 further comprises a fisheye stereo camera 15 with two pairs 151, 152 on the one hand and 153, 154 on the other. Each of these fisheye cameras 15 i(i = 1 to 4) is used to detect the environment around the device 1 both in the forward direction 4 and in the lateral direction (cf. arrow 16 in Fig. 1). The two in the Fig. 1 fisheye cameras 151 and 152 arranged on the left side of the vehicle 3 have a main detection direction 17, which, like the main detection directions of the cameras 5 i and 6 i runs horizontally, but in the case of the fisheye cameras 151 and 152, forms an angle of 45° to the forward direction 4. These fisheye cameras 151, 152 thus cover both the forward direction 4 and the lateral direction opposite to the lateral direction 16 (left in the Fig. 1) of the vehicle 3 on which the device 1 is mounted. Accordingly, the two additional fisheye cameras 153, 154, whose main detection directions 18 also form an angle of 45° to the forward direction 4, cover not only the forward direction 4 but also a right-side area in the lateral direction 16.
[0078] The fisheye cameras 15 i have a detection angle of 180° around the main detection directions 17, 18. The limit angle of the detection ranges of these fisheye cameras is 15°. i are in the Fig. 1. These opening areas overlap in an overlap area 19, which is shown in the Fig. 1 is indicated by hatching. Object 2, in particular, is located in this overlap area 19.
[0079] The illumination light source 7 emits flash light of the illumination flashes along the forward direction in an illumination cone, the edge rays of which are in the Fig. 1 are illustrated at 8a. The object 2 and also the overlap area 19 lie within the illumination cone 8a of the flash light source 7.
[0080] The fisheye stereo camera 15 can optionally perform a triangulatory object distance determination using at least two of the four fisheye cameras 15 i This can be done with short baselines (distances between fisheye cameras 151, 152 on the one hand, and 153, 154 on the other), or with long baselines (distances between fisheye cameras 151 and 153, or 151 and 154, or 152 and 153, or 152 and 154). This, in turn, allows for redundant triangulation of the object distance.
[0081] The use of such additional triangulation baselines is described, for example, in WO 2022 / 069424 A1 and WO 2022 / 179998 A1.
[0082] The device 1 also has another fisheye stereo camera 20 with fisheye cameras 201 to 204, which correspond in their structure, orientation, and paired arrangement to the fisheye cameras 151 to 154. The fisheye stereo camera 20 can be used redundantly to the fisheye stereo camera 15. The above statements regarding the flash light sources 7 and 13 and the stereo cameras 5 and 14 apply accordingly.
[0083] The Cameras 15 i the fisheye stereo camera 15 in turn have sensor arrays similar to the sensor arrays 6 described above i , can therefore be used as part of a TOF acquisition, as already explained above.
[0084] The control device 10 is in signal communication with the stereo cameras 5, 14, 15, and 20 and with the flash light sources 7 and 13. Furthermore, the control device 10 can be in signal communication with components of the vehicle 3, for example, with a drive and / or with a braking system and / or with vehicle display devices, for example in a dashboard of the vehicle 3.
[0085] In real-time operation, the device 1 can record images at, for example, ten images per second.
[0086] Fig. Figure 3 shows a further embodiment of a device 21 that can be used instead of device 1 for spatial image capture of an environment moving relative to the device. Components and functions of device 21 that correspond to those of device 1 bear the same reference numerals and will not be discussed in detail again.
[0087] The device 21 is equipped with two stereo cameras 5, 14, which capture the surroundings of the device 21, which is in turn mounted on a vehicle 3, in the forward direction 4. Three surrounding objects 22, 23, and 24 are illustrated within this captured environment.
[0088] With the device 21, in particular, a z-buffered operation in the forward direction 4, which is also referred to as the z-direction, is possible, in which a total distance range 25 along the forward direction 4, within which the objects 22 to 24 lie, is divided into a plurality of distance ranges 221 to 22 max , i.e. in distance ranges 22 i (i = 1 to max). The maximum number of these distance ranges is 22 i can be in the range between 3 and 50, for example in the range of 10. These distance ranges 22 i are also called gate zones (gz).
[0089] During the corresponding Z-buffering operation of the device 21, the image acquisition parameters illumination flash start (t = 0), illumination flash duration (Δt flash, gz-1 ), a detection start (t = t E ) and a detection time (Δt exp, gz-1 ) specified.
[0090] This image acquisition parameter specification is again carried out by means of the control device 10.
[0091] The distance range gz-1, for example the distance range 22 max , is then captured using the specified image acquisition parameters.
[0092] Further image acquisition parameters are specified via the control device 10, namely a further flash start (t = t flash ), another flash duration (Δ flash, gz ), another detection start (t = t E1 ) and a detection time (Δt exp, gz ). This specification is chosen so that a delay period (Δt lag , gz) with these additional predefined image acquisition parameters is slightly shorter than the initially predefined delay period Δt lag, gz-1 This means that when capturing with the last specified image acquisition parameters, a further detection range gz is captured, which exceeds the initially captured distance range gz-1, i.e. the distance range 22 max , adjacent to the device 21 at shorter distances.
[0093] The further distance range gz borders on or partially overlaps the previous distance range gz-1 along the forward direction 4 in a detection progress direction +z.
[0094] Object 24 is located in this now recorded detection area gz.
[0095] The last acquired image in the distance range gz is now digitally filtered to filter out structureless image areas, leaving only the object 24. Subsequently, the image acquired in the acquisition range preceding the acquisition of the distance range gz (acquisition of the distance range gz-1) is overwritten with the last acquired image (distance range gz) in the unfiltered image areas, i.e., in the area of the object 24, of this last acquired image, so that an updated image is stored in the control device 10, which contains the information from the two distance ranges gz-1 and gz.
[0096] During operation, the control device 10 specifies further image acquisition parameters so that a further distance range gz+1 is acquired, which in turn adjoins the distance range gz opposite to the forward direction 4 or partially overlaps with it.
[0097] Now the last described steps of capturing, filtering, overwriting and specifying image acquisition parameters are repeated for this distance range gz+1 as well as further distance ranges 224, 223, 222 and 221 until the specified total distance range 25 is covered, which results from the superposition of the captured distance ranges 221 to 22 max In the course of these sequences, the distance range is 22 i updated image, the image areas at the location of the other objects 23 (distance range 222) and 22 (distance range 221) are overwritten, so that in the last updated image all three objects 22 to 24 with their assignment to the respective distance range 22 i certain object distances to device 1.
[0098] The number i (i = 1 to max) of distance ranges 22 i, which are recorded during the operating procedure, can be, for example, 50. The operating procedure with scanning of all distance ranges 22 max to 221 can be performed in real time at 10 Hertz. The result of such a complete scanning process of the distance ranges 22 max to 221 is a 2D image with the non-filtered objects 22, 23 and 24 in all distance ranges 22 recorded within the total distance range 25 i . Overwritten information of the distance ranges 22 initially recorded within the total distance range 25 max to 222 can be retained in the last updated output image after detection of the distance range 221 for an independent spatial (3D) evaluation.
[0099] As the Fig. As illustrated in Figure 4, successive detection steps for distance ranges gz-1 and gz can overlap in time. More than two such detection steps can also overlap in time, so that, for example, during a delay period in the detection of a first distance range gz, a flash of at least one further detection of a further distance range gz+1, gz+2 begins. A detection period of a previous detection can also overlap, for example, with a flash period of a subsequent detection.
[0100] Distance extensions Δz of the different distance ranges 22 i , 22 j may differ from one another. Therefore, the detection times may vary between the various acquisition steps of the operating procedure.
[0101] After capturing a respective distance range 22 iAdditionally, a triangulation determination of the distance of the objects 22 to 24 in the non-filtered image areas can be carried out using the at least one stereo camera 5, 14. In this case, a stereo correspondence determination can again take place, as already explained above.
[0102] Before each acquisition step of the operating procedure, a Region of Interest (ROI) can be created as part of a camera 5 i , 14 i detectable image field can be specified, for example a road area identified by appropriate object filtering in the forward direction 4.
[0103] Fig. Figure 5 shows a side view of another embodiment of the image capturing device 1. Components and functions corresponding to those described above with reference to the Fig. 1 to 4 have already been explained, have the same reference numbers and will not be discussed in detail again.
[0104] Based on the Fig. 5 explains the use of the device 1 for determining and detecting a diffuse scattering medium along a flash light path 28 between the flash light source 7 and an exemplary environmental object 2. Shown in the Fig. 5 shows, for example, the camera 51 of the stereo camera 5, which enables not only the triangulatory distance determination but also a TOF distance determination.
[0105] In the example according to Fig. 5, the device 1 captures a total of three distance ranges 221, 222, 223 by specifying corresponding TOF image acquisition parameters. The first, closest distance range 221 extends by a first distance d1. The next, further distance range 222 extends by a second distance d2, for which d2 = 2 × d1 applies. The next, third distance range 223 extends by a third distance d3, for which d3 = 3 × d1 applies.
[0106] The specification of the image acquisition parameters for the three distance ranges 221 to 223 includes, in addition to the specification of the TOF image acquisition parameters “start of illumination”, “duration of illumination”, “start of detection” and “duration of detection”, a specification of a flash intensity I1 for illuminating the respective distance range 22 i . The flash intensity I2 for illuminating the distance range 222 is four times as large as the flash intensity I1 for illuminating the first, nearest distance range 221. The flash intensity I3 for illuminating the distance range 221 in the example Fig. 5 of the third distance range 223 is nine times as large as the flash intensity I1. The flash intensity I1 is therefore quadratically dependent on a distance d i within a total distance range of 25 to be recorded.
[0107] Assuming a constant reflectivity or a constant scattering power of an ambient surface 29, this quadratic dependence of the illumination intensity I results in i from the distance d i that the sensor array 61 of the camera 51 experiences a constant intensity illumination when capturing images in the three distance ranges 221, 222, and 223. This constant intensity illumination of the sensor array 61 only occurs when there is no significant scattering of the illumination flash 8 by the diffuse scattering medium 27 along the flash light paths 311, 312, 313 between the flash light source 7 and the surroundings in the distance ranges 221 to 223. In other words: inequalities in the intensity illumination of the sensor array 61 along the detection paths 30 i with the given quadratic dependence of the illumination intensity I i from the distance d iwithin the distance range to be detected 22 i are an indication of the presence of the diffuse scattering medium 27.
[0108] The following cases can be distinguished: a) The diffuse scattering medium 27 lies within the currently recorded distance range, in the Fig. 5, i.e., within the distance range 222. In this case, the diffuse scattering medium 27 is directly illuminated along the illumination path 312, and a corresponding scattering of the illumination light 8 by the diffuse scattering medium 27 into the sensor array 61 is measured during the TOF detection, which leads to an increase in the intensity illumination along the detection path 302. b) The diffuse scattering medium 27 lies between the flash light source 7 and the sensor array 61 on the one hand, and the currently detected distance range, for example, the distance range 223, on the other. In this case, the illumination light along the flash light path 313 is attenuated upon passing through the diffuse scattering medium 27. The illumination light backscattered from the detection range 223 toward the sensor array 61 along the detection path 303 is further attenuated due to repeated scattering by the diffuse scattering medium 27. As a result, the intensity illumination of the sensor array 61 when detecting the distance range 223 is attenuated due to the cloud of the diffuse scattering medium 27 present in the intermediate distance range 222.
[0109] A combination of an increased intensity illumination of the sensor array 61 when detecting the distance range 222 compared to the expected constant intensity illumination and a reduction in the intensity illumination of the sensor array 61 when detecting the distance range 223 thus enables the identification of the diffuse scattering medium 27 within the distance range 222.
[0110] In the above-explained embodiments of the image capture device 1 or 21, a pixel sensitivity of sensor pixels of the various sensor arrays 61, ... can be controlled as a function of an incident light intensity, i.e., as a function of an intensity illumination of the sensor array 61, ... In particular, a γ correction can be performed during image processing using the respective sensor array 61, ... The exponent γ used for the γ correction can vary in the range between, for example, 0.2 and 5. Depending on the value of γ, a pixel sensitivity for adapting a high dynamic range can then be achieved either at high or low intensity illumination of the sensor array.
[0111] A selection of object distance ranges to be recorded 22 iIn certain operating modes of the detection device 1 or 21, the detection range 25 can be adapted to a movement mode of the device 1 or 21 relative to the environment. In particular, the total distance range 25 can be adapted to a relative speed of the device 1, 21 to the environment. Adaptation to a reverse or cornering mode is also possible.
[0112] Alternatively or in addition to the above using the Fig. 3 and Fig. 4, the respective device 1 or 21 can also be operated as follows: By correspondingly assigning, in particular, the flash illumination duration and the detection duration, a TOF distance of the at least one object 2 or 22 to 24 is detected using the TOF principle with the aid of the respective device 1, 21. In parallel or sequentially, a triangulatory distance of the respective object 2 or 22 to 24 is detected using the triangulatory principle using the one stereo camera 5, ... of the device 1, 21. The detection values "TOF distance" and "triangulatory distance" are then compared during operation of the device 1, 21.
[0113] In particular, it is possible to use the determined TOF distance as an input variable for the triangulation detection. This can facilitate correspondence analysis during the triangulation detection. In particular, a disparity range to be covered by the correspondence analysis can be limited. The reverse case is also possible, i.e., the triangulation-determined distance is used as an input variable for the distance detection using the TOF principle. This can be used to determine or divide a given total distance range into segmented distance ranges. i as well as for distance-adjusted specification of the TOF image acquisition parameters.
[0114] An illumination intensity by the flash light source 7 can in particular be specified in such a way that a common and in particular scalar brightness control of all images of the Z-buffering of the first Fig. 3 and Fig. 4 is carried out with, in particular, constant intensity illumination of the respective sensor arrays 61, ... This brightness control can be used in the form of a control loop in the device 1, 21.
[0115] A calibration of a dependence of the illumination intensity of the flash light source on the distance, i.e. an illumination intensity function, can be carried out using a calibration setup using a scattering medium with a homogeneous scattering particle distribution.
[0116] The pixel sensitivity as a function of the incident light intensity can be set as a proportional function or as a non-linear function.
[0117] During operation of the device 1 or 21, selected objects 2 or 22 to 24 can be tracked. As soon as one of the objects 2 or 22 to 24 has been detected during the triangulatory distance detection and / or during the TOF distance detection, the TOF image detection parameters in particular, but also the triangulatory image detection parameters (for example, the length of the baseline of a respectively selected stereo camera, a lens focal length of the cameras of the respective stereo camera, or a disparity range to be monitored) can be specified such that the object detected once is also detected again in subsequent image captures.In this case, a relative movement of the respectively detected object to the device 1 or 21 can be taken into account, so that, for example, if it is expected that the object and the device will approach each other, the distance of the respectively selected distance range 22 is taken into account during tracking. i is reduced during TOF detection. The light intensity of the flash source can be adjusted and, in particular, controlled to a distance range of the object to be tracked.
[0118] Depending on the operating method of the device 1, 21, exactly one object can be tracked or several objects can be tracked simultaneously.
[0119] If at least one object is tracked, a tracking detection of the at least one tracked object and a scan over the entire distance range 25 can be carried out intermittently, which increases the overall detection reliability with a reasonable detection effort.
[0120] The image acquisition parameters can be adjusted for TOF acquisition of multiple distance ranges 22 i , 22 i+1 be adjusted so that a detection via the respective camera during exactly one detection period (same detection start, same detection duration) for these distance ranges 22 i , 22 i+1 In extreme cases, this can be the case for all distance ranges 22 i within the total distance range 25.
[0121] When tracking objects, the extent of the respective distance range 22 iThe distance in which the object is expected can be determined by adjusting the flash duration accordingly. For more precise object tracking, shorter flash durations and shorter detection times can be specified, resulting in a corresponding increase in distance accuracy for TOF distance determination.
[0122] The device 1 can contain a neural network or an AI module, which can be used to train the respective operating procedure and to improve image recognition accordingly. QUOTES CONTAINED IN THE DESCRIPTION
[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature
[0000] WO 2022 / 069 424 A1 [0002, 0020, 0021, 0039, 0070, 0071, 0081] WO 2022 / 069 425 A2 [0002, 0020, 0070] WO 2022 / 179 998 A1 [0002, 0029, 0081] DE 10 2011 080 102 B3 [0002, 0020, 0071] US 2019 / 056 498 A1
[0007] US 2022 / 0174255 A1
[0007] US 2015 / 296 200 A1
[0008] DE 10 2018 213 976 A1
[0024] WO 2012 / 095 258 A1
[0024] US 9,451,237
[0024] WO 2013 / 020 872 A1
[0024] Zitierte Nicht-Patentliteratur
[0000] T. Swedish et al.: „Beyond the Line of Sight? What's New in Optical Perception“, AUVSI XPONENTIAL 2022-SWEDISH (Konferenz-Handout), S. 1-8, https: / / www.xponential.org / xponential2022 / Custom / Handout / Speaker5158 4_Session4220_1.pdf
[0007]
Claims
[1] Device (1; 21) for spatial image capture of an environment, - with at least one stereo camera (5, 14, 15, 20; 5, 14) with at least two spaced-apart cameras (51, 52, 141, 142, 151, 152, 153, 154, 201, 202, 203, 204) for triangulatory object distance determination within the detected environment, - with at least one flash light source (7, 13) for ambient lighting with controlled specification of a flash lighting start (t = 0, t = t flash ) and a flash illumination duration (Δt flash ) successive flashes of illumination, - wherein the cameras (51, 52, 141, 142, 151, 152, 153, 154, 201, 202, 203, 204) of the stereo camera (5, 14, 15, 20; 5, 14) each comprise a sensor array (6) with controlled specification of a detection start (t = t E , t = t E1 ) and a detection time (Δt exp ), - with a control device (10) which is in signal connection with the cameras (51, 52, 141, 142, 151, 152, 153, 154, 201, 202, 203, 204) and the flash light source (7, 13), for the time-synchronized specification of the following image acquisition parameters: - the start of flash lighting (t = 0, t = t flash ) and the flash illumination duration (Δt flash ) of the flash light source (7, 13) and - the start of detection (t = t E , t = t E1 ) and the detection time (Δt exp ) of the cameras (51, 52, 141, 142, 151, 152, 153, 154, 201, 202, 203, 204). [2] Device according to claim 1 - with at least one further flash light source (13) with controlled specification of a flash illumination start (t = 0, t = t flash ) and a flash illumination duration (Δt flash ) successive illumination flashes (8, 81) and / or - with at least one further stereo camera (14) with two further cameras (141, 142) spaced apart from one another for triangulatory object distance determination within the enclosed environment. [3] Device according to claim 1 or 2, characterized by at least one fisheye stereo camera (15, 20) with at least two fisheye cameras (151 to 154, 201 to 204), wherein each of the fisheye cameras (151 to 154, 201 to 204) is aligned to capture the surroundings both in a main capture direction (4) and longitudinally thereto in a lateral direction (16), wherein capture areas of the fisheye cameras (151, 201) overlap in an overlap area (19) in the main capture direction (4). [4] Device according to one of claims 1 to 3, characterized by that the control device (10) has an image processor for real-time processing. [5] Method for operating a device (1; 21) according to one of claims 1 to 4, comprising the following steps: - specifying the image acquisition parameters such that a first distance range (gz-1) is captured, - Capturing the distance range (gz-1) with the specified image acquisition parameters using the cameras of at least one stereo camera, - specifying the image acquisition parameters such that a further distance range (gz) is acquired which borders on or partially overlaps the previous distance range (gz-1) along the forward direction (4) in a detection progress direction (+z), - Capturing the further distance range (gz) with the last specified image acquisition parameters with the cameras of at least one stereo camera, - digital filtering of the last captured image to filter out structureless image areas, - Overwriting the image acquired in the acquisition step preceding the last acquisition with the last acquired image in non-filtered image areas of the last acquired image to generate an updated image, - specifying the image acquisition parameters such that a further distance range (gz+1) is acquired which borders on or partially overlaps the previous distance range (gz) along the forward direction (4) in the acquisition progress direction (-z), - Repeat the last steps of acquisition, digital filtering, overwriting and presetting for further distance ranges (224 to 221) until a predefined total distance range (25) is covered, which results from a superposition of the acquired distance ranges (221 to 22 max ) results, - Output the last updated image. [6] Method according to claim 5, characterized bythat at least two of the consecutive recording steps overlap in time. [7] Method according to claim 5 or 6, characterized by that distance extensions (Δz) of the distance ranges (22 i ) differ from each other. [8] Method according to one of claims 5 to 7, characterized by that after capturing the respective distance range (gz-1, gz, gz+1; 22 i ) a triangulation determination of a distance of objects (22 to 24) in non-filtered image areas is carried out by means of the at least one stereo camera (5, 14). [9] Method according to one of claims 5 to 8, characterized by that before the capture, a region of interest (ROI) is specified as part of an image field that can be captured by the stereo cameras (5, 14). [10] Method for operating a device (1; 21) according to one of claims 1 to 4, comprising the following steps: - Determining a Time of Flight (TOF) distance of at least one object (2; 22 to 24) by correspondingly assigning the flash illumination duration and the detection duration using the TOF principle, - detecting a triangulatory distance of the object (2; 22 to 24) by means of the at least one stereo camera using the triangulatory principle, - Matching the TOF distance and the triangulation distance. [11] Method according to claim 10, characterized by that the TOF distance is used as an input for the triangulation detection. [12] Method according to one of claims 5 to 11, characterized by that an illumination intensity of the flash light source (7, 13) is limited to a distance range (22 i ) is voted on. [13] Method according to claim 12, characterized by that the flash intensity depends quadratically on a distance (d i) within the distance range to be detected (22 i ) depends. [14] Method according to one of claims 5 to 13, characterized by that a pixel sensitivity of a sensor pixel of the sensor array (6) is controlled as a function of an incident light intensity of the flash light source (7, 13). [15] Method according to one of claims 5 to 14, characterized by that a selection of object distance ranges to be recorded (22 i ) is adapted to a movement mode of the device (1; 21) relative to the environment.
Citation Information
Patent Citations
Device for rotating flat objects
DE102011080102A1
Device for measuring the distance of an object to a sensor
DE102018213976A1
Detection of a scrolling gesture
US20140282224A1
Stereo gated imaging system and method
US20150296200A1
Gated imaging apparatus, system and method
US20190056498A1
Cited By
Device for capturing a spatial image of a surrounding area which is moving relative to the device, and method for operating same
WO2024251542A1
