Method for calibrating a camera system

Abstract

In order to calibrate a camera system having at least three cameras (51, 52, 53) with known focal lengths (f), which are arranged next to one another in the region of a common camera arrangement plane (xy) along a camera arrangement coordinate (x) and the object-side opening angle of which is in each case at most 45°, at least three calibration markings are first selected. The calibration markings are used to specify a respective calibration marking point PA, PB, PC, wherein the calibration marking points (PA, PB, PC) are arranged at different distances (dA, dB, dC) from the camera arrangement plane (xy). The calibration marker points (PA, PB, PC) are mapped into a camera image plane of the respective camera (51, 52, 53) using the cameras (51, 52, 53) to capture calibration marker image points (PA', PB', PC'). For each pairing of two of the cameras (51, 52, 53), a disparity of the image of the respective calibration marker image point (PA', PB', PC') is measured in the respective camera image plane. From the at least nine measured disparities and the camera focal lengths (f), the distances (xi - xk) between the cameras (5i, 5k) of a camera pair along the camera arrangement coordinate (x) as well as position errors of each camera pair (5i, 5k) are determined, which result from arrangement deviations of the cameras (51, 52, 53) relative to the camera arrangement plane (xy). This results in a calibration method that ensures precise and reliable operation of the camera system.

Description

[0001] Method for calibrating a camera system

[0002] The present patent application claims priority from German patent application DE 10 2024 206 379.7, the contents of which are incorporated herein by reference.

[0003] The invention relates to a method for calibrating a camera system with at least three cameras with known focal lengths. The invention further relates to a camera system calibrated according to such a method, and to methods for operating such a camera system.

[0004] Camera systems are known from WO 2022 / 069 424 Al, WO 2022 / 069 425 A2, WO 2022 / 179 998 Al and DE 10 2011 080 702 B3. US 2019 / 0158813 Al discloses a method for recalibrating stereo cameras in real time.

[0005] It is an object of the present invention to ensure precise and safe operation of a camera system with multiple cameras, which in particular can be components of stereo cameras for triangulatory object distance determination within an image-capturing environment.

[0006] This problem is solved according to the invention by a calibration method with the features specified in claim 1.

[0007] According to the invention, it was discovered that with the aid of such a calibration method, even cameras with a comparatively small object-side opening angle can be calibrated reliably and precisely, so that these cameras can then be used in the camera system, for example as components of stereo cameras, after calibration has been carried out. The calibration method utilizes the fact that the cameras to be calibrated, arranged side by side along the camera arrangement coordinate of the camera system, see the selected calibration marker points with defined different disparities, so that even small positional or disparity errors can be determined.

[0008] The at least three calibration markers selected during the calibration process can be natural marker points, i.e., natural features of a scene. Alternatively or additionally, markers can also be provided as part of the calibration process.

[0009] The calibration marker points are arranged at different depths.

[0010] The object-side opening angle of the cameras in the camera system is a maximum of 45°. Depending on the camera system design, the opening angle of the cameras can also be smaller, for example, a maximum of 40°, 35°, or even 30°. The maximum object-side opening angle of the cameras in the camera system can also be smaller. This opening angle is typically greater than 5°.

[0011] The calibration markers shown in the calibration procedure are all located within the opening angles of all cameras in the camera system to be calibrated. In principle, the calibration procedure can also be used with a camera system containing at least one camera with an opening angle greater than 45°. In particular, the calibration procedure can also be used with a camera system using fisheye cameras or fisheye lenses.

[0012] The camera system to be calibrated can contain more than three cameras, for example, four, five, or even more. The cameras in the system to be calibrated do not need to be arranged exactly in the same plane. The distance of each camera from the nearest common camera arrangement plane is typically much smaller than the typical distance between two adjacent cameras in the system to be calibrated.

[0013] Distance-relative differences according to claim 2 improve a calibration result. The distances between the calibration marker points can differ from each other by more than 20%, for example by at least 25%, at least 50%, or even at least 100%.

[0014] Specifying more than three calibration measurement points according to claim 3 improves calibration precision. Alternatively or additionally, it is possible to specify more than one calibration marker point for each selected or provided calibration marker.

[0015] The advantages of a camera system according to claim 4 correspond to those already explained above with reference to the calibration method. Such a camera system, designed for spatial image acquisition, can be well adapted to practical applications, particularly image acquisition for ensuring assistive or autonomous driving. The respective telephoto lens camera can have a focal length in the range between 6 mm and 100 mm, for example, in the range between 8 mm and 50 mm.

[0016] With a pixel pitch of 5 pm per pixel for the respective camera and a focal length of 10 mm, the relative focal length (focal length / pixel pitch) is 2000 pixel pitches (in short: 2000 pixels). The pixel pitch can also be smaller and, for example, 3.45 pm.

[0017] With a wide-angle lens with a 2.1mm focal length and 3.45pm per pixel, the relative focal length is 609 pixels (2.1 / 3.45*1000). Depending on the camera system design, all cameras in the system can also be configured as corresponding telephoto lenses.

[0018] When the camera pairs are designed as stereo cameras according to claim 5, the advantages of calibration are particularly evident.

[0019] In the camera system according to claim 6, the functions of at least one stereo camera and, on the other hand, an image acquisition system using a time-of-flight method are advantageously complementary. Both the stereo camera and the components enabling a time-of-flight method—the flash light source and the sensor array, via which time-of-flight method image acquisition parameters can be specified—allow the determination of the distance to objects within the environment.

[0020] The camera system can be used for spatial image capture of an environment moving relative to the camera system. The at least one stereo camera can be used for triangulatory object distance determination within the captured environment in the forward direction of any relative movement of the camera system to the environment. This can be used to improve the safety of environmental perception, particularly in connection with driver-assisted or autonomous driving. The camera system can be integrated into a vehicle, such as a car or truck.

[0021] The control unit can, in particular, specify a delay period between the start of flash illumination from the at least one flash source and the start of detection by the respective sensor array of the cameras of the at least one stereo camera. The distance between the cameras of the stereo camera results in a baseline length, i.e., the distance between the centers of the entrance pupils of the cameras of the at least one stereo camera. The camera system can combine the principles of a time-of-flight (TOF) camera with a triangulatory distance-measuring stereo camera. A corresponding TOF image acquisition system is described, for example, in US 2019 / 056 498 AL. The flash source can be designed in the manner generally known from TOF cameras or LiDAR systems. The same applies to the sensor arrays of the cameras. A sensor array used in TOF image acquisition can be configured as a single-photon avalanche diode (SPAD) array.Such a SPAD array and its application in connection with a TOF measurement is described in a technical article by T. Swedish et al.: “Beyond the Line of Sight? What's New in Optical Perception”, AUVSI XPONENTIAL 2022-SWEDISH (conference handout), pp. 1-8, https: / / www.xponential.org / xponential2022 / Custom / Handout / Speaker51584_Session4220_l.pdf and in US 2022 / 0174255 AL. For triangulatory object distance determination, the two cameras of the stereo camera are synchronized with each other via a synchronization unit. This synchronization enables, in particular, precisely simultaneous acquisition of the respective object by the cameras of the stereo camera. This distinguishes a stereo camera for triangulatory object distance determination from a stereoscopic camera, which exclusively creates a spatial impression of an image capture and is described, for example, in US 2015 / 296 200 Al.The use of a SPAD array can lead to a reduction in motion blur and can be used to improve the contrast of image capture.

[0022] Synchronization, and in particular simultaneous object detection in triangulatory object distance determination, makes it possible to precisely determine the distance to objects that are moving independently, since the influence of an additional relative velocity due to the object's movement does not affect the distance determination.

[0023] A focal length f of the respective camera of a stereo camera, the baseline b of the stereo camera, and a disparity D, i.e., a difference between image coordinates of imaging positions of the same point on the sensor arrays of the cameras of the stereo camera, are coordinated such that, taking into account the relationship d = fb / D for the object distance value d to be captured, the following applies:

[0024] 3 m < d < 300 m Additionally, the camera system can be designed so that a corresponding object distance range is captured via the TOF image capture functionality of the camera system.

[0025] The flash source can be a controlled flash lamp or a time-controlled LED or laser light source. Flash sources known to be used, for example, in LID AR devices, can be employed.

[0026] The flash illumination duration of the flash light source can be in the range between 10 ns and 200 ns, for example in the range of 100 ns.

[0027] The detection duration of the sensor arrays of at least one stereo camera is typically set to be twice the flash duration of the flash source. The detection duration can range from 20 ns to 400 ns, for example, around 200 ns.

[0028] Both the flash illumination duration and the detection duration can be set and preset via the control unit.

[0029] The synchronization unit can be part of the control unit. A storage unit, particularly in the form of flash memory, can be part of the control unit. The synchronization unit can be part of a synchronization device for synchronizing the flash source and the sensor arrays of the cameras of the at least one stereo camera. The synchronization device, in turn, can be part of the control unit of the camera system. Synchronization provided by the synchronization device can be better than 5 ns, better than 2 ns, better than 1 ns, and even better.

[0030] The cameras of the stereo camera can be spaced apart laterally to the forward direction. Depending on the camera's detection angle, the intended use, and the main detection direction, other camera arrangements are also possible.

[0031] The cameras of at least one stereo camera can have a telephoto lens for capturing objects at distances between 30 m and 300 m. Depending on the application, the cameras of the stereo camera can also be equipped with lenses of other focal lengths. These camera lenses are designed to cover an entire predefined distance range, particularly between 3 m and 300 m. In principle, even shorter distances can be covered.

[0032] The distance ranges of triangulatory object detection on the one hand and TOF distance detection on the other are coordinated and can, for example, be the same. Partial overlap of these distance ranges is also possible, so that both distance determination principles, "triangulatory" and "TOF," are available in one distance range, and one of these principles is available in other distance ranges.

[0033] Using the cameras of at least one stereo camera, a correspondence analysis can be performed to identify objects within a field of interest that differ from a background. Methods for such a correspondence analysis are known from WO 2022 / 069 424 A1 and WO 2022 / 069 425 A2. To prepare the camera system for operation, the cameras of at least one stereo camera can be stereo calibrated, a procedure that is generally known from WO 2022 / 069 424 A1 and DE 10 2011 080 702 B3.

[0034] The at least one stereo camera can have more than two cameras spaced apart from each other, which in particular allows switching between different baselines, as is also described, for example, in WO 2022 / 069 424 Al.

[0035] The camera system can include at least two stereo cameras with a relatively large baseline and at least one additional stereo camera with a significantly smaller baseline, where the smaller baseline can be at most 50%, 25%, or 10% of the larger baseline. The camera system can also include multiple stereo cameras with the smaller baseline.

[0036] Further object information, particularly object distance information, can be obtained by analyzing the shadow cast by an object within the captured environment. This shadow is created by the flash illumination and is depicted differently by the two cameras of the at least one stereo camera, which are spaced apart from each other. In particular, the object's height can be determined because the image of an object shadow depends on the object's height and lateral position. The camera system enables spatial image capture of the environment, whether moving or stationary relative to the camera system. This allows for driver assistance functions or even autonomous driving capabilities, for example, when the camera system is used as part of a vehicle.

[0037] The flash source of the camera system, or an additional light source for ambient illumination, can have a measuring beam that is textured or structured across its cross-section. Such texturing / structuring of the measuring beam can be achieved by constructing it as a multitude of individual beams that capture the surroundings in a grid pattern. Such a light source for generating a textured / structured measuring beam can be a laser.Examples of a detection device that operates using such a textured / structured measuring light beam can be found in DE 10 2018 213 976 A1 and the references cited therein, in particular in WO 2012 / 095 258 A1, US 9,451,237 B2, and WO 2013 / 020 872 A1. Using such textured / structured illumination, optically unstructured, especially diffuse, scattering media within the encompassed environment can be measured with respect to their distance. Such diffuse scattering media can be, for example, fog, smoke, or dust.

[0038] The camera system can achieve form redundancy and / or functional redundancy, particularly when used for spatial image acquisition. Form redundancy occurs when different technologies are used to determine the same measurement. Functional redundancy exists when multiple devices using the same technology are employed to measure the same quantity. These terms "form redundancy" and "functional redundancy" are primarily used in the aerospace industry. There, a combination of form and functional redundancy is required to prevent the multiple consequences of a failure. Depending on the application, such redundancies may be legally mandated.

[0039] At least one additional flash source and / or at least one additional stereo camera according to claim 7 enable additional functional redundancy of the camera system. For example, if at least one additional flash source is provided, the system can switch to the additional flash source after the failure of the initially used flash source, thus avoiding a prolonged downtime of the camera system. If at least one additional stereo camera is used, for example, during parallel triangulation measurements using two stereo cameras, further redundancy in object distance determination can be achieved.

[0040] The cameras of the second stereo camera can each have a sensor array with a controlled specification of a detection start and duration, so that the second stereo camera can essentially replace the originally used stereo camera in its function. Redundant time-of-flight (TOF) determination of object distances is also possible using two corresponding stereo cameras with controlled specification of the detection start and duration, which provides additional reliability in object distance determination and helps, for example, to detect synchronization problems. When using multiple stereo cameras, it is also possible to assign cameras from different stereo cameras to a new "virtual" stereo camera with a correspondingly modified baseline, which enables further redundancy and reliability in object distance determination.For example, the camera pair that proves to be particularly advantageous for triangulation determination with regard to a specific environmental object can be selected as the baseline specification.

[0041] The camera system can, in principle, also include at least one fisheye stereo camera. Such a fisheye stereo camera can include at least two fisheye cameras, each of which is oriented to capture the environment both in a primary detection direction and laterally along this direction, with the detection areas of the fisheye cameras overlapping in the primary detection direction. At least one such fisheye stereo camera enables the use of additional triangulation baselines, which can be used to capture further environmental objects and / or to check object distance results from the other stereo cameras and / or to calibrate the other stereo cameras, particularly those implemented with telephoto lenses. The primary detection direction can be the forward direction of a relative movement of the camera system to the environment.The use of fisheye cameras is known from the references mentioned above as well as from WO 2022 / 179 998 A1. A fisheye camera, mounted as rigidly as possible to a telephoto camera, allows for more precise calibration of the stereo cameras relative to each other due to the wider field of view of the fisheye camera. An inertial measurement unit (IMU) in the cameras of the respective stereo camera allows for further optimization of the relative calibration. In particular, the IMU can be used to determine the rotation rate, i.e., the relative tilt of each camera around a defined axis.

[0042] The camera system can include multiple fisheye stereo cameras, providing additional redundancy. The cameras of the at least one fisheye stereo camera can each have sensor arrays with controlled predefined detection start and duration, enabling time-of-flight (TOF) determination of object distances. The at least one fisheye stereo camera can also be configured without TOF functionality. If the device includes multiple fisheye stereo cameras, it is also possible for at least one of the fisheye stereo cameras to have TOF functionality and at least one other fisheye stereo camera to lack it.

[0043] In principle, the device according to the above claims can be used in such a way that exactly a distance range is detected by means of the TOF components, for example in the distance range between 30 m and 300 m, in particular between 100 m and 200 m, to the device.

[0044] The advantages of an operating method according to claim 9 correspond to those already explained above with reference to the camera system. Such an operating method enables real-time operation of the device, thus making autonomous driving possible, for example, when the device is used as part of a vehicle. In this operating method, a total covered distance range can be subdivided into a plurality of adjacent or partially overlapping distance ranges. For example, three to 100 such distance ranges can be defined via corresponding image acquisition parameters, for example, 50 distance ranges that lie within the total distance range. During operation of the camera system, the total distance range can be captured several times per second via the defined individual distance ranges, for example, at a rate of 10 Hertz.During operation, the process can begin with the furthest object within the overall detection range. This ensures that when images captured in a previous acquisition step are overwritten by the most recently captured image, nearby objects will overwrite more distant objects in areas of the image that are not filtered out. Therefore, the result of the operation—namely, the most recently updated and output image—always represents the objects closest to the camera system. This allows for adjustments, such as influencing the camera system's movement. Intermediate storage of images captured in previous acquisition steps is then unnecessary, thus increasing the speed of the process.

[0045] The most recently updated, output image is a 2D image with unfiltered objects in all distance areas captured within the overall range. Information from previously captured distance areas within the overall range, even if overwritten during operation, can be retained for independent spatial (3D) analysis. For each unfiltered object associated with a particular capture step, the process yields a distance area in which the object is located relative to the camera system. This resulting object distance information, determined using time-of-flight (TOF) image acquisition, combined with triangulatory distance acquisition enabled by at least one stereo camera, provides form redundancy for the operating procedure.

[0046] Distance information can be output as either the smallest distance of the detected object or a distance per object pixel of the respective sensor array.

[0047] Overall, the result is a safe and forward-looking image acquisition system that is well adapted to the requirements of driver assistance or autonomous driving in particular.

[0048] When digitally filtering to extract featureless image areas, a stereo correspondence determination can be used, which is described, for example, in WO 2022 / 069 424 AE.

[0049] At least two of the successive detection steps can overlap in time. Such a temporal overlap enables a rapid execution of the operating procedure. The temporal overlap can be such that at least two distance ranges with simultaneous detection start and the same detection duration are detected simultaneously by the sensor arrays, whereby these detection ranges are defined by different specifications of the respective flash illumination start and / or flash illumination duration with regard to their image acquisition parameters. It is also possible to design the repetition sequence of the operating procedure so that a detection actually takes place at precisely one time period, i.e., at a predetermined detection start and with a predetermined detection duration, for all predetermined distance ranges.This simultaneous detection of at least two detection areas reduces image processing effort and accelerates the entire process.

[0050] The distance extents of the detection zones can differ from one another. Such differing distance extents of the detection zones make it possible, in particular, to define a distance sensitivity depending on the distance, which also helps to shorten the operating time of the process. For example, with increasing distance, the distance extent of the respective detection zone can be increased, especially progressively.

[0051] After capturing the respective distance range, a triangulation can be performed to determine the distance to objects in unfiltered image areas using at least one stereo camera. This additional triangulation allows for a comparison of object distance values ​​determined on the one hand by distance-range-resolved acquisition using the TOF image acquisition parameters and on the other hand by triangulation. Before acquisition, a Region of Interest (ROI) can be predefined as part of the image field captured by the stereo cameras. Predefining the ROI reduces the amount of image data to be processed, which can also shorten the procedure. The predefined ROI could be a section of road in the forward direction.

[0052] An operating method according to claim 10 can be used as an alternative or in addition to the operating method described above. Such an operating method enables redundant distance determination of a given object's position. In particular, this can provide form redundancy, i.e., redundancy achieved by using different technologies for distance detection.

[0053] The time-of-flight (TOF) distance can be used as an input for triangulation. Using the measured TOF distance as input for triangulation can facilitate correspondence analysis during triangulation. In particular, it can help define the disparity range to be covered during correspondence analysis. Conversely, the triangulated distance can also be used as an input for TOF acquisition. The use of measured object distance can be employed in the operation of the camera system to track detected objects. It is possible to track exactly one such object or to track multiple objects simultaneously. Object tracking steps can alternate with acquisition steps covering an entire predefined distance range.For object tracking, image acquisition parameters of TOF distance detection and / or triangulatory distance detection can be adjusted to the object distance expected during tracking.

[0054] The flash intensity can be adjusted to the distance being captured. Such a flash intensity / distance range adjustment makes it possible, in particular, to infer the presence of diffuse scattering media in the flash and / or camera's field of view. This enables reliable operation of the device even in foggy, dusty, or smoky environments.

[0055] The flash intensity can depend quadratically on the distance within the detection range. Such a quadratic dependence of the flash intensity on the detection range allows, particularly when the illumination intensity increases quadratically with increasing distance, for adjustment such that, with identical ambient backscatter and negligible scattering along a flash path and along a detection path between source and object, a constant intensity illumination of the at least one sensor array can be achieved. This can be used to determine diffuse scattering media within the detected detection range.

[0056] The pixel sensitivity of a sensor pixel in the sensor array can be controlled as a function of the incident light intensity from the flash source. Such control of pixel sensitivity can, in particular, be non-linear with respect to the incident light intensity. Specifically, gamma correction (γ) can be applied. The sensitivity of the at least one sensor array can then be specifically adapted to the lighting and / or detection conditions present in the application.

[0057] A selection of object distance ranges to be captured can be adapted to a movement mode of the camera system relative to the environment. Such an adapted distance range-movement mode selection enables efficient utilization of the camera system's computing power and can also increase operational reliability.

[0058] When evaluating images using triangulatory stereoscopy with binocular vision, two images are compared, so that any changes in brightness due to altered density distributions can be neglected to a first approximation. When comparing the images with trained data (neural networks, AI) with monocular vision, a homogeneous brightness distribution over distance increases detection performance, while changing density distributions make reliable object recognition more difficult.

[0059] Exemplary embodiments of the invention are explained in more detail below with reference to the drawing. This drawing shows:

[0060] Fig. 1 schematically shows a top view of a device for spatial image acquisition of an environment moving relative to the device, with an environmental object in the forward direction of a relative movement of the device to the environment being illustrated by way of example and perspective; Fig. 2 shows a time diagram to illustrate image acquisition parameters during operation of the device according to Fig. 1;

[0061] Fig. 3, in a representation similar to Fig. 1, shows a further embodiment of a device for spatial image acquisition of an environment moving relative to the device, wherein environment objects are illustrated by way of example and perspective in the forward direction of a relative movement of the device to the environment;

[0062] Fig. 4 shows a temporal sequence of an operating procedure of the device according to Fig. 3 in a representation similar to Fig. 2;

[0063] Fig. 5 shows a side view of the main components of an embodiment of the device for spatial image acquisition of an environment within three exemplary distance parameters, wherein a cloud-like diffuse scattering medium is present within the acquired environment; and

[0064] Fig. 6 schematically shows parameter relationships in a calibration situation of a calibration procedure for a camera system with three cameras.

[0065] Fig. 1 shows a top view of a device 1 for spatial image acquisition of an environment moving relative to the device 1, illustrated by an environment object 2 in the form of a cylindrical body. The device 1 represents a camera system. The device 1 can be part of a vehicle 3, for example, a car or a truck. A forward direction of the relative movement of the device 1 to the environment is illustrated in Fig. 1 by an arrow 4.

[0066] The device 1 has several stereo cameras for triangulatory object distance measurement of objects within the detected environment in the forward direction 4, i.e. in particular for determining the distance of the object 2.

[0067] A first stereo camera 5 of these stereo cameras has two cameras 51, 52 spaced apart from each other. A distance A between these cameras 51, 52 of the stereo camera 5 results in a baseline length of the stereo camera 5, i.e., a distance between the centers of the entrance pupils of cameras 51, 52 of the stereo camera. The cameras 5i, 52 of the stereo camera 5 each have a sensor array 61, 62 with controlled specification of a detection start time and a detection duration for the respective camera 5i and 52. The sensor arrays 61, 62 are designed like corresponding sensors of Time-of-Flight (TOF) cameras and can achieve a detection duration in the range between 1 ns and 500 ns. A typical detection time is in the range between 50 ns and 400 ns, for example between 150 ns and 250 ns.

[0068] The sensor arrays 61 and 62 can each have 100 x 100 pixels or 200 x 200 pixels. In principle, a higher pixel count is also possible, in particular 640 x 480 or 1440 x 1080 pixels.

[0069] The sensor arrays 61 and 62 can be implemented as CMOS arrays. The cameras 5i and 52 have telephoto lenses for detecting objects at distances between 50 m and 300 m.

[0070] Device 1 also includes a flash light source 7, which can also be designed like the light source of a TOF camera and allows controlled specification of the start time and duration of successive flashes. The flash duration can be in the range between 5 ns and 150 ns, for example, in the range of 100 ns.

[0071] Fig. 2 illustrates the time sequence during the operation of the device 1 with the stereo camera 5 and the flash source 7. At time t = 0, an illumination flash from the flash source 7 begins. t = 0 thus represents the start of an illumination flash 8, schematically depicted as a signal box in Fig. 2. The duration of the illumination flash 8 is indicated in Fig. 2 by the letter Atfiash.

[0072] Also at time t = 0, a delay time Atiag starts, which is also called the delay time. This delay time Atiag is in the range between 300 ns and 2 ms and can, for example, be in the range between 500 ns and 1 ms, for example in the range between 700 ns and 900 ns, for example at 800 ns.

[0073] After the delay period Atiag has elapsed, a detection 9 begins at time t = tE by the respective sensor array 6i or 62 of the stereo camera 5. tE thus represents the start of the detection. The respective sensor array 61, 62 then detects incident light from the flash source 7 for a detection duration At. exp . Detection 9 by the respective sensor array 6i, 62 of the stereo camera

[0074] Figure 5 is again illustrated by a time box in Fig. 2.

[0075] The image acquisition parameters explained above, namely the flash illumination start (t = 0), the flash illumination duration Atfiash, the detection start (t = 1E) and the detection duration At exp The timing is synchronized by a control unit 10 of the device 1. For this purpose, a synchronization unit of a synchronization device of the control unit 10 can be used.

[0076] The control unit 10 has an image processor for real-time data processing. This can be a processor with a field-programmable gate array (FPGA). The image processor can be configured for ultra-fast image grabbing of the individual images captured by the sensor arrays 6i, triggered by the control unit 10.

[0077] Due to the synchronized specification of the flash illumination start and the detection start, the control unit 10 inevitably also specifies the delay period Ati ag between the start of the illumination flash (t = 0) and the start of the detection (t = 1E) of the device 1.

[0078] Correlating with the speed of light, a distance range 11 is detected by the stereo camera 5 in the forward direction 4 of the device 1. The object 2 is at a distance from the device 1 along the forward direction 4 that lies within this distance range 11, so that the object 2 is detected by the stereo camera 5. The detected distance range 11 is approximately 100 m to 150 m from the device 1 and has an extent of, for example, 30 m along the forward direction 4.

[0079] This distance is shown at 12 in Fig. 1 and is not to scale compared to the distance range 11.

[0080] Based on the acquisition by the two cameras 5i and 52 of the stereo camera 5, a spatial image of object 2 is obtained. Triangulation can then be used redundantly to determine the reference distance of object 2 to device 1 over the delay period Atiag, in addition to the TOF (time-of-flight) position measurement. Device 1 is therefore essentially a TOF camera with added triangulation capabilities.

[0081] The stereo camera 5 can also perform a correspondence analysis of individual camera images captured by the two cameras 5i and 52 in order to separate stereoscopically captured objects from background noise that is present only in one of the two cameras 51 and 52. Such a correspondence analysis is described, for example, in WO 2022 / 069424 A1 and WO 2022 / 069425 A2.

[0082] Initially, a stereo calibration of the stereo camera 5, in particular of the orientations and positions of the cameras 5i, 52 of the stereo camera 5, can be carried out using a method that is also described in WO 2022 / 069424 AL. With regard to the calibration, reference is also made to DE 10 2011 080 702 B3.

[0083] In spatial image acquisition using device 1, a time-of-flight (TOF) distance of at least one captured object, acquired using the TOF principle, can be compared with a triangulatory distance of the object, acquired using the triangulatory principle with the at least one stereo camera. The TOF distance can, for example, be used as an input parameter for triangulatory acquisition. This can facilitate correspondence analysis during triangulatory acquisition, for example, by limiting the disparity range to be covered in the correspondence analysis.

[0084] Using the images of object 2, illuminated by flash source 7 and captured by cameras 5i and 52 of the stereo camera 5, it is also possible to detect the shadows cast by object 2 for the illumination light of the flash source 7. By evaluating the different shadows cast by object 2, as recorded by the two cameras 5i and 52 of the stereo camera 5, an additional redundant determination of the distance between object 2 and device 1 is possible.

[0085] The device 1 has a further flash source 13, whose function corresponds to that of the flash source 7, i.e., which also serves to control the start and duration of successive flashes. The further flash source 13 can be used redundantly to the flash source 7 described above, for example, in the event of a failure of the flash source 7 described above. While the further flash source 13 is operating, the flash source 7 described above can then be replaced, for example, so that uninterrupted operation of the device 1 is possible, for example, during a journey of the car or truck on which the device 1 is mounted that is being monitored. The device 1 has a further stereo camera 14 with cameras 14i, 142, whose construction and function correspond to what has already been explained above with reference to the stereo camera 5.Stereo camera 14i is positioned directly adjacent to stereo camera 5i and covers practically the same field of view in the forward direction 4 as camera 5i. Camera 142, in turn, is positioned directly adjacent to camera 52 and covers the same field of view in the forward direction 4 as camera 52.

[0086] Stereo camera 14 is redundant to stereo camera 5. The same applies here as was stated for flash sources 7 and 13.

[0087] The device 1 further comprises a fisheye stereo camera 15 with two pairs 15i, 152 on the one hand and 15s, 154 on the other. Each of these fisheye cameras 15i (i = 1 to 4) serves to capture the environment around the device 1 both in the forward direction 4 and in a lateral direction (see arrow 16 in Fig. 1). The two fisheye cameras 151 and 152 arranged on the left side of the vehicle 3 in Fig. 1 have a main detection direction 17, which, like the main detection directions of cameras 5i and 6i, is horizontal, but in the case of fisheye cameras 151 and 152 forms an angle of 45° to the forward direction 4. These fisheye cameras 151, 152 therefore cover both the forward direction 4 and the lateral direction opposite to the lateral direction 16 (left in Fig. 1) of the vehicle 3 on which the device 1 is mounted.Accordingly, the two other fisheye cameras 153, 154, whose main detection directions 18 also form an angle of 45° to the forward direction 4, cover not only the forward direction 4 but also a right-side area in the lateral direction 16. The fisheye cameras 15i have a detection angle of 180° around the main detection directions 17, 18. The limit angles of the detection areas of these fisheye cameras 15i are indicated by dashed lines in Fig. 1. These aperture areas overlap in an overlap area 19, which is indicated by hatching in Fig. 1. The object 2 lies particularly within this overlap area 19.

[0088] The illumination light source 7 emits flashes of light along the forward direction in an illumination cone, the marginal rays of which are illustrated at 8a in Fig. 1. The object 2 and also the overlap area 19 lie within the illumination cone 8a of the flash light source 7.

[0089] The fisheye stereo camera 15 can optionally perform triangulatory object distance determination using at least two of the four fisheye cameras 15i, employing either short baselines (distances between fisheye cameras 15i and 152 on the one hand, and 15s and 154 on the other) or long baselines (distances between fisheye cameras 15i and 153, 151 and 154, 152 and 15s, or 152 and 154). This allows for redundant triangulatory object distance determination.

[0090] The use of such additional triangulation baselines is described, for example, in WO 2022 / 069424 Al and in WO 2022 / 179998 AE.

[0091] The device 1 also has another fisheye stereo camera 20 with

[0092] Fisheye cameras 201 to 204, which correspond in their construction, orientation, and paired arrangement to fisheye cameras 15i to 154. The fisheye stereo camera 20 can be used redundantly with the fisheye stereo camera 15. The same applies here as described above for flash sources 7 and 13 and stereo cameras 5 and 14.

[0093] The cameras 15i of the fisheye stereo camera 15 in turn have sensor arrays of the type of the sensor arrays 6i described above, and can therefore again be used as part of a TOF recording, as already explained above.

[0094] The control unit 10 is in signal communication with the stereo cameras 5, 14, 15 and 20 and with the flash sources 7 and 13. Furthermore, the control unit 10 can be in signal communication with components of the vehicle 3, for example with a drive system and / or with a braking system and / or with vehicle display devices, for example in a dashboard of the vehicle 3.

[0095] In real-time operation, the device 1 can take an image at, for example, ten images per second.

[0096] Fig. 3 shows another embodiment of a device 21, which can be used instead of device 1 for spatial image acquisition of an environment moving relative to the device. Components and functions of device 21 that correspond to those of device 1 bear the same reference numerals and are not discussed again in detail. Device 21 is equipped with two stereo cameras 5, 14, which capture the environment of device 21, which is mounted on a vehicle 3, in the forward direction 4. Three environmental objects 22, 23 and 24 are illustrated within this captured environment.

[0097] With the device 21, in particular a z-buffered operation in the forward direction 4, which is also referred to as the z-direction, is possible, in which a total distance range 25 along the forward direction 4, within which the objects 22 to 24 lie, is divided into a plurality of distance ranges 22i to 22ma x is subdivided into distance zones 22i (i = 1 to max). The maximum number of these distance zones 22i can be between 3 and 50, for example, in the range of 10. These distance zones 22i are also referred to as gate zones gz.

[0098] During the corresponding Z-buffering operation of the device 21, the image acquisition parameters illumination flash start (t = 0), illumination flash duration (Atflash, gz-i), a detection start (t = 1E) and a detection duration (Atexp, gz-i) are initially specified.

[0099] This image acquisition parameter specification is again made using control unit 10.

[0100] The distance range gz-1, for example the distance range 22max, is then captured using the image acquisition parameters specified in this way.

[0101] Further image capture parameters are specified via the control unit 10, namely a further flash start (t = tfiash), a further flash duration (Afiash, g z), another detection start (t = tEi) and a detection duration (At) exp , gz This setting is chosen such that the delay period (Atiag, gz) for these additional specified image acquisition parameters is slightly shorter than the initially specified delay period Atiag, gz-i. This means that when acquiring the last specified image acquisition parameters, a further acquisition area gz is captured, which extends beyond the initially captured distance area gz-1, i.e., distance area 22. ma x, is adjacent to device 21 at shorter distances.

[0102] The further distance range gz borders on or partially overlaps the preceding distance range gz-1 along the forward direction 4 in a detection progress direction +z.

[0103] Object 24 is located within this now recorded recording area gz.

[0104] The last captured image in the distance range gz is now digitally filtered to remove unstructured image areas, leaving object 24. Subsequently, the image captured in the preceding capture range (capture of distance range gz-1) is overwritten with the last captured image (distance range gz) in the unfiltered image areas, i.e., in the area of ​​object 24, so that an updated image containing the information from both distance ranges gz-1 and gz is stored in the control unit 10.

[0105] During operation, the control unit 10 specifies further image acquisition parameters, so that another distance range gz+1 is acquired, which again adjoins or partially overlaps the distance range gz in the opposite direction to the forward direction 4. Now, the previously described steps of acquiring, filtering, overwriting, and specifying image acquisition parameters are repeated for this distance range gz+1 as well as for further distance ranges 224, 223, 222, and 221 until the specified total distance range 25 is covered, which results from the superposition of the acquired distance ranges 22i to 22max.During these sequences, in the image updated for distance range 22i, the image areas at the location of the other objects 23 (distance range 222) and 22 (distance range 22i) are overwritten, so that in the last updated image all three objects 22 to 24 are output with their object distances to the device 1 determined by the assignment to the respective distance range 22i.

[0106] The number i (i = 1 to max) of distance ranges 22i that are detected during the operating procedure can, for example, be 50. The operating procedure with scanning of all distance ranges 22 ma x to 22i can take place in real time in 10 Hz operation. The result of such a complete scanning process of the distance ranges 22 max to 22i is a 2D image with the unfiltered objects 22, 23, and 24 in all distance ranges 22i captured within the total distance range 25. Also included is overwritten information from the distance ranges 22 initially captured within the total distance range 25. max Up to 222 can be retained for independent spatial (3D) evaluation in the last updated, output image after the distance range 22i has been captured.

[0107] As illustrated in Fig. 4, successive detection steps for distance ranges gz-1 and gz can overlap in time. More than two such detection steps can also overlap, so that, for example, during a delay period when detecting a first distance range gz, the flash of at least one further detection of a further distance range gz+1, gz+2 may occur. Similarly, the detection period of a previous detection can overlap with the flash period of a subsequent detection.

[0108] The distance extents Az of the different distance ranges 22i, 22j can differ from each other. Therefore, the detection durations for the various acquisition steps of the operating procedure can differ.

[0109] After capturing a respective distance range 22i, a triangulation determination of the distance of objects 22 to 24 in the unfiltered image areas can additionally be carried out using the at least one stereo camera 5, 14. Here, a stereo correspondence determination can again take place, as already explained above.

[0110] Prior to each acquisition step of the operational procedure, a Region of Interest (ROI) can be specified as part of an image field that can be acquired via the respective camera 5i, 14i, for example a road area identified by appropriate object filtering in the forward direction 4.

[0111] Fig. 5 shows a side view of another embodiment of the image acquisition device 1. Components and functions corresponding to those already explained above with reference to Figures 1 to 4 bear the same reference numbers and are not discussed again in detail.

[0112] The use of the device 1 for determining and detecting a diffuse scattering medium along a flash path 28 between the flash source 7 and an exemplary surrounding object 2 is explained with reference to Fig. 5. The camera 5i of the stereo camera 5 is shown as an example in Fig. 5, which enables both triangulatory distance determination and time-of-flight (TOF) distance determination.

[0113] In the acquisition example shown in Fig. 5, the device 1 acquires a total of three distance ranges 22i, 222, 223 by specifying corresponding TOF image acquisition parameters. The first, nearest distance range 22i extends by a first distance di. The next further distance range 222 extends by a second distance d2, for which d2 = 2 x di. The next furthest, third distance range 22s extends by a third distance di, for which di = 3 x di.

[0114] The specification of image acquisition parameters for the three distance ranges 22i to 22s includes, in addition to the specification of the TOF image acquisition parameters "illumination start", "illumination duration", "detection start" and "detection duration", a specification of a flash intensity li for illuminating the respective distance range 22i. The flash intensity I2 for illuminating distance range 222 is four times greater than the flash intensity li for illuminating the first, nearest distance range 22i. The flash intensity I3 for illuminating the third, most distant distance range 223 in the example according to Fig. 5 is nine times greater than the flash intensity li. The flash intensity li therefore depends quadratically on a distance di within a total distance range 25 to be acquired.Assuming a constant reflectivity or scattering power of an ambient surface 29, the quadratic dependence of the illumination intensity L on the distance di results in the sensor array 6i of the camera 5i experiencing constant intensity illumination during image acquisition in the three distance ranges 22i, 222, and 223. This constant intensity illumination of the sensor array 6i only occurs if, along flash paths 3h, 3h, 313 between the flash source 7 and the environment in the distance ranges 22i to 22s, there is no significant scattering of the illumination flash 8 by the diffuse scattering medium 27.In other words: Inequalities in the intensity illumination of the sensor array 61 along the detection paths 3 Oi with the given quadratic dependence of the illumination intensity L on the distance di within the detection range 22i are an indication of the presence of the diffuse scattering medium 27.

[0115] The following cases can be distinguished: a) The diffuse scattering medium 27 is located within the currently detected distance range, i.e., within the distance range 222 in the case shown in Fig. 5. In this case, the diffuse scattering medium 27 is directly illuminated along the illumination path 3 h, and a corresponding scattering of the illumination light 8 through the diffuse scattering medium 27 into the sensor array 61 is measured during TOF detection, leading to an increase in the intensity of the illumination along the detection path 302. b) The diffuse scattering medium 27 is located between the flash light source 7 and the sensor array 6i on the one hand, and the currently detected distance range, for example, the distance range 223, on the other. In this case, the illumination light is attenuated along the flash light path 3 h as it passes through the diffuse scattering medium 27.The illumination light backscattered from the detection area 22s towards the sensor array 61 along the detection path 30s is further weakened due to additional scattering by the diffuse scattering medium 27. As a result, the intensity illumination of the sensor array 61 when detecting the distance area 223 is weakened due to the cloud of diffuse scattering medium 27 present in the intermediate distance area 222.

[0116] A combination of an increased intensity illumination of the sensor array 61 when detecting the distance range 222, compared to the expected constant intensity illumination, and a reduction in the intensity illumination of the sensor array 61 when detecting the distance range 223 thus enables the identification of the diffuse scattering medium 27 within the distance range 222.

[0117] In the image acquisition devices 1 and 21 described above, the pixel sensitivity of sensor pixels of the various sensor arrays 61, ... can be controlled as a function of the incident light intensity, i.e., as a function of the intensity illumination of the sensor array 61, ... . In particular, y-correction can be applied during image processing using the respective sensor array 61, ... . The exponent y used in the y-correction can vary in the range between, for example, 0.2 and 5. Depending on the value of y, a pixel sensitivity suitable for adapting to high dynamic range can then be achieved either at high or low intensity illumination of the sensor array.

[0118] A selection of object distance ranges 22i to be detected can be adapted to a movement mode of the detection device 1 or 21 relative to the environment in certain operating modes of the detection device 1 or 21. In particular, the total distance range 25 can be adapted to a relative speed of the device 1, 21 to the environment. Adaptation to a reverse or cornering mode is also possible.

[0119] Alternatively or in addition to the operating procedure described above with reference to Figures 3 and 4, the respective device 1 or 21 can also be operated as follows:

[0120] By appropriately assigning, in particular, the flash illumination duration and the detection duration, the respective device 1, 21 uses the time-of-flight (TOF) principle to detect the time-of-flight (TOF) distance of at least one object 2 or 22 to 24. Parallel or sequentially, the stereo camera 5, ... of the device 1, 21 uses the triangulatory principle to detect the triangulatory distance of the respective object 2 or 22 to 24. The detection values ​​"TOF distance" and "triangulatory distance" are then compared during operation of the device 1, 21.

[0121] It is particularly possible to use the determined TOF distance as an input for triangulation. This can especially facilitate correspondence analysis during triangulation. In particular, it allows for the narrowing of the disparity range to be covered in the correspondence analysis. The reverse is also possible, i.e., using the triangulated distance as an input for distance measurement using the TOF principle. This can be used to determine or divide a given total distance range into segmented distance ranges and to adjust the TOF image acquisition parameters based on the distance.

[0122] The illumination intensity from the flash source 7 can be specified in such a way that a common and, in particular, scalar brightness control of all images of the Z-buffering of the first figures 3 and 4 is carried out, with a constant intensity illumination of the respective sensor arrays 6i, ... This brightness control can be used in the form of a control loop in the device 1, 21.

[0123] Calibration of the dependence of the illumination intensity of the flash light source on the distance, i.e., an illumination intensity function, can be carried out using a calibration setup with a scattering medium with a homogeneous scattering particle distribution.

[0124] The pixel sensitivity as a function of the incident light intensity can be set in the form of a proportional function or in the form of a non-linear function.

[0125] With reference to Fig. 6, a method for calibrating a camera system with three cameras 5i, 52, and 5s is explained below, where these cameras can be either device 1 or device 21. Each camera has a sensor array 6i, 62, 63, corresponding to what has already been explained above in connection with the embodiments of devices 1 and 21. The sensor array 6i can also be a CCD array.

[0126] To illustrate spatial relationships, an xyz coordinate system is used below in connection with Figure 6. The x-direction in Figure 6 runs to the right. The y-direction runs perpendicular to the plane of Figure 6 and into it. The z-direction in Figure 6 runs upwards.

[0127] The cameras 5i to 5s are arranged side by side in the area of ​​a common camera arrangement plane, namely the xy-plane of figure 6, along a camera arrangement coordinate x.

[0128] The cameras 5i are designed as cameras with an object-side opening angle s of at most 30°. Edge-side limits 351, 352, 35s of this opening angle s of the respective cameras 5i, 52 and 5s are shown in Figure 6.

[0129] The 5i cameras are specifically designed as telephoto lens cameras with a focal length in the range of 10 mm.

[0130] To calibrate cameras 5i to 5s, three calibration markers are selected to define calibration marker points PA, PB, and Pc, respectively. These calibration marker points PA, PB, and Pc are positioned at different distances dA, dß, and dc from the camera's arrangement plane xy. The calibration marker points PA, PB, and Pc can be natural, depth-graded marker points within a scene or artificially created, depth-graded marker points.

[0131] The calibration marker points PA, PB and Pc lie within all three opening angles, i.e. within the respective opening angle limits 351, 352 and 35s of the cameras 5i, 52 and 5s.

[0132] With regard to the relationship of the distances dA, dß, dc to the other length parameters shown in Figure 6, Figure 6 is not to scale. The distances dA, dß, and dc are depicted as excessively small in Figure 6.

[0133] The calibration marker points PA, PB, and Pc are all positioned at the same x- and y-coordinates. The calibration procedure does not require a specific degree of accuracy regarding the alignment of the x- and y-coordinates of these marker points. The depth of alignment is what matters.

[0134] For example, the distances dA, dß and dc can be: dA= 50 m, dß = 100 m, dc= 200 m.

[0135] This distance depth layering corresponds to a typical street scene in front of a vehicle carrying the camera system to be calibrated. The x-distances between the cameras can be, for example, 2 m (b = 2 m). The focal length of each camera 5i can be in the range of 10 mm. With a typical camera pixel size, corresponding to a camera pixel pitch in the range of 3 pm to 10 pm, a disparity of, for example, 2 mm results, which corresponds, for example, to a distance that sweeps out 200 pixels of each camera 5i. The relative disparity is then 200 pixels. At a distance dß of 100 m, the relative disparity is accordingly 100 pixels, and at a distance dc = 200 m, the relative disparity is 50 pixels.

[0136] The distances dA, dß, and dc each differ from each other by at least 20%. In the embodiment shown in Figure 6, this difference between the distances is even at least 50%.

[0137] Depending on the specific calibration procedure, more than three calibration marker points (Pi) can be specified, e.g., four, five, six, eight, ten, or even more. This can improve the accuracy of the calibration.

[0138] The focal length f of each camera 5i is illustrated in Figure 6 in comparison to the distances d to the marker points P. In the embodiment shown in Figure 6, cameras 5i, 52, and 5s each have the same focal length f. Depending on the design of the camera system, the focal lengths of cameras 5i can also differ from one another.

[0139] In the arrangement shown in Figure 6, the distances d are each more than four times the focal length f. Other ratios between the focal lengths f of the cameras 5i and the distances d are also possible.

[0140] The entrance pupils of cameras 5i, illustrated in Figure 6 at Oi, O2, and O3, have baseline lengths corresponding to the x-distances between these entrance pupils Oi. In Figure 6, the x-coordinate of each camera entrance pupil Oi is labeled xi, X2, and X3, respectively, and assigned to the corresponding cameras 5i.

[0141] During the calibration procedure, the cameras 5i image the calibration marker points PA, PB, and Pc and capture them in a camera image plane of the respective camera 5i, i.e., in the xy-plane. This is illustrated in Figure 6 for camera 5s by a schematic image construction, in which the images P'A, 3, P'B, 3, and P'c, 3, i.e., the calibration marker image points, are shown on the sensor arrays 6i as intersection points between rays that extend from the entrance pupils Oi to the respective calibration marker points PA, PB, and PC.

[0142] The differences Xi - Xk between the respective x-coordinates of the entrance pupils of cameras 5i and 5j are the baselines corresponding to the stereo cameras to which these two cameras 5i and 5j belong. This difference Xi - xk is also referred to as bienik.

[0143] One and the same calibration marker point, e.g., marker point PA, is depicted shifted by a disparity shift UA, depending on the distance dA, the focal length f, and the relationship between the x-coordinates of this marker point PA and the entrance pupil of camera 5i compared to this x-coordinate Xi. These disparity shifts for marker points PA, PB, and Pc are highlighted as examples for camera 53 in Figure 6 at UA,3, UB,3, and uc,3.

[0144] The difference between the disparity shifts when mapping each of the marker points by two cameras 5i and 5k of a camera pair is thus measured, using marker point PA as an example, as the disparity difference duA,ik = UA,i - UA,k = A. Corresponding disparities B and C result when mapping marker points PB and PC with the respective cameras 5i, 52, and 5s. These disparity differences are also simply referred to as disparities.

[0145] During the calibration procedure, for each pairing i and k of two cameras, 5i, 5k, the disparity difference duA,ik (= UA,i - UA,k), duB.ii,- and duc.ii, of the image of the respective calibration marker point in the camera image plane xy, is measured. A total of nine measured disparities du result for the three marker points PA, PB, Pc and the three possible camera pairings 5ik, namely 512, 5 and 523.

[0146] Since the camera 52 is positioned at exactly the same x-coordinate X2 as the three calibration marker points PA, PB, PC, when imaging these calibration marker points PA, PB, PC by the camera 52, there are no x-distances of the calibration marker image points from this x-coordinate X2, so that by definition the camera 52 has disparity shift values ​​UA, 2, UB, 2 and uc, 2 = 0.

[0147] The stereo imaging formula yields: d = fblen / du (I)

[0148] Solving for the disparity du, this yields, for example, for the calibration marker point PA and cameras 5i and 5k: dUA,ik = UA,i - UA,k = f (Xi - Xk) / dA (II) The three cameras 51, 52, and 5s are not perfectly aligned with the camera arrangement plane xy, but there are deviations from it, which is the basis for the calibration. These deviations can be described analogously to the description of a ship's motion by a pitch angle, a roll angle, and a yaw angle of the camera system with cameras 5i to 63 compared to the arrangement plane xy, or by a corresponding pitch, roll, and yaw of each of these cameras 5i to 5s individually.

[0149] A corresponding positional error can be accounted for in the stereo imaging description of the equations above by including a tilting error 5ui - 5uk normalized to the focal length, i.e., a relative tilt between cameras 5i and 5k, each of the camera pairs. From equation (II) above, the following results, again using the calibration marker PA and cameras 5i and 5k as examples:

[0150] (UA,i - UA,k) / f = (Xi - Xk) / dA - (5ui - 5uk) (III)

[0151] The left side of the above equation (III) is known, with respect to the disparity, by measurement and, with respect to the camera focal length, by pre-calibration.

[0152] For the three calibration marker points PA, PB and Pc and the three possible camera pairings 512, 513 and 523, a total of nine equations of the above type (III) with nine unknowns result, namely the respective x-coordinates xi, X2, X3 of the entrance pupils Oi, O2 and O3 of the three cameras 5i, 52, 53, the respective rotation errors 5ui, 5u2 and Sus and the z-distances dA, dß, dc of the calibration marker points PA, PB and Pc.

[0153] After measuring the disparities, it is therefore possible to determine these x-coordinates, the rotation errors and the distances of the calibration marker points using the additionally known camera focal lengths f.

[0154] During operation of device 1 or 21, selected objects 2 or 22 to 24 can be tracked after the cameras have been calibrated. Once one of the objects 2 or 22 to 24 has been detected using triangulatory distance detection and / or time-of-flight (TOF) distance detection, the TOF image acquisition parameters, in particular, but also the triangulatory image acquisition parameters (for example, the length of the baseline of a selected stereo camera, a lens focal length of the cameras of the respective stereo camera, or a disparity range to be monitored), can be specified so that the object, once detected, is also detected again in subsequent image acquisitions.Here, the relative movement of the detected object to the device 1 or 21 can be taken into account, so that, for example, if it is expected that the object and device will approach each other, the distance of the selected tracking range 22i is reduced during TOF detection. The light intensity of the flash source can be adjusted and, in particular, controlled according to the tracking range of the object being tracked. Depending on the operating procedure of the device 1, 21, exactly one object can be tracked, or several objects can be tracked simultaneously.

[0155] Provided that at least one object is tracked, intermittent tracking of the at least one tracked object and a scan over the entire distance range 25 can be performed, which increases the overall recording reliability with reasonable recording effort.

[0156] The image acquisition parameters for TOF acquisition of multiple distance ranges 22i, 22i+i can be adjusted so that detection via the respective camera occurs for exactly one detection period (same detection start time, same detection duration) for these distance ranges 22i, 22i+i. In extreme cases, this can apply to all distance ranges 22i within the total distance range 25.

[0157] When tracking objects, the extent of the respective distance range 22i, in which the object is expected, can be adjusted by appropriately modifying the flash illumination duration. For more precise object tracking, shorter flash illumination durations and shorter detection durations can be specified, with a corresponding increase in distance accuracy during time-of-flight (TOF) distance determination.

[0158] The device 1 can include a neural network or an AI module, which can be used to train the respective operating procedure and to improve image germination accordingly.

Claims

Patent claims 1. Method for calibrating a camera system with at least three cameras (5 i, 52, 5s) with known focal lengths (f), which are arranged next to each other in the area of ​​a common camera arrangement plane (xy) along a camera arrangement coordinate (x) and whose object-side opening s angle is at most 45°, comprising the following steps: Selecting at least three calibration markers to specify one calibration marker point (PA, PB, PC), wherein the calibration marker points (PA, PB, PC) are arranged at different distances (dA, dß, dc) from the camera arrangement plane (xy), Imaging the calibration marker points (PA, PB, PC) with the cameras (5i, 52, 5s) to capture calibration marker image points (PA , PB', PC') in a camera image plane of the respective camera (5i, 52, 5s), For each pairing of two of the cameras (5i, 52, 5s), measure a disparity (A, B, C) of the image of the respective calibration marker pixel (PA , PB', PC') in the respective camera image plane, Determine, from the at least nine measured disparities (Aik, Bik, Cik) and the camera focal lengths (f), both — of distances (xi - Xk) between each of the cameras (5i, 5k) of a camera pair along the camera arrangement coordinate (x) as well as — due to positional errors (5ui - 5uk) of each camera pair (5 5k) resulting from deviations in the arrangement of the cameras (5i, 52, 5s).

2. Method according to claim 1, characterized in that the distances (A, dß, dc) of the calibration marker points (PA, PB, PC) differ from each other by at least 20%.

3. Method according to claim 1 or 2, characterized in that more than three calibration marks are provided for specifying more than three calibration mark points (PA, PB, PC,. ..) which are arranged at different distances ( A, dß, dc,. . .) to the camera arrangement plane (xy).

4. Camera system calibrated according to a method according to one of claims 1 to 3, wherein at least one of the cameras (5i, 52, 5s) is designed as a telephoto lens camera.

5. Camera system according to claim 4, wherein the camera pairs (5i, 5k) are designed as stereo cameras for triangulatory object distance determination within a spatially imaged environment.

6. Camera system according to claim 5, - with at least one flash light source (7, 13) for ambient illumination with controlled specification of a flash illumination start (t = 0, t = Uash) and a flash illumination duration (Atfiash) of successive illumination flashes, - where the cameras (5i, 52, 14i, 142, 15i, 152, 15s, 154, 20i, 202, 2O3, 2O4) of the stereo camera (5, 14, 15, 20; 5, 14) each a sensor array (6) with controlled specification of a detection start (t = tß, t = tEi) and a detection duration (At) exp ) exhibit, - with a control unit (10) that is connected to the cameras (5i, 52, 14i, 142, 15i, 152, 153, 154, 20i, 202, 2O3, 2O4) and the flash source (7, 13) is in signal connection, for the time-synchronized specification of the following image acquisition parameters: the flash illumination start (t = 0, t = tiiash) and the flash illumination duration (Atiiash) of the flash source (7, 13) as well as the detection start (t = tE, t = tEi) and the detection duration (Atexp) of the cameras (5i, 52, 14i, M2, 15i, 152, 153, 154, 20i, 202, 203, 204).

7. Camera system according to claim 6 - with at least one additional flash light source (13) with controlled Specification of a flash illumination start time (t = 0, t = tfiash) and a flash illumination duration (Atfiash) of successive illumination flashes (8, 81) and / or - with at least one additional stereo camera (14) with two additional cameras spaced apart from each other (14i, M2) for triangulatory object distance determination within the encompassed environment.

8. Camera system according to claim 6 or 7, characterized in that the control unit (10) has an image processor for real-time processing.

9. Method for operating a camera system according to any one of claims 6 to 8, comprising the following steps: - Specifying the image acquisition parameters such that a first distance range (gz-1) is captured, - Capturing the distance range (gz-1) with the specified image capture parameters using the cameras of at least one stereo camera, - Specifying the image acquisition parameters such that a further distance area (gz) is captured which borders on or partially overlaps the previous distance area (gz-1) along the forward direction (4) in a capture progress direction (+z), - Capturing the further distance range (gz) with the last specified image capture parameters using the cameras of at least one stereo camera, - digital filtering of the last captured image to remove unstructured image areas, - Overwriting the image captured in the previous capture step with the most recently captured image in unfiltered image areas of the most recently captured image to generate an updated image, - Specifying the image acquisition parameters such that a further distance range (gz+1) is captured, which borders on or partially overlaps the previous distance range (gz) along the forward direction (4) in the acquisition progress direction (-z), - Repeat the last steps: capture, digital filter, overwrite and specify further distance ranges (224 to 22i) until a specified total distance range (25) is covered, which results from an overlay of the captured distance ranges (22i to 22max), - Displaying the most recently updated image.

10. Method for operating a camera system according to any one of claims 6 to 8, comprising the following steps: - Recording a Time of Flight (TOF) distance of at least one Object (2; 22 to 24) via a corresponding assignment of the flash illumination duration and the detection duration using the TOF principle, - Determining a triangulatory distance of the object (2; 22 to 24) using at least one stereo camera using the triangulatory principle, - Comparison of TOF distance and triangulatory distance.

11. Method according to claim 10, characterized in that the TOF distance is used as an input variable for the triangulatory acquisition.

12. Method according to one of claims 9 to 11, characterized in that the illumination intensity of the flash light source (7, 13) is adjusted to a distance range (220) to be detected.

13. Method according to claim 12, characterized in that the flash intensity depends quadratically on a distance (di) within the distance range (220) to be detected.

14. Method according to one of claims 9 to 13, characterized in that a pixel sensitivity of a sensor pixel of the sensor array (6) is controlled as a function of an incident light intensity of the flash light source (7, 13).

15. Method according to one of claims 9 to 14, characterized in that a selection of object distance ranges (22i) to be detected is adapted to a movement mode of the camera system (1; 21) relative to the environment.

Images