Method and device for detecting an environment, and vehicle with such a device

Abstract

Method for detecting an environment, in particular a vehicle (3), wherein - the environment in a first detection area (7) is periodically detected by means of a laser scanner (5), wherein - the environment in a second detection area (11) is detected by means of an optical camera (9), wherein - the first detection area (7) and the second detection area (11) at least overlap, wherein - an optical sensor (19) of the camera (9) is exposed at least twice within a period of the laser scanner (5), and wherein - a first exposure time for a first exposure of the at least two exposures of the optical sensor (19) is selected and synchronized with the laser scanner (5) such that - the first exposure takes place within a first time window in which the laser scanner (5) detects the first detection area (7), and wherein - a second exposure time for a second exposure of the at least two exposures of the optical sensor (19) is selected to be greater than the first exposure time for the first exposure.

Description

[0001] The invention relates to a method and a device for detecting an environment, in particular a vehicle environment, as well as a vehicle with such a device.

[0002] It is known to combine optical cameras with laser scanning methods or laser scanners for environmental detection, particularly in vehicles. For example, the US patent application US 2016 / 0180177 A1 discloses a system for estimating a road boundary arranged in a motor vehicle, which system comprises a camera on the one hand and a lidar detector on the other. A first probabilistic model for the road boundary is determined from camera data, a second probabilistic model for the road boundary is determined from the lidar data, and the probabilistic models thus determined are fused to obtain a fused probabilistic model, from which the road boundary is estimated.

[0003] German patent application DE 10 2017 108 248 A1 discloses a computer-implemented method for road feature recognition. An image originating from a camera system connected to a vehicle on a road is received and further processed. The data thus obtained can be merged with other sensor data, for example, from lidar sensors, to improve detection and classification accuracy and reliability.

[0004] In general, optical cameras provide high-resolution images of the captured environment, but without distance information. Laser scanners, on the other hand, create 3D point clouds of the scanned environment, capable of providing highly accurate distance information. However, they have a significantly lower resolution.

[0005] It is also known to semantically segment images captured by optical cameras, particularly using neural networks. Objects in the images are recognized and each pixel is assigned a class label, such as "roadway," "vehicle," or "pedestrian." In this way, the image is divided into semantic segments. It would be advantageous to be able to combine this semantic segmentation information with the 3D point clouds obtained by a laser scanner, so that the 3D points in the point clouds are also assigned corresponding class labels. This would significantly improve scene interpretation in a vehicle and the prediction of the behavior of other road users.However, this requires precise synchronization of the scanning of the environment by means of the laser scanner on the one hand and the imaging of the environment in the optical camera on the other hand, as well as the selection of the shortest possible exposure time for the optical camera in order to obtain sufficiently sharp images even when the vehicle is moving.

[0006] Such short exposure times, however, are problematic if the camera images are also to be used preferentially for the detection of light signals, for example traffic lights, traffic signs designed as light signals, brake lights, indicators, and the like. This is particularly due to the fact that modern light signals are typically operated with light-emitting diodes, which only emit short, rapidly successive light pulses, operated in particular by means of pulse-width modulation. If exposure times are too short, there is a risk that the exposure may occur between the light pulses of the light signal - i.e., during a dark phase - in which case the state of the light signal, for example, the switching state of a traffic light, the type of traffic sign displayed, and the like, can no longer be recognized in the image.

[0007] A reliable semantic segmentation of 3D point clouds using camera images on the one hand and a reliable recognition of light signals on the other hand therefore appear to be opposing, incompatible goals.

[0008] EP 2 389 007 A2 discloses a method and apparatus for temporally interpolating a three-dimensional depth image to generate an intermediate depth image at a desired time. The apparatus interpolates depth images generated by a depth camera using a temporal interpolation method, generates an intermediate depth image at a new time using the interpolated depth images, and combines the generated intermediate depth image with color images to generate a high-precision 3D image.

[0009] US 2016 / 0116593 A1 discloses a device for object detection and a method for its operation. The device comprises a sensor unit with a light distance and range sensor (LiDAR) for acquiring LiDAR data and a camera sensor for acquiring image data, an area division unit configured to divide a common detection area of ​​the LiDAR sensor and the camera sensor into a plurality of areas, a data analysis unit configured to analyze the LiDAR data to extract first object information and / or to analyze the image data to extract second object information, and a sensor signal convergence unit configured to determine, based on the first object information and / or the second object information, whether a dangerous object is present in the respective divided area.

[0010] The invention is based on the object of creating a method and a device for detecting an environment, in particular a vehicle with such a device, wherein the aforementioned disadvantages do not occur.

[0011] The problem is solved by creating the subject matter of the independent claims. Advantageous embodiments emerge from the subclaims.

[0012] The object is achieved in particular by providing a method for detecting an environment, in particular an environment of a vehicle, wherein the environment is periodically detected in a first detection range by means of a laser scanner, wherein the environment is detected in a second detection range by means of an optical camera. The first detection range and the second detection range overlap at least with one another. An optical sensor of the camera is exposed to light at least twice within a period of the laser scanner, and a first exposure time for a first exposure of the at least two exposures of the optical sensor is selected and synchronized with the laser scanner such that the first exposure occurs within a first time window in which the laser scanner detects the first detection range.This ensures that the image capture by the optical camera is synchronized with the capture of the first capture area, which at least overlaps the second capture area, so that the laser scanner, on the one hand, and the optical camera, on the other hand, at least partially image the same area of ​​the surroundings. The first exposure time within the first time window is selected to be sufficiently short to obtain sharp images from the camera even when the vehicle is moving. The camera's exposure is therefore synchronized with the laser scanner, and the camera is exposed at least twice within the laser scanner's period, with one of the exposures occurring simultaneously with the scanning of the surroundings by the laser scanner, in particular simultaneously with the capture of the common, overlapping capture area.By providing a second exposure of the optical sensor within the laser scanner's period, reliable detection of pulse-width modulated light signals is also possible, since a second exposure time for the second exposure can be selected arbitrarily within the laser scanner's period and, in particular, longer than the first exposure time within the first time window. Thus, within one laser scanner period, both data that enables semantic segmentation of the optical camera image and the 3D point cloud captured by the laser scanner, as well as optical data that allows for reliable detection of light signals, can be obtained.

[0013] A detection range is understood here in particular to be an angular range around a vertical axis, in particular the vertical axis of a vehicle, and therefore an azimuth angular range. The first detection range is a corresponding angular range in which the laser scanner actually detects the surroundings. The first detection range is preferably smaller than a scan range of the laser scanner that the laser scanner covers within a period. In particular, it is possible for the laser scanner to be designed as a laser scanner rotating about an axis, which preferably rotates about a vertical axis, i.e. a vertical axis, wherein it covers a scan range of 360° within a period. However, it preferably does not detect data in the entire scan range, but rather only in the smaller, first detection range.

[0014] The second detection range is accordingly an angular range given by an aperture angle of the optics of the optical camera.

[0015] The fact that the first detection area and the second detection area at least overlap means that they are at least partially congruent.

[0016] Preferably, the detection areas are substantially congruent; more preferably, they overlap completely. In particular, one detection area selected from the first detection area and the second detection area can lie entirely within the other detection area selected from the second detection area and the first detection area. It is also possible for the detection areas to be completely congruent, meaning they overlap with each other everywhere. The terms "overlap" and "congruent" are used in particular with reference to the azimuth.

[0017] A vertical axis or a vertical axis is understood here to mean, in particular, an axis that points in the direction of a gravitational vector and / or that is perpendicular to a support plane or roadway plane on which a vehicle that is set up to carry out the method stands or travels.

[0018] A Lidar detector (Lidar - Light Detection And Ranging) is preferably used as a laser scanner.

[0019] An optical camera is an optical image recording device designed to capture static or moving images, particularly two-dimensional static or moving images. The optical camera can be configured, in particular, as a still camera or video camera. However, it is also possible to use an optical 3D camera.

[0020] The laser scanner preferably continuously scans the environment periodically. The camera's optical sensor is preferably exposed at least twice in a plurality of laser scanner periods. Particularly preferably, the optical sensor is exposed at least twice in each laser scanner period.

[0021] Preferably, the camera's optical sensor is exposed more than twice within one period of the laser scanner. Multiple exposures can be advantageously used to obtain additional or more precise information about the environment, in particular to reduce dead times for image acquisition. In particular, more than one exposure outside the first time window can be provided. In particular, more than one image acquisition can be provided with an exposure time that is longer than the first exposure time within the first time window.

[0022] According to a further development of the invention, the second exposure of the at least two exposures of the optical sensor takes place outside the first time window in a second time window, wherein the laser scanner preferably does not detect the first detection area in the second time window. In general, any time window within the period of the laser scanner can be selected for the second exposure, since in this case no synchronization with the detection by the laser scanner is necessary. However, temporal positioning of the second exposure outside the first time window enables a particularly clear separation between those data that are intended for synchronization with the scanning by the laser scanner and those data that are not intended for this purpose.

[0023] According to a further development of the invention, the first exposure time takes place in a central region of the first time window, in particular symmetrically to one half of the first time window. Since, according to a preferred embodiment, the laser scanner captures the first detection area at a constant detection speed, in particular at a constant angular speed, this selection results in the first exposure taking place precisely when the laser scanner passes the center of the first detection area. This results in a particularly good match between the recorded optical image on the one hand and the captured 3D points of the laser scanner on the other. This is particularly the case when the optical camera is also aligned with its second detection area centrally on the first detection area.

[0024] According to the invention, the second exposure time for the second exposure is selected to be longer than the first exposure time for the first exposure. In particular, the second exposure time for the second exposure is selected to be long enough to reliably detect pulse-width modulated light signals, in particular traffic lights, illuminated traffic signs, brake lights, indicators, and the like, so that, in particular, even for the shortest known pulse-width modulated traffic signal, at least one bright phase is detected within the second exposure time.

[0025] The first exposure time is preferably chosen to be short enough to capture sufficiently sharp images even when the vehicle is moving, enabling meaningful semantic segmentation and their assignment to the 3D point clouds captured by the laser scanner.

[0026] According to a further development of the invention, the first exposure time and / or the second exposure time are adapted to the period of the laser scanner. This ensures, on the one hand, precise synchronization of the optical detection with the laser scanner, and, on the other hand, it is ensured that, in particular, the second exposure time does not exceed the period of the laser scanner.

[0027] The first exposure time is preferably at most 10 ms, preferably at most 8 ms, preferably at most 7 ms, preferably at most 6 ms, preferably at most 5 ms, preferably at most 4 ms, preferably 4 ms.

[0028] The second exposure time is preferably at most 45 ms, preferably at most 40 ms, preferably at most 30 ms, preferably at most 20 ms, preferably at most 15 ms, preferably 12 ms, preferably more than 10 ms, preferably from at least 11 ms to at most 45 ms, preferably from at most 11 ms to at most 40 ms, preferably from at least 11 ms to at most 30 ms, preferably from at least 11 ms to at most 20 ms, preferably from at least 11 ms to at most 15 ms.

[0029] In a preferred embodiment, the period of the laser scanner is 90 ms, with the laser scanner covering a scanning range of 360° in these 90 ms. The first detection range is preferably an angular range of 120°, which is therefore scanned within 30 ms at a constant angular speed of the laser scanner. The laser scanner requires the remaining 60 ms of the period to complete the rotation. The first detection range lies, without loss of generality, in an angular range from 0° to 120°. If the first exposure time is 4 ms, the first exposure of the optical camera preferably occurs when the laser scanner is at approximately 60°, i.e., in the center of the first detection range. If, for example, the laser scanner begins scanning at 0° at a time t = 0, without loss of generality, the camera's exposure preferably begins at time t = 13 ms and lasts 4 ms.During this time, the laser scanner passes the center of the first detection zone at t = 15 ms. After 30 ms, the entire first detection zone is scanned.

[0030] When the second exposure of the optical sensor occurs is fundamentally irrelevant. For example, if the second exposure begins at t = 58 ms, a constant camera cycle time of 45 ms can be achieved. To capture currently known pulse-width modulated light signals, a duration of 12 ms for the second exposure time is advantageous. A new measurement cycle begins after 90 ms.

[0031] The exposure and exposure time of the optical camera and the scanning by the laser scanner are coordinated as follows: During a scan by the laser scanner, the optical camera is briefly exposed precisely when the currently scanned area of ​​the laser scanner and the viewing direction of the camera coincide. Since the period of the laser scanner is usually longer than the exposure time and dead time between two exposures of the optical camera due to the rotating units, the camera can be exposed again with a long exposure time when the laser scanner is not currently scanning the surroundings or is scanning areas outside the second detection area of ​​the camera. This results in a shorter exposure image synchronized with the laser scanner and a longer exposure image for light character recognition.

[0032] It is not mandatory for the laser scanner to not perform environmental detection outside the first detection area. In principle, the laser can also scan the environment within the entire scanning area. In this case, however, the first detection area is the area scanned by the laser scanner that overlaps with the second detection area of ​​the optical camera.

[0033] According to a further development of the invention, at least one image of the optical sensor captured with the first exposure time is semantically segmented, wherein the semantic segmentation is applied to data from the laser scanner. In particular, the semantic segmentation of an image captured with the first exposure time is applied to the laser scanner data acquired during the same laser scanner period in which the image captured with the first exposure time was captured. This enables a precise assignment of the semantic segmentation of the optical image to the laser scanner data.

[0034] The semantic segmentation is preferably carried out using a neural network, in particular a so-called convolutional neural network, in particular using deep learning.

[0035] Finally, the object is also achieved by providing a device for detecting an environment, in particular an environment of a vehicle, which device has a laser scanner that is configured to periodically detect a first detection area of ​​the environment, wherein the device further has an optical camera that is configured to detect the environment in a second detection area, wherein the first detection area and the second detection area at least overlap. The device also has a control device that is operatively connected on the one hand to the optical camera and on the other hand to the laser scanner in order to control the optical camera and the laser scanner, wherein the control device is configured to carry out a method according to the invention or a method according to one of the previously described embodiments.In connection with the device, the advantages that have already been explained in connection with the method arise in particular.

[0036] The optical camera and the laser scanner are preferably aligned parallel to each other with respect to their main axes. The main axes are those axes that point toward the detection zones. In particular, the main axis of the laser scanner is an axis that points symmetrically toward the first detection zone. The main axis of the optical camera is, in particular, the optical axis of the camera's optics. If the main axes of the optical camera and the laser scanner are oriented parallel to each other, they have the same viewing direction.

[0037] The object is also achieved by providing a vehicle having a device according to the invention or a device according to one of the previously described embodiments. In connection with the vehicle, the advantages already described in connection with the method are particularly evident.

[0038] According to a further development of the invention, it is provided that the vehicle is designed as a motor vehicle, in particular as a passenger car, as a truck or as a commercial vehicle.

[0039] The invention is explained in more detail below with reference to the drawing.

[0040] Showing: Fig. 1 a schematic representation of an embodiment of a device for detecting an environment, and Fig. 2 a schematic representation of an embodiment of a method for detecting an environment.

[0041] Fig. 1 shows a schematic representation of an embodiment of a device 1 for detecting an environment, in particular the environment of a schematically indicated vehicle 3, which can preferably be designed as a motor vehicle, in particular as a passenger car, as a truck or as a commercial vehicle.

[0042] The device 1 has a laser scanner 5, which is configured to periodically capture the surroundings—here, the surroundings of the vehicle 3—in a first capture area 7. The device 1 also has an optical camera 9, which is configured to capture the surroundings in a second capture area 11. The first capture area 7 and the second capture area 11 overlap at least partially. The device 1 also has a control device 13, which is operatively connected to the laser scanner 5 on the one hand and to the optical camera 9 on the other hand, such that the control device 13 can control the laser scanner 5 and the optical camera 9. The control device 13 is configured to carry out a method that will be described below.

[0043] The optical camera 9 and the laser scanner 5 are aligned parallel to each other with respect to their main axes 15, 17. Fig. 1 shows a first main axis 15 of the laser scanner 5 and a second main axis 17 of the camera 9, which are oriented parallel to each other, so that the laser scanner 5 on the one hand and the optical camera 9 on the other hand have the same viewing direction.

[0044] The detection areas 7, 11 are each angular ranges of an azimuth angle.

[0045] The laser scanner 5, in particular, has a scanning range that is larger than the first detection range 7. In particular, the scanning range of the laser scanner 5 covers a full azimuth angle range of 360°. The laser scanner 5 periodically sweeps the entire scanning range, scanning the first detection range 7 of the surroundings within such a period.

[0046] The first detection area 7 is, for example, an angular range of 120°, whereby it extends, without loss of generality, into Fig. 1 extends from 0° to 120°. The first main axis 15 divides the first detection area 7 in half, i.e., at 60°.

[0047] The laser scanner 5 preferably scans the scanning area at a constant angular velocity.

[0048] As part of the method for detecting the environment, an optical sensor 19 of the camera 9 is exposed at least twice within one period of the laser scanner 5. A first exposure time for a first exposure of the at least two exposures of the optical sensor 19 is selected and synchronized with the laser scanner 5 such that the first exposure occurs within a first time window in which the laser scanner 5 scans and thus detects the first detection area 7.

[0049] The second exposure of the at least two exposures of the optical sensor 19 can take place outside the first time window in a second time window in which the laser scanner 5 preferably does not sweep over the first detection area 7.

[0050] Preferably, the first exposure takes place in a central area of ​​the first time window, in particular symmetrically to one half of the first time window. This ensures that the first exposure takes place when the laser scanner 5 is just reaching the central area of ​​the first detection area 7, preferably in Fig. 1 - in particular symmetrically - around the 60° marking, that is to say in particular symmetrically to the first main axis 15.

[0051] The second exposure time for the second exposure is preferably selected to be longer than the first exposure time for the first exposure. In particular, by specifically selecting the exposure times, it can be ensured that, on the one hand, the first exposure time is short enough to enable sufficiently sharp images for semantic segmentation and for applying this semantic segmentation to the 3D data acquired by the laser scanner 5, while, on the other hand, the second exposure time is selected to be long enough to reliably capture pulse-width-modulated light signals.

[0052] The first exposure time and / or the second exposure time is / are adapted to the period of the laser scanner 5.

[0053] At least one image of the optical sensor 19 or the optical camera 9 recorded with the first exposure time is semantically segmented, wherein this semantic segmentation is applied to data of the laser scanner 5, in particular to data obtained in the same period of the laser scanner 5 as the image recorded with the first exposure time.

[0054] Fig. Figure 2 shows a schematic representation of an embodiment of the method. The time t in ms is plotted on the bottom axis. An exposure K of the camera 9 is shown on a top axis; the second axis from the top shows the scanning of the environment by the laser scanner 5 in the first detection area 7 as a laser scan L; the third axis from the top shows the scanning angle S of the laser scanner 5 over the entire scanning range from 0° to 360°.

[0055] Purely by way of example and without loss of generality, the period of laser scanner 5 here is 90 ms, within which the full scanning range from 0° to 360° is covered. The first detection range 7 of 120° is covered within 30 ms. Without loss of generality, the first detection range 7 here begins at 0° and ends at 120°.

[0056] The first exposure time for the optical sensor 19 is 4 ms. It is positioned such that it is temporally located in a middle range of the first time window, i.e., from t = 0 ms to t = 30 ms, with the first exposure time beginning at t = 13 ms and ending at t = 17 ms. Thus, the temporal center of the first exposure time coincides precisely with the time at which the laser scanner 5 reaches the 60° mark of the first detection area 7, namely t = 15 ms.

[0057] The second exposure time is 12 ms. It is located outside the first time window in the period of laser scanner 5, beginning here at t = 58 ms, i.e., with a time lag of 45 ms from the start of the first exposure time.

[0058] After 90 ms or 360° of the scan angle S, the process is continued periodically.

[0059] Overall, with the method proposed here, the device 1 and the vehicle 3, both a meaningful semantic segmentation of camera images and 3D data of the laser scanner 5 can be carried out, and pulse-width modulated light signals can be reliably detected.

Claims

[1] Method for detecting an environment, in particular a vehicle (3), wherein - the environment in a first detection area (7) is periodically detected by means of a laser scanner (5), wherein - the environment in a second detection area (11) is detected by means of an optical camera (9), wherein - the first detection area (7) and the second detection area (11) at least overlap, wherein - an optical sensor (19) of the camera (9) is exposed at least twice within a period of the laser scanner (5), and wherein - a first exposure time for a first exposure of the at least two exposures of the optical sensor (19) is selected and synchronized with the laser scanner (5) such that - the first exposure takes place within a first time window in which the laser scanner (5) detects the first detection area (7), and wherein - a second exposure time for a second exposure of the at least two exposures of the optical sensor (19) is selected to be greater than the first exposure time for the first exposure. [2] Method according to claim 1, characterized by that the second exposure of the at least two exposures of the optical sensor (19) takes place outside the first time window in a second time window in which the laser scanner (5) preferably does not detect the first detection area (7). [3] Method according to one of the preceding claims, characterized by that the first exposure takes place in a central region of the first time window, in particular symmetrically to one half of the first time window. [4] Method according to one of the preceding claims, characterized by that the first exposure time and / or the second exposure time is / are adapted to the period of the laser scanner (5). [5] Method according to one of the preceding claims, characterized by that at least one image of the optical sensor (19) recorded with the first exposure time is semantically segmented, wherein the semantic segmentation thus obtained is applied to data of the laser scanner (5). [6] Device (1) for detecting an environment, in particular a vehicle (3), with - a laser scanner (5) which is designed to periodically detect the environment in a first detection area (7), - an optical camera (9) which is arranged to detect the surroundings in a second detection area (11), wherein - the first detection area (7) and the second detection area (11) at least overlap, and with - a control device (13) which is operatively connected to the laser scanner (5) and to the optical camera (9) in order to control the laser scanner (5) and the optical camera (9), wherein - the control device (13) is configured to carry out a method according to one of claims 1 to 5. [7] Device (1) according to claim 6, characterized by that the optical camera (9) and the laser scanner (5) are aligned parallel to each other with respect to their respective main axes (15,17). [8] Vehicle (3) comprising a device (1) according to one of claims 6 or 7. [9] Vehicle (3) according to claim 8, characterized by that the vehicle (3) is designed as a motor vehicle, in particular as a passenger car, as a lorry or as a commercial vehicle.

Images