Method for identifying at least one region in optical image information, optical capturing system, and vehicle

Abstract

The invention relates to a method for identifying at least one region (44, 46) in optical image information, having the steps of: ascertaining first optical image information of a surrounding area (22) using a transmitted and received optical signal (LS), said first optical image information comprising a distance image (34) of the surrounding area (22), ascertaining second optical image information of the surrounding area (22) using received light (L), said second optical image information comprising a two-dimensional image (36) of the surrounding area, ascertaining the contour of at least one object (O) in the first and second image information, and identifying the at least one region (44, 46) using a comparison of the contour in the first and second image information. The invention additionally relates to an optical capturing system (10), to a vehicle (20), and to a computer program product.

Description

[0001] Method for identifying at least one area in optical image information, optical detection system and vehicle

[0002] Technical area

[0003] The application relates to a method for identifying at least one area in optical image information and to an optical detection system for a vehicle.

[0004] background

[0005] Modern vehicles (cars, vans, trucks, motorcycles, etc.) are equipped with a multitude of sensor systems whose data is used to inform drivers and / or provide them to driver assistance systems. These sensor systems record the vehicle's surroundings and other road users. Based on the recorded data, a model of the vehicle's environment can be created, and changes in this environment can be responded to.

[0006] Sensor systems are constantly being developed for various functions, e.g., for capturing environmental information in the near and far range of vehicles, such as passenger cars or commercial vehicles. Sensor systems can also be used for driver assistance systems, particularly assistance systems for autonomous or semi-autonomous vehicle control. They can be used, in particular, to detect obstacles and / or other road users in the front, rear, or blind spot area of ​​a vehicle. Sensor systems can be based on various sensor principles, such as radar, ultrasound, and optics.

[0007] An optical detection system for environmental detection, e.g., of vehicles, can be based on LIDAR technology (Light Detection and Ranging). A LIDAR system has an optical transmitter and an optical receiver. The transmitter can emit an optical transmission signal in the form of light, which can be continuous or pulsed. The optical transmission signal can also be modulated. In a LIDAR sensor, light in the form of laser beams in the ultraviolet, visible, or infrared range can be used. The receiver can receive the light after it has been reflected from an object in a detection area in the vicinity of the LIDAR sensor. The optical reception signal can be evaluated using the optical transmission signal, e.g., using a time-of-flight method, and the spatial position and distance of the object from which the reflection occurred can be determined.In this context, reflection or reflected light is understood to mean any light that is thrown back and is intended to include, in particular, light that is thrown back by scattering or absorption-emission.

[0008] US10564267B2 describes an optical detection system in which a light source emits light pulses into the surroundings, and a pixel-based image sensor receives reflected pulses and generates a main image. An image processor identifies oversaturated image portions of the main image and interprets the oversaturated image portions using additional image information captured by the image sensor. The additional image information can be obtained from a secondary image with low illumination.

[0009] Overview

[0010] A method for identifying at least one area in optical image information comprises:

[0011] Determining a first optical image information of an environment using a transmitted and received optical signal, wherein the first optical image information comprises a distance image of the environment,

[0012] Determining a second optical image information of the environment by received light, wherein the second optical image information comprises a two-dimensional image of the environment,

[0013] Determining a contour of at least one object in the first and second image information,

[0014] Identifying the at least one region using a comparison of the contour in the first and second image information. The contour of the object, also called outline or silhouette, refers to a curve that delimits the object from its surroundings. The contour refers to the outer boundary or outline of the object. It is the line that defines the edge or shape of an object and delimits it from its surroundings. In image information, the contour can be viewed as a sequence of points or vectors that describe the outline of an object. The contour can be obtained, for example, using mathematical algorithms and can be part of object detection. Contour detection in image information can be based, for example, on edge-based or threshold-based algorithms that use intensity values ​​or amplitude values ​​of image information.

[0015] The first and second pieces of image information relate at least partially to the same environmental scene. This allows the same edge to be identified in both the first and second pieces of image information, and at least one area can be identified that depicts the same environmental scene, e.g., the same object.

[0016] To identify the at least one region of interest, at least one distance image generated with active illumination by an optical signal and at least one two-dimensional image determined using received light are used. The received light can, in particular, differ from the optical signal. The differences in exposure for determining the first and second pieces of image information can then be used to identify the at least one region of interest. For this purpose, the same contour of the same object in the surrounding area is determined and compared in the first and second pieces of image information. This facilitates the identification of areas with unwanted optical artifacts.

[0017] The term artifact refers to unwanted effects or disturbances that can occur in optical systems and affect the quality or accuracy of images or measurements. These artifacts can have various causes, including the properties of the optical components, the design of the system, or the way the light is captured and processed. Optical artifacts in optical detection systems can include blooming and / or ghosting. Unwanted blooming or ghosting effects can occur in optical detection systems that work with emitted optical signals, such as lidar systems, for example, due to highly reflective or retroreflective objects in the environment and / or due to objects that are very close to the detection system.

[0018] Such objects, such as highly reflective, retroreflective, or close objects, can obscure areas larger than themselves due to excessive reflections in the image information, resulting in a bright halo around the highly reflective object in the image information. Other objects in the immediate vicinity of the highly reflective or retroreflective object may then also no longer be visible. This effect is known as blooming. The highly reflective or retroreflective objects can also appear in the image information in the form of phantom copies at a different location than where they actually exist. This effect is known as ghosting.

[0019] In one embodiment of the method, the identified region at least partially includes the contour. The differences in exposure used to determine the first and second pieces of image information can be used to identify the at least one region of interest based on the determined contour. For example, the second piece of image information can be selected such that saturation of a receiving sensor is unlikely. Contours detected in the second piece of image information can thus be assumed to be highly accurate. This facilitates the identification of regions with contours where undesirable artifacts, such as blooming and / or ghosting, may occur.

[0020] In particular, the comparison can be used to determine a difference between the contour in the first image compared to the contour in the second image. The difference in the contour can be used, for example, to determine whether the first and / or second image needs to be corrected. With knowledge of possible overexposure in the first image, the additional information in the second image can be helpful for further image processing.

[0021] Differences in the at least one contour between the first image information and the second image information can be determined, for example, by the first image information exceeding a certain distance value in the area of ​​the at least one contour and the second image information exceeding a certain distance value in the area of ​​the at least one contour. Alternatively or additionally, other image processing methods can also be used to determine differences in the image information.

[0022] By comparing the at least one contour, an optical artifact in the first and / or second image information can be detected. The contour in, for example, the first image information can thus be identified as requiring correction, for example, because an optical artifact is present in the area containing at least part of the contour. The second image information can be used for this correction.

[0023] In one embodiment of the method, after identifying the at least one region, a third piece of image information is determined using the first and second pieces of image information. The third piece of image information can be calculated, for example, with the aim of correcting an artifact present in the first and / or second pieces of image information using the respective other piece of image information.

[0024] The third piece of image information can, in particular, comprise the first piece of image information modified in the identified region using the second piece of image information. In particular, an artifact present in the first piece of image information, e.g., in the distance image, can be corrected using information from the second piece of image information.

[0025] In one embodiment of the method, the second image information comprises a gray image of the surroundings. A gray image is a two-dimensional image that contains only gray tones, without color information. It can also be referred to as a black and white image or monochrome image. In contrast to a color image, a gray image consists only of different gradations of gray tones, with each pixel having a brightness value between black (no brightness) and white (full brightness). By concentrating on the brightness values, gray images can be used, for example, when details, contrasts or, in particular, contours of a scene are important. Method according to one of the preceding claims, wherein the received light comprises background light and the second image information is determined by a pixel-by-pixel noise measurement. The received light can, for example,include the background light of the environment, and the two-dimensional image may be a passive image created without active illumination.

[0026] The optical signal and the light are received via the receiving sensor, which has an array of pixels that converts the optical signal and the light into electrical sensor signals for reception. For this purpose, the respective pixel can have, for example, photosensitive elements, e.g. photosensitive semiconductor elements. A pixel can also be referred to as a picture element. The pixel-based evaluation comprises the evaluation of at least one pixel of the sensor signal, i.e. the evaluation of at least one respective electrical sensor signal of the at least one pixel. A respective pixel or a respective group of pixels of the receiving sensor can in particular be designed to receive light from a specific direction. This directional information can also be referred to as angular information, since it indicates the angular direction in space from which the reflection of the received light occurred. For example,The direction and distance of the location where the reflection occurred can be used to create a 3D point cloud of the environment of the lidar system.

[0027] Pixel-based noise measurement analyzes the distribution of points in the pixel array of the receiving sensor. The points in the point cloud are counted within each pixel and statistically evaluated. By analyzing these statistical values, a pixel-by-pixel noise level can be determined. A higher number of points in a pixel can indicate higher measurement accuracy, while a greater variation in point density can indicate noise.

[0028] In the present case, the second image information therefore comprises a gray image which is obtained by a pixel-based noise measurement. The noise measurement can, for example, be based on background light and can be obtained passively, i.e. without an additional light source. In this embodiment, the received light corresponds to background light. In one embodiment, the received light comprises the received optical signal and the second image information is determined using the first image information. This has the advantage that the evaluation of the optical signal can simultaneously be used to determine the second image information. No additional receiving process has to be carried out. The evaluation of the optical signal to determine the second image information can, in particular, be selected such that the artifacts in the second image information are avoided. In this case, the determination of the second image information can, for example,an averaging of received amplitude values ​​of the optical signal over a section of the measuring distance, in particular over the entire measuring distance.

[0029] An optical detection system comprises an optical transmitting device configured to transmit the optical signal. The optical detection system further comprises an optical receiving device configured to receive the optical signal and may include the receiving sensor for this purpose. The optical detection system further comprises a computing device.

[0030] The computing device is configured to determine the first optical image information of the surroundings using the transmitted and received optical signal, wherein the first optical image information comprises the distance image of the surroundings. The computing device is further configured to determine the second optical image information of the surroundings as a function of received light, wherein the second optical image information comprises a two-dimensional image of the surroundings, e.g., a gray image. The computing device is configured to determine a contour of the at least one object in the first and second image information, and to determine the at least one region using a comparison of the contour in the first and second image information.

[0031] The optical receiving device can be configured to receive the light for determining the second piece of image information. For this purpose, the optical receiving device can have the receiving sensor, which can have a pixel array for receiving the light and the optical signal. The optical transmitting device can be configured to emit the light for determining the second piece of image information. The optical signal emitted by the optical transmitting device can in particular comprise the light for determining the second piece of image information. The optical transmitting device can have a laser light source configured to emit the optical signal. The optical transmitting device can optionally have a further laser light source configured to emit a further optical signal. The received light, which is used to determine the second image data, can comprise the further optical signal.Preferably, the further optical signal is weaker than the optical signal.

[0032] The optical detection system may be intended for use in a vehicle, wherein the vehicle may have an additional light source, e.g., its headlights. The light from this additional light source may be intended to be received as the light used to determine the second image information.

[0033] A computer program product contains instructions that can be executed by the computing device of the optical detection system, so that the described method is carried out by the computing device.

[0034] Fiourenliste

[0035] In the following, embodiments of this application are further explained and described with reference to the figures.

[0036] Fig. 1 schematically shows a method for identifying an area in image information,

[0037] Fig. 2 schematically shows a vehicle with an optical detection system,

[0038] Figs. 3 +4 example image information.

[0039] The same reference numerals are used in the figures for identical or similar elements. Representations in the figures may not be to scale. Figure description

[0040] Fig. 1 schematically illustrates a method for identifying an area in first and second image information.

[0041] In A, a first optical image information of an environment 22 is determined using a transmitted and received optical signal LS. The first optical image information comprises a distance image 34 of the environment 22. A distance image 34 is a three-dimensional image in which each pixel is assigned a distance value.

[0042] In B, a second optical image information of the environment 22 is determined by received light L, wherein the second optical image information comprises a gray image 36 of the environment 22.

[0043] In C, a contour of at least one object 0 is determined in the first and in the second image information.

[0044] In D, the contour in the first and second image information is compared, and the region 44, 46 is identified using the comparison. The identified region 44, 46 at least partially includes the contour. The comparison determines an optical artifact in the first and / or second image information.

[0045] In E, the artifact is corrected by determining a third image information using the first and second image information such that the artifact is corrected in the third image information.

[0046] The described method allows for the identification and correction of measurement errors from optical detection systems 10, in particular lidar systems. Such measurement errors can arise, for example, due to blooming and / or ghosting effects, e.g., on highly reflective, retroreflective, and / or nearby objects 0. For this purpose, at least one distance image 34 generated with active illumination by an optical signal LS and at least one gray image 36 determined using received light L are used.

[0047] The gray image 36, which is determined using the received light L, is selected such that the measurement errors to be corrected do not occur in the gray image 36. For example, a passive gray image 36 can be used which was generated without the use of active illumination. This passive gray image 36 is then generated using received light L, which includes, for example, the background light. It is also possible for the received light L to generate the gray image 36 to include reflected light originating from another light source, such as the headlights of a vehicle. It is also possible for the received light L to generate the gray image 36 to include reflected light L, which includes the reflected optical signal LS. In this case, the evaluation by the receiving device 14 is carried out such that the measurement error to be corrected does not occur in the gray image 36.

[0048] Fig. 2 schematically shows a vehicle 20, for example a passenger car, with an optical detection system 10, e.g. a lidar system. The optical detection system 10 is arranged in the front region of the vehicle 20. The surroundings 22 detected by the optical detection system 10 are located in front of the vehicle 20 in the direction of travel. An object O is schematically shown in the surroundings 22. The optical detection system 10 has an optical transmitting device 12 for emitting the optical signal LS. The transmitting device 12 can in particular have a light source for emitting laser light. The optical detection system 10 further has an optical receiving device 14 for receiving the reflected optical signal LS and optionally other light L. The transmitted optical signal LS and the received optical signal LS can be evaluated in a computing device 18 in order to detect the surroundings 22 and, for example,to capture the object O.

[0049] The illustrated optical detection system 10 is designed as a scanning detection system 10. The optical detection system 10 has an optical deflection device 16, which deflects the optical signal LS transmitted by the transmitting device 12 and the optical signal LS reflected from the environment 22 and optionally other light L in the direction of the receiving device 14. The deflection device 16 can change the angle at which the optical signal LS is deflected and thus realize, for example, a step-by-step scanning of the environment 22. A possible scanning movement of the optical signal LS is shown in Fig. 2 by the arrow 24. The described method can also be applied to optical detection systems 10 that are not scanning and / or do not have an optical deflection device 16.

[0050] The receiving unit 14 has a receiving sensor for receiving the optical signal LS and optionally the light L. The receiving sensor has pixels by means of which the light L and / or optical signals LS can be converted into electrical sensor signals.

[0051] The method described with reference to Fig. 1 can be carried out in the computing device 18. The transmitting process in the transmitting device 12, the receiving process in the receiving device 14, and, if applicable, the deflection effect of the deflection device 16 can also be monitored and controlled in the computing device 18.

[0052] In the example shown, the surroundings 22 in front of the vehicle 20 can be monitored in the direction of travel. It is also possible to arrange the optical detection system 10 in other areas of the vehicle 20, for example, in the rear area and / or in the side areas. It is also possible to arrange multiple optical detection systems 10 on the vehicle 20, in particular in corner areas of the vehicle 20.

[0053] With the optical detection system 10, stationary or moving objects O, in particular vehicles, persons, animals, plants, obstacles, road surface irregularities, in particular potholes or stones, road markings, traffic signs, open spaces, in particular parking spaces, precipitation or the like, in the environment 22 can be detected.

[0054] In the vehicle 20, the information generated by the optical detection system 10 can be used to implement autonomous or semi-autonomous driving functions.

[0055] Fig. 3 shows exemplary image information which can be determined using the optical detection system 10.

[0056] An amplitude image 32 represents the environment 22 in relation to the signal amplitude. Thus, the amplitude of the received optical signal LS is displayed for each pixel. Particularly in a lidar system, such an image 32 with the signal amplitude is also referred to as an intensity image 32. It provides information about the strength of the reflected optical signal LS, which is bounced back by various objects 0. By analyzing the signal amplitude, one can obtain information about the reflection properties of various objects 0. The intensity image 32 can be used together with the distance image 34 to obtain better information about the environment 22.

[0057] Fig. 3 also shows the distance image 34 and the grayscale image 36. In the distance image 34, each pixel is assigned a distance value. The distance value can be determined, for example, in the computing device 18 by evaluating the optical signal LS. The distance value refers to the distance from the optical detection system 10 at which the transmitted optical signal LS was reflected.

[0058] To generate the gray image 36, the received light L is evaluated. The received light L may, for example, comprise ambient light, from which a passive gray image 36 can then be generated, the generation of which does not require an additional light source.

[0059] The received light L may also include light originating from an external light source, such as a headlight of the vehicle 20. This may then result in greater brightness in the gray image 36 than in a passive gray image 36.

[0060] Furthermore, the received light L can be contained in the optical signal LS, which is evaluated anyway by the optical receiving device 14. This is particularly advantageous because the gray image 36 is recorded directly via the same signal path as used to determine the distance image 34 and / or the intensity image 32. This can refer to both the optical and the electrical portion of the signal path. By using this method, effects that would result from the use of other signal paths can be excluded. Furthermore, this method is advantageous in scanning detection systems 10 in which the detected environment 22 continues to rotate, for example, in the direction 24.

[0061] For this purpose, for example, the gray image 36 can be generated from the intensity image 32. The intensity image 32 already contains brightness information. If the intensity image 32 is an image with spatial information, the amplitude values ​​can be statistically processed over a certain distance range, e.g., the entire detected distance range of the environment 22. This can be done, for example, by averaging the amplitude values ​​over the entire detected distance in the environment 22.

[0062] It is also possible to generate the gray image 36 from a further optical signal that is weaker than the optical signal LS. For this purpose, the laser light source of the optical transmitting device 12 can, for example, additionally emit the further optical signal in addition to the optical signal LS. Alternatively or additionally, the optical transmitting device 12 can, for example, have a further laser light source that emits the further optical signal. The further optical signal can, for example, be emitted as a weak continuous light. The further optical signal can then be evaluated by the optical receiving device 14 to generate the gray image 36. In this exemplary embodiment, the received light L comprises the further optical signal.

[0063] The contours of one object 0 or multiple objects 0, which are provided by the grayscale image 36, are combined with the at least one distance image 34. For this purpose, the regions 44, 46 with the contours are identified, and the respective regions 44, 46 from the distance image and the two-dimensional image are compared. Through the comparison, the incorrect measurements caused by artifacts such as blooming and ghosting can be identified and subsequently corrected. Particularly with passive grayscale images 36, the incorrect measurements to be corrected, such as blooming or ghosting, do not occur or occur to a lesser extent due to the lack of illumination.

[0064] Exemplary areas 42, 44, 46 are shown in Fig. 4. These areas 42, 44, 46 correspond to the areas at the left edge of images 32, 34, 36. These areas 42, 44, 46 have the contour of object 0, which includes a traffic sign with a post.

[0065] Since the gray image 36 is captured using the received light L, it may be less susceptible to artifacts that can arise from using the optical signal LS to determine the distance image 34 and that can lead to incorrect measurements. In particular, objects 0 with high reflectivity and / or objects 0 in the vicinity of the optical detection system 10 are therefore less susceptible to incorrect measurements in the gray image 36.

[0066] In the distance image 34, in area 44 near object 0, the post of the traffic sign is outshone by the reflection of the retroreflective surface of the sign itself. This artifact leads to incorrect measurements that are not generated in the gray image 36 in area 46, for example, due to the lack of or much lower illumination. This means that in the gray image 36 in area 46, the post of the sign of object 0 is recognizable. This makes it possible to combine the contours provided by the gray image 36 in area 46 with area 44 of the distance image 34 and to identify the incorrect measurements caused, for example, by blooming and / or ghosting. After identification, these can then be calculated out and thus corrected.

[0067] Amplitude values ​​or noise measurements for determining the gray image 36 can also be reconstructed from accumulated measurements from the same or adjacent areas 44, 46 of the environment 22. This can be particularly advantageous in scanning detection systems 10, where the currently detected area changes continuously.

Claims

CLAIMS 1. A method for identifying at least one area (44, 46) in optical image information, comprising: Determining a first optical image information of an environment (22) using a transmitted and received optical signal (LS), wherein the first optical image information comprises a distance image (34) of the environment (22), Determining a second optical image information of the environment (22) by received light (L), wherein the second optical image information comprises a two-dimensional image (36) of the environment, Determining a contour of at least one object (0) in the first and second image information, Identifying the at least one region (44, 46) using a comparison of the contour in the first and second image information.

2. The method according to claim 1, wherein the identified region (44, 46) has the contour at least partially.

3. The method according to claim 1 or 2, wherein the comparison determines a difference between the contour in the first image information and the contour in the second image information.

4. The method of claim 3, wherein the at least one region (44, 46) is identified when the difference in the contour exceeds a certain distance value.

5. Method according to one of the preceding claims, wherein an optical artifact in the first and / or the second image information is determined by comparing the contour.

6. Method according to one of the preceding claims, wherein after identifying the at least one area (44, 46), a third image information is determined using the first and the second image information.

7. The method according to claim 6, wherein the third image information is determined such that the artifact in the third image information is corrected.

8. The method according to any one of the preceding claims, wherein the third image information comprises the first image information which is modified in the identified area (44, 46) using the second image information.

9. Method according to one of the preceding claims, wherein the second image information comprises a gray image (36) of the environment.

10. Method according to one of the preceding claims, wherein the received light comprises background light and the second image information is determined by a pixel-by-pixel noise measurement.

11. Method according to one of the preceding claims, wherein the received light (L) comprises the received optical signal (LS) and the second image information is determined using the first image information.

12. The method according to claim 11, wherein the determination of the second image information comprises an averaging of received amplitude values of the optical signal over a portion of the measuring distance, in particular over the entire measuring distance.

13. An optical detection system comprising an optical transmitting device (12) configured to transmit an optical signal (LS), an optical receiving device (14) configured to receive the optical signal (LS), and a computing device (18) configured to: determine a first piece of optical image information of an environment using the transmitted and received optical signal, wherein the first piece of optical image information comprises a distance image (34) of the environment (22), determine a second piece of optical image information of the environment (22) as a function of received light (L), wherein the second piece of optical image information comprises a two-dimensional image of the environment, to determine a contour of at least one object (0) in the first and second image information, and to determine at least one region (44, 46) using a comparison of the contour in the first and second image information.

14. The optical detection system according to claim 13, wherein the optical receiving device is configured to receive the light (L) for determining the second image information.

15. Optical detection system according to claim 13 or 14, wherein the optical transmitting device is arranged to emit the light (L) for determining the second image information.

16. Vehicle (20) with an optical detection system according to one of claims 13 to 15.

17. A computer program product which includes instructions which, when executed by a computing device (18), cause the computing device (18) to perform a method according to any one of claims 1 to 11.

Images