Methods for optimizing the function of visual sensors
By adjusting camera sensitivity and exposure time in sync with headlight light distribution, the method enhances night-time object detection for autonomous vehicles, improving safety and reliability without perceptible light changes.
Patent Information
- Application Number
- DE102019001259
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2019-02-21
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2039-02-21
AI Technical Summary
Visual sensors, particularly cameras, struggle to effectively detect unlit objects at night due to insufficient lighting, necessitating adaptations in lighting and image processing that can be disruptive to human drivers.
Adjust the sensitivity and exposure time of visual sensors like cameras in coordination with the existing headlight system's light distribution, using feedback mechanisms to optimize detection without altering the light distribution perceptibly, and adapt image evaluation to enhance recognition quality.
Improves the detection of unlit objects at night by optimizing camera performance with minimal disruption to human drivers, enhancing driving safety and reliability.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] The invention relates to a method for optimising the function of visual sensors, such as cameras in particular, in an at least partially autonomously driving vehicle according to the type defined in more detail in the preamble of claim 1.
[0002] In the current state of the art, visual sensors typically function very well, particularly in sufficient lighting, and thus essentially during the day. However, when used in the dark, and thus essentially at night, their function can be further optimized. In the current state of the art, visual sensors, such as cameras in particular, detect objects in the dark in two ways. Based on the lighting, when the object itself is illuminated, for example based on the tail lights or headlights of vehicles, or based on the detection of objects in the recorded images, whereby patterns and objects must be recorded, trained, and classified at night to ensure sufficient quality of object detection.
[0003] It is now also known in the prior art to adapt a vehicle's lighting at night to the respective situation. For example, DE 10 2016 014 708 A1, filed by the applicant, describes how the headlights of a vehicle can be adjusted differently during manual driving, in which a person driving the vehicle assesses the surroundings, than when the vehicle is traveling autonomously and the surroundings are detected by a visual sensor, possibly in combination with other sensors. The document thus describes an adjustment of the light distribution and / or light intensity of the vehicle's headlight system in order to achieve better detection of the surroundings with the camera.Individual objects that have been detected, for example, by radar but not by the camera itself, can then be specifically illuminated, provided that suitable pixel headlights for precise illumination of the surroundings are present in the vehicle's headlight system, in order to be able to visually recognize these objects.
[0004] A further adaptation of the ambient lighting to the requirements of a camera, or an adaptation that facilitates the camera's work as a visual sensor, is described in DE 10 2015 008 774 A1. It describes how a pattern can be temporarily projected into the vehicle's surroundings to facilitate the camera's detection of three-dimensional objects at night, and in particular, the differentiation between three-dimensional objects and flat objects.
[0005] It is also known from the general state of the art that active glare control, or a variation of the light / dark boundary in a vehicle's headlights, can occur. The camera serves as a visual sensor to detect oncoming vehicles. According to DE 10 2016 013 510 A1, the area of the headlight system containing the oncoming vehicle is then masked out, so that the surroundings are brightly illuminated but the oncoming vehicle is not dazzled.
[0006] DE 10 2017 202 980 A1 describes a variable cut-off / dark threshold. The primary objective of the application is not to reduce glare, but rather to provide the driver with the least possible fatigue-free lighting.
[0007] DE 10 2015 216 907 A1 describes an adjustment of a visual sensor. This occurs depending on the engaged gear, in this case, reverse gear.
[0008] The object of the invention is to provide a method for further optimizing the function of visual sensors in an at least partially autonomously driving vehicle according to the preamble of claim 1. According to the invention, this object is achieved by a method having the features in claim 1, and in particular in the characterizing part of claim 1. Advantageous embodiments and further developments of the method according to the invention are set out in the dependent claims.
[0009] With the method according to the invention, the detection of unlit objects at night can be further improved. When the headlight system is activated, the visual sensors, in particular one or more cameras, are adjusted to the light distribution generated by the headlight system. Unlike the methods according to the prior art, such adjustment of the sensors themselves to the light distribution generated by the headlight system does not initially intervene in the light distribution of the headlight system. Rather, the visual sensor, and in particular the control parameters of a camera, which can be seen as the primary visual sensor, are adjusted so that they can achieve the best possible results with the existing light distribution of the headlight system.
[0010] According to a very advantageous development of the invention, the sensitivity and / or exposure time of at least one visual sensor is adapted to the light distribution generated by the headlight system. This behavior, which could also be referred to as a type of feedback, can then adjust the sensitivity and / or exposure time of the camera as a visual sensor accordingly, depending on the currently existing settings of the headlight system, such as low beam or high beam. With high beam, for example, significantly higher intensities occur on reflective surfaces than with low beam. With appropriate adjustment, the visual sensor or camera can now react in a timely manner to such blending areas, so that the quality of detection can be optimized, particularly in the vicinity of such areas.This allows for better recognition of the surroundings and ultimately improves driving safety. Since the light distribution itself remains unaffected, this approach is particularly suitable for vehicles that are not or only partially autonomous, as a person using the vehicle is not irritated by a change in the light distribution. This significantly improves the detection of objects in the dark without requiring any intervention in the light distribution.
[0011] According to a very advantageous development of the idea, the ambient brightness and / or glaring objects can be detected and taken into account when adjusting the visual sensor in order to further increase the quality of the adjustment.
[0012] An advantageous embodiment of the method according to the invention further provides that in the case of an at least briefly activated long-range light, also referred to as long-range light or URHB, or in the case of an at least briefly upwardly adjusted light / dark boundary of a light distribution, a visual sensor with a telephoto lens is activated or the lens of a visual sensor is switched to a telephoto mode. In this case, which, in contrast to the feedback described above, could also be referred to as feedforward coupling, the light is also changed accordingly.If, for example, there is a long-range light or the light / dark boundary is temporarily shifted in order to be able to see further with the visual sensor, then at least during this time a visual sensor with a telephoto lens is used if several visual sensors are present, or in the case of a visual sensor with an adjustable lens, this is switched to telephoto mode in order to be able to benefit maximally from the at least temporarily changed light distribution.
[0013] According to another very advantageous embodiment, it can also be provided that, in the event of a cornering light being activated at least briefly, a visual sensor with wide-angle mode is activated, or the lens of a visual sensor is switched to a wide-angle lens. Thus, if the cornering light is activated, at least briefly, this can be used to activate one of several visual sensors with a wide-angle lens, or, in the case of visual sensors with variable lenses, to switch this or at least one of these to wide-angle mode in order to benefit from the changed light distribution with regard to detection with the visual sensor.
[0014] Another extremely advantageous embodiment of the concept provides for the image analysis of the images captured by the visual sensors to be adapted to the light distribution. Image analysis can be optimized by adapting it to the area illuminated by the light distribution of the headlight system. This means that only the brightly illuminated areas of the captured images are evaluated accordingly, so that in the case of long-range illumination, only a relatively small section of the image needs to be evaluated, and only in the case of correspondingly broad illumination, for example, via a cornering light, does the entire area captured by a wide-angle lens need to be evaluated. This optimizes image analysis with regard to recognition quality.It also improves detection accuracy, as it eliminates the need to evaluate and assume potential objects in critical and / or poorly lit peripheral areas. This ultimately results in a more reliable picture of the camera's surroundings for the person using the vehicle, which increases confidence in the technology.
[0015] An exceptionally advantageous refinement of the method according to the invention now also provides for the light distribution of the headlight system to be adjusted, at least during autonomous driving in the dark. In addition to adjusting the camera, the light distribution of the headlight system can now also be adjusted. The steps must be carried out consecutively to avoid mutual interference, so that in each individual step, one partner and then the other sets the requirements, until the overall system comprising the headlight system and visual sensors is optimized for the current situation.
[0016] A very advantageous further development of this involves removing dimming curves from the light distribution of the headlight system. In practice, and in vehicles driven by a person, the headlight system uses dimming curves for each low and high beam distribution to achieve the most homogeneous light distribution possible within the legal limits and to suggest homogeneous illumination to the person driving the vehicle. This benefits the acceptance of the lighting on the one hand and the concentration and attention of the person driving the vehicle on the other. However, this is irrelevant for an autonomous vehicle and the evaluation of camera images.Adapting the headlights for autonomous driving can therefore do without dimming curves, resulting in a light distribution that appears inhomogeneous to a person sitting in the vehicle, but still illuminates the surroundings within legal regulations so that the illumination is optimal for object detection via a visual sensor. In this case, depending on whether the vehicle is being driven manually or autonomously, it is possible to switch back and forth between the supposedly inhomogeneous light distribution without dimming curves and a light distribution with dimming curves, which appears homogeneous to the person driving the vehicle. This makes it possible for the camera's detection quality to improve during autonomous driving, which is sensible in this situation and contributes to road safety.
[0017] Furthermore, with this embodiment of the invention, the intensity of the light distribution of the headlight system can be adjusted depending on the exposure time of the visual sensor by adjusting the light distribution. This has the advantage that individual areas are less overexposed, allowing the camera to evaluate dark areas much more effectively. This is a crucial advantage when the vehicle is driving purely autonomously, but should be avoided when driving manually, as the light switching could impact the person driving the vehicle by limiting their vision, which is ultimately determined by the adaptability of their eyes.
[0018] As already mentioned, cameras or camera systems with multiple cameras can be used as visual sensors in the vehicle. In principle, the method can be used in all vehicle environments, not just in the direction of travel in the so-called destination area, but it has its primary advantages in this destination area in particular.
[0019] Further advantageous embodiments of the method according to the invention will also become clear from the exemplary embodiment which is illustrated and described in more detail below with reference to the figure.
[0020] The only attached figure shows a schematic of a vehicle with a visual sensor and a headlight system.
[0021] The figure shows a vehicle 1. This comprises a headlight system designated 2, of which a headlight is shown as an example and provided with the reference numeral . The headlight system 2 can be designed in particular as a so-called pixel light system, for example with an LED array, or with a light source and an array of digital micromirrors, an LCD or LCOS shutter or the like. The headlight system 2 is then capable of being controlled with pixel precision in order to emit almost any light distribution 3, as indicated by the dashed lines, into the environment continuously or in rapid temporal alternation between different light distributions 3. In particular, the change can occur so quickly, as is also known from the prior art mentioned at the beginning, that it is imperceptible to a person driving the vehicle 1 due to the inertia of their eyes.
[0022] The vehicle 1 further has at least one visual sensor, which is indicated here in the form of a camera designated 4. The viewing angle 5 of the camera 4 overlaps at least partially with the light distribution 3 of the headlight system 2. Until now, in the dark and with the headlight system 2 activated, non-self-luminous objects can only be detected with difficulty and possibly inaccurately. This can be improved if the exposure time and / or sensitivity of the camera 4 is adjusted accordingly in order to obtain the highest possible contrast image even at night and with unlit objects in the vicinity of the vehicle 1. The adjustment can take additional environmental parameters, such as ambient brightness or glaring objects, into account and can react precisely to the current illumination situation using the light distribution 3 of the headlight system 2.
[0023] The sensitivity and / or exposure time of camera 4 can therefore be adjusted, in particular, depending on the current light setting of the headlight system 2, for example, depending on the low beam or high beam or even a so-called long-range beam (URHB). This is because, for example, with the high beam or long-range beam, a significantly higher intensity occurs on reflective surfaces than in the low beam mode. By appropriately adjusting camera 4, it is now possible to react in a timely manner to such blending areas and, particularly in the vicinity of such areas, an improved and higher-contrast image can be achieved with camera 4, which improves the detection of objects, in particular but not exclusively for autonomous driving of vehicle 1.Since in this variant, which can also be referred to as feedback, the light distribution 3 remains essentially constant, this variant is particularly suitable for optimizing the quality of the surrounding images captured by the camera 4 when the vehicle 1 is only partially driven autonomously. This can be the case, for example, when a person using the vehicle 1 is driving using a camera-based distance assistant.
[0024] In addition, feedforward coupling is also conceivable as a supplement to the adaptation of the camera parameters to the light distribution 3. Depending on the exposure time of the camera 4 that is set or determined based on the light distribution 3, the intensity of the low beam or high beam of the headlight system 2 is adjusted across the entire light distribution 3 or just across individual sections thereof if the headlight system 2 has pixel headlights and is capable of doing so. By making such an adjustment, for example by reducing the intensity of the light in individual areas, an image with fewer overexposed areas can be achieved, in which darker areas can be better evaluated by the camera.in which areas that are too dark can be better recognized by the camera, which may require further readjustment of the camera 4 and / or the light distribution 3 in order to further improve the quality of the captured images. Since in this case of so-called feedforward coupling, an active light switching of the light distribution 4 of the headlight system 2 must take place, this becomes visible to a person driving the vehicle 1 and may possibly impair this person or impair the person's visibility of the surroundings. The method is therefore particularly suitable for an autonomous driving mode of the vehicle 1 in the dark.
[0025] In particular, when using the pixel light systems just described in the headlight system 2, it is then possible, as also described in the aforementioned DE 10 2016 014 708 A1, to illuminate individual areas particularly intensively, for example if the camera 4 is to be used to validate objects which have already been detected, for example, by a radar sensor.
[0026] In a further embodiment of the method, any dimming curves are removed from the low beam and / or high beam distributions so that the best possible light distribution 3 is achieved within the legal limits. If no dimming curves are present, the light distribution 3 appears very inhomogeneous to a person driving the vehicle 1 due to visible edges and transitions. However, this is not relevant for an autonomously driving vehicle 1. It therefore makes sense, especially when the vehicle 1 is driven autonomously, to remove the dimming curves of the light distribution 3 in addition to adapting the camera 4 to the light distribution 3. In particular, it is possible to switch between the "homogeneous" and "inhomogeneous" light distribution 3 depending on whether the vehicle 1 is driven manually or autonomously.
[0027] When using camera systems, the method can also be used to utilize different cameras 4 of the camera system with different optical systems. Depending on the light distribution, different cameras 4 of the camera system can then be used. For example, if a sensor function system of an autonomously driving vehicle 1 requests validation of radar or lidar data, a camera 4 with a telephoto lens and a long-range light (URHB) can be activated simultaneously. This would make it possible, for example, if a radar has detected lost cargo on the road, to use camera 4 as a visual sensor to assess its condition and accurately classify it.To ensure the best possible detection of objects even at night, the long-range light or a spot illumination of the objects can be activated briefly, as is known from the state of the art mentioned above. Alternatively, the cut-off point of the low beam could be adjusted accordingly and shifted upwards for a short or longer period. A similar approach would, of course, also work if one of the lenses of camera 4 is variable. Instead of activating a telephoto lens, the telephoto mode would then be selected, for example, by adjusting the lens accordingly.
[0028] Furthermore, all of this is of course also possible when using a wide-angle lens or a camera with a wide-angle lens in the camera system. For optical validation of objects in the peripheral area, which can be reliably detected with such a wide-angle lens, the cornering lights of headlight system 2 could then be activated, for example.
[0029] The image analysis, which evaluates the data captured by camera 4, can always be adapted to the illumination area of the light distribution 3 of the headlight system 2. This means, for example, that when the long-range light is activated, a smaller image section is evaluated by the data from camera 4 than when the cornering lights are activated, as the opposite example during wide-angle detection.
[0030] This method can also be combined with the variant also described in the prior art mentioned at the beginning in the form of DE 10 2015 008 774 A1, in which a special light pattern is projected into the environment, which accordingly improves the recognition of objects, in particular the recognition of three-dimensional objects, by the camera 4.
Claims
[1] Method for optimising the function of visual sensors (4) in an at least partially autonomously driving vehicle (1), with at least one visual sensor (4) for detecting the surroundings of the vehicle (1) and a headlight system (2) for illuminating this surroundings, characterized by that when the headlight system (2) is activated, the at least one visual sensor (4) is adjusted to the light distribution (3) generated by the headlight system (2). [2] Method according to claim 1, characterized by that the sensitivity and / or exposure time of the at least one visual sensor (4) is adapted to the light distribution (3) generated by the headlight system (2). [3] Method according to claim 1 or 2, characterized by that the ambient brightness and / or glare objects are also detected and taken into account during the adjustment. [4] Method according to claim 1, 2 or 3, characterized bythat in the case of a long-range light being activated at least briefly or a light / dark boundary of the light distribution (3) being adjusted upwards at least briefly, a visual sensor (4) with a telephoto lens is activated or the lens of a visual sensor (4) is brought into a telephoto mode. [5] Method according to one of claims 1 to 4, characterized by that in the event of a cornering light being activated at least briefly, a visual sensor (4) with a wide-angle lens is activated or the lens of a visual sensor (4) is put into a wide-angle mode. [6] Method according to one of claims 1 to 5, characterized by that an image evaluation of the recorded images of the at least one visual sensor (4) is adapted to the light distribution (3). [7] Method according to one of claims 1 to 6, characterized bythat the light distribution (3) of the headlight system (2) is additionally changed, at least during autonomous driving in the dark. [8] Method according to claim 7, characterized by that any dimming curves of the light distribution (3) of the headlight system (2) are removed. [9] Method according to claim 7 or 8, characterized by that the intensity of the light distribution (3) of the headlight system (2) is adjusted depending on the exposure time of the at least one visual sensor (4). [10] Method according to one of claims 1 to 9, characterized by that cameras (4) are used as visual sensors.
Citation Information
Patent Citations
Method and device for detecting a vehicle environment
DE102015008774A1
digital rearview mirror
DE102015216907A1
headlight system for vehicles of all kinds, with at least one lamp and a camera for detecting the road
DE102016013510A1
Method for controlling the front light distribution of a vehicle
DE102016014708A1
Changing the brightness gradient of a headlight by dimming light points
DE102017202980A1
