Device for determining visibility in a motor vehicle, assistance system, motor vehicle and method for determining visibility in a motor vehicle

Abstract

A device for determining visibility in a motor vehicle is described, comprising a first camera, an image recognition module which detects objects in the data of a first camera sensor, a second camera which is sensitive in a frequency range different from that of the first camera, an image recognition module which detects objects in the data of a second camera sensor, the second camera which generates distance information, and a computing unit which is configured to determine visibility based on the objects detected by the first camera and the objects detected by the second camera and the distance information.

Description

[0001] The text describes a device for determining visibility in a motor vehicle, an assistance system, a motor vehicle, and a method for determining visibility in a motor vehicle.

[0002] Devices for determining visibility in a motor vehicle, assistance systems, motor vehicles and methods for determining visibility in a motor vehicle of the type mentioned above are known in the prior art.

[0003] Autonomous vehicles must be able to handle dynamic traffic situations, such as suddenly appearing obstacles, unpredictable movements of other road users, and changing traffic signs. This requires not only precise perception of the current environment but also the ability to predict future developments and react accordingly. Furthermore, autonomous systems must be able to function reliably at night or in tunnels, which places additional demands on sensors and data processing. These capabilities are crucial for ensuring the safety of occupants and other road users while simultaneously optimizing traffic flow efficiency.

[0004] Autonomous vehicles use a variety of sensors, including cameras, lidar, radar, and ultrasonic sensors, to perceive their surroundings. Integrating and processing the data from these diverse sources, known as sensor fusion, is complex and requires powerful algorithms and computing resources. Each sensor type has its own strengths and weaknesses that must be considered. For example, cameras often offer high resolution but are susceptible to poor lighting conditions, while lidar provides precise distance data but can be affected by rain or fog.

[0005] Varying weather conditions such as rain, snow, fog, and strong sunlight can significantly impair sensor performance. These conditions pose a particular challenge because they can reduce visibility and decrease the accuracy of object detection and tracking.

[0006] Sight-range determination in motor vehicles is therefore one of the central challenges for the development and implementation of autonomous driving systems. It is a complex and multidisciplinary problem encompassing technological, environmental, dynamic, and regulatory challenges. The development of robust and reliable sight-range determination systems is crucial for the safe and efficient introduction of autonomous vehicles into road traffic.

[0007] From DE 10 2019 134 539 A1, a device for determining the viewing range of a vehicle's camera is known, wherein the device is configured to capture images of the vehicle's surroundings, the vehicle comprising a lighting element, in particular a headlight, which is designed to illuminate at least a portion of the surroundings captured by the camera. The device is configured to determine illumination information for a portion of the camera's image that is illuminated by the lighting element. Furthermore, the device is configured to determine an estimated value of the camera's viewing range based on the illumination information.

[0008] German patent DE 199 28 915 A1 discloses a method for precisely determining the visibility range within a vehicle's field of vision, in order to encourage the driver to adjust their driving style. According to one initial proposal, a monocular video sensor measures the contrast of an object detected by a radar or lidar sensor, and the visibility range is then determined from the measurements of both sensors. Alternatively, a binocular video sensor measures the distance and contrast of an object. The visibility range is then calculated from these measurements.

[0009] The method proposed in DE 10 2012 215 465 A1 serves to filter object information by combining information from two sensors with different operating principles. The first sensor detects at least one object, while the second sensor detects at least two objects, with at least one of the objects detected by the second sensor also being detected by the first sensor. The aim is to output filtered object information that represents only the objects detected exclusively by the second sensor.

[0010] Currently, no passive camera systems are known that are capable of reliably determining visibility ranges.

[0011] Known methods are often not precise enough for modern assistance systems, especially those for semi-autonomous or autonomous driving.

[0012] The task therefore is to further develop devices for determining visibility in a motor vehicle, assistance systems, motor vehicles and methods for determining visibility in a motor vehicle of the type mentioned above in such a way that a more precise measurement of visibility in a motor vehicle is possible.

[0013] The problem is solved by a device for determining a visibility range in a motor vehicle according to claim 1, an assistance system according to dependent claim 3, a motor vehicle according to dependent claim 4, and a method for determining a visibility range in a motor vehicle according to dependent claim 5. Further embodiments and developments are the subject of the dependent claims.

[0014] A device for determining the viewing range of a first camera in a motor vehicle is described, wherein the first camera has a first camera sensor sensitive in a first frequency range, wherein the first camera captures a first image area, wherein an image recognition module is provided that receives data from the first camera, wherein the image recognition module is configured to recognize objects in the data of the first camera sensor, wherein a second camera is provided that has a second camera sensor sensitive in a second frequency range, wherein the first frequency range and the second frequency range are different, wherein the second camera captures a second image area, wherein the second image area at least partially overlaps with the first image area, wherein an image recognition module is provided that receives data from the second camera, wherein the image recognition module is configured toto detect objects in the data of the second camera sensor, wherein the second camera generates distance information, wherein a computing unit is provided which is configured to determine a viewing range of the first camera on the basis of the objects detected by the first camera and the objects detected by the second camera and the distance information.

[0015] The first and second cameras can be oriented forward in the direction of travel. They can have essentially the same focal length relative to the sensor size, or capture essentially the same field of view. The first and second cameras can be configured to record moving images or a series of still images.

[0016] The alignment of the first camera and second camera can be precisely defined or calibrated.

[0017] The first camera can be a driving camera used to monitor the area in front of the vehicle, e.g. for traffic sign recognition, traffic detection, autonomous driving and the like, whereby the camera's sensor signal is additionally used as described below.

[0018] The first camera sensor can be, for example, a CCD or a CMOS sensor and be sensitive in a frequency range that roughly corresponds to that of visible light, i.e., approximately 400-700 nm. The first camera sensor can be a color sensor, for example, by placing a Bayer filter in front of the photosensitive layer, or a monochrome sensor.

[0019] The second camera can use the same or a different sensor principle and can be positioned adjacent to the first camera. The second camera sensor can be a color sensor or a monochrome sensor.

[0020] The sensitive frequency range of the second camera differs and extends beyond the frequency range of the first camera at least on one side of the spectrum. In one embodiment, the second camera sensor can exhibit greater sensitivity in the ultraviolet region of the spectrum, and in an alternative embodiment, in the infrared region. In a third embodiment, the second camera sensor can be more sensitive than the first camera sensor in both directions.

[0021] The distance information from the second camera, especially regarding the objects it detects, can be obtained in various ways.

[0022] One technical method for this is phase shift measurement. A continuous, modulated light beam is emitted. The phase shift of the reflected light compared to the emitted light is measured. This phase shift is distance-dependent and can be used to calculate the distance.

[0023] Another possibility is structured lighting. Here, a known pattern, such as a grid or stripes, is projected onto the scene, preferably using infrared light. The distortion of this pattern by the objects in the scene is detected and analyzed by a camera to calculate the distance.

[0024] Another option is the use of a rangefinder system.

[0025] Furthermore, distances can be determined by evaluating different intensity and / or contrast values ​​of images.

[0026] In various specific embodiments, each camera may have its own image recognition module, or a single, shared image recognition module may be used to evaluate the data from both the first and second cameras. The image recognition modules may be implemented in hardware and / or software.

[0027] Depending on the distance measurement principle used, the second camera can be configured to directly measure distances to the detected objects or to estimate distances. In some configurations, it may be possible to precisely specify the distance to an object, while in others, the object can be categorized into certain distance ranges, for example, into five-meter-deep segments.

[0028] By comparing the data from the first and second cameras, and especially by comparing the objects detected by each camera, it's possible to identify objects that were missed by the first camera but still detected by the second. In other words, the first camera's range is limited when an object is no longer visible to it. The second camera provides the corresponding range value for the first camera by measuring the distance to the object at that point. The range of the first camera, which is used for object detection and classification, is particularly important because it affects the availability and settings of driver assistance systems. For example, in automated driving, the speed is significantly dependent on the first camera's range.

[0029] Since distance ranges or distances are stored for the respective objects based on the information from the second camera, it is possible to precisely determine how far the field of view is, which is determined by the viewing distance of the first camera.

[0030] For distance measurement, the position of the second camera relative to the vehicle can be taken into account and the corresponding measured values ​​can be compensated so that they are detected from a front of the vehicle.

[0031] The motor vehicle can be a land vehicle, a water vehicle, or an aircraft, for example a car, a truck, a train, a ship or boat, or an airplane.

[0032] In a first further embodiment, it is provided that the second frequency range of the second camera extends into or covers the near-infrared range.

[0033] The near-infrared range is well suited because the corresponding light is scattered to a lesser extent by water droplets, which are often responsible for limiting visibility, for example in the form of rain or fog.

[0034] This means that, at least under limited visibility conditions, the second camera can look deeper into the room than the first camera and thus detect objects that the first camera can no longer perceive due to its more limited range of vision.

[0035] In a further, more advanced embodiment, a lighting device for emitting light pulses in the near-infrared range is provided.

[0036] This enables a highly precise measurement method that allows time-of-flight (ToF) measurement. The light pulse is emitted by an illumination device connected to the second camera via an evaluation unit; preferably, the illumination device is integrated into the second camera, which is configured as a ToF camera. A light pulse is emitted, and the evaluation unit measures the time it takes for the light to travel to the object and back to the camera. This time, known as the time of flight, is directly proportional to the object's distance. ToF cameras typically illuminate the scene with a light pulse and measure the return time for each pixel. This method allows for the capture of an entire scene in three dimensions and is particularly useful for applications requiring fast and precise distance measurement.

[0037] Near-infrared light also penetrates water droplets and is therefore reflected by objects located beyond the range of vision, and can be captured and evaluated by the second camera.

[0038] In a further, more advanced embodiment, the second camera is intended to be part of a range-gated camera system.

[0039] A range-gated camera system is a complex system for precise distance measurement, consisting of several essential components.

[0040] A laser source that emits short, intense light pulses can be used as a light source. These pulses have a defined duration and wavelength, optimized for the specific application.

[0041] The second camera is equipped with a fast shutter that can open and close in nanoseconds and operates synchronously with the laser source. This synchronization can be handled by a control unit, which ensures that the camera shutter opens at precisely the right moment to capture the reflected light from specific distances.

[0042] The system can be supplemented by optical components such as lenses and filters that focus the light and filter out unwanted wavelengths or scattered light.

[0043] A recording with a range-gated camera system begins with the emission of a laser pulse. This pulse propagates at the speed of light and encounters various objects in the environment. The light is reflected by these objects and returns to the camera. During this time, the camera shutter remains closed to prevent stray light and reflections from atmospheric particles. The control unit opens the camera shutter precisely at the moment the reflected light returns from the desired distance.

[0044] This allows the camera to capture and process only light from that specific distance. The camera uses a pulsed laser for illumination and a shutter in the image sensor that opens and closes synchronously with the light pulses. By changing the laser's pulse frequency and the camera's opening and closing times, which are adjusted accordingly, objects at different distances can be detected.

[0045] The camera captures the reflected light and creates an image containing distance information. This data is then processed by the control unit to calculate the precise distance. Distance measurement is achieved by calculating the time the laser pulse takes to travel from the transmitter to the object and back to the camera. This time is called the time-of-flight (ToF).

[0046] Through precise control of the camera shutter and synchronization with the laser pulse, the range-gated camera system can perform accurate distance measurements, even under challenging conditions such as fog, rain, or darkness. This technology thus offers a reliable solution for applications requiring precise and rapid distance measurement and represents a significant advancement in sensor technology.

[0047] A first independent item concerns an assistance system with a device of the type described above.

[0048] The assistance system in question could be, for example, a semi-autonomous or autonomous driving system of the vehicle, which is based at least partially on information from camera systems, including, for example, the front camera. This means that visibility is a crucial criterion for the current configuration of the assistance system. For instance, the maximum possible speed that the assistance system can achieve may depend on the visibility range.

[0049] Even less complex assistance systems can be controlled by information about the current visibility range, for example the activation of fog lights and / or rear fog lights.

[0050] Another independent item concerns a motor vehicle with an assistance system and / or a device of the type described above.

[0051] Another independent subject matter relates to a method for determining the visibility in a motor vehicle with a device of the type described above, wherein at least one first object is detected with the first camera, wherein the at least one first object and a first distance information to the first object are detected with the second camera, and at least one second object is detected which is not detected by the first camera, wherein at least a second distance information to the at least one second object is detected, and wherein the visibility is determined on the basis of the first distance information and the second distance information.

[0052] The viewing range can be conservatively determined, for example, by the furthest object detected by the first and second cameras.

[0053] Alternatively, the line of sight can also be determined by the furthest object detected by both the first and second cameras and the nearest object detected solely by the second camera. The average of these two distances can then be calculated. For example, if the furthest object detected jointly is 48 meters away and the nearest object detected only by the second camera is 62 meters away, then the line of sight can be set at 55 meters.

[0054] According to a further embodiment, the viewing distance can be determined by the nearest object that was identified exclusively by the second camera, i.e., not identified by the first camera.

[0055] If visibility exceeds a certain threshold, for example 200 meters, the visibility can be set to unlimited. This threshold may depend on the speed or the maximum permitted speed on a section of the route.

[0056] In another further embodiment, it is provided that the first camera detects several objects, the second camera detects these objects and records distance information to the objects detected by the first camera and the second camera, the viewing range being determined based on the distance information of the furthest object detected by the first camera and the second camera and the second distance information.

[0057] In the event that the first and second cameras detect the same set of objects, meaning they appear to have the same field of view, a fallback option in further training is to select the distance to the furthest object. This situation can occur, for example, if there are no other detectable objects beyond those detected together, depending on factors such as the surroundings or road layout.

[0058] In a further, more advanced embodiment, it is planned that the data from the first camera and the data from the second camera will be transferred into a common coordinate system.

[0059] In this way, parallax errors between the two cameras can be reduced and the assignment of jointly detected objects simplified, especially if the first camera and second camera have different resolutions and / or viewing angles.

[0060] In a further, more advanced embodiment, it is provided that the specific visibility range is transmitted to an assistance system, whereby the assistance system adjusts driving parameters to the visibility range.

[0061] One such driving parameter could be, for example, speed.

[0062] Another independent subject matter relates to a computer program product, comprising a computer-readable storage medium on which instructions are embedded which, when executed by at least one computing unit, cause that at least one computing unit to be equipped to execute the procedure of the aforementioned type.

[0063] The process can be executed on one or more computing units, so that certain process steps are executed on one computing unit and other process steps on at least one other computing unit, whereby calculated data can be transmitted between the computing units if necessary.

[0064] Further advantages, features, and details will become apparent from the following description, in which – possibly with reference to the drawing – at least one embodiment is described in detail. Identical, similar, and / or functionally equivalent parts are marked with the same reference numerals.

[0065] They show schematically: Fig. 1A, Fig. 1B a side view of a motor vehicle in a foggy environment; Fig. 2 a top view of a section of the motor vehicle Fig. 1; Fig. 3 a schematic representation of an assistance system, as well as Fig. 4 a flowchart of a procedure for determining visibility.

[0066] Fig. Figure 1A shows a side view of a motor vehicle 2 in a foggy environment.

[0067] The vehicle 2 is equipped with an assistance system 4 that allows autonomous driving at a higher SAE level, for example level 4.

[0068] The assistance system 4 features a camera system 6 with a first camera 6.1 and a [missing information] in Fig. 1A second camera not shown, which captures and records an area in front of the motor vehicle 2 in the direction of travel.

[0069] The first camera 6.1 detects not only the road 8 and the road markings on it, but also objects such as a motorcyclist 10 and a tree 12. Due to the fog, the visibility is limited to a visibility range of 14, so that a person 16 and a tree 18, which are further away than the visibility range of 14, are no longer detected by the first camera 6.1.

[0070] The first camera, 6.1, is a CMOS camera with a Bayer filter and is suitable for capturing color images and videos. The first camera, 6.1, has high resolution.

[0071] Data from the first camera 6.1 can also be used for other purposes, such as traffic sign recognition, hazard detection, parking assistance and / or for an environment camera.

[0072] The assistance system 4 may contain additional components related to Fig. 3 are explained in more detail.

[0073] Fig. 1B shows the same situation as Fig. 1A, however, instead of the first camera 6.1, a second camera 6.2 is shown.

[0074] The second camera 6.2 is part of a range-gated camera system 20, which includes, among other things, an illumination device 22 with a laser light operating in the near-infrared range at approximately 850 nm. The sensitivity of the second camera 6.2 is greater in the infrared direction than that of the first camera 6.1, so that the second camera 6.2 is also sensitive to light rays in this infrared frequency range.

[0075] Since light in the near-infrared range is not scattered as strongly by water droplets as visible light, the viewing range of the second camera 6.2 in conjunction with the lighting device 22 is greater than that of the first camera 6.1, so that the person 16 and the tree 18 are clearly recognizable by the second camera 6.2 of the range-gated camera system 20.

[0076] Furthermore, by means of the setup of the range-gated camera system 20, it is possible to measure the distance of the objects 10, 12, 16 and 18 (d10, d12, d16 and d18) by measuring the time of flight and to link it to the objects 10, 12, 16 and 18.

[0077] By comparing the objects 10, 12, which were detected using the first camera 6.1, and the objects 10, 12, 16, 18, which were detected using the second camera 6.2, it is possible to identify those objects that are only detected by the second camera 6.2, here the person 16 and the tree 18.

[0078] Using the corresponding distance information d16, d18, the visibility range 14 can thus be precisely estimated. The tree 12 is the furthest object detected by the first camera 6.1 and the second camera 6.2, whereas the person 16 is the first object detected exclusively by the second camera 6.2. Therefore, the visibility range 14 can be determined to be the distance of the first person 16 from the vehicle 2, in this case d16.

[0079] Fig. Figure 2 shows a top view of a section of the motor vehicle 2, wherein the first camera 6.1 and the second camera 6.2 of the camera system 6 are arranged adjacent to each other on a windshield of the motor vehicle 2.

[0080] The first camera 6.1 has a first image area 24, whereas the second camera 6.2 has a second image area 26. Image areas 24 and 26 largely overlap.

[0081] This ensures that the first camera 6.1 and the second camera 6.2 can capture essentially the same objects. Non-overlapping image areas can be excluded from the analysis.

[0082] The relative positions and orientations of the first camera 6.1 and the second camera 6.2 can be calibrated, for example, during installation or maintenance. Using advanced algorithms, it is also possible for the corresponding cameras 6.1 and 6.2 to calibrate themselves or recalibrate as needed. Such calibration can be performed at regular intervals or on demand.

[0083] The corresponding image data from the two cameras 6.1 and 6.2 can be transferred into a common coordinate system in order to better align object recognition and the assignment of the two data streams.

[0084] Fig. Figure 3 shows a schematic representation of the assistance system 4.

[0085] The assistance system 4 includes the camera system 6 with a first camera 6.1 and the range-gated camera system 20 with a second camera 6.2 and lighting device 22. The first camera 6.1 has a first camera sensor 28, and the second camera 6.2 has a second camera sensor 30. The first camera 6.1 and the second camera 6.2 can have different sampling rates.

[0086] The corresponding data streams from the first camera 6.1 and the second camera 6.2 are transmitted to a first image recognition module 32 and a second image recognition module 34, respectively, and evaluated there. The image recognition modules 32 and 34 are configured to identify objects in the data streams. For this purpose, algorithms trained using artificial intelligence can be employed. The second image recognition module 34 is also capable of evaluating the distance information determined from the range-gated camera system 20 and assigning it to the corresponding detected objects.

[0087] The corresponding data on the evaluated identified objects from the first image recognition module 32 and the second image recognition module 34 are fed to a computing unit 36, which derives the visibility range 14 from the data as described above.

[0088] The visibility range 14 thus determined is fed to an assistance system control unit 38, which in turn controls at least one actuator 40 accordingly.

[0089] A concrete example could be that the assistance system 4 is a system for autonomous driving, wherein the assistance system control unit 38 is configured to move the motor vehicle by controlling appropriate components such as steering, power, brakes, and the like. A corresponding actuator 40 could, for example, be a brake.

[0090] Fig. Figure 4 shows a flowchart of the procedure.

[0091] Image data from the first camera 6.1 and image data from the second camera 6.2 are evaluated in parallel, with object recognition taking place in the first camera 6.1 and both object recognition and distance determination taking place in the second camera 6.2.

[0092] The objects detected by the first camera 6.1 and the second camera 6.2 are compared in a common coordinate system and the nearest object detected only by the second camera 6.2 is determined.

[0093] The visibility range 14 is then set to the distance of the object in question and transmitted to the assistance system 4.

[0094] The process takes place continuously at a predetermined frequency, so that the visibility range of 14 is always up-to-date.

[0095] Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims. Reference symbol list 2 motor vehicles 4 Assistance systems 6-camera system 6.1 first camera 6.2 second camera 8th Street 10 motorcyclists 12 Tree 14 Visibility 16 people 18 Tree 20 Range-Gated Camera System 22 Lighting equipment 24 first image area 26 second image area 28 first camera sensor 30 second camera sensor 32 first image recognition module 34 second image recognition module 36 computing units 38 Assistance system control unit 40 actuator d10 - d18 distance information

Images