METHOD AND DEVICE FOR THE SITE-RESOTATED DETECTION OF AN EXTERNAL OBJECT USING A SENSOR INSTALLED IN A VEHICLE

Description

[0001] The present invention relates to a method for the spatially resolved detection of an object external to a vehicle using a sensor installed in a vehicle. The present invention further relates to a device for carrying out the method and a vehicle that is configured to carry out such a method or that includes such a device.

[0002] A wide variety of driver assistance systems are now available to support drivers in motor vehicles. Some systems are limited to signaling or warning functions, while others intervene in the vehicle's drive system or steering. Both require knowledge of the current vehicle situation, which is obtained from data provided by suitable sensors. These sensors can detect technical parameters of the vehicle itself, such as those for an electronic stability program (ESP) or anti-lock braking system (ABS), as well as information about the vehicle's surroundings. Various technologies, often in combination, are used to monitor the vehicle's surroundings and to implement assistance functions based on this data, such as parking assistance, automatic distance warnings and control, traffic sign recognition, blind spot monitoring, and emergency braking systems for pedestrian protection.Ultrasound, radar, lidar, and camera systems are particularly common for identifying objects in the vehicle's vicinity, such as pedestrians, other vehicles, traffic signs, etc. Here, the sensor data from multiple sensors can first be evaluated individually, objects in the vehicle's environment can be identified within these individual sensor data, and then the information provided by the various sensors about the detected objects can be combined into a comprehensive environmental model.

[0003] The aggregation or linking of output data from multiple sensors, also known as sensor data fusion, is performed with the aim of improving the quality of the information obtained, whether to increase the detection range or to make the detection and, if applicable, classification of objects more accurate or reliable. For example, if the information obtained from one or more sensors is unreliable, this can be partially or completely compensated for using data from other sensors. Likewise, existing information can also be confirmed using data from additional sensors.

[0004] Another approach to optimizing object detection and modeling in the vehicle's environment involves defining one or more regions of interest (ROls) during the evaluation of sensor signals. These ROls are believed to contain relevant information. By selecting these ROls appropriately, false detections can be reduced, system robustness increased, and decision-making accelerated. For example, when analyzing camera image signals, the image data can first be searched for ROls, which are then subjected to more detailed image processing and object recognition. Tracking methods are a known technique for this, where detected objects are tracked over time using models to predict ROls for future points in time. However, to track a new object, an initial setup is required.

[0005] A method for creating an environment model for a vehicle is known from DE 10 2013 201 545 A1. In this method, sensor data from several sensors are evaluated to detect objects, and one or more areas of interest are also determined within the environment model and used for the detection of an object or a property of the object. DE 10 2014 109079 A1 describes a device and a method for detecting a driver's interest in a visual advertisement by tracking the driver's gaze direction, so that an audio advertising message can be played for the driver, which is generally associated with the visual display.

[0006] It is an object of the invention to provide an improved method for the spatially resolved detection of an object external to a vehicle using a sensor installed in a vehicle.

[0007] This problem is solved by a method having the features of claim 1, and by a corresponding device according to claim 7. Preferred embodiments of the invention are the subject of the dependent claims.

[0008] In the inventive method for spatially resolved detection of an object external to a vehicle using a sensor installed in a vehicle, the gaze direction of a vehicle occupant is recorded. The recorded gaze direction is analyzed over a period of time, and an area of ​​interest outside the vehicle is determined from this. The sensor is then spatially sensitized and / or aligned in the direction of the determined area of ​​interest.

[0009] This improves both the identification of areas of interest outside the vehicle and the initial tracking of a newly tracked object. Furthermore, when multiple objects are detected, it provides information about which object has priority for the vehicle occupant.

[0010] According to one embodiment of the invention, in addition to the vehicle occupant's gaze direction, the convergence of the vehicle occupant's eyes is also determined in order to ascertain the spatial distance of the area of ​​interest. This allows the sensor for detecting the object outside the vehicle to be sensitized or aligned even more precisely.

[0011] According to a further embodiment of the invention, a confidence value is determined for a spatial area, wherein the confidence value depends on the duration and / or frequency of the vehicle occupant's observation. The spatial area is defined as an area of ​​interest if the confidence value exceeds a threshold. Such a confidence value allows for particularly good quantification of the gaze direction and thus makes the determination of an area of ​​interest especially reliable.

[0012] Advantageously, the threshold depends on the physical or mental state of the vehicle occupant and / or external parameters such as road, traffic, or weather conditions. This allows the definition of an area of ​​interest to be adapted to changing parameters, thus further increasing reliability.

[0013] According to a further embodiment of the invention, the area of ​​interest or an object located within that area of ​​interest is indicated to the vehicle occupant by a display, for example, by a projected frame around the area of ​​interest or the object. In this way, it can be made clear to the vehicle occupant that the object in question has been detected and is being taken into account by the vehicle's sensors and, if applicable, an assistance system, thereby increasing confidence in the assistance system.

[0014] The method can be particularly advantageous when the driver's gaze direction is captured, as the driver is most attentive to the vehicle's surroundings in order to drive safely. Furthermore, the driver knows in which direction they will steer the vehicle and therefore also in which direction external objects are particularly relevant, because they could, for example, cross the future vehicle trajectory. Finally, camera systems that capture the driver's eyes may already be integrated into the vehicle to detect driver fatigue and react accordingly, and can therefore be used for the invention without the additional cost of a further camera system.

[0015] Accordingly, a device according to the invention for spatially resolved detection of an object external to a vehicle using a first sensor installed in a vehicle comprises: a second sensor for detecting the gaze direction of a vehicle occupant; an evaluation and control unit to which sensor data from the second sensor about the detected gaze direction is fed and which analyzes the sensor data over a period of time and from this determines an area of ​​interest outside the vehicle and controls the first sensor so that it is spatially sensitized and / or aligned in the direction of the determined area of ​​interest.

[0016] Preferably, the first sensor is a camera, radar, lidar or ultrasonic sensor directed towards the vehicle environment and the second sensor is a camera sensor directed towards the vehicle occupants inside the vehicle.

[0017] Advantageously, a display device, for example a head-up display, is arranged in the vehicle by means of which the area of ​​interest or an object that is located in the area of ​​interest is made known to the vehicle occupants.

[0018] Preferably, the method or device according to the invention is used in a motor vehicle.

[0019] Further features of the present invention will become apparent from the following description and the claims in conjunction with the figures. Fig. 1 schematically shows the inventive method for optimizing vehicle sensors based on the detected gaze direction of a vehicle occupant; Fig. 2 shows the application of the invention using the example of a traffic intersection where an object is not yet detected by the vehicle sensors but is already known to the system due to the driver's observation; and Fig. 3 shows the application of the invention using the example of a traffic situation with bad weather conditions; and Fig. 4 schematically shows a block diagram of a device according to the invention.

[0020] To better understand the principles of the present invention, embodiments of the invention are explained in more detail below with reference to the figures. It is understood that the invention is not limited to these embodiments and that the described features can also be combined or modified without departing from the scope of protection of the invention as defined in the claims.

[0021] Figure 1Figure 1 schematically illustrates the inventive method for the spatially resolved detection of an object outside the vehicle using a sensor installed in the vehicle. According to method step 1, the gaze direction of a vehicle occupant, in particular the driver, is detected. For this purpose, the eye position, the gaze direction of the vehicle occupant, and, if applicable, the convergence of the vehicle occupant's eyes are determined using an eye-tracking method, and a position outside the vehicle is derived from this. The detected gaze direction is analyzed over a certain period of time in method step 2. If it is found that the vehicle occupant continuously views a specific area outside the vehicle or an object, such as a car, over this period, the system detects the object's position.If a vehicle is following a vehicle ahead, this area is defined as the area of ​​interest (ROI) for the sensors, since an area to which the vehicle occupant is paying attention is highly likely to contain information that is also relevant for the sensors monitoring the surroundings. Subsequently, in process step 3, the sensor is spatially sensitized and / or aligned in the direction of the defined area of ​​interest (ROI).

[0022] The vehicle occupant can observe an object even before it enters the detection range of the existing sensors. This means that this area is already known as the area of ​​interest (ROI) for the sensors, so that an object in this area can be immediately detected by the sensors when it enters the detection range.

[0023] This is for illustrative purposes in Figure 2As shown, a vehicle 4 equipped with suitable sensors for detecting its surroundings is located at an intersection, with the detection area 5 of the sensors shown hatched. A passing vehicle 7 is located within this detection area and is therefore detected by the sensors of the vehicle 4. A cyclist 6, however, is not yet within the detection area. If the driver focuses their gaze on this cyclist, for example, to check whether the cyclist is crossing the road in front of the vehicle, the cyclist is detected according to the invention, and an area of ​​interest (ROI) in the vicinity of the cyclist is transmitted to the external sensors. Thus, the external sensors are already aware, before the actual detection, that a relevant object is located within the area of ​​interest (ROI).

[0024] Figure 3This illustrates another scenario where the recorded gaze direction of the vehicle occupant can be used to optimize the detection of the vehicle's surroundings. In this scenario, poor visibility leads to conflicting hypotheses about the presence of objects between the radar and camera sensor systems. While the radar sensor system detects vehicle 8, which is in front of the vehicle in the same lane, the camera sensor system only detects vehicle 7 to its left. For certain driver assistance systems, it may be necessary for the system to choose a hypothesis even though the interpretation of the surrounding observations based on the sensor data is ambiguous. This can be optimized by analyzing the driver's gaze direction.If the driver concentrates his gaze on the vehicle 8 driving in front of him for most of the time, the radar hypothesis is supported.

[0025] Analyzing the direction of gaze can also be advantageous for prioritizing multiple objects detected by the vehicle's sensors. If the detected objects are of equal relevance to the vehicle's sensors, but the driver only looks at one of these objects, or looks at it for significantly longer than the others, it can be concluded that this object is particularly relevant from the driver's perspective.

[0026] This can be particularly relevant when using sensors that cannot distinguish between different detected objects. For example, a radar or lidar system – unlike a camera with downstream image processing algorithms – cannot differentiate whether an object at the roadside is stationary, such as a traffic sign or a trash can, or whether it might be about to move into the road, such as a cyclist or a playing child.

[0027] Even when a camera is used to capture the vehicle's surroundings, analyzing the occupant's gaze can be advantageous. This is because image analysis, especially with high-resolution cameras and sufficient frame rates, is very complex and therefore extremely expensive in real time due to the resulting volume of video data. By analyzing the occupant's gaze, however, it is possible to limit the image analysis to pixels within an area of ​​interest corresponding to that gaze direction.

[0028] Figure 4Figure 1 schematically shows a block diagram of a device according to the invention. Several sensors 9, 10, 11 installed in a vehicle are provided for detecting the vehicle's surroundings; these include, for example, camera, radar, lidar, and / or ultrasonic sensors. The sensor data are fed to an evaluation and control unit 12, where the sensor data are evaluated in order to control a driver assistance system 13 accordingly. Additionally, another sensor 14 is provided in the vehicle, which detects the gaze direction of a vehicle occupant. In particular, the driver's head position can be detected by means of a camera arranged in the vehicle interior, and the driver's eye position and gaze direction can be determined using suitable image analysis methods. Preferably, this is a near-infrared (NIR) camera, which detects short-wave infrared radiation immediately adjacent to the visible spectrum.In this case, illumination, particularly in the dark, can be provided by one or more NIR LEDs (not shown) without disturbing or dazzling the driver. The NIR camera and the NIR LEDs can be installed separately or in a single sensor component. The detected gaze direction is analyzed in the evaluation and control unit 12 over a certain period of time. From this, an area of ​​interest outside the vehicle is determined, and based on this, at least one of the sensors 9, 10, or 11 is spatially sensitized and / or aligned towards this area of ​​interest.

[0029] If there is only one occupant in the vehicle, namely the driver, then it necessarily follows that only their gaze direction can be evaluated. However, even with multiple occupants, it is particularly advantageous to capture the driver's gaze direction, as they are most attentive to the vehicle's surroundings in order to drive safely. Furthermore, the driver knows in which direction they will steer the vehicle and therefore also in which direction external objects are particularly relevant, because they could, for example, cross the future vehicle trajectory. Finally, camera systems that capture the driver's eyes may already be installed in the vehicle to detect driver fatigue and react accordingly. However, the gaze direction of other vehicle occupants, especially the front passenger, can also be captured and used according to the invention.This can be useful, for example, if an external object is partially or completely obscured from the driver's position but is more visible to the passenger. For instance, the driver might be obscured by other external objects such as traffic signs or traffic lights, or even by parts of the vehicle itself, such as the A-pillar on the passenger side, while the passenger, due to their different seating position, would not be. Finally, the direction of a vehicle occupant's gaze can also be analyzed when the vehicle is in autonomous driving mode.

[0030] In addition to objects in areas in front of or beside the vehicle, objects located behind the vehicle can also be detected by spatially resolving and evaluating the driver's glances in the rearview or side mirrors.

[0031] The invention can be used in any area of ​​vehicle engineering. Reference symbol list

[0032] 1. Process step with gaze direction detection 2. Process step with gaze direction analysis and determination of an area of ​​interest 3. Process step with sensor adaptation to the determined area of ​​interest 4. Own vehicle 5. Sensor detection range 6, 7, 8. External objects 9, 10, 11. Sensors 12. Evaluation and control unit 13. Assistance system 14. Camera for monitoring the vehicle occupant ROI Area of ​​interest

Claims

1. Method for the spatially resolved detection of an object (6, 7, 8) external to the vehicle using a sensor (9, 10, 11) installed in a vehicle, in which - the viewing direction of a vehicle occupant is detected (1); - the detected viewing direction is analyzed (2) over a period of time, and from this a region of interest (ROI) outside the vehicle is determined (2); and - the sensor (9, 10, 11) is spatially aligned (3) in the direction of the specific region of interest (ROI), and - a confidence value is determined for a spatial region, wherein the confidence value depends on the duration and / or frequency of viewing by the vehicle occupant and the spatial region is determined as a region of interest (ROI) if the confidence value exceeds a threshold value, and wherein the threshold value depends on the physical or mental state of the vehicle occupant and / or vehicle-external parameters such as road, traffic or weather conditions.

2. Method according to claim 1, wherein the convergence of the eyes of the vehicle occupant is determined and the spatial distance of the region of interest (ROI) is determined from the convergence.

3. Method according to either of the preceding claims, wherein the region of interest (ROI) or an object (6, 8) located in the region of interest (ROI) is made known to the vehicle occupant by a display, for example by a projected frame around the region of interest or the object.

4. Method according to any of the preceding claims, wherein the viewing direction of the driver of the vehicle is detected.

5. Device for the spatially resolved detection of an object (6, 7, 8) external to the vehicle using a first sensor (9, 10, 11) installed in a vehicle, the device comprising - a second sensor (14) for detecting the viewing direction of a vehicle occupant; - an evaluation and control unit (10) to which sensor data from the second sensor (14) about the detected viewing direction are fed and which analyses the sensor data over a period of time, determines a region of interest (ROI) outside the vehicle from this, and controls the first sensor (9, 10, 11) such that it is spatially aligned (3) in the direction of the determined region of interest (ROI), wherein a confidence value is determined for a spatial region, wherein the confidence value depends on the duration and / or frequency of viewing by the vehicle occupant, and the spatial region is determined as a region of interest (ROI) if the confidence value exceeds a threshold value, and wherein the threshold value depends on the physical or mental state of the vehicle occupant and / or vehicle-external parameters such as road, traffic or weather conditions.

6. Device according to claim 5, wherein the first sensor (9, 10, 11) is a camera, radar, lidar or ultrasonic sensor directed toward the vehicle surroundings and the second sensor (14) is a camera sensor directed toward the vehicle occupant in the vehicle.

7. Device according to claim 5 or 6, wherein a display device, for example a head-up display, is arranged in the vehicle, by means of which the region of interest (ROI) or an object (6, 8) located in the region of interest (ROI) is made known to the vehicle occupant.

8. Motor vehicle which has a device according to any of claims 5 to 7 or is designed to execute a method according to any of claims 1 to 4.