Method and apparatus for determining the probability that an object is located in the field of vision of a driver of a vehicle.

Abstract

Method for determining the probability that an object (O) is located in the field of vision of a driver of a vehicle, wherein: - depending on provided raw data (R) of a three-dimensional scene representing a given environment inside and / or outside the vehicle, one or more objects (O) of the three-dimensional scene are identified; - depending on the provided raw data (R) of the three-dimensional scene, a two-dimensional image is generated such that the two-dimensional image maps the identified objects (O) of the three-dimensional scene onto a curved plane, wherein the two-dimensional image has a set of pixels (P) each representing at least a part of one or more of the identified objects (O) of the three-dimensional scene; - at least a part of the pixels (P) has a tuple of object identifiers.- the two-dimensional image is determined depending on an occlusion and / or translucency of at least one of the identified objects (O) of the three-dimensional scene; - data (FOV) are provided that represent at least one identified field of view of the driver, wherein the at least one identified field of view is determined depending on at least one predefined detection parameter, which is detected in the vehicle with a predefined detection sensor system; - for at least one of the identified objects (O), the probability with which the at least one identified object (O) is located in the driver's field of view is determined, depending on the provided data (FOV) and the two-dimensional image.

Description

[0001] The invention relates to a method for determining the probability that an object is located in the driver's field of vision. The invention further relates to a device for determining the probability that an object is located in the driver's field of vision. The invention further relates to a computer program and a computer program product for determining the probability that an object is located in the driver's field of vision.

[0002] While early driver assistance systems such as anti-lock braking systems (ABS) and electronic stability programs (ESP) were limited to directly supporting vehicle control, a multitude of driver assistance systems now exist that actively warn the driver of existing hazards. Thanks to improved environmental perception, these systems can detect and predict significantly more potential conflict situations and alert the driver to impending dangers at an early stage. However, an excess of warnings and alerts can lead to cognitive overload for the driver and even distract them from the actual driving task. Furthermore, it is likely that a large number of unnecessary warnings will disturb the driver or even make them feel insecure about their driving skills, leading them to deactivate the system.Therefore, it can be advantageous to deliberately refrain from issuing warnings if an adequate driver reaction can be expected even without them.

[0003] Modern driver assistance systems for semi-automated or highly automated driving therefore increasingly utilize an assessment of the driver's situational awareness.

[0004] Such an assessment of situational awareness is often based on an evaluation of where the driver is focusing their attention. Since the driver is surrounded by many visual stimuli while driving, it can be important, depending on the situation, to be able to assess whether a particular object is visible to the driver and / or whether the driver can perceive that particular object.

[0005] The object underlying the invention is to provide a method and a device that makes it possible to reliably determine the probability with which an object is located in the field of vision of a driver of a vehicle.

[0006] The problem is solved by the features of the independent patent claims. Advantageous embodiments are characterized in the dependent claims.

[0007] According to a first aspect, the invention is characterized by a method for determining the probability that an object is located in the driver's field of vision. According to a second aspect, the invention is characterized by a device for determining the probability that an object is located in the driver's field of vision.

[0008] The device is designed to carry out the process or a specific embodiment of the process.

[0009] In this process, one or more objects from a three-dimensional scene representing a predefined environment inside and / or outside the vehicle are identified based on provided raw data. A two-dimensional image is then generated based on this raw data, mapping the identified objects onto a curved plane. This two-dimensional image contains a set of pixels, each representing at least a portion of one or more of the identified objects. Furthermore, data representing at least one identified field of vision for the driver is provided. This field of vision is determined based on at least one predefined measurement parameter captured within the vehicle by a predefined sensor system.For at least one of the identified objects, the probability is determined with which the at least one object is located in the driver's field of vision, which can also be referred to as the driver's real or actual field of vision, depending on the data provided and the two-dimensional image.

[0010] Various methods can be used for rendering, such as raycasting and / or rasterization. Raycasting is a method that involves tracing possible lines of sight. Rasterization, also called scan conversion, refers to the conversion of a vector graphic into a raster graphic.

[0011] The objects in the three-dimensional scene include, for example, a traffic sign and / or a traffic light and / or a vehicle and / or a person and / or a control element in the vehicle, and so on.

[0012] The advantage of mapping or rendering is that even hidden objects can be taken into account.

[0013] Furthermore, the method according to the invention enables a probabilistic statement regarding the probability that at least one object lies within the driver's field of vision. Preferably, for all objects known to the vehicle, the probability with which each object is located within the driver's field of vision is determined.

[0014] The driver's field of vision, unlike the calculated driver's field of vision, corresponds to the driver's actual field of vision. The calculated field of vision may differ from the actual field of vision due to detection tolerances and / or model tolerances.

[0015] The visual field can comprise one or more visual areas. In particular, the visual field can include a central visual area and / or a peripheral visual area. In the central visual area, a central point on the retina is used to fixate on an object. In the peripheral visual area, extrafoveal areas are used for perception.

[0016] This allows, for example, more accurate statements to be made about which objects the driver is focusing on and / or where their visual attention lies. In particular, this ensures higher quality assessments of whether the driver is adequately attentive in a given driving situation.

[0017] In an advantageous embodiment according to the first and second aspects, the curved plane comprises or is at least part of the surface of a sphere, with a cyclopean eye of the driver forming the center of the sphere. Here, at least some of the detected objects of the three-dimensional scene are mapped or rendered onto the surface of the sphere. The diameter of the sphere is of secondary importance as long as the diameter is sufficiently small, i.e., the sphere is so small that it does not encompass the objects that are to be mapped onto the surface.

[0018] In a further advantageous embodiment according to the first and second aspects, an area is determined in the two-dimensional image that corresponds to the driver's field of vision, which has been determined at least once. Furthermore, depending on the object's position relative to the determined area, it is determined whether the object, which has been determined at least once, is at least partially located within the driver's field of vision, which has been determined at least once.

[0019] This advantageously allows for a simple assignment of the at least one object to the at least one determined field of vision of the driver. The at least one determined field of vision of the driver is stored, for example, in a bitmap. To assign the at least one object to the at least one determined field of vision of the driver, the two-dimensional image is overlaid accordingly on the bitmap.

[0020] In a further advantageous embodiment according to the first and second aspects, the data represent a first determined field of vision and at least one further determined field of vision of the driver. Here, the position of the first determined field of vision deviates from the position of the respective further determined field of vision within a predefined tolerance range. A weighting factor is assigned to each of the first determined field of vision and each of the respective further determined fields of vision, and the probability with which the at least one object is located in the driver's "actual" field of vision is determined depending on the respective weighting factors.

[0021] In particular, the first and at least one further field of view are determined in such a way that their respective positions are ascertained. The first determined field of view and the at least one further determined field of view may overlap.

[0022] The field of vision to be determined can be calculated based on the recorded head pose, gaze direction, and / or position of the driver. Due to specific anatomical and physiological characteristics, errors can occur when recording the head pose, gaze direction, and / or position of the driver.

[0023] For example, the first determined field of vision corresponds to the first recorded and / or determined direction of the driver's gaze, and at least one further determined field of vision corresponds to another recorded and / or determined direction of the driver's gaze. For example, a hypothesis is formulated as to which direction of gaze is most likely, and the weighting factor is assigned accordingly to each determined field of vision.

[0024] The weighting factors preferably indicate how frequently the first determined field of view corresponds to the driver's "real field of view," compared to the case where at least one other field of view corresponds to the driver's "real field of view." The first determined field of view and the at least one other determined field of view can each be stored in a bitmap. The probabilistic assignment can be performed by repeatedly overlaying the bitmaps with weighted values.

[0025] In a further advantageous embodiment according to the first and second aspects, at least some of the pixels have a tuple of object identifiers. In particular, transparent and / or opaque objects can be very easily represented in the two-dimensional image using these tuples.

[0026] In a further advantageous embodiment according to the first and second aspects, the set of pixels comprises initial pixels, each representing at least a subset of at least two of the identified objects in the three-dimensional scene. Each of these at least subset of identified objects in the three-dimensional scene can be assigned an object identifier. The initial pixels can each comprise a tuple with at least two object identifiers assigned to different objects.

[0027] In a further advantageous embodiment according to the first and second aspects, the two-dimensional image is determined depending on the occlusion and / or translucency of at least one of the identified objects in the three-dimensional scene. Occlusion and / or translucency, in particular the transparency of objects, can be taken into account. Specifically, a transparent windshield can be considered in each pixel description of the objects by assigning an object identifier to the windshield. An object visible through the windshield can thus be described by a pixel with at least two object identifiers. The tuple also allows for the simultaneous consideration of an order of the objects.

[0028] In a further advantageous embodiment according to the first and second aspects, the set of pixels includes second pixels that represent at least part of an object that is obscured by another object. Alternatively or additionally, the second pixels can represent at least part of a translucent or transparent object. The first pixels and the second pixels can be identical pixels.

[0029] In a further advantageous embodiment according to the first and second aspects, the visibility of at least one of the identified objects in the three-dimensional scene is determined depending on the probability with which that at least one of the identified objects is located within the driver's field of vision. It is assumed here that the visibility of objects decreases with increasing deviation from the driver's line of sight.

[0030] In a further advantageous embodiment according to the first and second aspects, the determined field of vision comprises several fields of vision, and the probability with which at least one object is located in a specific field of vision of the driver is determined. Dividing the determined field of vision into different fields of vision makes it possible to differentiate the visibility and / or the visual perception of the driver more precisely.

[0031] In a further advantageous embodiment according to the first and second aspects, the respective determined field of vision represents a determined central field of vision area, and the probability with which the at least one object is located in an "actual" central field of vision area of ​​the driver is determined. Alternatively, the respective determined field of vision represents a determined peripheral field of vision area, and the probability with which the at least one object is located in an "actual" peripheral field of vision area of ​​the driver is determined.

[0032] In a further advantageous embodiment according to the first and second aspects, a driver's perception probability is determined for at least one of the identified objects in the three-dimensional scene, depending on the probability that at least one of the identified objects is located within the driver's field of vision. Additionally, the perception probability can be determined depending on the object's visibility.

[0033] For example, if an eye movement of the driver is detected, the probability of the driver perceiving the object can be determined based on the eye movement, the corresponding measured field of vision, and the visibility of the object. This probability can be determined using a predefined model based on object-based visual attention. The theory of object-based attention assumes that attention is not directed toward an abstract location, but rather toward a specific object at a specific location, or that attention can only be directed toward one or a few objects at any given time.

[0034] The periphery is given preferential treatment over the central retina during information processing. For example, if an object or movement suddenly appears in the periphery, central perception is suppressed in favor of the information from the periphery, and attention is directed to the new detail. This involves a reflexive reorientation of the head and eyes (with the help of eye movements) towards the object to be analyzed. The superior spatial resolution of central vision can then be used to analyze the situation. Peripheral vision thus assumes a detection function.

[0035] In a further advantageous embodiment according to the first and second aspects, the determined probability and / or the determined visibility and / or the determined perception probability for a given driver assistance function are provided, and the driver assistance function is carried out depending on the determined probability and / or the determined visibility and / or the determined perception probability. This allows the probability and / or the determined visibility and / or the determined perception probability to be provided for any driver assistance function, such as driver assistance functions for partially automated driving, where the human driver monitors the driving environment but is supported in certain driving situations by one or more driver assistance systems.

[0036] According to a third aspect, the invention is characterized by a computer program, wherein the computer program is configured to carry out the method for determining a probability with which an object is located in a driver's field of vision of a vehicle, or an optional embodiment of the method.

[0037] According to a fourth aspect, the invention is characterized by a computer program product comprising an executable program code, wherein the program code, when executed by a data processing device, performs the method for determining a probability with which an object is located in a driver's field of vision of a vehicle, or an optional embodiment of the method.

[0038] The computer program product includes, in particular, a medium readable by the data processing device on which the program code is stored.

[0039] Exemplary embodiments of the invention are explained in more detail below with reference to the schematic drawings. These show: Fig. 1. A flowchart for determining the probability that an object is located in the field of vision of a driver of a vehicle. Fig. 2 an exemplary traffic scene, Fig. 3 an intermediate image and Fig. 4 a two-dimensional image.

[0040] The Fig. Figure 1 shows a flowchart of a program for determining the probability with which an object O is located within the driver's field of vision. The program can be used by a device to determine the probability with which an object is located. O The tasks that are located within the driver's field of vision of a vehicle will be processed.

[0041] The device comprises, in particular, a processing unit, a program and data memory, and, for example, one or more communication interfaces. The program and data memory and / or the processing unit and / or the communication interfaces can be integrated into a single unit and / or distributed across multiple units.

[0042] The program and data memory of the device contains, in particular, the program for determining the probability with which an object O stored within the driver's field of vision of a vehicle.

[0043] The program will be completed in one step S1 started, in which variables can be initialized if necessary.

[0044] In one step S3 will depend on the raw data provided Ra three-dimensional scene representing a predefined environment inside and / or outside the vehicle, one or more objects O the three-dimensional scene is determined. Preferably, all objects are included. O the three-dimensional scene is determined. Identifying the objects O It may, but does not have to, include object recognition, where, for example, one object is recognized as a vehicle and another object as a traffic light.

[0045] The raw data R The three-dimensional scene is determined and provided, for example, based on a captured three-dimensional image. The three-dimensional image is typically acquired by at least one image acquisition device, such as a camera, on the vehicle. The image acquisition device is preferably designed to capture three-dimensional image data that includes distance and / or depth information. The raw dataR This may include a virtual spatial model that at least represents an object O as well as defining the position and viewing direction of a viewer. Additionally, the virtual model can define material properties of the objects and light sources.

[0046] In one step S7 will depend on the raw data provided R a two-dimensional image is determined from the three-dimensional scene, such that the two-dimensional image represents the determined objects. O the three-dimensional scene is mapped onto the surface of a sphere, with the two-dimensional image having a number of pixels. P exhibits, each of which contains at least a part of one or more of the identified objects O to represent the three-dimensional scene. Preferably, a cyclopean eye of the driver forms the center of the sphere, on whose surface the detected objects are projected.

[0047] The number of pixelsP includes, for example, first pixels that each represent at least a part of at least two of the identified objects O The three-dimensional scene is represented. Preferably, the two-dimensional image depends on the occlusion and / or translucency of at least one of the identified objects. O the three-dimensional scene is determined. The image acquisition device is preferably located in an interior space of the vehicle. The objects O Therefore, objects outside the vehicle are only visible through the windshield. Depending on the area of ​​the windshield, it can exhibit varying degrees of translucency.

[0048] At least some of the pixels P Therefore, for example, a tuple has at least two object identifiers that distinguish the different objects. O are assigned.

[0049] In one step S9 data FOV , which represent at least one determined field of vision of the driver, are provided.

[0050] For this purpose, the driver's gaze direction is determined, for example, based on a measured gaze vector or nose vector, using an interior camera. Modern eye-tracking devices typically use a camera sensor that outputs a three-dimensional view of the user's gaze direction. Additionally, the user's head position can also be output.

[0051] Depending on the driver's gaze direction, the driver's field of vision can be determined. Additionally, a recorded head pose and / or head position of the driver can also be used for this purpose.

[0052] In one step S11 will be for at least one of the identified objects O the probability with which the at least one identified objectO located in the driver's "actual" field of vision, determined based on the data provided. FOV and the two-dimensional image.

[0053] To determine the probability, preferably an area is identified in the two-dimensional image that corresponds to the determined field of vision of the driver and depends on the position of the object. O With regard to the identified area, it is determined whether at least one of the identified objects is present. O is located within the driver's determined field of vision.

[0054] The data FOV , which represent at least one determined field of view, are stored, for example, in a bitmap. To the objects O To assign at least one determined field of view, for example the two-dimensional image and the bitmap are superimposed and the areas in the further bitmap that are controlled by the objects are determined. Obe superimposed.

[0055] Even with a highly precise camera sensor, a vector describing the driver's gaze direction cannot be precisely determined due to individual eye anatomy (for example, the position of the fovea). The eye geometry cannot be captured by the camera sensor. Therefore, if eye geometry is not taken into account, gaze direction measurement is inaccurate.

[0056] Gaze-tracking devices predominantly use head models. Such a head model is usually based on specific anatomical or physiological features such as the corners of the eyes and mouth, and the tip of the nose, the relationship of which to each other generally differs across various head types. Therefore, errors can occur when determining the gaze direction and / or head pose, which can impair the visual field to be determined.

[0057] Due to the possible detection and determination errors of the eye-tracking device, the position of the determined field of vision generally deviates from the position of the "real" field of vision or the "actual" field of vision of the driver.

[0058] Preferably, a first field of view and at least one further field of view are determined, which deviates from the first field of view within a predefined tolerance range. The first and the at least one further field of view are stored, for example, in a bitmap. A probabilistic assignment of the objects to the driver's "actual" field of view is carried out, for example, by repeatedly overlaying the bitmaps with weighted values.

[0059] For example, the first determined field of view has a weighting factor. 70 The further determined field of view, for example, has a weighting factor. 15This is because it is assumed that the recorded gaze direction, on the basis of which the further determined field of vision was calculated, is more likely to deviate from the driver's actual gaze direction than the first determined field of vision. If the object can only be assigned to one field of vision, for example, to at least one further determined field of vision, the probability that the object lies within the driver's actual field of vision is lower than if the object can be assigned to the first determined field of vision or even both fields of vision.

[0060] In an optional step S13 will visibility S and / or a probability of perception Z of at least one object O determined, depending on the determined probability.

[0061] In a further optional step S15 The determined probability and / or the determined visibility will beS and / or the determined probability of perception Z provided for a predefined driver assistance function.

[0062] In a further step S17 The program will end.

[0063] Fig. Figure 2 shows an exemplary traffic scene 1 The traffic scene 1 The vehicle's shape is captured, for example, using an image capture device, such as a 3D camera. Alternatively, a 2D camera can be used, and the depth information is captured using other sensor devices, such as a radar device and / or a lidar device.

[0064] The traffic scene 1 is captured, for example, by an image capture device located inside the vehicle. In the Fig. In example 2, the image capture device is aligned such that part of a dashboard22 and a hood 20 the vehicle is recognizable. The traffic scene 1 includes several objects O : an oncoming vehicle 24 , a vehicle ahead 26 , a crossing vehicle 28 , a traffic light 30 , several pylons 32 and a dog 34 .

[0065] Fig. 3 shows an intermediate image 5 , which contains the rendered two-dimensional image. The objects O the traffic scene 1 They are represented in a simplified way. For example, in the three-dimensional scene, the objects are depicted as cuboids. This simplified representation is then transformed into the two-dimensional image. The pixels P The two-dimensional image, for example, each contains tuples with two object identifiers. A large proportion of the pixels PThis also represents, for example, the windshield. Obscured objects are represented, for example, in the tuple of pixels. P not listed.

[0066] Fig. Figure 4 shows the rendered two-dimensional image with an assignment of the objects. O to the surface of the sphere. Fig. Figure 4 shows the rendered two-dimensional image with a superimposed field of view. The driver's gaze is directed, for example, at the vehicle ahead. The oncoming vehicle is predominantly within a field of view of 30° to 45°. The dog is located within an area of ​​approximately 60° and is therefore only perceptible through peripheral vision. This demonstrates that the driver's attention is not (yet) focused on the dog. Reference symbol list 1 traffic scene 5 Intermediate image 7 two-dimensional image 20 Front hood 22 Dashboard 24 oncoming vehicle 26 preceding vehicle 28 crossing vehicle 30 traffic light 32 pylons 34 dogs FOV data that is representative of a determined field of view O object P pixels R Raw data Visibility W probability Z Perception probability

Claims

[1] Method for determining the probability that an object (0) is located in the field of vision of a driver of a vehicle, in which - depending on the raw data (R) provided of a three-dimensional scene representing a predefined environment inside and / or outside the vehicle, one or more objects (O) of the three-dimensional scene are identified, - depending on the raw data (R) provided of the three-dimensional scene, a two-dimensional image is determined such that the two-dimensional image maps the determined objects (O) of the three-dimensional scene onto a curved plane, wherein the two-dimensional image has a set of pixels (P) that each represent at least a part of one or more of the determined objects (O) of the three-dimensional scene, - Data (FOV) are provided that represent at least one determined field of view of the driver, wherein the at least one determined field of view is determined depending on at least one predefined detection parameter, which is detected in the vehicle with a predefined detection sensor system, - for at least one of the identified objects (O), the probability with which the at least one identified object (O) is located in the driver's field of vision is determined, depending on the provided data (FOV) and the two-dimensional image. [2] Method according to claim 1, wherein the curved plane comprises at least a part of a surface of a sphere, wherein a cyclops eye of the driver forms a center of the sphere. [3] Method according to claim 1 or 2, wherein - an area is determined in the two-dimensional image that corresponds to at least one determined field of vision of the driver, and - depending on the position of the object (O) in relation to the determined area, it is determined whether the at least one determined object (O) is at least partially located in the at least one determined field of vision of the driver. [4] Method according to any of the foregoing claims, wherein - the data (FOV) represent a first determined field of view and at least one further determined field of view of the driver, wherein a position of the first determined field of view deviates from a position of the respective further determined field of view within a specified tolerance range, - a weighting factor is assigned to the first determined field of vision and to each subsequent determined field of vision, and the probability with which at least one object (O) is located in the driver's field of vision is determined depending on the respective weighting factors. [5] Method according to any of the preceding claims, wherein at least a part of the pixels (P) comprise a tuple of object identifiers. [6] Method according to one of the preceding claims, wherein the set of pixels (P) comprises first pixels which each represent at least a part of at least two of the identified objects (O) of the three-dimensional scene. [7] Method according to one of the preceding claims, wherein the two-dimensional image is determined depending on an occlusion and / or translucency of at least one of the identified objects (O) of the three-dimensional scene. [8] Method according to one of the preceding claims, wherein the visibility (S) of at least one of the identified objects (O) of the three-dimensional scene is determined depending on the probability with which the at least one of the identified objects (O) is located in the driver's field of vision. [9] Method according to one of the preceding claims, wherein the determined field of vision has several fields of vision and a probability is determined with which the at least one object (O) is located in a certain field of vision of the driver. [10] Method according to any of the preceding claims, wherein - the determined field of vision has a central field of vision area or represents the determined central field of vision area and the probability with which at least one object (O) is located in a central field of vision area of ​​the driver is determined or - the respective determined field of vision has a determined peripheral field of vision area or represents the determined peripheral field of vision area and the probability with which at least one object (O) is located in a peripheral field of vision area of ​​the driver is determined. [11] Method according to one of the preceding claims, wherein a perception probability (Z) of the driver is determined for at least one of the identified objects (O) of the three-dimensional scene, depending on the probability with which the at least one of the identified objects (O) is located in the driver's field of vision. [12] Method according to one of the preceding claims, wherein the determined probability and / or the determined visibility (S) and / or the determined perception probability (Z) is provided for a given driver assistance function and the driver assistance function is carried out depending on the determined probability and / or the determined visibility (S) and / or the determined perception probability (Z). [13] Device for determining the probability that an object (O) is in the field of vision of a driver of a vehicle, wherein the device is configured to carry out a method according to any one of claims 1 to 12. [14] Computer program for determining the probability that an object (O) is in the field of vision of a driver of a vehicle, wherein the computer program is configured to carry out a method according to any one of claims 1 to 12 when executed on a data processing device. [15] Computer program product comprising executable program code, wherein the program code, when executed by a data processing device, performs the method according to any one of claims 1 to 12.

Images