Methods for operating a vehicle assistance system and assistance system

Abstract

Method for operating an assistance system (10) for a vehicle, comprising an object sensor device and a gaze sensor device, wherein in the method: - an object characteristic value is determined by the object sensor device, which is representative of the coordinates of an object (20) in the vicinity of the vehicle, wherein the vicinity denotes a three-dimensional space external to the vehicle; - a direction characteristic value is determined by the gaze sensor device, which is representative of a gaze direction (2) of a driver of the vehicle, and a position characteristic value is determined by the gaze sensor device, which is representative of a head position (3) of the driver; - depending on the position characteristic value and the object characteristic value, a projection characteristic value is determined, which is representative of a vector (4) that connects the head position (3) with the object (20); - depending on the direction and position characteristic value, a first area characteristic value is determined.which is representative of an estimated primary field of vision (B1) of the driver, - depending on the direction and position parameter, a second area parameter is determined which is representative of an estimated peripheral field of vision of the driver, and - depending on the first area parameter, the second area parameter and the projection parameter, an attention parameter is determined which is representative of a probability that the object (20) is at least partially in the driver's field of vision, and - depending on the attention parameter, a warning function of the vehicle is activated.

Description

The invention relates to a method for operating an assistance system for a vehicle as well as a corresponding assistance system, computer program and computer program product. Modern vehicles can include sensor units such as a camera sensor that detects a user's head pose, gaze direction, and / or position, for example, based on facial features. This data can then be used to determine, for instance, whether the driver is fatigued. DE 10 2015 206 200 A1 describes a method for determining the attention of a driver which includes the detection of a potentially dangerous situation by a system of the vehicle; the estimation of the driver's visual field of vision; and the determination of the driver's attention by comparing an orientation or position of the potentially dangerous situation with the driver's visual field of vision. DE 10 2013 217 405 A1 describes methods and systems for detecting whether a driver of a vehicle has detected an object outside the vehicle. The method comprises: receiving external sensor data representing a scene outside the vehicle; receiving internal sensor data representing an image of the driver; determining whether the driver has detected the object based on the external and internal sensor data; and optionally generating a control signal based on whether the driver has detected the object. DE 10 2012 022 691 A1 describes a head-up display and a method for controlling such a head-up display in a motor vehicle. The head-up display is integrated into a windshield of the motor vehicle. An object detection device in the motor vehicle can detect objects in the vehicle's surroundings. DE 10 2014 214 088 A1 describes a driver information system in a vehicle comprising a gaze direction detection unit and a display unit for indicating a critical event if the deviation between the driver's gaze direction and the critical event exceeds a limit value. DE 10 2009 002 979 A1 describes a projection display device for vehicles, comprising: optical projection means for displaying projection data, in particular image data, numbers, characters, symbols, information, or the like; means for detecting the driver's gaze direction; sensor means for detecting the environment outside the vehicle and for detecting sensor values; and a control unit, which is connected to the sensor means and to the means for detecting the gaze direction and is configured for processing, generating an image display, and forwarding the image display to the optical projection means. The object underlying the invention is to create a method for operating a vehicle assistance system and a corresponding assistance system that helps to precisely determine the driver's attention. The problem is solved by the independent patent claims. Advantageous embodiments are characterized in the dependent claims. According to a first aspect, the invention relates to a method according to claim 1 for operating an assistance system for a vehicle. The assistance system comprises an object sensor device for determining an object characteristic value that is representative of the coordinates of an object in the vicinity of the vehicle, and a gaze sensor device for determining a direction characteristic value that is representative of a gaze direction of a driver of the vehicle, as well as for determining a position characteristic value that is representative of a head position of the driver. The procedure determines the object identifier, the direction identifier, and the position identifier; depending on the position identifier and the object identifier, a projection identifier is determined that is representative of a vector connecting the head position to the object; depending on the direction and position identifiers, a first area identifier is determined that is representative of an estimated primary field of vision of the driver; subsequently, depending on the first area identifier and the projection identifier, an attention identifier is determined that is representative of the probability that the object is at least partially in the driver's field of vision, in particular in the estimated primary field of vision of the driver; finally, depending on the attention identifier, a warning function of the vehicle is activated. This advantageously allows the vehicle's warning function and corresponding warning times to be tailored to the driver. In particular, the attention index can indicate how well the driver can perceive individual objects in the environment, thus ensuring that no unnecessary warnings are issued or that other warnings are given too late. The gaze sensor device can be, in particular, a camera. The direction parameter can describe or include a gaze direction vector. Alternatively or additionally, the direction parameter can describe a head pose of the driver. The head pose can include a rotation and / or tilt of the head. For example, the head pose can be described by a nose vector. The object sensor device can also be a camera. Alternatively or additionally, the object sensor device can include a radar and / or lidar sensor unit or similar sensors. The object identifier can describe or include the coordinates of the object's center point. Alternatively or additionally, the object identifier can also include the coordinates of a contour of the object facing the vehicle or its entire surface facing the vehicle. The object can be, for example, another vehicle, an obstacle, a road boundary, or traffic signs. In particular, the object is an object that stands out from the road and requires the driver's attention. In this context, other vehicles or road users, such as cross traffic, are particularly relevant.The term "vehicle environment" here and in the following refers to the three-dimensional space external to the vehicle, in particular a sub-area within the driver's line of sight and / or direction of view. The driver's estimated primary field of view corresponds, for example, to the base of a circular or elliptical cone centered on the nose vector. The cone's opening angle can range from 0° to 90°, specifically between 0° and 2° for a region of sharpest vision, between 5° and 10° for deciphering signs, between 5° and 30° for perceiving outlines, or between 30° and 60° for perceiving colors. The horizontal opening angle (yaw angle) relative to the vehicle can differ from the vertical opening angle (pitch angle) relative to the vehicle. For example, the horizontal opening angle can be up to 90°, while the vertical opening angle can be only up to 70°. The estimated primary field of view can also be referred to as the useful field of view.In particular, the estimated primary field of vision can describe an area that the driver looks at 90% of the time while driving. In other words, the estimated primary field of vision describes an area in which the driver is likely to perceive objects. The vehicle's warning function can take the form of acoustic, haptic, or visual cues. For example, a visual cue can be projected directly into the driver's field of vision, making the object visibly more noticeable. In an advantageous embodiment according to the first aspect, an angle parameter is determined based on the direction parameter and the projection parameter. This angle parameter is representative of the smallest angle between the gaze direction and the vector connecting the head position to the object. The attention parameter is then determined based on this angle parameter. In this context, the smallest angle is understood to be the angle between the line of sight and the vector that connects the head position and the coordinates of the object that are closest to the area of ​​the estimated primary field of vision. In other words, the coordinates can describe a point on the object's contour that is closest to the point of origin of the line of sight or nose vector in the object's plane. In a further advantageous embodiment according to the first aspect, the directional parameter is representative of the driver's head position. Specifically, the directional parameter describes the nose vector, but not the actual gaze direction vector. Advantageously, this eliminates the need for a high-resolution camera as a gaze sensor device. This contributes to the cost-effective production of the assistance system. In a further advantageous embodiment according to the first aspect, the first area characteristic value is representative of an area around the viewing direction with a maximum horizontal deviation of 5° to 10° inclusive. In this context, the gaze direction parameter refers specifically to the nose vector. The first area parameter describes, in particular, the aforementioned base area of ​​the circular or elliptical cone with the deviation as the (horizontal) opening angle. In a further advantageous embodiment according to the first aspect, a second area parameter is determined based on the direction and position parameter, which is representative of the driver's estimated peripheral field of vision. The attention parameter is then determined based on this second area parameter. This advantageously allows for a gradation in the assessment of how well the driver can perceive individual objects in the environment. In particular, this enables improved coordination of the vehicle's warning function and corresponding warning times with the driver. The (horizontal) opening angle of the cone around the nose vector for peripheral perception of the object can, for example, be between 60° and 90° in this case. The attention metric can, in this context, indicate, in particular, the probability that the object lies within the driver's estimated primary field of vision, as well as the probability that the object lies within the driver's estimated peripheral field of vision. In a further advantageous embodiment according to the first aspect, the second area parameter is representative of an area around the viewing direction with a maximum horizontal deviation of 90° inclusive and a maximum vertical deviation of 70° inclusive. In a further advantageous embodiment according to the first aspect, a third area parameter is determined based on the direction and position parameter, which is representative of a field of vision not perceptible to the driver. The attention parameter is then determined based on this third area parameter. This advantageously allows for a more nuanced assessment of how well the driver can perceive individual objects in the environment. In particular, this enables improved coordination of the vehicle's warning function and corresponding warning times with the driver. The third area parameter, for example, is representative of an area around the nose vector with a horizontal deviation of over 90° and / or a vertical deviation of over 70°. In this context, the attention parameter can, in particular, indicate a probability that the object lies within the driver's estimated primary field of vision, a probability that the object lies within the driver's estimated peripheral field of vision, and a probability that the object lies outside the driver's field of vision. In a further advantageous embodiment according to the first aspect, a first state is assigned to the attention parameter if the angle parameter is representative of an angle outside the second range; a second state is assigned to the attention parameter if the angle parameter is representative of an angle both within the second range and outside the first range; and a third state is assigned to the attention parameter if the angle parameter is representative of an angle within the first range. Depending on the state assigned to the attention parameter, a warning function of the vehicle is activated. If the third state is assigned to the attention indicator, the warning function can be omitted, for example. In this case, it can be assumed that the driver has perceived the object. If the first state is assigned to the attention indicator, it can be assumed that the driver has not perceived the object. In this case, acoustic and / or haptic and / or visual cues are issued, for example. If the second state is assigned to the attention indicator, it is assumed that the object is at least peripherally perceived by the driver. A warning can be accordingly softened or suppressed. This can advantageously prevent sensory overload for the driver; in dangerous situations or similar circumstances, it can help ensure that the driver reacts to the vehicle's warnings. In a further advantageous embodiment according to the first aspect, the direction parameter is representative of the driver's gaze direction vector. In particular, the direction parameter comprises the actual gaze direction vector. Furthermore, the projection parameter is representative of a vector field that connects the head position with a contour of the object. Depending on the projection parameter and the direction parameter, a fourth state of the attention parameter is determined, which is representative of a probability that the driver's gaze direction lies within the contour of the object. Depending on the projection parameter and the direction parameter, alternatively or additionally, a fifth state of the attention parameter is determined, which is representative of a probability that the object lies at least partially within the driver's peripheral field of vision. This allows for a precise assessment of the driver's attention. The vector field comprises, in particular, one or more vectors. The fourth state, for example, describes the case where the gaze direction vector intersects the surface of the object. If the gaze direction vector is determined instead of the nose vector, the fourth state can provide a more precise indication than the third state. The fifth state, for example, describes the case where the gaze direction vector does not intersect the surface of the object, but the object lies within a region of peripheral perception around the gaze direction vector. If the gaze direction vector is determined instead of the nose vector, the fifth state can provide a more precise indication than the second state. In particular, the second and / or third states can serve as optional fallback positions in this case. According to a second aspect, the invention relates to an assistance system according to claim 9 for a vehicle. The assistance system comprises an object sensor device for determining an object characteristic value that is representative of the coordinates of an object in the vicinity of the vehicle, and a gaze sensor device for determining a direction characteristic value that is representative of the gaze direction of a driver of the vehicle, as well as for determining a position characteristic value that is representative of the head position of the driver. Furthermore, the assistance system comprises a control unit that is configured to carry out the method according to the first aspect. According to a third aspect, the invention relates to a computer program according to claim 10 for operating an assistance system. The computer program is configured to perform a method according to the first aspect when executed on a data processing device. According to a fourth aspect, the invention relates to a computer program product according to claim 11 comprising executable program code. When executed by a data processing device, the program code performs the method according to the first aspect. Exemplary embodiments of the invention are explained in more detail below with reference to the schematic drawings. Figure 1 shows a vehicle with the assistance system according to the invention; Figure 2 shows an exemplary driver's field of vision in top view; Figure 3 shows an exemplary driver's field of vision in perspective view; and Figure 4 shows an exemplary flowchart of a method for operating the assistance system according to Figure 1. Elements of the same construction or function are provided with the same reference symbols across all figures. Fig. 1 shows a vehicle with an assistance system 10, which is configured to detect a likely direction in which the driver is looking (hereinafter referred to as "gaze direction") and to initiate further steps depending on this. For example, the determined gaze direction can be used to monitor the driver's attention and, if necessary, to issue warnings. The assistance system 10 comprises a gaze sensor device (not shown in detail) with a camera positioned facing the driver and configured to detect the driver's gaze direction 2 as a directional parameter and the driver's head position 3 as a position parameter, for example, based on facial features such as the eye and / or nose position. In particular, the driver's head pose is determined, i.e., the tilt and / or yaw angle of the head, in order to determine in which areas the driver has good visibility. For example, the gaze direction corresponds to a nose vector 2a (see Fig. 3). The gaze sensor unit may also include an optional lighting device. The assistance system 10 further comprises an object sensor device (not shown in detail) with a camera, radar or lidar sensor unit, which is arranged facing away from the driver and is configured to detect an object 20 and to determine its spatial position relative to the vehicle or the driver as an object identifier. In particular, coordinates of a contour of the object or at least of its center point can be provided here. The assistance system 10 further comprises a control unit (not shown in detail) with a data and program memory in which a program for operating the assistance system 10 is stored, which is explained in more detail below with reference to the flowchart of Fig. 2. The program starts in step S1, in which, for example, variables are initialized. Furthermore, an object identifier is determined by the object sensor, and a direction and position identifier is determined by the gaze sensor. The program then continues in step S3. In step S3, depending on the position identifier and the object identifier, a projection identifier is determined that is representative of a vector 4 connecting the head position 3 with the object 20 (see Fig. 1). The program then continues in step S5a. In step S5a, a first area parameter is determined based on the direction and position parameter. This parameter is representative of the driver's estimated primary field of vision B1 (see Fig. 3-4). In this case, the primary field of vision B1 is an area around the nose vector 2a with a maximum horizontal deviation of approximately 60°. This area can also be considered the base of a circular or elliptical cone. Depending on the required accuracy, the primary field of vision B1 can comprise several sub-areas.The primary visual field B1 can thus be subdivided into an area B1' with a maximum horizontal deviation of 5° to 10°, in which the driver can (just barely) decipher characters; into an area B1'' with a maximum horizontal deviation of 5° to 30°, in which the driver can (just barely) perceive outlines; and into an area B1''' with a maximum horizontal deviation of 30° to 60°, in which the driver can (just barely) perceive colors. In the present embodiment, the primary field of view B1 designates an area with a maximum horizontal deviation of 7° inclusive and a maximum vertical deviation of 9° inclusive. This area can also be referred to as the useful field of view. In this execution variant, the program then continues in step S5b. In other execution variants, however, the program can also continue in step S5c or S7. In step S5b, a second area parameter is determined based on the direction and position parameter. This parameter is representative of the driver's estimated peripheral field of vision B2. In this case, the peripheral field of vision B2 is an area around the nose vector 2a with a maximum horizontal deviation of up to 94°. As shown in Fig. 3, the horizontal deviation can be asymmetrical around the nose vector 2a. However, the restricted field of vision of one eye towards the nose is compensated for by the corresponding other eye. The upper limit of the primary field of view B1 can be used as the lower limit of the horizontal deviation, for example, a maximum of 60°. In the present embodiment, the lower limit is 7°. A maximum vertical deviation is also 70°. A lower limit of the deviation of the peripheral field of view B2 can be specified analogously to the description above and is, for example, 9°. In this execution variant, the program then continues at step S5c. In other execution variants, however, the program can also continue at step S7. In step S5c, a third area parameter is determined based on the direction and position parameters. This parameter is representative of a field of vision B3 that is not perceptible to the driver. The unperceptible field of vision B3 is, in particular, an area outside the peripheral field of vision B2. The horizontal deviation in this context is, in particular, greater than 90°, for example, greater than 94°. Alternatively or additionally, the vertical deviation is, for example, greater than 70°. As shown in Fig. 4, object 20 is outside the primary and peripheral fields of vision B1 and B2 and is therefore not perceptible to the driver. The program then continues in step S7. In step S7, depending on the direction parameter and the projection parameter, an angle parameter is determined that is representative of a smallest angle Φ between the viewing direction 2 and the vector 4, which connects the head position 3 with the object 20 (see Fig. 1). The program then continues in step S9. In step S9, an attention parameter is determined based on the angle parameter and the corresponding area parameter. This parameter is representative of the probability that object 20 is at least partially within the driver's field of vision. Specifically, step S9 checks whether: - the angle Φ lies outside the peripheral field of vision B3. In this case, the attention parameter is assigned a first state; - the angle Φ lies within the peripheral field of vision B2 and outside the primary field of vision B1. In this case, the attention parameter is assigned a second state; or - the angle Φ lies within the primary field of vision B1. In this case, the attention parameter is assigned a third state. As an example, only the determined angle Φ and the limiting angles listed above are compared. The program then continues in step S11. In step S11, a warning function of the vehicle is activated depending on the attention indicator. For example, a warning is only issued if the attention indicator is assigned the first state. The program then terminates. In a second embodiment (not shown), the gaze sensor is configured to determine, in addition to the head pose, an actual gaze direction vector as gaze direction 2, exemplified by the driver's pupil position. Program steps S1 and S3 are executed analogously to the flowchart in Fig. 2. In this case, however, the projection parameter is representative of a vector field that connects the head position 3 with a contour of the object 20. The program continues in step S4 following step S3. In step S4, a fourth state of the attention parameter is determined based on the projection parameter and the direction parameter. This state is representative of the probability that the driver's gaze direction lies within the object's contour. Specifically, step S4 determines whether the gaze direction vector intersects a surface of object 20 facing the vehicle. For this purpose, the corresponding angles between the individual vectors 4 of the vector field and the gaze direction vector can also be determined and compared. If the fourth state is assigned to the attention parameter, it can be assumed that the driver has perceived the object with a high degree of certainty. A warning is then unnecessary, and the program can subsequently be terminated. Alternatively, if the gaze direction vector does not intersect the surface of object 20 facing the vehicle, it is checked whether object 20 lies within the driver's peripheral field of vision B2. Analogous to steps S5a to S9, in this context, it is specifically checked whether the angle Φ between the actual gaze direction vector 2 and vector 4 lies within the peripheral field of vision B2 and outside the primary field of vision B1. If this is the case, it can be assumed with a high degree of certainty that the driver has perceived the object peripherally. In this context, a fifth state is assigned to the attention metric. A warning can then also be omitted, for example. The program can then be terminated. In the event that the gaze sensor device degrades during operation, for example, if a reliable statement about the gaze direction vector can no longer be made, but the head pose can still be determined, the program can be continued in the corresponding program step of the flowchart according to Fig. 2 in order to determine the attention score based solely on the first, second and third states. Advantageously, the described methods provide the vehicle with information about which objects in the vehicle's surroundings are perceptible or likely to be perceived by the driver. The listed states (first to fifth states) can, for example, serve as the basis for issuing a warning function and / or be provided as output signals to other vehicle assistance systems. This allows, in particular, the adaptation of information and entertainment displays to the driver's surroundings. In summary, this is made possible by determining how well the driver can perceive individual objects in the environment. This is done by determining the driver's head pose and position using a camera pointed at the driver. To ensure the product's cost-effectiveness, determining the driver's viewing angle, which would require a higher-resolution camera, is omitted for this example. The driver's head pose and position are used to calculate the smallest angle to the object measured by the vehicle. This smallest angle is then used to determine whether the object lies within one of the following three areas: - likely seen by the driver: if the angle to the object is small enough that the object overlaps with the useful field of view, i.e., the area where the driver's eyes are located 90% of the time while driving.- probably seen in the peripheral area by the driver: if the object is not in the first area, but still in an area where it can be perceived peripherally by the driver. - not seen: if the angle to the object is large enough that it cannot be perceived by the driver. The assistance system can be advantageously scalable to accommodate adjustments to the sensor characteristics. If the gaze sensor is configured to determine the driver's viewing angle, this angle is used for evaluation instead of the head pose. This allows the three basic states listed above to be extended to include the following: - seen: the driver's gaze overlaps the object's surface. - seen peripherally: the driver's gaze does not overlap the object's surface, but the object is definitely located in the peripheral area. In the event that the gaze sensor device degrades during operation to a state in which only the head pose can be determined, the three basic states listed above can serve as a fallback position for evaluation. Reference symbol list 10 Assistance system 20 Object 2 Direction of gaze 2a Nose vector 3 Head position 4 Vector Φ Angle B1...B3 Field of view S1...S11 Program steps

Claims

Method for operating an assistance system (10) for a vehicle, comprising an object sensor device and a gaze sensor device, wherein in the method: - an object characteristic value is determined by the object sensor device, which is representative of the coordinates of an object (20) in the vicinity of the vehicle, wherein the vicinity denotes a three-dimensional space external to the vehicle; - a direction characteristic value is determined by the gaze sensor device, which is representative of a gaze direction (2) of a driver of the vehicle, and a position characteristic value is determined by the gaze sensor device, which is representative of a head position (3) of the driver; - depending on the position characteristic value and the object characteristic value, a projection characteristic value is determined, which is representative of a vector (4) that connects the head position (3) with the object (20); - depending on the direction and position characteristic value, a first area characteristic value is determined.which is representative of an estimated primary field of vision (B1) of the driver, - depending on the direction and position parameter, a second area parameter is determined which is representative of an estimated peripheral field of vision of the driver, and - depending on the first area parameter, the second area parameter and the projection parameter, an attention parameter is determined which is representative of a probability that the object (20) is at least partially in the driver's field of vision, and - depending on the attention parameter, a warning function of the vehicle is activated. Method according to claim 1, wherein, depending on the direction parameter and the projection parameter, an angle parameter is determined which is representative of a smallest angle (Φ) between the viewing direction (2) and the vector (4) which connects the head position (3) with the object (20), and, depending on the angle parameter, the attention parameter is determined. Method according to one of the preceding claims, wherein the direction parameter is representative of a head pose of the driver. Method according to one of the preceding claims, wherein the first area characteristic value is representative of an area around the viewing direction with a maximum horizontal deviation of 5° to 10° inclusive. Method according to one of the preceding claims, wherein the second area characteristic value is representative of an area around the viewing direction with a maximum horizontal deviation of inclusive 90° and a maximum vertical deviation of inclusive 70°. Method according to one of the preceding claims, wherein, depending on the direction and position parameter, a third area parameter is determined which is representative of a field of vision not perceptible to the driver, and, depending on the third area parameter, the attention parameter is determined. A method according to one of the preceding claims, wherein: - in the case that the angle characteristic is representative of an angle outside the second range, a first state is assigned to the attention characteristic; - in the case that the angle characteristic is representative of an angle within the second range and outside the first range, a second state is assigned to the attention characteristic; and - in the case that the angle characteristic is representative of an angle within the first range, a third state is assigned to the attention characteristic, wherein - depending on the state assigned to the attention characteristic, a warning function of the vehicle is activated. A method according to one of the preceding claims, wherein: - the direction parameter is representative of a gaze direction vector of the driver, - the projection parameter is representative of a vector field that connects the head position with a contour of the object, - depending on the projection parameter and the direction parameter, a fourth state of the attention parameter is determined which is representative of a probability that the driver's gaze direction lies within the contour of the object, and - depending on the projection parameter and the direction parameter, a fifth state of the attention parameter is determined which is representative of a probability that the object lies at least partially within the driver's peripheral field of vision. Assistance system for a vehicle, comprising: - an object sensor device for determining an object characteristic value that is representative of the coordinates of an object in the vicinity of the vehicle, wherein the vicinity denotes a three-dimensional space external to the vehicle, - a gaze sensor device for determining a direction characteristic value that is representative of a gaze direction of a driver of the vehicle, as well as for determining a position characteristic value that is representative of a head position of the driver, and - a control unit that is configured to carry out the method according to any one of claims 1 to 8. Computer program for operating an assistance system, wherein the computer program is configured to perform a method according to one of claims 1 to 8 when executed on a data processing device. A computer program product comprising executable program code, wherein the program code, when executed by a data processing device, performs the method according to any one of claims 1 to 8.

Images