METHOD FOR OPERATING A VEHICLE ASSISTANCE SYSTEM AND ASSISTANCE SYSTEM

Description

[0001] The invention relates to a method for operating an assistance system for a vehicle as well as a corresponding assistance system, computer program and computer program product.

[0002] Today's vehicles can incorporate sensor units such as a camera sensor that detects a user's head pose, gaze direction, and / or position, for example, based on facial features. Based on this, driver fatigue can be determined, for example.

[0003] Methods for operating an assistance system for a vehicle are known, for example, from the documents JP 2009 237776 A, EP 1 484 014 A1, JP 2014 120112 A or JP 2015 125686 A.

[0004] The object underlying the invention is to provide a method for operating an assistance system for a vehicle and a corresponding assistance system which contributes to precisely determining the driver's attention.

[0005] The problem is solved by the independent patent claims. Advantageous embodiments are characterized in the subclaims.

[0006] According to a first aspect, the invention relates to a method for operating an assistance system for a vehicle. The assistance system comprises an object sensor device for determining an object characteristic value comprising coordinates of a center point of an object in the surroundings of the vehicle, wherein the surroundings denote a three-dimensional space external to the vehicle, and a gaze sensor device for determining a direction characteristic value comprising a gaze direction vector and representative of a gaze direction of a driver of the vehicle, as well as for determining a position characteristic value representative of a head position of the driver.

[0007] In the method, the object characteristic value, the direction characteristic value and the position characteristic value are determined; depending on the position characteristic value and the object characteristic value, a projection characteristic value is determined which is representative of a vector connecting the head position with the object; depending on the direction characteristic value and the projection characteristic value, an angle characteristic value is determined which is representative of a smallest angle (Φ) between the viewing direction (2) and the vector (4) connecting the head position (3) with the object (20); depending on the direction and position characteristic value, a first area characteristic value is determined which is representative of an estimated primary field of view of the driver, wherein the first area characteristic value is representative of a first area around the viewing direction with a maximum horizontal deviation of 5° to 10° inclusive; then, depending on the direction and position characteristic value, a second area characteristic value is determined,which is representative of an estimated peripheral field of vision of the driver, wherein the second area characteristic value is representative of a second area around the line of sight with a maximum horizontal deviation of 90° and inclusive and a maximum vertical deviation of 70° and inclusive, and depending on the direction and position characteristic value, a third area characteristic value is determined which is representative of a third area with a field of vision not perceivable by the driver, and depending on the first area characteristic value, the second area characteristic value, the third area characteristic value and the angle characteristic value, an attention characteristic value is determined which is representative of a probability that the object is at least partially in the driver's field of vision, and in the case that the angle characteristic value is representative of an angle within the third area, a first state is assigned to the attention characteristic value; in the case,that the angle value is representative of an angle within the second range and outside the first range, a second state is assigned to the attention value; and in the case that the angle value is representative of an angle within the first range, a third state is assigned to the attention value; and finally, depending on the state assigned to the attention value, a warning function of the vehicle is activated, whereby the warning function can be dispensed with if the third state is assigned to the attention value; the warning function comprises acoustic and / or haptic and / or visual indications if the attention value is assigned the first state; and the warning function comprises indications of the second state in a mitigated form if the attention value is assigned the second state.

[0008] This advantageously allows the vehicle's warning function and corresponding warning times to be tailored to the driver. In particular, the attention score can indicate how well the driver can perceive individual objects in the surrounding area, thus ensuring that no unnecessary warnings are issued or that other warnings are issued too late.

[0009] The gaze sensor device can, in particular, be a camera. The directional characteristic value can describe or include a gaze direction vector. Alternatively or additionally, the directional characteristic value can describe the driver's head pose. The head pose can include a rotation and / or tilt of the head. For example, the head pose can be described using a nose vector.

[0010] The object sensor device can also be a camera. Alternatively or additionally, the object sensor device can comprise a radar and / or lidar sensor unit or similar sensor technology. The object characteristic value can describe or comprise coordinates of a center point of the object. Alternatively or additionally, the object characteristic value can also comprise coordinates of a contour of the object facing the vehicle or of its entire surface facing the vehicle. The object can be, for example, another vehicle, an obstacle, a road barrier or even traffic signs. In particular, the object is an object that stands out from the course of the road and requires the driver's attention. In this context, particular mention should be made of other vehicles or road users, for example cross traffic.Here and in the following, the surroundings of the vehicle are used to describe the three-dimensional space external to the vehicle, in particular a partial area within the driver's field of vision and / or direction of vision.

[0011] The estimated primary field of view of the driver corresponds, for example, to the base area of a circular or elliptical cone around the nose vector. The cone's aperture angle can, for example, be between 0° and 90°, in particular, for example, between 0° and 2° for a region of sharpest vision, between 5° and 10° for deciphering characters, between 5° and 30° for perceiving outlines, or between 30° and 60° for perceiving colors. The horizontal aperture angle (yaw angle) with respect to the vehicle can differ from the vertical aperture angle (pitch angle) with respect to the vehicle. For example, the horizontal aperture angle is up to 90°, whereas the vertical aperture angle is only up to 70°. The estimated primary field of view can also be referred to as the useful field of view.In particular, the estimated primary field of view can describe an area that the driver views for 90% of the driving time. In other words, the estimated primary field of view describes an area in which the driver is likely to perceive objects.

[0012] The vehicle's warning function can include acoustic, haptic, or visual cues. For example, a visual cue can be projected directly into the driver's field of vision, visibly highlighting the object.

[0013] Based on the direction and projection values, an angle value is determined that represents the smallest angle between the gaze direction and the vector connecting the head position to the object. The attention value is determined based on the angle value.

[0014] In this context, the smallest angle is understood to be the angle between the gaze direction and the vector connecting the head position and the object's coordinates closest to the area of the estimated primary field of view. In other words, the coordinates can describe a point on the object's contour that is closest to the point of origin of the gaze direction or nose vector in the object's plane.

[0015] In a further advantageous embodiment according to the first aspect, the directional characteristic value is representative of the driver's head pose. In particular, the directional characteristic value describes the nose vector, but not the actual gaze direction vector. This advantageously eliminates the need for a high-resolution camera as a gaze sensor device. This contributes to the cost-effective production of the assistance system.

[0016] In this context, the gaze direction parameter is specifically the nose vector. The first area parameter describes, in particular, the aforementioned base area of the circular or elliptical cone, with the deviation as the (horizontal) aperture angle.

[0017] According to the first aspect, a second area value is determined based on the direction and position value, which is representative of an estimated peripheral field of vision of the driver. The attention value is determined based on the second area value.

[0018] This advantageously allows for gradual assessment of how well the driver can perceive individual objects in the surroundings. In particular, it enables improved coordination of the vehicle's warning function and corresponding warning times with the driver. The (horizontal) opening angle of the cone around the nose vector for peripheral perception of the object can, for example, be between 60° and 90° in this case. In this context, the attention index can, in particular, indicate the probability that the object is in the driver's estimated primary field of vision and the probability that the object is in the driver's estimated peripheral field of vision.

[0019] Based on the direction and position values, a third area value is determined, which is representative of a field of vision not perceivable by the driver. The attention value is determined based on the third area value.

[0020] This advantageously enables further gradation in the assessment of how well the driver can perceive individual objects in the environment. In particular, this enables improved coordination of the vehicle's warning function and corresponding warning times with the driver. The third area characteristic value is, for example, representative of an area around the nose vector with a horizontal deviation of more than 90° and / or a vertical deviation of more than 70°. In this context, the attention characteristic value can, in particular, indicate a probability with which the object is in the driver's estimated primary field of vision, a probability with which the object is in the driver's estimated peripheral field of vision, and a probability with which the object is outside the driver's field of vision.

[0021] According to the first aspect, If the angle value is representative of an angle outside the second range, a first state is assigned to the attention value; if the angle value is representative of an angle within the second range and outside the first range, a second state is assigned to the attention value; and if the angle value is representative of an angle within the first range, a third state is assigned to the attention value. Depending on the state assigned to the attention value, a warning function of the vehicle is activated.

[0022] If the attention index is assigned to the third state, the warning function can be dispensed with, for example. In this case, it can be assumed that the driver has perceived the object. If, on the other hand, the attention index is assigned to the first state, it can be assumed that the driver has not perceived the object. In this case, acoustic and / or haptic and / or visual cues are issued, for example. If the attention index is assigned to the second state, it is assumed that the object is at least peripherally perceived by the driver. A warning can be mitigated or suppressed accordingly. This advantageously prevents the driver from being overwhelmed by stimuli; in dangerous or similar situations, this can help ensure that the driver reacts to the vehicle's warnings.

[0023] In a further advantageous embodiment according to the first aspect, the directional characteristic value is representative of a gaze direction vector of the driver. In particular, the directional characteristic value comprises the actual gaze direction vector. Furthermore, the projection characteristic value is representative of a vector field that connects the head position with a contour of the object. Depending on the projection characteristic value and the directional characteristic value, a fourth state of the attention characteristic value is determined, which is representative of a probability that the driver's gaze direction lies within the contour of the object. Depending on the projection characteristic value and the directional characteristic value, alternatively or additionally a fifth state of the attention characteristic value is determined, which is representative of a probability that the object lies at least partially within the driver's peripheral field of vision.

[0024] This advantageously allows a precise statement to be made about the driver’s attention.

[0025] The vector field comprises, in particular, one or more vectors. The fourth state describes, for example, the case in which the gaze direction vector hits the surface of the object. If the gaze direction vector is determined instead of the nose vector, the fourth state can provide a more precise statement than the third state. The fifth state describes, for example, the case in which the gaze direction vector does not hit the surface of the object, but the object lies in a region of peripheral perception around the gaze direction vector. If the gaze direction vector is determined instead of the nose vector, the fifth state can provide a more precise statement than the second state. In particular, the second and / or third state can serve as optional fallback positions in this case.

[0026] According to a second aspect, the invention relates to an assistance system for a vehicle. The assistance system comprises an object sensor device for determining an object characteristic value, which comprises coordinates of a center point of an object in the surroundings of the vehicle, wherein the surroundings denote a three-dimensional space external to the vehicle, and a gaze sensor device for determining a direction characteristic value, which comprises a gaze direction vector and is representative of a gaze direction of a driver of the vehicle, as well as for determining a position characteristic value, which is representative of a head position of the driver. Furthermore, the assistance system comprises a control unit configured to carry out the method according to the first aspect.

[0027] According to a third aspect, the invention relates to a computer program for operating an assistance system. The computer program is designed to carry out a method according to the first aspect when executed on a data processing device.

[0028] According to a fourth aspect, the invention relates to a computer program product comprising executable program code. When executed by a data processing device, the program code carries out the method according to the first aspect.

[0029] Embodiments of the invention are explained in more detail below with reference to the schematic drawings.

[0030] They show: Figure 1 shows a vehicle with the assistance system according to the invention; Figure 2 shows an exemplary field of vision of a driver in plan view; Figure 3 shows an exemplary field of vision of the driver in perspective view; and Figure 4 shows an exemplary flow diagram of a method for operating the assistance system according to Figure 1 .

[0031] Elements of the same construction or function are provided with the same reference symbols throughout the figures.

[0032] Figure 1 shows a vehicle with an assistance system 10 configured to detect a likely direction in which the driver is looking (hereinafter "viewing direction") and initiate further steps based on this. For example, the determined viewing direction can be used to monitor the driver's attention and issue warnings if necessary.

[0033] The assistance system 10 comprises a gaze sensor device (not shown in detail) with a camera that is arranged facing the driver and configured to detect a gaze direction 2 of the driver as a directional characteristic and a head position 3 of the driver as a position characteristic, for example based on facial features such as the eye and / or nose position. In particular, a head pose of the driver is determined, i.e., an inclination and / or yaw angle of their head, in order to be able to determine in which areas the driver has good vision. For example, the gaze direction corresponds to a nose vector 2a (cf. Figure 3 ). The gaze sensor unit may also include an optional lighting device.

[0034] The assistance system 10 further comprises an object sensor device (not shown in detail) with a camera, a radar, or a lidar sensor unit, which is arranged facing away from the driver and configured to detect an object 20 and determine its spatial position relative to the vehicle or the driver as an object characteristic value. In particular, coordinates of a contour of the object or at least its center point can be provided here.

[0035] The assistance system 10 further comprises a control unit (not shown in detail) with a data and program memory in which a program for operating the assistance system 10 is stored, which program is described in the flow chart of the Figure 2 is explained in more detail below.

[0036] The program starts in step S1, where, for example, variables are initialized. Furthermore, an object characteristic value is determined by the object sensor device, and a direction and position characteristic value is determined by the gaze sensor device. The program then continues in step S3.

[0037] In step S3, a projection characteristic value is determined depending on the position characteristic value and the object characteristic value, which is representative of a vector 4 connecting the head position 3 with the object 20 (cf. Fig. 1 ). The program then continues in step S5a.

[0038] In step S5a, a first area characteristic value is determined depending on the direction and position characteristic value, which is representative of an estimated primary field of view B1 of the driver (cf. Fig. 3-4). In this case, the primary field of vision B1 is an area around the nose vector 2a with a maximum horizontal deviation of approximately 60°. The area can also be viewed as the base area of a circular or elliptical cone. Depending on the accuracy requirements, the primary field of vision B1 can comprise several sub-areas. For example, the primary field of vision B1 can be divided into an area B1' with a maximum horizontal deviation of 5° to 10°, in which the driver can (just about) decipher characters; an area B1" with a maximum horizontal deviation of 5° to 30°, in which the driver can (just about) perceive outlines; and an area B1‴ with a maximum horizontal deviation of 30° to 60°, in which the driver can (just about) perceive colors.

[0039] In this embodiment, the primary field of view B1 refers to an area with a maximum horizontal deviation of 7° and a maximum vertical deviation of 9°. This area can also be referred to as the useful field of view.

[0040] In this embodiment, the program then continues in step S5b. However, in other embodiments, the program can also continue in step S5c or S7.

[0041] In step S5b, a second area characteristic value is determined depending on the direction and position characteristic value, which is representative of an estimated peripheral field of vision B2 of the driver. The peripheral field of vision B2 is an area around the nose vector 2a with a maximum horizontal deviation of up to 94°. As shown in Figure 3As shown, the horizontal deviation may be asymmetrical around the nose vector 2a. However, the restricted field of view of one eye toward the nose is compensated by the corresponding other eye.

[0042] The upper limit of the primary field of view B1 can be used as the lower limit of the horizontal deviation, for example, a maximum of 60°. In the present embodiment, the lower limit is 7°. A maximum vertical deviation is also 70°. A lower limit of the deviation of the peripheral field of view B2 can be specified analogously to the above description and is, for example, 9°.

[0043] In this embodiment, the program then continues in step S5c. However, in other embodiments, the program can also continue in step S7.

[0044] In step S5c, a third area characteristic value is determined depending on the direction and position characteristic value, which is representative of a field of vision B3 that is not perceptible to the driver. The imperceptible field of vision B3 is, in particular, an area outside the peripheral field of vision B2. The horizontal deviation in this context is, in particular, more than 90°, for example, more than 94°. Alternatively or additionally, the vertical deviation is, for example, more than 70°. As in Figure 4 As shown, object 20 is outside the primary and peripheral fields of vision B1, B2 and therefore not perceptible to the driver. The program then continues in step S7.

[0045] In step S7, an angle characteristic value is determined depending on the direction characteristic value and the projection characteristic value, which is representative of a smallest angle Φ between the viewing direction 2 and the vector 4 connecting the head position 3 with the object 20 (cf. Fig. 1 ). The program then continues in step S9.

[0046] In step S9, an attention value is determined depending on the angle value and the corresponding area value, which is representative of the probability that the object 20 is at least partially in the driver's field of vision. In this context, in step S9, it is checked in particular whether the angle Φ lies outside the peripheral field of vision B3. In this case, the attention index is assigned a first state; the angle Φ lies within the peripheral field of vision B2 and outside the primary field of vision B1. In this case, the attention index is assigned a second state; or whether the angle Φ lies within the primary field of vision B1. In this case, the attention index is assigned a third state.

[0047] For example, only the determined angle Φ and the limit angles listed above are compared. The program then continues in step S11.

[0048] In step S11, a vehicle warning function is activated based on the attention level. For example, a warning is only issued if the attention level is assigned to the first state. The program then terminates.

[0049] In a second embodiment (not shown), the gaze sensor device is configured to determine, in addition to the head pose, an actual gaze direction vector as gaze direction 2, for example based on a pupil position of the driver. Program steps S1 and S3 are executed analogously to the flowchart according to Figure 2 performed. However, in this case, the projection characteristic is representative of a vector field connecting head position 3 with a contour of object 20. Following step S3, the program continues in step S4.

[0050] In step S4, a fourth state of the attention index is determined based on the projection index and the direction index, which is representative of the probability that the driver's line of sight lies within the contour of the object. In particular, step S4 determines whether the line of sight vector strikes a surface of the object 20 facing the vehicle. For this purpose, the corresponding angles between the individual vectors 4 of the vector field and the line of sight vector can also be determined and compared, for example. If the fourth state is assigned to the attention index, it can be assumed that the driver has perceived the object with a high degree of certainty. A warning can then be omitted. The program can then be terminated.

[0051] Alternatively, if the gaze direction vector does not impinge on the surface of object 20 facing the vehicle, a check is made to determine whether object 20 is located in the driver's peripheral field of vision B2. Analogous to steps S5a to S9, in this context, a check is made in particular to determine whether the angle Φ between the actual gaze direction vector 2 and vector 4 lies within the peripheral field of vision B2 and outside the primary field of vision B1. If this is the case, it can be assumed with a high degree of certainty that the driver perceived the object peripherally. In this context, a fifth state is assigned to the attention index. A warning can then also be omitted, for example. The program can then be terminated.

[0052] In the event that the gaze sensor device degrades during operation, for example a reliable statement about the gaze direction vector can no longer be made, but the head pose can still be determined, the program can be stopped in the corresponding program step of the flow chart according to Figure 2 to determine the attention score based only on the first, second and third states.

[0053] Advantageously, the described methods provide the vehicle with information about which objects in the vehicle's surroundings are perceptible or likely to be perceptible by the driver. The listed states (first to fifth states) can thus serve as a basis for issuing a warning function and / or be provided as an output signal to other assistance systems in the vehicle. This enables, in particular, adaptation of the playback of information and entertainment offerings to the driver's surroundings.

[0054] In summary, this is achieved by determining how well the driver can perceive individual objects in the environment. This is done by determining the driver's head pose and position using a camera aimed at the driver. To ensure the product's cost-effectiveness, determining the driver's viewing angle is omitted, which would require a higher-resolution camera. The driver's head pose and position are used to calculate the smallest angle to the object measured by the vehicle. The smallest angle to the object is used to determine whether the object lies in one of the following three areas: Likely seen by the driver: when the angle to the object is small enough that the object overlaps with the useful field of view, i.e. the area in which the driver's eyes are located for 90% of the driving time. Likely seen in the peripheral field by the driver: when the object is not in the first field, but nevertheless in an area where it can be perceived peripherally by the driver. Not seen: when the angle to the object is large enough that it cannot be perceived by the driver.

[0055] Advantageously, the assistance system can be scalable for adjustments to the sensor properties. If the gaze sensor device is configured to determine a driver's viewing angle, this viewing angle is used for evaluation instead of the head pose. This allows the three basic states listed above to be expanded to include the following states: Seen: the driver's gaze overlaps the object area. Peripherally seen: the driver's gaze does not overlap the object area, but the object is clearly in the peripheral area.

[0056] In the event that the gaze sensor device degrades during operation to a state in which only the head pose can be determined, the three basic states listed above can serve as a fallback position for evaluation. List of reference symbols

[0057] 10Assistance system 20Object 2Direction of view 2aNose vector 3Head position 4Vector ΦAngle B1... B3Field of view S1... S11Program steps

Claims

1. Method for operating an assistance system (10) for a vehicle comprising an object sensor device and a viewing sensor device, the method including - ascertaining by way of the object sensor device an object characteristic value which comprises coordinates of a centre point of an object (20) in the vicinity of the vehicle, the vicinity designating a three-dimensional space externally with respect to the vehicle, - ascertaining by way of the viewing sensor device a direction characteristic value which comprises a viewing direction vector and is representative of a viewing direction (2) of a driver of the vehicle, and ascertaining by way of the viewing sensor device a position characteristic value which is representative of a head position (3) of the driver, - depending on the position characteristic value and the object characteristic value, ascertaining a projection characteristic value which is representative of a vector (4) connecting the head position (3) with the object (20), - depending on the direction characteristic value and the projection characteristic value, ascertaining an angle characteristic value which is representative of the smallest angle (Φ) between the viewing direction (2) and the vector (4) connecting the head position (3) with the object (20), - depending on the direction and position characteristic values, ascertaining a first region characteristic value which is representative of an estimated primary field of view (B1) of the driver, wherein the first region characteristic value is representative of a first region around the viewing direction with a maximum horizontal deviation of including 5° to 10°, - depending on the direction and position characteristic values, ascertaining a second region characteristic value which is representative of an estimated peripheral field of view of the driver, wherein the second region characteristic value is representative of a second region around the viewing direction with a maximum horizontal deviation of including 90° and a maximum vertical deviation of including 70°, and - depending on the direction and position characteristic values, ascertaining a third region characteristic value which is representative of a third region with a field of view that is not perceivable by the driver, and - depending on the first region characteristic value, the second region characteristic value, the third region characteristic value and the angle characteristic value, ascertaining an attention characteristic value which is representative of a probability that the object (20) is at least partially within the field of view of the driver, and -- in the case that the angle characteristic value is representative of an angle within the third region, assigning the attention characteristic value a first state; -- in the case that the angle characteristic value is representative of an angle within the second region and outside the first region, assigning the attention characteristic value a second state; and -- in the case that the angle characteristic value is representative of an angle within the first region, assigning the attention characteristic value a third state, and - depending on the state assigned to the attention characteristic value, activating a warning function of the vehicle, wherein -- the warning function can be omitted if the attention value is assigned the third state; -- the warning function comprises audible and / or haptic and / or visual indications if the attention value is assigned the first state; and -- the warning function comprises indications of the second state in a mitigated form if the attention value is assigned the second state.

2. Method according to Claim 1, wherein the direction characteristic value is representative of a head position of the driver.

3. Method according to either of the preceding claims, wherein - the direction characteristic value is representative of a viewing direction vector of the driver, - the projection characteristic value is representative of a vector field that connects the head position with a contour of the object, - depending on the projection characteristic value and the direction characteristic value, a fourth state of the attention characteristic value is ascertained, which is representative of a probability that the viewing direction of the driver is within the contour of the object, and - depending on the projection characteristic value and the direction characteristic value, a fifth state of the attention characteristic value is ascertained, which is representative of a probability that the object is at least partially within the peripheral field of view of the driver.

4. Assistance system for a vehicle, comprising - an object sensor device for ascertaining an object characteristic value which comprises coordinates of a centre point of an object in the vicinity of the vehicle, the vicinity designating a three-dimensional space externally with respect to the vehicle, - a viewing sensor device for ascertaining a direction characteristic value which comprises a viewing direction vector and which is representative of a viewing direction of a driver of the vehicle, and for ascertaining a position characteristic value which is representative of a head position of the driver, and - a control unit, which is set up to carry out the method according to any of Claims 1 to 3.

5. Computer program for operating an assistance system, wherein the computer program is designed to carry out a method according to any of Claims 1 to 3 when it is executed in an assistance system according to Claim 4.

6. Computer program product comprising executable program code, wherein the program code carries out the method according to any of Claims 1 to 3 when it is executed in an assistance system according to Claim 4.