Method for determining reactivity

Abstract

The invention relates to a method for determining the reaction capability of a vehicle user (2) in the automated driving of a vehicle (1) from image data acquired by a driver monitoring camera. According to the invention, it is provided to determine whether at least one predetermined relevant object (O) in the current vehicle environment is located within the field of view of the vehicle user (2) from digital map data present on the vehicle side and / or from sensor data acquired on the vehicle side; in the case of at least one detected predetermined relevant object (O) in the current vehicle environment, to determine with which duration and / or frequency the line of sight of the vehicle user (2) is directed at the detected at least one predetermined relevant object (O) from the acquired image data of the driver monitoring camera; and in the case of a determined duration below a predetermined minimum duration and / or in the case of a determined frequency of the line of sight wandering below a minimum frequency, the reaction capability of the vehicle user (2) is rated as nil.

Description

Technical Field

[0001] This invention relates to a method for determining the reaction ability of a vehicle user in autonomous driving based on image data collected by a driver monitoring camera. Background Technology

[0002] DE 10 2015 001 686 B4 discloses a method and apparatus for identifying a driver's responsiveness during autonomous driving of a motor vehicle. Here, a first assessment of responsiveness is performed using camera monitoring of the driver, and a second assessment of responsiveness is performed by recognizing the driver's actions on an operating unit. When the first and / or second assessments have identified the presence of responsiveness, responsiveness is classified as present. In the first assessment of responsiveness using camera monitoring of the driver, responsiveness is classified as present when at least one of the driver's eyes is open for a minimum period of a predetermined time interval. Responsiveness is not classified as present when the operation of the operating unit is performed by a person other than the driver or in a periodic manner.

[0003] Additionally, DE 10 2016 011 246 A1 describes a method for monitoring the state of at least one passenger in a vehicle. The method specifies that the passenger's head is illuminated by at least one alternately on / off light source. Light reflected from the passenger's eyes is detected by at least one sensor, wherein, additionally, it is examined whether the reflection changes according to the illumination of the light source, and at least one eyelid movement and / or head movement of the passenger is detected. Summary of the Invention

[0004] The objective of this invention is to describe a method for determining the responsiveness of a vehicle user during autonomous driving.

[0005] According to the present invention, this task is accomplished by a method having the features described below.

[0006] Advantageous designs of the present invention are also described below.

[0007] According to the present invention, a method for determining the responsiveness of a vehicle user in autonomous driving based on image data acquired by a driver monitoring camera specifies that, based on digital map data present on the vehicle side and / or sensor data acquired on the vehicle side, it is determined whether at least one predetermined object in the current vehicle environment is within the vehicle user's field of vision. In the case of at least one detected predetermined object in the current vehicle environment, based on the image data detected by the driver monitoring camera, it is determined at what duration and / or frequency the vehicle user focuses their gaze on the detected at least one predetermined object, and the vehicle user's responsiveness is assessed as zero when the determined duration is less than a predetermined minimum duration and / or when the determined frequency of gaze wandering is less than a minimum frequency.

[0008] By using this method, the reaction capabilities of the vehicle's users during fully automated driving operations can be determined. Here, "vehicle users" refers to the personnel performing the driving tasks in manual driving.

[0009] In particular, assessing responsiveness is crucial to determining whether the vehicle user can correctly respond to system-determined takeover requests related to the driving task and, consequently, accurately grasp the current situation. For this, situational awareness presupposes the ability to see and perceive relevant objects and / or areas outside the vehicle that depend on the situation.

[0010] In one embodiment of the method, an object is pre-determined as relevant when it is in the field of vision of the vehicle user during manual driving of the vehicle, i.e. when the vehicle user directs their gaze at the object for a predetermined duration and / or frequency, and when the object is considered by the vehicle user during vehicle control.

[0011] If the captured image data indicates that the vehicle user's gaze was not aligned with, or was only briefly aligned with, a predetermined object, it is interpreted that the vehicle user did not perceive the predetermined object and therefore had at least a low responsiveness.

[0012] In one possible implementation of the method, at least one other vehicle, traffic sign, light signaling device, warning light, lane marking, reflector post, guide rail, crash barrier and / or bridge pier located in front of, to the side and / or behind the vehicle is pre-determined and / or considered as a relevant object when determining the vehicle user's reaction capability.

[0013] Therefore, such an object is considered relevant and pre-determined as an object that can and / or should influence the driving style of the vehicle user in the manual driving of the vehicle.

[0014] An improved version of this method specifies that the size, visibility, relative position, and / or absolute position of a predetermined object related to the vehicle user's field of vision are identified and / or taken into account. This allows detection of whether the vehicle user can fully see this predetermined object from the vehicle.

[0015] To this end, one design provision of this method is that the vehicle user's gaze direction is determined as a line-of-sight vector, which is inferred to determine the field of view. In particular, the line-of-sight vector is inferred to be oriented outwards from the vehicle, and it is determined whether the inferred line-of-sight vector is aligned with a predetermined relevant object, and whether the predetermined relevant area is therefore within the vehicle user's field of view.

[0016] In particular, another possible design provision of this method is to predetermine the center point between the pupils of the vehicle user as the starting point of the gaze vector. This predetermining of the center point between the pupils is especially important when the eyes are looking straight ahead with the vehicle user's head oriented in a forward orientation. Determining the starting point of the gaze vector is used to project a pre-defined relevant object, specifically a region, onto the windshield and to determine whether the vehicle user is within that region.

[0017] In another design, the surrounding area of ​​a pre-determined object is defined based on its shape and / or size. In particular, the surrounding area is defined when the pre-determined object has a relatively complex shape and / or size. The smaller the defined area, the more accurately the object can be assigned to a line of sight, thereby enabling the detection of whether the pre-determined object is within the vehicle user's field of vision and whether the pre-determined object can be identified by the vehicle user.

[0018] If multiple objects are measured, the visibility of these objects relative to pre-determined relevant objects is assessed in a possible design based on their location, size, and / or material properties. That is, the visibility of pre-determined relevant objects is checked to essentially ensure that these objects are within the vehicle user's field of vision and are not obstructed by other objects. In other words, it is checked whether the pre-determined relevant objects are visible to the vehicle user.

[0019] In one possible improvement, perception and / or situational awareness are determined in a vehicle user-specific manner based on glance behavior analysis, wherein the vehicle user's glance behavior is compared with common glance behaviors. This comparison is particularly based on glance behaviors detected during manual driving and glance behaviors during autonomous driving, thereby determining whether the vehicle user has perceived predetermined relevant objects, such as traffic signs, during autonomous driving.

[0020] If it is determined, based on the vehicle user's glancing behavior, that there is no reaction capability specifically for taking over driving tasks, then at least one notification is issued inside the vehicle, and the vehicle user should respond to the notification within a predetermined time period.

[0021] This notification is issued to generate the vehicle user's responsiveness so that he can grant a takeover request. Attached Figure Description

[0022] The embodiments of the present invention will be explained in detail below with reference to the figures, wherein:

[0023] Figure 1 The diagram schematically illustrates three views showing the line-of-sight vectors of a vehicle occupant inside the vehicle.

[0024] Figure 2 The illustration shows vehicle users in two different locations and their corresponding line-of-sight vectors.

[0025] Figure 3 The front and side views schematically illustrate the starting points of the markers for the vehicle user and the line-of-sight vector.

[0026] Figure 4 This schematically illustrates the projection area onto the windshield of a vehicle user and an object in two different locations.

[0027] Figure 5 The illustration schematically depicts a method and process for determining the responsiveness of a vehicle user during autonomous driving.

[0028] Corresponding components are labeled with the same reference numerals in all the accompanying drawings. Detailed Implementation

[0029] Figure 1 Three different views are shown of a road segment F containing two lanes F1 and F2, in which a vehicle 1 is traveling in the right lane F1, and the... Figure 1 Shown in Figures 2 to 4 The image shows the line-of-sight vector V of vehicle user 2 (who is the driver of vehicle 1 in manual driving). Other vehicle 3 is also traveling in the right lane F1 in the top and middle views and in the left lane F2 in the bottom view.

[0030] The gaze vector V of vehicle user 2 is the same in all three views, where vehicle user 2 views different objects O respectively.

[0031] In the first view A1, the line-of-sight vector V is aligned with another vehicle 3 traveling in front of vehicle 1, which is object O.

[0032] In the second view A2, vehicle 1 intends to change lanes to the left lane F2, where vehicle 1 has already moved to the side of the road, so that the line of sight vector V is aligned with the left lane F2.

[0033] In the third view A3, other vehicle 3 is traveling in the left lane F2, and vehicle 1 is traveling in the right lane F1. The line of sight vector V of vehicle user 2 is straight forward, that is, aligned with the right lane F1.

[0034] The following describes a method that specifies: determining the responsiveness of vehicle user 2 in the autonomous driving of vehicle 1, wherein the glancing behavior of vehicle user 2 is detected and analyzed.

[0035] A driver monitoring camera (not shown in detail) is installed in vehicle 1, which continuously acquires image data during driving (whether automatically or manually), wherein the acquisition area of ​​the driver monitoring camera is directed at vehicle user 2.

[0036] The direction of view, i.e., the line-of-sight vector V, of vehicle user 2 is determined based on the image data captured by the driver's monitoring camera.

[0037] exist Figure 1 In particular, with the aid of three views A1 to A3, it is shown that the gaze vector V is insufficient to identify the object O and / or area that the vehicle user 2 is looking at outside the vehicle 1.

[0038] Based on the gaze vector V and the geometric data of vehicle 1, it can be determined whether vehicle user 2 is looking out of vehicle 1, for example, through the windshield 4 of vehicle 1, i.e., whether their gaze is directed towards the windshield 4. However, it is not possible to determine whether vehicle user 2's gaze is directed towards a specific object O and / or a specific area outside vehicle 1.

[0039] Using this method, it is determined whether vehicle user 2 sees object O based on the determined gaze vector V of vehicle user 2 and the object O detected outside vehicle 1 and its characteristics, such as its size and position.

[0040] When vehicle user 2 sees object O and / or area, it means that vehicle user 2's line of sight is directed at object O and / or area, and object O and / or area is within vehicle user 2's field of vision for the shortest possible duration.

[0041] The following describes a method related to the detection of object O by a vehicle user, wherein the detection of a region is performed similarly by means of this method.

[0042] Since it is necessary to determine how long vehicle user 2's gaze is focused on object O outside vehicle 1, it can be largely ruled out that vehicle user 2's unconscious, relatively rapid gaze, which contributes little or no to situational awareness, has not been taken into consideration.

[0043] To determine whether there is an object O outside of vehicle 1 within the field of vision of vehicle user 2, the gaze vector V is inferred to be directed outwards from vehicle 1. Specifically, when the inferred gaze vector V is as follows... Figure 2 When aligned with object O, object O is detected and perceived by vehicle user 2.

[0044] Figure 2 Specifically, the vehicle user 2 is shown in two seating positions S1 and S2, along with the corresponding inferred gaze vector V, a very simplified windshield 4 of vehicle 1, and two objects O outside vehicle 1. Here, vehicle user 2 is looking at the speed limit traffic sign 5, which is object O, in a lower seating position S1 and closer to the windshield 4, and is looking at other vehicles 3, which are objects, in another seating position S2, which is shown by dashed lines.

[0045] Vehicle user 2 looks at another object O in a corresponding sitting position, wherein their line of sight is aligned with the same point P about the windshield 4.

[0046] In cases where the object O has a relatively complex shape and / or size, the area surrounding the object O can be defined, such as... Figure 2 As shown. According to Figure 2 In this embodiment, the traffic sign 5, which is object O, is surrounded by a virtual rectangle R. Alternatively, other shapes can be set to define the area surrounding object O, wherein the smaller the area, the more accurate the object assignment.

[0047] Additionally, the visibility of object O is checked, especially when multiple objects O are located in the same area. For example, if two objects O are within the field of vision of vehicle user 2, then vehicle user 2's inferred line-of-sight vector V can be aligned with the two objects O. In this case, the more distant object O may be occluded by the closer object O belonging to vehicle user 2's line of sight.

[0048] The visibility of object O and its occlusion by another object O can be assessed using the location, size, and / or material properties of object O.

[0049] In one embodiment, object O or a region of object O is projected onto surfaces A1, A2, particularly the windshield 4, to determine whether object O is within the field of vision of vehicle user 2. Here, as... Figure 4 As shown, "the projection of object O onto surfaces A1 and A2 of the windshield 4" depends on many parameters, such as the line-of-sight vector V. Figure 3 The starting point C, the object O, especially their relative positions with respect to the starting point C, and the geometric data of surface A are shown.

[0050] Figure 3An exemplary determination of the starting point C of the gaze vector V is shown, wherein when the eyes 7 are looking straight ahead with the head 8 oriented forward, the starting point C is predetermined as the center point between the pupils 6 of the vehicle user 2.

[0051] If the position of the starting point C changes due to the positioning of the head 8, for example, changing laterally, then surfaces A1 and A2 will be positioned accordingly on the windshield 4. Figure 4 As shown in detail. Therefore, the size and posture of the vehicle user 2 are irrelevant when the object O is projected onto the surfaces A1 and A2 of the windshield 4.

[0052] Figure 4 The areas projected onto surfaces A1 and A2 of the windshield 4 by a vehicle user 2 in two different sitting positions S1 and S2 and an object O in the form of another vehicle 3 are shown.

[0053] Figure 4 A front view of the windshield 4 with two surfaces A1 and A2 of object O is also shown.

[0054] If the vehicle user 2 is in a seated position S1, the object O in the form of another vehicle 3 is projected onto surface A1 of the windshield 4, while the object O in another seated position S2 is projected onto another surface A2 of the windshield 4. Therefore, the object O is projected onto multiple surfaces A1, A2 of the windshield 4, particularly in relation to the corresponding seated positions S1, S2.

[0055] This method specifies glance behavior analysis in order to determine the degree and / or quality of perception. For this purpose, the glance behavior of vehicle user 2 is compared with their common glance behavior, particularly during manual driving of vehicle 1. Therefore, it can be determined, in particular, whether and / or with what degree and / or with what quality, vehicle user 2 perceives a predetermined relevant object O, such as a specific traffic sign 5, during autonomous driving of vehicle 1. To this end, during multiple instances of manual driving, vehicle user 2's glance behavior regarding the category of the predetermined relevant object O, such as traffic sign 5, can be compared with their glance behavior regarding object O.

[0056] If, for example, no glance is detected, or a brief glance toward a predetermined relevant object O is detected, especially a glance of less than the minimum duration, then it can be inferred that the vehicle user 2 did not perceive the predetermined relevant object O, and / or the perception level and / or perception quality is relatively low. In this case, it can be stipulated that the predetermined relevant object O, especially a traffic sign, is displayed in the display area of ​​the dashboard.

[0057] If, based on the glance behavior analysis, it is determined that vehicle user 2 has fully perceived the pre-determined relevant object O, then it is not displayed.

[0058] It is also conceivable that common glancing behaviors are identified not only in a way specific to vehicle users, but also by location and / or by situation. If vehicle user 2 frequently drives through the same road segment, then deviations in vehicle user 2's glancing behavior at specific locations within that road segment can be determined.

[0059] In one possible implementation, this method specifies that glancing behavior analysis is performed under certain working states and / or conditions of vehicle user 2. For example, vehicle user 2's glancing behavior may be analyzed to assess their perception and / or the quality of their perception or their state awareness when the system determines to request vehicle user 2 to take over the driving task.

[0060] The corresponding results can be taken into account by the system when transferring driving responsibility to vehicle user 2. For example, vehicle user 2 decides to perform a specific action, such as changing lanes, relatively suddenly. Here, based on glance behavior analysis, it is determined whether vehicle user 2 has not detected, i.e., seen, all objects O related to that action. For example, vehicle user 2 has not seen other vehicles 3 approaching from behind, in which case vehicle user 2 is consequently notified and / or the intended action is prohibited or not supported by the system.

[0061] Another implementation specifies that gaze data from multiple vehicle users 2 are used to determine the glancing behavior of the majority. This allows it to identify which objects O can be pre-determined as relevant objects O and with what perception level and / or quality of perception the majority of vehicle users 2 observe these objects O. For example, the objects O are other vehicles 3, traffic signs 5, light signaling devices, warning lights, lane markings, reflective posts, crash barriers, guide rails, bridge piers, etc.

[0062] Therefore, the vehicle user 2's glancing behavior is used to determine whether the vehicle user 2 has sufficient perception and / or sufficient perceived quality or sufficient situational awareness.

[0063] If this is the case, then vehicle user 2's responsiveness is assessed as present. If there is suspicion of reduced perception and / or reduced perceived quality or reduced situational awareness, then vehicle user 2's responsiveness is assessed as absent.

[0064] In particular, when vehicle user 2 fails to direct their gaze at one of the predetermined relevant objects O or for less than the minimum duration associated with the glance time, the responsiveness is rated as zero.

[0065] Alternatively or additionally, when the frequency of eye movement is below a minimum frequency, i.e., the determined frequency of eye movement is below a predetermined frequency threshold, responsiveness is rated as zero. The frequency of eye movement is determined such that the average duration of eye movement between different objects O is determined. The shorter this duration, the higher the frequency of eye movement.

[0066] A relatively low frequency of eye wandering could be a sign of distraction for vehicle user 2, which could stem from extra actions that are not permitted by autonomous driving. Extra actions are not allowed by autonomous driving because vehicle user 2 may be too focused on extra actions and therefore rarely, if ever, direct their gaze toward a predetermined, relevant object O outside vehicle 1.

[0067] As described above, the predetermined relevant object O is the object that vehicle user 2 sees under similar driving conditions during manual driving of vehicle 1 and is related to decisions about manual driving, i.e., vehicle control.

[0068] The location of such a related object O on the road segment currently being traveled by vehicle 1 is determined based on digital map data available on the vehicle side and / or sensor data collected on the vehicle side.

[0069] If it is determined, as described above, that the vehicle user 2 lacks responsiveness, then measures should be taken, particularly on the vehicle side, to regain or regenerate responsiveness.

[0070] The proposed action is to send a notification to vehicle user 2 within vehicle 1. Vehicle user 2 must respond to the notification within a predetermined time period to confirm their responsiveness. If no response is received within the predetermined time period and after one or more possible escalation phases, vehicle user 2 is requested to take over driving duties. If vehicle user 2 also fails to respond to the request within the predetermined time period, vehicle 1, which is in autonomous driving mode, will be selectively transitioned to a safe stop.

[0071] The same procedure is followed when vehicle user 2 is identified as dozing off based on image data collected by the driver's surveillance camera. Dozing off is indicated when vehicle user 2 closes their eyes 7 for several seconds (microsleep) or longer.

[0072] In addition to determining the frequency of vehicle user 2's eye movement, the image data captured by the driver's monitoring camera can also be evaluated to determine whether vehicle user 2 is in a position where he can take over the driving task of vehicle 1.

[0073] Based on facial recognition, the head 8 of vehicle user 2 is identified in the captured image data, and it is checked whether the head 8 is located within a predetermined effective area.

[0074] Furthermore, based on image processing methods related to the captured image data, illusions (Täuschung), such as those caused by images, dolls, etc., are identified to prevent misuse where vehicle user 2 pretends to be in their seat. Therefore, for example, it can be largely prevented that vehicle user 2 leaves their seat and climbs into the rear seat area of ​​vehicle 1.

[0075] If it is determined that vehicle user 2 is not in their seat or in a position where they can take over, then autonomous driving is terminated, or activation is not permitted if they are not activated.

[0076] It can also be stipulated that, as described in DE 10 2015 001 686 B4, the operation of vehicle user 2 is detected and the blinking and movement rate of vehicle user 2's head 8 are detected in order to determine the activity of vehicle user 2 and thereby infer its reaction ability.

[0077] Figure 5 This document presents an overview of a process for determining the responsiveness of a vehicle user 2 in autonomous driving of a vehicle 1 based on image data collected from a driver monitoring camera.

[0078] In the first method step V1, the presence of a predetermined relevant object O in the immediate vicinity in front of vehicle 1 is determined based on map data present on the vehicle side and / or sensor data collected on the vehicle side.

[0079] The second method step V2 specifies that the direction of the vehicle user 2's gaze is determined based on image data from the driver's monitoring camera. In the third method step V3, the pre-determined relevant objects O are classified, and in the fourth method step V4, the gaze objects are assigned. Specifically, the classification involves determining whether the relevant object O is a traffic sign 5, particularly a signpost, i.e., a common signpost, a traffic sign 5 with speed limits, or a traffic sign 5 with curve speed limits, or a vehicle ahead. In the case of a vehicle ahead, it is determined whether other vehicles 3 are traveling in vehicle 1's lane F1.

[0080] If it can be determined from the vehicle user 2's gaze direction that the predetermined relevant object O is located in the vehicle user 2's field of vision, then in the fifth method step V5, it is determined whether the vehicle user 2 has already grasped the characteristics related to the predetermined relevant object O based on the vehicle user 2's glancing behavior.

[0081] The memory of vehicle 1 stores data on common glancing behaviors of vehicle user 2, which is invoked in the sixth method step V6, wherein common glancing behaviors of vehicle user 2 related to the classified relevant object O are determined in the seventh method step V7.

[0082] In the eighth method step V8, the current glancing behavior of vehicle user 2 in autonomous driving is compared with the common glancing behavior, especially in manual driving of vehicle 1.

[0083] Next, in the ninth method step V9, the perception level and / or perception quality are evaluated, and the perception level and / or perception quality are compared with the target perception level or target perception quality in the tenth method step V10.

[0084] The eleventh method step V11 specifies that the perception of a predetermined relevant object O is confirmed, wherein, in the absence of confirmation, the responsiveness of the vehicle user 2 is assessed as zero.

Claims

1. A method for determining the reaction ability of a vehicle user (2) in autonomous driving of a vehicle (1) based on image data collected by a driver monitoring camera, characterized in that, - Determine whether at least one pre-determined relevant object (O) in the current vehicle environment is within the field of vision of the vehicle user (2) based on digital map data present on the vehicle side and / or sensor data collected on the vehicle side. - For at least one predetermined relevant object (O) detected in the current vehicle environment, determine, based on image data collected by the driver monitoring camera, the duration and / or frequency at which the vehicle user (2) directs their gaze toward the at least one predetermined relevant object (O) detected, and -Based on the comparison of the glancing behavior of the vehicle user (2) in autonomous driving according to a predetermined location or condition with the glancing behavior of the vehicle user (2) in manual driving according to a predetermined location or condition, the vehicle user (2)'s reaction ability is rated as zero when the duration is determined to be less than the predetermined minimum duration and / or when the frequency of gaze wandering is determined to be less than the minimum frequency.

2. The method according to claim 1, characterized in that, An object (O) is pre-determined as relevant when it is in the field of vision of the vehicle user (2) for a predetermined duration and / or frequency during manual driving of the vehicle (1) and when the object (O) is considered by the vehicle user (2) during vehicle control.

3. The method according to claim 1 or 2, characterized in that, At least one other vehicle (3), traffic sign (5), light signaling device, warning light, lane marking, reflector post, guide rail, crash barrier and / or bridge pier located in front of, to the side and / or behind the vehicle (1) is pre-determined and / or considered as a relevant object (O) in determining the responsiveness of the user (2) of the vehicle.

4. The method according to claim 1 or 2, characterized in that, The size, visibility, relative position and / or absolute position of a predetermined object (O) related to the field of vision of the vehicle user (2) are identified and / or considered.

5. The method according to claim 1 or 2, characterized in that, The gaze direction of the vehicle user (2) is determined as a line-of-sight vector (V), which is inferred to determine the field of view.

6. The method according to claim 5, characterized in that, The center point between the pupils (6) of the vehicle user (2) is predetermined as the starting point (C) of the gaze vector (V).

7. The method according to claim 1 or 2, characterized in that, Define the area around the relevant object (O) based on its predetermined shape and / or size.

8. The method according to claim 1 or 2, characterized in that, For multiple known objects (O), the visibility associated with a pre-determined relevant object (O) is evaluated based on the object (O)’s location, size and / or material properties.

9. The method according to claim 1 or 2, characterized in that, Based on glance behavior analysis, perception and / or situational awareness are determined in a way that is specific to vehicle users.

10. The method according to claim 1 or 2, characterized in that, When it is determined that the vehicle user (2) is unresponsive, at least one notification is output to the vehicle (1), and the vehicle user (2) should respond to the at least one notification within a predetermined time period.

Images