Methods for determining responsiveness

Abstract

Method for determining the reaction capability of a vehicle user (2) in the automated driving mode of a vehicle (1) using image data captured by a driver observation camera, wherein, - based on digital map data available on the vehicle and / or sensor data recorded on the vehicle, it is determined whether at least one predefined relevant object (O) is located in the vehicle user's (2) field of vision in a current vehicle environment, - for at least one recorded predefined relevant object (O) in the current vehicle environment, it is determined, based on recorded image data from the driver observation camera, for what duration and / or frequency the vehicle user (2) directs his gaze to the recorded at least one predefined relevant object (O) and - if the reaction time of the vehicle user (2) is determined to fall below a specified minimum duration and / or if the frequency of eye movements is determined to fall below a minimum threshold, the vehicle user's (2) reaction time is assessed as not being possible, characterized in that an object (O) is specified as relevant if such an object (O) is in the vehicle user's (2) field of vision for a specified duration and / or frequency during manual operation of the vehicle (1) and the object (O) is taken into account when the vehicle user (2) controls the vehicle.

Description

[0001] The invention relates to a method for determining the reaction capability of a vehicle user in automated driving mode of a vehicle using image data captured by a driver observation camera.

[0002] German patent application DE 10 2015 001 686 B4 discloses a method and a device for detecting a driver's reaction time during automated driving of a motor vehicle. The method involves a first assessment of reaction time by means of camera observation of the driver, followed by a second assessment by detecting the driver's operation of a control unit. Reaction time is classified as present if the first and / or the second assessment determines that it is present. In the first assessment of reaction time by means of camera observation, it is classified as present if at least one of the driver's eyes is open for a minimum portion of a predetermined time interval. Reaction time is not classified as present if the control unit is operated by someone other than the driver or if the operation is rhythmic.

[0003] Furthermore, DE 10 2016 011 246 A1 describes a method for monitoring the condition of at least one occupant of a vehicle. The method involves illuminating the occupant's head with at least one light source, which is alternately switched on and off. Light reflected from the occupant's eyes is detected by at least one sensor, which also checks whether the reflection changes depending on the illumination by the light source and whether at least eyelid activity and / or head movement of the occupant is detected.

[0004] The invention is based on the objective of providing a method for determining the reaction capability of a vehicle user in the automated driving operation of a vehicle.

[0005] German patent DE 10 2019 201 407 A1 discloses a method for controlling the attention of a driver of an ego vehicle. If it is detected that a driver is not perceiving critical traffic objects, a warning signal is issued.

[0006] According to DE 10 2019 115 455 A1, data from a vehicle's sensors are collected along with data tracking the driver's gaze. The route traveled by the vehicle can also be recorded. The driver's gaze is analyzed in relation to the sensor data to determine a feature on which the driver was focused. A focus dataset is created for this feature. Focus datasets for multiple drivers can be aggregated to determine the frequency of viewing of the feature. A machine learning model can be trained using these focus datasets to identify an area of ​​interest for a given scenario, enabling faster detection of relevant hazards.

[0007] DE 10 2011 084 367 A1 describes a method for determining whether a driver has perceived an object in the vicinity of the motor vehicle. The method checks whether the driver's gaze is directed at the object. The determination of whether the driver has perceived the object depends on: -a duration of a gaze directed at the object, -a number, how often the gaze falls on the same object or -a frequency, how often the driver's gaze falls on the same object per unit of time.

[0008] German patent DE 10 2019 109 375 A1 discloses a method for processing attention data. The method comprises receiving object data associated with at least one object in the vehicle's external environment, receiving impending behavior data determined from a planned route of the vehicle, and receiving gaze data captured by a vehicle occupant. To determine an occupant's attention score, a processor processes the object data, the impending behavior data, and the gaze data.

[0009] From DE 10 2017 214 535 A1, a method is known for providing a set of predefined areas, each representative of a specific viewing area. The driver's gaze direction is determined. A set of attention values ​​is continuously determined, with each attention value representing the driver's attention to a specific area within the set of predefined areas. The set of attention values ​​is determined by ascertaining, based on the determined gaze direction, whether the driver is looking into the respective predefined area. If it is determined that the driver is looking into the respective predefined area, the corresponding attention value is calculated using a predefined loading function.If it is determined that the driver is not looking in the designated area, the corresponding attention score is calculated using a predefined discharge function. The calculated amount of attention scores is then used for a specific driver assistance system.

[0010] DE 10 2017 202 194 A1 discloses a method for determining a virtual distraction of a vehicle driver. The method models the driver's attention by creating an attention map using driver information that represents the driver's gaze direction and / or head position and / or upper body orientation, as detected by a sensor in the vehicle. The attention map represents the actual distribution of the driver's attention in a two-dimensional coordinate system, dependent on the gaze direction and / or head position, and the two-dimensional coordinate system depicts at least a sub-area of ​​the vehicle's environment.The procedure further involves modeling attention requirements by creating an attention requirement map, where the attention requirement map represents a target distribution of the driver's attention in the two-dimensional coordinate system. The attention map is then compared with the attention requirement map to determine a distraction value representing the driver's visual distraction.

[0011] The problem is solved according to the invention by a method which has the features specified in claim 1.

[0012] Advantageous embodiments of the invention are the subject of the dependent claims.

[0013] A method for determining a vehicle user's reaction time during automated driving, based on image data from a driver observation camera, involves using digital map data and / or sensor data from the vehicle to determine whether at least one predefined object is within the user's field of vision in the current vehicle environment. If at least one predefined object is detected in the current vehicle environment, the image data from the driver observation camera is used to determine the duration and / or frequency with which the vehicle user focuses their gaze on this object. If the duration and / or frequency of gaze changes falls below a predefined minimum, the vehicle user's reaction time is assessed as insufficient.

[0014] By applying this method, the vehicle operator's reaction time can be determined during automated driving mode, in which the vehicle fully performs a driving task. A vehicle operator is defined as a person who performs a driving task of the vehicle in manual driving mode.

[0015] In particular, reaction time is assessed to determine whether the vehicle user can react appropriately to a system-induced takeover request regarding the driving task and correctly assess the existing situation. Situational awareness requires the ability to see and perceive relevant objects and / or areas outside the vehicle, depending on the situation.

[0016] According to the invention, an object is deemed relevant if such an object is in the vehicle user's field of vision for a predetermined duration and / or frequency during manual driving of the vehicle, i.e., if the vehicle user focuses their gaze on the object for the predetermined duration and / or frequency, and if the object is taken into account when the vehicle is controlled by the vehicle user.

[0017] If no glance or a comparatively short glance by the vehicle user at a given relevant object is detected based on the recorded image data, it is interpreted that the vehicle user did not perceive the given relevant object and therefore exhibits at least a low level of responsiveness.

[0018] In one possible embodiment of the method, relevant objects include at least one other vehicle located in front of, to the side of and / or behind the vehicle, a traffic sign, a traffic signal system, a warning light, a lane marking, a guidepost, guardrails, guide beacons and / or bridge piers, which are specified and / or taken into account when determining the vehicle user's reaction capability.

[0019] Therefore, objects are considered relevant and are specified as such if they can and / or should influence vehicle control and thus the driving style of the vehicle user in manual driving mode.

[0020] A further development of the procedure involves determining and / or taking into account the size, visibility, relative position, and / or absolute position of a given object in relation to the vehicle user's field of vision. This allows for the assessment of whether the vehicle user is actually able to see such a given object from inside the vehicle.

[0021] One design of the procedure involves determining the vehicle user's gaze direction as a gaze vector, which is then extrapolated to determine the field of vision. Specifically, the gaze vector is extrapolated outside the vehicle, and it is determined whether the extrapolated gaze vector intersects the specified relevant object and whether the specified relevant area is therefore within the vehicle user's field of vision.

[0022] In particular, a possible further development of the method envisages specifying the midpoint between the vehicle occupant's pupils as the starting point of the gaze vector. Specifically, the midpoint between the pupils is used as the starting point when the vehicle occupant's eyes are looking straight ahead with their head facing forward. Defining the starting point of the gaze vector serves to transfer the specified relevant object as an area, particularly onto a windshield, and to determine, based on this area, whether the vehicle occupant is looking at this area.

[0023] In further training, a zone around a given relevant object is defined based on its shape and / or size. In particular, a zone is defined around the given relevant object if it has a comparatively complex shape and / or size. The smaller this defined zone, the more precise the gaze-to-object correlation, allowing detection of whether the given relevant object is within the vehicle user's field of vision and can be recognized as such by the vehicle user.

[0024] If multiple objects are detected, then, in a possible training scenario, visibility in relation to a predefined relevant object is assessed based on the object's position, size, and / or material properties. This means that the visibility of the predefined relevant object is checked to ensure, as far as possible, that the predefined relevant object is within the vehicle user's field of vision and not obscured by another object. In other words, it is checked whether the predefined relevant object is visible to the vehicle user.

[0025] In a potential training course, a user-specific level of perception and / or situational awareness is determined using gaze behavior analysis, comparing the user's gaze behavior with typical gaze behavior. This comparison is based specifically on the user's detected gaze behavior during manual driving and during automated driving, allowing it to be determined whether the user has perceived a given relevant object, such as a traffic sign, while the vehicle is in automated driving mode.

[0026] If, based on the vehicle user's gaze behavior, it is determined that the ability to react, in particular to take over the driving task, is not present, at least a message will be issued in the vehicle, which the vehicle user must acknowledge within a specified time period.

[0027] The message is issued to restore the vehicle user's ability to respond, so that they are able to comply with a takeover request.

[0028] Exemplary embodiments of the invention are explained in more detail below with reference to drawings.

[0029] This shows: Fig. 1. Schematic representation of three views showing the viewing vector of a vehicle user in a vehicle. Fig. 2 schematically depicts a vehicle user in two different positions with their respective viewing vectors, Fig. 3. Schematic representation of a front and side view of a vehicle user with a marked starting point of a viewing vector. Fig. 4 schematically depicts a vehicle user in two different positions and a transferred area of ​​an object on a windshield and Fig. 5 schematically illustrates a procedure for determining the reaction capability of a vehicle user in automated driving mode of a vehicle.

[0030] Corresponding parts are marked with the same reference symbols in all figures.

[0031] In Fig. Figure 1 shows three different views of a road section F comprising two lanes F1 and F2, with a vehicle 1 driving in a right-hand lane F1 and a view vector V of a vehicle in the Fig. 2, Fig. 3 to Fig. The vehicle user 2 shown in Figure 4 is the driver of vehicle 1 in manual mode. Another vehicle 3 is also shown driving in the right lane F1 in a top view and a middle view, and in the left lane F2 in a bottom view.

[0032] The viewing vector V of vehicle user 2 is the same in all three views, with vehicle user 2 observing different objects O in each view.

[0033] In the first view A1, the viewing vector V is directed towards the other vehicle 3 as object O, which is driving in front of vehicle 1.

[0034] In the second view A2, vehicle 1 intends to change lanes to the left lane F2, whereby vehicle 1 has already swerved out, so that the viewing vector V is directed towards the left lane F2.

[0035] In the third view A3, the other vehicle 3 is driving on the left lane F2 and the vehicle 1 is driving on the right lane F1, with the viewing vector V of the vehicle user 2 directed straight ahead, i.e. towards the right lane F1.

[0036] The following describes a procedure which aims to determine the reaction capability of the vehicle user 2 in the automated driving operation of the vehicle 1, whereby the gaze behavior of the vehicle user 2 is recorded and analyzed.

[0037] The vehicle 1 is equipped with a driver observation camera (not shown in detail) which continuously records image data during driving, regardless of whether the driving is automated or manual, with one of the camera's detection ranges directed towards the vehicle user 2.

[0038] Based on the image data recorded by the driver observation camera, the direction of view of the vehicle user 2, i.e. the viewing vector V, is determined.

[0039] In Fig. Figure 1 illustrates, in particular, by means of the three views A1 to A3, that the gaze vector V is insufficient for the recognition of an object O and / or an area outside the vehicle 1, towards which the gaze of the vehicle user 2 is directed.

[0040] Based on the gaze vector V and geometric data of vehicle 1, it can be determined whether the vehicle user 2 is looking outside of vehicle 1, e.g., through a windshield 4 of vehicle 1, i.e., whether their gaze is directed towards the windshield 4. However, it is not possible to determine whether the vehicle user 2's gaze is directed at a specific object O and / or at a specific area outside of vehicle 1.

[0041] Using the method, based on the determined gaze vector V of the vehicle user 2 and an object O detected outside the vehicle 1 and its properties, such as its size and position, it is determined whether the vehicle user 2 has seen this object O.

[0042] The fact that the vehicle user 2 sees an object O and / or an area means that the vehicle user 2's gaze is directed towards this object O and / or this area and that the object O and / or the area is in the vehicle user 2's field of vision for a minimum period of time.

[0043] The procedure for detecting an object O by the vehicle user is described below, whereby the detection of an area is carried out analogously using the procedure.

[0044] By determining how long the vehicle user 2's gaze is directed at the object O outside the vehicle 1, it can be largely ruled out that unconscious, comparatively quick glances by the vehicle user 2, which contribute little or not at all to the perception of a situation, are not taken into account.

[0045] To determine whether an object O is located outside vehicle 1 and within the field of vision of vehicle user 2, the gaze vector V is extrapolated to outside vehicle 1. Specifically, an object O is detected and perceived by vehicle user 2 when the extrapolated gaze vector V intersects the object O, as shown in Fig. 2 is shown.

[0046] In particular, it shows Fig. 2 the vehicle user 2 in two seating positions S1, S2 and corresponding extrapolated viewing vectors V, a highly simplified windshield 4 of the vehicle 1 and two objects O outside the vehicle 1. In this case, the vehicle user 2, in a lower seating position S1 located closer to the windshield 4, observes a traffic sign 5 for speed limit as object O and, in the other further seating position S2 shown by a dashed line, another vehicle 3 as object.

[0047] The vehicle user 2 observes a different object O in the respective seating position, whereby his gaze is directed towards one and the same point P in relation to the windshield 4.

[0048] For objects O with a relatively complex shape and / or size, a region around the object O can be defined, such as... Fig. 2 shows. According to the present embodiment in Fig. 2. Traffic sign 5 is defined as object O and surrounded by a virtual rectangle R. Alternatively, a different shape can be defined as the area surrounding object O; the smaller this area, the more accurate the view-to-object correspondence.

[0049] Additionally, the visibility of an object O is checked, especially if several objects O are located in the same area. For example, if two objects O are within the field of vision of vehicle user 2, the extrapolated gaze vector V of vehicle user 2 can intersect both objects O. In such a case, a more distant object O is presumably obscured by a closer object O, to which the gaze of vehicle user 2 is attributed.

[0050] To assess the visibility and occlusion of an object O by another object O, the position, size and / or material properties of the objects O can be used.

[0051] In one embodiment, the object O or the area of ​​the object O is transferred to a surface A1, A2, in particular the windshield 4, in order to determine whether the object O is in the field of vision of the vehicle user 2. The transfer of the object O to the surface A1, A2 of the windshield 4 is as described in Fig. As shown in section 4, depending on a number of parameters, such as one in Fig. 3 shown starting point C of the viewing vector V and a relative position of the object O, in particular to the starting point C, as well as geometric data of the area A.

[0052] An exemplary definition of the starting point C of the viewing vector V shows Fig. 3, where the starting point C is defined as the midpoint between the pupils 6 of the vehicle user 2 when the eyes 7 are looking straight ahead with a frontally oriented head 8.

[0053] If the position of the starting point C changes due to the positioning of the head 8, for example in a lateral direction, the surface A1 A2 on the windshield 4 positions itself accordingly, as shown in Fig. 4 is shown in more detail. For this reason, the size and seating position of the vehicle user 2 play an indirect role in this transfer of object O onto the surface A1, A2 of the windshield 4.

[0054] Fig. Figure 4 shows the vehicle user 2 in two different seating positions S1, S2 with the area of ​​object O in the form of another vehicle 3 transferred onto a surface A1, A2 of the windshield 4.

[0055] Furthermore, in Fig. 4 a front view of the windshield 4 with the two surfaces A1, A2 of the object O shown.

[0056] If the vehicle user 2 is in seating position S1, the object O, in the form of the other vehicle 3, is transferred to a surface A1 of the windshield 4, whereas in the other seating position S2, the object O is transferred to another surface A2 of the windshield 4. The object O is thus transferred to several surfaces A1, A2 of the windshield 4, particularly with respect to the respective seating positions S1 and S2.

[0057] The procedure involves an analysis of gaze behavior to determine the degree and / or quality of perception. For this purpose, the gaze behavior of vehicle user 2 is compared with their usual gaze behavior, particularly during manual driving of vehicle 1. This allows, in particular, the determination of whether and / or with what degree and / or quality of perception vehicle user 2 perceived a given relevant object O, e.g., a specific traffic sign 5, during automated driving of vehicle 1. To this end, the gaze behavior of vehicle user 2 with regard to a category of the given relevant object O, e.g., traffic sign 5, can be compared with their gaze behavior with regard to object O during several manual driving sessions.

[0058] If, for example, no glance at all or a comparatively brief glance at the specified relevant object O is detected, especially a glance that falls below the minimum duration, it can be deduced that the vehicle user 2 did not perceive the specified relevant object O and / or that the degree and / or quality of perception is comparatively low. In this case, it may be possible to display the specified relevant object O, in particular a traffic sign, in a display area of ​​an instrument panel.

[0059] However, if the gaze behavior analysis determines that the vehicle user 2 has sufficiently perceived the specified relevant object O, it will not be displayed.

[0060] It is also conceivable that typical gaze behavior is determined not only in relation to the vehicle user but also depending on location and / or situation. If vehicle user 2 frequently drives the same routes, a different gaze behavior of vehicle user 2 at a specific point on each route can be observed.

[0061] In one possible implementation, the procedure envisages conducting the gaze behavior analysis in specific operating states and / or states of vehicle user 2. For example, the gaze behavior of vehicle user 2 can be analyzed to assess their level of perception and / or their perception quality or their situational awareness in the event of a system-induced takeover request of the driving task by vehicle user 2.

[0062] A corresponding result can be taken into account by the system when driving responsibility is transferred to vehicle user 2. For example, vehicle user 2 decides relatively suddenly to perform a specific action, such as a lane change. Based on, in particular, gaze behavior analysis, it is determined that vehicle user 2 has not perceived, i.e., seen, all objects O relevant to this action. For example, vehicle user 2 has not seen another vehicle 3 approaching from behind, in which case vehicle user 2 is informed of this and / or the intended action is suppressed or not supported by the system.

[0063] Another embodiment involves using gaze data from multiple vehicle users 2 to determine the gaze behavior of a majority. This allows for the identification of which objects O can be defined as relevant objects O and with what degree and / or quality of perception most vehicle users 2 view these objects O. These objects O include, for example, other vehicles 3, traffic signs 5, traffic lights, warning lights, lane markings, guideposts, guide beacons, guardrails, bridge piers, etc.

[0064] The gaze behavior of vehicle user 2 is used to determine whether vehicle user 2 has a sufficient level of perception and / or a sufficient quality of perception or a sufficient awareness of the situation.

[0065] If this is the case, the reaction ability of vehicle user 2 is assessed as sufficient. If there is suspicion of a reduced level and / or quality of perception or reduced situational awareness, the reaction ability of vehicle user 2 is assessed as insufficient.

[0066] In particular, the ability to react is assessed as not being given if the vehicle user 2, for example, does not direct his gaze towards one of the specified relevant objects O or if the minimum time period with regard to a gaze duration is not met.

[0067] Alternatively or additionally, responsiveness is assessed as not being present if the frequency of gaze shifts falls below a minimum threshold, i.e., if the measured frequency of gaze shifts falls below a predefined threshold. The frequency of gaze shifts is determined by calculating the average time O between shifts of gaze to different objects. The shorter this time, the higher the frequency of gaze shifts.

[0068] A comparatively low frequency of gaze shifts can indicate that the vehicle user 2 is distracted, and this distraction may result from a secondary activity that is not permitted in automated driving mode. A secondary activity is not permitted in automated driving mode because the vehicle user 2 may be concentrating too heavily on the secondary activity and thus rarely, if ever, directs their gaze to a given relevant object O outside the vehicle 1.

[0069] As described above, the specified relevant objects O are objects O that the vehicle user 2 observes in a similar driving situation during manual driving operation of the vehicle 1 and that are relevant for decision-making regarding manual driving operation, i.e., for vehicle control.

[0070] The location of such relevant objects O on a section of road currently being traveled by vehicle 1 is determined using digital map data available on the vehicle and / or sensor data recorded on the vehicle.

[0071] If, as described above, it is determined that the vehicle user 2 is unable to react, measures will be initiated to regain or restore the ability to react, in particular on the vehicle side.

[0072] One measure to be initiated is the issuance of a message from vehicle 1 to vehicle user 2, which vehicle user 2 must acknowledge within a specified time period to confirm their ability to respond. If the required acknowledgment is not received within the specified time period, and even after one or more escalation levels, vehicle user 2 is prompted to take over the driving task. If vehicle user 2 also fails to comply with this prompt within a specified time period, vehicle 1 is brought to a controlled and safe standstill in automated driving mode.

[0073] The same procedure is followed if a sleep event of vehicle user 2 is detected based on image data recorded by the driver observation camera. A sleep event occurs when vehicle user 2 has closed their eyes 7 for a period of seconds (microsleep) or longer.

[0074] In addition to determining the frequency of the vehicle user 2's gaze changes, the recorded image data from the driver observation camera can also be evaluated to determine whether the vehicle user 2 is in a takeover position, i.e., whether he is in a position from which the vehicle user 2 can take over the driving task of vehicle 1.

[0075] Using facial recognition, the head 8 of the vehicle user 2 is recognized in the recorded image data and it is checked whether the head 8 is located within a predefined valid area.

[0076] Furthermore, image processing techniques are used to detect deceptions in the recorded image data, e.g., through the use of pictures, mannequins, etc., in order to prevent cases of misuse where vehicle user 2 feigns their presence in their vehicle seat. This makes it possible, for example, to largely rule out the possibility that vehicle user 2 leaves their vehicle seat and climbs into the rear compartment of vehicle 1.

[0077] If it is determined that the vehicle user 2 is not in their vehicle seat or in a takeover position, automated driving operation will be terminated, or if it is not activated, it will not be enabled for activation.

[0078] Additionally, it may be provided that, as described in DE 10 2015 001 686 B4, operating actions of the vehicle user 2 are recorded and a rate of eye blinks and head movements 8 of the vehicle user 2 are recorded in order to determine an activity of the vehicle user 2 and to draw conclusions about his reaction ability.

[0079] In Fig. Figure 5 is an overview of the procedure for determining the reaction capability of the vehicle user 2 in the automated driving operation of the vehicle 1 using image data recorded by the driver observation camera.

[0080] In a first process step V1, it is determined, based on map data available on the vehicle and / or on sensor data recorded on the vehicle, that a predefined relevant object O is located in the immediate vicinity in front of the vehicle 1.

[0081] A second process step, V2, involves determining the direction of view of the vehicle user 2 based on the image data from the driver observation camera. In a third process step, V3, the captured, predefined relevant object O is classified, and in a fourth process step, V4, a view-to-object assignment is performed. Specifically, the classification determines whether the relevant object O is a traffic sign 5, in particular a sign (i.e., a standard sign), a traffic sign 5 with a speed limit, or a traffic sign 5 with a speed limit in a curve, or a vehicle ahead. In the case of a vehicle ahead, a distinction is made as to whether the other vehicle 3 is traveling in a lane F1 shared with vehicle 1.

[0082] If it can be determined from the direction of view of the vehicle user 2 that the specified relevant object O is in the field of view of the vehicle user 2, in a fifth procedure step V5 it is determined whether it can be determined from the gaze behavior of the vehicle user 2 whether the vehicle user 2 has perceived properties in relation to the specified relevant object O.

[0083] In a memory of the vehicle 1, data of the ordinary viewing behavior of the vehicle user 2 are stored, which are called up in a sixth procedure step V6, wherein in a seventh procedure step V7 the ordinary viewing behavior of the vehicle user 2 in relation to the classified relevant object O is determined.

[0084] In an eighth process step V8, the current gaze behavior of the vehicle user 2 in automated driving operation is compared with the ordinary gaze behavior, especially in manual driving operation of the vehicle 1.

[0085] Subsequently, in a ninth procedural step V9, the degree of perception and / or the quality of perception is assessed, which is then compared or compared with a target degree of perception or a target quality of perception in a tenth procedural step V10.

[0086] An eleventh procedural step V11 provides for confirmation of the perception in relation to the specified relevant object O, whereby if confirmation does not occur, the reaction capability of the vehicle user 2 is assessed as not being given. Reference symbol list 1 vehicle 2 vehicle users 3 more vehicles 4 Windscreen 5 traffic signs 6 Pupil 7 Eye 8 heads A1, A2 1Area C Starting point F Roadway section F1, F2 lane O object R rectangle S1, S2 Seating position V View vector V1 to V11 Process step

Claims

[1] Method for determining the reaction capability of a vehicle user (2) in automated driving mode of a vehicle (1) using image data captured by a driver observation camera, wherein, - based on digital map data available on the vehicle and / or sensor data recorded on the vehicle, it is determined whether at least one predefined relevant object (O) is located in the vehicle user's (2) field of vision in a current vehicle environment, - for at least one recorded predefined relevant object (O) in the current vehicle environment, it is determined, based on recorded image data from the driver observation camera, for what duration and / or frequency the vehicle user (2) directs his gaze to the recorded at least one predefined relevant object (O) and - if the duration of eye movements falls below a specified minimum and / or the frequency of eye movements falls below a minimum, the vehicle user's (2) reaction time is assessed as not being sufficient, characterized by , that an object (O) is presumed to be relevant if such an object (O) is in the field of vision of the vehicle user (2) for a predefined period of time and / or frequency during manual driving of the vehicle (1) and the object (O) is taken into account when the vehicle user (2) controls the vehicle. [2] Method according to claim 1, characterized by, that as relevant objects (O) at least one other vehicle (3) located in front of, to the side of and / or behind the vehicle (1), a traffic sign (5), a traffic signal system, a warning light, a lane marking, a guidepost, guardrails, guide beacons and / or bridge piers are specified and / or taken into account as objects (O) when determining the reaction capability of the vehicle user (2). [3] Method according to any one of the preceding claims, characterized by , that a size, a visibility, a relative position and / or an absolute position of a given object (O) in relation to the field of vision of the vehicle user (2) is determined and / or taken into account. [4] Method according to any one of the preceding claims, characterized by , that a direction of view of the vehicle user (2) is determined as a viewing vector (V), which is extrapolated to determine the field of view. [5] Method according to claim 4 characterized by, that the starting point (C) of the gaze vector (V) is specified as a midpoint between the pupils (6) of the vehicle user (2). [6] Method according to any one of the preceding claims, characterized by , that depending on the shape and / or size of a given relevant object (O), an area is defined around it. [7] Method according to any one of the preceding claims, characterized by , that for multiple detected objects (O), visibility in relation to a given relevant object (O) is evaluated based on the position, size and / or material properties of the object (O). [8] Method according to any one of the preceding claims, characterized by , that a level of perception and / or situational awareness specific to vehicle users is determined or will be determined based on an analysis of eye-tracking behavior. [9] Method according to any one of the preceding claims, characterized by, that if it is determined that the vehicle user (2) is unable to react, at least a message will be issued in the vehicle (1), which the vehicle user (2) must acknowledge within a specified time period.

Images