Device and method for operating a vehicle

Abstract

The invention relates to a device (100) for operating a vehicle (102), comprising a surroundings detection unit (106), which is designed to detect at least one item of surroundings information relating to the surroundings of the vehicle (102) and to generate surroundings data corresponding to the surroundings information, and an occupant detection unit (114), which is designed to detect the surroundings of the vehicle (102), the occupant detection unit is designed to capture at least one image depicting a vehicle driver (104) of the vehicle (102) and to generate driver image data corresponding to the image. Furthermore, the device (100) comprises a control unit (116) which is designed to ascertain at least one driving maneuver of the vehicle (102) on the basis of the surroundings data, the at least one driving maneuver is to be carried out by the vehicle (102) in an assisted or automated driving framework and maneuver data corresponding to the ascertained driving maneuver is generated. Furthermore, the control unit (116) is designed to determine, on the basis of the maneuver data, at least two maneuver-specific attention areas and at least one maneuver-specific attention criterion relating to a gaze activity of the vehicle driver (104) in the field of view of the vehicle driver (104), and to determine, on the basis of the maneuver data, at least two maneuver-specific attention areas and at least one maneuver-specific attention criterion relating to a gaze activity of the vehicle driver (104), on the basis of the driver image data, it is ascertained whether and how long the line of sight of the vehicle driver (104) is directed towards the region of interest, gaze activity data corresponding to the result of this gaze activity ascertainment is generated, the gaze activity data is compared with the attention criterion, and the attention criterion is determined as a result of this gaze activity ascertainment. And generating a control command for the assisted or automated driving of the vehicle (102) at least on the basis of such a comparison.

Description

Technical Field

[0001] The invention relates to a device for operating a vehicle. The invention also relates to a method for operating a vehicle. Background Art

[0002] In assisted driving or automated driving, specific driving functions of the vehicle are assumed by the driving assistance system. Unlike autonomous driving, in assisted or automated driving, the vehicle driver must be able to take over the driving task in a short time. For this reason, during assisted or automated driving, the vehicle driver must always pay attention to the traffic conditions around the vehicle. For this reason, the driving assistance system for assisted driving or automated driving usually has a system for monitoring the attention of the vehicle driver.

[0003] A lane change assistant for a vehicle is known from FR 3 047 957 A1. The lane change assistant is designed to inhibit certain maneuvers of the vehicle depending on the driver's viewing direction.

[0004] DE 10 2021 201 062 A1 discloses a method for operating a motor vehicle. In the method, a comparison is made to see whether the driver's line of sight is consistent with a set line of sight. If the driver's line of sight is inconsistent with the set line of sight, a warning sound is output and / or a vehicle function, in particular a partially automated lane change, is not triggered.

[0005] A system for operating a motor vehicle based on the direction of sight is known from DE 10 2013 007 257 A1. The system is designed to determine whether the driver's direction of sight is directed to a predetermined rear area when the reverse gear is engaged. Summary of the invention

[0006] The object of the present invention is to specify a device and a method for operating a vehicle which, compared to previously known devices and methods, allow better monitoring of the attentiveness of a vehicle driver during assisted or automated driving.

[0007] This object is achieved by a device having the features of claim 1 and a method having the features of the independent method claim. Advantageous developments are given in the dependent claims.

[0008] The proposed device for operating a vehicle comprises an environment detection unit and an occupant detection unit, wherein the environment detection unit is designed to detect at least one environment information related to the surroundings of the vehicle and generate environment data corresponding to the environment information, and the occupant detection unit is designed to detect at least one image of a vehicle driver of the vehicle and generate driver image data corresponding to the image. In addition, the device comprises a control unit, which is designed to determine at least one driving maneuver of the vehicle based on the environment data, the at least one driving maneuver to be performed by the vehicle within the framework of assisted driving or automated driving, and to generate maneuver data corresponding to the determined driving maneuver. In addition, the control unit is designed to determine at least two maneuver-specific attention areas in the field of vision of the vehicle driver and at least one maneuver-specific attention criterion related to the gaze activity of the vehicle driver based on the maneuver data, determine whether and for how long the vehicle driver's line of sight is respectively directed to the attention areas based on the driver image data, generate gaze activity data corresponding to the result of the gaze activity determination, compare the gaze activity data with the attention criterion, and generate a control command for assisted driving or automated driving based at least on the basis of the comparison.

[0009] The device monitors whether the vehicle driver is attentively following the assisted or automatic driving of the vehicle, so that he can intervene quickly if necessary. This monitoring is carried out specifically for the maneuver in that the control unit determines a specific attention area for each maneuver to be performed. Since the attention monitoring is carried out specifically for the maneuver, the attention monitoring can be carried out more targeted and therefore more reliably. The dynamic generation of the attention area also makes it possible to better take into account specific given conditions in the surroundings of the vehicle than in the case of known devices, in particular in urban traffic at intersections, in roundabouts and in the case of maneuvers at low speeds.

[0010] In order to perform attention monitoring specific to maneuvers, the control unit first determines, based on the surrounding data, which maneuver the vehicle should perform. To this end, the control unit determines the future trajectory of the vehicle, for example, based on the road course and the traffic conditions in the surroundings of the vehicle. Then, the control unit generates an attention area in the driver's field of vision according to the maneuver to be performed. Here, in this article, the field of vision of the vehicle driver is understood as such an area inside and outside the vehicle: the area can be seen by the vehicle driver, even if the vehicle driver must, for example, turn his head for this purpose. In addition, the control unit determines an attention criterion. Here, the attention criterion can be specified in advance for a certain maneuver or generated dynamically by the control unit, especially on the basis of a certain predetermined parameter. In particular, the attention criterion gives which of these attention areas the vehicle driver's attention must be directed to. However, the attention criterion can also give which of these attention areas the vehicle driver's attention is prohibited from being directed to during the execution of the maneuver. Examples of such attention areas are the footwell of the vehicle or the central information display screen of the vehicle. Then, the control unit compares the vehicle driver's gaze activity with the attention criterion and regulates the vehicle accordingly.

[0011] In one example, a vehicle is approaching an intersection and should automatically turn right. The control unit determines an attention area on the right side of the vehicle, for example, an attention area on the right side of the front windshield of the vehicle and on the right window glass. The vehicle turns right only when the vehicle driver looks at least to the right, otherwise the vehicle is stationary at the intersection. In another example, the vehicle should be assisted in changing lanes. In this example, the control unit determines in particular an attention area on one of the side windows and on the rear window glass in order to check whether the vehicle driver has performed a head turn before the lane change is performed. Different types of head turns can be identified by defining multiple attention areas. For example, not every vehicle driver will turn his head completely backwards to perform a head turn.

[0012] In this context, assisted driving is understood to mean automation levels 1 and 2 according to the Federal Highway Research Institute. In this context, automated driving is understood to mean automation levels 2+ and 3 according to the Federal Highway Research Institute. The surroundings detection unit is in particular a sensor of the vehicle and is also described in more detail in connection with some embodiments.

[0013] The occupant detection unit includes in particular an internal camera of the vehicle and is in particular designed to detect a two-dimensional image of the interior space of the vehicle. In addition, the occupant detection unit can be designed to detect a plurality of images that are sequential in time in order to generate an image sequence, each of which includes an image of the vehicle driver. In such an embodiment, the driver image data is equivalent to an image sequence. The occupant detection unit in particular detects light in the spectrum and / or in the infrared spectrum. The detection of light in the infrared spectrum has the additional advantage that the occupant detection unit can also detect images or image sequences in the case of a poor light ratio, for example at night. As an alternative to an optical camera, the occupant detection unit can also include a time-of-flight camera, a radar system or a lidar system, which can generate a three-dimensional image of the interior space of the vehicle, in particular an image in the form of a point cloud.

[0014] In one embodiment, the control unit is designed to determine whether the vehicle driver is attentive at least on the basis of a comparison of the gaze activity data with an attention criterion and to generate a control command such that the vehicle performs a maneuver only if the control unit determines that the vehicle driver is attentive according to the attention criterion. In this embodiment, the vehicle performs a maneuver only if the vehicle driver is attentive according to the attention criterion. Otherwise, the control unit generates a control command, for example, such that the vehicle stops safely or outputs a warning signal to the vehicle driver. This improves traffic safety during assisted or automated driving of the vehicle.

[0015] In a further embodiment, the device comprises an input unit of the vehicle. The control unit is designed to generate a control command to implement the maneuver when the vehicle driver completes a predetermined input into the input unit, after the control unit has first determined that the vehicle driver is not paying attention according to the attention criterion. In this embodiment, the driver must explicitly confirm that the maneuver is to be performed. This diverts the attention of the vehicle driver to the assisted driving or automatic driving of the vehicle and thus improves traffic safety. The implementation of the maneuver can be associated with other conditions, for example, now the vehicle driver is paying attention according to the attention criterion. In particular, the control unit can be designed to control the display unit of the vehicle in order to output such a message to the vehicle driver that the vehicle driver is not paying attention and / or must confirm the implementation of the maneuver.

[0016] In another embodiment, the attention criterion includes at least one of the following criteria: the minimum time and / or maximum time that the sight of the vehicle driver must be directed to a certain and / or arbitrary attention area in the attention area. In particular, the control unit is designed to determine whether the sight of the vehicle driver is directed to the corresponding attention area for a longer time than the minimum time and / or shorter time than the maximum time as part of the determination of the gaze activity for each attention area. In this embodiment, the control unit determines such an attention area: before the control unit determines that the vehicle driver is focused according to the attention criterion, the vehicle driver must at least look at the attention area for at least a predetermined time, that is, a minimum time. In addition, when the vehicle driver looks at one of the attention areas for too long, because the vehicle driver, for example, stares absent-mindedly at the corresponding attention area, the control unit can also determine that the vehicle driver is not focused according to the attention criterion. Through the previously mentioned criteria, the attention of the vehicle driver can also be monitored more reliably.

[0017] In another embodiment, the control unit is designed to perform scoring as part of the gaze activity determination, and to score and / or deduct points according to whether and how long the vehicle driver's line of sight is respectively directed at the attention area. The attention standard includes at least one minimum score, which must at least be reached as part of the gaze activity determination. If the vehicle driver reaches the minimum score, he is regarded as focused in the sense of the attention standard. The minimum score can be specifically predetermined for the maneuver or dynamically determined by the control unit. Scoring can include, for example, scoring according to the time of each attention area. In particular, scoring can only start after the vehicle driver's line of sight is directed to the corresponding attention area for the shortest time. If the vehicle driver's line of sight is directed to the corresponding attention area for longer than the longest time, points can be deducted to identify absent-minded gaze. Some attention areas can be deducted, especially such attention areas: the attention area is located in the interior space of the vehicle, such as the foot space or the central information display screen. In addition, if the vehicle driver's line of sight is directed to an area outside the attention area, points can be deducted. The gaze activity determination by scoring has such an advantage that the very different gaze activities of the vehicle driver can also be simply covered by the minimum score. Attention areas of equal value can be assigned the same score, whereas in other embodiments the same scene must be described by a complex either-or statement. This allows a particularly simple expression of the attention criterion, thereby making the device simpler to implement.

[0018] In another embodiment, the control unit is designed to determine at least one maneuvering focus in the driver's field of vision based on the surrounding environment data and the maneuvering data, and determine the attention area in such a way that at least one of the attention areas includes the maneuvering focus. The maneuvering focus corresponds to such an area in the surrounding environment of the vehicle: before and / or during the execution of the maneuvering action, the attention of the vehicle driver must be specifically directed to the said area. If scoring is performed, as long as the vehicle driver's line of sight is directed to the attention area near the maneuvering focus, more points can be obtained. Correspondingly, the farther the attention area currently being watched by the vehicle driver is from the maneuvering focus, the fewer points can be obtained. In addition, if the vehicle driver's line of sight is too far away from the maneuvering focus, points can also be deducted. Therefore, the maneuvering focus is an important criterion for determining the attention area and the attention standard, and this determination can be performed more reliably according to this important criterion. The more reliably the attention area and the attention standard can be determined, the more reliably the device can monitor the attention of the vehicle driver.

[0019] In a further embodiment, the control unit is designed to determine the position of at least one other traffic participant in the field of vision of the vehicle driver on the basis of the surroundings data and to determine at least one attention area corresponding to the position of the other traffic participant in the field of vision of the vehicle driver. In this embodiment, the other traffic participant can constitute one of the maneuvering foci. In particular, in environments with vulnerable traffic participants (such as cyclists, pedestrians, scooter riders, etc.), the attention of the vehicle driver must be directed to these traffic participants in order to be able to perform maneuvers in a traffic-safe manner. This embodiment implements this knowledge by determining at least one attention area corresponding to the position of the other traffic participant. As a result, vulnerable traffic participants can be better protected in particular.

[0020] In another embodiment, the control unit is designed to determine the traffic situation in the surroundings of the vehicle based on the surrounding environment data, generate traffic situation data corresponding to the traffic situation, and determine the attention area under consideration of the traffic situation data. The traffic situation includes, for example, whether the lane in front of the vehicle is blocked, whether the vehicle is in a jam, or whether and / or how many other vehicles and / or other traffic participants are around the vehicle. In this embodiment, the control unit determines the attention area under consideration of the traffic situation in the surroundings of the vehicle and can thus react dynamically to unexpected situations, such as in the surroundings of an city. As a result, traffic safety can be further improved when the vehicle is driven assisted or automatically.

[0021] In another embodiment, the surrounding environment detection unit includes at least one image detection unit of the vehicle, which is designed to detect at least one image with imaging of the surrounding environment of the vehicle as surrounding environment information and generate external image data corresponding to the image. The surrounding environment data includes external image data. The image detection unit of the vehicle is especially a camera of the vehicle. Alternatively, the image detection unit can also include a time-of-flight camera, a radar system or a laser radar system, which can generate a three-dimensional image of the surrounding environment of the vehicle, especially an image in the form of a point cloud. A large number of important features can be reliably extracted from the image of the surrounding environment of the vehicle, such as traffic signs, traffic light systems, lane markings, positions of other traffic participants and other information related to traffic conditions. For example, feature extraction can be performed with the help of machine learning methods, especially machine learning methods specifically used for machine vision, such as convolutional neural networks.

[0022] In another embodiment, the surroundings detection unit includes at least one navigation unit, which is designed to detect and / or receive at least one navigation information related to the vehicle and generate navigation data corresponding to the navigation information. The surroundings data include navigation data. For example, the navigation unit can be a navigation unit of the vehicle. Alternatively or additionally, the navigation unit can also be designed to receive navigation data, for example, from a mobile terminal device installed in the vehicle. The navigation information is especially a road map of the vehicle's surroundings, but can also be the position of the vehicle in a world coordinate system or traffic information, such as congestion information. Based on the navigation information, the control unit can particularly reliably determine the maneuver to be performed. For example, the control unit can plan a trajectory or determine that an intersection is to be crossed based on a road map of the vehicle's surroundings.

[0023] In another embodiment, the control unit is designed to determine the gaze direction and / or head posture of the vehicle driver on the basis of the driver image data, to obtain gaze direction data corresponding to the gaze direction and / or head posture of the vehicle driver, and to perform gaze activity ascertainment under consideration of the gaze direction data. The gaze direction and / or head posture, that is, the orientation and position of the occupant's head, can be obtained from the driver image data in particular by using a suitable machine learning method. In particular, the gaze direction can be obtained from the pupil orientation of the vehicle driver. Not only the gaze direction but also the head posture of the vehicle driver makes it possible to determine very accurately what the vehicle driver's line of sight is directed to. As a result, the gaze activity ascertainment and thus the monitoring of the vehicle driver's attention can be performed more reliably.

[0024] In a further embodiment, the control unit is designed to store the gaze direction data in a memory area of ​​the control unit and to determine the attention area taking into account the gaze direction data stored in the past. In this embodiment, the device learns the specific gaze habits of the vehicle driver, for example, what the vehicle driver's head turns to look like. Taking this information into account, the attention area can be determined in a targeted manner according to the vehicle driver. This makes the monitoring of the vehicle driver's attention even more reliable.

[0025] In another embodiment, the control unit is designed to determine at least one feature of the vehicle driver related to the driving task of the vehicle driver based on the driver image data, generate feature data corresponding to the feature, and generate control instructions for assisted driving or automated driving of the vehicle under consideration of the feature data. For example, the feature may be related to whether the vehicle driver holds the steering wheel or whether the vehicle driver is currently actively controlling the vehicle. However, the feature may also be related to a physiological variable, such as the blinking frequency, from which the attention of the vehicle driver can be determined. If scoring is performed, the control unit can already consider the feature data when the gaze activity is determined. The presence of a specific feature (for example, whether the vehicle driver holds the steering wheel) can generate additional points. Other features (for example, the micro-sleep state of the vehicle driver identified according to the blinking frequency) may lead to deductions. In such an embodiment, the feature data is jointly merged into the generation of control instructions for assisted or automated driving of the vehicle through the gaze activity data. By considering the characteristics of the vehicle driver, the attention of the vehicle driver can be determined more reliably.

[0026] In another embodiment, the control unit is designed to control at least one display unit of the vehicle in order to display the state of the attention area and / or the gaze activity acquisition to the vehicle driver. For example, the display unit can be a screen, in particular a screen of the dashboard of the vehicle or a central information display screen. However, the display unit can also be a head-up display, in particular a panoramic head-up display. In addition, the display unit can also be a light strip at the bottom of the window glass of the front windshield, which displays the state of the attention area and / or the gaze activity acquisition in different colors. The display on the display unit directs the attention of the vehicle driver to the attention area or the gaze activity acquisition, thereby improving traffic safety.

[0027] In addition, the present invention relates to a method for operating a vehicle. In the method, at least one surrounding information related to the surroundings of the vehicle is detected, and surrounding data corresponding to the surrounding information is generated. At least one image with an image of a vehicle driver is detected, and driver image data corresponding to the image is generated. Based on the surrounding data, at least one driving maneuver of the vehicle is determined, which should be performed by the vehicle within the framework of assisted driving or automated driving. Maneuver data corresponding to the determined driving maneuver is generated. Based on the maneuver data, at least two attention areas specific to the maneuver and at least one attention standard specific to the maneuver related to the gaze activity of the vehicle driver are determined in the field of vision of the vehicle driver. Based on the driver image data, it is determined whether and for how long the sight line of the vehicle driver is respectively directed to the attention area. Gaze activity data corresponding to the result of the gaze activity determination is generated. The gaze activity data is compared with the attention standard. In addition, a control instruction for assisted driving or automated driving of the vehicle is generated at least on the basis of this comparison.

[0028] The method has the same advantages as the claimed device. In particular, the method can be extended by the features of the dependent claims directed to the device. In addition, the device described above can be extended by the features described in this article in connection with the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The following is a more detailed explanation of the embodiments of the present invention based on the accompanying drawings. In the accompanying drawings:

[0030] Figure 1 A schematic diagram shows a device for operating a vehicle; and

[0031] Figure 2 A flow chart of a method for operating a vehicle is shown. DETAILED DESCRIPTION

[0032] Figure 1 A schematic illustration shows a device 100 for operating a vehicle 102 .

[0033] During the assisted or autonomous driving of the vehicle 102, the device 100 monitors the attention of the vehicle driver 104 of the vehicle 102. To this end, the device 100 first determines which maneuver the vehicle 102 should perform within the framework of the assisted or autonomous driving. The device 100 then determines a specific attention area for the maneuver to be performed in the field of vision of the vehicle driver 104. Next, the device 100 determines whether and for how long the vehicle driver 104's line of sight is directed toward the attention area, and compares the determined gaze activity of the vehicle driver 104 with the previously specified attention criterion for the maneuver. The maneuver will only be performed as planned if the vehicle driver 104 is attentive in the sense of the attention criterion.

[0034] Device 100 includes a surroundings detection unit 106 for detecting at least one surroundings information related to the surroundings of vehicle 102. Surroundings detection unit 106 includes an image detection unit 108, which is designed purely by way of example as a camera. Image detection unit 108 is arranged and designed to detect an image with a representation of an area outside vehicle 102 as surroundings information and to generate image data corresponding to the image. Figure 1 In the example shown in , the image includes an image of another road user 110, which is shown purely by way of example as a pedestrian. Image acquisition unit 108 can also be designed to acquire an image sequence formed by a plurality of images that follow one another in time and to generate image data corresponding to the image sequence.

[0035] In addition, the surrounding environment detection unit 106 includes a navigation unit 112, which is used to detect and / or provide navigation information related to the vehicle 102, such as a road map of the surrounding environment of the vehicle 102 or the current position of the vehicle 102 in the world coordinate system. The navigation unit 112 generates navigation data corresponding to the navigation information. The surrounding environment detection unit 106 provides the navigation data together with the image data therefrom as surrounding environment data.

[0036] Furthermore, the device 100 includes an occupant detection unit 114 for detecting at least one image with an image of the vehicle driver 104. The occupant detection unit 114 is designed purely by way of example as an interior camera of the vehicle 102, which is oriented and designed to detect at least one image with an image of the vehicle interior and to generate driver image data corresponding to the image. The occupant detection unit 114 can also be designed to detect an image sequence formed by a plurality of images that are successive in time and to generate driver image data corresponding to the image sequence.

[0037] Furthermore, the device 100 comprises a control unit 116, which is shown purely by way of example as part of the vehicle 102. The control unit 116 comprises a memory area 118, in which the control unit 116 can store data and from which the control unit 116 can read the stored data. The control unit 116 is further designed to control at least the previously mentioned functional units of the device 100 in order to carry out a method for operating the vehicle 102. In the method, the control unit 116 first determines a driving maneuver to be carried out based on the surroundings data and generates maneuver data corresponding to the driving maneuver. Then, the control unit 116 determines at least a maneuver-specific attention area and a maneuver-specific attention criterion based on the maneuver data. Then, the control unit 116 determines the gaze activity of the vehicle driver 104 based on the driver image data and generates gaze activity data corresponding to the gaze activity. The control unit 116 compares the gaze activity data with the attention criterion and generates control instructions for assisted or autonomous driving of the vehicle 102 based on this comparison. The control unit 116 then transmits the control command to the functional unit 120 of the vehicle 102, in particular to the actuator, the drive system, the brake system and / or the steering system. Figure 2 A method for operating vehicle 102 is also described in more detail.

[0038] exist Figure 1 In the exemplary embodiment shown in FIG. 1 , the control unit 116 is designed to control a display unit 122 of the vehicle 102 . The display unit 122 is designed purely as a head-up display. The control unit 116 can control the display unit 122 in order to display the state of the attention area and / or the gaze activity ascertainment, for example. Figure 1 In the exemplary embodiment shown in FIG, control unit 116 is furthermore designed to receive inputs from vehicle driver 104 from input unit 124 of device 100 .

[0039] Figure 2 A flow chart of a method for operating vehicle 102 is shown.

[0040] The method is started in step S200. In step S202, the control unit 116 controls the surrounding environment detection unit 106 to detect surrounding environment information and to provide surrounding environment data. For example, the control unit 116 controls the image detection unit 108 to detect an image of the surrounding environment of the vehicle 102 and provide it as image data. In another example, the control unit 116 controls the navigation unit 112 to detect a road map of the surrounding environment of the vehicle 102 and provide it as navigation data. In step S204, the control unit 116 controls the occupant detection unit 114 to detect an image of the vehicle driver 104 and to provide driver image data. Steps S202 and S204 can be performed simultaneously or in sequence in any arbitrary order.

[0041] In step S206, the control unit 116 determines the maneuver to be performed by the vehicle 102 within the framework of assisted driving or autonomous driving based on the surrounding environment data. In one example, the control unit 116 determines based on the image data, for example, by using a machine learning method specifically for machine vision, that the vehicle 102 is driving towards a red traffic light. In another example, the control unit 116 determines based on the navigation data that the vehicle 102 is driving towards an intersection. The control unit 116 generates maneuver data corresponding to the determined maneuver.

[0042] In an optional step S208, the control unit 116 determines at least one maneuvering focus in the field of vision of the vehicle driver 104 based on the surrounding data and the maneuvering data. The maneuvering focus is a point or area in the field of vision of the vehicle driver 104 that the vehicle driver 104 must see in order to be able to perform a maneuver in a traffic-safe manner. In an optional step S210, the control unit 116 determines the position of at least one other traffic participant 110 in the field of vision of the vehicle driver 104 based on the surrounding data. In one example, the control unit 116 determines based on the image data that a pedestrian is crossing the pedestrian crossing and the vehicle 102 is driving towards the pedestrian crossing. In an optional step S212, the control unit 116 determines the traffic situation in the surroundings of the vehicle 102 based on the surrounding data. In one example, the control unit 116 determines based on the image data that the roadway in front of the vehicle 102 is temporarily blocked by a construction site. The optional steps S208 to S212 can be performed simultaneously or in sequence in any arbitrary order.

[0043] In step S214, control unit 116 determines at least two attention areas in the field of vision of vehicle driver 104 and at least one attention criterion associated with the gaze activity of vehicle driver 104 based on at least the maneuver data. Both the attention areas and the attention criterion are maneuver-specific, that is, they are associated with a specific maneuver to be performed by vehicle 102. In step S214, control unit 116 can control display unit 122 in order to display the attention areas to vehicle driver 104. If step S208 has been performed, control unit 116 determines the attention areas in such a way that at least one attention area includes the maneuver focus. If step S210 has been performed, control unit 116 determines the attention areas in such a way that at least one attention area includes the position of another road user 110 in the field of vision of vehicle driver 104. If step S212 has been performed, control unit 116 determines the attention areas and the attention criterion taking into account the ascertained traffic situation.

[0044] In one example, vehicle 102 is to be turned right with assistance or automatically. A cyclist approaches vehicle 102 from behind on the right. In step S210, control unit 116 identifies the cyclist as an additional traffic participant. In step S214, control unit 116 determines the attention area in such a way that the right side window of vehicle 102, the rearview mirror of vehicle 102, the right side mirror of vehicle 102 and the cyclist are each covered by at least one attention area. Control unit 116 determines the attention criterion in such a way that vehicle driver 104 must at all events notice the cyclist before vehicle 102 performs a turning maneuver.

[0045] In step S216, the control unit 116 performs a gaze activity determination, in which the control unit 116 determines, based on the driver image data, whether and how long the sight line of the vehicle driver 104 is directed to the attention area. To this end, the control unit 116 can determine the sight line direction of the vehicle driver 104 from the image data. In particular, when the vehicle driver 104, for example, turns his head to avoid the occupant detection unit 114 when looking around, it may not be possible to determine the exact sight line direction from the image data. Therefore, alternatively or additionally, the control unit 116 can also determine the head posture of the vehicle driver 104. In addition, the control unit 116 generates gaze activity data corresponding to the determined gaze activity of the vehicle driver 104. Optionally, the control unit 116 can store these gaze activity data in the memory area 118 of the control unit 116 for a longer period of time. If there is a stored gaze activity data of the vehicle driver 104, the control unit 116 can determine the attention area in step S214 while taking into account the gaze activity data of the vehicle driver 104. Device 100 may thus take into account specific gaze habits of vehicle driver 104 when determining the attention zone.

[0046] In optional step S218, the control unit 116 determines at least one feature of the vehicle driver 104 based on the driver image data, the feature being related to the driving task of the vehicle driver 104. For example, the control unit 116 determines based on the driver image data whether the vehicle driver 104 is holding the steering wheel of the vehicle 102. The control unit 116 generates feature data corresponding to the feature.

[0047] In step S220, the control unit 116 compares the gaze activity data with the attention standard to determine whether the vehicle driver 104 is focused and can perform a maneuver in a traffic-safe manner according to the attention standard. For this reason, the control unit 116 can perform scoring and give a score to each time unit in which the vehicle driver 104 looks at a specific attention area. In particular, the control unit 116 can score more points for the attention area closer to the maneuvering focus and / or other traffic participants. Correspondingly, the control unit 116 can score fewer points for the attention area farther from the maneuvering focus and / or other traffic participants. When the sight line of the vehicle driver 104 is too far from the maneuvering focus and / or other traffic participants or when the vehicle driver 104 looks at the attention area corresponding to the foot space of the vehicle 102 or the central information display screen, the control unit 116 can also deduct some points. In such an embodiment, the attention standard can correspond to the minimum score that the vehicle driver 104 must reach in order to be regarded as focused in the sense of the attention standard. However, additional conditions may also be provided, for example, the vehicle driver 104 must necessarily look at the attention area corresponding to the position of the other traffic participant 110 in his field of vision, that is, the vehicle driver 104 must see the other traffic participant at all costs. If step S218 has been performed, additional points can be obtained based on the characteristic data. In addition, the attention criterion may specify a minimum time and / or a maximum time that the vehicle driver 104 must look at a specific attention area in order to be considered to be attentive. In addition, the attention criterion may also specify that the vehicle driver 104 must see a certain percentage of the attention area within a predetermined duration so that the vehicle driver 104 is considered to be attentive. In step S220, the control unit 116 may adjust the display unit 122 so as to display the status of the gaze activity to the vehicle driver 104.

[0048] In step S222, control unit 116 generates control commands for assisted or autonomous driving of vehicle 102 based on the result of step S220 and transmits the control commands to functional unit 120. In particular, control unit 116 generates control commands in such a way that vehicle 102 performs a maneuver only if control unit 116 determines in step S220 that vehicle driver 104 is attentive according to the attentiveness criterion. If control unit 116 determines in step S220 that vehicle driver 104 is not attentive according to the attentiveness criterion, the execution of the maneuver can be made dependent on an input by vehicle driver 104 in input unit 124, by which vehicle driver 104 confirms that the maneuver is to be performed. The method ends in step S224.

[0049] In particular, a single driving maneuver, such as driving through an intersection, can be decomposed into a plurality of maneuvers within the meaning of the method described here, for which steps S202 to S222 are respectively repeated. In a first maneuver, the vehicle 102 approaches the intersection and stops at the stop line. In step S214, the control unit 116 determines a notice area that covers a wide field of view, because the vehicle driver 104 has to monitor the entire traffic space around the vehicle 102. In a second maneuver, the vehicle 102 approaches. In step S214, the control unit 116 determines a notice area that is narrowly focused to the right front, because when the vehicle 102 approaches, people who are located in front of the vehicle 102 when approaching, such as pedestrians crossing the roadway, should not be endangered. In a third maneuver, the vehicle 102 finally drives through the intersection. In step S214, the control unit 116 determines a notice area that again covers a wide field of view. In addition, control unit 116 determines attention zones that respectively correspond to the positions of further road users along the trajectory of vehicle 102 in the field of vision of vehicle driver 104 .

[0050] In reference Figure 1 and 2 In the illustrated exemplary embodiment, at least the surroundings detection unit 106, the occupant detection unit 114 and the control unit 116 form the device 100 for operating the vehicle 102. Figure 1 and 2 The elements and features shown in and mentioned in the previous description may be part of the claimed apparatus 100. Likewise, the method steps described with reference to the apparatus 100 may be part of the claimed method.

[0051] Reference numerals list

[0052] 100 devices

[0053] 102 vehicles

[0054] 104 Vehicle Driver

[0055] 106 Environmental detection device

[0056] 108 Image acquisition device

[0057] 110 Traffic Participants

[0058] 112 Navigation Unit

[0059] 114 Occupant Detection Unit

[0060] 116 Control Unit

[0061] 118 memory area

[0062] 120 functional units

[0063] 122 display units

[0064] 124 input units

Claims

1. A device (100) for operating a vehicle (102), comprising: A surrounding environment detection unit (106), the surrounding environment detection unit being designed to detect at least one surrounding environment information related to the surrounding environment of the vehicle (102) and to generate surrounding environment data corresponding to the surrounding environment information; an occupant detection unit (114) configured to detect at least one image of a vehicle driver (104) of the vehicle (102) and to generate driver image data corresponding to the image; and A control unit (116) is provided, the control unit being designed to: determine at least one driving maneuver of the vehicle (102) based on the surrounding environment data, the at least one driving maneuver to be performed by the vehicle (102) within the framework of assisted driving or automated driving; and to generate maneuver data corresponding to the determined driving maneuver, The control unit (116) is further designed to: determine, on the basis of the maneuver data, at least two maneuver-specific attention areas in the field of vision of the vehicle driver (104) and at least one maneuver-specific attention criterion related to the gaze activity of the vehicle driver (104); and to determine, on the basis of the driver image data, whether and for how long the gaze of the vehicle driver (104) is respectively directed toward the at least two attention areas; generating gaze activity data corresponding to the result of the gaze activity; comparing the gaze activity data with the attention standard; And generating control instructions for the assisted driving or automated driving of the vehicle (102) at least based on the comparison.

2. The device (100) according to claim 1, wherein the control unit (116) is designed to: determine whether the vehicle driver (104) is attentive at least on the basis of the comparison of the gaze activity data with the attention standard; and generate the control instruction so that the vehicle (102) performs the maneuver only when the control unit (116) determines that the vehicle driver (104) is attentive according to the attention standard.

3. The device (100) according to claim 2 comprises an input unit (124) of the vehicle (102), wherein the control unit (116) is designed to generate the control instruction for implementing the maneuver when the vehicle driver (104) completes a predetermined input into the input unit (124) after the control unit (116) has first determined that the vehicle driver (104) is not attentive according to the attention criterion.

4. A device (100) according to any of the preceding claims, wherein the attention criterion includes at least one of the following criteria: a minimum time and / or a maximum time that the vehicle driver (104) must keep his or her line of sight directed toward a determined and / or arbitrary attention area in the attention area.

5. The device (100) according to any one of the preceding claims, wherein the control unit (116) is designed to: perform scoring as part of the gaze activity determination; and to score and / or deduct points depending on whether and for how long the sight line of the vehicle driver (104) is respectively directed to the attention area, And wherein the attention criterion comprises at least one minimum score, which must at least be achieved as part of the gaze activity being sought.

6. A device (100) according to any one of the preceding claims, wherein the control unit (116) is designed to: determine at least one maneuvering focus in the field of vision of the vehicle driver (104) based on the surrounding environment data and the maneuvering data; and determine the attention area so that the maneuvering focus is included in at least one of the attention areas.

7. A device (100) according to any one of the preceding claims, wherein the control unit (116) is designed to: determine the position of at least one other traffic participant (110) in the field of view of the vehicle driver (104) based on the surrounding environment data; and determine at least one attention area corresponding to the position of the other traffic participant (110) in the field of view of the vehicle driver (104).

8. A device (100) according to any one of the preceding claims, wherein the control unit (116) is designed to: determine the traffic conditions in the surroundings of the vehicle (102) based on the surrounding environment data; generate traffic condition data corresponding to the traffic conditions; and determine the attention area while taking into account the traffic condition data.

9. The device (100) according to any one of the preceding claims, wherein the surroundings detection unit (106) comprises at least one image detection unit (108) of the vehicle (102), the image detection unit being designed to: detect at least one image having an image of the surroundings of the vehicle (102) as the surroundings information; and generate external image data corresponding to the image, And wherein the surrounding environment data includes the external image data.

10. The device (100) according to any one of the preceding claims, wherein the surrounding environment detection unit (106) comprises at least one navigation unit (112), wherein the at least one navigation unit (112) is designed to: detect and / or receive at least one navigation information related to the vehicle (102); and generate navigation data corresponding to the navigation information, And wherein the surrounding environment data includes the navigation data.

11. A device (100) according to any one of the preceding claims, wherein the control unit (116) is designed to: determine the gaze direction and / or head posture of the vehicle driver (104) based on the driver image data; obtain gaze direction data corresponding to the gaze direction and / or head posture of the vehicle driver (104); and perform the gaze activity acquisition while taking into account the gaze direction data.

12. The device (100) according to claim 11, wherein the control unit (116) is designed to: store the gaze direction data in a memory area (118) of the control unit (116); and to determine the attention area while taking into account the gaze direction data stored in the past.

13. A device (100) according to any one of the preceding claims, wherein the control unit (116) is designed to: determine at least one feature of the vehicle driver (104) related to the driving task of the vehicle driver (104) based on the driver image data; generate feature data corresponding to the feature; and generate the control instructions for the assisted driving or automated driving of the vehicle (102) taking into account the feature data.

14. A device (100) according to any of the preceding claims, wherein the control unit (116) is designed to regulate at least one display unit (122) of the vehicle (102) so as to display the state of the attention area and / or the gaze activity to the vehicle driver (104).

15. A method for operating a vehicle (102), wherein: Detecting at least one surrounding environment information related to the surrounding environment of the vehicle (102), and generating surrounding environment data corresponding to the surrounding environment information; detecting at least one image having an image of a vehicle driver (104) of the vehicle (102) and generating driver image data corresponding to the image; determining at least one driving maneuver of the vehicle (102) on the basis of the surrounding environment data, the at least one driving maneuver to be performed within the framework of assisted driving or automated driving, and generating maneuver data corresponding to the determined driving maneuver; determining, on the basis of the maneuver data, at least two maneuver-specific attention areas in the field of vision of the vehicle driver (104) and at least one maneuver-specific attention criterion, the at least one attention criterion being related to a gaze activity of the vehicle driver (104); determining, on the basis of the driver image data, whether and for how long the sight of the vehicle driver (104) is directed towards the at least two attention areas in each case, and generating gaze activity data corresponding to the result of this gaze activity determination; comparing the gaze activity data to the attention criterion; and Control instructions for assisted or automated driving of the vehicle (102) are generated at least on the basis of the comparison.

Images