Device and method for operating a vehicle

Abstract

Provided are an apparatus and a method for operating a vehicle that can better monitor the attention of a vehicle driver of the vehicle. 【Solution means】 A surrounding detection unit 106 that detects surrounding information of the vehicle 102 and generates surrounding information, an occupant detection unit 114 that detects an image of the vehicle driver 104 and generates driver image data, and a control unit 116 that identifies the driving operation of the vehicle 102 based on the surrounding data and generates operation data. The control unit 116 determines, based on the operation data, an attention range in the field of view of the vehicle driver 104 and an attention standard regarding the visual activity of the vehicle driver 104, identifies whether the line of sight of the vehicle driver 104 is directed to the attention range based on the driver image data, generates visual activity data, compares the visual activity data with the attention standard, and generates a control command for the assisted driving or automated driving of the vehicle 102 based on the comparison.

Description

Technical Field

[0001] The present invention relates to a device for operating a vehicle. The present invention further relates to a method for operating a vehicle.

Background Art

[0002] During assisted driving or automated driving, a predetermined driving function of the vehicle is taken over by the vehicle assist system. Contrary to autonomous driving, the vehicle driver needs to be in a state of temporarily taking over the driving task in some cases during assisted driving or automated driving. For this purpose, the vehicle driver needs to constantly pay attention to the traffic situation around the vehicle during assisted driving or automated driving. For this reason, the vehicle assist system often includes a system for monitoring the attention (ability) of the vehicle driver for assisted driving or automated driving.

[0003] Lane change assist for vehicles is known from Patent Document 1. The lane change assist is configured to prevent a predetermined maneuver of the vehicle depending on the driver's line of sight direction.

[0004] Patent Document 2 discloses a method for operating a motor vehicle. In the method, it is compared whether the driver's line of sight direction coincides with a target line of sight direction. When the driver's line of sight direction does not correspond to the target line of sight direction, a warning sound is emitted and / or a vehicle function, particularly a partially automated lane change, is not activated.

[0005] From Patent Document 3, a system for operating a motor vehicle depending on the line of sight direction is known. The system is configured to determine whether the driver's line of sight direction is directed to a preset rear range when the vehicle is in reverse gear.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] French Patent Application Publication No. 3047957 [Patent Document 2] German Patent Application Publication No. 102021201062 [Patent Document 3] German Patent Application Publication No. 102013007257 [Summary of the Invention] [Problems to be Solved by the Invention]

[0007] The problem of the present invention is to provide an apparatus and a method for operating a vehicle that can monitor the attention of a vehicle driver of the vehicle better than known apparatuses and methods during assisted driving or autonomous driving. [Means for Solving the Problems]

[0008] The problem is solved by an apparatus having the features according to claim 1 and a method having the features of the independent claims. Advantageous developments are described in each dependent claim.

[0009] The proposed device for operating a vehicle includes a surrounding detection unit configured to detect at least one piece of surrounding information regarding the surroundings of the vehicle and generate surrounding data corresponding to the surrounding information, and an occupant detection unit configured to detect at least one image having a depiction of the vehicle driver of the vehicle and generate driver image data corresponding to the image. The device further includes a control unit configured to identify at least one driving maneuver of the vehicle to be executed by the vehicle in assisted driving or automated driving based on the surrounding data and generate control data corresponding to the identified driving maneuver. The control unit is further configured to determine, based on the control data, at least two maneuver-specific areas of attention in the field of vision of the vehicle driver and at least one maneuver-specific attention criterion regarding the visual activity of the vehicle driver, identify, based on the driver image data, whether the line of sight of the vehicle driver is directed towards each area of attention and for how long (in terms of time), generate visual activity data corresponding to the specific result of this visual activity, compare the visual activity data with the attention criterion, and generate, at least based on the comparison, a control command for the assisted driving or automated driving of the vehicle.

[0010] The device monitors whether the vehicle driver attentively follows the assisted driving or automated driving of the vehicle so that it can intervene quickly if necessary. This monitoring is carried out in a maneuver-specific manner by the control unit determining the area of attention specific to each maneuver to be executed. By carrying out the attention monitoring in a maneuver-specific manner, it is possible to carry out the attention monitoring purposefully and thus reliably. In addition, the dynamic generation of the area of attention makes it possible to take into account the specific situation in the surroundings of the vehicle better than in the case of known devices, especially at intersections and roundabouts in urban traffic and during maneuvers at low speed.

[0011] To perform driving-specific attention monitoring, the control unit first determines, based on the ambient data, what driving maneuvers the vehicle should execute. For this purpose, the control unit determines the future trajectory of the vehicle based on the road progression and the traffic situation around the vehicle. Then, the control unit generates an area of attention in the driver's field of view depending on the maneuvers to be executed. At this time, in this specification, the driver's field of view of the vehicle is understood to be the range inside and outside the vehicle that the driver can visually observe even if the driver needs to turn their head, for example. The control unit further determines attention criteria. At this time, the attention criteria can be preset for a given maneuver or can be dynamically generated by the control unit based on a particular setting. The attention criteria particularly indicate where the driver's attention must necessarily be directed within the area of attention. However, the attention criteria can also indicate where the driver's attention does not have to be directed within the area of attention during the execution of the maneuver. Examples of such attention criteria are the footwell of the vehicle or the central information display of the vehicle. Then, the control unit compares the driver's visual activity with the attention criteria and controls the vehicle accordingly.

[0012] In one example, the vehicle is traveling towards an intersection and is to turn right automatically. The control unit determines the area of attention on the right side of the vehicle, for example, on the right side of the windshield and the right side window of the vehicle. Then, the vehicle turns right only if the driver visually observes at least to the right; otherwise, the vehicle stops at the intersection. In another example, the vehicle is to perform a lane change with assistance. In this example, the control unit determines the area of attention particularly in the side window and the rear window to check whether the driver has performed a visual check (over the shoulder) before the execution of the lane change. By defining multiple areas of attention, it is possible to recognize different types of (over-the-shoulder) visual checks. That is, for performing a (over-the-shoulder) visual check, for example, not all drivers turn their heads completely backwards.

[0013] In this specification, assisted driving is understood as automation levels 1 and 2 according to the (German) Federal Highway Research Institute. In this specification, automated driving is understood as automation levels 2+ and 3 according to the (German) Federal Highway Research Institute. The surrounding detection unit is particularly a vehicle sensor and will be described in more detail in relation to the embodiments.

[0014] The occupant detection unit particularly includes an in-vehicle camera and is particularly configured to detect a two-dimensional image of the vehicle interior. Further, the occupant detection unit can be configured to detect a plurality of temporally consecutive images each including a depiction of the vehicle driver in order to generate an image sequence. In such an embodiment, the driver image data corresponds to the image sequence. The occupant detection unit particularly detects light in the optical spectrum and / or the infrared spectrum. Detection of light in the infrared spectrum has the further advantage that the occupant detection unit can detect an image or an image sequence even under poor light conditions, for example at night. The occupant detection unit can include a ToF (Time-of-Flight) camera, a radar system, or a lidar system instead of an optical camera, and particularly generates a three-dimensional image of the vehicle interior in the form of a point cloud.

[0015] In one embodiment, the control unit is configured to determine whether the vehicle driver is paying attention based at least on a comparison of the visual activity data with an attention criterion, and to generate the control command such that the vehicle executes a maneuver only when the control unit determines that the vehicle driver is paying attention according to the attention criterion. In this embodiment, the vehicle executes a maneuver only when the vehicle driver is paying attention according to the attention criterion. Otherwise, the control unit generates a control command such that, for example, the vehicle stops securely or a warning signal is issued to the vehicle driver. Thereby, traffic safety in the assisted driving or autonomous driving of the vehicle is improved.

[0016] In another embodiment, the apparatus includes an input unit of the vehicle. The control unit is configured to generate a control command for executing the maneuver when the vehicle driver operates a predefined input to the input unit after first determining that the vehicle driver is not paying attention in accordance with the attention criterion. In this embodiment, the vehicle driver needs to clearly confirm that the maneuver should be executed. Thereby, the attention of the vehicle driver is directed to the assisted driving or the local driving of the vehicle, and thus the traffic safety is improved. The execution of the maneuver can be associated with another condition, for example, that the vehicle driver is paying attention in accordance with the attention criterion. In particular, it is possible to be configured to control a display unit of the vehicle that issues a notification to the vehicle driver that the vehicle driver is not paying attention and / or that the vehicle driver needs to confirm the execution of the maneuver.

[0017] In another embodiment, the attention criterion includes at least one of the following conditions: a predetermined range and / or an arbitrary range within the attention range, and the minimum time and / or the maximum time for which the line of sight of the vehicle driver needs to be directed. In particular, the control unit is configured to identify, as a specific part of the visual activity for each attention range, whether the line of sight of the vehicle driver is directed to each attention range for a time longer than the minimum time and / or shorter than the maximum time. In this embodiment, the control unit determines an attention range that the vehicle driver must visually observe for at least a predetermined time, that is, the minimum time, before the control unit identifies that the vehicle driver is paying attention in accordance with the attention criterion. In addition, the control unit can also determine that the vehicle driver is not paying attention in accordance with the attention criterion when the vehicle driver visually observes one of the attention ranges for an excessively long time, for example, by staring blankly at each attention range. According to the above criteria, it is possible to monitor the attention of the vehicle driver with higher reliability.

[0018] In another embodiment, the control unit performs point assignment as a specific part of the visual activity, and is configured to assign and / or subtract points according to whether the line of sight of the vehicle driver is directed towards each attention range and for how long. The attention criteria include at least the minimum number of points that need to be achieved at least as a specific part of the visual activity. When the vehicle driver reaches the minimum number of points, the vehicle driver is considered to be paying attention (having attention) in the sense of the attention criteria. The minimum number of points can be set specifically for the operation or can be determined dynamically by the control unit. The point assignment can include, for example, point assignment based on time for each attention range. In particular, the point assignment can be started only after the line of sight of the vehicle driver has been directed towards each attention range for a minimum time. If the line of sight of the vehicle driver is directed towards each attention range for longer than the maximum time to recognize a vague stare, points can be subtracted. A predetermined attention range can subtract points, especially those in the interior space of the vehicle, such as the footwell or the central information display. Also, points can be subtracted when the line of sight of the vehicle driver is directed towards a range outside the attention range. The specification of the visual activity by point assignment has the advantage that very different visual activities of the vehicle driver can be simply covered by the minimum number of points. Thus, the same attention range can be added to the same point value, while similar situations (scenarios) in other embodiments need to be described by complex alternative instructions. This makes it possible to formulate the attention criteria particularly simply, thereby making it easier to implement the device.

[0019] In another embodiment, the control unit is configured to identify at least one driving focus in the field of view of the vehicle driver based on the ambient data and the driving data, and to determine the attention range such that at least one of the attention ranges includes the driving focus. The driving focus corresponds to a range around the vehicle where the attention of the vehicle driver should be particularly directed before or during the execution of driving. If point assignment is performed, more points can be assigned as long as the line of sight of the vehicle driver is directed towards the attention range near the driving focus. The farther the attention range currently being visually observed by the vehicle driver is from the driving focus, the correspondingly fewer points can be assigned. Also, if the line of sight of the vehicle driver is too far from the driving focus, it is possible to subtract points. Therefore, the driving focus is an important criterion for determining the attention range and the attention standard, and based on this, it is possible to make the determination more reliably. The more reliable the attention range and the attention standard are, the more reliably the device can monitor the attention of the vehicle driver as well.

[0020] In another embodiment, the control unit is configured to identify at least one other traffic participant in the field of view of the vehicle driver based on the ambient data, and to identify at least one of the attention ranges corresponding to the position of the other traffic participant in the field of view of the vehicle driver. In this embodiment, the other traffic participant can form one of the driving focuses. In particular, in the field of view having vulnerable road users such as cyclists, pedestrians, moped riders, etc., the attention of the vehicle driver needs to be directed towards the vulnerable road users so that driving can be executed safely on the road. This embodiment implements the recognition that at least one of the attention ranges is identified corresponding to the position of the other traffic participant. Therefore, it is possible to better protect vulnerable road users in particular.

[0021] In another embodiment, the control unit is configured to identify a traffic situation around the vehicle based on ambient data, generate traffic situation data corresponding to the traffic situation, and identify an attention range in consideration of the traffic situation data. The traffic situation includes, for example, whether the lane in front of the vehicle is closed, whether the vehicle is stuck in traffic, or whether there are other vehicles and / or other traffic participants around the vehicle, and / or how many other vehicles and / or other traffic participants there are. In this embodiment, the control unit identifies the attention range in consideration of the traffic situation around the vehicle, and thus, for example, in an urban area, it is possible to dynamically respond to unexpected situations. Thereby, it is possible to further improve the traffic safety in the assisted driving or autonomous driving of the vehicle.

[0022] In another embodiment, the ambient detection unit includes at least one image detection unit of the vehicle, and the image detection unit is configured to detect at least one image having a depiction of the surroundings of the vehicle as ambient information and generate external image data corresponding to the image. The ambient data includes the external image data. The image detection unit of the vehicle is particularly a camera of the vehicle. Instead of this, the occupant detection unit can include a ToF (Time-of-Flight) camera, a radar system, or a lidar system, and the system generates a three-dimensional image of the surroundings of the vehicle, particularly in the form of a point cloud. Based on the image of the surroundings of the vehicle, it is possible to reliably extract a number of important features, such as traffic signs, signal equipment, road surface markings, the positions of other traffic participants, and further information about the traffic situation. The extraction of features can be performed, for example, using a machine learning method, particularly a machine learning method specialized for machine vision, such as a convolutional neural network.

[0023] In another embodiment, the surrounding detection unit includes at least one navigation unit of the vehicle. The navigation unit is configured to detect and / or receive at least one piece of navigation information regarding the vehicle and generate navigation data corresponding to the navigation information. The surrounding data includes the navigation data. For example, the navigation unit may be a navigation unit of the vehicle. Alternatively or in addition, the navigation unit may be configured to receive navigation data by, for example, a mobile terminal installed in the vehicle. The navigation information is particularly a road map around the vehicle, but may also be the position of the vehicle in the world coordinate system or traffic information, particularly traffic jam information. Based on the navigation information, the control unit can determine the operation to be executed, particularly with high reliability. For example, the control unit can plan a trajectory or identify that it can cross an intersection based on the road map around the vehicle.

[0024] In another embodiment, the control unit determines the line-of-sight direction and / or the head posture of the vehicle driver based on the driver image data, identifies line-of-sight direction data corresponding to the line of sight and / or the head posture of the vehicle driver, and is configured to perform the identification of the visual activity in consideration of the line-of-sight direction data. The line-of-sight direction and / or the head posture, that is, the orientation and position of the head of the occupant, can be identified based on the driver image data, particularly using an appropriate machine learning method. The line-of-sight direction can be particularly identified based on the orientation of the pupils of the vehicle driver. Both the line-of-sight direction of the vehicle driver and the head posture enable very accurately determining where the line of sight of the vehicle driver is directed. Thereby, it is possible to perform the identification of the visual activity and thus the monitoring of the attention of the vehicle driver with higher reliability.

[0025] In another embodiment, the control unit is configured to store the line-of-sight direction data in the storage range of the control unit and to specify the attention range in consideration of the previously stored line-of-sight directions. In this embodiment, the device learns the special visual habits (visual quirks) of the vehicle driver, for example, what the vehicle driver's over-the-shoulder visual confirmation is like. Considering this information, it is possible to determine the attention range according to the vehicle driver. Thereby, the reliability of monitoring the vehicle driver's attention is further enhanced.

[0026] In another embodiment, the control unit specifies at least one characteristic of the vehicle driver regarding the driving task of the vehicle driver based on the driver image data, generates characteristic data corresponding to the characteristic, and is configured to generate a control command for the assisted driving or automated driving of the vehicle in consideration of the characteristic data. The characteristic can be, for example, whether the vehicle driver is holding the steering wheel or whether the vehicle driver is currently actively controlling the vehicle. However, the characteristic can be a physiological quantity, such as the frequency of blinking, and based on this blinking frequency, it is possible to determine the attention of the vehicle driver. If point assignment is performed, the control unit can already consider the characteristic data when specifying the visual activity. The presence of a predetermined characteristic, such as whether the vehicle driver is holding the steering wheel, can be an additional point. The presence of other characteristics, such as momentary drowsiness (microsleep) of the vehicle driver recognized based on the blinking frequency, can lead to a reduction in points. In such an embodiment, the characteristic data is introduced together via the visual activity data for generating control commands for the assisted driving or automated driving of the vehicle. By considering the characteristics of the vehicle driver, it is possible to determine the attention of the vehicle driver with higher reliability.

[0027] In another embodiment, the control unit is configured to actuate at least one display unit of the vehicle in order to display the area of attention and / or a specific state of the visual activity to the vehicle driver. The display unit may be, for example, a monitor screen, particularly a monitor screen of the dashboard of the vehicle or a monitor screen of a central information display. However, the display unit may also be a head-up display, particularly a panoramic head-up display. Further, the display unit may be a luminous elongated line at the window base of the windshield that displays the area of attention and / or a specific state of the visual activity in various colors. By the display in the display unit, the attention of the vehicle driver is directed to the area of attention or to the specification of the visual activity, thereby improving traffic safety.

[0028] The present invention further relates to a method for operating a vehicle. In the method, at least one piece of ambient information regarding the surroundings of the vehicle is detected, and ambient data corresponding to the ambient information is generated. At least one image having a depiction of the vehicle driver of the vehicle is detected, and driver image data corresponding to the image is generated. Based on the ambient data, at least one driving maneuver of the vehicle to be executed by the vehicle in assisted driving or automated driving is specified. Maneuvering data corresponding to the specified driving maneuver is generated. Based on the maneuvering data, at least two maneuver-specific areas of attention in the field of view of the vehicle driver and at least one maneuver-specific attention criterion regarding the visual activity of the vehicle driver are determined. Based on the driver image data, it is specified whether the line of sight of the vehicle driver is directed to each area of attention and for how long. Visual activity data corresponding to the specific result of the visual activity is generated. The visual activity data is compared with the attention criterion. Further, based at least on the comparison, a control command for assisted driving or automated driving of the vehicle is generated.

[0029] The method has the same advantages as the (patent) claimed device. In particular, the method can be developed by the features of the dependent claims that focus on the device. Also, the above-described device can be developed by the features described herein in relation to the method.

[0030] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

Brief Description of the Drawings

[0031]

Figure 1

Figure 2

Mode for Carrying Out the Invention

[0032] In FIG. 1, a device 100 for operating a vehicle 102 is schematically illustrated.

[0033] The device 100 monitors the attention (force) of the vehicle driver 104 of the vehicle 102 during assisted driving or autonomous driving of the vehicle 102. For this purpose, the device 100 first identifies which maneuvers the vehicle 102 should execute during assisted driving or autonomous driving. Then, the device 100 determines a specific attention range for the maneuvers to be executed within the field of view of the vehicle driver 104. Then, the device 100 identifies whether the line of sight of the vehicle driver 104 is directed towards the attention range and for how long the line of sight is directed towards the attention range, and also compares the identified visual activity of the vehicle driver 104 with a preset attention criterion specific to the maneuver. The maneuver is executed as planned only when the vehicle driver 104 is paying attention in the sense of the attention criterion.

[0034] The device 100 includes a surrounding detection unit 106 that detects at least one piece of surrounding information regarding the vehicle 102. The surrounding detection unit 106 includes, by way of example only, an image detection unit 108 formed as a camera. The image detection unit 108 is arranged and formed to detect an image having a depiction of an external range of the vehicle as surrounding information and generate image data corresponding to the image. In the example shown in FIG. 1, the image includes a depiction of another traffic participant 110 shown as a pedestrian by way of example only. The image detection unit 108 can also be formed to detect an image sequence consisting of a plurality of images that are temporally consecutive with each other and generate image data corresponding to the image sequence.

[0035] The surrounding detection unit 106 further includes a navigation unit 112 that detects and / or provides navigation information regarding the vehicle 102, such as a road map around the vehicle or the current position of the vehicle 102 in a world coordinate system. The navigation unit 112 generates navigation data corresponding to the navigation information. The surrounding detection unit 106 provides the navigation data together with the image data as surrounding data.

[0036] The device 100 further includes an occupant detection unit 114 that detects at least one image having a depiction of the vehicle driver 104. The occupant detection unit 114 is formed as an interior camera of the vehicle 102 by way of example only, and the interior camera is configured and formed to detect at least one image having a depiction of the interior of the vehicle cabin and generate driver image data corresponding to the image. The occupant detection unit 114 can also be formed to detect an image sequence consisting of a plurality of images that are temporally consecutive with each other and generate driver image data corresponding to the image sequence.

[0037] The device 100 further includes a control unit 116, which is merely exemplarily shown as part of the vehicle 102. The control unit 116 includes a storage area 118 in which the control unit 116 can store data and from which the control unit 116 can read the stored data. Further, the control unit 116 is configured to control (operate) at least the above-described functional units of the device 100 in order to execute a method for operating the vehicle 102. In the method, first, based on ambient data, the control unit 116 identifies a driving maneuver to be executed and generates maneuver data corresponding to the driving maneuver. Then, based at least on the maneuver data, the control unit 116 identifies a maneuver-specific attention range and a maneuver-specific attention criterion. Then, based on the driver image data, the control unit 116 identifies the visual activity of the vehicle driver 104 and generates visual activity data corresponding to the visual activity. The control unit 116 compares the visual activity data with the attention criterion and, based on the comparison, generates a control command (control instruction) for assisted driving or autonomous driving of the vehicle 102. Then, the control unit 116 transmits the control command to the functional units 120 of the vehicle, in particular to the actuator, drive system, brake system and / or steering system. Below, the method for operating the vehicle 102 will be described in more detail based on FIG. 2.

[0038] In the embodiment shown in FIG. 1, the control unit 116 is further configured to control (operate) the display unit 122 of the vehicle 102. The display unit 122 is formed, for example, as a head-up display. The control unit 116 can be controlled (operated) to, for example, indicate the state of the attention range and / or visual activity identification. Further, according to the embodiment shown in FIG. 1, the control unit 116 is configured to receive an input from the vehicle driver 104 from the input unit 124 of the device 100.

[0039] FIG. 2 shows a flowchart of a method for operating the vehicle 102.

[0040] The method starts at step S200. In step S202, the control unit 116 controls (operates) the ambient detection unit 106 that detects ambient information and provides ambient data. For example, the control unit 116 controls (operates) the image detection unit 108 that detects an image around the vehicle 102, and provides the image as image data. In another example, the control unit 116 controls (operates) the navigation unit 112 that detects a road map around the vehicle 102, and provides the road map as navigation data. In step S204, the control unit 116 controls (operates) the occupant detection unit 114 that detects an image of the vehicle driver 104 and provides driver image data. Steps S202 and S204 can be executed simultaneously or continuously in any order respectively.

[0041] In step S206, the control unit 116 identifies the maneuvers to be executed by the vehicle 102 in assisted driving or autonomous driving based on the ambient data. In one example, the control unit 116 identifies, based on the image data, that the vehicle 102 is traveling towards a red signal, using, for example, a machine learning method specialized in machine vision. In another example, the control unit 116 identifies, based on the navigation data, that the vehicle 102 is traveling towards an intersection. The control unit 116 generates maneuver data corresponding to the identified maneuvers.

[0042] In optional step S208, the control unit 116 identifies at least one steering focus in the field of view of the vehicle driver 104 based on the ambient data and the driving data. The steering focus is a point or range in the field of view of the vehicle driver 104 that the vehicle driver 104 needs to recognize so that the driving can be carried out safely in traffic. In optional step S210, the control unit 116 identifies the position of at least one other traffic participant 110 in the field of view of the vehicle driver 104 based on the ambient data. In one example, the control unit 116 identifies that a pedestrian is just crossing a crosswalk that the vehicle 102 is heading towards based on the image data. In optional step S212, the control unit 116 identifies the traffic situation around the vehicle driver 102 based on the ambient data. In one example, the control unit 116 identifies that the lane in front of the vehicle 102 is temporarily closed due to a construction site based on the image data. Optional steps S208 to S212 can be executed simultaneously or consecutively in any order respectively.

[0043] In step S214, the control unit 116 determines at least two attention ranges in the field of view of the vehicle driver 104 and at least one attention criterion regarding the visual activity of the vehicle driver 104 based on at least the driving data. Both the attention ranges and the attention criteria are specific to the driving, that is, they relate to the specific driving to be executed by the vehicle 102. In order to display the attention ranges to the vehicle driver 104, the control unit 116 can control (operate) the display unit 122 in step S214. When step S208 is performed, the control unit 116 determines the attention ranges such that at least one of the attention ranges includes the steering focus. When step S210 is performed, the control unit 116 determines the attention ranges such that at least one of the attention ranges includes the position of another traffic participant 110 in the field of view of the vehicle driver 104. When step S212 is performed, the control unit 116 determines the attention ranges and the attention criteria taking into account the identified traffic situation.

[0044] In one example, vehicle 102 will be assisted or will automatically turn right. A motorcyclist approaches vehicle 102 from the right rear. In step S210, the control unit 116 recognizes the motorcyclist as another traffic participant. In step S214, the control unit 116 determines the attention range such that the right side window of vehicle 102, the rearview mirror of vehicle 102, the side mirror on the right side of vehicle 102, and the motorcyclist are each covered (hidden) by at least one of the attention ranges. The control unit 116 determines an attention criterion such that the vehicle driver 104 must notice the motorcyclist in any case before the vehicle performs a turning maneuver (right or left turn).

[0045] In step S216, the identification of the visual activity is performed. In this step, the control unit 116 identifies, based on the driver image data, whether the line of sight of the vehicle driver 104 is directed towards each attention range and for how long. For this purpose, the control unit 116 is able to identify the line of sight direction of the vehicle driver 104 based on the image data. In particular, when the vehicle driver 104 looks over the shoulder, for example, if the head of the vehicle driver moves away from the occupant detection unit 114, it may not be possible to accurately identify the line of sight direction based on the image data. Therefore, instead of or in addition to this, the control unit 116 is also able to identify the head pose of the vehicle driver 104. Further, the control unit 116 generates visual activity data corresponding to the identified visual activity of the vehicle driver 104. Optionally, the control unit 116 can store the visual activity data in the storage area 118 of the control unit 116 for a long time. If there is stored visual activity data of the vehicle driver 104, the control unit 116 can determine the attention range in step S214 taking into account the visual activity data of the vehicle driver 104. Thereby, the device 100 can take into account the specific line of sight habits of the vehicle driver 104 when determining the attention range.

[0046] In option step S218, the control unit 116 identifies at least one characteristic of the vehicle driver 104 regarding the driving task of the vehicle driver 104 based on the driver image data. For example, the control unit 116 identifies whether the vehicle driver 104 is holding the steering wheel of the vehicle 104 based on the driver data. The control unit 116 generates characteristic data corresponding to the characteristic.

[0047] In step S220, in order to determine whether the vehicle driver 104 is paying attention in accordance with the attention standard and whether the operation can be carried out safely in traffic, the control unit 116 compares the visual activity data with the attention standard. For this purpose, the control unit 116 can specifically perform scoring and assign points (scores) per unit time for which the vehicle driver 104 looks within a predetermined attention range. In particular, the control unit 116 can assign more points for the attention range closer to the steering focus and / or another road user. Correspondingly, the control unit 116 can assign fewer points for the attention range further away from the steering focus and / or another road user. If the line of sight of the vehicle driver 104 is too far away from the steering focus and / or another road user, or if the vehicle driver 104 is looking, for example, at the footwell of the vehicle 102 or the attention range corresponding to the central information display, the control unit 116 can also reduce the points. In such an embodiment, the attention standard can correspond to the minimum number of points that the vehicle driver 104 needs to reach in order to be considered as paying attention in the sense of the attention standard. However, it is also possible to set additional conditions, for example, that the vehicle driver 104 must always direct their line of sight to the attention range corresponding to the position of another road user in their field of vision, that is, the vehicle driver 104 must always be looking at another road user. When step S218 is executed, it is possible to assign additional points based on the characteristic data. Also, the attention standard can be set to the minimum time and / or maximum time for which the vehicle driver 104 needs to direct their line of sight to a predetermined attention range in order to be considered as paying attention. Also, the attention standard can be set such that a certain percentage of the attention range must be visually observed by the vehicle driver 104 within a predefined time so that the vehicle driver 104 is considered as paying attention. In order to indicate a specific state of the visual activity to the vehicle driver 104, the control unit 116 can control (operate) the display unit 122 in step S220.

[0048] In step S222, the control unit 116 generates a control command for the assisted driving or autonomous driving of the vehicle 102 based on the result of step S220, and transmits the control command to the functional unit 120. In particular, the control unit 116 generates a control command such that the vehicle 102 executes steering only when the control unit 116 determines in step S220 that the vehicle driver 104 is paying attention in accordance with the attention standard. If the control unit 116 determines in step S220 that the vehicle driver 104 is not paying attention in accordance with the attention standard, it is possible to make the execution of steering depend on the input of the vehicle driver 104 to the input unit 124, where the vehicle driver 104 confirms that steering should be executed. The method ends in step S224.

[0049] Individual driving maneuvers, such as passing through an intersection, can be decomposed into a plurality of maneuvers in the sense of the method described individually, and steps S202 to S222 are repeated for each of these maneuvers. In the first maneuver, the vehicle 102 approaches the intersection and stops at the stop line. The control unit 116 identifies in step S214 an attention range covering a distant field of view. This is because the entire traffic space around the vehicle 102 should be monitored by the vehicle driver 104. In the second maneuver, the vehicle 102 starts moving. The control unit 116 determines in step S214 an attention range focused on the vicinity of the right front. This is because when the vehicle 102 starts moving, a person located in front of the vehicle 102 at the start, such as a pedestrian just crossing the lane, should not be put at risk. In the third maneuver, the vehicle 102 finally passes through the intersection. The control unit 116 determines in step S214 an attention range covering a distant field of view again. Further, the control unit 116 identifies an attention range corresponding to the position of each other traffic participant along the trajectory of the vehicle 102 in the field of view of the vehicle driver 104.

[0050] In the embodiment described based on FIGS. 1 and 2, at least a surrounding detection unit 106, an occupant detection unit 114, and a control unit 116 form a device 100 for operating a vehicle 102. The other elements and features shown in FIGS. 1 and 2 and mentioned in the above description may be part of the device 100. Similarly, the method steps described based on the device 100 may also be part of the claimed method.

Description of Reference Numerals

[0051] 100 Device 102 Vehicle 104 Vehicle driver 106 Surrounding detection unit 108 Image detection unit 110 Traffic participant 112 Navigation unit 114 Occupant detection unit 116 Control unit 118 Memory range 120 Functional unit 122 Display unit 124 Input unit

Claims

1. An apparatus (100) for operating a vehicle (102), comprising: a surrounding detection unit (106) configured to detect at least one piece of surrounding information regarding the surroundings of the vehicle (102) and generate surrounding information corresponding to the detected surrounding information; an occupant detection unit (114) configured to detect at least one image having a depiction of a vehicle driver (104) of the vehicle (102) and generate driver image data corresponding to the detected image; a control unit (116) configured to identify at least one driving maneuver of the vehicle (102) to be performed by the vehicle (102) in assisted driving or automated driving based on the surrounding data and generate control data corresponding to the identified driving maneuver; wherein the control unit (116) is further configured to: determine at least two maneuver-specific attention ranges in the field of view of the vehicle driver (104) and at least one maneuver-specific attention criterion regarding the visual activity of the vehicle driver (104) based on the control data, identify whether and for how long the line of sight of the vehicle driver (104) is directed to each of the attention ranges based on the driver image data, generate visual activity data corresponding to the result of the identification of the visual activity, compare the visual activity data with the attention criterion, and generate a control command for the assisted driving or automated driving of the vehicle (102) based at least on the comparison. The apparatus (100) according to claim 1, characterized in that it is formed as such.

2. The control unit (116) is configured to: identify whether the vehicle driver (104) is paying attention based at least on the comparison of the visual activity data with the attention criterion, and generate the control command such that the vehicle (102) performs the maneuver only when the control unit (116) determines that the vehicle driver (104) is paying attention in accordance with the attention criterion. The apparatus (100) according to claim 1, characterized in that it is formed as such.

3. including an input unit (124) of the vehicle (102), wherein the control unit (116) is configured to: After first determining that the vehicle driver (104) is not paying attention in accordance with the attention criteria, when the vehicle driver (104) operates a predefined input to the input unit (124), so as to generate the control command for executing the operation The device (100) according to claim 2, characterized in that it is formed as such.

4. The attention criteria are the following conditions: The minimum time and / or maximum time for which the line of sight of the vehicle driver (104) needs to be directed to a predetermined range and / or any range within the attention range The device (100) according to any one of claims 1 to 3, characterized in that it includes at least one of them.

5. The control unit (116) is As a specific part of the visual activity, while giving points, according to whether the line of sight of the vehicle driver (104) is directed to the attention range respectively, and for how long, give points and / or subtract points as such Formed, and the attention criteria include at least one minimum number of points that need to be achieved at least as a specific part of the visual activity. The device (100) according to any one of claims 1 to 4 is characterized in that.

6. The control unit (116) is Based on the ambient data and the operation data, identify at least one operation focus in the field of view of the vehicle driver (104), and determine the attention range so that at least one of the attention ranges includes the operation focus as such The device (100) according to any one of claims 1 to 5, characterized in that it is formed as such.

7. The control unit (116) is Based on the ambient data, identify the position of at least one other traffic participant (110) in the field of view of the vehicle driver (104), and identify at least one of the attention ranges corresponding to the position of the other traffic participant (110) in the field of view of the vehicle driver (104) as such The device (100) according to any one of claims 1 to 6, characterized in that it is formed as such.

8. The control unit (116) is Based on the ambient data, identify the traffic situation around the vehicle (102), generate traffic situation data corresponding to the traffic situation, and identify the attention range in consideration of the traffic situation data as such The apparatus (100) according to any one of claims 1 to 7, characterized in that it is formed.

9. The surrounding detection unit (106) includes at least one image detection unit (108) of the vehicle (102), and the image detection unit detects at least one image having a description of the surrounding of the vehicle (102) as surrounding information and generates external image data corresponding to the image. The apparatus (100) according to any one of claims 1 to 8, characterized in that the surrounding data includes the external image data.

10. The surrounding detection unit (106) includes at least one navigation unit (112), and the navigation unit detects and / or receives at least one navigation information regarding the vehicle (102) and generates navigation data corresponding to the navigation information. The apparatus (100) according to any one of claims 1 to 9, characterized in that the surrounding data includes the navigation data.

11. The control unit (116) determines the line-of-sight direction and / or head posture of the vehicle driver (104) based on the driver image data, specifies line-of-sight direction data corresponding to the line of sight and / or the head posture of the vehicle driver (104), and performs the specification of the visual activity in consideration of the line-of-sight direction data. The apparatus (100) according to any one of claims 1 to 10, characterized in that it is formed.

12. The control unit (116) stores the line-of-sight direction data in the storage range (118) of the control unit (116) and is formed to specify the attention range in consideration of the line-of-sight direction stored in the past. The apparatus (100) according to claim 11, characterized in that it is formed.

13. The control unit (116) specifies at least one characteristic of the vehicle driver (104) regarding the driving task of the vehicle driver (104) based on the driver image data, generates characteristic data corresponding to the characteristic, and generates a control command for the assisted driving or automated driving of the vehicle (102) in consideration of the characteristic data. The apparatus (100) according to any one of claims 1 to 12, characterized in that it is formed.

14. The apparatus (100) according to any one of claims 1 to 13, characterized in that the control unit (116) is configured to activate at least one display unit (122) of the vehicle (102) in order to display the attention range and / or a specific state of the visual activity to the vehicle driver (104).

15. A method of operating a vehicle (102), comprising: detecting at least one piece of ambient information regarding the surroundings of the vehicle (102) and generating ambient information corresponding to the detected ambient information; detecting at least one image having a depiction of a vehicle driver (104) of the vehicle (102) and generating driver image data corresponding to the detected image; identifying at least one driving maneuver of the vehicle (102) to be executed by the vehicle (102) in assisted driving or automated driving based on the ambient data and generating maneuver data corresponding to the identified driving maneuver; determining at least two maneuver-specific attention ranges in the field of view of the vehicle driver (104) and at least one maneuver-specific attention criterion regarding the visual activity of the vehicle driver (104) based on the maneuver data; identifying whether the line of sight of the vehicle driver (104) is directed towards each of the attention ranges and for how long, based on the driver image data, and generating visual activity data corresponding to the specific result of the visual activity; comparing the visual activity data with the attention criterion, and generating a control command for assisted driving or automated driving of the vehicle (102) based at least on the comparison characterized by the above method.

Images