OCCUPANT ATTENTION METHOD

Abstract

The invention discloses a computer-implemented method 140 for directing the attention of an occupant 168 of a vehicle 168, wherein the vehicle 168 is configured to drive automatically. The method 140 comprises steps 142, 144 in which first sensor data 112 is obtained, on the basis of which a current attention state 160 of the occupant 168 is determined. The method further comprises steps 146, 148 in which second sensor data 128 is obtained, on the basis of which a future desired attention state 162 of the occupant 168 is determined. If the current attention state 160 does not correspond to the future desired attention state 162, an attention directing function 164 is selected, which directs the attention of the occupant 168 to a specific object and / or a specific activity.

Description

Technical area

[0001] The present invention relates to the control of automated vehicles. In particular, the present invention relates to computer-implemented methods and devices for directing the attention of a vehicle occupant. Accordingly, the present invention also relates to a computer program product, a computer-readable storage medium, a data carrier signal, and a vehicle. Technical background and task

[0002] The SAE J3016 standard describes the classification and definition of terms for on-road vehicles with automated driving systems. It defines six levels: Level 0 corresponds to no driving automation, Level 1 to assisted driving, Level 2 to partial driving automation, Level 3 to limited driving automation, Level 4 to highly automated driving, and Level 5 to fully automated driving.

[0003] As semi-autonomous driving methods (SAE Levels 3 and 4) become increasingly mature, it is essential to ensure that the driver can take control of the vehicle if necessary. The driver must therefore be able to focus their attention on the surroundings and the vehicle situation for a limited time in certain driving situations. For example, the vehicle's automated system may be overwhelmed by certain traffic situations, such as complicated intersections, or the driver may be required to drive at an average speed that the vehicle's automated system does not allow.

[0004] Current systems attempt to achieve this by issuing warnings. Advanced systems can also take the driver's current level of attention into account, thus reducing the warnings or adapting them to the situation. However, these systems have the disadvantage of operating reactively; they only act when the driver needs to take over control of the vehicle.

[0005] Another problem with existing systems is that they are based on a warning. However, a warning is perceived as a strong negative stimulus and therefore reduces driving comfort. Furthermore, reactive notifications can arrive at inconvenient times for the driver, for example, because they are engrossed in a demanding task.

[0006] It is therefore the object of the present invention to provide computer-implemented methods and devices for directing the attention of a vehicle occupant, which eliminates at least one of the aforementioned disadvantages. Furthermore, the object of the invention is to provide a computer program product, a computer-readable storage medium, a data carrier signal, and a vehicle. Disclosure of the invention

[0007] The object is achieved according to the invention by the features of the main claims. Advantageous embodiments can be found in the subclaims.

[0008] According to a first aspect of the invention, a computer-implemented method for directing the attention of an occupant of a vehicle configured for automated driving comprises a step in which first sensor data is obtained. A vehicle can be a motor vehicle, a truck, an aircraft, a helicopter, a train, a bus, or the like. Directing attention is understood to mean the intention of directing the occupant's concentration and perception toward specific stimuli in the environment. The occupant is, for example, a driver of the vehicle.

[0009] The method comprises a further step in which the occupant's current level of attention is determined based on the first sensor data. The first sensor data must therefore be suitable for determining the occupant's current level of attention. For example, this can be achieved by analyzing image data of the occupant using a neural network to identify the current mood and / or activity. For example, a bored facial expression and no specific activity could indicate that the occupant is bored. Or the occupant is reading a book, which is captured and recognized via the image data. In this case, it can be assumed that the occupant's level of attention is low with regard to the current driving situation. A heart rate sensor or a skin perspiration sensor can also acquire data that allows conclusions to be drawn about the occupant's current level of attention.For example, a very low heart rate suggests that the occupant is very relaxed and not paying much attention.

[0010] The method includes a further step in which second sensor data is obtained. The second sensor data is generally not identical to the first sensor data.

[0011] The method comprises a further step in which a desired future attention level is determined based on the second sensor data. For example, the second sensor data can first be used to predict a future traffic situation, for example by using position tracking sensors such as GPS and suitable route planning programs with access to traffic density data. A decision can then be made as to whether the future traffic situation and the associated driving maneuvers to be performed can be handled by the vehicle in automated driving mode. If this is not the case, the driver should be informed early on that they will have to assume manual control of the vehicle in the future. "Early on" here refers to the fact that an acute warning and the associated negative stimulus should be avoided.

[0012] If the current attention level does not correspond to the desired future attention level, the method comprises a further step in which an attention-directing function is selected that directs the occupant's attention to a specific object and / or a specific activity. The attention-directing function serves to adapt the current attention level (e.g., "low") to the desired future attention level (e.g., "high"). This is achieved, for example, through an activity that can be performed in the vehicle. For example, a game can be started that requires a high level of attention and is displayed directly in the occupant's field of vision via a head-up display.The game can be overlaid with additional information such as characters or images to further prepare the occupant for the imminent need to take control of the vehicle. The attention-guiding function can also gradually trigger various activities to overcome the discrepancy between the current and the desired future level of attention.

[0013] In an advantageous embodiment, the attention-directing function is performed by the vehicle. The attention-directing function can, for example, be performed by a combination of a central computer (HPC, or high-performance computer) and a display device, such as a head-up display or an infotainment system.

[0014] In an advantageous embodiment, the desired future attention level correlates with a future driving maneuver of the vehicle. In other words, the occupant is induced to maintain a level of attention necessary to execute a future driving maneuver. This is relevant, for example, if the vehicle cannot perform the driving maneuver itself and the driver must therefore assume control. For example, the attention level corresponds to a specific SAE level or a related requirement (e.g., the driver must assume control in X seconds, namely as soon as traffic reaches an average speed of >Y km / h).

[0015] In an advantageous training, the attention-directing function corresponds to one of the following actions: reading and / or answering a text message, viewing a website, or starting a game.

[0016] Through the described procedures, the vehicle retains control over the driver's secondary activities, which allows the transfer of control to be initiated more reliably and more comfortably for the driver.

[0017] According to a second aspect of the invention, a computer program product comprises instructions which, when executed by a computer, cause the computer to execute a computer-implemented method as described above. The computer program product is typically written in a programming language such as Python or C++.

[0018] According to a third aspect of the invention, a computer-readable storage medium comprises instructions that, when executed by a computer, cause the computer to perform a computer-implemented method as described above. The computer-readable storage medium is typically a non-volatile memory such as an SSD (solid-state disk) or a flash memory.

[0019] According to a fourth aspect of the invention, a data carrier signal transmits the computer program product as described above. The transmission can be wired, for example, via a CAN bus. Wireless transmission via WLAN (wireless local area network), mobile communications such as 5G or 6G, or Bluetooth are also possible.

[0020] According to a fifth aspect of the invention, a device for directing the attention of a vehicle occupant has a first sensor unit, which is arranged in the interior of the vehicle and generates first sensor data that correlates with a current state of attention of the occupant. In other words, the first sensor unit provides the first sensor data for the method. The first sensor unit can be a single sensor or a group of sensors, which do not necessarily have to be embodied in a single component, but rather can also be distributed throughout the interior of the vehicle. These are arranged in the interior of the vehicle. For example, this can include a camera or a mobile device connected to the vehicle, such as a smartphone. The current state of attention of the occupant is determined using the first sensor data from the first sensor unit.

[0021] The device further comprises a second sensor unit, which generates second sensor data and is arranged on the vehicle in such a way that the second sensor data correlates with a current and / or future driving situation. The second sensor unit can also be a single sensor or consist of a group of sensors, which do not necessarily have to be embodied in a single component but can also be distributed throughout the vehicle. Examples of sensors in the second sensor unit are GPS (global positioning system) sensors, radar sensors, or lidar sensors. Using such sensors, the immediate surroundings of the vehicle as well as the current traffic situation at a distance can be determined.

[0022] Knowledge of the current traffic situation, near or far away, allows a prediction of whether the vehicle will be able to cope with it in the future, i.e., whether it can perform the necessary driving maneuvers; or whether the driver should assume control in the future. The term "future" typically refers to a time horizon of approximately 10 seconds or more, to allow sufficient time for the driver's attention level to adjust to the necessary driving maneuver or the future degree of automation before assuming control. However, the time horizon depends on the current attention level. For example, a sleeping driver needs more than 10 seconds to regain full attention than a driver reading a book.

[0023] The device further comprises an evaluation unit configured to execute a computer-implemented method as described above. For this purpose, the evaluation unit comprises, for example, a processor and at least one memory. Some of the steps of the method can be executed, for example, by neural networks stored in the at least one memory. The various components of the evaluation unit can be arranged in the vehicle or implemented decentrally, for example in the form of a server.

[0024] The device further comprises an action unit configured to perform an attention-directing function. The attention-directing function selected during the method can, for example, be a game that is displayed to the occupant on the action unit, for example, a head-up display.

[0025] All units are communicatively connected to the evaluation unit to enable the data flow necessary for a process described above.

[0026] In an advantageous embodiment, the first sensor unit comprises one or more sensors from the following group: camera, heart rate sensor, blood oxygen sensor, skin perspiration sensor, acceleration sensor. Some of the sensors, such as the camera, are often already installed in the vehicle for other purposes. Sensors for monitoring vital functions, such as blood oxygen sensors and skin perspiration sensors, will be increasingly installed in vehicles in the future. Furthermore, many of these sensors can be found in mobile devices such as smartphones, which are frequently connected to the vehicle by passengers. Such sensors can also provide the data of the first sensor unit.

[0027] In an advantageous embodiment, the second sensor unit comprises one or more sensors from the following group: radar, lidar, position tracking sensor, ultra-wideband sensor, or cellular sensor. Some sensor data from these sensors is suitable for recording the current traffic situation in the immediate vicinity of the vehicle by detecting the position and speed vector of other road users. These sensors can also detect the density of the traffic flow, differences in vehicle speeds, or the behavior of other drivers.

[0028] Other sensor data from these sensors is suitable for recording the current traffic situation at a greater distance, for example, by receiving traffic jam data. In conjunction with map data from a navigation system for a current route, values ​​such as maximum speed, average speed, traffic jam probability, or upcoming intersections can be assessed to define the desired future attention level. This allows, for example, a prediction of whether and when current SAE requirements may be violated.

[0029] According to a sixth aspect of the invention, a vehicle comprises a device as described above.

[0030] The present invention provides a positive interaction experience for the vehicle driver, as their needs are addressed empathetically. Takeover warnings can be reduced and made less invasive, as the driver is usually already engaged in an activity that allows for a "soft" takeover. Takeovers can be made more reliable and pleasant, as the vehicle retains control of the secondary activity. This means that the vehicle can potentially integrate warnings into the activity or, in an emergency, forcibly terminate it.

[0031] Further examples of the use of the present invention are: 1. The occupant is stuck in a traffic jam, but the traffic is expected to end soon. According to SAE levels, the driver must be able to take control of the situation at short notice. The occupant is offered the opportunity to read and respond to a missed text message, as this activity can be completed at short notice. 2. The occupant is stuck in a traffic jam that will continue for some time. According to SAE levels, the driver must be able to take control of the situation for a short time. The occupant is offered the opportunity to browse their current favorite websites, as this activity can be completed quickly but should take some time. 3. The occupant is on an autonomous long-distance journey, and the SAE level allows for a takeover period of several minutes. Furthermore, the occupant is clearly not challenged enough. A challenging video game is suggested, which can be played directly on a heads-up display. Summary of the characters

[0032] The invention is explained in more detail below using exemplary embodiments and figures. The figures show: Fig. 1: A device for directing the attention of a vehicle occupant; Fig. 2: A flowchart of a method for directing the attention of a vehicle occupant; Fig. 3: An embodiment as a block diagram of the device from Fig. 1, which describes the procedure Fig. 2; and Fig. 4: A vehicle with a device for directing the attention of an occupant. Detailed description of the characters

[0033] Fig. 1 shows a device 100 for directing the attention of an occupant 168 of a vehicle 166.

[0034] The device comprises a first sensor unit 102, which in turn comprises an interior camera 104 and a heart rate sensor 106. The interior camera 104 continuously generates image data 110, and the heart rate sensor 106 continuously generates heart rate data 110. The first sensor unit 102 merges the image data 110 and the heart rate data 110 into first sensor data 112.

[0035] The device has a second sensor unit 114, which in turn has a lidar 116, a radar 118, and a GPS sensor 120. The lidar 116 continuously generates lidar data 122, the radar 118 continuously generates radar data 124, and the GPS sensor 120 continuously generates GPS data 126. The second sensor unit 114 merges the lidar data 122, the radar data 124, and the GPS data 126 into second sensor data 118.

[0036] The device comprises an evaluation unit 130, which in turn comprises a processor 132 and a non-volatile memory 134. A computer program product 136 is stored on the non-volatile memory 134, the steps of which are described in detail in Fig. 2 is explained.

[0037] The device has an action unit 138 which serves to indicate an activity to the occupant 168 of the vehicle 166.

[0038] All devices 102, 114, 130, 138 are communicatively connected to each other, for example in a CAN-BUS system.

[0039] Fig. 2 shows a flowchart of a method 140 for directing the attention of an occupant 168 of a vehicle 166.

[0040] In a first acquisition step 142, the first sensor data 112 are obtained. These are used in a first determination step 144 to determine a current attention state 160 of the occupant 168.

[0041] In a second acquisition step 146, the second sensor data 128 are obtained. These are used in a second determination step 148 to determine a future desired attention state 162.

[0042] In a decision step 150, it is checked whether the current attention state 160 and the future desired attention state 162 match. The match does not have to be binary, but can also occur within a defined range of values ​​along a continuous scale.

[0043] If the current attention state 160 and the future desired attention state 162 match, the process can be restarted (t-branch).

[0044] If the current attention state 160 and the future desired attention state 162 match (f-branch), an attention steering function is selected in a selection step 152. This can be based on the discrepancy between the current attention state 160 and the future desired attention state 162. It can also be taken from an activity catalog 156, which, for example, links a future desired attention state 162 to a specific activity or in which a future SAE level, with which the vehicle 166 will be operated in the future, is linked to a specific activity.

[0045] In an execution step 154, the selected attention-direction function, which is intended to entice the driver 168 to take an action, is executed in the vehicle 166.

[0046] Fig. 3 shows an embodiment as a block diagram of the device 100 from Fig. 1, which describes the procedure 140 from Fig. 2 executes.

[0047] As in Fig. 1 shows the first sensor unit 102 with interior camera 104 and heart rate sensor 106. The interior camera 104 continuously generates image data 110, and the heart rate sensor 106 continuously generates heart rate data 110. The first sensor unit 102 merges the image data 110 and the heart rate data 110 to form first sensor data 112. These sensor types allow easy conclusions to be drawn about the current alertness level 160 of the occupant 168.

[0048] As in Fig. 1 shows the second sensor unit 114 with the lidar 116, the radar 118, and the GPS sensor 120. The lidar 116 continuously generates lidar data 122, the radar 118 continuously generates radar data 124, and the GPS sensor 120 continuously generates GPS data 126. The second sensor unit 114 merges the lidar data 122, the radar data 124, and the GPS data 126 to form second sensor data 118. These sensor types allow for easy inferences about the future desired alertness state 162 of the occupant 168.

[0049] Further shown is the evaluation unit 130 with the processor 132 and the non-volatile memory 134. Furthermore, the evaluation unit 130 has a volatile memory 158, for example, RAM. The non-volatile memory 134, the volatile memory 158, and the processor 132 can communicate bidirectionally with each other. Data can therefore be freely shared among them.

[0050] A computer program product 136 is stored on the non-volatile memory 134, which implements the method 140 of Fig. 2. The execution of the steps of method 140 occurs in processor 132 using volatile memory 158.

[0051] The first sensor data 112 and the second sensor data 128 are forwarded to the evaluation unit 130. Based on the first sensor data 112, the current attention state 160 of the occupant 168 is calculated in the processor 132 by the computer program product 136. For this purpose, the computer program product 136 has a correspondingly trained neural network. This can be realized, for example, by a CNN (convolutional neural network). Other forms of artificial intelligence or machine learning are, of course, conceivable for an alternative implementation. In the example of Fig. 3, the neural network recognizes on the basis of the image data 108 in the first sensor data 112 that the occupant 168 is reading a book and determines a “low” current attention state 160.

[0052] Based on the second sensor data 128, the future desired attention state 162 of the occupant 168 is calculated in the processor 132 by the computer program product 136. For this purpose, the computer program product 136 has an appropriately trained additional neural network. This can be implemented, for example, by a CNN. Other forms of artificial intelligence or machine learning are, of course, conceivable for an alternative implementation. In the example of Fig. 3, the further trained neural network uses the GPS data 126 within the second sensor data 128 to detect a traffic jam 5 km away along the planned route on which the vehicle 166, currently traveling at SAE level "4," is located. The further trained neural network then determines the future desired attention level 162 required for the traffic jam in the form of an SAE level "2."

[0053] The further trained neural network can abstract map data from a navigation unit and the environmental sensors. For example, information is automatically summarized to obtain a compact description of the environmental situation. This description concerns, for example, complexity, required reaction time, travel time until the situation changes, and the like.

[0054] The sensor types are chosen so that the neural networks are easy to train to recognize the respective attention states 160, 162. Experts have a variety of sensors at their disposal that they can use for this purpose.

[0055] Now, in the processor 132, the computer program product 136 selects an attention-guiding function 164 from an activity catalog 156 stored on the non-volatile memory 134. In the activity catalog 156, the future SAE level at which the vehicle 166 will be operated in the future, i.e. SAE level “2,” is linked to a specific activity with which the occupant 168 can be guided from his current “low” current attention state 160 to the future desired attention state 162, depending on the remaining time until the occupant 168 takes control of the vehicle 166. In the example of Fig. 3, the attention-direction function 164 is selected from the activity catalogue 156 to read out and answer missed short messages.

[0056] The activity catalog 156 can correlate a list of possible activities with a description of the complexity / attention required, as well as an expected timeframe for assuming control. This activity catalog 156 can, for example, be sorted and modified as a list based on user preferences. This list and preferences can be personalized, for example, based on the occupant's mobile device.

[0057] The attention-directing function 164 is then executed on the action unit 138 in the form of a head-up display in combination with a voice output from the vehicle's infotainment system 166. In other words, the driver 168 is prompted via voice output to respond to missed text messages read aloud by the infotainment system. At the same time, the message is displayed as text in the virtual image of the head-up display. Between the first and second messages, and also in the image of the head-up display, the driver 168 is also informed that they must take control of the vehicle 166 in approximately 5 km.

[0058] In this way, the driver 168 is gently guided to take control of the vehicle 166 by specifically controlling his attention and encouraging him to maintain the required level of attention until he takes control.

[0059] According to the invention, the attention level of the occupant 168 is continuously determined, and it is recognized when the occupant 168 begins to become bored or their attention level drops to such an extent that a potentially necessary assumption of control is no longer possible in accordance with SAE. To prevent the occupant 168 from turning to tasks where termination by warning would result in a strong, negative stimulus, a proactive activity is suggested. This activity is selected from the activity catalog 156 based on the environmental situation and the condition of the occupant 168. The complexity of the activity should be sufficient to challenge the occupant 168 to such an extent that they do not turn to other activities, yet be low-threshold enough to ensure problem-free assumption within the framework of the current SAE level and the environmental situation.If several activities meet these conditions, they can be sorted or filtered according to user preference.

[0060] Fig. 4 shows a vehicle 166 with a device 100 for directing the attention of an occupant 168. The device 100 is not designed as a single component.

[0061] The vehicle has an interior camera 104, whose field of view 170 can capture the occupant 168. Furthermore, the vehicle has a lidar 116 and a radar 118, which detect the immediate surroundings of the vehicle 166. Furthermore, the vehicle has a GPS 120, which can determine the current position of the vehicle 166.

[0062] In an evaluation unit 130, the sensor data of the sensors 104,116,118,120 can be evaluated, as for Fig. 3. The attention-direction function 164 selected in this way can be executed on the head-up display 138 mounted in the vehicle 166. List of reference symbols 100 device 102 First sensor unit 104 Interior camera 106 Heart rate sensor 108 image data 110 heart rate data 112 First sensor data 114 Second sensor unit 116 Lidar 118 radars 120 GPS 122 Lidar data 124 radar data 126 GPS data 128 Second sensor data 130 Evaluation unit 132 processor 134 Non-volatile memory 136 Computer program product 138 Action Unit / HUD 140 procedures 142 First conservation step 144 First investigative step 146 Second conservation step 148 Second investigation step 150 decision step 152 Selection step 154 Execution step 156 Catalogue of activities 158 Volatile memory 160 Current state of attention 162 future desired state of attention 164 Attention-directing function 166 vehicles 168 inmates 170 field of view

Claims

[1] Computer-implemented method (140) for directing the attention of an occupant (168) of a vehicle (166), wherein the vehicle (166) is designed to drive automatically, the method (140) comprising the following steps: a) receiving (142) first sensor data (112), b) determining (144) a current state of attention (160) of the occupant (168) on the basis of the first sensor data (112), c) receiving (146) second sensor data (128), d) determining (148) a future desired attention state (162) based on the second sensor data (128), and, if the current attention state (160) does not correspond to the future desired attention state (162): e) selecting (152) an attention-directing function (164) which directs the attention of the occupant (168) to a specific object and / or activity. [2] Computer-implemented method according to claim 1, characterized by that the attention-directing function (164) is carried out by the vehicle (166). [3] Computer-implemented method according to claim 1 or 2, characterized by that the future desired state of attention (162) correlates with a future driving maneuver of the vehicle (168). [4] Computer-implemented method according to one of the preceding claims, characterized by that the attention-directing function (164) corresponds to one of the following actions: reading and / or answering a short message, displaying a website or starting a game. [5] A computer program product (136) comprising instructions which, when executed by a computer, cause the computer to execute a computer-implemented method (140) according to any one of the preceding claims. [6] A computer-readable storage medium (134) comprising instructions which, when executed by a computer, cause the computer to perform a computer-implemented method (140) according to any one of claims 1 to 4. [7] A data carrier signal which transmits the computer program product (136) according to claim 5. [8] Device (100) for directing the attention of an occupant (168) of a vehicle (166), comprising: a) A first sensor unit (102) which is arranged in the interior of the vehicle (166) and generates first sensor data (112) which correlate with a current state of attention (160) of the occupant (168), b) A second sensor unit (114) which generates second sensor data (128) and is arranged on the vehicle (166) in such a way that the second sensor data (128) correlate with a current and / or future driving situation, c) An evaluation unit (130) designed to carry out a computer-implemented method (140) according to one of claims 1 to 4, and d) An action unit (138) designed to perform an attention-directing function (164), wherein all units (102,114,130) are communicatively connected to the evaluation unit (138). [9] Device according to claim 8, characterized by that the first sensor unit (102) has one or more sensors from the following group: camera (104), heart rate sensor (110), blood oxygen sensor, skin sweat sensor, acceleration sensor. [10] Device according to claim 8 or 9, characterized by that the second sensor unit (114) comprises one or more sensors from the following group: radar (118), lidar (116), position tracking sensor (120), ultra-wideband sensor, mobile radio sensor. [11] Vehicle (166) comprising a device (100) according to any one of claims 8 to 10.

Images