Handover assistant for machine-to-driver transitions

Abstract

The invention relates to a handover assistant for machine-to-driver transitions. A vehicle handover system that monitors an environment of a vehicle is disclosed herein. A vehicle handover system receives a transition request to change vehicle control from an automated driving mode to a passenger of a vehicle. The vehicle handover system detects a critical event that may be related to the transition request, and the detection of the critical event is based on the monitored environment. The vehicle handover system may generate a handover scene that includes an image associated with the key event, and the image includes a sequence of images over a time period of the key event. A handover scenario is displayed to the passenger before the vehicle handover system changes vehicle control from an automated driving mode to the passenger.

Description

This application is a divisional application of the invention patent application filed on July 28, 2022, with a priority date of August 23, 2021, application number 202210898630.X, entitled "Transfer Assistant for Machine-to-Driver Transformation". Technical Field

[0001] This disclosure generally relates to autonomous driving systems, and more specifically to driving systems that can transfer control of a vehicle from autonomous driving mode to a human driver. Background Technology

[0002] Vehicles with autonomous or partially autonomous driving modes are becoming increasingly popular. These vehicles typically include various monitoring systems equipped with cameras and other sensors to observe information related to the vehicle's interior, monitor its movement, and scan for objects outside the vehicle. When a vehicle operates in autonomous (or partially autonomous) driving mode, it may become necessary to return control (or partial control) of the vehicle to the human driver. Attached Figure Description

[0003] In the accompanying drawings, the same reference numerals generally refer to the same parts throughout the different views. These drawings are not necessarily to scale, but generally focus on illustrating exemplary principles of this disclosure. In the following description, various exemplary aspects of this disclosure are described with reference to the following drawings, in which: Figure 1 The diagram illustrates an example flowchart of a vehicle handover system that allows a vehicle to switch from autonomous driving mode to manual driving mode. Figure 2 An exemplary vehicle handover system is shown that presents transition information to a human driver during vehicle handover. Figure 3 An exemplary flowchart depicts an event summary that prioritizes key events in an enhanced image sequence; Figure 4 Exemplary summary information is shown relating to multiple key events displayed in a single frame image of an enhanced image sequence. Figure 5 The illustration shows an exemplary enhanced image sequence that can depict one or more key events and associated metadata; Figure 6 An exemplary flowchart is shown for a response time system used to predict the expected response time of passengers and to determine the required response time in a specific geographical location and situation of a vehicle. Figure 7 An exemplary flowchart is depicted for determining the response time of passengers to detected events; Figure 8 The diagram illustrates an exemplary schematic of a device for transferring control of a vehicle from an automated driving mode to a manual driving mode. Figure 9 A schematic flowchart depicts a method for transferring control of a vehicle to passengers. Detailed Implementation

[0004] The accompanying drawings are used in the following detailed description and illustrate exemplary details and features by way of illustration.

[0005] The term “exemplary” is used in this document to mean “serving as an example, instance, or illustration.” Any aspect or design described as “exemplary” in this document should not be construed as being preferred or advantageous over other aspects or designs.

[0006] Throughout the accompanying drawings, it should be noted that, unless otherwise stated, the same reference numerals are used to depict the same or similar elements, features, and structures.

[0007] The phrases “at least one” and “one or more” can be understood to include a quantity greater than or equal to one (e.g., one, two, three, four, [...] etc.). The phrase “at least one of…” relating to a group of elements can be used herein to mean at least one element from the group consisting of these elements. For example, the phrase “at least one of…” relating to a group of elements can be used herein to mean a selection of: one of the listed elements, one of the multiple listed elements, multiple individual listed elements, or multiple of the multiple individual listed elements.

[0008] The terms "plural" and "multiple" in the specification and claims explicitly refer to a quantity greater than one. Therefore, any phrase that explicitly invokes the above terms to refer to a certain number of elements (e.g., "plural [elements]", "multiple [elements]") explicitly refers to more than one of the stated elements. For example, the phrase "aplurality" can be understood to include a quantity greater than or equal to two (e.g., two, three, four, five, [...] etc.).

[0009] The phrases “(...) group,” “(...) set,” “(...) collection,” “(...) series,” “(...) sequence,” “(...) grouping,” etc. (if present) in the specification and claims refer to a quantity equal to or greater than one, i.e., one or more. The terms “appropriate subset,” “reduced subset,” and “smaller subset” refer to a subset of a set that is not equal to that set; illustratively, they refer to a subset of a set that contains fewer elements than that set.

[0010] As used herein, the term "data" can be understood to include information provided in any suitable analog or digital form, such as as a file, a portion of a file, a collection of files, a signal or stream, a portion of a signal or stream, a collection of signals or streams, and so on. Furthermore, the term "data" can also be used to refer to information, for example, in the form of a pointer. However, the term "data" is not limited to the examples above and can take various forms and represent any information as understood in the art.

[0011] For example, the terms "processor" or "controller" as used herein can be understood as any kind of technical entity that allows the disposal of data. Data can be disposed of according to one or more specific functions performed by the processor or controller. Further, as used herein, a processor or controller can be understood as any kind of circuit, such as any kind of analog or digital circuit. A processor or controller can therefore be or may include analog circuits, digital circuits, mixed-signal circuits, logic circuits, processors, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), field-programmable gate arrays (FPGAs), integrated circuits, application-specific integrated circuits (ASICs), etc., or any combination thereof. Any other kind of implementation of the corresponding functions, which will be described in further detail below, can also be understood as a processor, controller, or logic circuit. It should be understood that any two (or more) processors, controllers, or logic circuits detailed herein may be implemented as a single entity with equivalent functionality, etc., and conversely, any single processor, controller, or logic circuit detailed herein may be implemented as two (or more) separate entities with equivalent functionality, etc.

[0012] As used herein, “memory” is understood to be a computer-readable medium (e.g., a non-transitory computer-readable medium) in which data or information can be stored and retrieved. The reference to “memory” as used herein is therefore understood to refer to volatile or non-volatile memory, including random access memory (RAM), read-only memory (ROM), flash memory, solid-state storage, magnetic tape, hard disk drive, optical drive, 3DXPoint, etc. TMetc., or any combination thereof. In this document, registers, shift registers, processor registers, data buffers, etc., may also be included by the term "memory". The term "software" refers to any type of executable instructions, including firmware.

[0013] Unless explicitly specified, the term "transmit" encompasses both direct (point-to-point) and indirect (via one or more intermediate points) transmission. Similarly, the term "receive" encompasses both direct and indirect reception. Furthermore, the terms "transmit," "receive," "transfer," and other similar terms encompass both physical transmission (e.g., transmission of radio signals) and logical transmission (e.g., transmission of digital data via a logical software-level connection). For example, a processor or controller may transmit or receive data in the form of radio signals via a software-level connection to another processor or controller, wherein physical transmission and reception are handled by radio layer components such as RF transceivers and antennas, and logical transmission and reception via a software-level connection are performed by the processor or controller. The term "transfer" encompasses one or both of transmission and reception, i.e., one-way or bidirectional transmission in one of the incoming and outgoing directions or in both directions. The term "computation" encompasses both 'direct' computation via mathematical expressions / equations / relations and 'indirect' computation via lookup tables or hash tables and other array indexing or search operations.

[0014] "Vehicle" can be understood to include any type of driven object. For example, a vehicle can be a driven object having an internal combustion engine, a reaction engine, an electric drive, a hybrid drive, or a combination thereof. Vehicles can be or can include automobiles, buses, minibuses, vans, trucks, motorhomes, vehicle trailers, motorcycles, bicycles, tricycles, locomotives, train carriages, mobile robots, personal transport aircraft, boats, ships, submersibles, submarines, drones, airplanes, or rockets, etc.

[0015] "Passenger" can be understood to include any person inside the vehicle. As an example, a passenger may be seated in a driver's seat (e.g., behind the steering wheel) or a passenger seat (e.g., not behind the steering wheel). A passenger can be understood as the "driver" of the vehicle, whether the driver is actively controlling the vehicle (e.g., the vehicle may be controlled by autonomous driving mode or partial autonomous driving mode) or simply allows autonomous mode to control the vehicle.

[0016] The devices and methods described herein can be implemented using a hierarchical architecture, for example, by prioritizing access based on spectrum priority, or by introducing hierarchical usage priority ranking for different types of users (e.g., low / medium / high priority, etc.) (e.g., where the highest priority is given to tier 1 users, followed by tier 2 users, then tier 3 users, and so on).

[0017] Vehicles, including autonomous or partially autonomous driving modes, are typically equipped with monitoring systems that are usually associated with safety systems used to control the autonomous driving mode or to warn the driver of objects that may be present in the vicinity of the vehicle. Monitoring systems typically include various inputs, sensors, cameras, and other information-gathering devices to assist autonomous driving modes in making movements, following specific routes, avoiding collisions, and generally operating the vehicle safely as the environment around it dynamically changes. When a vehicle is operating in a fully autonomous or partially autonomous driving mode, it may become necessary to transfer control of the vehicle back to the human driver. However, current systems do not adequately assess whether the human driver is ready to assume control of the vehicle. As discussed in detail below, this disclosure provides a system for monitoring, recording, and displaying recent events that can inform the human driver about the current status of the vehicle at the time of handover. The system can also assess the expected reaction time of passengers in order to notify the handover process and / or other vehicle safety / monitoring systems of the expected reaction time of passengers. The system can also assess whether the driver is confused about how to react correctly to the current situation of the vehicle or how to operate the vehicle safely, and the system can subsequently use this information during the handover process or notify other safety / monitoring systems on the vehicle. Therefore, the system provides relevant past events that may not yet be perceived by passengers about to take control of the vehicle from automated driving mode, thereby improving the safety of the handover. Safety can be further improved by customizing how and on what basis information is rendered, to enhance the transfer of knowledge to passengers before and during the handover.

[0018] Figure 1The diagram illustrates an example flowchart 100 of a vehicle handover system for a vehicle transitioning from an autonomous driving mode to a manual driving mode. Reading flowchart 100 from left to right, the vehicle can operate in an autonomous driving mode 120. Autonomous driving mode 120 may include fully or partially automated control of the vehicle, where an automated vehicle system (such as automated vehicle system 110) can control the vehicle's movement along a route by monitoring various sensors and inputs that provide information about the vehicle and its surrounding environment. When in autonomous driving mode 120, automated vehicle system 110 can sense, record, and track information related to the environment around the vehicle, the vehicle's movement, and the movement of objects near the vehicle. The automated driving system 110 can use any number of sensors to collect such environmental information, including, for example, cameras, light detection and ranging (LiDAR) sensors, vehicle position sensors, vehicle speed sensors, accelerometers, gyroscopes, etc.

[0019] Before the vehicle relinquishes control from autonomous driving mode 120 to the driver in manual driving mode 140 (e.g., to a human driver 150), there may be a transition period 130 during which the system can transmit critical event information (e.g., events that may affect the safety of the handover) to the human (e.g., in information transmission 132), informing the human driver of the current situation of the vehicle and facilitating a safe handover. Information transmission 132 may be designed to inform the driver of the current situation of the vehicle and recent events that may be important for the driver to safely regain control of the vehicle. The system may identify critical event information (e.g., critical event information 115) associated with the current situation or recent events and display that critical event information (or a portion thereof) as text and / or images on a screen in the vehicle (e.g., on screen 135), verbally express / read the critical event information to the driver via the vehicle's audio system, provide a text message or alert with the critical event information on the driver's smartphone, and / or notify the driver of critical events that may be relevant to a safe transition in any other way.

[0020] Figure 2The figure illustrates an exemplary vehicle handover system 200 for presenting transition information to a human driver during the handover of a vehicle from autonomous driving mode to manual driving mode. When in autonomous driving mode, in module 210, the vehicle continuously monitors its motion and surrounding environment to ensure safe operation. Vehicle / environment sensors may include any number of sensors and combinations of sensor / inputs, including, for example, cameras, light detection and ranging (LiDAR) sensors, vehicle position sensors, vehicle speed sensors, accelerometers, gyroscopes, etc. A critical event detection module 220 receives sensor information from the vehicle / environment sensor module 210 and uses this information to determine whether it may be related to a critical event that may be relevant to the vehicle handover.

[0021] Critical events detected by the critical event detection module 220 may include movement events that, while in autonomous driving mode, cause or may cause the vehicle to change its trajectory (e.g., a change from a previously planned trajectory), such as when the vehicle detects an unexpected object on the road (e.g., an object falling from another vehicle in front of it) or when the vehicle detects an unmapped temporary road sign (e.g., a person waving a flag signaling slowdown in a construction zone). Critical events may also include unexpected behavioral events associated with other objects or vehicles approaching and / or in the monitored environment surrounding the vehicle. This may include, for example, vehicles violating traffic rules (e.g., ignoring a red traffic light) or vehicles whose trajectories do not follow the expected behavior / route (e.g., irregular movement along a straight road, unwarranted or irresponsible lane changes, etc.). Critical events may also include out-of-domain events, where the vehicle detects a situation indicating that it is operating outside a specific driving profile / domain that is safe for operation in a particular geographic location and / or condition (e.g., detecting snow on a mountain at night without street lighting, making it possible that the current domain / profile is no longer satisfied). The critical event detection module 220 may use a data-driven model (e.g., a deep neural network (DNN)) to analyze sensor data of the external environment to identify any events that may be related to the transfer of control of the vehicle to a human driver.

[0022] The critical event detection module 220 can detect critical events using dedicated modules associated with each category of critical event described above. For example, the critical event detection module 220 can detect critical events by analyzing sensor data (such as video frames from an external camera used for object detection) that may be received from the vehicle / environment sensor module 210. The critical event detection module 220 can utilize vehicle preprocessing algorithms (e.g., vehicle tracking and trajectory analysis) to coordinate with other event detectors. Additionally, the critical event detection module 220 can use diagnostic or output signals from other vehicle modules to detect other types of events. The critical event detection module 220 can buffer sensor / signal information associated with critical events into a scene summary, which includes: (i) a processed environmental model at a specified viewpoint; (ii) raw sensor / signal information received from the vehicle / environment sensor module 210 or other modules; and / or (iii) a pre-assessment of the importance of the sensor / signal information to the transition. The critical event detection module 220 can buffer scene summary information associated with critical events for potential use in the critical event storage module 230, which is discussed below.

[0023] Once the critical event detection module 220 has detected and buffered the critical event and its associated information, it passes the information to the critical event storage module 220. The critical event storage module 220 can store each critical event as an enhanced image sequence, which can be a data structure containing information related to the critical event over a time period and designed to convey how the critical event evolves over time. The enhanced image sequence can include multiple different viewpoints, such as a forward-facing vehicle camera view, a bird's-eye view, a rearview mirror view, a map view, etc. Each viewpoint can include any information associated with the critical event and can be arranged in different spatial perspectives. The critical event storage module 220 can generate any number of spatial perspectives, each based on information associated with the critical event. Additionally, the enhanced image sequence may include metadata information associated with key events, including, for example, information identifying when and where the event occurs, the type of event, the duration of the event and / or the enhanced image sequence, the image viewpoint of each image, and / or the pre-assessed importance of each frame (if provided by the key event detection module 220), which can enable the selection of a subset of frames or the emphasis of more important frames.

[0024] Additionally, the key event storage module 230 can make decisions based on information received from the passenger monitoring module 240 regarding which events to store, which metadata should be stored along with the events, and the time period of the stored events. The passenger monitoring module 240 can monitor passenger attributes and provide this information to the key event storage module 230. As will be discussed in detail later, the monitored passenger attributes may include, for example, passenger attention, passenger location within the vehicle, passenger estimated reaction time, passenger pattern confusion, and / or any other passenger attributes that may affect which events, the type of information related to the events, and the level of detail associated with the events that may need to be stored.

[0025] Next, the critical event storage module 230 can pass the stored information to the critical event prioritization module 250. The critical event prioritization module 250 can assess each critical event among the critical events by considering any number of variables to determine the importance of the event to the transition. As an example, the critical event prioritization module 250 can use time (e.g., comparing the time elapsed since the critical event) to rate more recent events as more important or less important (e.g., the older the event, the less important it is). As another example, the critical event prioritization module 250 can identify the categories of events and highlight events belonging to higher priority categories (e.g., highly dynamic events, highly unsafe events) as more important events. To achieve this, the critical event prioritization module 250 can use a fixed event category priority table to categorize events into one category within the event categories. As another example, the critical event prioritization module 250 can use the spatial location of the event (e.g., its proximity to the vehicle) to prioritize those events that are closer to the vehicle or on the planned route of the vehicle as more important events.

[0026] As another example, the critical event prioritization module 250 can prioritize its passenger attributes provided by the passenger monitoring module 240. For instance, the passenger monitoring module 240 can provide passenger focus information related to whether the driver is likely to have observed the critical event, missed it, or only observed a portion of it. For example, if the driver is looking to the right of the vehicle when the critical event occurs on the left side, the passenger monitoring module 240 can determine that the driver is less likely to be aware of the event. As another example, the passenger monitoring module 240 can determine that the driver is looking in the same direction as the critical event, but their attention is primarily focused on billboard advertisements. Therefore, the passenger monitoring module 240 can determine that the driver may be aware of some aspects of the critical event, but may not possess all the essential information. This type of passenger information can be used by the critical event prioritization module 250 to prioritize the importance of specific events to the transition and / or determine the scope of information needed regarding the critical event.

[0027] One objective of the Prioritizing Critical Events Module 250 is to summarize / reduce information related to critical events, focusing on information relevant to events that will be of utmost importance to the driver for consumption as part of the handover. The Prioritizing Critical Events Module 250 may use a selection heuristic to determine which events and which pieces of information related to those events are important to include in the summary. This selection heuristic may be a function of: (i) the urgency of the event to the current situation, which may be assessed, for example, by time dependence and spatial dependence; and (ii) the driver's potential cognitive gaps regarding the event (or gaps in information related to the event), in order to prioritize events (or information related to a given event) that the driver is most likely to miss.

[0028] To this end, the critical event prioritization module 250 can consider any of the dimensions discussed above (e.g., time, event category, spatial location, driver focus / field of view, etc.). Using this information (also known as the critical event heuristic), the critical event prioritization module 250 can make decisions about which perspectives / viewpoints should be included in the enhanced image sequence (e.g., sensor stream), which text / metadata should be displayed, and / or the length of the summary (e.g., number of frames). The result is an event summary that includes a sequence outline, which may include summarized images / videos and associated text descriptions of one or more critical events.

[0029] To perform prioritized event generalization, the priority critical event module 250 can use a custom network on top of any of several image recognition and generalization algorithms known to those skilled in the art. For example, the priority critical event module 250 can use a three-dimensional (3D) convolutional neural network (e.g., a 3D “Conv-Net” network for time series following a C3D architecture) to obtain a key feature set. The priority critical event module 250 can also use a two-dimensional (2D) convolutional neural network (e.g., a Conv-Net for action recognition following an I3D architecture) to obtain a key feature set. The priority critical event module 250 can also provide the obtained key feature set to a principal component analysis (PCA) module, which has a target length trigger that fuses the feature sets from the 3D and 2D Conv-Nets. The priority critical event module 250 can provide the fused feature set to (i) a decoder network that generates an image sequence 350 for rendering and (ii) a long short-term memory (LSTM) network that generates a text description of the sequence.

[0030] Figure 3 An exemplary flowchart 300 is shown, illustrating an event generalization that the critical event module 250 can perform on information related to critical events. For example, flowchart 300 illustrates a multimodal generalization system where the critical event module 250 can feed a time series 310 into a 3D convolutional neural network, such as 3D Conv-Net 320, which outputs a set of key features associated with the time series 310. Similarly, the critical event module 250 can feed an image frame 315 into a two-dimensional convolutional neural network (e.g., 2D Conv-Net 325) for action recognition and output a set of key features associated with the image frame 315. The key feature sets associated with the time series 310 and the key feature sets associated with the image frame 315 can flow into a principal component analysis module, such as PCA 330, which has a target length trigger that fuses the feature sets from 3D Conv-Net 320 and 2D Conv-Net 325. The output of PCA 330 can then flow into a decoder network, such as decoder 340, to produce an image sequence for rendering. The output of PCA 330 can also be fed into a long short-term memory network, such as LSTM 360, that generates a text description (e.g., text description 370) of the sequence.

[0031] Return to Figure 2The critical event prioritization module 250 can summarize multiple critical events into a single frame or as frames of overlaid images. The critical event prioritization module 250 can enhance each of the multiple critical events using text, descriptions, or other metadata. For example, once the critical event prioritization module 250 has generated a text description of the sequence, it can select events that can be displayed together in the same environment (e.g., the current environment of the vehicle), which may include events occurring, for example, within a relatively close proximity of the vehicle. The critical event prioritization module 250 can then place a portion of the generated text (e.g., a description of the event) and / or additional metadata (e.g., timestamps, time elapsed since the event occurred, etc.) at or near the current location of the object associated with the event (e.g., the vehicle that caused the unsafe situation event).

[0032] In this context, the current location may depend on whether the image is a bird's-eye view / map view or an augmented reality image (e.g., a forward-facing camera view). If the image is a bird's-eye view, the generated text and / or attached metadata may be located near the corresponding vehicle. If the image is augmented reality (e.g., a head-mounted display or inside a vehicle), the generated text and / or attached metadata can be attached as text labels pointing to the appropriate 3D space of the corresponding object associated with the event. Additionally, the priority critical event module 250 can adjust the transparency of objects (in conjunction with or as an alternative to text) as a way to indicate, for example, the amount of time that has elapsed since the event associated with the object occurred (e.g., objects associated with events that occurred long ago are more transparent (and therefore less important) than objects associated with recent events).

[0033] Figure 4An exemplary single frame (or frame of overlaid images) 400 is shown displaying summarized information related to multiple key events in a single image, as discussed in the following paragraphs. The single frame 400 contains three key events, all of which are displayed in the same environment as the self-vehicle vehicle 410. These three events are suitable for display in a single frame because they occur along road 440 within a relatively close proximity to the self-vehicle vehicle 410. One event relates to a nearby vehicle 430 that exhibits unsafe / unexpected behavior in the form of a braking event. For this event, the priority key event module 250 summarizes the event data by placing an image of the nearby vehicle 430 on road 440 at its position relative to the self-vehicle vehicle 410 and a message box 435 indicating the type / name of the event (e.g., braking event) and the time elapsed since the event (e.g., 5 seconds). Another event relates to a nearby vehicle 420 exhibiting an unsafe driving event, where vehicle 420 violates a safe distance between itself and an object ahead. For this event, the priority critical event module 250 summarizes the event data by placing an image of a nearby vehicle 420 on road 440 at its position relative to the autonomous vehicle 410, a double arrow indicating a measurement of the unsafe distance between the vehicle 420 and the object ahead, and a message box 425 indicating the type / name of the event (e.g., violation of safe distance) and the time elapsed since the event (e.g., 3 seconds). Another event is related to the autonomous vehicle 410 experiencing a sensor malfunction (incidentally, this is what triggers the autonomous driving system to determine to transfer control of the autonomous vehicle 410 to the human driver). For this event, the priority critical event module 250 summarizes the event data by placing an image of the autonomous vehicle 410 on road 440 at its current position relative to the vehicle 420, and a message box 415 indicating the type / name of the event (e.g., sensor malfunction) and the time elapsed since the event (e.g., now).

[0034] As can also be seen in a single frame 400, the Prioritization of Critical Events module 250 alters the transparency of each vehicle associated with a critical event based on the time elapsed since the event occurred. Thus, vehicle 410 uses thick lines in darker colors to visually indicate its association with a more recent event, while vehicle 420 uses thin lines in lighter colors to visually indicate its association with an event that occurred a few seconds ago. Of course, a single frame 400 is merely an exemplary way of summarizing multiple critical events (and the data associated with these events) in a single image, and it should be understood that the Prioritization of Critical Events module 250 can use different texts, perspectives, visual aids, etc., to summarize information associated with multiple critical events in order to effectively convey information to a human driver.

[0035] Once summarized, prioritize critical event module 250 (return to...). Figure 2 The summarized key events can be passed to the presentation key event module 260 for delivery to the driver (e.g., displaying them on a screen (e.g., on an in-vehicle infotainment system)) for the driver to consume, as discussed in more detail below.

[0036] The presentation critical event module 260 can receive a summarized sequence of images and / or a text description of the sequence from the priority critical event module 250 and prepare it in an efficient manner for presentation to the driver. As part of the preparation, the presentation critical event module 260 may determine or be notified of the urgency of the handover. For example, if the handover must be completed quickly, the available time for presenting the information is shorter, and the presentation must be reduced to fit within a short time frame. The presentation critical event module 260 may need to compress the information using, for example, a shorter, less detailed version of the information while still conveying the critical information in a minimally obstructive manner. If the handover is not urgent, the presentation critical event module 260 can present the information in a longer, more detailed manner.

[0037] The critical event presentation module 260 can use urgency to determine the boundaries of the presentation time window. The maximum boundary of the time window may depend on, for example, the amount of time available for handover (e.g., the maximum amount of time it would be safe to present information before the vehicle must relinquish control to a human driver). The minimum boundary of the time window may depend on the minimum time required to deliver the critical event to the driver with sufficient quality (e.g., 100%, 75%, or 50% quality), taking into account the number of events that must be displayed and the length of each event. If the minimum time required for replay exceeds the maximum boundary, the critical event presentation module 260 can compress the required time by removing event-related information or by, for example, following a priority scheme, to include only the highest priority events within the maximum boundary. The critical event presentation module 260 removes event-related information by sampling a subset of frames in the sequence (e.g., by fast sampling using isomorphic subsampling and / or by priority-based sampling based on the frame importance of each frame, removing less important frames). The presentation key event module 260 can select key events to be included in the presentation by using the priority of each event (e.g., selecting the highest priority event to be included in the presentation, and selecting the next event (or multiple events) in the priority hierarchy until the maximum boundary of the presentation time is reached).

[0038] The presentation key event module 260 can adapt the presentation format based on passenger attributes already monitored by, for example, the passenger monitoring module 240. For instance, the presentation key event module 260 can update the format based on the current location of passengers within the vehicle. Therefore, if the passenger monitoring module 240 reports that the driver is sitting in the driver's seat and looking forward when a handover is requested, the presentation key event module 260 can adapt the presentation format for the key event to be rendered from a vehicle-based perspective in the vehicle display area. Similarly, the passenger monitoring module 240 can report to the presentation key event module 260 that a passenger is looking at a personal device (e.g., a smartphone or laptop) when a handover is requested, so the presentation key event module 260 can adapt the format to a bird's-eye view and send it to the user's personal device.

[0039] The presentation key event module 260 can accept requests from passengers to view the presentation and / or re-render the presentation from an alternative viewpoint or in an alternative form or format. Additionally, if the presentation key event module 260 receives information from the passenger monitoring module 240 that the passenger's attributes have changed or that the passenger may have missed the presentation (e.g., the passenger was moving towards the driver's seat at the time of presentation), the presentation key event module 260 can replay the presentation, taking into account the passenger's updated attributes (e.g., a format adapted to the presentation for the passenger's new location, the vehicle's current location, etc.).

[0040] Figure 5 Displays an exemplary enhanced image sequence 500 that can depict key events and associated metadata. The system can generate image sequence 500 in a storage module (e.g., the key event storage module 230 described above with respect to system 200) and can modify image sequence 500 in a priority ordering (e.g., the priority treatment key event module 250 described above with respect to system 200) and / or a presentation module (e.g., the presentation key event module 260 described above with respect to system 200). In this particular example, it should be understood that the key event is another vehicle (vehicle 515) on the road in front of the self-contained vehicle (vehicle 565), where the system detects that this other vehicle is exhibiting unexpected / unsafe behavior. Figure 5 The image sequence 500 depicted may consist of multiple perspectives of key events, one of which may be a front camera view from the perspective of a driver sitting in the driver's seat, and another perspective may be a bird's-eye view from above the vehicle. The front camera view is depicted as a series of images (front image sequence 510), each of which shows the vehicle 515 at a different point in time (where the increments in time are plotted from left to right on the horizontal axis at t=0, t=1, t=2, ..., t=5). Similarly, the bird's-eye view is depicted in a second series of images (bird's-eye view image sequence 550), each of which shows the vehicle 515 at a different point in time compared to the self-vehicle vehicle 565 (the self-vehicle vehicle 565 is at the bottom of each image).

[0041] It should be readily understood from the bird's-eye view image sequence 550 that the system can enhance the image sequence so that it not only displays the raw data (e.g., raw camera images from a camera on vehicle 565), but can be a description of the scene generated based on sensor data (multiple) of sensors. The system may, for example, create each of the enhanced images in the bird's-eye view image sequence 550 based on sensor data from multiple sensors to generate a bird's-eye view where no camera is present (e.g., there may be no bird's-eye view camera on the vehicle itself to capture that view). The system can enhance the image sequence or individual image frames in the sequence in other ways (including, for example, highlighting certain image frames, adding annotations to identify objects, adding tracking information for objects, etc.). The bird's-eye view image sequence 550 includes, for example, enhancement in the form of a trajectory 575, which depicts the movement trajectory of vehicle 565 over time, such that the length of the trajectory increases with time.

[0042] Additionally, the enhanced image sequence 500 may (e.g., within the enhanced image) include (e.g., depict) or utilize any metadata associated with the key event. For example, in the enhanced image sequence 500, the metadata is displayed as text boxes 360 below the foreground image sequence 510 and the bird's-eye view image sequence 550. The text box 360 displays, for example, the category of the event (e.g., “unstable transportation”), the duration of the event (e.g., 3 seconds), the time of occurrence (e.g., “12:00 AM”), the location of the event (e.g., latitude 20.66682 and longitude 103.39182), the relative importance of each frame in the image sequence (e.g., the frame at time 0 has a relative importance of 0, the frame at time 1 has a relative importance of 2, the frame at time 2 has a relative importance of 10, the frame at time 3 has a relative importance of 10, the frame at time 4 has a relative importance of 2, and the frame at time 5 has a relative importance of 10), and the viewpoint of the image (e.g., the front camera view is the first sequence (e.g., front image sequence 510), and the bird's-eye view is the second sequence (e.g., bird's-eye view image sequence 550)). Of course, the system can visually summarize the image frames using metadata, for example, through tagging, coloring, tracking, modification, highlighting, etc.

[0043] As explained above regarding the 200th explanation of the transportation transfer system (see...). Figure 2 The key event storage module 230, the key event priority treatment module 250, and the key event presentation module 260 can receive input from the passenger monitoring module 240. The passenger monitoring module 240 can monitor passenger attributes, including, for example, passenger attention, passenger position in the vehicle, passenger reaction time, passenger potential confusion state, and / or any other passenger attributes that may be related to the handover.

[0044] An example of passenger attributes that the passenger monitoring module 240 can monitor is passenger reaction time. For example, if an event occurs requiring the automated driving system to transfer control of the vehicle to the driver, the passenger monitoring module 240 can estimate the passenger's response time and also determine the required response time for that specific event. Expected response time and / or required response time can be used, for example, by the critical event storage module 230 to make decisions regarding which events to store, which metadata should be stored with the events, and the time period of the stored events. As another example, expected response time and / or required response time can be used by the critical event prioritization module 250 to make decisions regarding which events should be prioritized and how the data associated with each event should be generalized. As yet another example, the vehicle handover system 200 can use expected response time and / or required response time to determine, for example, the amount of time required for the handover and / or, given the expected / required response time(s), whether the handover should actually take place.

[0045] By observing how passengers respond to previous events (e.g., comparing the driver’s actual response with the expected response to an event or simulated event), observing the current environment of the vehicle (e.g., weather, lighting conditions, road geometry, nearby objects, etc.), the current state / attention of passengers in the vehicle (e.g., field of view, focus of attention, distracting noise in the vehicle cabin, driver’s profile, etc.), and / or by correlating the information with map-based information, the passenger monitoring module 240 can estimate the passenger’s response time and also determine the required response time for a specific event.

[0046] Figure 6 An exemplary flowchart is shown for a response time system 600 used to predict the expected response time of passengers and to determine the required response time in a specific geographical location and situation of a vehicle. This response time system 600 can be implemented as part of a passenger monitoring module 240 (see [link]). Figure 2 The response time system 600 can, for example, monitor passenger reactions to certain events in the response monitoring module 610 to provide the passenger's experienced response time to the response time calculation module 640. For example, the response monitoring module 610 can detect a monitored event and then measure how quickly the passenger responds to that particular event. To make this determination, the response monitoring module 610 can use defined event categories and a list of associated response actions for each category. The response monitoring module 610 can then calculate the response time as the time between the occurrence of the monitored event and the passenger's execution of the response action. To detect a monitored event, the response monitoring module 610 can use data from any of a plurality of vehicle sensors and / or vehicle systems (e.g., cameras, LiDAR sensors, vehicle position sensors, vehicle speed sensors, accelerometers, gyroscopes, etc.) that sense the environment, vehicle movement, or detected objects. Once the response monitoring module 610 detects a monitored event, it can (e.g., via a lookup table of events and / or event categories) determine a predefined response action associated with that event.

[0047] Figure 7 The monitoring reaction module 610 is shown (see Figure 610). Figure 6 An exemplary flowchart 700 illustrates how events can be used to obtain the empirical response time of a specific monitored event. In 710, for example, an event occurs, which may be a measurable event. As an example, this could be a traffic light changing from red to green when a vehicle stops. In terms of the system, in 725, the system (e.g., Figure 6The monitoring and response module 610 detects that the traffic light has turned green and, in 735, determines that the associated response action of the traffic light change is that the driver depresses the accelerator, causing the vehicle to begin moving. Regarding the human driver, after the event occurs, in 720, the driver spends a certain amount of time perceiving the event, and in 730, implements an appropriate response action. In 740, the response time is calculated by calculating the time spent by the human driver taking action after the monitored event occurred.

[0048] Return to Figure 6 The monitoring response module 610 can monitor many types of events, not just those mentioned above. Figure 7 The described example traffic light events also include more nuanced and / or more complex events / response actions. For example, monitored events may include the occurrence of new road conditions or new speed limit signs requiring speed changes (e.g., traffic sign events), approaching objects to which vehicles should yield (e.g., traffic conditions with higher priority or approaching vehicles (e.g., right-of-way events), unexpected objects requiring braking, deceleration, acceleration, or lane changing actions (e.g., stopping events, steering events, deceleration events, acceleration events, etc.), and / or approaching curve events requiring deceleration and steering wheel turning (e.g., steering events). In addition to actual events, these events can be simulated by the monitoring response module 610. For example, the vehicle may generate simulated events as warning messages (or audible tones or messages) displayed to the driver, and associated response actions may include pressing a requested button or providing an audible confirmation. Simulated events may also include questions related to a specific event scenario, where response actions are used to select the correct response to the question, and the empirical response time is the time it takes for the driver to respond correctly.

[0049] The monitoring response module 610 can measure any number of passengers' experienced response times and provide them to the response time calculation module 640. If the monitoring response module 610 performs numerous experienced response measurements for a given passenger, the response time calculation module 640 can estimate the average response time for that particular passenger. The average response time can be the average response time for all events, or the response time calculation module 640 can categorize each event into larger event categories (e.g., traffic sign / signal light events, intersection / priority events, unexpected / sudden braking events, etc.) so that the response time calculation module 640 can calculate the average response time for the event category. This average can be a simple average or a moving average, where a moving average helps reduce the impact of any single particular event (e.g., a single instance of slow reaction time) on the overall average. A possible moving average can be calculated as ρ. 新的 =αρ 事件+(1-α)ρ 旧的 Where α is the weighting factor, and ρ 新的 This is the new average response time, ρ 事件 It is the response time of the current event, and ρ 旧的 This is the previous average response time.

[0050] The response time system 600 can also determine an environmental rating. For example, in 620, the environmental rating can be provided to and used by the response time calculation module 640. Because human response time while driving may depend on the current environment of the vehicle (e.g., weather experienced or approaching at the vehicle, light levels / conditions at or approaching the vehicle, road geometry, road type, vehicle speed, air temperature at the vehicle, presence of nearby objects, etc.), the response time calculation module 640 can use input from the environmental rating module 620 to calculate the expected response time. An example of an environmental rating that can be provided by the environmental rating module 620 is a visibility rating. For example, the environmental rating module 620 can detect low-contrast areas (e.g., areas obscured by shadows or twilight) or areas that may affect the speed at which a human driver's eyes adjust (e.g., areas rapidly transitioning from a very bright environment to a very dark environment), and then calculate a visibility rating factor that the response time calculation module 640 can use to rate the response time.

[0051] Another example of environmental rating that can be provided by the environmental rating module 620 is a congestion rating. For example, the environmental rating module 620 can detect the number of objects in the environment, the number of moving objects, the distance between objects, object flow near vehicles, etc., and then calculate a congestion rating factor, which can be provided to the response time calculation module 640 for rating the response time. Another example of environmental rating that can be provided by the environmental rating module 620 is an occlusion rating. For example, the environmental rating module 620 can locate occluded areas that may have a higher risk of sudden object appearance due to obstructed driver visibility. An occlusion rating can (e.g., by tracking the driver's head, eyes, etc.) take into account whether the driver is focused or frequently checking the occluded area, because if the driver is focused on the occluded area, the driver may be better prepared to react to objects suddenly appearing from behind the occluded area (e.g., faster response time). On the other hand, if the driver does not spend much time looking at the occluded area, the driver may be less prepared to react to objects suddenly appearing from behind the occluded area (e.g., slower reaction time).

[0052] The response time system 600 can also determine in-vehicle ratings. For example, in 630, the in-vehicle ratings can be provided to and used by the response time calculation module 640. Because human response time while driving may depend on the driver's attention rating (e.g., based on observed characteristics related to passengers, such as head direction, eye direction, body posture, heart rate, non-driving activities (e.g., secondary activities such as making a phone call, changing the radio, talking to another passenger, drinking a beverage or eating food, etc.), noise within the vehicle cabin, driving habits, etc.), the response time calculation module 640 can use input from the in-vehicle rating module 630 to calculate the expected response time. For example, the driver's head direction can be detected by in-vehicle sensors (e.g., cameras) to estimate the direction in which the driver's attention is focused. Similar sensing can detect eye direction and body posture to rate the driver's attention based on these in-vehicle observations. For example, the in-vehicle rating module 630 can determine the driver's expected field of vision based on observed characteristics of the driver's head, eyes, body posture, etc., wherein the in-vehicle rating of response time may be faster for events occurring near the center of the field of vision, and slower for events occurring near the periphery of the field of vision. As another example, the in-vehicle rating module 630 can also monitor the driver's reaction to external events, for example, by observing whether the driver turns his / her head toward a sound (e.g., a horn or emergency traffic siren), to provide a rating of response time.

[0053] The in-vehicle rating module 630 can continuously / repeatedly monitor (e.g., in the current environment / operational context of the vehicle) the driver's state to determine an in-vehicle rating using any number of observations over time. For example, recognizing that response time may vary depending on age (e.g., younger passengers may have faster responses than older passengers), driving style, physiology (e.g., people with shorter legs / arms may not react quickly), or other driver observations / demographics, the in-vehicle rating module 630 can use a driver profile to provide an in-vehicle rating. These driver observations can be combined with other types of passenger observations to arrive at an in-vehicle rating. For example, a person's body posture or position relative to the steering wheel may improve or worsen the rating (e.g., a driver with shorter arms may have a slower rating unless they sit closer to the steering wheel to compensate). As another example, heart rate can affect the rating; if a person has a low heart rate, their rating may be lower because the person may be close to drowsy and have a slower reaction time. As another example, the activities of people inside the vehicle can affect the rating, because passengers who frequently control the radio, check their smartphones, sneeze, and / or are engaged in lively conversations may have longer response times, so the vehicle-in-vehicle rating module 630 can adjust the rating accordingly. As another example, the vehicle-in-vehicle rating module 630 can monitor the driver's driving behavior over time to inform the vehicle-in-vehicle rating. For example, if the vehicle-in-vehicle rating module 630 determines that the driver frequently switches between not turning and sudden turns, this indicates a high level of distraction and poor response time, so the vehicle-in-vehicle rating module 630 can adjust the rating accordingly.

[0054] The response time system 600 can also collect map-based data, such as in 650, which can be provided to and used by the response time calculation module 640. For example, the required response time may depend on road location, structure, and surrounding objects, where these map-based factors may affect a driver's ability to see and react to unforeseen events. For instance, the required response time on a straight road in a rural area may (e.g., due to a lack of stimulation) be longer than at an intersection or crosswalk in an urban area (e.g., where there are many stimuli to keep the driver focused on reacting to events). The required response time may also depend on other map-based data, such as whether the road is near a school, filled with billboards or other distracting objects, or part of a confusing intersection. The map-based prediction module 650 can use such map-based data to determine a map-based prediction of the required response time.

[0055] The map-based prediction module 650 can use a multi-layered approach to determine the required map-based response time. The first layer may include static map data (e.g., a pre-generated response time map), the second layer may include environmental data (e.g., an environment-dependent response time map), the third layer may include driver data / predictive data (e.g., a driver-dependent response time map for drivers of a vehicle), and the fourth layer may include other driver data / predictive data (e.g., a shared response time map based on the responses of other drivers at the map location (e.g., average driver response time)). Any, some, or all of these layers can be used to generate, validate, or adjust the resulting map-based required response time. The map-based prediction module 650 can use data stored locally (e.g., at the vehicle location) or data obtained from a cloud-based service. Specifically, the fourth layer, including other driver data / predictive data, may be particularly suitable for cloud-based services, given that this data may include shared data from other drivers and external sources.

[0056] In a static map layer (e.g., the first layer), a map-based prediction module 650 can generate the required response time for a given map location. The static map layer can utilize road geometry / layout, nearby objects (e.g., buildings, vegetation, billboards, etc., whose response time may be affected by occlusion, speed limits, or the frequency of events occurring at the map location). For example, at a signalized intersection with traffic lights, the required response time may be lower (faster) because drivers may need to react quickly when the traffic light turns red.

[0057] In the environmental data layer (e.g., the second layer), the map-based prediction module 650 can adjust the required response time based on the environment of the vehicle at a specific map location. The environmental data layer can receive sensed, recorded, and / or tracked information relating to the environment around the vehicle, the movement of the vehicle, and the movement of objects approaching the vehicle, obtained locally from any number of sensors (for example, including cameras, light detection and ranging (LiDAR) sensors, vehicle position sensors, vehicle speed sensors, accelerometers, gyroscopes, etc.) or from external sources. For example, the environmental data layer can adjust the required response time of the map location based on the time of day (e.g., a faster response time may be needed during the daytime (e.g., at / near a location with a school when students are likely to be present)), the weather (e.g., a faster response time may be needed due to lower visibility during heavy rain), and / or the landscape of the location (e.g., open roads versus forest roads). Of course, the environmental data layer can use any number of environmental factors to adjust the required response time at the map location, and each environmental factor may have different effects when considered in combination (for example, severe weather may have different effects on response time depending on the landscape at the location).

[0058] In the driver data / prediction layer (e.g., the third layer), the map-based prediction module 650 can adjust the required response time based on driver observations that may be relevant to a specific map location. For example, if the driver data / prediction layer learns that the driver might be distracted at a particular map location (e.g., the driver is interested in a brand advertised at that location, or the driver has a retail store at that location, or if the driver has a friend living near that map location), the driver data / prediction layer can adjust the required response time to account for the potential distraction at that location. The driver data / prediction layer can also be used to normalize the response times generated / adjusted in the static map layer (e.g., the first layer) and / or the environmental data layer (e.g., the second layer) based on driver-specific data in the driver data / prediction layer. For example, if the driver data / prediction layer determines that the driver is an older driver with a higher response time, the driver data / prediction layer can increase all required response times associated with the map data.

[0059] In another driver data / prediction layer (e.g., the fourth layer), the map-based prediction module 650 can adjust the required response time based on information relating to the reaction times of other drivers at a particular map location. As previously mentioned, this layer may be particularly well-suited to utilizing a cloud-based data repository, allowing driver response times to be aggregated from many other drivers (e.g., crowdsourced), and allowing the response times calculated by the response time calculation module 640 to be (anonymously) sent to the cloud-based repository, where the layer can apply learning algorithms (e.g., neural networks) to improve the adjustment of response times based on the impact of each measurement. The cloud-based data repository may include spatial, temporal, and event attributes associated with each driver's response time, allowing other driver data / prediction layers to use this data to refine the required response time for a specific map location at a specific time of day. For example, cloud-based data associated with a forest road at dusk could indicate that many other drivers experienced an emergency braking event or accident (e.g., due to a deer crossing the road) at that time of day associated with that map location. Therefore, for example, if similar events are anticipated for safety reasons, other driver data / prediction layers can reduce the required response time for that map location to below a threshold level. Other driver data / prediction layers can also utilize or provide user profile data associated with specific drivers, allowing driver-specific adjustments to follow the driver from one mode of transport to another.

[0060] The response time system 600 can, for example at 640, use information received from a map-based prediction module 650 (and / or ratings from a monitoring response module 610, an environmental rating module 620, and / or an in-vehicle rating module 610) to determine the predicted response time for passengers and the required response time for the vehicle / driver at a specific vehicle location. The response time calculation module 640 can be located within the vehicle (e.g., onboard processor / memory) or can be cloud-based. Based on information received from other modules, the response time calculation module 640 can use a prediction algorithm to calculate the predicted response time (ρ) for passengers according to, for example, the following equation: total ). ρ total =max{ρ min , (ρ driver -ρ driver，baseline )+ρ map +ρ in-vehicle}

[0061] In the above exemplary equations, ρ min This is the minimum response time (e.g., a minimum time based on a regulatory body's requirement for a minimum handover time from autonomous driving mode to manual driving mode). ρ driverA unique response time rating for a specific driver, ρ driver The baseline response time rating ρ was offset from (e.g., from the monitoring response module 610) the specific situation / environment. driver,baseline . ρ map This is an adjustment to the response time of map location / (e.g., environmental rating from environmental rating module 620), and ρ in-vehicle This is an adjustment to the vehicle rating (e.g., from the vehicle rating module 630).

[0062] As described above, the response time system 600 can be implemented as a passenger monitoring module 240 of a vehicle handover system (see above). Figure 2 As part of the vehicle handover system, the response time system 600 can also provide passenger response time to other types of vehicle systems, such as vehicle safety systems that determine whether a vehicle should take action (e.g., braking, steering, decelerating, etc.). For example, if the expected response time for a passenger to react to a particular event exceeds the safe response time determined by the vehicle safety system, the vehicle can perform the necessary action instead of waiting for the driver to react.

[0063] On the other hand, the response time system 600 can also provide the autonomous driving system with passenger response times for a given map location and a given vehicle situation to determine the parameters of the autonomous driving system. For example, if the predicted passenger response time is longer than the minimum required response time for the vehicle situation (e.g., slower), the autonomous driving system can use the response time to limit driving parameters (e.g., set a maximum speed) or completely disable the autonomous driving system. As an example, suppose the vehicle is in autonomous driving mode and approaches a curved road with obstructions. If the driver is distracted, the response time system 600 may report a passenger response time longer than the required response time for the current situation. Therefore, if necessary, the driver may not be able to safely take control of the vehicle in the curve. Therefore, the system can (e.g., using audible, visual, or tactile alarms) warn the driver, (e.g., by requesting the driver to press a button or speak an audible response) ask the driver to confirm his / her focus, and / or completely disable the autonomous driving function (e.g., bring the vehicle to a safe stop).

[0064] Although the response time system 600 has been described above in relation to determining the driver's response time of a self-contained vehicle, the response time system 600 is not limited to this aspect. The response time system 600 can analyze the response times of pedestrians or other traffic participants based on the same type of analysis. The response time system 600 can provide the response time information of these other targets (e.g., pedestrians or other traffic participants) to the vehicle's safety systems (e.g., to determine how quickly a particular object is expected to react to other objects in a particular scenario / cluster) or to infrastructure equipment (e.g., traffic monitoring equipment, traffic lights, etc.) so that they can analyze the overall situation and / or publish the calculated response times to other traffic participants.

[0065] As mentioned above, passenger monitoring module 240 (see Figure 2 It can monitor numerous types of events (including simulated events) and passenger reactions to these events to determine, for example, passenger response times (see above regarding...). Figure 6 and Figure 7 (Discussion). In addition to response time, passenger monitoring module 240 can use this event-based information to determine whether a passenger is confused about the current situation of the vehicle (e.g., passenger "pattern confusion"). For example, if a passenger is expected to react specifically to an event (e.g., turn the steering wheel at an approaching bend or an object appearing on the road), but instead reacts differently (e.g., performs an unsafe action or fails to perform an action), passenger monitoring module 240 can determine that the passenger is experiencing pattern confusion. As described above, passenger monitoring module 240 can generate simulated events to measure whether a passenger is aware of the current situation of the vehicle or whether the passenger is experiencing pattern confusion. For example, passenger monitoring module 240 can display a warning message (e.g., display a warning message on a display screen or display a warning message as an audible tone / voice) that requests a specific response (e.g., press a button or speak an audible response), and if the passenger provides an incorrect response (e.g., press a different button, do not respond, or provide an unexpected audible response), passenger monitoring module 240 can determine that the passenger is experiencing pattern confusion.

[0066] Simulated events may also include posed questions related to a specific event scenario, with expected answers (e.g., the correct answer from a multiple-choice list of possible answers), and if a passenger provides an incorrect answer (e.g., selecting the wrong answer from the multiple-choice list), the passenger monitoring module 240 can determine pattern confusion. If pattern confusion exists, the vehicle handover system 200 can refuse to transfer control of the vehicle from the autonomous driving mode to the pattern-confused passenger. Additionally, pattern confusion can be used by the key event storage module 230, the key event prioritization module 250, and / or key event presentation module 260 to determine which events should be stored, prioritized, or presented, and to what extent such events are stored, prioritized, or presented (e.g., event length, event priority, and the length / summary of information within the event, depending on whether the passenger frequently exhibits pattern confusion in recurring situations).

[0067] Figure 8 This is a schematic diagram illustrating a device 800 for assisting in the handover of vehicles. The device 800 may include any of the features described above. Figure 8 This can be implemented as an apparatus, method, and / or computer-readable medium that performs the above-described features when executed. It should be understood that device 800 is merely an example, and other configurations including, for example, different or additional components are possible.

[0068] Device 800 includes a vehicle handover system 810. Vehicle handover system 810 includes a processor 820. The processor 820 of vehicle handover system 810 is configured to monitor the environment of the vehicle, in addition to or in combination with any of the features described in the following paragraphs. The processor 820 is also configured to receive a transfer request to change control of the vehicle from automated driving mode to a passenger in the vehicle. The processor 820 is also configured to detect a critical event that may be associated with the transfer request. This critical event is based on the monitored environment. The processor 820 is also configured to generate a handover scene including an image associated with the critical event. This image includes a sequence of images within a time period of the critical event. The processor 820 is also configured to display the handover scene to the passenger before the control of the vehicle is changed from automated driving mode to passenger.

[0069] In addition to any features, or a combination thereof, of the features relating to the vehicle handover system 810 in this paragraph and / or the preceding paragraphs, critical events may include at least one of the following: movement events that have caused or may cause a change in the trajectory of the vehicle; unexpected behavior events of the vehicle observed in the monitored environment; and / or external operational domain events involving the vehicle that do not conform to the expected operating conditions of the vehicle at the operational setpoint. Furthermore, in addition to any features, or a combination thereof, of the features relating to the vehicle handover system 810 in this paragraph and / or the preceding paragraphs, the images may further include event metadata associated with the critical events. This event metadata includes at least one of the following: the type of critical event; the time of the critical event; the location of the critical event; the duration of the critical event; the importance of each image in the images associated with the critical event; and / or the image viewpoint of each image in the images associated with the critical event. Furthermore, in addition to or in combination with any features of the features relating to the vehicle handover system 810 in this paragraph and / or the preceding paragraphs, the image may be generated based on sensor data relating to the vehicle. Furthermore, in addition to or in combination with any features of the features relating to the vehicle handover system 810 in this paragraph and / or the preceding paragraphs, the image may include an enhanced image sequence generated from at least one camera image, which may be enhanced using event metadata.

[0070] Furthermore, in addition to or in combination with any features of the features relating to the vehicle handover system 810 in this paragraph and / or the preceding paragraphs, the handover scene may include multiple perspectives corresponding to the passenger's viewpoint. Furthermore, in addition to or in combination with any features of the features relating to the vehicle handover system 810 in this paragraph and / or the preceding paragraphs, the multiple perspectives may include a forward-facing camera view, a bird's-eye view, and / or a map-based view of the vehicle. Furthermore, in addition to or in combination with any features of the features relating to the vehicle handover system 810 in this paragraph and / or the preceding paragraphs, the handover scene may include static images formed from images and / or event metadata associated with multiple key events. In addition, apart from any feature of this paragraph and / or the preceding paragraphs regarding the vehicle handover system 810, or in combination with any feature of this paragraph and / or the preceding paragraphs regarding the vehicle handover system 810, the still image may include at least one of the following: a single frame of an image associated with a plurality of key events and / or an overlaid image based on overlaying one or more images associated with a plurality of key events.

[0071] Furthermore, in addition to any features of the features in this paragraph and / or the preceding three paragraphs regarding the vehicle handover system 810, or in combination with any features of the features in this paragraph and / or the preceding three paragraphs regarding the vehicle handover system 810, the handover scene may further include text associated with the image. This text may be generated from the image using a multimodal generalization system. Furthermore, in addition to any features of the features in this paragraph and / or the preceding three paragraphs regarding the vehicle handover system 810, or in combination with any features of the features in this paragraph and / or the preceding three paragraphs regarding the vehicle handover system 810, the text may include a description of key events and the time at which the key events occurred. Furthermore, in addition to any features of the features in this paragraph and / or the preceding three paragraphs regarding the vehicle handover system 810, or in combination with any features of the features in this paragraph and / or the preceding three paragraphs regarding the vehicle handover system 810, the text may include a description of key events that can be verbally expressed to passengers via the vehicle's audio system. In addition, time may be the elapsed time since the occurrence of the critical event, except for any feature of this paragraph and / or the preceding three paragraphs concerning the vehicle handover system 810, or a combination thereof.

[0072] Furthermore, in addition to any feature of the features in this paragraph and / or the preceding four paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding four paragraphs concerning the vehicle handover system 810, critical events may be assigned a priority corresponding to the importance of the critical event relative to the current status of the vehicle. Furthermore, in addition to any feature of the features in this paragraph and / or the preceding four paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding four paragraphs concerning the vehicle handover system 810, this priority may be based on at least one of the following: the time elapsed since the critical event occurred, the category of the critical event, the distance from the location of the critical event to the current location of the vehicle, and / or the likelihood that the passenger is aware of the critical event. Furthermore, in addition to any feature of the features in this paragraph and / or the preceding four paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding four paragraphs concerning the vehicle handover system 810, the current location of the vehicle may include the expected location of the vehicle after the elapsed time. In addition, apart from or in combination with any of the features in this paragraph and / or the preceding four paragraphs concerning the vehicle handover system 810, the processor 820 may be further configured to monitor passengers to determine the likelihood that a passenger is aware of a critical event.

[0073] Furthermore, in addition to any feature of the features in this paragraph and / or the preceding five paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding five paragraphs concerning the vehicle handover system 810, the processor 820 may be further configured to determine the presentation duration for presenting the handover scene to the passenger. The presentation duration may depend on the urgency of the transfer request. Furthermore, in addition to any feature of the features in this paragraph and / or the preceding five paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding five paragraphs concerning the vehicle handover system 810, the processor 820 may be further configured to compress the length of the handover scene to fit within the presentation duration. Furthermore, in addition to any feature of the features in this paragraph and / or the preceding five paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding five paragraphs concerning the vehicle handover system 810, the compressed length may be based on removing a portion of the image data associated with the image from the handover scene based on its importance to the critical event. Furthermore, in addition to or in combination with any of the features in this paragraph and / or the preceding five paragraphs concerning the vehicle handover system 810, the handover scenario may include multiple key events. The compressed length may be configured to remove at least one key event from the handover scenario based on the priority of at least one of the multiple key events.

[0074] Furthermore, in addition to or in combination with any of the features of this paragraph and / or the preceding six paragraphs regarding the vehicle handover system 810, the processor 820 may be further configured to store the handover scenario as a stored handover scenario and to display the stored handover scenario to the passenger based on a request to review the stored handover scenario. Furthermore, in addition to or in combination with any of the features of this paragraph and / or the preceding six paragraphs regarding the vehicle handover system 810, the generation of the handover scenario may depend on the passenger's expected response time. Furthermore, in addition to or in combination with any of the features of this paragraph and / or the preceding six paragraphs regarding the vehicle handover system 810, the expected response time may be based on the passenger's average response time to multiple monitored events. In addition, in addition to any feature of this paragraph and / or the preceding six paragraphs regarding the features of the vehicle handover system 810, or in combination with any feature of this paragraph and / or the preceding six paragraphs regarding the features of the vehicle handover system 810, the plurality of monitored events may include at least one of the following: traffic sign events, right-of-way events, stopping events, turning events, deceleration events, and / or acceleration events.

[0075] Furthermore, in addition to any feature of the features in this paragraph and / or the preceding seven paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding seven paragraphs concerning the vehicle handover system 810, the expected response time may be adjusted based on the environmental rating of the monitored environment of the vehicle. Furthermore, in addition to any feature of the features in this paragraph and / or the preceding seven paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding seven paragraphs concerning the vehicle handover system 810, the monitored environment may include at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density near the vehicle, vehicle speed, road geometry, road type, and / or air temperature at the vehicle. Furthermore, in addition to or in combination with any of the features of the vehicle handover system 810 in this paragraph and / or the preceding seven paragraphs, the expected response time may be adjusted based on a passenger attention rating, which depends on the characteristics observed by the passenger. Furthermore, in addition to or in combination with any of the features of the vehicle handover system 810 in this paragraph and / or the preceding seven paragraphs, the characteristics observed by the passenger may include at least one of the following: passenger gaze, passenger head movement, passenger age, passenger advertising preferences, passenger body posture, passenger body shape, passenger body size, passenger heart rate, passenger non-driving activities, passenger driving quality, and / or noise level inside the vehicle.

[0076] Furthermore, in addition to or in combination with any of the features of this paragraph and / or the preceding eight paragraphs regarding the vehicle handover system 810, the passenger's non-driving activities may include secondary activities in which the passenger participates. Furthermore, in addition to or in combination with any of the features of this paragraph and / or the preceding eight paragraphs regarding the vehicle handover system 810, the expected response time may be adjusted based on the vehicle's geographic location and map-based information associated with that geographic location. Furthermore, in addition to or in combination with any of the features of this paragraph and / or the preceding eight paragraphs regarding the vehicle handover system 810, the map-based information may include at least one of the following: road curvature, road visibility, density of nearby objects, and / or road quality at the geographic location. In addition, map-based information may include a baseline reaction time offset associated with the average reaction time of a geographic location, in addition to or in combination with any of the features in this paragraph and / or the preceding eight paragraphs concerning the vehicle handover system 810. Furthermore, the average response time may be stored in a passenger-associated average response time database, in addition to or in combination with any of the features in this paragraph and / or the preceding eight paragraphs concerning the vehicle handover system 810. The average response time database may be updated using adjusted expected response times.

[0077] Furthermore, in addition to or in combination with any of the features of this paragraph and / or the preceding nine paragraphs regarding the vehicle transfer system 810, the database of average response times may include average response times associated with other drivers. Furthermore, in addition to or in combination with any of the features of this paragraph and / or the preceding nine paragraphs regarding the vehicle transfer system 810, the processor 820 may be further configured to determine the minimum response time required to fulfill a transfer request to change control of the vehicle to a passenger. Furthermore, in addition to or in combination with any of the features of this paragraph and / or the preceding nine paragraphs regarding the vehicle transfer system 810, the image sequence within a time period of a critical event may be based on the minimum response time. Furthermore, apart from any feature of the features in this paragraph and / or the preceding nine paragraphs concerning the vehicle transfer system 810, or in combination with any feature of the features in this paragraph and / or the preceding nine paragraphs concerning the vehicle transfer system 810, if the expected response time is slower than the minimum response time, the vehicle transfer system 810 may refuse the transfer request. Furthermore, apart from any feature of the features in this paragraph and / or the preceding nine paragraphs concerning the vehicle transfer system 810, or in combination with any feature of the features in this paragraph and / or the preceding nine paragraphs concerning the vehicle transfer system 810, if the expected response time is faster than the minimum response time, the vehicle transfer system 810 may proceed with the requested transfer.

[0078] Furthermore, in addition to any feature of the features in this paragraph and / or the preceding ten paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding ten paragraphs concerning the vehicle handover system 810, the observed characteristics of the passenger may include passenger pattern confusion. This pattern confusion may be determined based on comparing the passenger's expected actions with the passenger's observed actions. Furthermore, in addition to any feature of the features in this paragraph and / or the preceding ten paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding ten paragraphs concerning the vehicle handover system 810, the expected actions may include the expected response to a question posed to the passenger in relation to the simulated driving situation. The observed actions may include the response provided by the driver to that question. Furthermore, in addition to any feature of the features in this paragraph and / or the preceding ten paragraphs concerning the vehicle handover system 810, or in combination with any feature of the features in this paragraph and / or the preceding ten paragraphs concerning the vehicle handover system 810, the expected actions may be based on the current operating state of the vehicle and the expected set of vehicle maneuvers associated with the current operating state. Pattern confusion can be based on whether the observed action is at least one of the expected vehicle maneuvers in the set of vehicle maneuvers. Furthermore, in addition to or in combination with any feature of this paragraph and / or the preceding ten paragraphs concerning the vehicle handover system 810, the processor 820 may be further configured to grant or reject a transfer request based on pattern confusion.

[0079] Figure 9 A schematic flowchart depicts a method 900 for assisting in the handover of vehicles. Method 900 can implement any of the features described above regarding the equipment 800.

[0080] Method 900 is a method for assisting the handover of a vehicle, including, in 910, monitoring the environment of the vehicle. Method 900 further includes, in 920, receiving a transfer request to change control of the vehicle from an automated driving mode to a passenger of the vehicle. Method 900 further includes, in 930, detecting a key event that may be related to the transfer request, wherein the key event is based on the monitored environment. Method 900 further includes, in 940, generating a handover scene including images associated with the key event, wherein the images include a sequence of images within a time period of the key event. Method 900 further includes, in 950, displaying the handover scene to the passenger before changing control of the vehicle from an automated driving mode to the passenger.

[0081] Example 1 is a vehicle handover system comprising a processor configured to monitor the environment of the vehicle. The processor is also configured to receive a transfer request to change control of the vehicle from an automated driving mode to a passenger. The processor is further configured to detect critical events that may be associated with the transfer request, wherein the critical events are based on the monitored environment. The processor is also configured to generate a handover scene including images associated with the critical events, wherein the images comprise a sequence of images within a time period of the critical events. The processor is further configured to display the handover scene to the passenger before the change of control of the vehicle from an automated driving mode to the passenger.

[0082] Example 2 is a vehicle handover system of Example 1, wherein key events include at least one of the following: movement events that have caused or may cause a change in the trajectory of the vehicle, unexpected behavior events of the vehicle observed in the monitored environment of the vehicle, and / or external operating domain events that include the vehicle's operating setpoints that do not conform to the expected operating conditions of the vehicle.

[0083] Example 3 is a vehicle handover system of either Example 1 or 2, wherein the images further include event metadata associated with key events, wherein the event metadata includes at least one of the following: the type of key event, the time of the key event, the location of the key event, the duration of the key event, the importance of each image in the images associated with the key event, and / or the image view of each image in the images associated with the key event.

[0084] Example 4 is a vehicle handover system of any of Examples 1 to 3, wherein the image is generated based on sensor data related to the vehicle.

[0085] Example 5 is a vehicle handover system of either Example 3 or 4, wherein the images include an enhanced image sequence generated from at least one camera image, which is enhanced using event metadata.

[0086] Example 6 is a vehicle handover system of any of Examples 1 to 5, wherein the handover scenario includes multiple perspectives corresponding to the passenger's viewpoint.

[0087] Example 7 is a vehicle handover system of Example 6, in which multiple perspectives include a forward-facing camera perspective of the vehicle, a bird's-eye view, and / or a map-based perspective.

[0088] Example 8 is a vehicle handover system of any of Examples 3 to 7, wherein the handover scenario comprises static images formed by images and / or event metadata associated with multiple key events.

[0089] Example 9 is a vehicle handover system of Example 8, wherein the static image includes at least one of the following: a single frame of an image associated with multiple key events and / or an overlaid image based on one or more images associated with multiple key events.

[0090] Example 10 is a vehicle handover system of any of Examples 1 to 9, wherein the handover scenario further includes text associated with an image, wherein the text is generated from the image using a multimodal generalization system.

[0091] Example 11 is the vehicle handover system of Example 10, and the text includes descriptions of key events and the times when the key events occurred.

[0092] Example 12 is a vehicle handover system of either Example 10 or 11, wherein the text includes a description of key events verbally communicated to passengers via the vehicle's audio system.

[0093] Example 13 is a vehicle handover system of either Example 11 or 12, where time is the elapsed time since the critical event occurred.

[0094] Example 14 is a vehicle handover system of any of Examples 1 to 13, in which critical events are assigned priorities corresponding to the importance of the critical events relative to the current state of the vehicle.

[0095] Example 15 is a vehicle handover system of Example 14, wherein the priority is based on at least one of the following: the time elapsed since the critical event occurred, the category of the critical event, the distance from the location of the critical event to the current location of the vehicle, and / or the likelihood that the passenger is aware of the critical event.

[0096] Example 16 is a vehicle handover system of Example 15, wherein the current location of the vehicle includes the expected location of the vehicle after the elapsed time.

[0097] Example 17 is a vehicle handover system of any of Examples 1 to 16, wherein the processor is further configured to monitor passengers to determine the likelihood that the passengers are aware of a critical event.

[0098] Example 18 is a vehicle handover system of any of Examples 1 to 17, wherein the processor is further configured to determine the presentation duration of the handover scene to be presented to the passenger, wherein the presentation duration depends on the urgency of the handover request.

[0099] Example 19 is a vehicle handover system of Example 18, wherein the processor is further configured to compress the length of the handover scene to fit within the presentation duration.

[0100] Example 20 is a vehicle handover system of Example 19, in which the compressed length is removed from the handover scene based on the importance of a portion of the image data associated with the image to the key event.

[0101] Example 21 is a vehicle handover system of Example 19, wherein the handover scenario includes multiple key events, and the compressed length is configured to remove at least one key event from the handover scenario based on the priority of at least one of the multiple key events.

[0102] Example 22 is a vehicle handover system of any of Examples 1 to 21, wherein the processor is further configured to store the handover scene as a stored handover scene, and wherein the processor is further configured to display the stored handover scene to passengers based on a request to review the stored handover scene.

[0103] Example 23 is a vehicle handover system of any of Examples 1 to 22, wherein the generation of the handover scenario depends on the passenger’s expected response time.

[0104] Example 24 is a vehicle handover system of Example 23, wherein the expected response time is based on the average response time of a passenger to multiple monitored events.

[0105] Example 25 is a vehicle handover system of Example 24, wherein multiple monitored events include at least one of the following: traffic sign event, right-of-way event, stopping event, turning event, deceleration event, and / or acceleration event.

[0106] Example 26 is a vehicle handover system of any of Examples 23 to 25, wherein the expected response time is adjusted based on the environmental rating of the monitored environment of the vehicle.

[0107] Example 27 is a vehicle handover system of any one of Examples 1 to 26, wherein the monitored environment includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, vehicle speed, road geometry, road type and / or air temperature at the vehicle.

[0108] Example 28 is a vehicle handover system of any of Examples 23 to 27, wherein the expected response time is adjusted based on a passenger’s attention rating, which depends on the passenger’s observed characteristics.

[0109] Example 29 is a vehicle handover system of Example 28, wherein the observed characteristics of a passenger may include at least one of the following: passenger gaze, passenger head movement, passenger age, passenger advertising preferences, passenger body posture, passenger body shape, passenger body size, passenger heart rate, passenger non-driving activities, passenger driving quality, and / or noise level inside the vehicle.

[0110] Example 30 is a vehicle handover system of Example 29, in which the non-driving activities of passengers include secondary activities in which passengers participate.

[0111] Example 31 is a vehicle handover system of any of Examples 23 to 30, wherein the expected response time is adjusted based on the geographic location of the vehicle and map-based information associated with that geographic location.

[0112] Example 32 is a vehicle handover system of Example 31, wherein map-based information includes at least one of the following: road curvature, road visibility, density of nearby objects, and / or road quality at a geographic location.

[0113] Example 33 is a vehicle handover system of either Example 31 or 32, wherein map-based information includes a baseline reaction time offset associated with the average reaction time of the geographic location.

[0114] Example 34 is a vehicle handover system of any of Examples 26 to 33, wherein the average response time is stored in a database of average response times associated with passengers, and the database of average response times is updated using adjusted expected response times.

[0115] Example 35 is a vehicle handover system of Example 34, wherein the database of average response times further includes average response times associated with other drivers.

[0116] Example 36 is a vehicle handover system of any of Examples 1 to 35, wherein the processor is further configured to determine the minimum response time required to fulfill a transfer request to change control of the vehicle to a passenger.

[0117] Example 37 is a vehicle handover system of Example 36, in which the image sequence within a time period of a critical event is based on the minimum response time.

[0118] Example 38 is a vehicle handover system of any of Examples 23 to 37, wherein a request for a transfer is rejected if the expected response time is slower than the minimum response time.

[0119] Example 39 is a vehicle handover system of any of Examples 23 to 38, wherein the requested transfer proceeds if the expected response time is faster than the minimum response time.

[0120] Example 40 is a vehicle handover system of any of Examples 1 to 39, wherein the observed characteristics of passengers include passenger pattern confusion, wherein pattern confusion is determined based on comparing the passenger’s expected action with the passenger’s observed action.

[0121] Example 41 is a vehicle handover system of Example 40, wherein the anticipated actions include the anticipated responses to questions posed to passengers in relation to a simulated driving situation, and wherein the observed actions include the responses provided by the driver to the questions.

[0122] Example 42 is a vehicle handover system of any one of Examples 40 or 41, wherein the expected action is based on the current operating state of the vehicle and a set of expected vehicle maneuvers associated with the current operating state, and wherein pattern confusion is based on whether the observed action is at least one vehicle maneuver in the set of expected vehicle maneuvers.

[0123] Example 43 is a vehicle handover system of either Example 41 or 42, wherein the processor is further configured to grant or reject a transfer request based on pattern confusion.

[0124] Example 44 is a system for assessing the attention level of passengers in a vehicle. The system includes a processor configured to determine the average response time of passengers within the vehicle based on passenger observations of multiple monitored events. The processor is further configured to determine an environmental rating of the vehicle based on a monitored environment outside the vehicle. The processor is further configured to determine an in-vehicle rating of the vehicle based on observed characteristics of passengers within the vehicle. The processor is further configured to determine the required response time for a geographical location of the vehicle based on received map information regarding response times associated with geographical locations. The processor is further configured to determine the expected response time of passengers at a geographical location based on the average response time, environmental rating, and in-vehicle rating. The system further includes a transmitter configured to provide the expected response time and the required response time to an automated driving system of the vehicle.

[0125] Example 45 is a system similar to Example 44, wherein multiple monitored events include at least one of the following: traffic sign event, right-of-way event, stopping event, turning event, deceleration event, and / or acceleration event.

[0126] Example 46 is a system of any one of Examples 44 or 45, wherein the monitored environment outside the vehicle includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, vehicle speed, road geometry, road type and / or air temperature at the vehicle.

[0127] Example 47 is a system of any one of Examples 44 to 46, wherein the observed characteristics of the passenger may include at least one of the following: the passenger's gaze, the passenger's head movement, the passenger's age, the passenger's advertising preferences, the passenger's body posture, the passenger's body shape, the passenger's body size, the passenger's heart rate, the passenger's non-driving activities, the passenger's driving quality, and / or the noise level inside the vehicle.

[0128] Example 48 is a system of any of Examples 44 to 47, wherein the expected response time is adjusted based on received map-based information, wherein the received map-based information further includes at least one of the following: road curvature, road visibility, density of nearby objects and / or road quality at the geographic location.

[0129] Example 49 is a system of any of Examples 44 to 48, wherein the map-based information further includes a baseline reaction time offset based on the average response time associated with other drivers at that geographic location.

[0130] Example 50 is a system of any of Examples 44 to 49, wherein the automated driving system of a vehicle includes a vehicle handover system that determines whether to transfer control of the vehicle from the automated driving mode to a passenger based on the expected response time and the required response time.

[0131] Example 51 is a device for transferring control of a vehicle to a passenger. The device includes a processor configured to monitor the environment of the vehicle. The processor is also configured to receive a transfer request from a passenger to transfer control of the vehicle from an automated driving mode. The processor is further configured to detect key events that may be associated with the transfer request, wherein the key events are based on the monitored environment. The processor is also configured to generate a handover scene including images associated with the key events, wherein the images comprise a sequence of images within a time period of the key events. The processor is further configured to display the handover scene to the passenger before transferring control of the vehicle from an automated driving mode to the passenger.

[0132] Example 52 is the device of Example 51, wherein the critical events include at least one of the following: a movement event that has caused or caused a change in the trajectory of the vehicle, an unexpected behavior event of the vehicle observed in the monitored environment of the vehicle, and / or an external operating domain event that includes the vehicle's operating setpoint that does not conform to the expected operating conditions of the vehicle.

[0133] Example 53 is a device of either Example 51 or 52, wherein the image further includes event metadata associated with a key event, wherein the event metadata includes at least one of the following: the type of key event, the time of the key event, the location of the key event, the duration of the key event, the importance of each image in the images associated with the key event, and / or the image view of each image in the images associated with the key event.

[0134] Example 54 is a device of any of Examples 51 to 53, wherein the image is generated based on sensor data related to the vehicle.

[0135] Example 55 is a device of either Example 53 or 54, wherein the image includes an enhanced image sequence generated from at least one camera image, which is enhanced using event metadata.

[0136] Example 56 is a device of any of Examples 51 to 55, wherein the handover scene includes multiple perspectives corresponding to the passenger's viewpoint.

[0137] Example 57 is the device of Example 56, where multiple perspectives include a forward-facing camera perspective of a vehicle, a bird's-eye view, and / or a map-based perspective.

[0138] Example 58 is a device of any of Examples 53 to 57, and the handover scenario includes a static image formed by images and / or event metadata associated with multiple key events.

[0139] Example 59 is the device of Example 58, wherein the still image includes at least one of the following: a single frame of an image associated with a plurality of key events and / or an overlaid image based on one or more images associated with a plurality of key events.

[0140] Example 60 is a device of any of Examples 51 to 59, wherein the handover scene further includes text associated with an image, wherein the text is generated from the image using a multimodal generalization system.

[0141] Example 61 is the device of Example 60, and the text includes a description of the key events and the time when the key events occurred.

[0142] Example 62 is a device of either Example 60 or 61, wherein the text includes a description of key events expressed verbally to passengers via the audio system of a vehicle.

[0143] Example 63 is a device of either Example 61 or 62, where time is the elapsed time since the critical event occurred.

[0144] Example 64 is a device of any of Examples 51 to 63, in which critical events are assigned priorities corresponding to the importance of the critical events relative to the current situation of the vehicle.

[0145] Example 65 is the device of Example 64, wherein the priority is based on at least one of the following: the time elapsed since the critical event occurred, the category of the critical event, the distance from the location of the critical event to the current location of the vehicle, and / or the likelihood that the passenger is aware of the critical event.

[0146] Example 66 is the device of Example 65, wherein the current location of the vehicle includes the expected location of the vehicle after the elapsed time.

[0147] Example 67 is a device of any of Examples 51 to 66, wherein the processor is further configured to monitor passengers to determine the likelihood that the passengers are aware of the critical event.

[0148] Example 68 is a device of any of Examples 51 to 67, wherein the processor is further configured to determine the presentation duration of the handover scene to be presented to the passenger, wherein the presentation duration depends on the urgency of the transition request.

[0149] Example 69 is a device of Example 68, wherein the processor is further configured to compress the length of the handover scene to fit within the rendering duration.

[0150] Example 70 is a device similar to Example 69, where the compression length is removed from the handover scene based on the importance of a portion of the image data associated with the image to the key event.

[0151] Example 71 is a device of Example 69, wherein the handover scenario includes multiple key events, and the compressed length is configured to remove at least one key event from the handover scenario based on the priority of at least one of the multiple key events.

[0152] Example 72 is a device of any of Examples 51 to 71, wherein the processor is further configured to store the handover scene as a stored handover scene, wherein the processor is further configured to display the stored handover scene to the passenger based on a request to review the stored handover scene.

[0153] Example 73 is a device of any of Examples 51 to 72, wherein the generation of the handover scenario depends on the passenger’s expected response time.

[0154] Example 74 is the device of Example 73, where the expected response time is based on the average response time of the passenger to multiple monitored events.

[0155] Example 75 is the device of Example 74, wherein the plurality of monitored events include at least one of the following: traffic sign event, right-of-way event, parking event, turning event, deceleration event, and / or acceleration event.

[0156] Example 76 is a device of any of Examples 73 to 75, wherein the expected response time is adjusted based on the environmental rating of the monitored environment of the vehicle.

[0157] Example 77 is a device of any one of Examples 51 to 76, wherein the monitored environment includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, speed of the vehicle, road geometry, road type and / or air temperature at the vehicle.

[0158] Example 78 is a device of any of Examples 73 to 77, wherein the expected response time is adjusted based on a passenger’s attention rating, which depends on the characteristics observed by the passenger.

[0159] Example 79 is the device of Example 78, wherein the characteristics observed by the passenger may include at least one of the following: the passenger's gaze, the passenger's head movement, the passenger's age, the passenger's advertising preferences, the passenger's body posture, the passenger's body shape, the passenger's body size, the passenger's heart rate, the passenger's non-driving activities, the passenger's driving quality, and / or the noise level inside the vehicle.

[0160] Example 80 is the device of Example 79, wherein the passenger’s non-driving activities include secondary activities in which the passenger participates.

[0161] Example 81 is a device of any of Examples 73 to 80, wherein the expected response time is adjusted based on the geographic location of the vehicle and map-based information associated with that geographic location.

[0162] Example 82 is the device of Example 81, wherein the map-based information includes at least one of the following: road curvature, road visibility, density of nearby objects, and / or road quality at a geographic location.

[0163] Example 83 is a device of either Example 81 or 82, wherein the map-based information includes a baseline reaction time offset associated with the average reaction time of the geographic location.

[0164] Example 84 is a device of any of Examples 76 to 83, wherein the average response time is stored in a database of average response times associated with passengers, and wherein the database of average response times is updated using adjusted expected response times.

[0165] Example 85 is the device of Example 84, wherein the database of average response time further includes average response time associated with other drivers.

[0166] Example 86 is a device of any of Examples 51 to 85, wherein the processor is further configured to determine the minimum response time required to implement a transfer request to change control of the vehicle to a passenger.

[0167] Example 87 is the device of Example 86, where the image sequence within a time period of a critical event is based on the minimum response time.

[0168] Example 88 is a device of any of Examples 73 through 87, where a request for a transition is rejected if the expected response time is slower than the minimum response time.

[0169] Example 89 is a device of any of Examples 73 through 88, in which the requested transformation proceeds if the expected response time is faster than the minimum response time.

[0170] Example 90 is a device of any of Examples 51 to 89, wherein the observed characteristics of the passenger include passenger pattern confusion, wherein the pattern confusion is determined based on comparing the passenger’s expected action with the passenger’s observed action.

[0171] Example 91 is the device of Example 90, wherein the expected actions include the expected response to a question posed to a passenger in relation to a simulated driving situation, and wherein the observed actions include the response provided by the driver to the question.

[0172] Example 92 is a device of any one of Examples 90 or 91, wherein the expected action is based on the current operating state of the vehicle and a set of expected vehicle maneuvers associated with the current operating state, and wherein the pattern confusion is based on whether the observed action is at least one vehicle maneuver in the set of expected vehicle maneuvers.

[0173] Example 93 is a device of either Example 91 or 92, wherein the processor is further configured to grant or deny a conversion request based on pattern obfuscation.

[0174] Example 94 is a device for assessing the attention level of passengers in a vehicle. The device includes a processor configured to determine the average response time of passengers within the vehicle based on passenger observations of multiple monitored events. The processor is further configured to determine an environmental rating of the vehicle based on a monitored environment outside the vehicle. The processor is further configured to determine an in-vehicle rating of the vehicle based on observed characteristics of passengers within the vehicle. The processor is further configured to determine the required response time for a geographical location of the vehicle based on received map information regarding response times associated with geographical locations. The processor is further configured to determine the expected response time of passengers at a geographical location based on the average response time, environmental rating, and in-vehicle rating. The system further includes a transmitter configured to provide the expected response time and the required response time to an automated driving system of the vehicle.

[0175] Example 95 is the device of Example 94, wherein the plurality of monitored events include at least one of the following: traffic sign event, right-of-way event, parking event, turning event, deceleration event, and / or acceleration event.

[0176] Example 96 is a device of any one of Examples 94 or 95, wherein the monitored environment outside the vehicle includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, speed of the vehicle, road geometry, road type and / or air temperature at the vehicle.

[0177] Example 97 is a device of any one of Examples 94 to 96, wherein the characteristics observed by the passenger may include at least one of the following: the passenger's gaze, the passenger's head movement, the passenger's age, the passenger's advertising preferences, the passenger's body posture, the passenger's body shape, the passenger's body size, the passenger's heart rate, the passenger's non-driving activities, the passenger's driving quality, and / or the noise level inside the vehicle.

[0178] Example 98 is a device of any of Examples 94 to 97, wherein the expected response time is adjusted based on received map-based information, wherein the received map-based information further includes at least one of the following: road curvature, road visibility, density of nearby objects and / or road quality at the geographic location.

[0179] Example 99 is a device of any of Examples 94 to 98, wherein the map-based information further includes a baseline reaction time offset based on the average response time associated with other drivers at that geographic location.

[0180] Example 100 is a device of any of Examples 94 to 99, wherein the device further includes a vehicle handover system that determines whether to transfer control of the vehicle from an automated driving mode to a passenger based on the expected response time and the required response time.

[0181] Example 101 is a method for transferring control of a vehicle to a passenger, the method including monitoring the environment of the vehicle. The method also includes receiving a transfer request from a passenger to change control of the vehicle from an automated driving mode to the passenger. The method further includes detecting key events that may be associated with the transfer request, wherein the key events are based on the monitored environment. The method also includes generating a handover scene including images associated with the key events, wherein the images comprise a sequence of images within a time period of the key events. The method further includes displaying the handover scene to the passenger before the transfer of control of the vehicle from the automated driving mode to the passenger.

[0182] Example 102 is the method of Example 101, wherein the key events include at least one of the following: a movement event that has caused or caused a change in the trajectory of the vehicle, an unexpected behavior event of the vehicle observed in the monitored environment of the vehicle, and / or an external operating domain event that includes the vehicle's operating setpoint that does not conform to the expected operating conditions of the vehicle.

[0183] Example 103 is a method of either Example 101 or 102, wherein the image further includes event metadata associated with a key event, wherein the event metadata includes at least one of the following: the type of key event, the time of the key event, the location of the key event, the duration of the key event, the importance of each image in the images associated with the key event, and / or the image view of each image in the images associated with the key event.

[0184] Example 104 is a method of any of Examples 101 to 103, wherein the image is generated based on sensor data related to the vehicle.

[0185] Example 105 is a method of either Example 103 or 104, wherein the image includes an enhanced image sequence generated from at least one camera image, which is enhanced using event metadata.

[0186] Example 106 is a method of any of Examples 101 to 105, wherein the handover scenario includes multiple perspectives corresponding to the passenger's viewpoint.

[0187] Example 107 is the method of Example 106, where multiple perspectives include a forward-facing camera perspective of a vehicle, a bird's-eye view, and / or a map-based perspective.

[0188] Example 108 is a method of any of Examples 103 to 107, where the handover scenario includes a static image formed from images and / or event metadata associated with multiple key events.

[0189] Example 109 is the method of Example 108, wherein the static image includes at least one of the following: a single frame of an image associated with multiple key events and / or an overlaid image based on one or more images associated with multiple key events.

[0190] Example 110 is a method of any of Examples 101 to 109, wherein the handover scenario further includes text associated with an image, wherein the text is generated from the image using a multimodal generalization system.

[0191] Example 111 is the method of Example 110, and the text includes a description of the key event and the time when the key event occurred.

[0192] Example 112 is a method of either Example 110 or 111, wherein the text includes a description of key events expressed verbally to passengers via the audio system of a vehicle.

[0193] Example 113 is a method of either Example 111 or 112, where time is the elapsed time since the critical event occurred.

[0194] Example 114 is a method of any of Examples 101 to 113, in which critical events are assigned priorities corresponding to the importance of the critical events relative to the current state of the vehicle.

[0195] Example 115 is the method of Example 114, wherein the priority is based on at least one of the following: the time elapsed since the critical event occurred, the category of the critical event, the distance from the location of the critical event to the current location of the vehicle, and / or the likelihood that the passenger is aware of the critical event.

[0196] Example 116 is a method of Example 115, wherein the current location of the vehicle includes the expected location of the vehicle after the elapsed time.

[0197] Example 117 is a method of any of Examples 101 to 116, wherein the method further includes monitoring passengers to determine the likelihood that the passengers are aware of the critical event.

[0198] Example 118 is a method of any of Examples 101 to 117, wherein the method further includes determining the presentation duration of presenting the handover scene to the passenger, wherein the presentation duration depends on the urgency of the transition request.

[0199] Example 119 is a method of Example 118, wherein the method further includes compressing the length of the handover scene to fit within the rendering duration.

[0200] Example 120 is a method of Example 119, in which the compression length is removed from the handover scene based on the importance of a portion of the image data associated with the image to the key event.

[0201] Example 121 is a method of Example 119, wherein the handover scenario includes multiple key events, and the compressed length is configured to remove at least one key event from the handover scenario based on the priority of at least one of the multiple key events.

[0202] Example 122 is a method of any one of Examples 101 to 121, wherein the method further includes storing the handover scene as a stored handover scene and displaying the stored handover scene to a passenger based on a request to review the stored handover scene.

[0203] Example 123 is a method of any of Examples 101 to 122, where the generation of the handover scenario depends on the passenger's expected response time.

[0204] Example 124 is the method of Example 123, where the expected response time is based on the average response time of the passenger to multiple monitored events.

[0205] Example 125 is the method of Example 124, wherein the multiple monitored events include at least one of the following: traffic sign event, right-of-way event, parking event, turning event, deceleration event, and / or acceleration event.

[0206] Example 126 is a method of any of Examples 123 to 125, wherein the expected response time is adjusted based on the environmental rating of the monitored environment of the vehicle.

[0207] Example 127 is a method of any one of Examples 101 to 126, wherein the monitored environment includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, speed of the vehicle, road geometry, road type and / or air temperature at the vehicle.

[0208] Example 128 is a method of any of Examples 123 to 127, wherein the expected response time is adjusted based on the passenger’s attention rating, which depends on the passenger’s observed characteristics.

[0209] Example 129 is a method of Example 128, wherein the observed characteristics of the passenger may include at least one of the following: the passenger's gaze, the passenger's head movement, the passenger's age, the passenger's advertising preferences, the passenger's body posture, the passenger's body shape, the passenger's body size, the passenger's heart rate, the passenger's non-driving activities, the passenger's driving quality, and / or the noise level inside the vehicle.

[0210] Example 130 is the method of Example 129, wherein the passenger’s non-driving activities include secondary activities in which the passenger participates.

[0211] Example 131 is a method of any of Examples 123 to 130, wherein the expected response time is adjusted based on the geographic location of the vehicle and map-based information associated with that geographic location.

[0212] Example 132 is a method of Example 131, wherein the map-based information includes at least one of the following: road curvature, road visibility, density of nearby objects, and / or road quality at a geographic location.

[0213] Example 133 is a method of either Example 131 or 132, wherein the map-based information includes a baseline reaction time offset associated with the average reaction time of the geographic location.

[0214] Example 134 is a method of any of Examples 126 to 133, wherein the average response time is stored in a database of average response times associated with passengers, and wherein the database of average response times is updated using adjusted expected response times.

[0215] Example 135 is the method of Example 134, wherein the database of average response times further includes average response times associated with other drivers.

[0216] Example 136 is a method of any of Examples 101 to 135, wherein the method further includes determining the minimum response time required to fulfill a transfer request to change control of the vehicle to a passenger.

[0217] Example 137 is the method of Example 136, where the image sequence within a time period of the key event is based on the minimum response time.

[0218] Example 138 is a method of any of Examples 123 through 137, in which a request for a transition is rejected if the expected response time is slower than the minimum response time.

[0219] Example 139 is a method of any of Examples 123 through 138, in which the requested transformation proceeds if the expected response time is faster than the minimum response time.

[0220] Example 140 is a method of any of Examples 101 to 139, wherein the observed characteristics of the passenger include passenger pattern confusion, wherein pattern confusion is determined based on comparing the passenger’s expected action with the passenger’s observed action.

[0221] Example 141 is a method of Example 140, wherein the expected action includes an expected response to a question posed to a passenger in relation to a simulated driving situation, and wherein the observed action includes a response from the driver to the question.

[0222] Example 142 is a method of either Example 140 or 141, wherein the expected action is based on the current operating state of the vehicle and a set of expected vehicle maneuvers associated with the current operating state, and wherein pattern confusion is based on whether the observed action is at least one vehicle maneuver in the set of expected vehicle maneuvers.

[0223] Example 143 is a method of either Example 141 or 142, wherein the method further includes granting or denying a conversion request based on pattern obfuscation.

[0224] Example 144 is a method for assessing the attention level of passengers in a vehicle. The method includes determining the average response time of passengers within the vehicle based on passenger observations of multiple monitored events. The method further includes determining an environmental rating of the vehicle based on a monitored environment outside the vehicle. The method further includes determining an in-vehicle rating of the vehicle based on observed characteristics of passengers within the vehicle. The method further includes determining the required response time for a geographical location of the vehicle based on received map information regarding response times associated with geographical locations. The method further includes determining the expected response time of passengers at a geographical location based on the average response time, environmental rating, and in-vehicle rating. The method further includes providing the expected response time and the required response time to an automated driving system of the vehicle.

[0225] Example 145 is a method of Example 144, wherein the plurality of monitored events include at least one of the following: traffic sign event, right-of-way event, stopping event, turning event, deceleration event, and / or acceleration event.

[0226] Example 146 is a method of any one of Examples 144 or 145, wherein the monitored environment outside the vehicle includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, vehicle speed, road geometry, road type and / or air temperature at the vehicle.

[0227] Example 147 is a method of any one of Examples 144 to 146, wherein the observed characteristics of the passenger may include at least one of the following: the passenger's gaze, the passenger's head movement, the passenger's age, the passenger's advertising preferences, the passenger's body posture, the passenger's body shape, the passenger's body size, the passenger's heart rate, the passenger's non-driving activities, the passenger's driving quality, and / or the noise level inside the vehicle.

[0228] Example 148 is a method of any of Examples 144 to 147, wherein the expected response time is adjusted based on received map-based information, wherein the received map-based information further includes at least one of the following: road curvature, road visibility, density of nearby objects and / or road quality at the geographic location.

[0229] Example 149 is a method of any of Examples 144 to 148, wherein the map-based information further includes a baseline reaction time offset based on the average response time associated with other drivers at that geographic location.

[0230] Example 150 is a method of any of Examples 144 to 149, wherein the method further includes determining whether to transfer control of the vehicle from the automated driving mode to the passenger based on the expected response time and the required response time.

[0231] Example 151 is one or more non-transitory computer-readable media configured to, when executed, cause one or more processors to perform a method for transferring control of a vehicle to a passenger. The method, stored in the non-transitory computer-readable medium, includes monitoring the environment of the vehicle. The method also includes receiving a transfer request from a passenger to change control of the vehicle from an automated driving mode. The method further includes detecting a critical event that may be associated with the transfer request, wherein the critical event is based on the monitored environment. The method also includes generating a handover scene including images associated with the critical event, wherein the images comprise a sequence of images within a time period of the critical event. The method further includes displaying the handover scene to the passenger before the control of the vehicle is transferred from the automated driving mode to the passenger.

[0232] Example 152 is a nontransitory computer-readable medium of Example 151, wherein the critical events include at least one of the following: a movement event that has caused or caused a change in the trajectory of the vehicle, an unexpected behavior event of the vehicle observed in the monitored environment of the vehicle, and / or an external operating domain event that includes the vehicle's operating setpoint that does not conform to the vehicle's expected operating conditions.

[0233] Example 153 is a non-transitory computer-readable medium of any of Examples 151 or 152, wherein the image further includes event metadata associated with a critical event, wherein the event metadata includes at least one of the following: the type of critical event, the time of the critical event, the location of the critical event, the duration of the critical event, the importance of each image in the images associated with the critical event, and / or the image view of each image in the images associated with the critical event.

[0234] Example 154 is a non-transitory computer-readable medium of any of Examples 151 to 153, wherein an image is generated based on sensor data relating to a vehicle.

[0235] Example 155 is a non-transitory computer-readable medium of any of Examples 153 or 154, wherein the image includes an enhanced image sequence generated from at least one camera image, which is enhanced using event metadata.

[0236] Example 156 is a non-transitory computer-readable medium of any of Examples 151 to 155, wherein the handover scene includes multiple perspectives corresponding to the passenger's viewpoint.

[0237] Example 157 is a non-transitory computer-readable medium of Example 156, wherein multiple perspectives include a forward-facing camera perspective of a vehicle, a bird's-eye view, and / or a map-based perspective.

[0238] Example 158 is a non-transitory computer-readable medium of any of Examples 153 to 157, the handover scenario comprising a static image formed of images and / or event metadata associated with multiple key events.

[0239] Example 159 is a non-transitory computer-readable medium of Example 158, wherein the still image includes at least one of the following: a single frame of an image associated with a plurality of key events and / or an overlaid image based on one or more images associated with a plurality of key events.

[0240] Example 160 is a non-transitory computer-readable medium of any of Examples 151 to 159, wherein the handover scenario further includes text associated with an image, wherein the text is generated from the image using a multimodal generalization system.

[0241] Example 161 is a non-transitory computer-readable medium of Example 160, and the text includes a description of the key events and the time when the key events occurred.

[0242] Example 162 is a non-transitory computer-readable medium of any of Examples 160 or 161, wherein the text includes a description of key events expressed verbally to passengers via an audio system of a vehicle.

[0243] Example 163 is a non-transitory computer-readable medium of either Example 161 or 162, wherein time is the elapsed time since the critical event occurred.

[0244] Example 164 is a non-transitory computer-readable medium of any of Examples 151 to 163, in which critical events are assigned priorities corresponding to the importance of the critical events relative to the current state of the vehicle.

[0245] Example 165 is a non-transitory computer-readable medium of Example 164, wherein the priority is based on at least one of the following: the time elapsed since the critical event occurred, the category of the critical event, the distance from the location of the critical event to the current location of the vehicle, and / or the likelihood that the passenger is aware of the critical event.

[0246] Example 166 is a non-transitory computer-readable medium of Example 165, wherein the current position of a vehicle includes the expected position of the vehicle after elapsed time.

[0247] Example 167 is a non-transitory computer-readable medium of any of Examples 151 to 166, wherein the method stored in the non-transitory computer-readable medium further includes monitoring passengers to determine the likelihood that the passengers are aware of the critical event.

[0248] Example 168 is a non-transitory computer-readable medium of any of Examples 151 to 167, wherein the method of storing in a non-transitory computer-readable medium further includes determining the presentation duration of presenting the handover scene to the passenger, wherein the presentation duration depends on the urgency of the transfer request.

[0249] Example 169 is a non-transitory computer-readable medium of Example 168, wherein the method of storing in a non-transitory computer-readable medium further includes compressing the length of the handover scene to fit within the presentation duration.

[0250] Example 170 is a non-transitory computer-readable medium of Example 169, wherein the compressed length is removed from the handover scene based on the importance of a portion of the image data associated with the image to a critical event.

[0251] Example 171 is a non-transitory computer-readable medium of Example 169, wherein a handover scenario includes a plurality of key events, wherein a compressed length is configured to remove at least one key event from the handover scenario based on the priority of at least one of the plurality of key events.

[0252] Example 172 is a non-transitory computer-readable medium of any of Examples 151 to 171, wherein the method of storing in the non-transitory computer-readable medium further includes storing the handover scene as a stored handover scene and displaying the stored handover scene to a passenger based on a request to review the stored handover scene.

[0253] Example 173 is a non-transitory computer-readable medium of any of Examples 151 to 172, wherein the generation of the handover scenario depends on the passenger’s expected response time.

[0254] Example 174 is a non-transitory computer-readable medium of Example 173, wherein the expected response time is based on the average response time of a passenger to multiple monitored events.

[0255] Example 175 is a non-transitory computer-readable medium of Example 174, wherein a plurality of monitored events include at least one of the following: traffic sign event, right-of-way event, stopping event, turning event, deceleration event, and / or acceleration event.

[0256] Example 176 is a nontransitory computer-readable medium of any of Examples 173 to 175, wherein the expected response time is adjusted based on the environmental rating of the monitored environment of the vehicle.

[0257] Example 177 is a non-transitory computer-readable medium of any of Examples 151 to 176, wherein the monitored environment includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, speed of the vehicle, road geometry, road type and / or air temperature at the vehicle.

[0258] Example 178 is a non-transitory computer-readable medium of any of Examples 173 to 177, wherein the expected response time is adjusted based on a passenger’s attention rating, which depends on the characteristics observed by the passenger.

[0259] Example 179 is a non-transitory computer-readable medium of Example 178, wherein the observed characteristics of a passenger may include at least one of the following: passenger gaze, passenger head movement, passenger age, passenger advertising preferences, passenger body posture, passenger body shape, passenger body size, passenger heart rate, passenger non-driving activities, passenger driving quality, and / or noise level inside the vehicle.

[0260] Example 180 is a non-transitory computer-readable medium of Example 179, wherein the passenger’s non-driving activities include secondary activities in which the passenger participates.

[0261] Example 181 is a non-transitory computer-readable medium of any of Examples 173 to 180, wherein the expected response time is adjusted based on the geographic location of the vehicle and map-based information associated with that geographic location.

[0262] Example 182 is a non-transitory computer-readable medium of Example 181, wherein map-based information includes at least one of the following: road curvature, road visibility, density of nearby objects, and / or road quality at a geographic location.

[0263] Example 183 is a non-transitory computer-readable medium of any of Examples 181 or 182, wherein the map-based information includes a baseline reaction time offset associated with the average reaction time of a geographic location.

[0264] Example 184 is a non-transitory computer-readable medium of any of Examples 176 to 183, wherein the average response time is stored in a database of average response times associated with passengers, wherein the database of average response times is updated using adjusted expected response times.

[0265] Example 185 is a non-transitory computer-readable medium of Example 184, wherein the database of average response times further includes average response times associated with other drivers.

[0266] Example 186 is a nontransitory computer-readable medium of any of Examples 151 to 185, wherein the method stored in the nontransitory computer-readable medium further includes determining the minimum response time required to implement a transfer request to change control of the vehicle to a passenger.

[0267] Example 187 is a non-transitory computer-readable medium of Example 186, wherein the sequence of images over a period of time of a critical event is based on a minimum response time.

[0268] Example 188 is a nontransitory computer-readable medium of any of Examples 173 to 187, wherein a request for a transition is rejected if the expected response time is slower than the minimum response time.

[0269] Example 189 is a nontransitory computer-readable medium of any of Examples 173 to 188, wherein the requested transformation proceeds if the expected response time is faster than the minimum response time.

[0270] Example 190 is a non-transitory computer-readable medium of any of Examples 151 to 189, wherein the observed characteristics of a passenger include passenger pattern confusion, wherein pattern confusion is determined based on comparing the passenger’s expected action with the passenger’s observed action.

[0271] Example 191 is a nontransitory computer-readable medium of Example 190, wherein the expected action includes an expected response to a question posed to a passenger in relation to a simulated driving situation, and wherein the observed action includes a response from the driver to the question.

[0272] Example 192 is a non-transitory computer-readable medium of any of Examples 190 or 191, wherein the expected action is based on the current operating state of the vehicle and a set of expected vehicle maneuvers associated with the current operating state, and wherein the pattern confusion is based on whether the observed action is at least one vehicle maneuver in the set of expected vehicle maneuvers.

[0273] Example 193 is a nontransitory computer-readable medium of any of Examples 191 or 192, wherein the method stored in the nontransitory computer-readable medium further includes granting or rejecting a conversion request based on pattern obfuscation.

[0274] Example 194 is one or more non-transitory computer-readable media configured to, when executed, cause one or more processors to perform a method for transferring control of a vehicle to a passenger. The method, stored in the non-transitory computer-readable medium, includes determining an average response time of a passenger within the vehicle based on passenger observations of multiple monitored events. The method further includes determining an environmental rating of the vehicle based on monitored environments outside the vehicle. The method further includes determining an in-vehicle rating of the vehicle based on observed characteristics of passengers within the vehicle. The method further includes determining a required response time for a geographical location of the vehicle based on received map information regarding response times associated with geographical locations. The method further includes determining an expected response time for a passenger at a geographical location based on the average response time, environmental rating, and in-vehicle rating. The method further includes providing the expected response time and the required response time to an automated driving system of the vehicle.

[0275] Example 195 is one or more non-transitory computer-readable media of Example 194, wherein the plurality of monitored events include at least one of the following: traffic sign event, right-of-way event, parking event, turning event, deceleration event, and / or acceleration event.

[0276] Example 196 is one or more non-transitory computer-readable media of any one of Examples 194 or 195, wherein the monitored environment outside the vehicle includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, speed of the vehicle, road geometry, road type and / or air temperature at the vehicle.

[0277] Example 197 is one or more non-transitory computer-readable media of any one of Examples 194 to 196, wherein the observed characteristics of a passenger may include at least one of the following: passenger gaze, passenger head movement, passenger age, passenger advertising preferences, passenger body posture, passenger body shape, passenger body size, passenger heart rate, passenger non-driving activities, passenger driving quality, and / or noise level inside the vehicle.

[0278] Example 198 is one or more non-transitory computer-readable media of any of Examples 194 to 197, wherein the expected response time is adjusted based on received map-based information, wherein the received map-based information further includes at least one of the following: road curvature, road visibility, density of nearby objects and / or road quality at a geographic location.

[0279] Example 199 is one or more non-transitory computer-readable media of any of Examples 194 to 198, wherein the map-based information further includes a baseline reaction time offset based on the average response time associated with other drivers at that geographic location.

[0280] Example 200 is one or more non-transitory computer-readable media of any of Examples 194 to 199, wherein the method stored in the non-transitory computer-readable medium further includes determining whether to transfer control of the vehicle from an automated driving mode to a passenger based on the expected response time and the required response time.

[0281] Example 201 is an apparatus for transferring control of a vehicle to a passenger, the apparatus including means for monitoring the environment of the vehicle. The apparatus also includes means for receiving a transfer request from a passenger to change control of the vehicle from an automated driving mode. The apparatus further includes means for detecting key events that may be associated with the transfer request, wherein the key events are based on the monitored environment. The apparatus also includes means for generating a handover scene including images associated with the key events, wherein the images comprise a sequence of images within a time period of the key events. The apparatus also includes means for displaying the handover scene to the passenger before the transfer of control of the vehicle from the automated driving mode to the passenger.

[0282] Example 202 is the device of Example 201, wherein the critical events include at least one of the following: a movement event that has caused or caused a change in the trajectory of the vehicle, an unexpected behavior event of the vehicle observed in the monitored environment of the vehicle, and / or an external operating domain event that includes the vehicle's operating setpoint that does not conform to the expected operating conditions of the vehicle.

[0283] Example 203 is a device of either Example 201 or 202, wherein the image further includes event metadata associated with a key event, wherein the event metadata includes at least one of the following: the type of key event, the time of the key event, the location of the key event, the duration of the key event, the importance of each image in the images associated with the key event, and / or the image view of each image in the images associated with the key event.

[0284] Example 204 is a device of any of Examples 201 to 203, wherein the image is generated based on sensor data related to the vehicle.

[0285] Example 205 is a device of either Example 203 or 204, wherein the image includes an enhanced image sequence generated from at least one camera image, which is enhanced using event metadata.

[0286] Example 206 is a device of any of Examples 201 to 205, wherein the handover scenario includes multiple perspectives corresponding to the passenger's viewpoint.

[0287] Example 207 is the device of Example 206, in which multiple perspectives include a forward-facing camera perspective of a vehicle, a bird's-eye view, and / or a map-based perspective.

[0288] Example 208 is a device of any of Examples 203 to 207, and the handover scenario includes static images formed by images and / or event metadata associated with multiple key events.

[0289] Example 209 is the device of Example 208, wherein the still image includes at least one of the following: a single frame of an image associated with a plurality of key events and / or an overlaid image based on one or more images associated with a plurality of key events.

[0290] Example 210 is a device of any of Examples 201 to 209, wherein the handover scenario further includes text associated with an image, wherein the text is generated from the image using a multimodal generalization system.

[0291] Example 211 is the device of Example 210, and the text includes a description of the key events and the time when the key events occurred.

[0292] Example 212 is a device of either Example 210 or 211, wherein the text includes a description of key events expressed verbally to passengers via the audio system of a vehicle.

[0293] Example 213 is a device of either Example 211 or 212, where time is the elapsed time since the critical event occurred.

[0294] Example 214 is a device of any of Examples 201 to 213, in which critical events are assigned priorities corresponding to the importance of the critical events relative to the current situation of the vehicle.

[0295] Example 215 is the device of Example 214, wherein the priority is based on at least one of the following: the time elapsed since the critical event occurred, the category of the critical event, the distance from the location of the critical event to the current location of the vehicle, and / or the likelihood that the passenger is aware of the critical event.

[0296] Example 216 is the device of Example 215, wherein the current location of the vehicle includes the expected location of the vehicle after the elapsed time.

[0297] Example 217 is the device of any of Examples 201 to 216, wherein the device further includes means for monitoring passengers to determine the likelihood that the passengers are aware of the critical event.

[0298] Example 218 is an apparatus of any of Examples 201 to 217, wherein the apparatus further includes means for determining the presentation duration of the handover scene to be presented to the passenger, wherein the presentation duration depends on the urgency of the transition request.

[0299] Example 219 is a device of Example 218, wherein the device further includes means for compressing the length of the handover scene to fit within the presentation duration.

[0300] Example 220 is the device of Example 219, wherein the compression length is removed from the handover scene based on the importance of a portion of the image data associated with the image to the key event.

[0301] Example 221 is a device of Example 219, wherein the handover scenario includes multiple key events, and the compressed length is configured to remove at least one key event from the handover scenario based on the priority of at least one of the multiple key events.

[0302] Example 222 is an apparatus of any one of Examples 201 to 221, wherein the apparatus further includes means for storing a handover scene as a stored handover scene and for displaying the stored handover scene to a passenger based on a request to review the stored handover scene.

[0303] Example 223 is a device of any of Examples 201 to 222, wherein the generation of the handover scenario depends on the passenger’s expected response time.

[0304] Example 224 is the device of Example 223, where the expected response time is based on the average response time of the passenger to multiple monitored events.

[0305] Example 225 is the device of Example 224, wherein the plurality of monitored events include at least one of the following: traffic sign event, right-of-way event, parking event, turning event, deceleration event, and / or acceleration event.

[0306] Example 226 is a device of any of Examples 223 to 225, wherein the expected response time is adjusted based on the environmental rating of the monitored environment of the vehicle.

[0307] Example 227 is a device of any one of Examples 201 to 226, wherein the monitored environment includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, speed of the vehicle, road geometry, road type and / or air temperature at the vehicle.

[0308] Example 228 is a device of any of Examples 223 to 227, wherein the expected response time is adjusted based on a passenger’s attention rating, which depends on the characteristics observed by the passenger.

[0309] Example 229 is the device of Example 228, wherein the characteristics observed by the passenger may include at least one of the following: passenger gaze, passenger head movement, passenger age, passenger advertising preferences, passenger body posture, passenger body shape, passenger body size, passenger heart rate, passenger non-driving activities, passenger driving quality and / or noise level inside the vehicle.

[0310] Example 230 is the device of Example 229, wherein the passenger’s non-driving activities include secondary activities in which the passenger participates.

[0311] Example 231 is a device of any of Examples 223 to 230, wherein the expected response time is adjusted based on the geographic location of the vehicle and map-based information associated with that geographic location.

[0312] Example 232 is the device of Example 231, wherein the map-based information includes at least one of the following: road curvature, road visibility, density of nearby objects and / or road quality at a geographic location.

[0313] Example 233 is a device of either Example 231 or 232, wherein the map-based information includes a baseline reaction time offset associated with the average reaction time of the geographic location.

[0314] Example 234 is a device of any of Examples 226 to 233, wherein the average response time is stored in a database of average response times associated with passengers, wherein the database of average response times is updated using adjusted expected response times.

[0315] Example 235 is the device of Example 234, wherein the database of average response time further includes average response time associated with other drivers.

[0316] Example 236 is an apparatus of any of Examples 201 to 235, wherein the apparatus further includes means for determining the minimum response time required to fulfill a transfer request to change control of the vehicle to a passenger.

[0317] Example 237 is the device of Example 236, wherein the image sequence within a time period of a critical event is based on the minimum response time.

[0318] Example 238 is a device of any of Examples 223 to 237, in which a request for a transition is rejected if the expected response time is slower than the minimum response time.

[0319] Example 239 is a device of any of Examples 223 to 238, wherein the requested transformation proceeds if the expected response time is faster than the minimum response time.

[0320] Example 240 is a device of any of Examples 201 to 239, wherein the observed characteristics of the passenger include passenger pattern confusion, wherein the pattern confusion is determined based on comparing the passenger’s expected action with the passenger’s observed action.

[0321] Example 241 is the device of Example 240, wherein the expected actions include an expected response to a question posed to a passenger in relation to a simulated driving situation, and wherein the observed actions include a response from the driver to the question.

[0322] Example 242 is a device of any one of Examples 240 or 241, wherein the expected action is based on the current operating state of the vehicle and a set of expected vehicle maneuvers associated with the current operating state, and wherein the pattern confusion is based on whether the observed action is at least one vehicle maneuver in the set of expected vehicle maneuvers.

[0323] Example 243 is an apparatus of Example 241 or 242, wherein the apparatus further includes means for granting or denying a conversion request based on pattern obfuscation.

[0324] Example 244 is a device for assessing the attention level of passengers in a vehicle. The device includes means for determining the average response time of passengers within the vehicle based on passenger observations of multiple monitored events. The device further includes means for determining an environmental rating of the vehicle based on a monitored environment outside the vehicle. The device further includes means for determining an in-vehicle rating of the vehicle based on observed characteristics of passengers within the vehicle. The device further includes means for determining the required response time for a geographical location of the vehicle based on received map information regarding response times associated with geographical locations. The device further includes means for determining the expected response time of passengers at a geographical location based on the average response time, environmental rating, and in-vehicle rating. The device further includes means for providing the expected response time and the required response time to an automated driving system of the vehicle.

[0325] Example 245 is the device of Example 244, wherein the plurality of monitored events include at least one of the following: traffic sign event, right-of-way event, parking event, turning event, deceleration event, and / or acceleration event.

[0326] Example 246 is a device of any one of Examples 244 or 245, wherein the monitored environment outside the vehicle includes at least one of the following: weather events experienced at the vehicle, light level events at the vehicle, traffic flow near the vehicle, traffic density of traffic near the vehicle, speed of the vehicle, road geometry, road type and / or air temperature at the vehicle.

[0327] Example 247 is a device of any one of Examples 244 to 246, wherein the characteristics observed by the passenger may include at least one of the following: the passenger's gaze, the passenger's head movement, the passenger's age, the passenger's advertising preferences, the passenger's body posture, the passenger's body shape, the passenger's body size, the passenger's heart rate, the passenger's non-driving activities, the passenger's driving quality, and / or the noise level inside the vehicle.

[0328] Example 248 is a device of any of Examples 244 to 247, wherein the expected response time is adjusted based on received map-based information, wherein the received map-based information further includes at least one of the following: road curvature, road visibility, density of nearby objects and / or road quality at the geographic location.

[0329] Example 249 is a device of any of Examples 244 to 248, wherein the map-based information further includes a baseline reaction time offset based on the average response time associated with other drivers at that geographic location.

[0330] Example 250 is the device of any of Examples 244 to 249, wherein the device further includes means for determining whether to transfer control of the vehicle from the automated driving mode to the passenger based on the expected response time and the required response time.

[0331] While this disclosure has been specifically shown and described with reference to particular aspects, those skilled in the art will understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of this disclosure as defined by the appended claims. Therefore, the scope of this disclosure is indicated by the appended claims and is thus intended to cover all changes falling within the equivalent meaning and scope of the claims.

Claims

1. A vehicle handover system, comprising: The processor is configured to be used for: Monitor the environment of vehicles; Detect critical events related to a request to change control of the vehicle from an automated driving mode to a user within the vehicle, wherein the critical events are based on the monitored environment; Provide the user with information associated with the key events; Monitor the user's attributes; and Based on the monitored attributes, it is determined whether to transfer control of the vehicle from the automated driving mode to the passenger.

2. The vehicle handover system as described in claim 1, wherein, The key events include at least one of the following: a movement event that has caused a change in the trajectory of the vehicle, an unexpected behavior event of the vehicle observed in the monitored environment, or an external operating domain event.

3. The vehicle handover system as described in claim 1, wherein, The processor is further configured to monitor the environment using sensor data from at least one of the following: a camera, a light detection and ranging (LIDAR) sensor, a vehicle position sensor, a vehicle speed sensor, an accelerometer, or a gyroscope.

4. The vehicle handover system as described in claim 1, wherein, The user's attributes are monitored using a camera.

5. The vehicle handover system as described in claim 1, wherein, Determining whether to change control of the vehicle is based on a comparison of the passenger’s expected actions with the user’s observed actions.

6. The vehicle handover system as described in claim 5, wherein, The observed actions include the user's non-driving activities.

7. The vehicle handover system as claimed in claim 1, wherein, Whether to change control of the vehicle is determined based on whether the user's response time is slower than the set response time.

8. The vehicle handover system as claimed in claim 7, wherein, If the response event is slower than the set response event, the processor is further configured to stop the vehicle.

9. The vehicle handover system as claimed in claim 1, wherein, The processor is further configured to warn the user using visual, auditory, or tactile warnings.

10. The vehicle handover system of claim 9, wherein the processor is further configured to warn the user in the event of user distraction.

11. The vehicle handover system as claimed in claim 1, wherein, The processor is configured to display the information before the control of the vehicle is transferred from the automated driving mode to the user.

12. A non-transitory computer-readable medium comprising instructions that, if executed by a processor, cause the processor to: Monitor the environment of vehicles; Detect critical events related to requests to transfer control of the vehicle from automated driving mode to a user within the vehicle, wherein... The key events are based on the monitored environment; Provide the user with information associated with the key events; Monitor the user's attributes; as well as Based on the monitored attributes, it is determined whether to transfer control of the vehicle from the automated driving mode to the passenger.

13. The non-transitory computer-readable medium of claim 12, wherein, The key events include at least one of the following: a movement event that has caused a change in the trajectory of the vehicle, an unexpected behavior event of the vehicle observed in the monitored environment, or an external operating domain event.

14. The non-transitory computer-readable medium of claim 12, wherein, The instructions further instruct the processor to monitor the environment using sensor data from at least one of the following: a camera, a light detection and ranging (LIDAR) sensor, a vehicle position sensor, a vehicle speed sensor, an accelerometer, or a gyroscope.

15. The non-transitory computer-readable medium of claim 12, wherein, The user's attributes are monitored using a camera.

16. The non-transitory computer-readable medium of claim 12, wherein, Determining whether to change control of the vehicle is based on a comparison of the passenger’s expected actions with the user’s observed actions.

17. The non-transitory computer-readable medium of claim 16, wherein, The observed actions include the user's non-driving activities.

18. The non-transitory computer-readable medium of claim 12, wherein, Whether to change control of the vehicle is determined based on whether the user's response time is slower than the set response time.

19. The non-transitory computer-readable medium of claim 18, wherein, If the response time is slower than the set response time, the instruction further causes the vehicle to stop.

20. The non-transitory computer-readable medium of claim 12, wherein, The instructions further enable the processor to warn the user using visual, auditory, or tactile warnings.

21. The non-transitory computer-readable medium of claim 20, wherein, The instruction further causes the processor to warn the user if the client is distracted.

22. The non-transitory computer-readable medium of claim 12, wherein, The instruction further instructs the processor to display the information before control of the vehicle is transferred from the automation mode to the user.

23. A method comprising: Monitor the environment of vehicles; Detect critical events related to a request to change control of the vehicle from an automated driving mode to a user within the vehicle, wherein the critical events are based on the monitored environment; Provide the user with information associated with the key events; Monitor the user's attributes; and Based on the monitored attributes, it is determined whether to transfer control of the vehicle from the automated driving mode to the passenger.

24. A means of transportation, comprising: A vehicle handover system, the vehicle handover system comprising: The processor is configured to be used for: Monitor the environment of vehicles; Detect critical events related to a request to change control of the vehicle from an automated driving mode to a user within the vehicle, wherein the critical events are based on the monitored environment; Provide the user with information associated with the key events; Monitor the user's attributes; and Based on the monitored attributes, it is determined whether to transfer control of the vehicle from the automated driving mode to the passenger.

25. The means of transport as claimed in claim 24, wherein, The key events include at least one of the following: a movement event that has caused a change in the trajectory of the vehicle, an unexpected behavior event of the vehicle observed in the monitored environment, or an external operating domain event.

Images