Real-time event-triggered feedback for autonomous vehicles

Abstract

The present disclosure relates to collecting feedback from passengers of an autonomous vehicle. For example, it can be determined that a triggering environment for triggering a feedback request has been met. The triggering environment can include a driving event, presence of other road users, or a trip status. Based on the determination, a display requirement and data collection parameters of the feedback request are identified. The display requirement defines when the feedback request is displayed, while the data collection parameters identify information to be collected by the feedback request. Based on the display requirement and the data collection parameters, the feedback request is provided for display. In response, feedback from passengers of the autonomous vehicle is received and stored for subsequent use.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 222,277, filed July 15, 2021, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] This invention relates to a method for collecting feedback from passengers in autonomous vehicles. Background Technology

[0004] Autonomous vehicles (such as those that do not require a human driver) can be used to help transport passengers or goods from one location to another. Such vehicles can operate in a fully autonomous mode, where the user provides initial input, such as a pick-up location or destination location, and the vehicle maneuvers itself to that location. When people (or users) want to physically travel and / or transport goods between two locations via vehicle, they can use any number of taxi or delivery services. To date, these services typically involve a human driver who is given dispatch instructions to arrive at a location to pick up and unload users and / or goods. In some cases, passengers are given the opportunity to “rate” their overall experience of the ride, for example, by providing the driver with prompts and / or star ratings or other ratings. Summary of the Invention

[0005] One aspect of this disclosure provides a method for collecting feedback from passengers of an autonomous vehicle. The method includes: one or more processors of a feedback system of the autonomous vehicle determining that a triggering environment for triggering a feedback request has been met, the triggering environment including one or more of a driving event, the presence of other road users, or a trip state; based on the determination, the one or more processors identifying the display requirements and data collection parameters of the feedback request, wherein the display requirements define when the feedback request should be displayed, and the data collection parameters identify information to be collected in the feedback request; the one or more processors providing the feedback request for display based on the display requirements and the data collection parameters; in response to the provision, the one or more processors receiving feedback from passengers of the autonomous vehicle; and the one or more processors storing the feedback for later use.

[0006] In one example, the driving event includes disengaging from autonomous driving mode to manual driving mode. In another example, the presence of other road users includes a predefined number of pedestrians. In another example, the presence of other road users includes a predefined number of cyclists. In another example, the trip status includes the time from when a passenger picks up. In another example, the trip status includes the time from when a passenger drops off. In another example, the triggering environment also includes a passenger lifecycle requirement. In this example, the passenger lifecycle requirement includes whether the passenger is on a specific ride number. Alternatively, the passenger lifecycle requirement includes whether the passenger has made a specific number of rides within a week. Alternatively, the passenger lifecycle requirement includes whether the passenger has a predefined amount of time between multiple rides. Alternatively, the passenger lifecycle requirement includes a predefined number of weeks since the first ride. In another example, the triggering environment also includes a support interaction requirement. In another example, the display requirement defines the delay time between when the triggering environment has been satisfied and when a display feedback request is made. In another example, the display requirement includes a vehicle action that must occur before the display feedback request is made. In another example, the method further includes: based on the determination, identifying a priority for the feedback request, and the priority corresponding to the reciprocal of the frequency at which the triggering environment might occur. In this example, the method further includes using the priority of the feedback request to determine when to display the feedback request. In another example, the provision includes providing a display on a display of an autonomous vehicle. In another example, the provision includes providing a display on a display of a client computing device associated with a passenger. In another example, the method further includes: based on the determination, identifying a frequency limit for the feedback request, and the provision is also based on the frequency limit. In this example, the frequency limit includes a fixed maximum number of feedback requests displayed during the trip. Attached Figure Description

[0007] Figure 1 This is a functional diagram of an example vehicle according to an exemplary embodiment.

[0008] Figure 2 These are example exterior views of a vehicle based on various aspects of this disclosure.

[0009] Figure 3 It is a functional diagram of an example system based on various aspects of this disclosure.

[0010] Figure 4 These are schematic diagrams of example systems based on various aspects of this disclosure.

[0011] Figure 5 Based on all aspects of this disclosure Figure 4 The system's functional diagram.

[0012] Figures 6A-6H These are examples of displays and information displayed according to various aspects of this disclosure.

[0013] Figures 7A-7D These are examples of displays and information displayed according to various aspects of this disclosure.

[0014] Figure 8 These are example flowcharts based on various aspects of this disclosure. Detailed Implementation

[0015] The technology involves collecting feedback from passengers during trips involving autonomous vehicles. While some feedback systems can request feedback from users once the mission is complete, for example, after the trip, this may not always be the most useful feedback in a timely manner. To achieve real-time feedback, a human operator (“internal stakeholder”) must first define the feedback request. Each feedback request can include three main characteristics: triggering environment, display requirements, and data collection parameters.

[0016] Triggering environments can include requirements such as driving events within a predetermined time period or number of times, the presence or actions of other road users, trip status, road map requirements, support interactions, user-generated / initiated feedback, passenger lifecycle, time of day, other driving conditions, or a combination of these. Road map requirements can include certain types of intersections, unprotected turns, narrow streets, construction zones, exit zones (where the autonomous vehicle will automatically disengage from autonomous driving mode to manual driving mode, or areas where parking is not permitted), etc. Display requirements can define when to display feedback requests in order to capture the most useful feedback in a timely manner.

[0017] Data collection parameters identify the information to be collected in a feedback request, or more precisely, parameters that define the information to be input into the feedback system. In some cases, the data collection parameters for a feedback request may also vary based on the passenger type and / or lifecycle.

[0018] Assuming feedback requests meet internal requirements for quality and usefulness (e.g., some internal standards), priorities can be assigned to them. In one instance, a priority corresponds to the inverse of the frequency at which a triggering environment might occur. As another instance, a priority could correspond to the need for a particular type of data, regardless of frequency.

[0019] Triggering conditions, display requirements, and data collection parameters can be stored in lookup tables, databases, or other storage systems. These lookup tables, databases, or other storage systems allow one or more computing devices in the autonomous vehicle's feedback system to precisely determine when a triggering condition is met and then identify any corresponding display requirements and data collection parameters.

[0020] The computing devices in the feedback system can monitor information from various systems within the autonomous vehicle. For example, they can continuously analyze road map features and location information from the autonomous vehicle's localization system, route and trajectory features from its planning system, detected objects and characteristics from its perception system, and behavior predictions from its behavior modeling system. Based on this, the feedback system can determine when a triggering environment occurs.

[0021] Once a triggering environment occurs, the feedback system's computing device can generate and display an interface for the user. This can be displayed on the vehicle's display or potentially on the user's phone, depending on the display requirements of the feedback request, with or without an audible notification. When multiple feedback requests exist simultaneously, prioritization allows the feedback system to select the most important request to display to the user, sacrificing others.

[0022] Feedback can be received and stored for subsequent analysis and / or automatically sent to a fleet management system for storage and further analysis. For example, a triage team can review feedback and create notifications for the human operator or another entity that initially requested it. Additionally, given the precise timestamp of when a problem occurred, the triage team can pinpoint the problem more quickly and efficiently. In some cases, feedback can be used in real-time to influence the driving behavior of autonomous vehicles.

[0023] The features described in this paper allow autonomous vehicles to collect passenger feedback in real time while events important to stakeholders are occurring. Therefore, the feedback can be more accurate and even more useful because it is not simply collected after the trip has ended. For example, passengers will have a better memory of the event than later in the trip, having just experienced it. Furthermore, subsequent negative or positive events during the ride will not affect the passenger's feedback on the current event. Additionally, some passengers may have a better experience if feedback is solicited multiple times during the ride compared to asking several questions simultaneously at the end of the ride (when passengers are unlikely to recall the answers correctly).

[0024] Example System

[0025] like Figure 1 As shown, an autonomous vehicle 100 according to one aspect of this disclosure includes various components. While certain aspects of this disclosure are particularly useful in combination with specific types of vehicles, the vehicle can be any type of vehicle, including but not limited to cars, trucks, motorcycles, buses, recreational vehicles, etc. The vehicle may have one or more computing devices, such as a computing device 110 containing one or more processors 120, memory 130, and other components typically found in general-purpose computing devices.

[0026] Memory 130 stores information accessible by one or more processors 120, including data 132 and instructions 134 that can be executed or otherwise used by the processor 120. Memory 130 can be any type capable of storing information accessible by a processor, including computing devices or computer-readable media, or other media that store data readable by means of electronic devices, such as hard disk drives, memory cards, ROM, RAM, DVDs or other optical discs, and other writable and read-only memories. Systems and methods may include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0027] Instructions 134 can be any set of instructions that can be executed directly by the processor (such as machine code) or indirectly (such as scripts). For example, instructions can be stored as computing device code on a computing device-readable medium. In this regard, the terms "instruction" and "program" are used interchangeably herein. Instructions can be stored in object code format for direct processor processing or in any other computing device language, including sets of scripts or stand-alone source code modules that are interpreted on demand or pre-compiled. The function, methods, and routines of instructions are explained in more detail below.

[0028] Data 132 can be retrieved, stored, or modified by processor 120 according to instructions 134. For example, although the claimed subject matter is not limited to any particular data structure, the data can be stored in a computing device register, as a table with multiple different fields and records, an XML document, or a flat file in a relational database. The data can also be formatted in any computing device-readable format.

[0029] One or more processors 120 can be any conventional processor, such as a commercially available CPU or GPU. Alternatively, one or more processors can be dedicated devices, such as ASICs or other hardware-based processors. Although Figure 1While the processor, memory, and other components of computing device 110 are shown functionally as being housed in the same block, those skilled in the art will understand that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be stored in the same physical housing. For example, memory may be a hard disk drive or other storage medium located in a different housing than computing device 110. Therefore, references to processors or computing devices will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0030] The computing device 110 may include all components typically used in conjunction with a computing device, such as the processor and memory described above, as well as user input devices 150 (e.g., one or more buttons, a mouse, a keyboard, a touchscreen, and / or a microphone), various electronic displays (e.g., a monitor with a screen or any other electrical device operable to display information), and speakers 154, to provide information to passengers or others of the autonomous vehicle 100 as needed. For example, an electronic display 152 may be located inside the passenger compartment of the autonomous vehicle 100 and may be used by the computing device 110 to provide information to passengers within the autonomous vehicle 100.

[0031] The computing device 110 may also include one or more wireless network connections 156 to facilitate communication with other computing devices, such as client computing devices and server computing devices described in detail below. The wireless network connection may include short-range communication protocols such as Bluetooth, Bluetooth Low Energy (LE), cellular connectivity, and various configurations and protocols including the Internet, World Wide Web, intranet, virtual private network, wide area network, local area network, private network using one or more proprietary communication protocols, Ethernet, WiFi, and HTTP, as well as various combinations thereof.

[0032] The computing device 110 may be part of an autonomous control system for the autonomous vehicle 100 and may be able to communicate with various components of the vehicle to control the vehicle in autonomous driving mode. For example, returning Figure 1 The computing device 110 can communicate with various systems of the autonomous vehicle 100, such as the deceleration system 160, acceleration system 162, steering system 164, signal system 166, planning system 168, routing system 170, positioning system 172, perception system 174, behavior modeling system 176, and power system 178, so as to control the movement, speed, etc. of the autonomous vehicle 100 according to the instructions 134 of the memory 130 in autonomous driving mode.

[0033] As an example, computing device 110 can interact with deceleration system 160 and acceleration system 162 to control the speed of the vehicle. Similarly, steering system 164 can be used by computing device 110 to control the direction of autonomous vehicle 100. For example, if autonomous vehicle 100 is configured for use on a road, such as a car or truck, the steering system may include components for controlling wheel angles to turn the vehicle. Computing device 110 may also use signaling system 166 to signal the vehicle's intentions to other drivers or vehicles, for example, by illuminating turn signals or brake lights when needed.

[0034] Routing system 170 can be used by computing device 110 to generate routes to destinations using map information. Planning system 168 can be used by computing device 110 to generate short-term trajectories that allow vehicles to follow routes generated by the routing system. In this regard, planning system 168 and / or routing system 170 can store detailed map information, such as pre-stored highly detailed maps identifying road networks, including road shapes and elevations, lane lines, intersections, pedestrian crossings, speed limits, traffic signals, buildings, signs, real-time traffic information (such as updates received from remote computing devices, such as computing device 410 or other computing devices discussed below), parking locations, vegetation, or other such objects and information.

[0035] In addition to the physical characteristics described above, the map information can be configured as a road map, comprising multiple graphical nodes and edges representing road or vehicle lane segments, which together constitute the road network of the map information. Each edge is defined by a starting graphical node with a specific geographic location (e.g., latitude, longitude, altitude, etc.), an ending graphical node with a specific geographic location (e.g., latitude, longitude, altitude, etc.), and a direction. This direction may refer to the direction in which the autonomous vehicle 100 must move to follow the edge (i.e., the direction of traffic flow). Graphical nodes can be located at fixed or variable distances. For example, the spacing between graphical nodes can range from a few centimeters to a few meters and may correspond to the speed limits of the roads where the graphical nodes are located. In this respect, a greater speed may correspond to a greater distance between graphical nodes. Edges may represent driving along the same lane or changing lanes. Each node and edge may have a unique identifier, such as the latitude and longitude location of the node or the start and end locations of the edge or node. In addition to nodes and edges, the map can identify additional information, such as the type of maneuver required at different edges and which lanes are drivable.

[0036] The routing system 170 can use the aforementioned map information to determine a route from the current location (e.g., the location of the current node) to the destination. Routes can be generated using cost-based analysis, which attempts to select the route to the destination with the lowest cost. Costs can be evaluated in any number of ways, such as time to reach the destination, distance traveled (each edge can be associated with a cost for crossing that edge), type of maneuver required, passenger or vehicle convenience, etc. Each route can include a list of multiple nodes and edges that a vehicle can use to reach the destination. The route can be recalculated periodically as the vehicle travels to the destination.

[0037] The map information used for route planning can be the same as or different from the map used for trajectory planning. For example, map information for route planning requires not only information about individual lanes but also the nature of lane boundaries (e.g., solid white, dashed white, solid yellow, etc.) to determine where lane changes are permitted. However, unlike the map used for trajectory planning, map information for routing does not need to include other details such as the location of pedestrian crossings, traffic lights, stop signs, etc., although some of this information may be useful for routing purposes. For example, between a route with many intersections with traffic control (such as stop signs or traffic lights) and a route with little or no traffic control, the latter route may have a lower cost (e.g., because it is faster) and is therefore preferred.

[0038] Positioning system 172 can be used by computing device 110 to determine the relative or absolute position of the vehicle on a map or the earth. For example, positioning system 172 may include a GPS receiver to determine the latitude, longitude, and / or altitude of the device. Other positioning systems, such as laser-based positioning systems, inertial-assisted GPS, or camera-based positioning, may also be used to identify the vehicle's position. The vehicle's position may include absolute geographic location, such as latitude, longitude, and altitude, the position of nodes or edges on a road map, and relative position information, such as its position relative to other vehicles in its immediate vicinity, which can typically be determined with less noise than absolute geographic location.

[0039] The positioning system 172 may also include other devices, such as accelerometers, gyroscopes, or other orientation / velocity detection devices, that communicate with the computing device 110 to determine the vehicle's orientation and speed, or changes thereof. By way of example only, the accelerometer may determine its pitch, yaw, or roll (or changes thereof) relative to the direction of gravity or a plane perpendicular to it. The device may also track increases or decreases in speed and the direction of such changes. The provision of position and orientation data by the devices described herein may be automatically made available to the computing device 110, other computing devices, and combinations thereof.

[0040] The perception system 174 also includes one or more components for detecting objects outside the vehicle, such as other vehicles, obstacles in the road, traffic signals, signs, trees, etc. For example, the perception system 174 may include LIDAR, sonar, radar, cameras, and / or any other detection devices that record data that can be processed by the computing device of computing device 110. In the case where the vehicle is a passenger car such as a minivan, the minivan may include lasers or other sensors mounted on the roof or in another convenient location.

[0041] For example, Figure 2 This is an example external view of an autonomous vehicle 100. In this example, the roof housing 210 and dome housing 212 may include LIDAR sensors as well as various camera and radar units. Additionally, housing 220 located at the front of the autonomous vehicle 100 and housings 230, 232 on the driver's and passenger's sides of the vehicle may each house LIDAR sensors. For example, housing 230 is located in front of the driver's door 260. The autonomous vehicle 100 also includes housings 240, 242 for radar units and / or cameras also located on the roof of the autonomous vehicle 100. Additional radar units and cameras (not shown) may be located at the front and rear ends of the autonomous vehicle 100 and / or at other locations along the roof or roof housing 210.

[0042] The computing device 110 may be able to communicate with various components of the vehicle to control the movement of the autonomous vehicle 100 according to the master vehicle control code in the memory of the computing device 110. For example, referring back to FIG1, the computing device 110 may include various computing devices that communicate with various systems of the autonomous vehicle 100, such as a deceleration system 160, an acceleration system 162, a steering system 164, a signaling system 166, a planning system 168, a routing system 170, a positioning system 172, a perception system 174, a behavior modeling system 176, and a power system 178 (i.e., the vehicle's engine or motor) to control the movement, speed, etc. of the autonomous vehicle 100 according to instructions 134 in the memory 130.

[0043] Various vehicle systems can be operated using autonomous vehicle control software to determine how to control the vehicle. As an example, the perception system software module of perception system 174 can use sensor data generated by one or more sensors of the autonomous vehicle (e.g., cameras, LiDAR sensors, radar units, sonar units, etc.) to detect and identify objects and their characteristics. These characteristics can include position, type, heading, orientation, speed, acceleration, changes in acceleration, size, shape, etc. In some cases, characteristics can be input into the behavior prediction system software module of behavior modeling system 176, which uses various behavior models based on object type to output predicted future behavior of the detected objects. In other cases, features can be placed into one or more detection system software modules, such as a traffic light detection system software module configured to detect the state of known traffic signals, a construction zone detection system software module configured to detect construction zones from sensor data generated by one or more sensors of the vehicle, and an emergency vehicle detection system configured to detect emergency vehicles from sensor data generated by the vehicle's sensors. Each of these detection system software modules can use various models to output the probability that a construction zone or object is an emergency vehicle. The detected objects, predicted future behaviors, various possibilities from the detection system software module, map information identifying the vehicle's environment, location information from the positioning system 172 identifying the vehicle's position and orientation, the vehicle's destination location or node, and feedback from various other systems of the vehicle can be input into the planning system software module of the planning system 168. The planning system 168 can use this input to generate a trajectory that the vehicle will follow over a short period of time in the future, based on a route generated by the routing module of the routing system 170. In this regard, the trajectory can define specific characteristics such as acceleration, deceleration, and velocity to allow the vehicle to follow a route toward its destination. The control system software module of the computing device 110 can be configured to control the vehicle's movement, for example, by controlling the vehicle's braking, acceleration, and steering, in order to follow the trajectory.

[0044] The computing device 110 can control the vehicle in autonomous driving mode by controlling various components. For example, as an example, the computing device 110 can autonomously navigate the vehicle to a destination location using detailed map information and data from the planning system 168. The computing device 110 can use the positioning system 172 to determine the vehicle's position and use the perception system 174 to detect objects and respond to them as needed to safely reach the location. Again, to do this, the computing device 110 and / or the planning system 168 can generate trajectories and make the vehicle follow these trajectories, for example, by accelerating the vehicle (e.g., by supplying fuel or other energy to the engine or power system 178 by the acceleration system 162), decelerating (e.g., by reducing the fuel supplied to the engine or power system 178, changing gears, and / or by applying braking by the deceleration system 160), changing direction (e.g., by turning the front or rear wheels of the autonomous vehicle 100 by the steering system 164), and signaling such changes using the signaling system 166 (e.g., by illuminating turn signals). Therefore, the acceleration system 162 and the deceleration system 160 can be part of a transmission system that includes various components between the vehicle's engine and the vehicle's wheels. Furthermore, by controlling these systems, the computing device 110 can also control the vehicle's transmission system to autonomously maneuver the vehicle.

[0045] The autonomous vehicle 100 may also include a feedback system. For example, turning to Figure 3 The feedback system 300 may include one or more computing devices 310 having one or more processors 320, memory 330, data 332, and instructions 334. Such processors, memory, data, and instructions may be configured similarly to one or more processors 120, memory 130, data 132, and instructions 134 of the computing device 110, respectively. The feedback system 300 may communicate with various systems of the autonomous vehicle 100, including the computing device 110, deceleration system 160, acceleration system 162, steering system 164, signal system 166, planning system 168, routing system 170, positioning system 172, perception system 174, behavior modeling system 176, power system 178, etc. Alternatively, the feedback system may be incorporated into or be part of the computing device 110.

[0046] Computing device 310 may include user input device 350, electronic display 352, speaker 354, and wireless network connection 356. Each of these may be configured to be the same as or similar to user input device 150, electronic display 152, speaker 154, and wireless network connection 156 as described above. In other cases, these may be identical (i.e., it is not necessary to replicate them all between computing device 110 and computing device 310).

[0047] The computing device 310 may also include one or more wireless network connections 356 to facilitate communication with other computing devices, such as client computing devices and server computing devices described in detail below. The wireless network connection may include short-range communication protocols such as Bluetooth, Bluetooth Low Energy (LE), cellular connectivity, and various configurations and protocols including the Internet, World Wide Web, intranet, virtual private network, wide area network, local area network, private network using one or more proprietary communication protocols, Ethernet, WiFi, and HTTP, as well as various combinations thereof.

[0048] Additionally, memory 330 and / or memory 130 may store information about feedback requests. Such information may include three main characteristics: triggering environment, display request, and data collection parameters. Triggering environment may include requests such as driving events within a predetermined time period or number of times (e.g., 10 horn blasts), the presence or actions of other road users, trip status, road map requests, support interactions, user-generated / initiated feedback, passenger lifecycle, time of day, other driving conditions (e.g., traffic or weather conditions), or a one-off combination of these. Examples of driving events may include emergency braking, turning or swaying, certain accelerations (e.g., rapid acceleration), disengaging from automatic driving mode to manual driving mode, etc. Examples of the presence or actions of other road users may include the presence of a single pedestrian, multiple pedestrians, a single cyclist, multiple cyclists, a single scooter, multiple scooters, horn blasting, being overtaken or caught by another road user, etc. Examples of trip status may include whether the passenger has just boarded or is about to alight, has begun moving after stopping for an emergency, or has had their trip canceled, etc.

[0049] Road map requirements may include certain types of intersections, unprotected turns, narrow streets, construction zones, exit zones (where the autonomous vehicle will automatically disengage from autonomous driving mode to manual driving mode), or areas where parking is not permitted. Examples of supported interactions may include calls or chats initiated by passengers and / or remote assistance operators between passengers and remote assistance operators. Examples of passenger lifecycle may include whether a passenger is at a specific ride number (e.g., first ride, 10th ride, etc.), a specific number of rides per week, a specific number of days or weeks between rides, a specific number of weeks since the first ride, etc.

[0050] Display requirements can define when to display a feedback request, ensuring the most useful feedback is captured promptly. In some cases, the default display requirement might be to display the feedback request immediately once the triggering condition is met. In other cases, display requirements can define the delay between a met triggering condition and the display of the feedback request. Still other cases, display requirements can define additional criteria, such as waiting until the autonomous vehicle moves again if an emergency occurs and the vehicle stops.

[0051] Data collection parameters identify the information to be collected in a feedback request, or more precisely, parameters that define the information to be input into the feedback system, such as binary (yes / no), sliding scales, and bucket responses (e.g., a smile to a frown to indicate pleasure or displeasure). In some cases, the data collection parameters for a feedback request can also vary based on passenger type and / or lifespan. For example, feedback requests to more experienced passengers (such as those who have taken more rides, are taking unpaid rides for testing purposes, or have previously provided direct feedback on the type of incident) may receive more direct questions compared to less experienced passengers. For instance, after an emergency braking event, feedback requests may be more general (e.g., “How is the driving comfort now?”) for less experienced passengers and more specific (e.g., “How was that braking?”) for more experienced passengers.

[0052] Assuming feedback requests meet internal requirements for quality and usefulness (e.g., some internal standards), they are assigned priorities. In one instance, priority corresponds to the reciprocal of the frequency of the triggering environment. Lower-frequency triggering environments result in higher priorities, and vice versa. This can be estimated based on log data. As another instance, priorities can correspond to the need for a specific type of data, regardless of frequency. For example, if it is necessary to measure vehicle responses to certain types of road users, such as responses to children or family groups (e.g., children and adults together), such feedback requests could be assigned a higher priority than other types of feedback requests.

[0053] The triggering environment, display requirements, and data collection parameters can be stored in a lookup table, database, or other storage configuration in memory 330, which allows one or more computing devices 310 of the feedback system to precisely determine when the triggering environment is met, and then identify any corresponding display requirements and data collection parameters.

[0054] The computing device 110 of the autonomous vehicle 100 can also receive information from or transmit information to other computing devices, such as those computing devices that are part of the transportation service and other computing devices. Figure 4 and Figure 5 These are schematic and functional diagrams of an example system 400, including multiple computing devices 410, 420, 430, 440 and a storage system 450 connected via a network 460. System 400 also includes autonomous vehicle 100A and autonomous vehicle 100B, which can be configured to be the same as or similar to autonomous vehicle 100. Although only a few vehicles and computing devices are depicted for simplicity, a typical system may include significantly more vehicles and computing devices.

[0055] like Figure 5 As shown, each of computing devices 410, 420, 430, and 440 may include one or more processors, memory, data, and instructions. Such processors, memory, data, and instructions may be configured similarly to one or more processors 120, memory 130, data 132, and instructions 134 of computing device 110.

[0056] Network 460 and intervening graphical nodes can include a variety of configurations and protocols, including short-range communication protocols such as Bluetooth, Bluetooth LE, the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local area networks, private networks using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, as well as various combinations thereof. This communication can be facilitated by any device capable of sending and receiving data to and from other computing devices, such as modems and wireless interfaces.

[0057] In one example, one or more computing devices 410 may include one or more server computing devices with multiple computing devices, such as a load balancing server cluster, which exchanges information with different nodes in the network for receiving data from other computing devices, processing data, and sending data to other computing devices. For example, one or more computing devices 410 may include one or more server computing devices capable of communicating via network 460 with computing device 110 of autonomous vehicle 100 or similar computing devices of autonomous vehicle 100A or 100B, as well as computing devices 420, 430, and 440. For example, vehicles 100, 100A, and 100B may be part of a fleet capable of being dispatched to various locations by the server computing devices. In this respect, server computing device 410 can be used as a fleet management system capable of arranging trips for passengers by allocating and dispatching vehicles such as vehicles 100, 100A, and 100B. These arrangements may include trips scheduled to different locations for passengers to board and alight. In this respect, server computing device 410 may be operated using scheduling system software to manage the aforementioned autonomous vehicle scheduling and dispatching. Additionally, computing device 410 can use network 460 to send and present information to users (such as users 422, 432, 442) on displays (such as displays 424, 434, 444 of computing devices 420, 430, 440). In this respect, computing devices 420, 430, 440 can be considered as client computing devices.

[0058] like Figure 4 As shown, each client computing device 420, 430 may be a personal computing device intended for use by users 422, 432, and has all the components typically used in conjunction with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory for storing data and instructions (e.g., RAM and internal hard disk drives), displays such as displays 424, 434, 444 (e.g., a monitor with a screen, a touchscreen, a projector, a television, or other devices operable to display information), and user input devices 426, 436, 446 (e.g., a mouse, keyboard, touchscreen, or microphone). The client computing device may also include a camera for recording video streams, speakers, network interface devices, and all components for connecting these elements to each other.

[0059] While client computing devices 420 and 430 may both include full-size personal computing devices, they may alternatively include mobile computing devices capable of wirelessly exchanging data with a server via a network such as the Internet. By way of example only, client computing device 420 may be a mobile phone or a device such as a wireless-enabled PDA, tablet PC, wearable computing device or system, or a netbook capable of obtaining information via the Internet or other networks. In another example, client computing device 430 may be a wearable computing system, as shown in the example below. Figure 4 The wristwatch shown is an example. As an example, a user can input information using a keypad, a keyboard, a microphone, visual signals from a camera, or a touchscreen. As yet another example, the client computing device 440 can be a desktop computing system that includes a keyboard, mouse, camera, and other input devices.

[0060] Each of the client computing devices can be a remote computing device used by a person (e.g., a human operator or user 422, 432, 442) to review and analyze sensor data and other information generated by the vehicle's perception system (such as perception system 174 of the autonomous vehicle 100). Although in Figure 3 and Figure 4 Only a few remote computing devices are shown, but a typical system could include any number of such workstations.

[0061] Similar to memory 130, storage system 450 can be any type of computerized memory capable of storing information accessible by server computing device 410, such as hard disk drives, memory cards, ROM, RAM, DVDs, CD-ROMs, writable and read-only memories. Additionally, storage system 450 can include a distributed storage system where data is stored on multiple different storage devices, which may be physically located in the same or different geographical locations. Storage system 450 can be configured via... (as shown in Figure 4 and...) Figure 5 The network 460 shown is connected to a computing device, and / or can be directly connected to or incorporated into any of the computing devices 110, 410, 420, 430, 440, etc.

[0062] As described in more detail below, storage system 450 can store various types of information. This information can be retrieved or otherwise accessed by server computing devices (such as one or more server computing devices 410) to perform some or all of the features described herein. For example, storage system 450 can store log data. Log data can include, for example, sensor data generated by a perception system (such as perception system 174 of autonomous vehicle 100). The perception system can include multiple sensors that generate the sensor data. As an example, sensor data can include raw sensor data as well as data identifying defined characteristics of the perceived objects (including other road users), such as the shape, position, orientation, speed, etc., of the objects (such as vehicles, pedestrians, cyclists, vegetation, curbs, lane lines, sidewalks, crosswalks, buildings, etc.). Log data may also include “event” data that identifies different types of events, such as collisions or near-collisions with other objects, planned trajectories describing the planned terrain and / or speed of the autonomous vehicle 100's potential path, the vehicle's actual position at different times, the vehicle's actual orientation / direction of travel at different times, the vehicle's actual speed, acceleration, and deceleration (and / or changes in speed, orientation / direction of travel / steering angle, acceleration, etc. over time) at different times, the classification and response of perceived objects, predictions of the behavior of perceived objects, and the state of the vehicle's various systems (such as acceleration, deceleration, perception, steering, signaling, routing, power, etc.) at different times (including log errors, inputs and outputs of the vehicle's various systems at different times, etc.). Therefore, this event and sensor data can be used to “reconstruct” the vehicle's environment, including perceived objects and the vehicle's behavior in the simulation.

[0063] At least some of the log data can be “ride data,” that is, log data generated during a specific trip or ride by a passenger in an autonomous vehicle, such as autonomous vehicle 100. Ride data can include all the features described above in the log data, or it can include only specific types of data, such as motion planning commands from the vehicle’s planning system (such as planning system 168), telemetry from the vehicle, context from map information used to control the vehicle, processed or raw sensor data from other road users (such as vehicles, cyclists, pedestrians, etc.) from the vehicle’s perception system (such as perception system 174), acceleration information, jerk (or derivative of acceleration) information, etc. Ride data can also be associated with ride feedback provided by one or more passengers for a specific ride. As discussed further below, at least some of this feedback can be provided by passengers in response, for example, immediately during or after an unpleasant event. Additionally or alternatively, as discussed further below, feedback can include an overall ride quality value after the ride has been completed.

[0064] Example Method

[0065] In addition to the operations shown above and in the accompanying drawings, various other operations will now be described. It should be understood that the following operations need not be performed in the exact order described below. Instead, various steps can be processed in different orders or simultaneously, and steps can be added or omitted.

[0066] Figure 8 Example flowchart 800 includes some examples of collecting feedback from passengers in an autonomous vehicle, which can be executed, for example, by one or more processors of the autonomous vehicle's feedback system (such as processor 320 of the feedback system 300 of autonomous vehicle 100 and / or processor 120 of computing device 110). For example, at block 810, a triggering environment has been determined that has been met to trigger a feedback request. The triggering environment includes one or more of the following: a driving event, the presence of other road users, or the trip state.

[0067] The computing device 310 of the feedback system 300 can monitor information from various systems of the autonomous vehicle 100. For example, the computing device of the feedback system can continuously analyze road map features and location information from the autonomous vehicle's positioning system 172, route and trajectory features from the autonomous vehicle's routing system 170 and planning system 168, detected objects and characteristics from the autonomous vehicle's perception system 174 (e.g., the state of yellow lights, pedestrians in crosswalks, etc.), and behavior predictions (e.g., trajectory predictions) from the autonomous vehicle's behavior modeling system 176. Based on this, the feedback system can determine when the triggering environment defined in the memory 330 occurs.

[0068] At block 820, based on the determination, display requirements and data collection parameters for the feedback request are determined. The display requirements define when the feedback request should be displayed, while the data collection parameters identify the information to be collected in the feedback request. For example, once a triggering environment is identified in memory 330 and / or memory 130, computing device 310 can identify any associated display requirements and data collection parameters.

[0069] At box 830, a feedback request is provided for display based on display requirements and data collection parameters. Once a triggering environment occurs, the computing device of the feedback system is able to generate and display an interface for the user. This can be achieved by sending the feedback request to the passenger's client computing device via a network such as network 460, on a display in the vehicle (such as display 152 or display 352), or potentially on the passenger's client computing device (e.g., client computing device 420). The feedback request can be displayed with or without an audible notification (e.g., a "ding").

[0070] Figures 6A-6H Example representations of information that can be displayed on displays 152 or 352 by computing devices 110 and / or 310 during a trip are provided. Figure 6A In one example, after a passenger has initiated a trip to their destination, an information screen is depicted indicating that the autonomous vehicle 100 is en route to the destination. Figure 6B Depicting what can be done Figure 6A The example shown after the example is a sample representation of the feedback request. In this example, Figure 6B The feedback request includes information for the user to provide feedback (“Kat, did you get on the bus smoothly?”) and a request with multiple options (angry face, frowning face, neutral face, and smiling face, etc.). In this example, the feedback request has been customized to include the passenger's name (“Kat”), although this may not be required.

[0071] Figure 7A and Figure 7B An example is provided of how feedback requests can be displayed on a passenger's client-side computing device. Figure 7A In this scenario, the feedback request is initially displayed as a notification on the passenger's client computing device 420's "lock screen". Once the passenger unlocks the notification, the rest of the feedback request is displayed.

[0072] As mentioned above, multiple options can be defined by the data collection parameters of the feedback request. Figure 6C In the example, the passenger may have already selected an angry face (here, "bad") using, for example, a touchscreen or another user input device in the vehicle (e.g., user input device 150). Figure 6C and Figure 7C As shown, computing devices 110 and / or 310 can automatically highlight the selected option.

[0073] However, as mentioned above, when to display a feedback request depends on the display requirements of that request. In this regard, display requirements can identify how long after a triggering condition is met should a feedback request be displayed. For example, display requirements could be "15 seconds after entering a narrow lane," "just after leaving an intersection," etc. Some feedback requests may be better suited to one type of input (e.g., voice for vehicles and text for telephone calls).

[0074] In some situations, there may be multiple feedback requests that need to be displayed simultaneously or within a short period of time. For example, within a 10-second timeframe, the feedback system may determine that the triggering conditions for two or more different feedback requests have been met. Prioritization allows the feedback system to select the most important requests to display to the user, sacrificing others. Alternatively, lower-priority requests may potentially be delayed until a later ride.

[0075] At box 840, in response to the provision, feedback from a passenger in the vehicle is received. For example, the passenger may provide feedback via user input device 150 and / or user input device 350 or via a user input device on the passenger's client computing device. If a client computing device (e.g., the passenger's mobile phone) is used, the feedback may be sent to the vehicle (e.g., via Bluetooth, Bluetooth Low Energy (LE), Near Field Communication, via network 460, or other communication protocols), or it may be sent to a server computing device (such as server computing device 410) and subsequently forwarded by the server computing device to the passenger's client computing device (e.g., computing device 420).

[0076] In some cases, feedback requests can include multiple "layers" of information requests in order to gather as much information as possible about the passenger experience. In other words, additional feedback requests can be displayed based on the passenger's response. For example, going to... Figure 6D and Figure 7C It can respond to the user's selection of Figure 6C or Figure 7C The frowning face in the image displays a second feedback request (“Tell us what’s wrong”). The second feedback request can also include multiple options (“Feel unsafe,” “Wrong location,” and “Away from the curb”) again defined by the data collection parameters of the second feedback request. In this example, the passenger can select the “Feel unsafe” option, such as... Figure 6E It is highlighted in the middle.

[0077] like Figure 6E As shown, once the passenger has made a second feedback request (or potentially as...) Figure 6B The initial feedback request shown provides feedback, and computing devices 110 and / or 310 can provide additional options, such as an "Add Comment" option that allows passengers to record and submit audible or video messages, and a "Submit" option that allows passengers to submit a response to the feedback request. Similarly, as Figure 7D As shown, the client computing device 420 can display a text box (“Anything to add”) for the user to provide text as well as additional options (options 710 and 720, respectively) for providing image or audio recordings. Figure 7D Option 730 is also provided for submitting passenger feedback.

[0078] For example, in Figure 6F and Figure 6G In the example, as indicated by notification 620, audio is being recorded (“listened to”). This example also provides the passenger with some guidance (attempted explanation, such as “difficult to enter the car”) along with additional options for “cancelling” or “submitting” the audible recording. In the example of Figure 6G, computing devices 110 and / or 310 have transcribed the audible message and displayed it again on the display, along with additional options for “cancelling” or “submitting” the audible recording.

[0079] At box 850, feedback is stored for later use. Feedback results can be received and stored, for example, in memory 130 and / or memory 330, for subsequent analysis and / or automatically sent to the fleet management system for storage and further analysis. In this regard, feedback can be sent to server computing device 410 for storage in storage system 450. Once the passenger selects any of the above submission options, such as... Figure 6HAs shown, computing devices 110 and / or 310 can provide a summary of user-provided feedback and an indication 630 that the feedback is being submitted (e.g., stored for later use and / or automatically sent). Once submitted and / or after an appropriate time period has elapsed, displays 152 and / or 352 can return to... Figure 6A The information screen in the middle or another information screen, Figure 6A The information screen indicates that the autonomous vehicle 100 is en route to its destination.

[0080] In some cases, the consultation team can review feedback and create notifications (e.g., errors) for the human operator or another entity that initially requested the feedback. Additionally, where there is an accurate timestamp of when the problem occurred (e.g., "a car stopped us a few seconds ago" compared to "a car stopped us at some point during my 20-minute ride"), the consultation team can be more quickly and effectively pinpointing the problem.

[0081] In some situations, feedback can be used in real time to influence the driving behavior of autonomous vehicles. For example, if a passenger has a strong negative reaction to a moment when an autonomous vehicle suddenly accelerates, makes an unprotected turn, or performs some other maneuver, the autonomous vehicle can be controlled more conservatively from that moment on, thus optimizing less for time / route efficiency and potentially even avoiding unauthorized maneuvers. As another example, if a passenger reports a negative boarding experience in real time, the fleet management system can disable that boarding location for all other autonomous vehicles in the fleet and avoid other undesirable boardings in that area, for example, at least until the cause of the negative boarding experience can be determined.

[0082] Feedback requests can also have frequency and passenger-specific limitations. For example, a fixed maximum number of feedback requests can be displayed per trip, or this can vary depending on the trip length. In some cases, the number of feedback requests for a particular trip can vary based on the passenger's past history of providing responses (e.g., more responses from the passenger during past trips correspond to more feedback requests in the current trip). In some cases, machine learning methods can be used to determine the number and frequency of feedback requests displayed during a trip based on different groups or types of passengers. For example, passengers testing a vehicle can be expected to provide more feedback than paying passengers. Furthermore, different types, quantities, and qualities of feedback can be expected from passengers on their first ride (newer passengers), 10th ride, or 100th ride (experienced passengers). These frequency requirements can be associated with the triggering environment and are also stored in memories 130 and / or 330.

[0083] In some cases, passengers may “mute” or not request “this” additional feedback request, or not request additional feedback requests at all. For example, a passenger may make this change by selecting to mute feedback at the start of the trip via user input device 150 or from the passenger’s own client computing device (e.g., a mobile phone), indicating that the passenger does not want to be disturbed during the trip. Passengers may also use one of those channels to respond to specific feedback requests, thereby selecting to suspend said kind or type of feedback or to suspend feedback for the duration of the trip. Passengers may also be able to select feedback preferences across trips (e.g., in account settings, via the passenger’s client computing device, which allows it to be used during the trip but is not required, or via the vehicle’s user input device). This preference information may be sent to computing devices 110 and / or 310 to control the frequency of requests for the current trip (or even just complete silence). Alternatively, preference information may be sent to server computing devices, such as server computing device 410, to store preferences for future feedback requests along with the passenger’s account information. Furthermore, if a client computing device (e.g., a passenger's mobile phone) is used, preference information can be sent to the vehicle (e.g., via Bluetooth, Bluetooth Low Energy (LE), Near Field Communication, or other communication protocols), or it can be first sent to a server computing device (such as server computing device 410) and then forwarded by the server computing device to the vehicle's computing devices (e.g., computing devices 110 and / or 310).

[0084] The features described in this paper allow autonomous vehicles to collect passenger feedback in real time as events important to stakeholders are occurring. Therefore, the feedback can be more accurate and even more useful because it is not simply collected after the trip has ended. For example, a passenger will have just experienced the event and will therefore have a better memory of it than later in the trip. Furthermore, other negative or positive events that occur later in the ride will not affect the passenger's feedback on the current event. Additionally, some passengers may have a better experience if feedback is solicited multiple times during the ride compared to asking several questions simultaneously at the end of the ride (when passengers are unlikely to recall the answers correctly).

[0085] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the features described above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be understood by way of illustration rather than by way of limiting the subject matter defined by the claims. Furthermore, the provision of examples described herein and terms such as “e.g.,” “comprising,” etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, these examples are intended only to illustrate one of many possible embodiments. Moreover, the same reference numerals in different figures can identify the same or similar elements.

Claims

1. A method of collecting feedback from a passenger of an autonomous vehicle, the method comprising: determining, by one or more processors of a feedback system of the autonomous vehicle, that a triggering environment for triggering a feedback request has been met, the triggering environment comprising one or more of a driving event, a presence of other road users, or a trip status; based on the determining, identifying, by the one or more processors, a display requirement and data collection parameters for the feedback request, wherein, the display requirement defines a delay time between when the triggering environment has been met and when the feedback request is displayed, thereby defining when the feedback request is displayed, and the data collection parameters identify information to be collected by the feedback request; providing, by the one or more processors, the feedback request for display based on the display requirement and the data collection parameters; in response to the providing, receiving, by the one or more processors, feedback from the passenger of the autonomous vehicle; and storing, by the one or more processors, the feedback for subsequent use.

2. The method of claim 1, wherein, the driving event comprises a disengagement from an autonomous driving mode to a manual driving mode.

3. The method of claim 1, wherein, the presence of other road users comprises a predefined number of pedestrians.

4. The method of claim 1, wherein, the presence of other road users comprises a predefined number of cyclists.

5. The method of claim 1, wherein, the trip status comprises a time since the passenger boarded the vehicle.

6. The method of claim 1, wherein, the trip status comprises a time since the passenger exited the vehicle.

7. The method of claim 1, wherein, the triggering environment further comprises a rider lifecycle requirement.

8. The method of claim 7, wherein, the rider lifecycle requirement comprises whether the passenger is on a particular ride number.

9. The method of claim 7, wherein, the rider lifecycle requirement comprises whether the passenger has made a particular number of rides within a week.

10. The method of claim 7, wherein, the rider lifecycle requirement comprises whether the passenger has a predefined amount of time between rides.

11. The method of claim 7, wherein, the rider lifecycle requirement comprises a predefined number of weeks since a first ride.

12. The method of claim 1, wherein, the triggering environment further comprises a support interaction requirement.

13. The method of claim 1, wherein, the display requirement comprises a vehicle action that must occur before the feedback request is displayed.

14. The method of claim 1, further comprising: based on the determining, identifying a priority of the feedback request, wherein the priority corresponds to an inverse of a frequency at which the triggering environment is likely to occur.

15. The method of claim 14, further comprising: using the priority of the feedback request to determine when the feedback request is displayed.

16. The method of claim 1, wherein, the providing comprises providing a display on a display of the autonomous vehicle.

17. The method of claim 1, wherein, the providing comprises providing a display on a display of a client computing device associated with the passenger.

18. The method of claim 1, further comprising: based on the determining, identifying a frequency limit for the feedback request, and wherein the providing is further based on the frequency limit.

19. The method of claim 18, wherein, the frequency limit comprises a fixed maximum number of feedback requests displayed during a trip.

Images