Simulation of modified traffic for testing autonomous vehicle software

Abstract

The present disclosure relates to testing software for operating an autonomous vehicle. In one instance, a simulation can be run using log data collected by a vehicle (100, 100A, 100B) operating in an autonomous driving mode. The simulation can be run using software that controls the simulated vehicle and by modifying characteristics of a traffic body (such as a vehicle (620)) identified in the log data. During the running of the simulation, it can be determined that a first type of interaction between the first simulated vehicle and the modified traffic body (620') will occur. In response to determining that a particular type of interaction will occur, the modified traffic body can be replaced with an interactive traffic body (620") that simulates a road user corresponding to the modified traffic body, the interactive traffic body being capable of responding to actions performed by the simulated vehicle. It can be determined that a particular type of interaction between the simulated vehicle and the interactive traffic body has occurred in the simulation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of U.S. patent application Ser. No. 16 / 697,428, filed on Nov. 27, 2019, the entire disclosure of which is incorporated herein by reference. Technical Field

[0003] The present application relates to simulation with modified traffic for testing autonomous vehicle software. Background Art

[0004] Autonomous vehicles, such as those that do not require a human driver, can be used to help transport passengers or items from one location to another. Such vehicles can operate in a fully autonomous mode, where a passenger can provide some initial input, such as a pickup or destination location, and the vehicle maneuvers itself to that location, for example, by determining and following a route that may require the vehicle to respond to and interact with other road users (such as vehicles, pedestrians, cyclists, etc.). It is crucial that the autonomous control software used by these vehicles to operate in autonomous mode is tested and validated before such software is actually used to control the vehicle in areas where the vehicle is interacting with other objects. Summary of the Invention

[0005] One aspect of the present disclosure provides a method for testing software for operating a vehicle in an autonomous driving mode. The method includes: running a simulation by one or more processors by modifying characteristics of a traffic body identified in log data and using log data collected by the vehicle operating in the autonomous driving mode, wherein the simulation is run using software to control the simulated vehicle; during the simulation, determining by the one or more processors that a first type of interaction will occur between a first simulated vehicle and the modified traffic body; in response to determining that a first specific type of interaction will occur between the simulated vehicle and the modified traffic body, replacing the modified traffic body with an interactive traffic body simulating a road user corresponding to the modified traffic body, the interactive traffic body being capable of responding to actions performed by the simulated vehicle; and determining by the one or more processors that a second specific type of interaction has occurred between the simulated vehicle and the interactive traffic body in the simulation.

[0006] In one example, the first specific type of interaction is a collision. In this example, the second specific type of interaction is a collision, and the method further includes: in response to determining that a collision has occurred between the simulated vehicle and the interactive traffic body, marking the simulation for further review. Alternatively, the second specific type of interaction is one of the simulated vehicle making a sharp turn or the interactive traffic body making a sharp turn, and the method further includes: in response to determining that a collision has occurred between the simulated vehicle and the model traffic body, marking the simulation for further review. Alternatively, the second specific type of interaction is one of the simulated vehicle decelerating at a specific rate or the interactive traffic body decelerating at a specific rate, and the method further includes: in response to determining that a collision has occurred between the simulated vehicle and the interactive traffic body, marking the simulation for further review. In another example, the method further includes analyzing various log data to identify situations in which the autonomous vehicle is exhibiting a first type of behavior and the traffic body is exhibiting a second type of behavior. In this example, the first type of behavior is changing lanes, and the second type of behavior is passing the autonomous vehicle. Alternatively, the first type of behavior is making an unprotected turn, and the second type of behavior is exceeding the speed limit for the traffic body when approaching the autonomous vehicle in an oncoming lane. In another example, the characteristics include the size of the vehicle. In another example, the characteristics include the shape of the vehicle. In another example, the vehicle is a first type of road user, and the vehicle is modified so that the modified vehicle appears to the software as a second type of road user, different from the first type of road user. In this example, the first type of road user is a car, and the second type of road user is a motorcycle. Alternatively, the first type of road user is a car, and the second type of road user is a truck with specific characteristics. Alternatively, the first type of road user is a pedestrian, and the second type of road user is a cyclist. Alternatively, the first type of road user is a pedestrian, and the second type of road user is an animal. Alternatively, the first type of road user is a cyclist, and the second type of road user is a scooter rider. Alternatively, the first type of road user is a pedestrian with a shopping cart, and the second type of road user is a pedestrian with a stroller. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a functional diagram of an example vehicle according to an exemplary embodiment.

[0008] Figure 2 is an example of map information according to aspects of the present disclosure.

[0009] Figure 3 are example exterior views of a vehicle according to aspects of the present disclosure.

[0010] Figure 4 is a schematic diagram of an example system according to aspects of the present disclosure.

[0011] Figure 5 According to various aspects of the present disclosure Figure 4 Functional diagram of the system.

[0012] Figure 6 is an example representation of log data according to aspects of the present disclosure.

[0013] Figure 7 is another example representation of a simulation according to aspects of the present disclosure.

[0014] Figure 8 is another example representation of simulations and data according to aspects of the present disclosure.

[0015] Figure 9 is an example flow chart according to aspects of the present disclosure. DETAILED DESCRIPTION

[0016] Overview

[0017] This technology involves evaluating interactions in log-based simulations using software for autonomously operating vehicles. Log-based simulations are simulations run using log data collected from a vehicle operating in autonomous mode over a short period of time, such as one minute or more or less. The log data may include information from various vehicle systems, including perception, routing, planning, and localization. Simultaneously, the actual vehicle is replaced with a simulated vehicle that can make decisions using the software for autonomously controlling the vehicle. This allows for rigorous testing of the software. For example, simulations can be used to determine whether specific types of events or interactions with other traffic agents have occurred, such as specific types of behavior, collisions, or near-collisions. These events and interactions can be used for various purposes, such as determining whether the software can "pass" a given simulation without a collision or near-collision, and pinpointing potentially concerning behaviors and / or software modules to improve performance and safety, without requiring the vehicle to physically drive "realistic" miles or having to "create" situations in the real world.

[0018] However, when running simulations using log data, there may be a finite number of situations with certain characteristics. For example, it may be necessary to test software under "unusual" situations (for which there may be few examples). As an example, testing software in situations with a particular type of road user (such as a motorcyclist) may be difficult because there may be few examples of motorcycles in the vicinity of the vehicle for which the log data was recorded. Therefore, certain actions that the software may be able to cause the vehicle to perform (such as changing lanes or approaching a merge) may not have been adequately tested to confidently use these actions near motorcycles. As a result, it may be very difficult to verify the safety and effectiveness of a vehicle's software for such situations. To address these issues, more common road users or traffic bodies can be replaced by less common road users when running simulations.

[0019] To identify relevant log data, the log data can be analyzed to identify situations where the autonomous vehicle is exhibiting one or more specific types of behavior and another vehicle is exhibiting one or more specific types of behavior. Using the identified log data, a simulation can be run. In the simulation, the vehicle exhibiting the specific type of behavior for which log data was identified can be modified to provide a modified simulation.

[0020] During the modified simulation, the system can determine whether a trigger will be present to replace the modified traffic entity. The trigger for replacing the modified traffic entity can include some type of interaction between the simulated vehicle and the modified traffic entity. If not, the simulation will proceed without replacing the modified traffic entity with the interactive traffic entity. If present, the modified traffic entity can be replaced with a traffic entity that has certain responsiveness. As a result, the interactive traffic entity may be able to respond to the autonomous vehicle's actions in the modified simulation.

[0021] The modified simulation can be analyzed to identify specific types of interactions with other traffic, such as collisions or near collisions between the simulated vehicle and the interactive traffic. If no collisions or near collisions occur, the software can be considered to have "passed" the modified simulation, or the software can be considered validated for that particular modified simulation. If a collision or near collision with traffic is identified, the modified simulation can be annotated or marked for additional consideration, etc. The simulation can be analyzed to view measurements other than collision data (such as comfort, stranding, safety risk, etc.) to evaluate the software.

[0022] The features described herein can provide a safe, effective and realistic way to test the software for autonomous vehicles, using software to identify potential critical vulnerabilities (bugs) for abnormal situations. For example, the traffic body modified can be used to test software in hundreds of thousands of situations based on actual sensor data. In addition, those annotated or marked simulations may be more crucial for determining how to revise or update the tested software. In addition, by adding interactive traffic bodies, a "new" simulation that allows analysis of multiple potential safety issues can be created without requiring new log data for such simulations. Other benefits may include that software developers experiment with behavior changes and use them to show the ability of many realistic examples of how their changes progress (playout). This can allow much faster feature development because alternatives will involve developers screening a large number of pseudo-failures themselves, waiting for operators to do the same thing, or driving realistic vehicles around with behavior changes to log the response of reality.

[0023] Example System

[0024] like Figure 1 As shown, a vehicle 100 according to one aspect of the present disclosure includes various components. Although certain aspects of the present disclosure are particularly useful in conjunction with specific types of vehicles, the vehicle can be any type of vehicle, including but not limited to cars, trucks, motorcycles, buses, recreational vehicles, etc. The vehicle can have one or more computing devices, such as a computing device 110 that includes one or more processors 120, memory 130, and other components typically found in a general-purpose computing device.

[0025] Memory 130 stores information accessible by one or more processors 120, including instructions 134 and data 132 that can be executed or otherwise used by processor 120. Memory 130 can be any type of memory capable of storing information accessible by a processor, including computing device-readable media, or other media that stores data that can be read by an electronic device, such as a hard drive, memory card, ROM, RAM, DVD or other optical disk, and other writable and read-only memory. Systems and methods may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.

[0026] Instructions 134 may be any set of instructions that are executed by a processor directly (such as machine code) or indirectly (such as a script). For example, the instructions may be stored as computing device code on a computing device readable medium. In this regard, the terms "software," "instructions," and "program" are used interchangeably herein. The instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language (including a collection of independent source code modules or scripts that are interpreted on demand or pre-compiled). The functions, methods, and routines of the instructions are explained in more detail below.

[0027] Processor 120 may retrieve, store, or modify data 132 according to instructions 134. For example, although the claimed subject matter is not limited to any particular data structure, the data may be stored in a computing device register, in a relational database as a table with multiple different fields and records, in an XML document, or in a flat file. The data may also be formatted in any computing device readable format.

[0028] The one or more processors 120 may be any conventional processor, such as a commercial CPU. Alternatively, the one or more processors may be a dedicated device, such as an ASIC or other hardware-based processor. Figure 1 The processor, memory, and other elements of computing device 110 are functionally shown as being within the same block, but one of ordinary skill in the art will appreciate that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be housed within the same physical housing. For example, the memory may be a hard drive or other storage medium located in a separate housing from the housing of computing device 110. Thus, references to a processor or computing device will be understood to include references to a collection of processors, computing devices, or memories that may or may not operate in parallel.

[0029] The computing device 110 may include all components typically used in conjunction with a computing device, such as the processor and memory described above, as well as user input 150 (e.g., a mouse, keyboard, touch screen, and / or microphone) and various electronic displays (e.g., a monitor with a screen or any other electrical device operable to display information). In this example, the vehicle includes an interior electronic display 152 and one or more speakers 154 to provide information or an audio-visual experience. In this regard, the interior electronic display 152 may be located within the cabin of the vehicle 100 and may be used by the computing device 110 to provide information to passengers within the vehicle 100.

[0030] The computing device 110 may also include one or more wireless network connections 156 to facilitate communication with other computing devices (such as client computing devices and server computing devices described in detail below). The wireless network connections may include short-range communication protocols such as Bluetooth, Bluetooth Low Energy (LE), cellular connections, and various configurations and protocols including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more company-specific communication protocols, Ethernet, WiFi, and HTTP, as well as various combinations of the foregoing.

[0031] In one example, computing device 110 may be a control computing device of an autonomous driving computing system or incorporated into vehicle 100. The autonomous driving computing system may be capable of communicating with various components of the vehicle to control the movement of vehicle 100 according to autonomous control software of memory 130 as discussed further below. For example, Figure 1 The computing device 110 can communicate with various systems of the vehicle 100 to control the movement, speed, etc. of the vehicle 100 according to the instructions 134 of the memory 130, such as the deceleration system 160, the acceleration system 162, the steering system 164, the signal system 166, the routing system 168, the positioning system 170, the perception system 172, and the power system 174 (i.e., the engine or motor of the vehicle). Similarly, although these systems are shown as being external to the computing device 110, in reality, these systems can also be incorporated into the computing device 110, also serving as an autonomous driving computing system for controlling the vehicle 100.

[0032] As an example, computing device 110 can interact with one or more actuators of deceleration system 160 and / or acceleration system 162 (such as the vehicle's brakes, accelerator pedal, and / or engine or motor) to control the vehicle's speed. Similarly, computing device 110 can use one or more actuators of steering system 164 (such as a steering wheel, steering shaft, and / or a rack and pinion system) to control the direction of vehicle 100. For example, if vehicle 100 is configured for on-road use, such as a car or truck, the steering system may include one or more actuators to control the angle of the wheels to turn the vehicle. Computing device 110 can use signaling system 166 to signal the vehicle's intentions to other drivers or vehicles (e.g., by illuminating a turn signal or brake light, if necessary).

[0033] Computing device 110 may use routing system 168 to determine and follow a route to a location. In this regard, routing system 168 and / or data 132 may store detailed map information, such as a highly detailed map that identifies roads, lane markings, intersections, crosswalks, speed limits, traffic signals, buildings, signs, real-time traffic information, the shape and height of vegetation, or other such objects and information.

[0034] Figure 2 202, 204. In this example, map information 200 includes information on the shape, location, and other characteristics of lane markings 210, 212, 214, traffic lights 220, 222, crosswalks 230, sidewalks 240, stop signs 250, 252, and yield signs 260. Although map information is described herein as an image-based map, map information need not be fully image-based (e.g., raster). For example, map information may include one or more road graphs or graph networks of information such as roads, lanes, intersections, and the connections between these features. Each feature may be stored as graph data and may be associated with information such as a geographic location and whether it is linked to other related features (e.g., a stop sign may be linked to a road and intersection, etc.). In some examples, associated data may include a grid-based index of a road graph to allow efficient search for certain road graph features.

[0035] The computing device 110 may use the positioning system 170 to determine the relative or absolute position of the vehicle on a map or on the Earth. For example, the positioning system 170 may include a GPS receiver to determine the latitude, longitude, and / or altitude position of the device. Other positioning systems, such as laser-based positioning systems, inertial assisted GPS, or camera-based positioning, may also be used to identify the location of the vehicle. The vehicle's location may include absolute geographic location information, such as latitude, longitude, and altitude, as well as relative location information, such as relative to other cars immediately surrounding it, which can generally be determined with less noise than the absolute geographic location.

[0036] The positioning system 170 may also include other devices (such as accelerometers, gyroscopes, or other direction / speed detection devices) in communication with the computing device 110 to determine the vehicle's direction and speed, or changes thereto. By way of example only, an acceleration device may determine its pitch, yaw, or roll (or changes thereto) relative to the direction of gravity or relative to a plane perpendicular to the direction of gravity. The device may also track increases or decreases in speed and the direction of such changes. Provision of position and orientation data by a device as described herein may be automatically provided to the computing device 110, other computing devices, and combinations thereof.

[0037] The perception system 172 also includes one or more components for detecting objects outside the vehicle, such as other vehicles, obstacles in the road, traffic signals, signs, trees, etc. For example, the perception system 172 may include lasers, sonar, radar, cameras, and / or any other detection devices that record data that can be processed by the computing device 110. In the case where the vehicle is a passenger vehicle such as a minivan, the minivan may include lasers or other sensors mounted on the roof or other convenient location. For example, Figure 3 is an example exterior view of vehicle 100. In this example, roof-top housing 310 and dome housing 312 can include LIDAR (Light Detection and Ranging) sensors and various cameras and radar units. In addition, housing 320 located at the front of vehicle 100 and housings 330, 332 on the driver's and passenger sides of the vehicle can each house a LIDAR sensor. For example, housing 330 is located in front of driver's door 350. Vehicle 100 also includes housings 340, 342 for radar units and / or cameras, which are also located on the roof of vehicle 100. Additional radar units and cameras (not shown) can be located at the front and rear ends of vehicle 100 and / or at other locations along the roof or roof housing 310.

[0038] Computing device 110 can control the direction and speed of the vehicle by controlling various components. As an example, computing device 110 can use data from detailed map information and routing system 168 to fully autonomously navigate the vehicle to a destination location. Computing device 110 can use positioning system 170 to determine the vehicle's location and perception system 172 to detect and, if necessary, respond to objects to safely reach that location. To do so, computing device 110 can accelerate the vehicle (e.g., by increasing fuel or other energy supplied to the engine by acceleration system 162), decelerate the vehicle (e.g., by reducing fuel supplied to the engine, shifting gears, and / or by applying brakes by deceleration system 160), change direction (e.g., by turning the front or rear wheels of vehicle 100 by steering system 164), and signal such changes (e.g., by illuminating a turn signal from signal system 166). Thus, acceleration system 162 and deceleration system 160 can be part of a drivetrain system that includes various components between the vehicle's engine and the vehicle's wheels. Similarly, by controlling these systems, computing device 110 can also control the vehicle's drivetrain system to autonomously maneuver the vehicle.

[0039] The computing device 110 of the vehicle 100 may also receive information from or transmit information to other computing devices, such as those that are part of the transportation service and other computing devices. Figure 4 and Figure 5 4 and 5 are schematic and functional diagrams, respectively, of an example system 400 including a plurality of computing devices 410, 420, 430, 440 and a storage system 450 connected via a network 460. System 400 also includes a vehicle 100 and vehicles 100A, 100B that may be configured the same as or similar to vehicle 100. Although only a few vehicles and computing devices are depicted for simplicity, a typical system may include significantly more vehicles and computing devices.

[0040] like Figure 4 As shown, each of computing devices 410, 420, 430, 440 may include one or more processors, memory, data, and instructions. Such processors, memory, data, and instructions may be configured similarly to the one or more processors 120, memory 130, data 132, and instructions 134 of computing device 110.

[0041] The network 460 and intermediary nodes may include various configurations and protocols, including short-range communication protocols such as Bluetooth, Bluetooth LE, the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more company-specific communication protocols, Ethernet, WiFi, and HTTP, as well as various combinations of the foregoing. Such communications may be facilitated by any device capable of transmitting data to and receiving data from other computing devices, such as a modem and a wireless interface.

[0042] In one example, the one or more computing devices 410 may include one or more server computing devices (e.g., a load balancing server farm) having multiple computing devices that exchange information with different nodes of a network for the purpose of receiving data from, processing data from, and transmitting data to other computing devices. For example, the one or more computing devices 410 may include one or more server computing devices capable of communicating with the computing device 110 of vehicle 100 or a similar computing device of vehicle 100A, as well as computing devices 420, 430, 440, via network 460. For example, vehicles 100, 100A may be part of a fleet that can be dispatched to various locations by server computing devices. In this regard, the server computing device 410 may be used as a verification computing system that can be used to verify that vehicles such as vehicle 100 and vehicle 100A can be used for autonomous control software operating in an autonomous driving mode. Furthermore, server computing device 410 may transmit information to users (such as users 422, 432, 442) using network 460 and present information to users (such as users 422, 432, 442) on displays (such as displays 424, 434, 444 of computing devices 420, 430, 440). In this regard, computing devices 420, 430, 440 may be considered client computing devices.

[0043] like Figure 4 As shown, each client computing device 420, 430, 440 can be a personal computing device intended for use by a user 422, 432, 442 and has all of the components typically used in conjunction with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory (e.g., RAM and an internal hard drive) to store data and instructions, a display such as a display 424, 434, 444 (e.g., a monitor with a screen, a touch screen, a projector, a television, or other device operable to display information), and a user input device 426, 436, 446 (e.g., a mouse, keyboard, touch screen, or microphone). The client computing device may also include a camera for recording a video stream, speakers, a network interface device, and all of the components used to connect these elements to each other.

[0044] Although each of client computing devices 420, 430, and 440 may comprise a full-size personal computing device, they may alternatively comprise a mobile computing device capable of wirelessly exchanging data with a server over a network such as the Internet. By way of example only, client computing device 420 may be a mobile phone, or a device capable of obtaining information via the Internet or other network, such as a wireless-enabled PDA, tablet PC, wearable computing device or system, or netbook. In another example, client computing device 430 may be a wearable computing system, such as a Figure 4 As examples, a user can enter information using a keypad, a keypad, a microphone, visual signals using a camera, or a touch screen.

[0045] In some examples, the client computing device 440 may be an operations workstation used by an administrator or operator to view scenario output, switch times, and verification information as discussed further below. Figure 4 and Figure 5 Only a single operator workstation 440 is shown in FIG, but any number of such workstations may be included in a typical system. In addition, although the operator workstation is depicted as a desktop computer, the operator workstation may include various types of personal computing devices, such as laptop computers, netbooks, tablet computers, etc.

[0046] As with memory 130, storage system 450 may be any type of computerized storage capable of storing information accessible by server computing device 410, such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, writable and read-only memory. In addition, storage system 450 may include a distributed storage system in which data is stored on multiple different storage devices that may be physically located in the same or different geographic locations. Figure 4 and Figure 5 As shown, storage system 450 may be connected to the computing device via network 460 and / or may be directly connected to or incorporated into any of computing devices 110 , 410 , 420 , 430 , 440 , etc.

[0047] The storage system 450 can store various types of information, as described in more detail below. A server computing device (such as one or more server computing devices 410) can retrieve or otherwise access this information in order to perform some or all of the features described herein. For example, the storage system 450 can store log data. The log data can include, for example, sensor data generated by a perception system (such as the perception system 172 of the vehicle 100). As an example, the sensor data can include raw sensor data as well as data identifying characteristics of objects that define perception (such as the shape, position, orientation, speed, etc. of objects such as vehicles, pedestrians, cyclists, vegetation, curbs, lane markings, sidewalks, crosswalks, buildings, etc.). The log data may also include "event" data identifying different types of events (such as collisions or near collisions with other objects), projected trajectories describing the planned geometry and / or velocity of potential paths for the vehicle 100, 100A, the actual position of the vehicle at different times, the actual heading / direction of the vehicle at different times, the actual speed, acceleration, and deceleration of the vehicle at different times, classifications of sensed objects and responses to sensed objects, predictions of the behavior of sensed objects, the states of the vehicle's various systems (such as acceleration, deceleration, perception, steering, signaling, routing, powertrain, etc.) at different times (including logged errors), inputs and outputs of the vehicle's various systems at different times, etc. Thus, these events and sensor data can be used to "recreate" the vehicle's environment, including the sensed objects, and the vehicle's behavior in simulation. In some cases, the log data can be annotated with information identifying the behavior of the autonomous vehicle (such as passing, changing lanes, merging, etc.) and with information identifying the behavior of other traffic entities in the log data (such as passing or overtaking the autonomous vehicle, changing lanes, merging, etc.).

[0048] The storage system may also store interactive traffic entities or data and instructions that can be used to generate simulated road users to interact with virtual vehicles in the simulation. Because there are different types of road users, there may be different types of interactive traffic entities. For example, there may be interactive traffic entities for vehicles (or specific types of vehicles, such as autonomous vehicles, buses, vans, cars, trucks, motorcycles, emergency vehicles (e.g., police cars, ambulances, etc.), and other larger vehicles), as well as non-vehicles (e.g., pedestrians, groups of pedestrians, pedestrians with strollers, children, scooters, wildlife, and pets). Because humans are generally unpredictable, models can be generated by establishing a set of characteristics. These may relate to, for example, reaction times for responding to visual or auditory stimuli by moving a foot or hand to change a vehicle's braking, acceleration, and / or steering behavior, as a human driver, pedestrian, or cyclist would. In other words, the model may include models of how an ideal, average, or below-average human would brake or swerve, as derived from existing human reaction research. In this regard, the model may be approximate and manually tuned, and may respond in a more predictable manner than a typical human driver. In some instances, the model may also have behavioral rules, such as how a typical driver would behave in a 4-way stop or respond to children in the environment.

[0049] In addition, the storage system 450 may also store autonomous control software to be used by a vehicle, such as the vehicle 100, to operate the vehicle in an autonomous driving mode. The autonomous control software stored in the storage system 450 may be a version that has not yet been tested or verified. Once verified, the autonomous control software may be sent, for example, to the memory 130 of the vehicle 100 for use by the computing device 110 to control the vehicle 100 in the autonomous driving mode.

[0050] Example Method

[0051] In addition to the operations described above and shown in the figures, various operations will now be described. It should be understood that the following operations do not have to be performed in the exact order described below. Instead, the various steps can be processed in a different order or simultaneously, and steps can also be added or omitted.

[0052] To test and / or validate the autonomous control software to be stored in memory 130 for use by computing device 110 of vehicle 100, server computing device 410 can run various simulations. These simulations can be log-based simulations generated from the information in the aforementioned log data stored in storage system 450. In this regard, server computing device 410 can access storage system 450 to retrieve the log data and run the simulations. For example, a portion of real-time one-minute log data corresponding to the autonomous vehicle that generated the log data can be retrieved from the storage system. This portion of log data can be "manually" selected by a human operator and / or computing device based on the type of events recorded in the log or more randomly (e.g., by selecting 1% or more or less of all autonomous driving logs).

[0053] As mentioned above, when running simulations using log data, there may be a limited number of scenarios with certain characteristics. For example, there may be a need to test software in "outlier" situations (for which there may be very few examples). As an example, testing software in situations with a specific type of road user (such as a motorcyclist) may be difficult because there may be very few examples of motorcycles near the vehicle for which the log data was recorded. Therefore, certain behaviors that the software may be able to cause the vehicle to perform (such as changing lanes or merging nearby) may not be sufficiently tested to confidently use these behaviors near motorcycles. As a result, verifying the safety and effectiveness of the vehicle's software in such situations may be very difficult. To address these issues, more common road users or traffic bodies can be replaced with less common road users when running simulations. Examples of such less common road users can include, for example, autonomous vehicles, motorcyclists, cyclists, emergency vehicles, trucks, or other larger vehicles. Other non-vehicle examples of less common road users can include, for example, pedestrians, groups of pedestrians, pedestrians with strollers, children, scooters, wild animals, and pets.

[0054] To identify relevant log data, the log data in storage system 450 can be analyzed to identify situations in which the autonomous vehicle is exhibiting one or more specific types of behavior and another object of traffic is exhibiting one or more specific types of behavior. For example, as described above, the log data can be annotated with information identifying the behavior of the autonomous vehicle (such as passing, changing lanes, merging, etc.) and information identifying the behavior of the other object of traffic in the log data (such as passing or passing the autonomous vehicle, changing lanes, merging, etc.). By way of example, using this identification information, log data can be identified for situations in which the autonomous vehicle intends to make a lane change and another object (in the same or a different lane as the autonomous vehicle) intends to pass the autonomous vehicle, or other situations in which an object cuts into the lane in front of, to the side of, or behind the autonomous vehicle, or in unprotected turns (left turns, right turns, U-turns, etc.). Other situations for other types of objects (such as pedestrians in a crosswalk or potentially turning into a crowd in or outside the crosswalk) may also be useful.

[0055] As an example, log data may be identified where traffic did not stop at a stop sign at an intersection and the autonomous vehicle is making a left turn. Figure 6 , provides an example of log data 600 corresponding to a road segment of map information 200. In this example, intersections 602 and 604 correspond to intersections 202 and 204, respectively. In this regard, the shapes, positions, and other characteristics of lane markings 210, 612, 614, traffic lights 616, 616, crosswalk 630, sidewalk 640, stop signs 650, 652, and yield sign 660 correspond to the shapes, positions, and other characteristics of lane markings 210, 212, 214, traffic lights 220, 222, crosswalk 230, sidewalk 240, stop signs 250, 252, and yield sign 260. In log data 600, vehicle 100 is approaching intersection 604, for example, to make a left turn. Traffic vehicles 620, 622, 624, 626, generated from event data and / or sensor data from the log data used for simulation, are also approaching or crossing intersection 604. In this example, the log data may indicate that vehicle 620 (depicted as a car) did not stop at stop sign 650, and therefore, this log data may be identified. Of course, any number of other types of simulations may be identified, depending on what type of scenario the human operator is interested in using for testing purposes. As another example, log data may be provided where the autonomous vehicle is making an unprotected turn (e.g., an unprotected left turn) and traffic is speeding (i.e., exceeding the posted speed limit for the lane in which the second vehicle is traveling) while approaching the autonomous vehicle from an oncoming lane (e.g., opposing traffic).

[0056] Figure 9An example flowchart 900 includes some examples for testing software for controlling a vehicle in autonomous driving mode, which can be executed by one or more processors, such as the processor of computing device 410. For example, at block 910, a simulation is run by modifying characteristics of traffic bodies identified in log data and using log data collected by a vehicle operating in autonomous driving mode. In this example, the simulation is run using software to control a simulated vehicle. For example, a simulation can be run using the identified log data. A portion of the retrieved log data can be used to run an initial simulation. When the autonomous control software is run using the portion of the log data, the details of the log data (sensor data and events) can be used to generate the simulation. In other words, the sensor data from the portion of the log data can be simply "played" as input to the perception system 172 of the simulated vehicle controlled by the autonomous control software. In this regard, the autonomous control software "experiences" or processes the log data as if the autonomous control software were actually running on vehicle 100. In other words, the simulation may include data defining characteristics of objects (such as the shape, position, orientation, speed, etc. of objects defined by the sensor data of the log data, such as vehicles, pedestrians, cyclists, vegetation, curbs, lane markings, sidewalks, crosswalks, buildings, etc.). Furthermore, the simulation may include characteristics of a virtual vehicle corresponding to vehicle 100, including the shape, position, orientation, speed, etc. of the virtual vehicle defined by the events of the log data.

[0057] However, in the simulation, the traffic object for which log data is identified and for which a specific type of behavior is being attempted can be modified to provide a modified simulation. For example, characteristics of the traffic object (such as its size (e.g., dimensions) and shape) can be adjusted to make the traffic object appear more like a specific type of traffic object to the simulated vehicle (or, more specifically, to the software being used in the simulation). In other words, the software will "perceive" the modified traffic object as if it were a different type of traffic object. This can involve generating sensor data for the modified traffic object and providing that sensor data to the software. For example, a vehicle can be modified to appear more like another vehicle, such as a specific type of autonomous vehicle, a bus, a van, a car, a truck, a motorcycle, an emergency vehicle (e.g., a police car, an ambulance, etc.), or other larger vehicles, as well as non-vehicles such as pedestrians, groups of pedestrians, pedestrians with strollers, children, scooters, wild animals, and pets. In yet other examples, vehicles such as cars can be modified to appear as trucks with specific size and / or shape characteristics (e.g., dump trucks, garbage trucks, tractor-trailers, etc.), pedestrians can be modified to appear as cyclists, cyclists can be modified to appear as scooters, pedestrians with shopping carts can be modified to appear as pedestrians with strollers, and so on.

[0058] Figure 7 An example of a simulation 700 run using log data from a modified version of example 600 is provided. In this example, simulated vehicle 770 can correspond to vehicle 100 or vehicle 100A, and software from storage system 450 can be used to determine how the simulated vehicle will behave in the simulation. Furthermore, vehicle 620 is replaced by modified traffic body 620′. In this example, the size and shape of vehicle 620 are modified so that modified traffic body 620′ behaves as a motorcyclist exhibiting the behavior of vehicle 620. Of course, as described above, the modified traffic body can correspond to any number of different types of road users, such as autonomous vehicles, buses, vans, cars, trucks, motorcycles, emergency vehicles (e.g., police cars, ambulances, etc.), and other larger vehicles, as well as non-vehicles such as pedestrians, groups of pedestrians, pedestrians with strollers, children, scooters, wildlife, and pets.

[0059] During the modified simulation, the server computing device 410 may determine whether there will be a trigger for replacing the modified traffic body with the interactive traffic body of the storage system 450. The trigger for replacing the modified traffic body may include some type of interaction between the simulated vehicle and the modified traffic body. Figure 9 At block 920, during the execution of the simulation, it is determined that a first type of interaction will occur between the first simulated vehicle and the modified traffic entity. Thereafter, at block 930, in response to determining that the first type of interaction will occur between the simulated vehicle and the modified traffic entity, the modified traffic entity is replaced with an interactive traffic entity simulating a road user corresponding to the modified traffic entity, the interactive traffic entity being capable of responding to actions performed by the simulated vehicle.

[0060] For example, the first type of interaction may include a collision or near collision. In other words, if there is a collision (or near collision) between the autonomous vehicle (or more precisely, the simulated vehicle) and the modified traffic body, this may be a trigger to replace the modified traffic body with the interactive traffic body stored in the storage system 450. This can be done by looking at the position and orientation of the simulated vehicle 770 and the position and orientation of the modified traffic body 620' and inferring whether these objects will intersect with each other at some point during the simulation. For example, this may include calculating whether a given trajectory of the modified traffic body and the trajectory generated by the software for the vehicle will intersect with each other (or be within a small distance, i.e., close). If not, the simulation will be run without replacing the modified traffic body with the interactive traffic body.

[0061] Likewise, if a collision or near collision is expected in the simulation, the modified traffic body may be replaced by a traffic body (e.g., an interactive traffic body) having some responsiveness by the server computing device 410. Thus, the interactive traffic body may be able to respond to the behavior of the simulated vehicle (e.g., vehicle 770) in the simulation. Figure 8 7 shows a comparison of the trajectories of simulation 700. In this example, the modified traffic body's trajectory 820 intersects the trajectory 870 generated by the software for the vehicle at point 810. Consequently, the server computing device 410 can automatically replace the modified traffic body 620' with the interactive traffic body 620" and continue the simulation. In this regard, the replacement occurs during the simulation. Furthermore, the interactive traffic body will have the physical and other characteristics of the modified traffic body, but the interactive traffic body 620" may now be able to respond to the behavior of the simulated vehicle 770 in the simulation.

[0062] In some examples, interactive traffic bodies can be given certain characteristics by the server computing device 410, such as increased or decreased aggressiveness. For example, more aggressive interactive traffic bodies may be more likely to get closer to the autonomous vehicle or change lanes in a shorter period of time than less aggressive interactive traffic bodies. In this regard, many different simulations can be run with different levels of aggressiveness of interactive traffic bodies. As a result, multiple different types of simulations can be created from the logged data of a single simulation. Furthermore, because the interactive traffic bodies are already adjusted traffic bodies, it is possible to generate situations that would otherwise be rare in the logged data for testing the software.

[0063] In some instances, when running a simulation or a modified simulation, all traffic entities in the log data can be replaced by interactive traffic entities by server computing device 410. For example, referring to simulation 700, all traffic entities, vehicles 622, 624, and 626, except for modified traffic entity 620', can be replaced by interactive traffic entities from storage system 450. However, in such a case, because the interactive traffic entities can be perfectly perceived by the simulated vehicles (i.e., having sharp, well-defined edges and boundaries), certain issues with the software's perception system (i.e., software corresponding to perception system 172) may be missed. Furthermore, using more data from the log data may result in higher fidelity of realism than using less data from the log data. In other words, depending on the interactive traffic entity, an aggressive driver may become more polite, or vice versa. Therefore, in order to make the simulation as realistic as possible, other methods can be used.

[0064] The server computing device 410 may analyze the results of the simulation to identify whether a particular type of interaction has occurred. In this regard, the server computing device may be able to determine whether the software was able to complete the simulation without the particular type of interaction occurring between the simulated vehicle and the interactive traffic body. Figure 9 At block 940 , a determination is made as to whether a second type of interaction has occurred between the simulated vehicle and the interactive traffic entity during the simulation. For example, specific types of maneuvers may include a collision or near collision between the simulated vehicle and the interactive traffic entity, as well as specific types of maneuvers (e.g., a sharp turn by the simulated vehicle that causes the interactive traffic entity to turn sharply or hard brake, hard braking or a specific rate of deceleration, risky behavior by the interactive traffic entity that causes the simulated vehicle to turn sharply or hard brake, etc.). If no collision or near collision occurs, the software may be considered to have “passed” the modified simulation, or the software may be considered validated for that particular modified simulation. If a collision or near collision with the traffic entity is identified, this may indicate that the software will not complete the modified simulation if a collision or near collision has not occurred. Therefore, the modified simulation may be annotated or flagged for additional consideration, etc. In addition to or as an alternative to collisions or near collisions and other interactions, the simulation may be analyzed to view other measures (such as comfort, stranding, safety risk, etc.) to evaluate the software.

[0065] While the examples herein involve modifying and replacing only one road user, the features described herein can be applied to multiple road users within the same simulation. For example, log data from storage system 450 can be analyzed to identify situations in which the autonomous vehicle is exhibiting one or more specific types of behavior while two or more other traffic entities are exhibiting one or more specific types of behavior. As an example, log data can be analyzed to identify situations in which the autonomous vehicle is making an unprotected left turn, a second vehicle approaching the autonomous vehicle from the oncoming lane is speeding (i.e., exceeding the posted speed limit for the lane in which the second vehicle is traveling), and a third vehicle (e.g., a truck) is blocking the autonomous vehicle's view of the second vehicle. One or both of the second and third vehicles can then be modified. For example, the second vehicle can be modified to appear as a motorcycle, and the third vehicle can be modified to appear as an even larger truck, thereby making the simulation more complex. For any of those modified second and / or third vehicles whose trajectories intersect with the trajectory of the autonomous vehicle, those second and / or third vehicles can be replaced with interactive traffic entities as described above. As another example, log data can be analyzed to identify situations in which the autonomous vehicle is approaching multiple jaywalkers. When running a simulation, the jaywalker can be modified to appear as another road user, such as a cyclist, a scooter rider, an animal, or another type of road user capable of jaywalking. For any of those modified other road users that have a trajectory that intersects the vehicle's trajectory, those other road users can be replaced with interactive traffic bodies as described above.

[0066] Although the examples herein rely on simulations based on log data collected from real vehicles, simulations may be generated from log data from previous simulations, or from simulations that may not even actually include such real-life driving data.

[0067] The features described herein can provide a safe, effective and realistic way to test software for autonomous vehicles, using software to identify potentially critical vulnerabilities for abnormal situations. For example, software can be tested in hundreds of thousands of situations using modified traffic bodies based on actual sensor data. In addition, those annotated or marked simulations may be more crucial for determining how to revise or update the software being tested. In addition, by adding interactive traffic bodies, "new" simulations that allow analysis of multiple potential safety issues can be created without requiring new log data for such simulations. Other benefits may include the ability of software developers to experiment with behavior changes and use them to show how many real-life examples of how their changes progress. This can allow much faster feature development because alternatives will involve developers screening a large number of pseudo-failures themselves, waiting for operators to do the same thing, or driving real vehicles around with behavior changes to log realistic responses.

[0068] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the above-described features can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be made by way of illustration, rather than by way of limitation, to the subject matter defined by the claims. Furthermore, the provision of examples described herein, as well as clauses worded “such as,” “including,” etc., should not be construed as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only one of a variety of possible embodiments. Furthermore, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A method of testing software for operating a vehicle in an autonomous driving mode, the method comprising: modifying, by one or more processors, a traffic body identified using log data collected by a vehicle operating in an autonomous driving mode, wherein the modified traffic body is perceived by the software as a different type of traffic body to the software than before the traffic body was modified; generating, by the one or more processors, modified sensor data of the traffic body; providing sensor data to the software by the one or more processors; running a first simulation using software by the one or more processors to control a simulated vehicle; During execution of the first simulation, determining, by the one or more processors, whether a collision or near collision has occurred between the simulated vehicle and the modified body of traffic; In response to determining that a collision or near collision has occurred between the simulated vehicle and the modified traffic entity, executing, by the one or more processors, a second simulation after replacing the modified traffic entity with an interactive traffic entity simulating a road user corresponding to the modified traffic entity, the interactive traffic entity being responsive to actions performed by the simulated vehicle; and During execution of the second simulation, a determination is made by the one or more processors as to whether a collision or near collision has occurred between the simulated vehicle and the interactive traffic volume.

2. The method of claim 1, further comprising: In response to determining that a collision or near collision between the simulated vehicle and the modified body of traffic has not occurred, the second simulation is validated.

3. The method of claim 1 , further comprising: In response to determining that a collision or near collision has occurred between the simulated vehicle and the interactive traffic body, the second simulation is marked for further review.

4. The method according to claim 3, wherein: The collision or near-collision includes a simulated sharp turn of the vehicle or a simulated sharp turn of the interactive traffic body.

5. The method according to claim 3, wherein: The collision or near collision includes a simulated vehicle decelerating at a specific rate or an interactive traffic body decelerating at a specific rate. 6 . The method of claim 1 , further comprising identifying the log data by analyzing the various log data to identify situations where the vehicle operating in the autonomous driving mode is exhibiting a first type of behavior and the body of traffic is exhibiting a second type of behavior.

7. The method according to claim 6, wherein: The first type of action is changing lanes.

8. The method of claim 7, wherein: The second type of activity is passing vehicles operating in autonomous driving mode.

9. The method of claim 6, wherein: The first type of behavior is to make an unprotected turn.

10. The method of claim 9, wherein: The second type of behavior is exceeding the speed limit for a body of traffic when approaching a vehicle operating in autonomous driving mode from an oncoming lane.

11. The method of claim 1, wherein: The characteristics of the traffic body include the size of the traffic body.

12. The method of claim 1, wherein: The characteristics of the traffic body include the shape of the traffic body.

13. The method of claim 1, wherein: The traffic entity is a first type of road user, and the traffic entity is modified so that the modified traffic entity appears to the software as a second type of road user different from the first type of road user.

14. The method of claim 13, wherein: The first type of road user is cars, and the second type of road user is motorcycles.

15. The method of claim 13, wherein: The first type of road user is cars, and the second type of road user is trucks having specific characteristics.

16. The method of claim 13, wherein: The first type of road user is pedestrians and the second type of road user is cyclists.

17. The method of claim 13, wherein: The first type of road users are pedestrians, and the second type of road users are animals.

18. The method of claim 13, wherein: The first type of road user is cyclists and the second type of road user is scooter riders.

19. The method of claim 13, wherein: The first type of road user is a pedestrian with a shopping cart, and the second type of road user is a pedestrian with a baby stroller.

20. A system for testing software for operating a vehicle in an autonomous driving mode, the system comprising: One or more processors, wherein the one or more processors are configured to: running a first simulation by modifying characteristics of a traffic body identified by using log data collected by a vehicle operating in an autonomous driving mode, wherein the first simulation is run using software to control the simulated vehicle; During the running of the first simulation, determining whether a collision or near collision has occurred between the simulated vehicle and the modified traffic body; In response to determining that a collision or near collision has occurred between the simulated vehicle and the modified traffic entity, running a second simulation after replacing the modified traffic entity with an interactive traffic entity simulating a road user corresponding to the modified traffic entity, the interactive traffic entity being responsive to actions performed by the simulated vehicle; and During execution of the second simulation, it is determined whether a collision or near collision has occurred between the simulated vehicle and the interactive traffic volume.

Images