Selective intelligent agent in simulation

Abstract

Evaluating performance of an autonomous vehicle includes: determining, from log data, a simulation scenario including a non-AV actor that is caused to selectively deviate from its recorded trajectory during the simulation; performing the simulation based on the simulation scenario; monitoring the performance of the simulation; based on the monitoring, determining whether a behavior switch condition for the non-AV actor associated with a traffic interaction involving the AV is satisfied during the simulation; in response to determining that the behavior switch condition for the non-AV actor associated with the traffic interaction involving the AV is satisfied during the simulation, modifying the behavior of the non-AV actor using a first behavior model to deviate from its recorded trajectory; and determining a performance metric based on the simulation.

Description

Background Technology

[0001] Autonomous vehicle technology faces a challenge in evaluating the performance of different subsystems of autonomous vehicles under various real-world driving environments. In practice, different subsystems of autonomous vehicles can be evaluated on multiple log-based simulation scenarios that attempt to simulate the real world. For example, the perception or planning subsystems of an autonomous vehicle can be evaluated on simulated scenarios to determine whether the autonomous vehicle navigates appropriately in the environment. There has long been a need for a technique to evaluate the performance of autonomous vehicles in simulated scenarios where the actor can selectively adapt to new environmental changes that occur during the simulation. Summary of the Invention

[0002] This disclosure describes a technique for evaluating the performance of autonomous vehicles in simulated scenarios using intelligent actors. For example, the intelligent actor can be one that is enabled to selectively adjust its responsiveness to new environmental changes occurring during the simulation. An existing method for evaluating the performance of autonomous vehicles is to test their behavior in simulated scenarios using scripted actors. The scripted actors follow an original trajectory (recorded trajectory) observed in log files collected from real-world driving. While scripted actors may have good accuracy in reproducing trajectories from the original logs, they may have poor responsiveness in adapting to the planned behavior exhibited by the autonomous vehicle in the simulation. For example, even if the autonomous vehicle is already present or will be present in a conflict lane, the scripted actor may still rear-end the autonomous vehicle or make a lane change. Therefore, existing methods reduce the signal-to-noise ratio of metrics measured from simulations and may not contribute to accurately evaluating the performance of autonomous vehicles. For example, the metrics may measure instances of suboptimal results in the performance of the autonomous vehicle. This disclosure is particularly advantageous for evaluating the performance of autonomous vehicles because it can equip intelligent actors with the ability to selectively deviate from the recorded trajectory in response to new environmental changes, such as the planned trajectory of the autonomous vehicle in the simulation. This disclosure also has another advantage, because the behavioral model parameters used in the intelligent agent are optimized to maintain minimal scenario divergence from the recorded trajectory once selective deviation from the recorded trajectory is initiated in the simulation.

[0003] This specification relates to methods and systems for evaluating the performance of autonomous vehicles in a simulation scenario using intelligent actors. According to one aspect of the subject matter described in this disclosure, a method includes: determining a simulation scenario from log data, the simulation scenario including a first non-autonomous vehicle (non-AV) actor, the first non-AV actor being enabled to selectively deviate from its recorded trajectory in the log data during simulation; performing a simulation based on the simulation scenario; monitoring the execution of the simulation; based on the monitoring, determining whether, during the simulation, behavior switching conditions for the first non-AV actor associated with traffic interactions including AV are met; in response to determining that the behavior switching conditions for the first non-AV actor associated with traffic interactions including AV are met during the simulation, modifying the behavior of the first non-AV actor to deviate from its recorded trajectory using a first behavior model; and determining performance metrics based on the simulation.

[0004] Generally, another aspect of the subject matter described in this disclosure includes a system comprising one or more processors and a memory operatively coupled to the one or more processors, wherein the memory stores instructions that, in response to execution by the one or more processors, cause the one or more processors to perform the following operations: determining a simulation scenario from log data, the simulation scenario including a first non-autonomous vehicle (non-AV) actor, the first non-AV actor being enabled to selectively deviate from its recorded trajectory in the log data during simulation; performing a simulation based on the simulation scenario; monitoring the execution of the simulation; determining, based on the monitoring, whether a behavior switching condition for the first non-AV actor associated with traffic interaction including AV is met during simulation; modifying the behavior of the first non-AV actor to deviate from its recorded trajectory using a first behavior model in response to determining that the behavior switching condition for the first non-AV actor associated with traffic interaction including AV is met during simulation; and determining performance metrics based on the simulation.

[0005] Other embodiments of one or more of these aspects include corresponding systems, apparatuses, and computer programs configured to perform the actions of these methods and encoded on computer storage devices.

[0006] These and other implementations may each optionally include one or more of the following aspects. For example, these aspects may further include: determining, in response to modifying the behavior of a first non-AV actor to deviate from its recorded trajectory, whether a behavior switching condition for a second non-AV actor associated with the first non-AV actor is met during the simulation, the second non-AV actor being included in the simulation scenario and enabled to selectively deviate from its recorded trajectory in log data during the simulation; and using a second behavior model to modify the behavior of the second non-AV actor to deviate from its recorded trajectory in response to determining that the behavior switching condition for the second non-AV actor associated with the first non-AV actor is met during the simulation. For example, these aspects may further include determining whether a behavior switching condition for a first non-AV actor associated with a traffic interaction including an AV is met during the simulation, including: determining the future recorded trajectory of the first non-AV actor in the simulation, determining the future recorded trajectory of the first non-AV actor in the simulation, and using the future recorded trajectory of the first non-AV actor and the future trajectory of the AV, determining whether there is a traffic conflict between the first non-AV actor and the AV at a future spatiotemporal point in the simulation. In another example, these aspects may further include defining the behavior switching condition based on a traffic conflict. For example, these aspects may further include determining whether the first non-AV actor has a traffic conflict with the AV at a future spatiotemporal point in the simulation using the future recorded trajectory of the first non-AV actor and the future trajectory of the AV, including projecting the future trajectory of the AV onto the reference frame of the first non-AV actor; and determining whether the first non-AV actor has a traffic conflict with the AV at each of a plurality of discrete time steps along the projected future trajectory of the AV. For example, these aspects may further include modifying the behavior of the first non-AV actor to deviate from its recorded trajectory, including modifying the acceleration of the first non-AV actor. For example, these aspects may further include modifying the acceleration of the first non-AV actor by: determining an acceleration that keeps the first non-AV actor at an acceptable following distance behind the AV in the same lane as the AV; determining the acceleration of the first non-AV actor to prevent a collision with the AV; determining whether the first non-AV actor must yield to the AV in a traffic conflict, said traffic conflict being a merging, and in response to determining that the first non-AV actor must yield to the AV in a traffic conflict, determining the acceleration of the first non-AV actor yielding to the AV; determining whether the first non-AV actor has the right-of-way in a traffic conflict with the AV, said traffic conflict being an intersection, and in response to determining that the first non-AV actor does not have the right-of-way in a traffic conflict with the AV, determining the acceleration at which the first non-AV actor stops at the intersection.For example, these aspects may also include evaluating behavior switching conditions selected from at least one of the following: deviation of the AV from its recorded trajectory in log data during traffic interaction with a first non-AV actor; a divergence threshold between the AV's planned trajectory and its recorded trajectory in log data; a current distance threshold between the AV and the first non-AV actor; changes in the state of dynamic elements other than non-AV actors; the presence of static elements; and proximity relative to traffic infrastructure. For example, these aspects may further include determining performance metrics based on simulation by: determining an evaluation set of simulation scenarios, each simulation scenario in the evaluation set including one or more non-AV actors enabled to selectively deviate from their recorded trajectories; performing a simulation for each simulation scenario in the evaluation set; and determining the precision and recall of the performance metrics based on the simulation. In another example, these aspects may also include instances of suboptimal results in AV performance measured by the performance metrics. In another example, these aspects may further include that the first behavior model and the second behavior model are machine learning models, each trained based on log records of multiple actors in real-world driving data. In another example, these aspects may also include using nonlinear least squares optimization methods to optimize the parameters of the first and second behavioral models to follow the recorded trajectory. Attached Figure Description

[0007] These and other aspects and features of this embodiment will become apparent after consulting the following description of specific embodiments in conjunction with the accompanying drawings, wherein: Figure 1 This is a block diagram illustrating an example hardware and software environment for an autonomous vehicle according to some implementations.

[0008] Figure 2 This is a block diagram illustrating an example computing system that uses intelligent actors to evaluate the performance of autonomous vehicles in a simulation scenario according to some implementations.

[0009] Figure 3A It is shown Figure 2 A block diagram of an example implementation of the simulation management engine referenced herein.

[0010] Figure 3B It is shown Figure 2 A block diagram of an example implementation of the simulation execution engine referenced herein.

[0011] Figure 3C It is shown Figure 2 A block diagram of an example implementation of the simulation verification engine referenced herein.

[0012] Figure 4 This is a flowchart illustrating a method for evaluating the performance of an autonomous vehicle according to some implementations.

[0013] Figure 5 This is a flowchart illustrating a method for modifying the behavior of a non-AV actor according to some implementations. Detailed Implementation

[0014] In the following disclosure, an intelligent log-based simulation (SFL) system 160 is used to evaluate the performance of an autonomous vehicle (AV) in a set of simulation scenarios using intelligent actors. In some implementations, intelligent actors may be used as replacements for existing scripted actors in the simulation scenarios. An intelligent actor can be defined as an integrated or reactive actor that is enabled to selectively deviate from a recorded trajectory based on the simulation scenario during simulation execution. For example, an intelligent actor can be a non-AV actor, such as an intelligent path follower (IPF) actor, which has a set of selective deviation capabilities from the recorded trajectory that are enabled during simulation scenario creation, such as merging interaction reasoning, adaptive cruise control (ACC), intersection navigation processing, collision avoidance, etc. The simulation scenario can describe a three-dimensional scene (e.g., a virtual scene) that simulates the behavior, attributes, and sensor configuration of the AV in specific encounters with an environment that includes other vehicle actors (autonomous and / or non-autonomous), pedestrian actors, wildlife actors, time of day, weather conditions, terrain, and road markings, etc., whether stationary or in motion. For example, simulation scenarios can include perception scenarios, perception simulation scenarios, motion planning simulation scenarios, vehicle detection and tracking (VDT) scenarios, etc.

[0015] The Intelligent SFL system 160 uses one or more intelligent non-AV actors to evaluate the performance of the AV in a simulated scenario. The intelligent actors can be made to selectively deviate from the recorded trajectory in response to new environmental changes, such as the AV's planned trajectory during simulation. For example, an intelligent actor can selectively deviate from the recorded trajectory in response to the AV's (also described herein as intelligent) planned trajectory in the simulation satisfying a behavior switching condition for that intelligent actor. The Intelligent SFL system 160 uses a behavior model to modify the behavior of the intelligent actors to deviate from the recorded trajectory. One existing method for evaluating AV performance is to use scripted actors in a simulated scenario. Scripted actors follow the original trajectory (recorded trajectory) observed in log files collected from real-world driving. In other words, scripted actors are configured to follow a predefined time trajectory in the simulation. However, when the AV is executing a planned trajectory that deviates from the original log (such as making a lane change, accelerating or decelerating along the current lane), a scripted actor faithfully reproducing the recorded trajectory in the simulation results in the observation of some irrational actor behaviors from that scripted actor, behaviors that a rational driver would not exhibit in a real-world driving scenario. This irrational behavior renders simulations used to evaluate the performance of autonomous vehicles ineffective due to errors.

[0016] This disclosure offers a particular advantage because it provides a system and method for evaluating AV performance in a simulated scenario (i.e., an intelligent simulation scenario) by using intelligent non-AV actors equipped with selective deviations from the recorded trajectory. For example, the “intelligence” of the intelligent actor’s selective deviation from the recorded trajectory allows the intelligent actor to avoid unrealistic behaviors exhibited by human drivers in the real world when interacting with the AV during traffic interactions. This improves the accuracy of metrics that measure suboptimal outcomes (e.g., collisions) in AV performance, determined from simulations of the intelligent simulation scenario, while maintaining metric recall. Furthermore, the actor model parameters used for the intelligent actor in the simulation scenario are optimized to maintain a minimum scenario divergence from the recorded trajectory during simulation once selective deviation from the recorded trajectory is initiated. By enabling non-AV actors in simulated scenarios to avoid unrealistic behaviors in their interactions with AV traffic while maintaining minimal scenario divergence from recorded trajectories, this disclosure reduces false positives of suboptimal results (i.e., where the performance or behavior of the AV is incorrectly attributed to the suboptimal result) and improves the reproducibility of recorded trajectories of intelligent actors in simulations (i.e., preserving actor behavior from the original logs to achieve minimal scenario divergence). Therefore, this disclosure improves the effectiveness of evaluating and validating AV performance in simulated scenarios. Furthermore, this disclosure enhances the accuracy and practicality of simulated scenarios in conducting effective testing of autonomous vehicles.

[0017] autonomous vehicles

[0018] Referring to the accompanying drawings, the same numbers in several views denote the same parts. Figure 1 Example hardware and software environments of autonomous vehicles in which various technologies disclosed herein can be implemented are shown. For example, vehicle 100 may include a powertrain 102 comprising a prime mover 104 powered by energy source 106 and capable of powering a drivetrain 108, and a control system 110 comprising steering control 112, powertrain control 114, and braking control 116. Vehicle 100 may be implemented as any number of different types of vehicles, including vehicles capable of transporting people and / or goods and capable of operating on land, sea, air, underground, underwater, and / or in space, and it is understood that the aforementioned components 102-116 may vary considerably depending on the type of vehicle using these components.

[0019] For simplicity, the embodiments discussed below will focus on wheeled land vehicles, such as cars, vans, trucks, buses, etc. In such embodiments, the prime mover 104 may include one or more electric motors and / or internal combustion engines (among others). The energy source 106 may include, for example, a fuel system (e.g., providing gasoline, diesel, hydrogen, etc.), a battery system, solar panels or other renewable energy sources, and / or a fuel cell system. The drivetrain 108 includes wheels and / or tires, a transmission, and / or any other mechanical drive components adapted to convert the output of the prime mover 104 into vehicle motion, one or more brakes configured to controllably stop or decelerate the vehicle 100, and directional or steering components adapted to control the trajectory of the vehicle 100 (e.g., a rack and pinion steering mechanism capable of pivoting one or more wheels of the vehicle 100 about a generally vertical axis to change the angle of the wheel's plane of rotation relative to the longitudinal axis of the vehicle). In some embodiments, a combination of powertrain and energy source may be used (e.g., in the case of an electric / gas hybrid vehicle), and in some embodiments, multiple electric motors (e.g., dedicated to individual wheels or axles) may be used as prime movers. In the case of a hydrogen fuel cell implementation, the prime mover 104 may include one or more electric motors, and the energy source 106 may include a fuel cell system powered by hydrogen fuel.

[0020] Steering control 112 may include one or more actuators and / or sensors for controlling the direction or steering components and receiving feedback therefrom to cause vehicle 100 to follow a desired trajectory. Powertrain control 114 may be configured to control the output of powertrain 102, such as controlling the output power of prime mover 104, controlling the gear positions in transmission 108, etc., thereby controlling the speed and / or direction of vehicle 100. Braking control 116 may be configured to control one or more brakes that decelerate or stop vehicle 100, such as disc or drum brakes coupled to the vehicle wheels.

[0021] Other vehicle types, including but not limited to aircraft, spacecraft, helicopters, drones, military vehicles, all-terrain or tracked vehicles, ships, submarines, and construction equipment, will inevitably use different power systems, transmission systems, energy sources, steering control, power system control, and braking control. Furthermore, in some embodiments, certain components can be combined; for example, vehicle steering control is primarily handled by altering the output of one or more prime movers. Therefore, the embodiments disclosed herein are not limited to the specific applications of the techniques described herein in autonomous wheeled land vehicles.

[0022] In the illustrated embodiment, fully automatic or semi-automatic control of vehicle 100 is implemented in vehicle control system 120, which may include one or more processors 122 and one or more memories 124, wherein each processor 122 is configured to execute program code instructions 126 stored in memory 124. The processor may include, for example, a graphics processing unit (“GPU”) and / or a central processing unit (“CPU”).

[0023] Sensor 130 may include various sensors suitable for collecting information from the vehicle's surrounding environment for controlling the operation of vehicle 100. For example, sensor 130 may include a radar (RADAR) sensor 134, a LiDAR (Light Detection and Ranging) sensor 136, a 3D positioning sensor 138, or satellite navigation systems such as GPS (Global Positioning System), GLONASS (Global Navigation Satellite System), BeiDou Navigation Satellite System (BDS), Galileo, a compass, etc. The 3D positioning sensor 138 can be used to determine the vehicle's position on Earth using satellite signals. Sensor 130 may optionally include a camera 140 and / or an IMU (Inertial Measurement Unit) 142. The camera 140 may be a monocular or stereo camera and capable of recording still images and / or video images. The IMU 142 may include multiple gyroscopes and accelerometers capable of detecting linear and rotational motion of vehicle 100 in three directions. One or more encoders 144, such as wheel encoders, may be used to monitor the rotation of one or more wheels of vehicle 100.

[0024] The output of sensor 130 can be provided to a set of control subsystems 150, including a positioning subsystem 152, a perception subsystem 154, a planning subsystem 156, and a control subsystem 158. The positioning subsystem 152 is primarily responsible for accurately determining the position and orientation (sometimes referred to as "attitude") of vehicle 100 within its surrounding environment and typically within a reference frame. The perception subsystem 154 is primarily responsible for detecting, tracking, and / or identifying objects in the environment surrounding vehicle 100. According to some embodiments, machine learning models can be used to track objects. Given a desired destination and static and moving objects in the environment, the planning subsystem 156 is primarily responsible for planning the trajectory or path of vehicle 100 within a certain time frame. According to some embodiments, machine learning models can be used to plan the vehicle trajectory. The control subsystem 158 is primarily responsible for generating appropriate control signals to control various control devices in the vehicle control system 120 to achieve the planned trajectory of vehicle 100. Similarly, machine learning models can be used to generate one or more signals to control the autonomous vehicle 100 to achieve the planned trajectory.

[0025] This is understandable. Figure 1 The component set of the vehicle control system 120 shown is only one example. Individual sensors may be omitted in some embodiments. Additionally or alternatively, in some embodiments, sensors may be used... Figure 1 Multiple sensors of the same type shown are used for redundancy and / or coverage of different areas around the vehicle. In addition to the sensors described above, there may be additional sensors to provide actual sensor data related to the operation and environment of the wheeled land vehicle. Similarly, different types and / or combinations of control subsystems may be used in other embodiments. Further, although subsystems 152-160 are illustrated as independent of processor 122 and memory 124, it will be understood that in some embodiments, some or all of the functions of subsystems 152-160 may be implemented by program code instructions 126 residing in one or more memories 124 and executed by one or more processors 122, and these subsystems 152-160 may, in some cases, be implemented using the same processor and / or memory. At least in part, subsystems may be implemented using various dedicated circuit logics, various processors, various field-programmable gate arrays (“FPGAs”), various application-specific integrated circuits (“ASICs”), various real-time controllers, etc.; as described above, multiple subsystems may utilize circuits, processors, sensors, and / or other components. Further, the various components in the vehicle control system 120 may be networked in various ways.

[0026] In some embodiments, vehicle 100 may further include an auxiliary vehicle control system (not shown), which can serve as a redundancy or backup control system for vehicle 100. In some embodiments, the auxiliary vehicle control system is fully capable of operating the autonomous vehicle 100 in the event of an adverse event in vehicle control system 120; while in other embodiments, the auxiliary vehicle control system may have only limited functionality, such as performing a controlled stop of vehicle 100 in response to an adverse event detected in primary vehicle control system 120. In other embodiments, the auxiliary vehicle control system may be omitted.

[0027] Generally speaking, countless different architectures, including various combinations of software, hardware, circuit logic, sensors, networks, etc., can be used to implement... Figure 1 The components are shown. For example, each processor can be implemented as a microprocessor, and each memory can represent a random access memory (“RAM”) device including a main storage device, as well as any supplementary levels of memory, such as cache memory, non-volatile or spare memory (e.g., programmable or flash memory), read-only memory, etc. Furthermore, each memory can be considered to include memory storage physically located elsewhere in vehicle 100, such as cache memory in any processor, and any storage capacity used as virtual memory, such as stored on a mass storage device or another computer controller. Figure 1 One or more processors 122 shown, or completely independent processors, can be used to implement additional functions of the vehicle 100 beyond automatic control purposes, such as controlling the entertainment system, operating the doors, lights, convenience functions, etc.

[0028] In addition, for additional storage, vehicle 100 may include one or more mass storage devices, such as removable disk drives, hard disk drives, direct access storage devices (“DASD”), optical drives (e.g., CD drives, DVD drives, etc.), solid-state storage drives (“SSD”), network-attached storage, storage area networks and / or tape drives, etc.

[0029] In addition, vehicle 100 may include a user interface 164 to enable vehicle 100 to receive and generate outputs from multiple inputs from a user or operator, such as one or more displays, touchscreens, voice and / or gesture interfaces, buttons, and other tactile controls. Alternatively, user input may be received via another computer or electronic device, such as through an application on a mobile device or through a web interface.

[0030] In addition, vehicle 100 may include one or more network interfaces, such as network interface 162, adapted to communicate with one or more networks 176 to allow information to be exchanged with other computers and electronic devices, including, for example, central services such as cloud services, from which vehicle 100 receives information and other data, including trained machine learning models, for its automatic control. The one or more networks 176 may be, for example, a communication network including a wide area network (“WAN”) (such as the Internet), one or more local area networks (“LAN”) (such as Wi-Fi LAN), a mesh network, and one or more bus subsystems. The one or more networks 176 may optionally utilize one or more standard communication technologies, protocols, and / or inter-process communication technologies. In some embodiments, data collected by one or more sensors 130 may be uploaded via network 176 to computing system 172 for further processing.

[0031] In the illustrated embodiment, vehicle 100 can communicate with computing system 172 via network 176 to enable various functions described below for evaluating the performance of autonomous vehicles in simulation scenarios using intelligent actors. In some embodiments, computing system 172 is a cloud-based computing device. See below for reference. Figure 2 As described in a more detailed description, computing system 172 includes an intelligent log-based simulation (SFL) system 160 and a machine learning engine 166. For example, in some embodiments, the intelligent SFL system 160 operates on computing system 172 to: determine a simulation scenario including non-AV actors; perform a simulation of the simulation scenario; monitor the execution of the simulation; determine whether behavior switching conditions for non-AV actors associated with AV are met; if the behavior switching conditions are met, use a behavior model trained by machine learning engine 166 to modify the behavior of non-AV actors to deviate from their recorded trajectories; and determine collision metrics associated with AV based on the simulation.

[0032] Figure 1 Each processor illustrated herein, along with the various additional controllers and subsystems disclosed herein, typically operates under the control of an operating system and executes or otherwise depends on various computer software applications, components, programs, objects, modules, data structures, etc., which will be described in more detail below. Furthermore, various applications, components, programs, objects, modules, etc., may also execute on one or more processors in another computer (e.g., computing system 172) connected to vehicle 100 via network 176, for example, in a distributed, cloud-based, or client-server computing environment, thereby distributing the processing required to realize the functions of the computer program across multiple computers and / or services via the network.

[0033] Generally, routines executed to implement the various embodiments described herein, whether implemented as part of an operating system or as a particular application, component, program, object, module, or sequence of instructions (or even a subset thereof), will be referred to herein as "program code." Program code typically comprises one or more instructions that reside at various times in various memory and storage devices, and when read and executed by one or more processors, perform steps necessary to implement the steps or elements embodying the various aspects of this disclosure. Furthermore, while embodiments have been and will hereafter be described in the context of fully functional computers and systems, it should be understood that the various embodiments described herein can be distributed as different forms of program products, and embodiments can be implemented regardless of the specific type of computer-readable medium actually used for performing the distribution.

[0034] Examples of computer-readable media include tangible, non-transient media such as volatile and non-volatile storage devices, floppy disks and other removable disks, solid-state drives, hard disk drives, magnetic tapes, and optical discs (such as CD-ROMs, DVDs, etc.).

[0035] Furthermore, the various program codes described below can be identified based on the application implementing the code in a particular implementation. However, it should be understood that any specific program terminology used subsequently is merely for convenience, and therefore this disclosure should not be limited to use only in any particular application identified and / or implied by these terms. Further, considering the infinite number of ways a computer program can be organized as routines, procedures, methods, modules, objects, etc., and the various ways program functionality can be distributed among various software layers residing within a typical computer (e.g., operating systems, libraries, APIs, applications, applets, etc.), it should be understood that this disclosure is not limited to the specific organization and distribution of program functionality described herein.

[0036] Figure 1 The example environments shown are not intended to limit the implementations disclosed herein. In fact, other alternative hardware and / or software environments may be used without departing from the scope of the implementations disclosed herein.

[0037] Intelligent Log-Based Simulation (SFL) System

[0038] Figure 2 This is a block diagram illustrating an example of a computing system 172 for evaluating the performance of an autonomous vehicle in a simulated scenario using intelligent actors, according to some implementations.

[0039] refer to Figure 2The illustrated example computing system 172 includes: one or more processors 210 that communicate with memory 260 via a communication system 240 (e.g., a bus); at least one network interface controller 230 having a network interface port connected to a network (e.g., connected to network 176 via signal line 178); a data storage device 280; and other components, such as an input / output (“I / O”) component interface 250 connected to a display (not shown) and an input device (not shown), an intelligent SFL system 160, and a machine learning engine 166. Typically, processor 210 executes instructions (or computer programs) received from memory 260. The illustrated processor 210 is incorporated with (or directly connected to) cache memory 220. In some cases, instructions are read from memory 260 into cache memory 220 and executed by processor 210 from cache memory 220.

[0040] More specifically, processor 210 can be any logic circuit that processes instructions, such as those fetched from memory 260 or cache 220. In some embodiments, processor 210 is a microprocessor unit or a dedicated processor. The computing system or device 172 can be based on any processor or collection of processors capable of operating as described herein. Processor 210 can be a single-core or multi-core processor. Processor 210 can be multiple independent processors.

[0041] Memory 260 can be any device suitable for storing computer-readable data. Memory 260 can be a device with fixed storage or a device for reading removable storage media. Examples include all forms of non-volatile memory, media and storage devices, semiconductor storage devices (e.g., EPROM, EEPROM, SDRAM and flash memory devices), magnetic disks, magneto-optical disks, and optical disks (e.g., CD ROM, DVD-ROM, or Blu-ray® discs). Computing system 172 can have any number of storage devices as memory 260. Although the intelligent SFL system 160 and machine learning engine 166 are illustrated as independent of processor 210 and memory 260, it will be understood that in some embodiments, some or all of the functionality of components 160 and 166 can be implemented by program code instructions residing in memory 260 and executed by processor 210.

[0042] Cache memory 220 is typically a form of computer memory placed close to processor 210 to achieve fast read times. In some embodiments, cache memory 220 is part of processor 210 or resides on the same chip as processor 210. In some embodiments, there are multiple levels of cache 220, such as L2 and L3 cache layers.

[0043] Network interface controller 230 manages data exchange via a network interface (sometimes referred to as a network interface port). Network interface controller 230 handles the physical and data link layers of the OSI model for network communication. In some embodiments, some tasks of the network interface controller are handled by one or more processors 210. In some embodiments, network interface controller 230 is part of processor 210. In some embodiments, computing system 172 has multiple network interfaces controlled by a single controller 230. In some embodiments, computing system 172 has multiple network interface controllers 230. In some embodiments, each network interface is a connection point of a physical network link (e.g., a Cat-5 Ethernet link). In some embodiments, network interface controller 230 supports wireless network connectivity, and the interface port is a wireless (e.g., radio) receiver / transmitter (e.g., for any IEEE 802.11 protocol, Near Field Communication "NFC", Bluetooth, ANT, WiMAX, 5G, or any other wireless protocol). In some embodiments, network interface controller 230 implements one or more network protocols, such as Ethernet. Typically, computing system 172 exchanges data with other computing devices via a network interface through a physical or wireless link (represented by signal line 178). The network interface can be directly linked to another device or linked to another device via an intermediary device (e.g., a network device that connects computing system 172 to a data network (such as the Internet) such as a hub, bridge, switch, or router).

[0044] Data storage device 280 may be a non-transient storage device that stores data to provide the functionality described herein. As will be defined below, data storage device 280 may store (among other data) log data 211, simulation registry 213, simulation log 215, AV performance metrics 217, and machine learning model 224.

[0045] The computing system 172 may include one or more input or output (“I / O”) devices 250, or provide interfaces thereto. Input devices include, but are not limited to, keyboards, microphones, touchscreens, foot pedals, sensors, MIDI devices, and pointing devices such as mice or trackballs. Output devices include, but are not limited to, video displays, speakers, refreshable Braille terminals, lights, MIDI devices, and 2-D or 3-D printers. Other components may include I / O interfaces, external serial device ports, and any additional coprocessors. For example, the computing system 172 may include interfaces (e.g., Universal Serial Bus (USB) interfaces) for connecting input devices, output devices, or additional storage devices (e.g., portable flash drives or external media drives). In some embodiments, the computing system 172 includes additional devices such as coprocessors, for example, a math coprocessor that can assist the processor 210 in performing high-precision or complex calculations.

[0046] In embodiments consistent with this disclosure, the intelligent SFL system 160 is used to evaluate the performance of an autonomous vehicle (AV) in a simulation scenario using intelligent actors. More specifically, this disclosure relates to evaluating AV performance using simulation scenarios in which intelligent actors can be used as alternatives to scripted actors. Intelligent actors can be defined as integrated or reactive actors that are enabled to selectively deviate from a recorded trajectory during simulation execution based on the simulation scenario. For example, intelligent actors can be non-AV actors, such as intelligent path follower (IPF) actors with a set of capabilities for selectively deviating from a recorded trajectory. This set of capabilities may include merging interactive inference, adaptive cruise control (ACC), intersection navigation processing, collision avoidance, etc., enabled when the simulation scenario is created.

[0047] In some implementations, the intelligent SFL system 160 includes a log data generator 202, a simulation management engine 204, a simulation execution engine 206, and a simulation verification engine 208. The log data generator 202, simulation management engine 204, simulation execution engine 206, and simulation verification engine 208 of the intelligent SFL system 160, as well as the separate machine learning engine 166, are example components that can implement the techniques and / or systems, components, and techniques described herein, and can be connected to. Although described within the context of computing system 172, it should be understood that... Figure 2 The operations performed by one or more components 202, 204, 206, 208, and 166 can be distributed across multiple computing systems. In some embodiments, one or more aspects of components 202, 204, 206, 208, and 166 can be combined into a single system, and / or one or more aspects can be implemented by computing system 172. For example, in some embodiments, multiple aspects of log data generator 202 can be combined with multiple aspects of simulation management engine 204. In another example, multiple aspects of simulation verification engine 208 can be combined with multiple aspects of simulation execution engine 206. Engines according to many embodiments can each be implemented in one or more computing devices that communicate, for example, via communication network 176.

[0048] Log data generator 202 can receive and store vehicle log data 211 collected during one or more driving sessions of an autonomous, semi-autonomous, or non-autonomous vehicle in the real world. For example, one or more sensors 130 of an autonomous vehicle (AV) 100 can collect a set of vehicle log data 211 along one or more driving routes in the real world and upload the collected data to computing system 172 via network 176. This set of vehicle log data 211 can represent typical situations or events that the autonomous vehicle 100 is expected to encounter along one or more driving routes in the real world. In some implementations, each instance of vehicle log data can be associated with a timestamp. Vehicle log data can include time-series log data, such as location data, tracking data, and optionally include other vehicle sensor data and environmental data. For example, during a driving session of AV 100, vehicle control subsystem 150 can collect data at different points in time and record the time of data collection. For example, each instance of time-series log data can contain the current position, orientation, and speed of AV 100 based on location data. Tracking data may include tracking of objects outside the autonomous vehicle 100, describing their location, range, orientation category, speed, and other tracking data or predictions. Information about static objects (such as road signs, lane markings, road surfaces, etc.) may also be recorded. In some embodiments, other forms of environmental data (e.g., weather conditions, lighting conditions, visibility, etc.) may also be recorded. In some embodiments, the log data generator 202 may process and convert timestamped vehicle log data 211 collected along a real-world driving route into a set of route segments or log segments. Each log segment in this set may include an event encountered by the autonomous vehicle 100. Depending on the event, the log segment may also include vehicle log data before and / or after the event. For example, a 30-second log segment may include 10 seconds of vehicle log data before and after the event. The log data generator 202 may store this set of log segments (e.g., a set of real-world driving log data segments) under log data 211 in the data storage device 280. For the purposes of this disclosure, the terms “log fragment,” “raw log,” and “log data” are used interchangeably to refer to the same thing: real driving data associated with the autonomous vehicle 100, collected on public roads under real-world driving conditions.

[0049] In some implementations, the log data generator 202 may receive one or more user annotations (e.g., tags) to label each route segment in the set of log data segments based on the presence of a specific operating environment (OE), vehicle maneuver (VM), or actor (A) or object and event detection and response (OEDR) within that route segment. OE elements may characterize the operating environment of the AV 100, including road type, geographic features, speed range, weather and environmental conditions, traffic rules, operating location, etc. VM elements may characterize the types of maneuvers initiated by the AV 100 itself, which are typically navigation-related, such as entering and exiting restricted highways, initiating turns, changing lanes, stopping, parking, turning on and off power, etc. OEDR or actor (A) elements may characterize the handling of external situations encountered by the AV 100, including actors and objects, perception, planning, and the implementation of autonomous vehicle actions. In some implementations, the log data generator 202 may generate and provide a user interface for users to label all relevant OE, VM, and OEDR elements in the log segments. Each log segment may include a label for speed limit, road type, lane description, direction, and at least one vehicle movement. Log data generator 202 stores labels provided as annotations for each log segment in log data 211. These labels make querying the set of log segments easier. For example, the set of log segments can also be categorized based on these labels. In some implementations, log data generator 202 manages a training dataset composed of log segments and manages the classification of OE, VM, and OEDR labels provided for each log segment. Log data generator 202 stores the training dataset containing log segments in data storage device 280 and makes it available for training one or more machine learning models.

[0050] Exemplary OE elements that can be tagged in log fragments include, but are not limited to: time of day (e.g., morning, noon, evening, night, etc.), speed limits (e.g., 25 mph, 30 mph, 40 mph, etc.), road type or surface (e.g., surface street, side road or auxiliary road, highway, entrance ramp, exit ramp, parking lot, etc.), straight lanes (e.g., single lane, two lanes, three lanes, four lanes), lane direction (e.g., one-way, two-way unseparated, two-way separated, etc.), and intersections (e.g., 4-way traffic light intersection, 3-way traffic light intersection, etc.). Traffic light types include: green light intersections, 2-way traffic light intersections, one-way traffic light intersections, 4-way or all-way stop sign intersections, 2-way stop sign intersections, one-way stop sign intersections, 2-way uncontrolled intersections, 3-way uncontrolled intersections, 4-way uncontrolled intersections, etc.; traffic light types (e.g., traffic lights with protected left-turn arrows, traffic lights with protected right-turn arrows, etc.); and road elements (e.g., shoulder on the left, shoulder on the right, shoulders on both sides, parallel parking lanes next to A / V lanes, bicycle lanes, pedestrian crossings, railway crossings, fire lanes, and two lanes merging into one).

[0051] Exemplary VM elements that can be tagged in log fragments include, but are not limited to: turning (e.g., unprotected left turn, right turn when the AV has no right-of-way, right turn at a red light, protected right turn, protected left turn, right turn at a stop sign or when the AV has the right-of-way, left turn at a stop sign or when the AV has the right-of-way, U-turn, etc.), lane changing (e.g., changing lanes to the right, changing lanes to the left), lane position (e.g., occupied lane position (counting from the rightmost lane = 1)), and additional AV behaviors (e.g., merging, maintaining a speed appropriate to the road speed limit and / or adjusting speed due to the vehicle in front, stopping, nudge, going straight after a green light, stopping at a stop sign and then going straight, or passing through an uncontrolled intersection, etc.).

[0052] Exemplary OEDR elements that can be tagged in log fragments include, but are not limited to: whether an actor / object exists on or near the AV path (e.g., pedestrians, motorcycles, cyclists, vehicles, foreign objects, construction areas, toll booths, police traffic stops, etc.), whether the actor / object is occluded or unoccluded on or near the AV path, whether the actor / object is compliant on or near the AV path, the type of actor / object on or near the AV path, and the motion state of the actor / object on or near the AV path (e.g., rate, speed, acceleration, etc.).

[0053] The simulation management engine 204 can access, process, and manage a set of base simulation scenarios sufficient to model a set of real-world situations used for testing the behavior of the AV 100. In some implementations, the simulation management engine 204 can access the base simulation scenarios and convert them into multiple simulation scenarios. For example, the simulation management engine 204 can use parameter scanning to adjust the value of a parameter in the base simulation scenario within a defined range and generate configurations for multiple different simulation scenarios. In another example, the simulation management engine 204 can use a Monte Carlo sampling method to randomly sample the value of a parameter in the base simulation from a probability distribution and generate configurations for various simulation scenarios. As an example, changing parameters in the base simulation scenario can include changing one or more configuration values ​​of vehicle platform parameters, map parameters, starting gate, starting speed, actor placement (e.g., bicycles, pedestrians, etc.), environmental parameters, or other autonomous parameters.

[0054] In some implementations, simulation management engine 204 may transform real-world driving log file data accessible in log data 211 (e.g., generated by log data generator 202) in different ways to generate simulation data. The simulation data may represent an editable source of facts defining multiple simulation scenarios. Simulation management engine 204 may use one or more components of the log data 211 instance to assist in creating at least one aspect of the simulation scenario. In some implementations, simulation management engine 204 may use vehicle log data 211 as a data source based on real-world conditions regarding real-world driving situations to adjust parameter values ​​in a base simulation scenario, thereby generating multiple varied simulation scenarios. For example, simulation management engine 204 may use vehicle log data 211 as an aid to generate a description of the environment in the simulation scenario, including the behavior of AVs (e.g., ego vehicles), vehicle configurations (e.g., AV position, platform, speed, or orientation), sensor configurations, and the environment, including actors (e.g., other vehicles, traffic, pedestrians, and static objects). However, more generally, in some implementations, other information available from vehicle log data 211 may be used as an aid in generating simulation scenarios. In general, vehicle log data 211 can be used as a resource in some implementations to provide a real sensor data source for simulation tasks that require a real sensor data source.

[0055] The simulation management engine 204 can register simulation scenarios by generating simulation identifiers (e.g., log-based simulation (SFL) identifiers), assigning simulation identifiers to simulation scenarios, and storing simulation scenarios in a simulation registry 213 indexed by the simulation identifiers in the data storage device 280. For example, the simulation identifier can be a globally unique identifier (GUID). The simulation registry 213 can be a database storing currently and previously available simulation scenarios, indexed by their respective simulation identifiers. In some implementations, the simulation management engine 204 can process simulation scenarios and derive one or more tags to associate with the simulation scenario in the simulation registry 214. For example, these tags can be based on one or more of the following: geographic location (e.g., San Francisco, New York, etc.), actor (e.g., other vehicles, bicycles, pedestrians, scooters, motorized scooters, etc.), behavior (e.g., lane changing, merging, turning, etc.), location (e.g., four-way parking, intersection, ramp, etc.), status (e.g., deprecated, isolated, etc.), vehicle brand and model of the actor, sensor configuration, etc. In some implementations, the simulation management engine 204 may also receive one or more user annotations for tagging each simulation scenario in the simulation registry 213 based on the presence of OE, VM, and OEDR elements. For example, the simulation management engine 204 generates and provides a user interface for users to tag all relevant OE, VM, and OEDR elements in a simulation scenario. Annotated tags make querying the simulation registry 213 and selecting simulation scenarios easier. Simulation scenarios in the simulation registry 213 can also be categorized by these annotated tags. In some implementations, the simulation management engine 204 generates and provides a user interface for querying the simulation registry 213 to select one or more simulation scenarios to execute in a simulation. For example, a query may include one or more phrases such as “pedestrians near AV path,” “speed limit = 55 mph,” “4-way traffic light intersection,” etc. The simulation management engine 204 matches the query against annotation tags associated with the simulation scenario and retrieves the matching simulation scenario from the simulation registry 213.

[0056] refer to Figure 3A The illustration shows an example implementation of the simulation management engine 204 in more detail. The simulation management engine 204 may include a simulation (SIM) annotation engine 302 and a SIM scene reconstruction engine 304.

[0057] In some implementations, the SIM annotation engine 302 may compile a list of simulation scenarios that include suboptimal outcomes in traffic interactions involving AVs and scripted actors. For example, the list of simulation scenarios may include existing simulation scenarios with observed AV-actuator collisions compiled from the simulation registry 213. Suboptimal outcomes can refer to autonomous driving behaviors or decisions made by the AV that might violate safety rules and regulations in the real world. Such safety violations by the AV could result in accidents, collisions, or other dangerous situations that could cause injury or discomfort to passengers, other vehicles, or pedestrians. The degree of suboptimal outcome can vary widely, ranging from a less-than-ideal (but acceptable) result to one that causes serious AV performance failure. Examples of suboptimal outcomes may include, but are not limited to, abrupt acceleration, speeding, sudden braking, deceleration, failure to yield, failure to recognize obstacles, failure to recognize traffic signs or traffic lights, close calls, crossing road boundaries, intrusion into oncoming lanes, lane drift, collisions, etc. The SIM annotation engine 302 may collaborate with the simulation execution engine 206 and the simulation verification engine 208 to collect a first set of labels for the list of simulation scenarios containing scripted actors. The SIM annotation engine 302 can instruct the simulation execution engine 206 to run simulations based on each simulation scenario in the list and collect manual labels regarding whether true positive suboptimal results in terms of AV performance were observed in the simulation. In one example, a true positive AV-actuator collision can be defined as a collision involving the AV and an actor, where the behavior or performance of the AV leads to the collision with the actor. A false positive AV-actuator collision can be defined as a collision involving the AV and an actor, where the irrational behavior of the actor leads to the collision with the AV. In some implementations, the SIM annotation engine 302 can instruct the simulation execution engine 206 to execute simulations of each scenario in the simulation scenario list using a degraded version of the AV motion planner to facilitate the collection of such labels indicating true positive suboptimal results in AV performance. For example, the simulation execution engine 206 can configure such a poorly performing motion planner for the AV by adjusting one or more parameters in the motion planner software stack.

[0058] In some implementations, the SIM annotation engine 302 receives simulation results and / or simulation logs from the simulation execution engine 206, based on the simulation execution of each scenario in the simulation scenario list. The SIM annotation engine 302 processes the simulation logs to generate tag metadata to be associated with each simulation scenario in the list. For example, a tagged simulation scenario may include a tag metadata tuple, such as: the task identifier, start time, and end time of the segment; the identifier of the simulation scenario from which the simulation originated; the commit hash (commithash, e.g., a unique identifier for the software version) of the motion planner used for the AV in the simulation scenario; a tag indicating whether suboptimal results occurred in the AV performance; and a suboptimal result descriptor, which includes, but is not limited to, the type of suboptimal result (e.g., collision, emergency braking, etc.), the actor identifier of the actor involved in the suboptimal result, the actor type of the actor involved in the suboptimal result, the timestamp of the suboptimal result, the severity of the suboptimal result, etc. The SIM annotation engine 302 may generate and provide a user interface for human reviewers to submit tags identifying suboptimal results in the AV performance of a particular simulation scenario. For example, a human reviewer can identify an AV-actuator collision observed in a simulation of a specific simulation scenario as a true positive collision, where the AV's behavior or performance is the cause of the collision with the actor. In another example, a human reviewer can identify an AV-actuator collision observed in a simulation of a specific simulation scenario as a false positive collision, where the actor's illogical behavior is the cause of the collision with the AV. The SIM annotation engine 302 stores the tag metadata for each scenario in the simulation scenario list in the simulation registry 213 of the data storage device 280.

[0059] The SIM annotation engine 302 can maintain a labeled dataset of simulation scenarios to facilitate the development of intelligent simulation scenarios and to evaluate the effect of changes in actor behavior on the signal-to-noise ratio of one or more AV performance metrics obtained in the simulation of intelligent simulation scenarios. For example, these metrics can measure instances of suboptimal results in AV performance. The SIM annotation engine 302 can keep the label metadata of each labeled simulation scenario in the list unchanged, used to evaluate changes in actor behavior, such as the precision and / or recall of metrics. A limitation may exist: only the latest version of the AV motion planner can be used in the simulation. This can be problematic because running the same simulation scenario with different commit hashes for the AV motion planner might invalidate the original labels of that simulation scenario. To prevent label invalidation, the SIM annotation engine 302 can collect new labels for each simulation scenario in the list each time the motion planner software version is updated. Unless a change is detected in the new simulation run, the SIM annotation engine 302 will retain the original labels of the simulation scenario by default. In some implementations, the SIM annotation engine 302 may rely on an event detector that operates on actors in the simulation based on the simulation scenario. The SIM annotation engine 302 examines the output of the event detector in the simulation log to see if its results have changed. For example, if an actor that cut in front of the AV in the simulation of the original annotated simulation scene no longer does so in a new simulation run, this result can be considered changed because the new version of the motion planner causes the AV to accelerate and stay in front of the actor. If a change is detected, the SIM annotation engine 302 processes the simulation log to generate new label metadata for the simulation scene. Examples of changes that may invalidate the original labels may include, but are not limited to: metadata associated with suboptimal results (such as collisions, sudden braking, violent acceleration, etc.) detected in the new simulation run, such as the identifier of the actor involved in the suboptimal result, the collision point, or the actor's position relative to the AV, and invariants (such as capabilities being tested in the simulation).

[0060] The SIM scene reconstruction engine 304 receives a labeled dataset from a list of simulation scenes provided by the SIM annotation engine 302. The SIM scene reconstruction engine 304 identifies simulation scenes from this list that have suboptimal results involving AV (Action-Active) collisions. The SIM scene reconstruction engine 304 reconstructs the simulation scene using intelligent actors to eliminate unrealistic actor behaviors that may lead to suboptimal results and improve the quality of the simulation scene. For example, the SIM scene reconstruction engine 304 can identify simulation scenes with label metadata indicating the presence of false-positive AV-actuator collisions and recreate the simulation scene to use intelligent actors. In some implementations, the SIM scene reconstruction engine 304 can receive user requests to update simulation scenes containing suboptimal results involving AV, thereby replacing scripted actors with intelligent actors. In some implementations, the SIM scene reconstruction engine 304 processes simulation logs of existing simulation scenes from the labeled dataset in the list of simulation scenes and identifies actors involved in suboptimal results. The SIM scene reconstruction engine 304 automatically constructs intelligent simulation scenes corresponding to existing simulation scenes by replacing actors involved in false-positive AV-actuator collisions with intelligent actors.

[0061] The SIM scene reconstruction engine 304 replaces one or more existing non-AV actors in the simulation scene with intelligent non-AV actors. For example, the SIM scene reconstruction engine 304 replaces existing scripted actors in the simulation scene with intelligent actors such as Intelligent Path Followers (IPFs). Scripted actors in the existing simulation scene are actors that follow predefined time trajectories in the simulation. While scripted actors may have good accuracy in replaying trajectories from the original logs, they may have poor responsiveness in adapting to the planning behavior exhibited by AVs in the simulation. For example, even if an AV is already present or will be present in a conflict lane, a scripted actor may rear-end the AV or make a lane change. Therefore, using scripted actors reduces the signal-to-noise ratio of metrics measuring suboptimal results and may not help accurately evaluate AV performance. IPF actors are a type of Path Follower (PF) actor. A PF actor is an integrated actor that follows a predefined spatial path. PF actors do not possess any intelligence, while IPF actors can be configured to have different intelligent capabilities that influence their movement (as described herein). The SIM scene reconstruction engine 304 can retrieve the platform file of an existing simulation scene from the simulation registry 213 in the data storage device 280, and update the platform file by replacing the parameters associated with scripted actors with IPF actors while retaining the remaining parameters. This ensures that the reconstructed simulation scene is generally consistent with the existing simulation scene. That is, parameters other than IPF actors are retained in the reconstructed simulation scene. Such a simulation scene reconstructed using intelligent actors can be called an intelligent simulation scene.

[0062] In intelligent simulation scenarios, intelligent actors that replace scripted actors accurately replay their recorded trajectories from the original logs during simulation. The SIM scenario reconstruction engine 304 enables a mode within intelligent actors in the intelligent simulation scenario to selectively deviate from their respective recorded trajectories during simulation. The SIM scenario reconstruction engine 304 equips intelligent actors in the intelligent simulation scenario with the ability to selectively deviate from recorded trajectories in response to new environmental changes, such as the planned trajectory of an AV in the simulation. For example, an IPF actor in an intelligent simulation scenario might be equipped with a set of capabilities for selectively deviating from recorded trajectories, such as merging interaction reasoning, adaptive cruise control (ACC), intersection navigation processing, and collision avoidance, to avoid unrealistic behaviors that could lead to suboptimal AV performance results in the simulation. This improves the accuracy of metrics used to measure suboptimal AV performance instances, determined based on simulations of intelligent simulation scenarios, while maintaining metric recall.

[0063] The SIM scene reconstruction engine 304 enables by default the ability of all non-AV actors in the intelligent simulation scene to selectively deviate from the recorded trajectory. In some implementations, the SIM scene reconstruction engine 304 can receive user input to select specific non-AV actors in the intelligent simulation scene to be equipped with the ability to selectively deviate from the recorded trajectory. The SIM scene reconstruction engine 304 uses the selected non-AV actors equipped with the ability to selectively deviate from the recorded trajectory to construct the intelligent simulation scene. In some other implementations, the SIM scene reconstruction engine 304 can receive user input to select that one or more non-AV actors in the intelligent simulation scene are not allowed to deviate from the recorded trajectory. For example, a user who wants to test AV in the simulation may observe an actor exhibiting non-compliant behavior (e.g., aggressive driving behavior) in an original log segment but without an AV-actuator collision. In the intelligent simulation scene corresponding to that log segment, the user may not want to equip that non-compliant actor with the ability to deviate from the recorded trajectory in the simulation. Instead, the user may want to retain the actor's non-compliant behavior to induce a collision with the AV or to test the AV's behavioral response to the behavior of the non-compliant actor in the simulation. The SIM scene reconstruction engine 304 utilizes selected actors with the ability to deviate from the recorded trajectory to construct intelligent simulation scenes.

[0064] Once a deviation from the recorded trajectory is initiated during simulation, the intelligent actor may begin to deviate significantly from its original recorded trajectory, subsequently failing to retain the original scene of the simulation scenario. To correct this, when reconstructing the simulation scene, the SIM scene reconstruction engine 304 optimizes one or more parameters associated with the actor's behavior model used internally by the intelligent actor to follow the original recorded trajectory. This actor model parameter optimization improves the reproducibility of the recorded trajectory of the intelligent actor in the simulation. The SIM scene reconstruction engine 304 uses optimization techniques to optimize the actor behavior model parameters and / or intelligent driver model (IDM) parameters of the intelligent actor in the corresponding intelligent simulation scene to maintain minimal scene divergence from the original log fragment. For example, the SIM scene reconstruction engine 304 may use a nonlinear least squares optimization method that considers gradients and curvatures in the optimization of actor behavior model parameters to better fit the recorded trajectory. In some implementations, the SIM scene reconstruction engine 304 collaborates with the machine learning engine 166 to train the actor behavior model using a training dataset of log fragments. For example, the actor behavior model can be trained using the recorded trajectories of multiple actors from real-world driving data.

[0065] The SIM scene reconstruction engine 304 can generate a list of intelligent simulation scenes corresponding to the labeled dataset of simulation scenes maintained by the SIM annotation engine 302. The SIM scene reconstruction engine 304 can identify a batch of simulation scenes and convert them into intelligent simulation scenes. For example, the SIM scene reconstruction engine 304 can identify a batch of frequently used and / or recently used simulation scenes (e.g., valuable simulation scenes) and immediately convert all scenes in that batch into intelligent simulation scenes. The SIM scene reconstruction engine 304 can collaborate with the simulation execution engine 206 and the simulation verification engine 208 to verify the intelligent simulation scenes from the list of intelligent simulation scenes. Once a smart simulation scene is verified, the SIM scene reconstruction engine 304 can deploy it as the default setting to replace existing simulation scenes.

[0066] The SIM annotation engine 302 can collaborate with the simulation execution engine 206 and the simulation verification engine 208 to collect a second set of labels for the list of intelligent simulation scenarios to evaluate actor behavior changes compared to the list of non-intelligent simulation scenarios. For example, the SIM annotation engine 302 can instruct the simulation execution engine 206 to run a simulation once based on each intelligent simulation scenario in the list and collect human labels indicating whether a true positive suboptimal result was observed in that simulation. In some implementations, the SIM annotation engine 302 can instruct the simulation execution engine 206 to perform a simulation for each scenario in the list of intelligent simulation scenarios using a degraded version of the AV motion planner to facilitate the collection of true positive suboptimal result labels. It should be understood that evaluating actor behavior changes in the context of a flawed AV motion planner will not provide the exact same signal regarding the retention rate of true positive safety violations in the AV compared to evaluating them in the context of a state-of-the-art, conventional motion planner. The SIM annotation engine 302 can be configured to continuously collect true positive safety violations using the latest version of the AV motion planner to ensure that the metric retention rate in the intelligent simulation scenarios does not decrease. The SIM scene reconstruction engine 304 stores the list of intelligent simulation scenes in the simulation registry 213 of the data storage device 280.

[0067] The simulation execution engine 206 can perform simulations for the control subsystem set 150 of the autonomous vehicle 100 based on one or more simulation scenarios in the simulation registry 213. For example, the simulation scenario may correspond to a perception simulation scenario, a motion planning simulation scenario, a vehicle detection and tracking scenario, etc. In some embodiments, the simulation management engine 204 sends a simulation identifier to the simulation execution engine 206. The simulation execution engine 206 uses the simulation identifier to extract the configuration of the matching simulation scenario from the simulation registry 213 and performs the simulation based on the extracted simulation scenario configuration. The simulation execution engine 206 can create a run identifier (run ID) to associate with the execution (run) of the simulation. In some embodiments, the simulation execution engine 206 can create a batch of multiple simulation scenario variants and execute the batch of multiple simulation scenario variants in a single execution. In such an embodiment, the simulation execution engine 206 can create a batch identifier (batch ID) to associate with the batch execution. The simulation execution engine 206 can generate simulation results and / or simulation logs during simulation execution and store them in the simulation log 215. In some implementations, simulation results and / or simulation logs may be one or more formatted messages that include or encode state information of AV 100 and other actors observed in the simulation. For example, state information may include event detections associated with AV 100, such as collisions, sudden braking, deceleration, and other potential critical events observed during the simulation run. Simulation log 215 may be a database storing historical logs of simulation runs indexed by corresponding run IDs and / or batch IDs. In some implementations, simulation execution engine 206 generates one or more formatted messages reflecting events observed in the simulation based on the simulation scenario and streams them in real-time to simulation verification engine 208 during simulation execution.

[0068] refer to Figure 3B An example implementation of the simulation execution engine 206 is shown in more detail. The simulation execution engine 206 may include SIM switching logic 306 and SIM actor intelligent logic 308 to assist in performing simulations based on intelligent simulation scenarios.

[0069] SIM switching logic 306 receives a list of intelligent simulation scenarios from SIM scene reconstruction engine 304. SIM switching logic 306 determines an intelligent simulation scenario from this list and performs a simulation based on that intelligent simulation scenario. An intelligent simulation scenario may contain one or more non-AV intelligent actors, such as IPF actors, which can selectively choose to exhibit different intelligent behaviors during the simulation. For the purposes of this disclosure, intelligent behavior can refer to the ability of an intelligent actor to selectively deviate from a recorded trajectory in response to changes in AV behavior (e.g., the planned trajectory of the AV) and / or the behavior of other intelligent actors during the simulation.

[0070] Different intelligent behaviors can influence the movement of IPF (Intelligent Power Function) actors in simulations under specific conditions. The intelligent behaviors of IPF actors may be constrained to approximate the behavior of ordinary human drivers in the real world. In some implementations, these constraints may be physical. For example, the constraints could specify a maximum acceleration rate, a minimum safe following distance to maintain, a minimum distance to travel before initiating a lane change, etc. An IPF actor may have a predefined path to follow, consisting of a series of connected route edges. Furthermore, the IPF actor may have a series of control points arranged along these route edges. The control points indicate the desired velocity of the IPF actor at various points along its route. In the simulation, the IPF actor perceives the world in an orthogonal curvilinear coordinate (OCC) reference system defined by its own route, rather than in a Cartesian coordinate system. For example, an IPF actor perceives its position in the simulation based on the longitudinal, lateral, and vertical displacements of other actors along its own route. IPF actors in the intelligent simulation scenario may have access to the ground truth acceleration curves exhibited by the original actors in the original logs. In some implementations, the SIM actor intelligence logic 308 can selectively choose whether to enable accelerated intelligent behavior of the IPF actor along the longitudinal component of its OCC reference frame.

[0071] The SIM switching logic 306 monitors the execution of the simulation to determine whether the behavior switching conditions for non-AV actors are met. The behavior switching conditions for non-AV actors may be evaluated in conjunction with behavioral changes (e.g., recorded trajectories) of AV and / or other non-AV actors during the simulation. The SIM switching logic 306 uses the behavior switching conditions to identify the time step in the simulation that allows a non-AV actor to switch from trajectory-following behavior to intelligent behavior. In some implementations, the behavior switching conditions evaluated by the SIM switching logic 306 may be defined by one or more of the following: traffic conflicts involving non-AV actors and AVs, a divergence threshold (e.g., a divergence threshold measured in meters) between the AV's planned trajectory and its recorded trajectory, a current distance threshold (e.g., a distance threshold measured in meters) between the intelligent AV and the intelligent non-AV actor, the background context of the scenario involving other non-AV actors in the simulation, changes in the state of dynamic elements (e.g., pedestrians, animals, birds, traffic lights, weather, time, etc.) other than non-AV actors in the simulation, the presence of static elements (e.g., road signs, unexpected obstacles, traffic barriers, etc.), and proximity relative to traffic infrastructure in the simulation (e.g., the proximity of the non-AV actor to the upcoming intersection, the proximity of the non-AV actor to the upcoming merging point, etc.). If the behavior switching conditions are met, the SIM switching logic 306 instructs the SIM actor intelligence logic 308 to modify the behavior of the non-AV actor in the simulation to deviate from its recorded trajectory.

[0072] The SIM switching logic 306 determines the log future trajectory of a non-AV actor. For example, the SIM switching logic 306 queries the original log file for the future trajectory of the non-AV actor during the simulation. The SIM switching logic 306 also estimates the predicted future trajectory of the AV actor in the simulation. For example, the SIM switching logic 306 estimates the predicted future trajectory of the AV actor by querying the predicted state of the AV actor in the simulation. The SIM switching logic 306 uses the log future trajectory of the non-AV actor and the predicted future trajectory of the AV actor to determine whether a traffic conflict will occur between the non-AV actor and the AV actor at a future spatiotemporal point in the simulation. Specifically, the SIM switching logic 306 reconstructs the predicted future trajectory of the AV actor into the OCC reference frame of the non-AV actor, thereby determining the longitudinal, lateral, and height displacements of the AV actor relative to the non-AV actor's route. The SIM switching logic 306 iteratively steps forward along the reconstructed predicted future trajectory of the AV actor at a given discretization level until a given future look-ahead time point is reached. In one example, the discretization level can be defined by a discrete time step. At each discretization level, the SIM switching logic 306 determines whether a traffic conflict will occur between the non-AV actor and the AV actor. For example, traffic conflicts may include impending collisions, merging navigation, intersection navigation, and following another vehicle in the same lane. If a traffic conflict is expected to occur in the simulation, the SIM handover logic 306 will identify groups containing both non-AV actors and AV actors in the simulation as part of the traffic conflict. The SIM handover logic 306 will then send this group to the SIM actor intelligence logic 308, thereby selectively enabling intelligent behaviors for the intelligent actors within that group.

[0073] The SIM actor intelligence logic 308 receives a group associated with traffic conflict from the SIM switching logic 306, identifies intelligent non-AV actors within that group, and modifies the behavior of the non-AV actors to deviate from the recorded trajectory to avoid unrealistic behaviors that could lead to suboptimal results involving AV in the simulation. The SIM actor intelligence logic 308 uses one or more machine learning models associated with the intelligence of the non-AV actors to modify their behavior. For example, the SIM actor intelligence logic 308 uses actor behavior models to modify the acceleration of the non-AV actor along its longitudinal component of the OCC reference frame in the simulation. In some implementations, the SIM actor intelligence logic 308 may have access to various different machine learning behavior models associated with the non-AV actors, which can elicit different intelligent behaviors in the simulation. The SIM actor intelligence logic 308 can select the minimum longitudinal acceleration for the non-AV actor from various intelligent behaviors. For example, the SIM actor intelligence logic 308 may select the minimum acceleration as a heuristic for determining the safest among different intelligent behaviors.

[0074] Assuming that traffic conflicts are associated with driving in the same conflicting lane, the SIM actor intelligence logic 308 will determine an acceleration for non-AV actors to maintain a safe following distance behind the leading vehicle (e.g., AV).

[0075] Assuming that a traffic conflict is associated with an impending collision, the SIM actor intelligence logic 308 will determine an acceleration for the non-AV actor to prevent a collision with the AV.

[0076] Assuming a traffic conflict is associated with merging in navigation, the SIM (Signaling Activated Vehicle) intelligent logic 308 will determine whether a non-AV (Action Vehicle) must yield to an AV in the merging navigation. If the non-AV must yield, the SIM intelligent logic 308 will determine the acceleration for the non-AV to yield to the AV.

[0077] Assuming a traffic conflict is associated with navigation through an intersection, the SIM (Signaling Entity) intelligent logic 308 determines whether a non-AV (Action Vehicle) has the right-of-way in the intersection navigation. If the AV has the right-of-way, the SIM intelligent logic 308 determines the acceleration required for the non-AV to stop at the stop line at the intersection, and then continues driving when it is the non-AV's turn to cross the intersection.

[0078] In implementations where there is a possibility of more than one of the aforementioned traffic conflict scenarios, the SIM actor intelligent logic 308 will select the minimum value among various different accelerations as the acceleration of the non-AV actor in the simulation.

[0079] Once the first non-AV actor selectively deviates from its recorded trajectory, a cascading effect can occur, forcing other non-AV actors in the simulation to adapt their behavior to that of the first. For example, a second non-AV actor might follow the first non-AV actor's recorded trajectory, thus becoming part of a group involved in traffic conflict in the simulation. SIM switching logic 306 determines whether the behavior switching conditions for the second non-AV actor associated with the first non-AV actor are met. If so, SIM actor intelligence logic 308 begins modifying the behavior of the second non-AV actor to deviate from its recorded trajectory in the simulation.

[0080] To further demonstrate the selective activation of intelligent behavior by non-AV actors in simulation, an example implementation of enabling non-AV actors with merging navigation intelligence is described in detail below.

[0081] SIM switching logic 306 detects whether there are merging traffic entities ahead in the simulation. These merging traffic entities may include both AV (Action Vehicle) and non-AV (Non-AV) actors. SIM switching logic 306 queries the predicted state of the AV (i.e., the merging traffic entity) in the simulation and estimates its predicted future trajectory. SIM switching logic 306 reconstructs the predicted future trajectory of the AV into the OCC (Optical Control Center) reference frame of the non-AV actors involved in the merging traffic conflict. In the simulation, SIM switching logic 306 iteratively steps forward along the reconstructed predicted future trajectory of the AV at a specific discretization level until a given future look-ahead time point is reached. During this discretization process, for each predicted state, SIM switching logic 306 determines whether the AV overlaps with the log future trajectory of a non-AV actor. That is, SIM switching logic 306 determines whether the following conditions are met: The first condition checks whether the absolute value of the lateral offset of the AV within the OCC reference frame of the non-AV actor is less than the sum of a first multiple of the width of the non-AV actor and a second multiple of the width of the AV. The second condition checks whether the predicted state orientation of the AV is consistent with the path direction of the non-AV actor. The second condition is to ensure that non-merging intersections are not considered merging situations. If the predicted state of the AV meets the above condition, the SIM switching logic 306 will detect a merging traffic conflict ahead in the simulation. In some implementations, the SIM switching logic 306 can use a geometry-heuristic model to detect whether there is a merging traffic ahead in the simulation.

[0082] The SIM actor intelligence logic 308 can use a machine learning decision model to determine discrete decisions for the non-AV actor involved in a traffic conflict. For example, in an implementation of merging navigation intelligence, the SIM actor intelligence logic 308 uses a merging decision model to determine whether the non-AV actor should yield to the merging actor (e.g., AV) in a merging traffic conflict detected by the SIM switching logic 306. Other example discrete decisions output by the decision model might include whether the non-AV actor should follow another intelligent actor, lead another intelligent actor, or travel alongside another intelligent actor. The SIM actor intelligence logic 308 inputs immediate features of the current and / or predicted states of the non-AV actor and the merging actor into the merging decision model, thereby outputting the probability that the non-AV actor needs to yield to the merging actor. For example, the current and / or predicted states might include speed, acceleration, distance from the merging point, etc., and the merging decision model might be a logistic regression model with several hidden layers between the input features and the logic output. In some implementations, the SIM actor intelligent logic 308 may also input other background input features (such as lane priority, speed limits, information about other actors, etc.) into the merging decision model to generate a logical output. The output decision of the merging decision model can also be parameterized by an assertiveness parameter. This parameter varies in value between 0.0 (always yield) and 1.0 (never yield), and represents the confidence threshold of the merging decision model when making a yielding decision about non-AV actors.

[0083] The SIM (Signal-Induced Traffic Flow) intelligent logic 308 can use machine learning behavioral models to modify the behavior of non-AV (Action-Induced) actors in traffic conflicts. The behavioral model can output continuous actions / reactions that modify the behavior of non-AV actors based on discrete decisions from a decision model. For example, in an implementation of merging navigation intelligence, the SIM intelligent logic 308 uses a merging behavior model of the non-AV actor to determine a merging acceleration for that non-AV actor that matches the yielding decision of the merging decision model. In some implementations, the SIM intelligent logic 308 also uses an IDM (Induced Traffic Flow) version implementing ACC (Adaptive Traffic Flow) behavior to determine the merging acceleration of the non-AV actor. If the yielding decision is not to yield, the SIM intelligent logic 308 ignores the merging actor and uses the normal acceleration curve of the non-AV actor in the simulation. If the yielding decision is to yield, the SIM intelligent logic 308 reconstructs the current state of the merging actor to the OCC (Operational Traffic Flow) reference frame of the non-AV actor. The SIM (Signaling Module) intelligent logic 308 determines the merging acceleration by projecting the pose of the merging behavior onto the path of the non-AV (Action Vehicle) behavior and using other data such as the current speed of the non-AV behavior, the desired speed of the non-AV behavior, the speed of the merging behavior, and the headway between the non-AV behavior and the merging behavior. The current speed of the non-AV behavior refers to its longitudinal speed in its own OCC (Optical Control Center) reference frame. The speed of the merging behavior refers to its longitudinal speed along the OCC reference frame of the non-AV behavior. The desired speed of the non-AV behavior refers to the speed specified at the next control point along the path of the non-AV behavior. The headway between the non-AV behavior and the merging behavior is the difference between the longitudinal offset of the non-AV behavior in its own OCC reference frame and the longitudinal offset of the merging behavior in the OCC reference frame of the non-AV behavior. In some implementations, the SIM intelligent logic 308 inputs immediate characteristics of the current and / or predicted states of the non-AV behavior and the current and / or predicted states of the merging behavior into the merging behavior model of the non-AV behavior to determine the merging acceleration. For example, a co-current behavior model might be trained based on recorded trajectories of actors in real-world driving data.

[0084] The simulation verification engine 208 can monitor and verify the simulations executed by the simulation execution engine 206 based on multiple simulation scenarios. Simulations typically have many different modules, and each module generates and sends several messages regarding the simulation execution status during execution. The simulation execution engine 206 may be configured to send messages to the simulation verification engine 208 for processing in real-time or with a predetermined delay during simulation execution. In some implementations, the simulation verification engine 208 may process the simulation results and / or simulation logs 215 after the simulation is completed. The simulation verification engine 208 processes the messages and automatically detects one or more events that occur during simulation execution. The simulation verification engine 208 can generate verification results associated with verifying the simulation scenario and store them in the simulation log 216. In some implementations, the simulation management engine 204 analyzes the verification results and identifies events or other interesting behaviors in the simulation based on the simulation scenario. The simulation management engine 204 may map the analysis of the verification results back to the simulation data 212, thereby reconfiguring and creating variations of the simulation scenario using intelligent actors. The intelligent simulation scenario can be executed again in the simulation by the simulation execution engine 206, and then verified by the simulation verification engine 208.

[0085] refer to Figure 3C This document details an example implementation of the simulation verification engine 208. The simulation verification engine 208 may include a SIM verifier 310 and a SIM metric generator 312 to verify multiple simulation scenarios.

[0086] The SIM verifier 310 receives multiple messages broadcast by the simulation execution engine 206. These messages may correspond to time-series messages originally generated during the execution of one or more simulations. In some implementations, the SIM verifier 310 may process the time-series data until a specific time point (latch point) is reached or a specific condition is met; subsequently, it will ignore subsequent messages based on the time-series data up to that time point. The SIM verifier 310 may be configured with verification conditions. Verification conditions can be used to check for suboptimal outcomes in traffic interactions involving the AV and other actors in the simulation environment. Examples of suboptimal outcomes include, but are not limited to: rapid acceleration, speeding, sudden braking, deceleration, failure to yield, failure to recognize obstacles, failure to recognize road signs, near collision, crossing road boundaries, intrusion into oncoming lanes, lane drift, collision, etc. The SIM verifier 310 performs verification checks on the received messages. The verification checks performed by the SIM verifier 310 determine whether the message value satisfies the constraints specified by the verification conditions. The components of the verification conditions may use Boolean operators or other operators such as "=", ">", "=", "<=", and "<". These Boolean operators and other operators can be binary operators (i.e., involving two operands), but are not necessarily limited to binary form; for example, "a <= x <= b" or "(x and (AND) y and z)". The SIM verifier 310 generates a status message indicating whether the verification check performed on the processed message was successful (passed) or failed (failed), and sends this status message to the SIM indicator generator 312. In other words, the SIM verifier 310 verifies whether its preset conditions have been met.

[0087] The SIM verifier 310 can implement a suboptimal result verifier, which assists in marking potential true positive suboptimal results in the simulation. The suboptimal result verifier receives a status message containing information about the state of the bounding boxes representing actors in the simulation and determines whether verification conditions are met. For example, a collision verifier can check for overlap between the bounding box of the AV and the bounding boxes of one or more other actors in the simulation. If an overlap is detected between the bounding box of the autonomous vehicle and the bounding boxes of other actors, the collision verifier determines that a collision has occurred. The collision verifier generates a status message indicating whether the collision check for the processed message was successful (i.e., a collision was detected) or failed (i.e., no collision was detected), and sends this status message to the SIM indicator generator 312.

[0088] The SIM verifier 310 can implement a scene divergence verifier, which verifies the scene divergence of an actor relative to a recorded trajectory in a simulation. The scene divergence verifier receives state messages containing the speed, distance, and travel time associated with the actor observed in the simulation before and after the actor's intelligence is selectively enabled. The scene divergence verifier determines whether the actor's scene divergence is above or below a preset threshold after the actor's intelligence is selectively enabled. For example, the scene divergence threshold could be 100 meters. If the scene divergence is above the threshold, the scene divergence verifier determines that the actor in the simulation has diverged from its original logs. The scene divergence verifier generates a state message indicating whether the scene divergence check for the actor was successful (i.e., scene divergence above the threshold was detected) or failed (i.e., scene divergence below the threshold was detected), and sends this state message to the SIM metric generator 312.

[0089] The SIM metric generator 312 determines the values ​​of one or more performance metrics associated with actors in the simulation scenario based on the execution of the simulation. Performance metrics can be statistical data determined for the AV 100 across multiple AV performance dimensions, such as safety and comfort. The SIM metric generator 312 receives state messages from the SIM verifier 310 and determines metrics associated with the AV's performance based on the execution of the simulation. These metrics are used to evaluate the performance of the tested AV capabilities (such as perception, path planning, or vehicle control). Metrics measure the number of suboptimal outcomes involving the AV during the simulation. In one example, the simulation metric generator 312 receives collision-related state messages from the simulation verifier 310 and determines a collision metric for AV-actuator collisions based on the execution of the simulation. This collision metric measures the number of AV-actuator collisions. In another example, the simulation metric generator 312 receives scene divergence-related state messages from the simulation verifier 310 and determines a scene divergence metric associated with the actor based on the execution of the simulation. This scene divergence metric quantitatively measures the degree to which the actor deviates from its recorded trajectory. The simulation metric generator 312 can be configured to extract and differentiate scenario divergence metrics before and after selectively enabling agent intelligence in a simulation.

[0090] It should be understood that the AV-actor collision index generated by the simulation index generator 312 is the sum of true positive AV-actor collisions and false positive AV-actor collisions. In some embodiments, the simulation index generator 312 receives tags collected for true positive AV-actor collisions regarding a first list of simulation scenarios using scripted actors (non-intelligent simulation scenarios) and a second list of simulation scenarios using intelligent actors (intelligent simulation scenarios). The simulation index generator 312 calculates the precision and recall of the collision index associated with these two lists of simulation scenarios for evaluation. The precision of the collision index can be defined as the proportion of true positive AV-actor collisions out of all detected AV-actor collisions (i.e., the sum of true positive AV-actor collisions and false positive AV-actor collisions). The recall of the collision index can be defined as the proportion of true positive AV-actor collisions out of the actual number of all detected true positive AV-actor collisions (i.e., the sum of true positive AV-actor collisions and false negative AV-actor collisions). The simulation metric generator 312 determines that the accuracy of the collision metrics for the intelligent simulation scene list is improved compared to the non-intelligent simulation scene list. This is because the number of false positive AV-actuator collisions is reduced by enabling actors to avoid unrealistic collisions. The recall rate of the collision metrics for the intelligent simulation scene list is at least maintained.

[0091] like Figure 2 As shown, the computing system 172 includes a machine learning engine 166 to train one or more machine learning models 224. For example, the one or more machine learning models 224 may be agent behavior models. In some implementations, the machine learning engine 166 receives training data from the intelligent SFL system 160 for training the machine learning models 224. For example, the training data may include features from this set of log segments and OE, VM, and OEDR label classifications provided for each log segment.

[0092] like Figure 2As shown, in some embodiments, once the intelligent SFL system 160 has verified that the intelligent simulation scenario is suitable for training the machine learning model 224, the machine learning engine 166 can use the intelligent simulation scenario as a training example to train the machine learning model 224. In some embodiments, if no intelligent agent is assigned to the simulation scenario, the corresponding simulation scenario and its simulation data may be disqualified from being used to train the machine learning model 224 of the autonomous vehicle 100. In some embodiments, the machine learning model 224 is a neural network model and includes one and / or multiple layers of storage units, each of which has corresponding weights. Various neural network models can be used, including feedforward neural networks, convolutional neural networks, recurrent neural networks, radial basis functions, other neural network models, and combinations of several neural networks. Additionally or alternatively, in addition to neural networks, the machine learning model 224 may also represent various machine learning techniques, such as support vector machines, decision trees, Bayesian networks, random decision forests, k-nearest neighbors, linear regression, least squares, other machine learning techniques, and / or combinations of machine learning techniques.

[0093] Machine learning model 224 can be trained for various autonomous vehicle tasks, including determining the location of a target autonomous vehicle, generating one or more signals to control the autonomous vehicle, tracking or identifying objects in the autonomous vehicle's environment, etc. For example, a neural network model can be trained to recognize traffic lights in an environment with autonomous vehicle 100 present. As a further example, a neural network model can be trained to predict the brand and model of other vehicles in an environment with autonomous vehicle 100 present. In many implementations, the neural network model can be trained to perform a single task. In other implementations, the neural network model can be trained to perform multiple tasks.

[0094] Machine learning engine 166 can generate training instances from a simulated scenario to train machine learning model 224. For example, training instances may include instances of simulated autonomous vehicle data, where the autonomous vehicle 100 is capable of detecting stop signs using simulated sensor data from one or more sensors, and labels corresponding to simulated outputs that stop the autonomous vehicle in the simulated scenario. Machine learning engine 166 can apply the training instances as input to machine learning model 224. In some implementations, machine learning model 224 can be trained using any of at least one of the following learning methods: supervised learning (e.g., support vector machines, neural networks, logistic regression, linear regression, stacking, gradient boosting, etc.), unsupervised learning (e.g., clustering, neural networks, singular value decomposition, principal component analysis, etc.), or semi-supervised learning (e.g., generative models, transductive support vector machines, etc.). Additionally or alternatively, the machine learning model according to some implementations may be a deep learning network including recurrent neural networks, convolutional neural networks (CNNs), networks that are combinations of multiple networks, etc. For example, machine learning engine 166 can generate predicted machine learning model outputs by applying training inputs to machine learning model 224. Additionally or alternatively, machine learning engine 166 may compare the predicted machine learning model output with known outputs from training instances (e.g., simulation outputs in a simulation scenario) and use this comparison to update one or more weights in machine learning model 224. In some implementations, one or more weights may be updated by backpropagating the differences across the entire machine learning model 224.

[0095] Machine learning engine 166 can test a trained machine learning model according to several implementations. Machine learning engine 166 can generate test instances using a simulation scenario and a simulated autonomous vehicle in the simulation scenario where machine learning model 224 has been trained for a specific autonomous vehicle task. Machine learning engine 166 can apply the test instances as input to the trained machine learning model 224. The predicted output generated by applying the test instances to the trained machine learning model 224 can be compared with the known output of the test instance (i.e., the simulation output observed in the simulation) to update the accuracy value (e.g., percentage accuracy) of machine learning model 224.

[0096] Now for reference Figure 4 The diagram illustrates a method 400 for evaluating the performance of an autonomous vehicle according to some embodiments. Method 400 may be a sequence of operations or process steps performed by a system of one or more computers at one or more locations, such as including… Figure 2 The intelligent SFL system 160 in the computing system 172 is independent of Figure 2The intelligent SFL system 160 may be another computer system, or any combination thereof. Furthermore, while in some embodiments the sequence of operations may be fully automated, in other embodiments some steps may be performed and / or guided by human intervention. It will also be understood that the order of operations in the sequence may be changed, and in some embodiments some operations may be performed in parallel and / or iteratively.

[0097] At block 402, the system determines the simulation scenario from the log data, which includes non-AV actors. These non-AV actors are enabled to selectively deviate from their recorded trajectories in the log data during simulation. For example, the non-AV actor could be an IPF actor with a set of selective deviation capabilities from recorded trajectories, such as merging interactive reasoning, adaptive cruise control (ACC), intersection navigation processing, collision avoidance, etc.

[0098] In block 404, the system performs the simulation based on the simulation scenario.

[0099] In block 406, the system monitors the execution of the simulation.

[0100] In block 408, the system determines, based on monitoring, whether behavior switching conditions for non-AV actors associated with traffic interactions including AV are met. The behavior switching conditions for non-AV actors are evaluated in conjunction with changes in the behavior of AV and / or other non-AV actors (e.g., recorded trajectories) during the simulation. The system uses the evaluation of the behavior switching conditions to identify the time step in the simulation that would cause a non-AV actor to switch from following a trajectory to intelligent behavior.

[0101] If, in block 408, the system determines that the behavior switching conditions for non-AV actors associated with traffic interactions including AVs are not met, the system loops back to block 406. The system continues to monitor the execution of the simulation.

[0102] On the other hand, if in block 408 the system determines that the behavior switching conditions for non-AV actors associated with traffic interactions including AV are met, the system proceeds to block 410.

[0103] In block 410, the system uses a behavior model to modify the behavior of the non-AV actor to deviate from its recorded trajectory. For example, in response to changes in the behavior of the AV during simulation (e.g., the planned trajectory of the AV deviates from its recorded trajectory in the log data) and / or the behavior of another intelligent actor, the non-AV actor decides to selectively deviate from the recorded trajectory.

[0104] In block 412, the system determines performance metrics based on simulation. These performance metrics are used to evaluate the AV's capabilities under test, such as perception, path planning, or vehicle control. The performance metrics measure the number of suboptimal AV performance outcomes that occur during simulation.

[0105] Now for reference Figure 5 The diagram illustrates a method 500 for modifying the behavior of a non-AV actor according to some embodiments. The method 500 may be a sequence of operations or process steps performed by a system of one or more computers at one or more locations, such as including… Figure 2 The intelligent SFL system 160 in the computing system 172 is independent of Figure 2 The intelligent SFL system 160 may be another computer system, or any combination thereof. Furthermore, while in some embodiments the sequence of operations may be fully automated, in other embodiments some steps may be performed and / or guided by human intervention. It will also be understood that the order of operations in the sequence may be changed, and in some embodiments some operations may be performed in parallel and / or iteratively.

[0106] In block 502, the system determines the future recorded trajectories of non-AV actors in the simulation. For example, the system performs a simulation based on an intelligent simulation scenario. The system queries the raw log file for the future recorded trajectories of non-AV actors in the simulation.

[0107] In block 504, the system determines the future trajectory of the AV in the simulation. For example, the system determines the future trajectory planned by the AV by querying the predicted state of the AV in the simulation.

[0108] In block 506, the system projects the future trajectory of the AV onto the reference frame of the non-AV actors. For example, the reference frame of the non-AV actors is an orthogonal curvilinear coordinate (OCC) reference frame. In the simulation, the non-AV actors perceive the world in the OCC reference frame defined by their path, rather than in a Cartesian coordinate system. The non-AV actors perceive their positions in the simulation based on the longitudinal, lateral, and height displacements of other actors along their own paths.

[0109] In block 508, the system determines the next discrete time step of the future trajectory projected along AV.

[0110] In block 510, the system determines whether a non-AV actor will have a traffic conflict with an AV. In some implementations, the system determines whether a non-AV actor will have a traffic conflict with an AV at a future spatiotemporal point in the simulation. For example, traffic conflicts may include impending collisions, merging navigation, intersection navigation, following another vehicle in the same lane, etc.

[0111] If at block 510 the system determines that there is no traffic conflict between a non-AV actor and an AV, the system loops back to block 508. The system continues to iteratively advance along the future trajectory of the AV's projection in discrete time steps during the simulation.

[0112] On the other hand, if in block 510 the system determines that a non-AV actor has a traffic conflict with an AV actor, the system proceeds to block 512.

[0113] In block 512, the system modifies the acceleration of non-AV actors during simulation. For example, the system uses actor behavior models to modify the longitudinal component of the acceleration of the non-AV actor along its OCC reference frame during simulation. In some implementations, the system can access different behavior models associated with the non-AV actor that can initiate different intelligent behaviors during simulation. The system can select the minimum longitudinal acceleration for the non-AV actor from different intelligent behaviors. For example, the system selects the minimum acceleration as a heuristic method for the safest case among different intelligent behaviors.

[0114] The preceding description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be understood, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are given the widest possible scope consistent with the language of the claims, wherein reference to an element in the singular is not intended to mean “one and only one” (unless otherwise stated), but rather “one or more.” Unless otherwise specifically stated, the term “some” means one or more. All structural and functional equivalents of the elements throughout the various aspects described in the foregoing description that are known or will be known by those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, whether such disclosures are expressly recited in the claims or not, nothing disclosed herein is intended to be offered to the public. No element of any claim may be construed as means plus function unless that element is expressly described using the phrase “means for…”.

[0115] It is understood that the specific order or hierarchy of the blocks in the disclosed process is an example of illustrative method. Based on design preferences, it is understood that the specific order or hierarchy of the blocks in the process can be rearranged while remaining within the scope of the foregoing description. The appended method claims present the elements of the blocks in the order of the examples, but this does not imply limitation to the specific order or hierarchy presented.

[0116] The prior description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the disclosed subject matter. Various modifications to these embodiments will be apparent, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the prior description. Therefore, the prior description is not intended to limit it to the embodiments shown herein, but should be given the maximum scope consistent with the principles and novel features disclosed herein.

[0117] The various examples illustrated and described are provided merely as examples illustrating the features of the claims. However, the features shown and described with respect to any given example are not necessarily limited to the relevant example and may be used in combination with or in use with other examples shown and described. Furthermore, the claims are not intended to be limited by any single example.

[0118] The foregoing method descriptions and process flowcharts are provided as illustrative examples only and are not intended to require or imply that the blocks in the examples must be executed in the order presented. As will be understood, the blocks in the preceding examples can be executed in any order. Words such as “afterward,” “then,” and “following” are not intended to restrict the order of these blocks; these words are only used to guide the reader through the description of these methods. Furthermore, any reference to singular claim elements, such as the use of the articles “a,” “an,” or “the,” should not be construed as limiting the element to the singular.

[0119] The various illustrative logic blocks, modules, circuits, and algorithm blocks described in conjunction with the examples disclosed herein can be implemented using electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionalities of the various illustrative components, blocks, modules, circuits, and blocks have been generally described above. Whether these functions are implemented in hardware or software depends on the specific application and the design constraints imposed on the system as a whole. A skilled craftsman can implement the described functions in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0120] The hardware used to implement the various illustrative logics, logic blocks, modules, and circuits disclosed herein may be implemented or performed using a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device designed to perform the functions described herein, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor may be a microprocessor, but in alternative embodiments, it may also be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors associated with a DSP core, or any other such configuration. Alternatively, certain blocks or methods may be executed by a circuit system dedicated to a specific function.

[0121] In some examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, these functions may be stored as one or more instructions or code on a non-transient computer-readable storage medium or a non-transient processor-readable storage medium. The blocks of the methods or algorithms disclosed in this invention may be implemented in processor-executable software modules residing on non-transient computer-readable or processor-readable storage media. Non-transient computer-readable or processor-readable storage media may be any storage medium accessible by a computer or processor. By way of example, but not limitation, such non-transient computer-readable or processor-readable storage media may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, or any other medium suitable for storing desired program code in the form of instructions or data structures accessible by a computer. The disks and discs used herein include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically copy data magnetically, while optical discs copy data optically using lasers. The above combinations are also included within the scope of non-transient computer-readable and processor-readable media. Furthermore, the operations of a method or algorithm may reside as a single, arbitrary combination, or set of code and / or instructions on a non-transient processor-readable and / or computer-readable storage medium, which may be incorporated into a computer program product.

[0122] The descriptions of the previously disclosed examples are provided to enable others to make or use this disclosure. Various modifications to these examples will readily become apparent to those skilled in the art, and the general principles defined herein may be applied to certain examples without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples shown herein, but should be accorded the widest scope consistent with the principles and innovative features disclosed herein.

Claims

1. A method for evaluating the performance of an autonomous vehicle (AV), the method comprising: The simulation scenario is determined from log data, the simulation scenario including a first non-AV actor, the first non-AV actor being enabled to selectively deviate from its recorded trajectory in the log data during simulation; The simulation will be performed based on the aforementioned simulation scenario; Monitor the execution of the simulation; Based on the monitoring, it is determined whether the behavior switching conditions associated with the first non-AV actor for traffic interaction including the AV are met during the simulation. In response to determining that the behavior switching conditions for the first non-AV actor associated with the traffic interaction including the AV are met during the simulation, a first behavior model is used to modify the behavior of the first non-AV actor to deviate from its recorded trajectory; as well as The performance metrics are determined based on the simulation.

2. The method according to claim 1, further comprising: In response to modifying the behavior of the first non-AV actor to deviate from its recording trajectory, it is determined whether a behavior switching condition for a second non-AV actor associated with the first non-AV actor is met during the simulation, the second non-AV actor being included in the simulation scenario and being enabled to selectively deviate from its recording trajectory in the log data during the simulation. as well as In response to determining that the behavior switching conditions for the second non-AV actor associated with the first non-AV actor are met during the simulation, a second behavior model is used to modify the behavior of the second non-AV actor to deviate from its recorded trajectory.

3. The method according to claim 1, wherein, Determining whether behavior switching conditions for the first non-AV actor associated with the traffic interaction including the AV are met during the simulation includes: Determine the future recorded trajectory of the first non-AV actor in the simulation; Determine the future trajectory of the AV in the simulation; and Using the future recorded trajectory of the first non-AV actor and the future trajectory of the AV, determine whether the first non-AV actor has a traffic conflict with the AV at a future spatiotemporal point in the simulation. The traffic conflict defines the behavior switching conditions.

4. The method according to claim 3, wherein, Determining whether the traffic conflict between the first non-AV actor and the AV exists at a future spatiotemporal point in the simulation using the future recorded trajectory of the first non-AV actor and the future trajectory of the AV includes: Projecting the future trajectory of the AV onto the reference frame of the first non-AV actor; and Along the projected future trajectory of the AV, at each of a plurality of discrete time steps, it is determined whether the first non-AV actor has the traffic conflict with the AV.

5. The method according to claim 1, wherein, Modifying the behavior of the first non-AV actor to deviate from its recorded trajectory includes: Modify the acceleration of the first non-AV actor.

6. The method according to claim 5, wherein, Modifying the acceleration of the first non-AV actor includes: Determine the acceleration required to keep the first non-AV actor at an acceptable following distance behind the AV in the same lane as the AV.

7. The method according to claim 5, wherein, Modifying the acceleration of the first non-AV actor includes: Determine the acceleration of the first non-AV actor used to prevent collision with the AV.

8. The method according to claim 5, wherein, Modifying the acceleration of the first non-AV actor includes: Determine whether the first non-AV actor must yield to the AV in a traffic conflict, wherein the traffic conflict is a merging; and In response to determining that the first non-AV actor must yield to the AV in the traffic conflict, the acceleration for the first non-AV actor to yield to the AV is determined.

9. The method according to claim 5, wherein, Modifying the acceleration of the first non-AV actor includes: Determine whether the first non-AV actor has the right-of-way in a traffic conflict with the AV, wherein the traffic conflict is at an intersection; and In response to determining that the first non-AV actor does not have the right-of-way in the traffic conflict with the AV, the acceleration for the first non-AV actor to stop at the intersection is determined.

10. The method according to claim 1, wherein, The behavior switching condition assessment is selected from at least one of the following groups: During the traffic interaction with the first non-AV actor, the AV deviates from its recorded trajectory in the log data. The divergence threshold between the planned trajectory of the AV and its recorded trajectory in the log data The current distance threshold between the AV and the first non-AV actor. Aside from changes in the state of dynamic elements other than non-AV actors. The existence of static elements, and Compared to proximity to transportation infrastructure.

11. The method according to claim 1, wherein, Determining performance metrics based on the simulation further includes: An evaluation set of simulation scenarios is determined, wherein each simulation scenario in the evaluation set includes one or more non-AV actors that are enabled to selectively deviate from their recorded trajectory; For each simulation scenario in the evaluation set, the simulation is executed; and Based on the simulation, the accuracy and recall of the performance metrics are determined.

12. The method according to claim 1, wherein, The performance metric measures an example of a suboptimal result in the performance of the AV.

13. The method according to claim 2, wherein, The first behavior model and the second behavior model are machine learning models, each of which is trained based on log records of multiple actors in real-world driving data.

14. The method according to claim 2, wherein, A nonlinear least squares optimization method is used to optimize the parameters of the first behavior model and the second behavior model to follow the recorded trajectory.

15. A system comprising one or more processors and a memory operatively coupled to said one or more processors, wherein, The memory stores instructions that, in response to execution by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 14.

Images