Autonomous Vehicle Verification System and Method

Abstract

An exemplary method comprises the steps of: obtaining log data describing the reference action of a reference vehicle in an environment—which is the reference action occurring in the initial state of the environment; determining the planned action of the vehicle being simulated in the initial state of the environment using an operating system; simulating the SUT state of the environment obtained by the vehicle being simulated executing the planned action in the initial state of the environment and an actor performing an actor action following the planned action, and simulating the reference state of the environment obtained by the vehicle being simulated executing the reference action in the initial state of the environment and an actor performing an actor action following the reference action; determining a test score based on the SUT state and a reference score based on the reference state; and evaluating the operating system based on the test score and the reference score.

Description

Technology Field

[0001] This application claims priority and interest to U.S. Provisional Application No. 63 / 491,644 (filed March 22, 2023) and U.S. Patent Application No. 18 / 309,494 (filed April 28, 2023). U.S. Provisional Application No. 63 / 491,644 and U.S. Patent Application No. 18 / 309,494 are incorporated herein by reference in their entirety. Background Technology

[0002] Autonomous platforms can process data to perceive the environment in which they are driving. For example, autonomous vehicles use various sensors to perceive their surroundings and identify objects around them. Through the perceived environment, autonomous vehicles can identify an appropriate path and travel along it with minimal or no human intervention. Prior art literature

[65535] US Patent Published US 2021-0403033 (Dec. 30, 2021) US Patent Published US 2021-0387628 (Dec. 16, 2021)

[0003] The present disclosure relates to techniques for verifying systems for operating autonomous vehicles. The verification techniques according to the present disclosure may comprehensively evaluate an end-to-end autonomous computing pipeline or a selected portion thereof. Exemplary embodiments may deploy a simulated autonomous vehicle into a driving scenario and compare the planned behavior of said autonomous vehicle with the behavior of a reference vehicle within the same scenario.

[0004] For example, log data can record the behavior of human reference subjects in various driving scenarios. Exemplary implementations can simulate the results obtained by a simulated autonomous vehicle executing planned actions, and can simulate the results obtained by a simulated autonomous vehicle executing reference actions. In this way, the results achieved by the operating systems of the autonomous vehicle can be verified against a baseline provided by human reference subjects.

[0005] The verification techniques according to the present disclosure can leverage the expertise of human drivers when facing specific scenarios (e.g., edge case scenarios). For example, a human driver may apply the brakes or reduce acceleration when driving through a road merging zone when there are other vehicles merging into the driver's lane, even though the driver generally has the right of way in the lane. The driver's choice reflects a complex balance of risk and cost: namely, the risk of sudden early entry by another vehicle, the cost of being unprepared for such sudden early entry, and the cost of the driver decelerating. Therefore, broadly speaking, the collective behavior of human reference subjects can provide a framework for understanding the expected behavior of vehicles in various driving scenarios. Accordingly, one exemplary technique for evaluating autonomous vehicle behavior (and the operating system controlling such behavior) involves comparisons with human reference subjects.

[0006] In some examples, directly comparing autonomous vehicle behavior to human reference objects involves counting the number of interventions by the human operator supervising the autonomous vehicle. The goal may be to reduce the number of interventions so that the autonomous vehicle drives naturally in a manner that the human operator expects and accepts.

[0007] In some examples, verification techniques according to the present disclosure can determine proxy indicators of human operator intervention events by comparing the behavior of reference targets recorded in a given driving scenario (e.g., using log data) with the planned behavior of an autonomous vehicle deployed in the same scenario. By evaluating the differences between the behaviors—and the potential consequences therefrom—the operating system of the autonomous vehicle can be evaluated against a baseline.

[0008] Advantageously, the verification technique according to the present disclosure can evaluate the performance of operating systems for autonomous vehicles without requiring manually adjusted, hard-coded complex costs and arbitrary thresholds and ranges. Exemplary embodiments can evaluate the performance of operating systems for autonomous vehicles without the need to explicitly model decision heuristics by utilizing a large dataset recording millions of expert decision cases.

[0009] The verification techniques according to the present disclosure also have the advantage of obtaining direct comparisons with reference targets without requiring extensive and unlimited accumulation of actual or simulated driving distances and corresponding data samples related to the occurrence of potential scenarios. Such accumulation, even if simulated, can consume significant resources (e.g., computing resources, energy resources, etc.) and be time-consuming. Advantageously, direct comparisons with reference behaviors can be obtained at a much higher density by immersing the simulated autonomous vehicle in scenarios derived from existing log data.

[0010] The verification techniques according to the present disclosure can also advantageously perform a comprehensive evaluation of the impact of individual model developments on the overall results achieved by an autonomous vehicle. For example, an autonomous computing pipeline may implement a number of complex systems and models to perceive and understand an environment and to generate and execute plans for traversing that environment. Through a comprehensive performance evaluation against a reference baseline, individual operating systems and changes to those systems can be evaluated based on their impact on the overall performance of the vehicle. In this way, performance differences that deviate from reference operation can be prioritized in development over performance differences that do not ultimately contribute to performance defects. Accordingly, the verification techniques according to the present disclosure can support more efficient and effective system and model development.

[0011] For example, in one embodiment, the present disclosure provides an exemplary method for verifying an operating system for operating an autonomous vehicle. In some embodiments, the exemplary method comprises the step of (a) obtaining log data describing an exemplar action of a reference vehicle in an environment—the exemplar action is an action occurring in the initial state of the environment. In some embodiments, the exemplary method comprises the step of (b) determining a planned action for a vehicle being simulated in the initial state of the environment using the operating system. In some embodiments, the exemplary method comprises the step of (c) simulating (i) a system under test (SUT) state of the environment obtained by the vehicle being simulated executing a planned action in the initial state of the environment and an actor performing an actor action following the vehicle being simulated executing a planned action, and (ii) a reference state of the environment obtained by the vehicle being simulated executing a exemplar action in the initial state of the environment and an actor performing an actor action following the vehicle being simulated executing a exemplar action. In some embodiments, the exemplary method comprises the step of (d) determining a test score based on the SUT state of the environment and a reference score based on the reference state of the environment. In some implementations, the exemplary method includes (e) a step of evaluating the operating system based on test scores and reference scores.

[0012] In some embodiments of the exemplary method, the actor action includes a risk affecting the simulated vehicle. In some embodiments of the exemplary method, the reference action is a defensive driving action to prepare for the occurrence of a risk.

[0013] In some implementations of the exemplary method, the actor action is selected from among a plurality of actions having a probability lower than a threshold probability.

[0014] In some embodiments of the exemplary method, (c) includes a step of determining a response to a risk affecting the simulated vehicle.

[0015] In some implementations of the exemplary method, the reference operation includes the vehicle state of a human-driven vehicle.

[0016] In some embodiments of the exemplary method, the operating system includes at least one model of an autonomous vehicle control system configured to receive sensor data and control the movement of the autonomous vehicle. In some embodiments of the exemplary method, the operating system includes a motion planning model. In some embodiments of the exemplary method, the operating system includes a perception model.

[0017] In some implementations of the exemplary method, the reference action is a defensive driving action.

[0018] In some embodiments of the exemplary method, (c) (e.g., simulation of a first state and a reference state) includes the step of simulating an action on an actor in the environment following the vehicle being simulated executing a planned action and a reference action. In some embodiments of the exemplary method, the action is selected from a plurality of actions having a probability lower than a threshold probability. In some embodiments of the exemplary method, the action on the actor includes a risk affecting the vehicle being simulated.

[0019] In some embodiments of the exemplary method, (c) (e.g., simulation of a first state and a reference state) includes the step of simulating a response to a risk affecting the simulated vehicle.

[0020] In some implementations of the exemplary method, the test score is based on the state of the vehicle simulated in the environment's SUT state, and the reference score is based on the state of the vehicle simulated in the environment's reference state.

[0021] In some embodiments of the exemplary method, (d) (e.g., determination of test score) includes the step of evaluating the state of the vehicle simulated in the environment's SUT state and the state of the vehicle simulated in the environment's reference state using a cost function. In some embodiments of the exemplary method, the cost function is determined based on the distance to the actor boundary. In some embodiments of the exemplary method, the cost function is determined based on a measure of the severity of the intersection with the actor boundary.

[0022] In some implementations of the exemplary method, (e) (e.g., evaluation of an operating system) includes the step of classifying test scores and reference scores based on one or more features of the initial state of the environment.

[0023] In some embodiments of the exemplary method, the exemplary method comprises: (f) determining one or more categories for improvement related to suboptional test scores; and (g) adjusting one or more parameters of an operating system corresponding to the operation of an autonomous vehicle in one or more categories for improvement.

[0024] For example, in one embodiment, the present disclosure provides one or more exemplary non-transient computer-readable media storing executable instructions to cause one or more processors to perform operations. In some embodiments, the operations include (a) acquiring log data describing a reference operation of a reference vehicle in an environment—the reference operation is an operation occurring in the initial state of the environment. In some embodiments, the operations include (b) using an operating system to determine a planned operation for a vehicle being simulated in the initial state of the environment. In some embodiments, the operations include (c) simulating (i) the SUT state of the environment obtained by the vehicle being simulated executing the planned operation in the initial state of the environment and an actor performing an actor operation following the vehicle being simulated executing the planned operation, and (ii) the reference state of the environment obtained by the vehicle being simulated executing the reference operation in the initial state of the environment and an actor performing an actor operation following the vehicle being simulated executing the reference operation. In some embodiments, the operations include (d) determining a test score based on the SUT state of the environment and a reference score based on the reference state of the environment. In some implementations, operations include (e) an operation that evaluates the operating system based on test scores and reference scores.

[0025] In some embodiments of one or more non-transient computer-readable media, the actor action includes a risk affecting the simulated vehicle. In some embodiments of one or more exemplary non-transient computer-readable media, the reference action is a defensive driving action to prepare for the occurrence of a risk.

[0026] In some embodiments of one or more exemplary non-transient computer-readable media, an actor action is selected from a plurality of actions having a probability lower than a threshold probability.

[0027] In some embodiments of one or more exemplary non-transient computer-readable media, (c) includes an action that determines a response to a risk affecting a simulated vehicle.

[0028] In some embodiments of one or more exemplary non-transient computer-readable media, the reference operation includes the vehicle state of a human-driven vehicle.

[0029] In some embodiments of one or more exemplary non-transient computer-readable media, the reference operation is a defensive operation.

[0030] In some embodiments of one or more exemplary non-transient computer-readable media, (c) (e.g., simulation of a first state and a reference state) includes simulating an action for an actor in an environment following the vehicle being simulated executing a planned action and a reference action.

[0031] For example, in one embodiment, the present disclosure provides an exemplary autonomous vehicle control system for controlling an autonomous vehicle. In some embodiments, the exemplary autonomous vehicle control system comprises one or more processors and one or more non-transient computer-readable media storing instructions executable by one or more processors to enable the autonomous vehicle control system to control the operation of the autonomous vehicle using an operating system. In some embodiments, the operating system is verified by (a) acquiring log data describing a reference operation of a reference vehicle in an environment—the reference operation is an operation occurring in the initial state of the environment. In some embodiments, the operating system is verified by (b) determining a planned operation for a vehicle simulated in the initial state of the environment using the operating system. In some embodiments, the operating system was verified by (c) simulating (i) the SUT state of the environment obtained by the simulated vehicle executing a planned action in the initial state of the environment and the actor performing an actor action following the simulated vehicle executing the planned action, and (ii) the reference state of the environment obtained by the simulated vehicle executing a reference action in the initial state of the environment and the actor performing an actor action following the simulated vehicle executing the reference action. In some embodiments, the operating system was verified by (d) determining a test score based on the SUT state of the environment and a reference score based on the reference state of the environment. In some embodiments, the operating system was verified by (e) evaluating the operating system based on the test score and the reference score.

[0032] Other exemplary aspects of the present disclosure relate to other systems, methods, vehicles, devices, tangible non-transient computer-readable media, and devices for performing the functions described herein. These and other features, aspects, and advantages of various embodiments will be better understood by reference to the following description and the appended claims. The appended drawings, incorporated into and constituting part of the present specification, illustrate embodiments of the present disclosure and are used to explain the relevant principles together with the description. Brief explanation of the drawing

[0033] Detailed descriptions of embodiments presented to a person skilled in the art are provided herein with reference to the accompanying drawings, in which: FIG. 1 is a block diagram of an exemplary operation scenario according to some embodiments of the present disclosure; FIG. 2 is a block diagram of an exemplary system according to some embodiments of the present disclosure; FIG. 3a is a schematic diagram of an exemplary operating environment according to some embodiments of the present disclosure; FIG. 3b is a schematic map of an exemplary operating environment according to some embodiments of the present disclosure; FIG. 3c is a schematic diagram of an exemplary operating environment according to some embodiments of the present disclosure; FIG. 3d is a schematic map of an exemplary operating environment according to some embodiments of the present disclosure; FIG. 4 is a block diagram of an exemplary system for performing system verification according to some embodiments of the present disclosure; FIG. 5a is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 5b is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 6a is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 6b is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 6c is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 7a is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 7b is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 7c is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 8a is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 8b is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 9a is a block diagram of an exemplary system for performing system verification according to some embodiments of the present disclosure; FIG. 9b is a diagram of an exemplary verification scenario according to some embodiments of the present disclosure; FIG. 10 is a decision tree for classifying verification results according to some embodiments of the present disclosure; FIG. 11 is a flowchart of an exemplary method for system verification according to some embodiments of the present disclosure; FIG. 12 is a flowchart of an exemplary method for system verification according to some embodiments of the present disclosure; FIG. 13 is a flowchart of an exemplary method for training and verifying a machine-learned operating system according to some embodiments of the present disclosure; and FIG. 14 is a block diagram of an exemplary computing system for performing system verification according to some embodiments of the present disclosure. Specific details for implementing the invention

[0034] The techniques of the present disclosure are described below in the context of autonomous vehicles only for exemplary purposes. As described herein, the techniques of the present disclosure are not limited to autonomous vehicles and may be implemented for or within other autonomous platforms and other computing systems.

[0035] Exemplary embodiments of the present disclosure are discussed in more detail with reference to FIG. 1-14. FIG. 1 is a block diagram of an exemplary operation scenario according to some embodiments of the present disclosure. In an exemplary operation scenario, the environment (100) includes a plurality of objects, including an autonomous platform (110) and a first actor (120), a second actor (130), and a third actor (140). In an exemplary operation scenario, the autonomous platform (110) can move through the environment (100) and interact with objects located within the environment (100) (e.g., the first actor (120), the second actor (130), the third actor (140), etc.). The autonomous platform (110) may optionally be configured to communicate with remote system(s) (160) via network(s) (170).

[0036] The environment (100) may be an indoor environment (e.g., an environment within one or more facilities, etc.) or an outdoor environment, or may include such. An indoor environment may be, for example, an environment enclosed by a structure such as a building (e.g., a service warehouse, a maintenance site, a manufacturing facility, etc.). An outdoor environment may be one or more areas of the outside world, such as, for example, one or more rural areas (e.g., areas with one or more rural roads, etc.), one or more urban areas (e.g., areas with one or more urban roads, highways, etc.), one or more suburban areas (e.g., areas with one or more suburban roads, etc.), or other outdoor environments.

[0037] The autonomous platform (110) may be any type of platform configured to operate within the environment (100). For example, the autonomous platform (110) may be a vehicle configured to perceive and operate autonomously within the environment (100). The vehicles may be ground-based autonomous vehicles, such as, for example, autonomous cars, trucks, vans, etc. The autonomous platform (110) may be an autonomous vehicle capable of controlling, connecting to, or otherwise associating with tools, attachments, and / or accessories for transporting people or cargo. The autonomous platform may include, for example, an autonomous tractor optionally coupled to a cargo trailer. Additionally or alternatively, the autonomous platform (110) may be any other type of vehicle, such as one or more aerial vehicles, water vehicles, space-based vehicles, other ground-based vehicles, etc.

[0038] The autonomous platform (110) may be configured to communicate with remote system(s) (160). For example, the remote system(s) (160) may communicate with the autonomous platform (110) for support (e.g., mobility support, situational response support, etc.), control (e.g., vehicle management, remote operation, etc.), maintenance (e.g., updates, monitoring, etc.), or other local or remote tasks. In some embodiments, the remote system(s) (160) may provide data instructing the autonomous platform (110) to perform tasks. For example, as further described herein, the remote system(s) (160) may provide data instructing the autonomous platform (110) to perform mobility / services, such as user transport mobility / services, delivery mobility / services (e.g., cargo, goods, etc.).

[0039] The autonomous platform (110) can communicate with the remote system(s) (160) using the network(s) (170). The network(s) (170) can facilitate the transmission of signals (e.g., electronic signals, etc.) or data (e.g., data from a computing device, etc.) and may include various wired (e.g., twisted pair cables, etc.) or wireless communication mechanisms (e.g., cellular, wireless, satellite, microwave, radio frequency, etc.) or any combination of any desired network topology (or topologies). For example, the network(s) (170) may include a local area network (e.g., intranet, etc.), a wide area network (e.g., the Internet, etc.), a wireless LAN network (e.g., via Wi-Fi, etc.), a cellular network, a SATCOM network, a VHF network, an HF network, a WiMAX-based network, or any other suitable communication network (or combination thereof) for transmitting data into or out of the autonomous platform (110).

[0040] For example, as illustrated in FIG. 1, the environment (100) may include one or more objects. The object(s) may be objects that are not moving or are not expected to move ("static objects") or objects that are moving or are expected to move ("dynamic objects" or "actors"). In some embodiments, the environment (100) may include any number of actor(s), such as one or more pedestrians, animals, vehicles, etc. The actor(s) may move within the environment along one or more actor trajectories. For example, movement may be made such that a first actor (120) may move along any one of the first actor trajectories (122A-C), a second actor (130) may move along any one of the second actor trajectories (132), and a third actor (140) may move along any one of the third actor trajectories (142).

[0041] As further described in this specification, the autonomous platform (110) may utilize its own autonomous system(s) to detect these actors (and their movements) and plan their actions to move through the environment (100) along one or more platform trajectories (112A-C). The autonomous platform (110) may include onboard computing system(s) (180). The onboard computing system(s) (180) may include one or more processors and one or more memory devices. The one or more memory devices may store instructions executable by one or more processors to enable the one or more processors to perform actions or functions related to the autonomous platform (110), including the implementation of the autonomous driving system(s) of the autonomous platform.

[0042] FIG. 2 is a block diagram of an exemplary autonomous system (200) for an autonomous platform according to some embodiments of the present disclosure. In some embodiments, the autonomous system (200) may be implemented by a computing system of the autonomous platform (e.g., onboard computing system(s) (180) of the autonomous platform (110)). The autonomous system (200) may operate to obtain inputs from sensor(s) (202) or other input devices. In some embodiments, the autonomous system (200) may further obtain platform data (208) (e.g., map data (210)) from a local or remote storage device. The autonomous system (200) may generate control outputs for controlling the autonomous platform (e.g., via platform control devices (212), etc.) based on sensor data (204), map data (210), or other data. The autonomous system (200) may include different subsystems for performing various autonomous operations. The subsystems may include a positioning system (230), a perception system (240), a planning system (250), and a control system (260). The positioning system (230) may determine the location of the autonomous platform within an environment; the perception system (240) may detect, classify, and track objects and actors within an environment; the planning system (250) may determine the trajectory of the autonomous platform; and the control system (260) may convert the trajectory into vehicle control actions for controlling the autonomous platform. The autonomous system (200) may be implemented by one or more onboard computing system(s). The subsystems may include one or more processors and one or more memory devices. One or more memory devices may store instructions executable by one or more processors to enable one or more processors to perform actions or functions related to the subsystems.The computing resources of the autonomous system (200) can be shared among the subsystems, or the subsystems can have a dedicated set of computing resources.

[0043] In some embodiments, the autonomous system (200) may be implemented for or by an autonomous vehicle (e.g., a ground-based autonomous vehicle). The autonomous system (200) may perform various processing techniques on inputs (e.g., sensor data (204), map data (210)) to generate an appropriate set of control outputs for implementing a vehicle motion plan (e.g., including one or more trajectories) to traverse the vehicle's surrounding environment (e.g., environment (100) of FIG. 1, etc.). In some embodiments, the autonomous vehicle implementing the autonomous system (200) may perform driving, movement, operation, etc. with minimal or no interaction with a human operator (e.g., driver, pilot, etc.).

[0044] In some embodiments, the autonomous platform may be configured to operate in multiple modes of operation. For example, the autonomous platform may be configured to operate in a fully autonomous (e.g., autonomous driving) mode of operation, which allows the autonomous platform to be controlled without user input (e.g., driving and moving without input from a human operator present in the autonomous vehicle or remotely from the autonomous vehicle). The autonomous platform may operate in a semi-autonomous mode of operation, in which the autonomous platform can operate with some input from a human operator present in the autonomous platform (or a human operator located far away from the autonomous platform). In some embodiments, the autonomous platform may enter a manual mode of operation, in which the autonomous platform can be fully controlled by a human operator (e.g., a human driver) and can be prohibited or disabled (e.g., temporarily, permanently, etc.) from performing autonomous movement (e.g., autonomous driving). The autonomous platform may be configured to operate in other modes, such as parking or power-saving modes (used between actions such as movement / service waiting, charging, etc.). In some embodiments, the autonomous platform may implement vehicle operation assistance techniques (e.g., collision mitigation systems, power-assisted steering, etc.) to assist a human operator of the autonomous platform (e.g., while in manual mode, etc.).

[0045] The autonomous system (200) may be located onboard (e.g., on or inside the autonomous platform) the autonomous platform and may be configured to operate the autonomous platform in various environments. The environment may be a real environment or a simulated environment. In some embodiments, one or more simulation computing devices may simulate one or more of the sensors (202), sensor data (204), communication interface(s) (206), platform data (208), or platform control devices (212) to simulate the operation of the autonomous system (200).

[0046] In some embodiments, the autonomous system (200) may communicate with one or more networks or other systems through communication interface(s) (206). The communication interface(s) (206) may include, for example, transmitters, receivers, ports, controllers, antennas, or any other suitable components that can facilitate communication, and may include any components suitable for interfacing with one or more network(s) (e.g., network(s) (170) of FIG. 1). In some embodiments, the communication interface(s) (206) may include multiple components (e.g., antennas, transmitters, or receivers) that enable the implementation and utilization of various communication techniques (e.g., multiple input, multiple output (MIMO) techniques, etc.).

[0047] In some embodiments, the autonomous system (200) may communicate with one or more remote computing devices from the autonomous platform (e.g., remote system (160)) via one or more networks (e.g., network(s) (170)) using communication interface(s) (206). For example, in some embodiments, one or more inputs, data, or functions of the autonomous system (200) may be supplemented or replaced by the remote system communicating via the communication interface(s) (206). For example, in some embodiments, map data (210) may be downloaded to the remote system via the network using the communication interface(s) (206). In some embodiments, one or more of the location system (230), perception system (240), planning system (250), or control system (260) may be updated, affected, nudged, or communicated with by the remote system for support, maintenance, situation response override, management, etc.

[0048] The sensor(s) (202) may be positioned onboard on an autonomous platform. In some embodiments, the sensor(s) (202) may include one or more types of sensor(s). For example, one or more sensors may include image capture device(s) (e.g., visible spectrum cameras, infrared cameras, etc.). Additionally or alternatively, the sensor(s) (202) may include one or more depth capture device(s). For example, the sensor(s) (202) may include one or more light detection and distance measurement (LIDAR) sensors or radio detection and distance measurement (RADAR) sensors. The sensor(s) (202) may be configured to generate point data describing at least a portion of a 360-degree view of the surrounding environment. The point data may be point cloud data (e.g., 3D LiDAR point cloud data, radar point cloud data). In some embodiments, one or more of the sensors (202) for capturing depth information may be fixed to a rotating device to rotate the sensor(s) (202) around an axis. As the sensor(s) (202) rotate around the axis, data may be captured in interval sector packets describing different parts of the 360-degree view of the surrounding environment of the autonomous platform. In some embodiments, one or more of the sensors (202) for capturing depth information may be solid state.

[0049] Sensor(s) (202) may be configured to capture sensor data (204) representing or associated with at least a portion of the environment of the autonomous platform. The sensor data (204) may include image data (e.g., 2D camera data, video data, etc.), radar data, lidar data (e.g., 3D point cloud data, etc.), audio data, or other types of data. In some embodiments, the autonomous system (200) may obtain input from additional types of sensors, such as inertial measurement units (IMUs), altimeters, inclinometers, odometers, position or positioning devices (e.g., GPS, compass), wheel encoders, or other types of sensors. In some embodiments, the autonomous system (200) may obtain sensor data (204) associated with specific component(s) or system(s) of the autonomous platform. Such sensor data (204) may indicate, for example, wheel speed, component temperature, steering angle, cargo or passenger status, etc. In some embodiments, the autonomous system (200) may acquire sensor data (204) related to ambient conditions, such as environmental or weather conditions. In some embodiments, the sensor data (204) may include multi-modal sensor data. Multi-modal sensor data may be acquired by at least two types of sensors (e.g., sensors among the sensors (202)) and may represent static object(s) or actor(s) within the environment of the autonomous platform. Multi-modal sensor data may include at least two types of sensor data (e.g., camera and LIDAR data). In some embodiments, the autonomous platform may utilize sensor data (204) from sensors located remotely (e.g., offboard) from the autonomous platform. This may include, for example, sensor data (204) captured by another autonomous platform.

[0050] The autonomous system (200) can acquire map data (210) related to the environment in which the autonomous platform is located, is located, or will be located. The map data (210) may provide information about the environment or geographical area. For example, the map data (210) may include the location and position of various travel paths (e.g., roads, etc.), travel path segments (e.g., road segments, etc.), buildings or other items or objects (e.g., streetlights, crosswalks, curbs, etc.); the location and direction of boundaries or boundary markers (e.g., traffic lanes, parking lanes, turning lanes, bicycle lanes, other lanes, etc.); traffic control data (e.g., location and instructions of signs, traffic lights, other traffic control devices); obstacle information (e.g., temporary or permanent blockages); event data (e.g., road closures / changes in traffic rules due to parades, concerts, sports events, etc.); and nominal vehicle route data (e.g., indicating an ideal vehicle route along the center of a specific lane). Alternatively, information regarding any other map data that helps the autonomous platform understand the surrounding environment and its relationships may be provided. In some embodiments, the map data (210) may include high-resolution map information. Additionally, or alternatively, the map data (210) may include sparse map data (e.g., lane graphs, etc.). In some embodiments, sensor data (204) may be fused with the map data (210) or used to update the map data (210) in real time.

[0051] The autonomous system (200) may include a positioning system (230) capable of providing an understanding of the location and orientation within the environment for the autonomous platform. In some examples, the positioning system (230) may support one or more other subsystems of the autonomous system (200) by providing an integrated local reference frame for performing, for example, cognitive actions, planning actions, or control actions.

[0052] In some embodiments, the positioning system (230) can determine the current position of the autonomous platform. The current position may include an absolute position (global position) (e.g., a position relative to a georeference anchor, etc.) or a relative position (e.g., a position relative to objects in the environment, etc.). The positioning system (230) may generally include or interface with any device or circuit for analyzing the position or position change of the autonomous platform (e.g., an autonomous ground-based vehicle, etc.). For example, the positioning system (230) may determine the position by using one or more of inertial sensors (e.g., inertial measurement unit(s), etc.), a satellite positioning system, radio receivers, networking devices (e.g., based on IP addresses, etc.), network access points, or other network components (e.g., cellular towers, Wi-Fi access points, etc.), triangulation or proximity to, or other suitable techniques. The location of the autonomous platform may be used by various subsystems of the autonomous system (200) or provided to a remote computing system (e.g., using communication interface(s) (206).

[0053] In some embodiments, the positioning system (230) may register the relative positions of elements of the surrounding environment of the autonomous platform along with the positions recorded in the map data (210). For example, the positioning system (230) may determine the position of the autonomous platform within the environment by processing sensor data (204) (e.g., LiDAR data, radar data, camera data, etc.) for alignment or other registration on a map of the surrounding environment (e.g., from the map data (210)). Thus, in some embodiments, the autonomous platform may identify its position within the surrounding environment (e.g., across six axes, etc.) based on a search of the map data (210). In some embodiments, given an initial position, the positioning system (230) may update the position of the autonomous platform through gradual realignment based on deviations recorded or estimated from the initial position. In some embodiments, the position may be registered directly within the map data (210).

[0054] In some embodiments, the map data (210) may include a large amount of data subdivided into geographic tiles so that a desired area of ​​the map stored in the map data (210) can be reconstructed from one or more tiles. For example, a plurality of tiles selected in the map data (210) may be stitched together by the autonomous system (200) based on a location obtained by the location identification system (230) (e.g., a plurality of tiles selected near the location).

[0055] In some embodiments, the positioning system (230) may determine the locations (e.g., relative or absolute locations) of one or more attachments or accessories to the autonomous platform. For example, the autonomous platform may be associated with a cargo platform, and the positioning system (230) may provide the locations of one or more points on the cargo platform. For example, the cargo platform may include a trailer or other device towed by the autonomous platform or otherwise attached or operated by the autonomous platform, and the positioning system (230) may provide data describing the location (e.g., absolute, relative location, etc.) of the autonomous platform as well as the cargo platform. This information may be obtained by another autonomous driving system to assist in the operation of the autonomous platform.

[0056] The autonomous system (200) may include a perception system (240) capable of enabling the autonomous platform to detect, classify, and track objects and actors within the environment. Environmental features or objects perceived within the environment may be within the field of view of the sensor(s) (202) or predicted to be obscured from the sensor(s) (202). This may include object(s) that are not moving or are not predicted to move (static objects) or objects that are moving or are predicted to move (dynamic objects / actors).

[0057] The perception system (240) can determine one or more states (e.g., current or past state(s), etc.) of one or more objects within the surrounding environment of the autonomous platform. For example, the state(s) may be the current or past location (also referred to as position) of an object (e.g., for a given time, period, etc.); current or past speed / velocity; current or past acceleration; current or past direction of travel; current or past orientation; size / footprint (e.g., size / footprint expressed as boundary shape, object highlighting, etc.); classification (e.g., pedestrian class vs. vehicle class vs. bicycle class, etc.); associated uncertainty; or estimates of other state information. In some embodiments, the perception system (240) may determine the state(s) using one or more algorithms or machine learning models configured to identify / classify objects based on inputs from the sensor(s) (202). The perception system may use various forms of sensor data (204) to generate a representation of the environment to be processed by one or more algorithms or machine learning models. In some embodiments, the state(s) for one or more identified or unidentified objects may be maintained and updated over time as the autonomous platform continues to perceive the objects or interacts with the objects (e.g., maneuvering, yielding, etc. with or around the objects). In this way, the perception system (240) may provide an understanding of the current state of the environment (e.g., a state including objects in the environment, etc.) known from a record of previous states of the environment (e.g., states including movement history of objects in the environment). This information may be useful when the autonomous platform plans actions through the environment.

[0058] The autonomous system (200) may include a planning system (250) that can be configured to determine how the autonomous platform interacts with the environment and moves within the environment. The planning system (250) may determine one or more motion plans for the autonomous platform. A motion plan may include one or more trajectories (e.g., motion trajectories) representing a path that the autonomous platform must follow. A trajectory may be a specific length or time range. The length or time range may be defined by a computational planning range of the planning system (250). A motion trajectory may be defined by one or more waypoints (including associated coordinates). The waypoint(s) may be future location(s) of the autonomous platform. Motion plans may be continuously generated, updated, and considered by the planning system (250).

[0059] The motion planning system (250) can determine a strategy for the autonomous platform. The strategy may be a series of individual decisions made by the autonomous platform (e.g., yielding to an actor, yielding to an actor, merging, changing lanes). The strategy may be selected from among multiple potential strategies. The selected strategy may be the lowest-cost strategy determined by one or more cost functions. The cost functions may, for example, evaluate the likelihood of a collision with another actor or object.

[0060] The planning system (250) can determine a desirable trajectory for executing a strategy. For example, the planning system (250) can obtain one or more trajectories for executing one or more strategies. The planning system (250) can evaluate the trajectories or strategies (e.g., using scores, costs, rewards, constraints, etc.) and rank them. For example, the planning system (250) can indicate the evaluation of candidate trajectories or strategies for autonomous platforms using prediction output(s) that represent the interactions (e.g., proximity, intersections, etc.) between the trajectories for autonomous platforms and one or more objects. In some embodiments, the planning system (250) can evaluate the trajectories of autonomous platforms (e.g., "avoiding lane boundaries," "minimizing jerk," etc.) by utilizing static cost(s). Additionally or alternatively, the planning system (250) may evaluate trajectories or strategies for the autonomous platform based on predicted outcomes for the current operational scenario (e.g., predicted trajectories or strategies leading to interactions between actors, predicted trajectories or strategies leading to interactions between actors and the autonomous platform, etc.) by utilizing dynamic cost(s). The planning system (250) may rank the trajectories based on one or more static costs, one or more dynamic costs, or a combination thereof. The planning system (250) may select an operation plan (and corresponding trajectory) based on the ranking of multiple candidate trajectories. In some embodiments, the planning system (250) may select the highest-ranked candidate or the highest-ranked feasible candidate.

[0061] Then, the planning system (250) can verify the selected trajectory against one or more constraints before the trajectory is executed by the autonomous platform.

[0062] To assist in making motion planning decisions, the planning system (250) may be configured to perform a prediction function. The planning system (250) may predict future state(s) of the environment. This may include predicting future state(s) of other actors within the environment. In some embodiments, the planning system (250) may predict future state(s) based on current or past state(s) (e.g., states developed or maintained by the perception system (240)). In some embodiments, the future state(s) may be or include predicted trajectories (e.g., locations over time) of objects within the environment, such as other actors. In some embodiments, one or more of the future state(s) may include one or more associated probabilities (e.g., marginal probabilities, conditional probabilities). For example, one or more probabilities may include one or more probabilities that are conditioned according to strategies or trajectory options available on the autonomous platform. Additionally or alternatively, the probabilities may include probabilities conditioned according to trajectory options available to one or more other actors.

[0063] In some embodiments, the planning system (250) may perform interactive prediction. The planning system (250) may determine a motion plan for the autonomous platform through an understanding of how the predicted future states of the environment may be affected by the execution of one or more candidate motion plans. For example, referring again to FIG. 1, the autonomous platform (110) may determine candidate motion plans corresponding to a set of platform trajectories (112A-C) corresponding to the first actor trajectories (122A-C) for the first actor (120), the trajectories (132) for the second actor (130), and the trajectories (142) for the third actor (140), respectively (e.g., each trajectory correspondence is indicated by a matching line style). For example, the autonomous platform (110) can predict that a platform trajectory (112A) that moves the autonomous platform (110) more quickly to the area in front of the first actor (120) (e.g., using the autonomous system (200)) is associated with the first actor (120) reducing its forward speed along the first actor trajectory (122A) and yielding more quickly to the autonomous platform (110). Additionally or alternatively, the autonomous platform (110) can predict that a platform trajectory (112B) that moves the autonomous platform (110) smoothly to the area in front of the first actor (120) is associated with the first actor (120) reducing its speed slightly along the first actor trajectory (122B) and yielding slowly to the autonomous platform (110). Additionally or alternatively, it can be predicted that the autonomous platform (110) has a platform trajectory (112C) that maintains parallel alignment with the first actor (120), and that the first actor (120) does not yield any distance to the autonomous platform (110) according to the first actor trajectory (122C).The planning system (250) can select a motion plan (and associated trajectory) by considering the interaction between the autonomous platform and the environment (100) by comparing the predicted scenarios (e.g., by assigning scores to the scenarios based on cost or reward) with a desired result set. In this way, for example, the autonomous platform (110) can interleave the prediction and motion planning functions.

[0064] To implement selected motion plan(s), the autonomous system (200) may include a control system (260) (e.g., a vehicle control system). Generally, the control system (260) may provide an interface between the autonomous system (200) and the platform control unit (212) to implement strategies and motion plan(s) generated by the planning system (250). For example, the control system (260) may implement the selected motion plan / trajectory to control the operation of the autonomous platform through the environment by following a selected trajectory (e.g., waypoints included in the trajectory). For example, the control system (260) may convert the motion plan into commands (e.g., acceleration control, braking control, steering control, etc.) for appropriate platform control units (212). For example, the control system (260) may convert the selected motion plan into commands such as adjusting a steering component (e.g., steering angle) by a specific number, applying a braking force of a specific magnitude, and increasing / decreasing speed. In some embodiments, the control system (260) may communicate with platform control units (212) through communication channels including, for example, one or more data buses (e.g., a controller area network (CAN), etc.), onboard diagnostic connectors (e.g., OBD-II), or a combination of wired or wireless communication links. The platform control units (212) may transmit or receive data, messages, signals, etc. to or from the autonomous system (200) or the autonomous system through the communication channel(s) (or vice versa).

[0065] The autonomous system (200) can receive support signal(s) from the remote support system (270) through the communication interface(s) (206). The remote support system (270) can communicate with the autonomous system (200) through a network (e.g., as the remote system (160) through the network (170)). In some embodiments, the autonomous system (200) can initiate a communication session with the remote support system (270). For example, the autonomous system (200) can initiate a session based on or in response to a trigger. In some embodiments, the trigger may be a warning, an error signal, a map feature, a request, a location, a traffic condition, a road condition, etc.

[0066] After initiating a session, the autonomous system (200) may provide context data to the remote support system (270). The context data may include sensor data (204) and state data of the autonomous platform. For example, the context data may include a live camera feed from the autonomous platform's camera and the autonomous platform's current speed. An operator of the remote support system (270) (e.g., a human operator) may use the context data to select support signals. The support signal(s) may provide values ​​or adjustments for various operating parameters or characteristics of the autonomous system (200). For example, the support signal(s) may include waypoints (e.g., obstacle path, lane change, etc.), speed or acceleration profiles (e.g., speed limit, etc.), relative motion instructions (e.g., convoy formation, etc.), operating characteristics (e.g., use of assistance systems, reduction of energy processing mode, etc.), or other signals to support the autonomous system (200).

[0067] The autonomous system (200) may use support signal(s) input to one or more autonomous subsystems to perform autonomous functions. For example, the planning subsystem (250) may receive support signal(s) as inputs for generating an operation plan. For example, the support signal(s) may include constraints for generating an operation plan. Additionally or alternatively, the support signal(s) may include cost or compensation adjustments that affect the operation plan by the planning subsystem (250). Additionally or alternatively, the support signal(s) may be considered by the autonomous system (200) as proposed inputs that may be additionally considered with other received data (e.g., sensor inputs, etc.).

[0068] The autonomous system (200) may be independent of the platform, and the control system (260) may provide control commands to platform control devices (212) for various platforms for autonomous movement (e.g., multiple different autonomous platforms equipped with autonomous control systems). This may include various different types of autonomous vehicles (e.g., sedans, vans, SUVs, trucks, electric vehicles, combustion-powered vehicles, etc.) from various different manufacturers / developers that operate in various different environments and perform one or more vehicle services in some embodiments.

[0069] For example, with reference to FIG. 3a, the operating environment may include a dense environment (300). The autonomous platform may include an autonomous vehicle (310) controlled by an autonomous system (200). In some embodiments, the autonomous vehicle (310) may be configured for maneuverability in a dense environment, for example, with a set wheelbase or other specifications. In some embodiments, the autonomous vehicle (310) may be configured to transport cargo or passengers. In some embodiments, the autonomous vehicle (310) (e.g., passenger van, shuttle, bus, etc.) may be configured to transport multiple passengers. In some embodiments, the autonomous vehicle (310) may be configured to transport cargo, for example, large cargo (e.g., truck, box van, step van, etc.) or small cargo (e.g., food, personal packages, etc.).

[0070] Referring to FIG. 3b, a selected top view (302) of a dense environment (300) is superimposed with an exemplary movement / service between a first location (304) and a second location (306). The exemplary movement / service may be assigned to an autonomous vehicle (320), for example, by a remote computing system. The autonomous vehicle (320) may be, for example, a vehicle of the same type as the autonomous vehicle (310). The exemplary movement / service may include transporting passengers or cargo between the first location (304) and the second location (306). In some embodiments, the exemplary movement / service may include moving to or through one or more intermediate locations, for example, to load or unload passengers or cargo. In some embodiments, the exemplary movement / service may be pre-booked (e.g., for regular movement such as a transport schedule). In some implementations, exemplary mobility / services may be on demand (e.g., taxi, car sharing, ride calling, courier, delivery services, etc., or for the performance thereof).

[0071] Referring to FIG. 3c, in another example, the operating environment may include an open travel path environment (330). The autonomous platform may include an autonomous vehicle (350) controlled by the autonomous system (200). This may include an autonomous tractor for an autonomous truck. In some embodiments, the autonomous vehicle (350) may be configured for high-volume transport (e.g., transport of a large volume of cargo or other goods or passengers), such as long-distance, high-volume transport. For example, the autonomous vehicle (350) may include one or more cargo platform attachments, such as a trailer (352). Although illustrated as a towable attachment in FIG. 3c, in some embodiments, one or more cargo platforms may be integrated into the autonomous vehicle (350) (such as attached to the chassis), as in a box van, step van, etc.

[0072] Referring to FIG. 3d, a selected top view of an open movement path environment (330) including movement paths (332), an interchange (334), transfer hubs (336 and 338), access movement paths (340), and locations (342 and 344) is shown. In some embodiments, an exemplary movement / service may be assigned to an autonomous vehicle (e.g., autonomous vehicle (310) or autonomous vehicle (350)) to traverse one or more movement paths (332) (optionally connected by an interchange (334)) to transport cargo between a transfer hub (336) and a transfer hub (338). For example, in some embodiments, the exemplary movement / service includes a cargo delivery / transport service such as a cargo delivery / transport service. The exemplary movement / service may be assigned by a remote computing system. In some embodiments, the transfer hub (336) may be the origin point of the cargo (e.g., warehouse, wholesale store, facility, etc.) and the transfer hub (338) may be the destination point of the cargo (e.g., retail store, etc.). However, in some embodiments, the transfer hub (336) may be an intermediate point along the final journey between each origin point and each destination of the cargo. For example, the origin point of the cargo may be located at a location (342) along the access route (340). Accordingly, the cargo may be transported to the transfer hub (336) for staging (e.g., by a human-driven vehicle, an autonomous vehicle (310), etc.). At the transfer hub (336), various cargo items may be grouped or staged for long-distance transport along the routes (332).

[0073] In some embodiments of the exemplary movement / service, a group of staged cargo items may be loaded onto an autonomous vehicle (e.g., autonomous vehicle (350)) for transport to one or more other transfer hubs, such as a transfer hub (338). For example, although not illustrated, it should be understood that an open movement path environment (330) may include more transfer hubs than the transfer hubs (336 and 338) and more movement paths (332) interconnected by more interchanges (334). A simplified map is presented here for the purpose of clarity only. In some embodiments, one or more cargo items transported to a transfer hub (338) may be distributed to one or more local destinations along access paths (340) to a location (344), for example, by a human-driven vehicle, an autonomous vehicle (310), etc. In some embodiments, exemplary movement / service may be pre-booked (e.g., in the case of regular transport, such as transport schedules). In some embodiments, exemplary movement / service may be on demand (e.g., for the performance of chartered passenger transport or cargo delivery services, or upon request for said services).

[0074] To improve the performance of an autonomous platform, such as an autonomous vehicle (e.g., an autonomous vehicle (310 or 350)) that is at least partially controlled using an autonomous system (200), the planning system (250) may implement verification techniques according to exemplary embodiments of the present disclosure.

[0075] FIG. 4 is a block diagram of a verification system (400) according to some embodiments of the present disclosure. FIG. 4 illustrates an exemplary embodiment of a verification system (400) having various components, but it should be understood that components may be rearranged, combined, omitted, etc. within the scope of the present disclosure and within the scope consistent with the present disclosure.

[0076] The verification system (400) may take log data (402) as input. The log data (402) may include data recorded during instances of actual or simulated driving. The recorded data may include data collected by sensors mounted on one or more vehicles (e.g., autonomous vehicles, non-autonomous vehicles, etc.). The recorded data may include data collected from other sources (e.g., roadside cameras, aerial vehicles, etc.). The log data (402) from simulated scenarios may include probabilistic data, such as data extracted from a distribution fitted to multiple observations.

[0077] The verification system (400) can extract a reference scenario (404) from log data. The reference scenario (404) may be part of log data (402) describing a scene or event of interest that includes a reference target. The scene or event of interest may include an environment that includes objects, actors, etc. The reference target may be an object or actor that is the subject of study during verification. The reference target may be an actor that demonstrates an action that can verify another system. The reference scenario (404) may include a driving scenario. The reference scenario (404) may be selected based on the possibility of exploring events of exceptional situations and the system's responses to them. For example, the reference scenario (404) may include cases where the reference vehicle drives past a vehicle parked on the roadside, where the reference vehicle joins traffic, or where the reference vehicle maintains a trailing distance while cruising in a normal state on a highway.

[0078] In a reference scenario (404), the reference vehicle may perform a reference action (410). For example, in a scenario involving the reference vehicle passing a vehicle parked on the roadside, the reference action (410) may include a deceleration amount (or a reduction in acceleration) of the reference vehicle as the reference vehicle approaches the vehicle parked on the roadside. In another example, in a scenario involving the reference vehicle maintaining normal cruise driving on a highway, the reference action (410) may include a distance maintained behind a preceding vehicle. In another example, in a scenario involving the reference vehicle merging (or being in a merging lane), the reference action (410) may include a amount of acceleration or deceleration applied in response to the merging. The reference action (410) may be a defensive maneuver performed to reduce the likelihood of a serious event or to reduce the severity of a serious event. For example, the reference action (410) may be a defensive driving maneuver.

[0079] The verification system (400) can verify the system under test (SUT) (412) by introducing the vehicle being simulated into a reference scenario (404) for comparison with a reference target. For example, considering the context of the reference scenario (404), the SUT action (414) may represent an action that the target vehicle implementing the SUT (412) would execute in the same situation where the reference target was made to execute the reference action (410) (e.g., an action of "teleporting" the SUT (412) from the log data to the exact time when the reference target was executing, or an interpolation action (of the times when the two targets were executing). In this way, for example, the SUT action (414)—and the resulting actions—can be compared with the reference action (410) and the resulting actions.

[0080] The SUT (412) may be or may include one or more operating systems of an autonomous vehicle. For example, the system (412) may include one or more autonomous systems or one or more systems operated to support the autonomous systems. For example, the SUT (412) may include one or more parts of the autonomous system (200), such as a positioning subsystem (230), a perception subsystem (240), a planning subsystem (250), a control subsystem (260), etc. In some examples, the SUT (412) may include sensor(s) (202), communication interface(s) (206), a remote support system (270), a platform control unit (212), etc. The SUT (412) may include one or more machine learning models.

[0081] The verification system (400) can evaluate changes to any part of the autonomous vehicle control system pipeline. Advantageously, the verification system (400) can facilitate evaluation in a common reference frame comparison with reference target(s). In this way, for example, work on changes to complex systems can be reduced to evaluating whether the SUT (412) operates more favorably or more disadvantageously than the reference target overall. For example, the speed error of a sensor (202) with a 200-meter range may have a larger magnitude than the speed error of a sensor (202) with a 1-meter range, but the verification system (400) can help evaluate whether the ultimate impact on the overall performance of the SUT (412) actually results in significantly different behavior compared to the reference targets. Similarly, the verification system (400) can help evaluate whether complex machine learning cognitive models of different machine learning model architectures have a significant impact on the overall system behavior compared to the reference targets.

[0082] The reference operation (410) and the SUT operation (414) may include current or future state data describing the vehicle state for the target vehicle. For example, the vehicle state may include position, speed, acceleration, jerk, direction, orientation, heading, etc.

[0083] In some embodiments, the verification system (400) may directly compare the reference action (410) with the SUT action (414). For example, the verification system (400) may score the reference action (410) and the SUT action (414) based on various costs or rules of thumb, so that the scores can be compared. For example, the SUT action (414) may include a motion plan for the target vehicle being simulated, and the planned motion may be compared with the reference action performed by the reference target.

[0084] Additionally, or alternatively, the verification system (400) may use a simulator (420) to allow the reference scenario (404) to unfold over time using both the reference action (410) and the SUT action (414) as different starting points or seed actions. For example, the simulator (420) may simulate what events are likely to occur in the environment when the target vehicle performs the reference action (410) (e.g., up to a given time range such as 1 second, 5 seconds, etc.). At various points in time, the simulator (420) may output an environment state (430) associated with the reference action (410). The environment state (430) may include state data related to object states and actor states for objects and actors within the environment over time.

[0085] Similarly, the simulator (420) can also simulate what events are likely to occur within the environment when the target vehicle performs the SUT operation (414). At various points in time, the simulator (420) can output an environment state (440) related to the SUT operation (414). The environment state (440) may include state data related to object states and actor states for objects and actors within the environment over time.

[0086] The simulator (420) may include simulations of multiple types or layers. For example, the simulator (420) may output planned SUT behaviors composed of environmental data (e.g., initial state data describing the environment). The simulator (420) may simulate the behaviors of the system under test (e.g., one or more components of an autonomous vehicle computing system) in the simulated environment. For example, the simulator (420) may input simulated sensor data into the simulated autonomous vehicle control system(s) to obtain simulated control decisions.

[0087] The simulator (420) may also be configured to perform lighter simulations. For example, the simulator (420) may compute simulations based on approximations. For example, the simulator (420) may compute simulations to obtain conservative estimates for stopping distance, deceleration speed, lane change speed, etc. For example, the simulator (420) may compute simulations to efficiently estimate the outer boundary of the response to actor action (e.g., upper limit of stopping time or distance). For example, the simulator (420) may compute simulations using simplified analysis equations that model target vehicle action (e.g., action of a vehicle controlled using the SUT). For example, target vehicle action may be modeled as a one-dimensional, two-dimensional, or three-dimensional projectile trajectory. In this case, for example, the simulator (420) may estimate the SUT state (440) and reference state (430) in a cost-effective manner. The simulator (420) can calculate SUT states and reference states for multiple actor actions. For example, for a given outer loop simulation stage (e.g., simulation of a target vehicle driving in an environment at a given time), the simulator (420) can iteratively run an inner loop simulation with lighter weights to calculate the target vehicle's ability to correspond to a range of various hazards within the environment.

[0088] The evaluator (450) can evaluate and optionally compare the environment states (430 and 440). In some embodiments, the evaluator (450) outputs only a relative evaluation of the SUT (412) compared with reference targets. In some embodiments, the evaluator (450) additionally or alternatively generates an evaluation of the environment states (430 and 440) over the entire corpus of log data (402).

[0089] Exemplary verification scenarios are illustrated in FIGS. 5a through 6c. For example, in FIG. 5a, the target vehicle (500) being simulated may be simulated driving in a lane of a road with an action (502) for execution. For example, the action (502) may be a reference action (410) derived from log data (402). In FIG. 5a, the action (502) may include a coasting action (translator's note: reduced power driving or inertia driving action) considering that the actor (510) approaches a merging zone of the road with an expected actor action (512) (e.g., an expected action based on the current state of the actor (510)). The action (502) may represent a defensive driving or protective action that a professional human driver might take in the scenario.

[0090] In FIG. 5b, at a point in time after the snapshot shown in FIG. 5a, both the simulated target vehicle (500) and the actor (510) are advancing along the road. In this case, this advance may be caused by a simulation by the simulator (420). At a later point in time, a risk may be simulated. For example, the verification system (400) may simulate the actor (510) performing an actor action (514). The actor action (514) may be a sudden cut-in.

[0091] In FIG. 5c, at a point in time after the snapshot shown in FIG. 5b, both the simulated target vehicle (500) and the actor (510) are further advanced along the road. For example, this advancement may be the result of a simulation by the simulator (420). At a later point in time, the target vehicle (500) may perform a defined action (506) in response to an actor action (514) or an actor action (516). For example, the defined action (506) may include appropriate braking in response to a sudden merging of the actor (510).

[0092] The environmental states illustrated in FIGS. 5a-5c may be environmental states output by the simulator (420). For example, the environmental state of FIG. 5a may be an initial state of the environment. If the action (502) is a reference action (410), the later resulting state of FIG. 5c may be an environmental state (430). The environmental state may include, for example, the distance between the target vehicle (500) and the actor (510), and data describing the braking action (506), such as speed, acceleration, jerk, etc., related to the execution of the braking action (506).

[0093] FIGS. 6a-6c illustrate an alternative scenario in which the simulated target vehicle (500) performs an initial action (602) instead of an action (502). The action (602) may be a SUT action (414) output by the system under test (412) in response to one or more inputs describing the environment of FIG. 6a (e.g., an action determined by a motion planner of an autonomous vehicle control system). The action (602) may be different from the action (502). For example, the action (602) may be associated with a different speed or acceleration than the action (502).

[0094] In FIG. 6b, at a point in time after the snapshot shown in FIG. 6a, both the simulated target vehicle (500) and the actor (510) are in a state of advancing along the road. For example, this advance may be the result of a simulation by the simulator (420). At a later point in time, the same danger simulated in FIG. 5b may be simulated here. For example, the verification system (400) may simulate the actor (510) performing an actor action (614). The actor action (614) may be a sudden cut-off such as the actor action (514).

[0095] In FIG. 6c, at a point in time after the snapshot shown in FIG. 6B, both the simulated target vehicle (500) and the actor (510) have advanced further along the road. For example, this advancement may be the result of a simulation by the simulator (420). At a later point in time, the target vehicle (500) may perform a defined action (606) in response to an actor action (614) or an actor action (616). For example, the defined action (606) may include sudden braking in response to a sudden merging of the actor (510). The defined action (606) may be the same as the defined action (506).

[0096] The environmental states illustrated in FIGS. 6a-6c may be environmental states output by the simulator (420). For example, the environmental state in FIG. 6a may be the initial state of the environment. If the action (602) is the SUT action (414), the resulting state in FIG. 6c may be the environmental state (440). The environmental state may include, for example, the distance between the target vehicle (500) and the actor (510), data describing the braking action (606), such as speed, acceleration, jerk, etc., related to the execution of the braking action (606). The verification system (400) may determine whether the action (602) has produced a result comparable to the action (502) by directly comparing the resulting environmental state in FIG. 5c with the resulting environmental state in FIG. 6c. In this way, the verification system (400) may directly measure the reference target (e.g., a professional human driver) against the SUT (e.g., an autonomous vehicle control system).

[0097] In some embodiments, the simulation generated by the reference action (410) and the SUT action (414) may encompass various time scales. For example, in some embodiments, the time scale of the simulation may extend over a period sufficient to capture interactions between the target vehicle and its environment. In some embodiments, the time scale of the simulation may be relatively short, for example, within the planning range of the motion planner of the SUT (412). For example, in some embodiments, the simulator (420) obtains the results of the simulation directly based on the motion plan output by the SUT (412) itself.

[0098] In this way, for example, the verification system (400) may provide a paired comparison of the reference target(s) and the SUT (412). The precision of the paired comparison may be configured to be based, for example, on a scenario level (e.g., comparing what the reference target performs in a given scenario with what the SUT proposes in a given scenario) or on a timestamp level (e.g., comparing potential results resulting from the action (410) of the reference target at a given moment with potential results resulting from the action (414) of the SUT at the same moment).

[0099] Accordingly, the verification system (400) can evaluate how well the SUT (412) has performed compared to the reference target by scoring the environmental states of FIG. 5c and FIG. 6c. Generally, the state results may be scored according to any desired metric or rule of thumb. For example, the score may indicate the severity of the result. The severity may relate to potential intersections between the target vehicle (500) and the boundary of the object or actor (e.g., bounding box, boundary line, etc.). For example, the severity may include the occurrence of an intersection, distance from the intersection (e.g., margin or buffer space, etc.), contact rate or intersection rate, degree of overlap, etc. In some embodiments, the severity may include energy-based evaluation techniques (e.g., kinetic energy-based, etc.). The severity may also be based on inherent characteristics of the target vehicle's own motion, such as acceleration, jerk, curvature, etc.

[0100] Score or severity values ​​may include temporal features. For example, if the simulated set actions proceed over a time interval, scores may be calculated at each time step of the interval, at time steps sampled from the interval, at the end of the interval, etc. In some implementations, values ​​may be averaged over the interval or combined in other ways.

[0101] Score or severity values ​​may include features based on the context of a given scenario. For example, some merging scenarios are often expected to involve short periods of close retracement followed by longer periods of longer retracement. By using the temporal context (e.g., the start of the merging interval), the calculation of the score can ensure that an overly close but short merging maneuver is not underestimated, as such maneuvers may have lower severity in multiple subsequent timeframes due to longer retracement. For example, the context can be used to assign different weights to different timestamps or different modalities of cost or score.

[0102] Similarly, scores or severity may include other contextual features. For example, right-of-way status, merging progress, etc., can selectively influence the severity of a given metric, such as distance determination.

[0103] In this way, for example, the verification system (400) can comprehensively evaluate the actions proposed by the system under test. For example, the verification system (400) can compare the immediate action performed by the reference target (e.g., reference action (410)) in the same situation with the immediately corresponding SUT action (414) performed by the SUT (412), as well as the downstream results caused by different actions.

[0104] In particular, this comprehensive evaluation has the advantage of facilitating verification by focusing the evaluation on more significant deviations of the SUT (412). For example, FIGS. 7a and FIGS. 8a illustrate two scenarios in which the reference operation (702) differs from the SUT operation (802). However, the difference is relatively insignificant compared to the differences illustrated in FIGS. 7b and FIGS. 8b. In FIGS. 7b and FIGS. 8b, the presence of the actor (510) increases the likelihood of negative consequences arising from the difference between the reference operation (702) and the SUT operation (802) (e.g., as illustrated in FIGS. 5a-6c). In contrast, in FIGS. 7a and FIGS. 8a, without the joining of the actor (510), there is no opportunity for the same sudden braking event that occurred in FIG. 6c to occur.

[0105] Another example includes a scenario in which the actor (510) changes lanes at a forward distance of the target vehicle (500). When the actor (510) is sufficiently far from the target vehicle (500) and is accelerating away from the target vehicle (500), the difference between the reference action (702) and the SUT action (802) regarding the target vehicle (e.g., whether to apply braking force) may not be associated with a meaningful difference in the desirability of the resulting state of the environment. In contrast, when the actor (510) is sufficiently close to the target vehicle (500), or when the distance between the actor (510) and the target vehicle (500) is decreasing, the difference between the reference action (702) and the SUT action (802) regarding the target vehicle (e.g., whether to apply braking force) may be associated with a meaningful difference in the desirability of the resulting state of the environment.

[0106] In this way, for example, by scoring the simulated results for differences from the reference object(s), the verification system (400) can effectively reduce the impact of differences that do not lead to negative results and increase the impact of differences that are more likely to lead to negative results. In this way, the verification system (400) can improve performance more efficiently by supporting the prioritization of verification tasks and SUT improvements.

[0107] In some embodiments, the simulator (420) may simulate additional actor actions to evaluate how well the target vehicle being simulated can respond when starting from other initial states or seed states. For example, the verification system (400) may simulate the target vehicle's response to other actor actions when starting from the reference action (410) compared to when starting from the SUT action (414).

[0108] FIGS. 9a and 9b illustrate simulated action-response pairs. For example, the simulator (420) can input an actor action (922-1) into a scenario and simulate a SUT response (924-1). Similarly, the simulator (420) can input an actor action (922-2) into a scenario and simulate a SUT response (924-2). Likewise, the simulator (420) can input an actor action (922-n) into a scenario and simulate a SUT response (924-n).

[0109] The simulator (420) can simulate each of the responses (924-1, 924-2, and 924-n) starting from each of the reference action (410) seed states and the SUT action (414) seed state. The simulator (420) can obtain the resulting environment states for each of the responses (924-1, 924-2, and 924-n), and the evaluator (450) can score those states (e.g., similar to what was discussed above in relation to FIGS. 5a-6c). In this way, for example, the verification system (400) can evaluate how well the SUT (412) is prepared or defended to avoid worse results than the reference target in response to the input actions (922-1, 922-2, 922-n), etc.

[0110] For example, referring to FIGS. 5b and FIGS. 6b, the actor actions (514 and 614) may be input actions. Due to different seed states (e.g., reference action (410) leading to FIG. 5b; SUT action (414) leading to FIG. 6b, etc.), the target vehicle (500) encounters the input actions differently. For example, since action (502) includes a coasting event defending against an approaching merging, the target vehicle (500) encounters the input actor action (514) at a much greater distance behind the actor (510). In contrast, since action (602) does not include a coasting event defending against a merging, the target vehicle (500) encounters the input event (614) at a smaller distance behind the actor (510). Therefore, the results differ: the action (506) (e.g., the response of the SUT (412) to the input actor action (514 / 516)) is less severe than the action (606) (e.g., the response of the SUT (412) to the input action (614 / 616). In this way, for example, the result may indicate that the reference result is less severe than the SUT result.

[0111] Exemplary pseudocode is provided in Algorithm 1.

[0112] Algorithm 1

[0113] For individual log interval(s) of log data {{

[0114] For individual timestamp(s) of individual log interval(s) {{

[0115] For individual injection operation(s) of one or more injection operations {{

[0116] Simulate and score the results of the initial state of the reference operation.

[0117] Simulate and score the results of the initial state of the SUT operation.

[0118] }

[0119] }

[0120] }

[0121] The simulator (420) can acquire input actions from various sources. The input actions may be generated or synthesized. For example, the input actions may be synthesized or extracted from one or more probability distributions of possible actions that can be taken. The list of actions may include exception scenarios that aid in verification. The list of actions may include low-probability actions that may not actually occur frequently (e.g., thus reducing the sample size of the training data and presenting scenarios that are difficult for machine learning models to learn). The list of actions may include a predetermined set of benchmark actions useful for investigating the actions of the SUT (412). Log data (402) may provide the basis for acquiring the list of actions. For example, in the case of an autonomous vehicle with a human operator (onboard, offboard), log data describing the vehicle operator's disengages (e.g., disengaging all or part of the autonomous system) may indicate scenarios where the autonomous system did not operate as expected or preferred, and thus the scenario may include actions of interest to be input into the simulation for evaluation.

[0122] Additionally, log data (402) may represent situations in which the autonomous vehicle performs alternative actions. These situations may provide a template for generating actions to be input into verification scenarios. For example, if the log data (402) represents lane braking patterns (e.g., sudden braking) in merging scenarios where actors cut in early, the early actor cutting action forms part of the list of actions to be input into verification scenarios. In this way, for example, the verification system (400) can directly evaluate and score the overall effect of the change in the SUT (412) on the score or severity of the target vehicle's response to the early cutting.

[0123] The results of individual verification scenarios can be combined with other results to provide verification results for the SUT (412). For example, in some situations, individual reference targets may be irregular reference targets (e.g., human drivers may exhibit various variations). Therefore, results for a wide range of verification scenarios can provide a more accurate assessment of the SUT (412).

[0124] In some implementations, the average value of performance metrics from all verification scenarios can provide a high level of evaluation of how the SUT (412) aligns with the entire set of reference targets. The difference between the average reference result and the average SUT result can provide a verification metric.

[0125] In some embodiments, the evaluator (450) can classify and count individual verification results by calculating a histogram using score / severity bins. In this way, for example, differences between reference targets and SUTs can be examined more finely based not only on the quantity of such differences but also on the severity level at which more differences occur.

[0126] In some embodiments, the evaluator (450) may divide or bin verification scenarios or results based on individual contexts and characteristics. Then, scores for the individual bins or bins may be averaged or otherwise combined to reduce noise in the evaluation signal while providing more granular insight into situations where the SUT differs from the entire set of reference targets.

[0127] FIG. 10 illustrates an exemplary decision tree (1000) for classifying verification results. The root (1002) may be a node of a larger tree. Based on the Boolean responses for the root (1002), additional nodes (1004 and 1006) may further classify verification scenarios. Intervals (1012, 1014, 1016, and 1018) may correspond to leaf nodes.

[0128] In some embodiments, verification scenarios are segmented based on interpretable features corresponding to indicators of interest (e.g., as in FIG. 10). In this way, classification can help identify areas for future performance improvement, current improvement, or regression. For example, whether a target vehicle has priority of way can influence the classification of behaviors observed in a set of reference targets. For example, if a reference target has priority of way, the reference targets may exhibit less defensive behavior. Thus, classifying segments (1012 and 1014) according to priority of way status can provide an interpretable understanding for calculating more granular performance indicators. For example, when evaluating the question "Does the AV protect similarly to the reference targets when the AV does not have priority of way?", it may involve calculating performance indicators that compare the reference behavior and the AV behavior in segment (1014). These granular performance indicators can aid in further refinement and verification of the AV systems under test.

[0129] However, generally, classifying verification scenarios based on the features of the scenarios can facilitate the prediction of performance defects (e.g., excessive severity) according to those features. These features may be potential features learned by machine learning models. For example, machine learning-based mixed models or other clustering models may be configured to describe the distribution of verification scenarios to identify groups of verification scenarios with similar characteristics and to perform clustering to make comparisons between reference targets and SUT (412).

[0130] Classification can be hierarchical. The desired granularity of the hierarchical structure can be determined by analyzing the resulting distribution(s) of the leaf nodes or intervals. For example, the variance of reference targets within an interval may indicate that multiple intervals are needed to better capture the variance and context-dependence of the SUT (412) and reference actions.

[0131] FIG. 11 is a flowchart of a method (1100) for performing system verification according to embodiments of the present disclosure. One or more parts(s) of the method (1100) may be implemented by a computing system comprising one or more computing devices, such as computing systems described with reference to other drawings (e.g., autonomous platform (110), vehicle computing system (180), remote system (160), system of FIG. 14, etc.). Each individual part of the method (1100) may be performed by any (or any combination of) of the one or more computing devices. Additionally, one or more parts(s) of the method (1100) may be implemented for hardware components of the device(s) described herein (e.g., as in FIG. 1, 2, FIG. 14, etc.). FIG. 11 illustrates elements performed in a specific order for the purpose of explanation and discussion. Those skilled in the art will understand that, using the disclosure provided herein, elements of any of the methods discussed herein may be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the disclosure. FIG. 11 is described by reference to elements / terms described in relation to other systems and drawings for the purposes of exemplary description and is not intended to be limiting. One or more parts of the method (1100) may be performed additionally or substituted by other systems.

[0132] In 1102, the exemplary method (1100) includes the step of obtaining log data describing a reference action of a reference vehicle in an environment—a reference action occurring in the initial state of the environment. The reference action (e.g., reference action (410)) may correspond to a protection action (e.g., protection action against the occurrence of an event such as a low-probability event). The reference action (410) may indicate the vehicle state of the reference vehicle at a specific point in time.

[0133] In 1104, the exemplary method (1100) includes the step of using an operating system to determine a planned action for a vehicle being simulated in an initial state of the environment. For example, based on a snapshot of the environment at the time of executing a reference action (410), the operating system may be a SUT (412) capable of generating (or being used to generate) a planned action for a target vehicle (406) being simulated. For example, the operating system may include at least one model of an autonomous vehicle control system configured to receive sensor data and control the movement of the autonomous vehicle. For example, the SUT (412) may include one or more parts of a perception system including perception sensors (e.g., cameras, lidar, etc.) or perception models (e.g., machine learning-based perception models for object recognition, tracking, etc.). For example, the SUT (412) may include a motion planning model.

[0134] In 1106, the exemplary method (1100) includes the step of simulating (i) an SUT state of the environment obtained by the simulated vehicle executing a planned action in an initial state of the environment and the actor performing an actor action following the simulated vehicle executing the planned action, and (ii) a reference state of the environment obtained by the simulated vehicle executing a reference action in an initial state of the environment and the actor performing an actor action following the simulated vehicle executing the reference action. For example, FIGS. 6a-6c illustrate an exemplary simulation of an SUT state of the environment (e.g., FIG. 6c) obtained by the simulated vehicle executing a planned SUT action (602) (e.g., FIG. 6a) and the actor performing an actor action (614) (e.g., FIG. 6b) following the simulated vehicle executing the planned SUT action (602). For example, FIGS. 5a-5c illustrate exemplary simulations of a reference state of an environment (e.g., FIG. 5c) obtained as a simulated vehicle performs a reference action (502) (e.g., FIG. 5a) and subsequently, an actor performs an actor action (614) (e.g., FIG. 5b) following the simulated vehicle performing the reference action (502).

[0135] In 1108, the exemplary method (1100) includes the step of determining a test score based on the SUT state of the environment and a reference score based on the reference state of the environment. For example, the score or severity may be determined based on the environment state (430) and the environment state (440). For example, the exemplary method (1100) may include the step of evaluating the state of the vehicle simulated in the SUT state of the environment and the state of the vehicle simulated in the reference state of the environment using a cost function. The cost function may be determined based on the distance to the actor boundary. The cost function may be determined based on the severity measurement of the intersection with the actor boundary.

[0136] In 1110, the exemplary method (1100) includes the step of evaluating an operating system based on test scores and reference scores. For example, the step of evaluating an operating system may include the step of measuring the difference between the scores. In some embodiments, the results of one or more verification scenarios may be combined, and the scores may be evaluated for groups or intervals of verification scenarios. For example, the exemplary method (1100) may include the step of classifying test scores and reference scores based on one or more features of the initial state of the environment.

[0137] An exemplary method (1100) may include a step of determining one or more categories for improvement related to lane test scores. For example, a verification system (400) may provide multiple test scores related to verification scenarios of multiple different categories. Differences between the scores of different categories may identify categories that may need improvement. An exemplary method (1100) may include a step of adjusting one or more parameters of an operating system corresponding to the operation of an autonomous vehicle in one or more categories for improvement (e.g., one or more hardware configurations, one or more machine learning model parameters, etc.).

[0138] In some embodiments of the exemplary method, the actor action includes a risk affecting the simulated vehicle. In some embodiments of the exemplary method, the actor action is selected from a plurality of actions having a probability less than a threshold probability. In some embodiments of the exemplary method, the exemplary method includes the step of determining a response to a risk affecting the simulated vehicle.

[0139] In some embodiments of the exemplary method, the reference action may be a defensive driving action to prevent the occurrence of a hazard. For example, the hazard may be an early cut-in by an actor in a merging zone. The defensive driving action may be a coasting action or a braking action.

[0140] FIG. 12 illustrates an exemplary embodiment of 1100 including a multi-layered simulation environment. (e.g., in 1104 to perform the exemplary method (1100)) the first simulation may process an environment (e.g., initial state data describing the environment) and output a planned action. For example, in 1204, the exemplary embodiment may include the step of using an operating system in the first simulation to determine a planned action for a vehicle being simulated from an initial state of the environment. The first simulation may simulate the operations of system(s) under test (e.g., one or more components of an autonomous vehicle computing system) in the environment being simulated. For example, the first simulation may input sensor data being simulated into the autonomous vehicle control system(s) being simulated to obtain simulation-based control decisions.

[0141] In the optional loop (1206), for individual actor actions of one or more actor actions by an actor in the environment, an exemplary embodiment in 1208 may include, from a second simulation: (i) a state of the environment obtained according to individual actor actions occurring subsequently as the vehicle being simulated executes a planned action in the initial state of the environment, and (ii) a reference state of the environment obtained according to individual actor actions occurring subsequently as the vehicle being simulated executes a reference action in the initial state of the environment. The second simulation may be the same as or different from the first simulation. The second simulation may be implemented by the same simulator as the first simulation. The second simulation may be implemented by a different simulator than the first simulation. The second simulation may be calculated based on approximations. For example, the second simulation may include calculating conservative estimates for stopping distance, deceleration speed, lane change speed, etc. For example, the second simulation may be configured to efficiently estimate the outer boundary of the response to the actor action (e.g., upper limit of stopping time or distance). For example, the second simulation may include calculating simplified analytical equations that model target vehicle motion as projectile trajectories. In this way, for example, the second simulation can estimate SUT states and reference states in a cost-effective manner. The second simulation can calculate SUT states and reference states for various actor motions. For example, given a planned motion and a reference motion, the readiness of the target vehicle to respond to various actor motions can be evaluated by estimating the resulting SUT states and reference states in the second simulation.

[0142] FIG. 13 illustrates a flowchart of a method (1300) for training one or more machine learning-based operational models according to embodiments of the present disclosure. For example, an operational system (e.g., SUT (412)) may include a machine learning-based operational model.

[0143] One or more parts(s) of the method (1300) may be implemented by a computing system comprising one or more computing devices, such as computing systems described with reference to other drawings (e.g., autonomous platform (110), vehicle computing system (180), remote system(s) (160), system of FIG. 14, etc.). Each individual part of the method (1300) may be performed by any (or any combination of) of the one or more computing devices. Additionally, one or more parts(s) of the method (1300) may be implemented for hardware components of the device(s) described herein (e.g., as in FIG. 1, 2, 14, etc.) to verify one or more systems or models. FIG. 13 depicts elements performed in a specific order for the purposes of explanation and discussion. Those skilled in the art will understand that, using the disclosure provided herein, elements of any of the methods discussed herein may be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the disclosure. FIG. 13 is described by reference to elements / terms described in relation to other systems and drawings for exemplary illustrative purposes and is not intended to be limiting. One or more parts of the method (1300) may be performed additionally or substituted by other systems.

[0144] In 1302, the method (1300) may include the step of obtaining training data for training a machine learning-based operational model. The training data may include a plurality of training instances (e.g., reference plan data such as labeled trajectories or strategies based on expert demonstrations).

[0145] Training data may be collected using one or more autonomous platforms (e.g., autonomous platform (110)) or when the autonomous platform is within the environment using the sensors. As an example, training data may be collected using one or more autonomous vehicle(s) (e.g., autonomous platform (110), autonomous vehicle (310), autonomous vehicle (350), etc.) or when the vehicle(s) operate along one or more travel paths using the sensors. In some examples, training data may be collected using other sensors, such as mobile device-based sensors, ground-based sensors, air-based sensors, satellite-based sensors, or any sensor interface configured to acquire and / or record measurement data substantially.

[0146] Training data may include multiple training sequences partitioned among multiple datasets (e.g., training dataset, validation dataset, or test dataset). Each training sequence may include multiple pre-recorded cognitive data points, point clouds, images, etc. In some embodiments, each sequence may include LiDAR point clouds (e.g., collected using LiDAR sensors of an autonomous platform) and images (e.g., collected using mono or stereo imaging sensors). For example, in some embodiments, multiple images may be scaled for training and evaluation.

[0147] In 1304, the method (1300) may include the step of selecting a training instance based on at least partially the training data.

[0148] In 1306, the method (1300) may include the step of inputting a training instance into a machine learning-based operational model.

[0149] In 1308, the method (1300) may include the step of generating one or more loss indicator(s) and / or one or more objective(s) for a machine learning-based operating model based on at least some of the output(s) and label(s) associated with training instances of the machine learning-based operating model.

[0150] In 1310, the method (1300) may include the step of modifying at least one parameter of at least a portion of a machine learning-based operating model based on at least one of loss indicator(s) and / or at least one of purpose(s). For example, a computing system may modify at least a portion of a machine learning-based operating model based on at least one of loss indicators and / or at least one of purpose(s).

[0151] In some implementations, machine learning-based operational models can be trained in an end-to-end manner. For example, in some implementations, machine learning-based operational models can be completely differentiated.

[0152] After being updated, the operating model or the operating system including the operating model may be provided for verification (e.g., according to exemplary implementations of the method (1100), etc.). In some implementations, the verification system (400) may evaluate or verify the operating system to identify underperforming areas in comparison to a whole set of reference targets. The verification system (400) may trigger the retraining, decommissioning, etc. of the operating system based on failure to satisfy verification thresholds in one or more areas (e.g., preventing one or more risks).

[0153] In some embodiments, the verification system (400) is implemented to provide cost functions for training the operating system to behave more similarly to reference targets (e.g., for training one or more components of a machine learning-based behavior planning system). In some embodiments, the verification system (400) is implemented to apply penalties to non-conforming behaviors that do not conform to the reference behavior.

[0154] FIG. 14 is a block diagram of an exemplary computing ecosystem (10) according to exemplary embodiments of the present disclosure. The exemplary computing ecosystem (10) may include a first computing system (20) and a second computing system (40) that are coupled to communicate through one or more networks (60). In some embodiments, the first computing system (20) or the second computing system (40) may implement one or more of the systems, operations, or functions described herein (e.g., remote system(s) (160), onboard computing system(s) (180), autonomous system (200), etc.) to verify one or more systems or operating systems.

[0155] In some embodiments, the first computing system (20) may be included in an autonomous platform and utilized to perform the functions of the autonomous platform as described herein. For example, the first computing system (20) may be located onboard in an autonomous vehicle and may implement autonomous system(s) for autonomously operating the autonomous vehicle. In some embodiments, the first computing system (20) may represent the entire onboard computing system or a part thereof (e.g., a positioning system (230), a perception system (240), a planning system (250), a control system (260), or a combination thereof). In other embodiments, the first computing system (20) may not be located onboard in an autonomous platform. The first computing system (20) may include one or more independent physical computing devices (21).

[0156] The first computing system (20) (e.g., the computing device(s) (21)) may include one or more processors (22) and memory (23). The one or more processors (22) may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or a plurality of processors operatively connected. The memory (23) may include one or more computer-readable storage media, e.g., RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc. and combinations thereof.

[0157] Memory (23) may store information that can be accessed by one or more processors (22). For example, memory (e.g., one or more non-transient computer-readable storage media, memory devices, etc.) may store data (24) that can be acquired (e.g., received, accessed, recorded, manipulated, formed, created, stored, pulled, downloaded, etc.). Data (24) may include, for example, sensor data, map data, data associated with autonomous functions (e.g., data associated with cognitive, planning, or control functions), simulation data, or any data or information described herein. In some embodiments, the first computing system (20) may acquire data from one or more memory devices remote from the first computing system (20).

[0158] Memory (23) can store computer-readable instructions (25) that can be executed by one or more processors (22). The instructions (25) may be software written in any suitable programming language or implemented in hardware. Additionally or alternatively, the instructions (25) may be executed in logically or virtually separated threads on the processor(s) (22).

[0159] For example, memory (23) may store instructions (25) executable by one or more processors (e.g., one or more processors (22), one or more other processors, etc.) to perform any of the operations, functions, or methods / processes (or parts thereof) described herein (e.g., computing device(s) (21), the first computing system (20), or other system(s) having processors that execute instructions). For example, operations may include implementing system verification (e.g., as described herein).

[0160] In some embodiments, the first computing system (20) may store or contain one or more models (26). In some embodiments, the models (26) may be one or more machine learning models (e.g., machine learning-based operating systems, etc.) or otherwise contain them. As examples, the models (26) may be various machine learning models, such as regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear and non-linear models, or otherwise contain them. Examples of neural networks include forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the first computing system (20) may include one or more models for implementing subsystems of autonomous system(s) (200), including any of a location identification system (230), a perception system (240), a planning system (250), or a control system (260).

[0161] In some embodiments, the first computing system (20) may acquire one or more models (26) that communicate with the second computing system (40) via the network(s) (60) using the communication interface(s) (27). For example, the first computing system (20) may store the model(s) (e.g., one or more machine learning models) in memory (23). Then, the first computing system (20) may use the models (26) or otherwise implement the models (e.g., by the processors (22)). For example, the first computing system (20) may implement the model(s) (26) to locate the autonomous platform within an environment, recognize the environment of the autonomous platform or objects within it, plan one or more future states of the autonomous platform for movement through the environment, and control the autonomous platform to interact with the environment.

[0162] The second computing system (40) may include one or more computing devices (41). The second computing system (40) may include one or more processors (42) and memory (43). The one or more processors (42) may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or a plurality of processors operatively connected. The memory (43) may include one or more computer-readable storage media, e.g., RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.

[0163] Memory (43) may store information that can be accessed by one or more processors (42). For example, memory (43) (e.g., one or more non-transient computer-readable storage media, memory devices, etc.) may store obtainable data (44). Data (44) may include, for example, sensor data, model parameters, map data, simulation data, simulated environmental scenes, simulated sensor data, data related to vehicle movement / service, or any data or information described herein. In some embodiments, the second computing system (40) may obtain data from one or more memory devices remote from the second computing system (40).

[0164] Memory (43) may also store computer-readable instructions (45) that can be executed by one or more processors (42). The instructions (45) may be software written in any suitable programming language or implemented in hardware. Additionally, or alternatively, the instructions (45) may be executed on logically or virtually separated threads on the processor (42).

[0165] For example, memory (43) may store instructions (45) executable (e.g., by one or more processors (42), one or more processors (22), one or more other processors, etc.) to perform any of the operations, functions, or methods / processes described herein (e.g., by one or more processors (42), one or more processors (22), one or more other processors, etc.) to perform any of the operations, functions, or methods / processes described herein (e.g., by one or more processors (42), one or more processors (22), one or more other processors, etc.). This may include, for example, functions of the autonomous system(s) (200) (e.g., location, perception, planning, control, etc.) or other functions related to the autonomous platform (e.g., remote assistance, mapping, vehicle management, movement / service allocation and matching, etc.). This may also include, for example, verifying a machine learning-based operating system.

[0166] In some embodiments, the second computing system (40) may include one or more server computing devices. If the second computing system (40) includes multiple server computing devices, the server computing devices may operate according to various computing architectures, such as sequential computing architecture, parallel computing architecture, or some combination thereof.

[0167] In addition to or alternatively to the model(s) (26) of the first computing system (20), the second computing system (40) may include one or more models (46). For example, the models (46) may be one or more machine learning models (e.g., machine learning-based operating systems, etc.) or otherwise include them. As examples, the models (26) may be various machine learning models, such as regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear and non-linear models or otherwise include them. Examples of neural networks include forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the second computing system (40) may include one or more models of the autonomous system(s) (200).

[0168] In some embodiments, the second computing system (40) or the first computing system (20) may train one or more machine learning models of model(s) (26) or model(s) (46) by using one or more model trainer(s) (47) and training data (48). The model trainer(s) (47) may train either of model(s) (26) or model(s) (46) using one or more training or learning algorithms. One example training technique is backpropagation of errors. In some embodiments, the model trainer(s) (47) may perform supervised training techniques using labeled training data. In other embodiments, the model trainer(s) (47) may perform unsupervised training techniques using unlabeled training data. In some embodiments, the training data (48) may include simulated training data (e.g., training data obtained from simulated scenarios, inputs, configurations, environments, etc.). In some embodiments, the second computing system (40) may implement simulations to acquire training data (48) or to implement model trainer(s) (47) for training or testing model(s) (26) or model(s) (46). For example, the model trainer(s) (47) may train one or more components of a machine learning model for autonomous system(s) (200) through unsupervised training techniques using an objective function (e.g., cost, reward, heuristics, constraints, etc.). In some embodiments, the model trainer(s) (47) may perform a number of generalization techniques to improve the generalization ability of the model(s) being trained. Generalization techniques include weight decays, dropouts, or other techniques.

[0169] For example, in some embodiments, the second computing system (40) may generate training data (48) according to exemplary aspects of the present disclosure. For example, the second computing system (40) may generate training data (48). For example, the second computing system (40) may implement a method according to exemplary aspects of the present disclosure. The second computing system (40) may train model(s) (26) using training data (48). For example, in some embodiments, the first computing system (20) may include a computing system mounted on or otherwise associated with an actual or simulated autonomous vehicle. In some embodiments, the model(s) (26) may include a cognitive or machine vision model(s) configured to be mounted on or distributed in a manner that is serviced for an actual or simulated autonomous vehicle. In this way, for example, the second computing system (40) may provide a training pipeline for training the model(s) (26).

[0170] The first computing system (20) and the second computing system (40) may each include communication interfaces (27, 49). The communication interfaces (27, 49) may be used to communicate with one or more other systems or devices, including systems or devices located remotely from the first computing system (20) or the second computing system (40). The communication interfaces (27, 49) may include any circuits, components, software, etc., for communicating with one or more networks (e.g., network(s) (60)). In some embodiments, the communication interfaces (27, 49) may include, for example, one or more of a communication controller, a receiver, a transceiver, a transmitter, a port, conductors, software or hardware for data communication.

[0171] Network(s) (60) may be any type of network or combination of networks that allows communication between devices. In some embodiments, network(s) may include one or more of a local area network, a wide area network, the Internet, a security network, a cellular network, a mesh network, a peer-to-peer communication link, or some combination thereof, and may include any number of wired or wireless links. Communication through network(s) (60) may be achieved, for example, through a network interface using any type of protocol, protection scheme, encoding, format, packaging, etc.

[0172] FIG. 14 illustrates an example computing ecosystem (10) that may be used to implement the present disclosure. Other systems may also be used. For example, in some embodiments, the first computing system (20) may include model trainer(s) (47) and training data (48). In these embodiments, the model(s) (26, 46) may all be trained and used locally in the first computing system (20). As another example, in some embodiments, the computing system (20) may not be connected to another computing system. Additionally, components exemplified or discussed as being included in one of the computing systems (20 or 40) may instead be included in the other of the computing systems (20 or 40).

[0173] Computing tasks discussed herein as being performed on a remote computing device(s) from an autonomous platform (e.g., an autonomous vehicle) may instead be performed on the autonomous platform (e.g., through the vehicle computing system of the autonomous vehicle), or vice versa. Such configurations may be implemented without departing from the scope of this disclosure. Using computer-based systems allows for a wide variety of possible configurations, combinations, and partitions between or among components. Computer-implemented operations may be performed on a single component or across multiple components. Computer-implemented tasks or operations may be performed sequentially or in parallel. Data and instructions may be stored in a single memory device or across multiple memory devices.

[0174] Various aspects of the present disclosure have been described in terms of exemplary embodiments. To those skilled in the art, a number of other embodiments, variations, or modifications may be devised within the scope and spirit of the appended claims by reviewing the present disclosure. Any features described in the following claims may be combined or rearranged in any possible manner. Accordingly, the scope of the present disclosure is illustrative rather than restrictive, and the present disclosure does not exclude the inclusion of such variations, modifications, or additions to the subject matter that may be obvious to those skilled in the art. Additionally, terms are described in this specification using lists of exemplary elements connected by conjunctions such as "and," "or," "but," etc. It should be understood that these conjunctions are provided for illustrative purposes only. For example, lists connected by specific conjunctions such as "or" may mean "at least one" or "any combination" of the exemplary elements listed therein, and unless otherwise specified, "or" is understood as "and / or." In addition, terms such as "based on" should be understood as "at least partially based."

[0175] Those skilled in the art will understand that, using the disclosure provided herein, elements of any of the claims, actions, or processes discussed herein may be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of the disclosure. Some claims are described with character references to claim elements for illustrative purposes, and this is not intended to be limiting. Character references do not imply a specific order of actions. For example, character identifiers such as (a), (b), (c), ..., (i), (ii), (iii), ..., etc. may be used to describe actions. These identifiers are provided for the convenience of the reader and do not indicate a specific order of steps or actions. An action exemplified by a list identifier such as (a), (i), etc. may be performed before, after, or in parallel with another action exemplified by a list identifier such as (b), (ii).

Claims

Claim 1 A method for verifying a System Under Test (SUT) for operating an autonomous vehicle, comprising: (a) acquiring log data describing an exemplar action of a reference vehicle in an environment—wherein the exemplar action is an action occurring in the initial state of the environment—; (b) determining a planned action for a vehicle simulated in the initial state of the environment using the operating system; (c) (i) simulating the response of the simulated vehicle to an actor performing an actor action following the execution of the planned action by the simulated vehicle in the initial state of the environment in order to acquire the SUT (system under test) state of the environment; (ii) simulating the response of the simulated vehicle to an actor performing an actor action following the execution of the exemplar action by the simulated vehicle in the initial state of the environment in order to acquire the exemplar state of the environment; (d) determining a test score based on the SUT state of the environment and a reference score based on the exemplar state of the environment; and (e) evaluating the operating system based on the test score and the exemplar score. Claim 2 A method according to claim 1, wherein the actor action includes a risk affecting the simulated vehicle. Claim 3 A method according to claim 1, wherein (c) comprises the step of simulating (i) and (ii) using a model that approximates the operation of the simulated vehicle. Claim 4 The method of claim 1, wherein (c) comprises the step of injecting the actor operation into the initial state of the environment. Claim 5 A method according to claim 2, wherein the reference operation includes a defensive operation that defends against the occurrence of the risk. Claim 6 A method according to claim 4, wherein the actor action is a composite actor action based on a set of possible actions that the actor can take in the initial state of the environment. Claim 7 The method of claim 1, wherein the operating system comprises at least one model of an autonomous vehicle control system configured to receive sensor data and control the movement of the autonomous vehicle. Claim 8 A method according to claim 4, wherein the actor action is based on a recorded actor action obtained from other log data. Claim 9 A method according to claim 1, wherein the test score is based on the state of the simulated vehicle in the SUT state of the environment, and the reference score is based on the state of the simulated vehicle in the reference state of the environment. Claim 10 A method according to claim 9, wherein (d) a step of evaluating the state of the simulated vehicle in the SUT state of the environment and the state of the simulated vehicle in the reference state of the environment using a cost function. Claim 11 In paragraph 10, the above cost function is determined based on the distance to the actor boundary. Claim 12 A method according to claim 10, wherein the cost function is determined based on a measurement of the severity of the intersection with the actor boundary. Claim 13 A method according to claim 1, wherein (e) comprises the step of classifying the test score and the reference score based on one or more features of the initial state of the environment. Claim 14 A method according to claim 13, comprising: (f) determining one or more categories for improvement associated with test scores; and (g) adjusting one or more parameters of the operating system corresponding to the operation of the autonomous vehicle in the one or more categories for improvement. Claim 15 One or more computer-readable media storing executable instructions that cause one or more processors to perform actions for verifying a System Under Test (SUT) for operating an autonomous vehicle, wherein the actions include: (a) acquiring log data describing an exemplar action of a reference vehicle in an environment—wherein the exemplar action is an action occurring in the initial state of the environment; (b) determining a planned action for a vehicle simulated in the initial state of the environment using the operating system; (c) (i) simulating the response of the simulated vehicle to an actor performing an actor action following the execution of the planned action by the simulated vehicle in the initial state of the environment in order to acquire the SUT (system under test) state of the environment; (ii) simulating the response of the simulated vehicle to an actor performing an actor action following the execution of the exemplar action by the simulated vehicle in the initial state of the environment in order to acquire the reference state of the environment; (d) determining a test score based on the SUT state of the environment and a reference score based on the reference state of the environment. A computer-readable medium comprising the steps of: (e) evaluating the operating system based on the test score and the reference score. Claim 16 A computer-readable medium according to claim 15, wherein the actor action includes a risk affecting the simulated vehicle. Claim 17 A non-transient computer-readable medium according to claim 15, wherein the actor action is selected from a plurality of actions having a probability smaller than a threshold probability. Claim 18 In claim 16, (c) a computer-readable medium comprising the step of determining a response to a risk where the simulated vehicle is affected. Claim 19 A computer-readable medium according to claim 16, wherein the reference operation is a defensive operation to prevent the occurrence of danger. Claim 20 As an autonomous vehicle control system for controlling an autonomous vehicle, the autonomous vehicle control system comprises: one or more processors; and includes one or more non-transient computer-readable media storing instructions executable by one or more processors for enabling the autonomous vehicle control system to control the operation of the autonomous vehicle using the operating system; the operating system comprises: (a) acquiring log data describing an exemplar action of a reference vehicle in an environment—the exemplar action is an action occurring in the initial state of the environment—; (b) determining a planned action for a vehicle simulated in the initial state of the environment using the operating system; (c) (i) simulating the response of the simulated vehicle to an actor performing an actor action following the execution of the planned action by the simulated vehicle in the initial state of the environment in order to acquire the SUT (system under test) state of the environment; (ii) simulating the response of the simulated vehicle to an actor performing an actor action following the execution of the exemplar action by the simulated vehicle in the initial state of the environment in order to acquire the reference state of the environment; (d) a test score based on the SUT state of the environment and the reference state of the environment An autonomous vehicle control system verified by the step of determining a reference score based on the test score and the reference score; and (e) the step of evaluating the operating system based on the test score and the reference score.

Images