Autonomous vehicle verification system and method

Abstract

An exemplary method includes the steps of: obtaining log data describing the reference behavior of a reference vehicle in an environment—a reference behavior occurring in the initial state of the environment; determining the planned behavior of a vehicle to be simulated in the initial state of the environment using an operating system; simulating the SUT state of the environment obtained by the simulated vehicle performing the planned behavior in the initial state of the environment and the actor performing an actor behavior following the planned behavior, and simulating the reference state of the environment obtained by the simulated vehicle performing the reference behavior in the initial state of the environment and the actor performing an actor behavior following the reference behavior; determining a test score based on the SUT state and a reference score based on the reference state; and evaluating the operating system based on the test score and the reference score.

Description

Technical Field

[0001] Priority This application claims priority and benefit to U.S. Provisional Application No. 63 / 491,644, filed on March 22, 2023, and U.S. Patent Application No. 18 / 309,494, filed on April 28, 2023. U.S. Provisional Application No. 63 / 491,644 and U.S. Patent Application No. 18 / 309,494 are hereby incorporated by reference in their entirety.

Background Art

[0002] An autonomous platform can process data to recognize the environment in which the autonomous platform operates. For example, an autonomous vehicle can use various sensors to recognize the surrounding environment and identify objects around the autonomous vehicle. The autonomous vehicle can identify an appropriate route through the recognized surrounding environment and move along the route with minimal or no human intervention.

Summary of the Invention

[0003] This disclosure relates to a technique for verifying a system for operating an autonomous vehicle. The verification technique according to this disclosure can comprehensively evaluate an end-to-end autonomous computing pipeline or a selected part thereof. Exemplary embodiments can put a simulated autonomous vehicle into a driving scenario and compare the planned actions of the autonomous vehicle with the actions of a reference vehicle within the same scenario.

[0004] For example, log data can record the actions of human reference subjects in various driving scenarios. Exemplary embodiments can simulate the results obtained by a simulated autonomous vehicle executing planned actions and can simulate the results obtained by the simulated autonomous vehicle executing reference actions. In this way, the results achieved by the operating system of the autonomous vehicle can be verified against a baseline provided by a human reference subject.

[0005] The verification techniques described herein can leverage the expertise of human drivers when encountering specific scenarios (e.g., edge case scenarios). For example, a human driver may, in general, brake or slow down when navigating a road merging area, even if they have priority in their lane, if another vehicle is merging into their lane. The driver's choice reflects a complex balance of factors such as the risk of an unexpected early cut-in by another vehicle, the cost of not being prepared for such an unexpected early cut-in, and the cost of slowing down. Thus, broadly speaking, the collective behavior of human references can provide a framework for understanding the expected behavior of a vehicle in various driving scenarios. Therefore, one exemplary technique for evaluating the behavior of an autonomous vehicle (and the operating system that controls that behavior) involves comparison with a human reference.

[0006] In some cases, directly comparing the behavior of an autonomous vehicle to a human reference involves counting the number of interventions by a human operator supervising the autonomous vehicle. The goal is to reduce the number of interventions and allow the autonomous vehicle to drive naturally in a manner that a human operator would expect and accept.

[0007] In some cases, the verification techniques described herein may determine a proxy for a human operator intervention event by comparing a reference behavior recorded (e.g., using log data) in a given driving scenario with the planned behavior of an autonomous vehicle placed in the same scenario. By evaluating the deviation between the behaviors—and the resulting potential consequences—the autonomous vehicle's operating system can be evaluated against a baseline.

[0008] Advantageously, the verification techniques described herein can evaluate the performance of an operating system for an autonomous vehicle without requiring manually adjusted, hard-coded, complex costs and arbitrary thresholds and ranges. Exemplary embodiments can evaluate the performance of an operating system for an autonomous vehicle without explicitly modeling decision heuristics by utilizing a large dataset recording millions of expert decision-making instances.

[0009] The verification techniques disclosed herein also advantageously allow for direct comparisons with reference data without requiring extensive and unlimited accumulation of actual or simulated mileage and data samples related to the occurrence of potential scenarios. Such accumulation, even when simulated, can consume considerable resources (e.g., computing resources, energy resources, etc.) and be time-consuming. Advantageously, direct comparisons with reference behavior can be obtained at a much higher density by deploying simulated autonomous vehicles into scenarios derived from existing log data.

[0010] The verification techniques disclosed herein can also be advantageous in comprehensively evaluating the impact of individual model developments on the overall results achieved by autonomous vehicles. For example, an autonomous computing pipeline can implement numerous complex systems and models to perceive and understand the environment and generate and execute plans for navigating that environment. Through a comprehensive performance evaluation of a reference baseline, the impact of changes to individual operating systems and systems on the overall performance of the vehicle can be assessed. In this way, performance differences that deviate from reference behavior can be prioritized in development over performance differences that ultimately do not contribute to performance defects. Therefore, the verification techniques disclosed herein can support more efficient and effective system and model development.

[0011] For example, in one embodiment, the Disclosure provides an exemplary method for verifying an operating system for operating an autonomous vehicle. In some embodiments, the exemplary method includes the step of (a) obtaining log data describing an Exemplar Action of an Exemplar Action of a reference vehicle in an environment—an Exemplar Action being an action that occurs in the initial state of the environment. In some embodiments, the exemplary method includes the step of (b) determining a planned action for a vehicle to be simulated in the initial state of the environment using the operating system. In some embodiments, the exemplary method includes the step of (c) simulating the System Under Test (SUT) state of the environment obtained by the simulated vehicle performing a planned action in the initial state of the environment and an actor performing an actor action following the simulated vehicle performing the planned action, and (ii) simulating the reference state of the environment obtained by the simulated vehicle performing an Exemplar Action in the initial state of the environment and an actor performing an actor action following the simulated vehicle performing the Exemplar Action. In some embodiments, the exemplary method includes the step of (d) determining a test score based on the SUT state of the environment and a reference score based on the reference state of the environment. In some embodiments, exemplary methods include (e) evaluating the operating system based on test scores and reference scores.

[0012] In some embodiments of the exemplary method, actor actions include hazards affecting the simulated vehicle. In some embodiments of the exemplary method, reference actions are defensive driving actions to prevent the occurrence of hazards.

[0013] In some embodiments of the exemplary method, the actor action is selected from a plurality of actions having a probability lower than a threshold probability.

[0014] In some embodiments of the exemplary method, (c) includes the step of determining a response to a hazard affecting the simulated vehicle.

[0015] In some embodiments of the exemplary method, the reference operation includes the vehicle state of a human-driven vehicle.

[0016] In some embodiments of the exemplary method, the operating system includes at least one model of an autonomous vehicle control system configured to receive sensor data and control the movement of the autonomous vehicle. In some embodiments of the exemplary method, the operating system includes a motion planning model. In some embodiments of the exemplary method, the operating system includes a perception model.

[0017] In some embodiments of the exemplary method, the reference operation is a defensive operation.

[0018] In some embodiments of the exemplary method, (c) (for example, simulating a first state and a reference state) includes the step of simulating an action toward an actor in the environment, following the simulated vehicle performing a planned action and a reference action. In some embodiments of the exemplary method, the action is selected from a plurality of actions having a probability lower than a threshold probability. In some embodiments of the exemplary method, the action toward an actor includes a hazard that affects the simulated vehicle.

[0019] In some embodiments of the exemplary method, (c) (for example, simulating a first state and a reference state) includes the step of simulating a response to a hazard affecting the simulated vehicle.

[0020] In some embodiments of the exemplary method, the test score is based on the vehicle state simulated in the SUT state of the environment, and the reference score is based on the vehicle state simulated in the reference state of the environment.

[0021] In some embodiments of the exemplary method, (d) (e.g., determining the test score) includes the step of evaluating the vehicle state simulated in the SUT state of the environment and the vehicle state simulated in the reference state of the environment using a cost function. In some embodiments of the exemplary method, the cost function is determined based on the distance to the actor boundary. In some embodiments of the exemplary method, the cost function is determined based on a severity measure of the intersection with the actor boundary.

[0022] In some embodiments of the exemplary method, (e) (e.g., evaluation of an operating system) includes the step of classifying test scores and reference scores based on one or more characteristics of the initial state of the environment.

[0023] In some embodiments of the exemplary method, the exemplary method includes (f) determining one or more categories for improvement related to a suboptimal test score, and (g) adjusting one or more parameters of the operating system corresponding to the behavior of the autonomous vehicle in one or more categories for improvement.

[0024] For example, in one embodiment, the Disclosure provides one or more exemplary non-temporary computer-readable media for storing executable instructions for causing one or more processors to perform an operation. In some embodiments, the operation includes (a) an operation to obtain log data describing a reference operation of a reference vehicle in an environment—a reference operation being an operation that occurs in the initial state of the environment. In some embodiments, the operation includes (b) an operation to determine a planned operation for a vehicle to be simulated in the initial state of the environment using an operating system. In some embodiments, the operation includes (c) an operation to simulate the SUT state of the environment obtained by the simulated vehicle performing a planned operation in the initial state of the environment and, following the simulated vehicle performing the planned operation, an actor performing an actor operation; and (ii) an operation to simulate the reference state of the environment obtained by the simulated vehicle performing a reference operation in the initial state of the environment and, following the simulated vehicle performing the reference operation, an actor performing an actor operation. In some embodiments, the operation includes (d) an operation to determine a test score based on the SUT state of the environment and a reference score based on the reference state of the environment. In some embodiments, the operation includes (e) an operation to evaluate the operating system based on the test score and the reference score.

[0025] In some embodiments of one or more non-temporary computer-readable media, actor actions include hazards affecting the simulated vehicle. In some exemplary embodiments of one or more non-temporary computer-readable media, reference actions are defensive driving actions to prevent the occurrence of hazards.

[0026] In some embodiments of one or more exemplary non-transient computer-readable media, the actor action is selected from a plurality of actions having a probability lower than a threshold probability.

[0027] In some embodiments of one or more exemplary non-transitory computer-readable media, (c) includes an operation of determining a reaction to a danger that affects the simulated vehicle.

[0028] In some embodiments of one or more exemplary non-transitory computer-readable media, the reference operation includes the vehicle state of a vehicle driven by a human.

[0029] In some embodiments of one or more exemplary non-transitory computer-readable media, the reference operation is an operation of defensive driving.

[0030] In some embodiments of one or more exemplary non-transitory computer-readable media, (c) (e.g., simulating the first state and the reference state) includes an operation of simulating an operation on an actor in the environment following the simulated vehicle executing the planned operation and the reference operation.

[0031] For example, in one embodiment, the Disclosure provides an exemplary autonomous vehicle control system for controlling an autonomous vehicle. In some embodiments, the exemplary autonomous vehicle control system includes one or more processors and one or more non-temporary computer-readable media that store instructions executable by the one or more processors, causing the autonomous vehicle control system to control the operation of the autonomous vehicle using an operating system. In some embodiments, the operating system is verified by (a) obtaining log data describing the reference operation of a reference vehicle in an environment—a reference operation being an operation that occurs in the initial state of the environment. In some embodiments, the operating system is verified by (b) determining a planned operation for a vehicle simulated in the initial state of the environment using the operating system. In some embodiments, the operating system is verified by (c) simulating (i) the SUT state of the environment obtained by the simulated vehicle performing the planned operation in the initial state of the environment and the actor performing an actor operation following the simulated vehicle performing the planned operation, and (ii) the reference state of the environment obtained by the simulated vehicle performing the reference operation in the initial state of the environment and the actor performing an actor operation following the simulated vehicle performing the reference operation. In some embodiments, the operating system was validated by (d) determining a test score based on the SUT state of the environment and a reference score based on the reference state of the environment. In some embodiments, the operating system was validated by (e) evaluating the operating system based on the test score and the reference score.

[0032] Other exemplary embodiments of this disclosure relate to other systems, methods, vehicles, apparatus, tangible non-temporary computer-readable media and devices for performing the functions described herein. These and other features, aspects and advantages of various embodiments will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and forming part of this specification are used to illustrate embodiments of this disclosure and to illustrate relevant principles together with the description. [Brief explanation of the drawing]

[0033] A detailed description of embodiments presented to those skilled in the art is given herein with reference to the accompanying drawings.

[0034] [Figure 1] This is a block diagram illustrating an exemplary operating scenario according to some embodiments of the present disclosure.

[0035] [Figure 2] This is a block diagram of an exemplary system according to some embodiments of the present disclosure.

[0036] [Figure 3a] This figure shows an exemplary operating environment according to some embodiments of the present disclosure.

[0037] [Figure 3b] This figure shows an exemplary operating environment map according to some embodiments of the present disclosure.

[0038] [Figure 3c] This figure shows an exemplary operating environment according to some embodiments of the present disclosure.

[0039] [Figure 3d] This figure shows an exemplary operating environment map according to some embodiments of the present disclosure.

[0040] [Figure 4]This is a block diagram of an exemplary system for performing system verification according to some embodiments of this disclosure.

[0041] [Figure 5a] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0042] [Figure 5b] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0043] [Figure 6a] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0044] [Figure 6b] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0045] [Figure 6c] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0046] [Figure 7a] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0047] [Figure 7b] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0048] [Figure 7c] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0049] [Figure 8a] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0050] [Figure 8b] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0051] [Figure 9a] This is a block diagram of an exemplary system for performing system verification according to some embodiments of this disclosure.

[0052] [Figure 9b] This is a diagram illustrating an exemplary verification scenario according to some embodiments of the present disclosure.

[0053] [Figure 10] This is a decision tree for classifying the verification results according to some embodiments of this disclosure.

[0054] [Figure 11] This is a flowchart illustrating an exemplary method for system verification according to some embodiments of the present disclosure.

[0055] [Figure 12] This is a flowchart illustrating an exemplary method for system verification according to some embodiments of the present disclosure.

[0056] [Figure 13] This is a flowchart illustrating an exemplary method for training and validating a machine learning-based operating system according to some embodiments of the present disclosure.

[0057] [Figure 14] This is a block diagram of an exemplary computing system for performing system verification according to some embodiments of the present disclosure. [Modes for carrying out the invention]

[0058] The following describes the technologies of this disclosure in the context of autonomous vehicles for illustrative purposes only. As described herein, the technologies of this disclosure are not limited to autonomous vehicles but can be implemented in or within other autonomous platforms and other computing systems.

[0059] Exemplary embodiments of the present disclosure will be described in further detail with reference to Figures 1 to 14. Figure 1 is a block diagram of an exemplary operating scenario according to some embodiments of the present disclosure. In the exemplary operating scenario, the environment 100 includes an autonomous platform 110 and a plurality of objects, including a first actor 120, a second actor 130, and a third actor 140. In the exemplary operating scenario, the autonomous platform 110 can move through the environment 100 and interact with objects located within the environment 100 (e.g., the first actor 120, the second actor 130, the third actor 140, etc.). The autonomous platform 110 may be selectively configured to communicate with a remote system 160 via a network 170.

[0060] Environment 100 is or may include an indoor environment (e.g., within one or more facilities) or an outdoor environment. For example, an indoor environment may be an environment enclosed by a structure such as a building (e.g., a service warehouse, maintenance area, manufacturing facility). An outdoor environment may be one or more areas in the outside world, such as one or more rural areas (e.g., having one or more rural travel routes), one or more urban areas (e.g., having one or more urban travel routes, highways, etc.), one or more suburban areas (e.g., having one or more suburban travel routes), or other outdoor environments.

[0061] The autonomous platform 110 may be any type of platform configured to operate within the environment 100. For example, the autonomous platform 110 may be a vehicle configured to autonomously perceive and operate within the environment 100. The vehicle may be, for example, a ground-based autonomous vehicle such as an autonomous car, truck, or van. The autonomous platform 110 may be an autonomous vehicle capable of controlling, connecting to, or otherwise associating with equipment, attachments, and / or accessories for transporting people or cargo. The autonomous platform may include, for example, an autonomous tractor selectively coupled to a cargo trailer. Additionally or alternatively, the autonomous platform 110 may be any other type of vehicle, such as one or more aircraft, water vehicles, space-based vehicles, or other ground-based vehicles.

[0062] The autonomous platform 110 may be configured to communicate with a remote system 160. For example, the remote system 160 may communicate with the autonomous platform 110 for assistance (e.g., navigation assistance, situational assistance, etc.), control (e.g., vehicle management, remote operation, etc.), maintenance (e.g., updates, monitoring, etc.), or other local or remote tasks. In some embodiments, the remote system 160 may provide data instructing the autonomous platform 110 to perform tasks. For example, as further described herein, the remote system 160 may provide data instructing the autonomous platform 110 to perform trips / services such as user transport trips / services, delivery trips / services (e.g., for cargo, transport, or goods).

[0063] The autonomous platform 110 can communicate with the remote system 160 using the network 170. The network 170 can facilitate the transmission of signals (e.g., electronic signals) or data (e.g., data from computing devices) and may include various wired (e.g., twisted-pair cables) or wireless communication mechanisms (e.g., cellular, radio, satellite, microwave, radio frequency, etc.) or any combination of any desired network topology (or multiple topologies). For example, the network 170 may include a short-range network (LAN) (e.g., an intranet), a wide-area network (e.g., the Internet), a wireless LAN network (e.g., via Wi-Fi, etc.), a cellular network, a SATCOM network, a VHF network, an HF network, a WiMAX-based network, or any other suitable communication network (or a combination thereof) for transmitting data into and out of the autonomous platform 110.

[0064] As shown in Figure 1, the environment 100 may include one or more objects. The objects may be objects that are not moving or are not expected to move ("static objects") or objects that are moving or are expected to move ("dynamic objects" or "actors"). In some embodiments, the environment 100 may include any number of actors, such as one or more pedestrians, animals, vehicles, etc. Actors may move within the environment following one or more actor trajectories. For example, a first actor 120 may move following one of the first actor trajectories 122A to C, a second actor 130 may move following one of the second actor trajectories 132, and a third actor 140 may move following one of the third actor trajectories 142.

[0065] As further described herein, the autonomous platform 110 can use its autonomous system to detect these actors (and their movements) and plan its actions so as to travel through the environment 100 according to one or more platform trajectories 112A-C. The autonomous platform 110 may include an onboard computing system 180. The onboard computing system 180 may include one or more processors and one or more memory devices. One or more memory devices can store instructions executable by one or more processors so that one or more processors perform actions or functions related to the autonomous platform 110, including the implementation of the autonomous driving system of the autonomous platform.

[0066] Figure 2 is a block diagram of an exemplary autonomous system 200 for an autonomous platform according to some embodiments of the present disclosure. In some embodiments, the autonomous system 200 can be implemented by the computing system of the autonomous platform (e.g., the onboard computing system 180 of the autonomous platform 110). The autonomous system 200 can operate to acquire input from a plurality of sensors 202 or other input devices. In some embodiments, the autonomous system 200 can further acquire platform data 208 (e.g., map data 210) from local or remote storage. The autonomous system 200 can generate control outputs for controlling the autonomous platform (e.g., via a platform control device 212, etc.) based on the sensor data 204, map data 210, or other data. The autonomous system 200 may include different subsystems for performing various autonomous operations. The subsystems may include a localization system 230, a perception system 240, a planning system 250, and a control system 260. The position estimation system 230 can determine the position of the autonomous platform within its environment; the recognition system 240 can detect, classify, and track objects and actors within the environment; the planning system 250 can determine the trajectory of the autonomous platform; and the control system 260 can translate the trajectory into vehicle control actions for controlling the autonomous platform. The autonomous system 200 can be implemented by one or more onboard computing systems. Subsystems may include one or more processors and one or more memory devices. One or more memory devices may store instructions executable by one or more processors so that one or more processors perform actions or functions related to the subsystem. The computing resources of the autonomous system 200 may be shared among its subsystems, or subsystems may have their own set of computing resources.

[0067] In some embodiments, the autonomous system 200 can be implemented for or by an autonomous vehicle (e.g., a ground-based autonomous vehicle). The autonomous system 200 can perform various processing techniques on inputs (e.g., sensor data 204, map data 210) to recognize and understand the vehicle's surrounding environment and generate a set of appropriate control outputs to implement a vehicle action plan (e.g., including one or more trajectories) for operating in the vehicle's surrounding environment (e.g., environment 100 in Figure 1). In some embodiments, an autonomous vehicle implementing the autonomous system 200 can drive, roam, operate, etc. with minimal or no interaction from a human operator (e.g., driver, pilot, etc.).

[0068] In some embodiments, the autonomous platform may be configured to operate in multiple operating modes. For example, the autonomous platform may be configured to operate in a fully autonomous operating mode (e.g., autonomous driving) where the autonomous platform can be controlled without user input (e.g., it can be driven and operated without input from a human operator, either present in or remotely from the autonomous vehicle). The autonomous platform may operate in a semi-autonomous operating mode where the autonomous platform can operate with some input from a human operator present in the autonomous platform (or a human operator located away from the autonomous platform). In some embodiments, the autonomous platform may enter a manual operating mode where the autonomous platform can be fully controlled by a human operator (e.g., a human driver) and autonomous driving (e.g., autonomous driving) can be prohibited or deactivated (e.g., temporarily, permanently, etc.). The autonomous platform may be configured to operate in other modes (e.g., modes for use between tasks such as trip / service standby, charging, etc.) such as parking mode or sleep mode. In some embodiments, for example, the autonomous platform may implement vehicle operation assistance techniques (e.g., collision mitigation systems, power-assisted steering, etc.) to assist a human operator of the autonomous platform (e.g., during manual mode).

[0069] The autonomous system 200 may be deployed onboard (e.g., on or inside the autonomous platform) of the autonomous platform and may be configured to operate the autonomous platform in various environments. The environment may be a real environment or a simulated environment. In some embodiments, one or more simulation computing devices may simulate one or more of the following to simulate the operation of the autonomous system 200: sensors 202, sensor data 204, communication interfaces 206, platform data 208, or platform control devices 212.

[0070] In some embodiments, the autonomous system 200 may communicate with one or more networks or other systems via a communication interface 206. The communication interface 206 may include, for example, a transmitter, receiver, port, controller, antenna, or other suitable components that can support the activation of communication, and may include any suitable components for interface connection with one or more networks (e.g., network 170 in Figure 1). In some embodiments, the communication interface 206 may include multiple components (e.g., antenna, transmitter, or receiver) that enable the implementation and use of various communication technologies (e.g., multi-input, multi-output (MIMO) technology).

[0071] In some embodiments, the autonomous system 200 may communicate with one or more computing devices located remotely from an autonomous platform (e.g., remote system 160) via one or more networks (e.g., network 170) using a communication interface 206. For example, in some embodiments, one or more inputs, data, or functions of the autonomous system 200 may be complemented or replaced by a remote system communicating via the communication interface 206. For example, in some embodiments, map data 210 may be downloaded to a remote system via the network using the communication interface 206. In some embodiments, one or more of the position estimation system 230, recognition system 240, planning system 250, or control system 260 may be updated, influenced, nudged, communicated, etc., by a remote system for support, maintenance, situational response override, management, etc.

[0072] Sensor 202 may be positioned on an autonomous platform. In some embodiments, sensor 202 may include one or more types of sensors. For example, one or more sensors may include image capture devices (e.g., visible spectrum cameras, infrared cameras, etc.). Additionally or alternatively, sensor 202 may include one or more depth capture devices. For example, sensor 202 may include one or more light detection and distance measuring (LIDAR) sensors or radio detection and distance measuring (RADAR) sensors. Sensor 202 may be configured to generate point data describing at least a portion of a 360-degree field of view of the surrounding environment. The point data may be point cloud data (e.g., 3D LIDAR point cloud data, RADAR point cloud data). In some embodiments, one or more of the sensors 202 for capturing depth information may be fixed to a rotating device to rotate the sensor 202 around an axis. While rotating around the axis, sensor 202 can capture data in spaced-sector packets describing different portions of a 360-degree field of view of the surrounding environment of the autonomous platform. In some embodiments, one or more of the sensors 202 for capturing depth information may be solid state.

[0073] Sensor 202 may be configured to capture sensor data 204 that represents or otherwise relates to at least a portion of the environment of the autonomous platform. Sensor data 204 may include image data (e.g., 2D camera data, video data, etc.), RADAR data, LIDAR data (e.g., 3D point cloud data, etc.), audio data, or other types of data. In some embodiments, the autonomous system 200 may receive input from additional types of sensors, such as inertial measuring units (IMUs), altimeters, inclinometers, odometers, positioning or positioning devices (e.g., GPS, compasses), wheel encoders, or other types of sensors. In some embodiments, the autonomous system 200 may receive sensor data 204 related to specific components or systems of the autonomous platform. This sensor data 204 may indicate, for example, wheel speed, part temperature, steering angle, cargo or passenger status, etc. In some embodiments, the autonomous system 200 may receive sensor data 204 related to ambient conditions, such as environmental or weather conditions. In some embodiments, sensor data 204 may include multi-modal sensor data. Multimodal sensor data can be acquired by at least two different types of sensors (e.g., sensor 202) and can represent static objects or actors in the environment of the autonomous platform. Multimodal sensor data may include at least two types of sensor data (e.g., camera and LIDAR data). In some embodiments, the autonomous platform can utilize sensor data 204 about sensors located remotely from the autonomous platform (e.g., off-board). This may include, for example, sensor data 204 captured by other autonomous platforms.

[0074] The autonomous system 200 may acquire map data 210 related to the environment in which the autonomous platform was located, its current location, or the environment in which it will be located in the future. The map data 210 may provide information about the environment or geographical area. For example, map data 210 may provide information on the identification and location of various driving routes (e.g., roads), driving route segments (e.g., road segments), buildings or other articles or objects (e.g., streetlights, crosswalks, curbs), the location and direction of boundaries or boundary markings (e.g., location and direction of lanes, parking lanes, turning lanes, bicycle lanes, and other lanes), traffic control data (e.g., location and direction of signs, traffic lights, and other traffic control devices), obstacle information (e.g., temporary or permanent blockages), event data (e.g., road closures / changes in traffic rules due to parades, concerts, sporting events, etc.), nominal vehicle route data (e.g., showing an ideal vehicle route, such as following the center of a particular lane), or any other map data that provides information to help the autonomous platform understand the surrounding environment and its relationships. In some embodiments, map data 210 may include high-resolution map information. Additionally or alternatively, map data 210 may include rare map data (e.g., lane graphs). In some embodiments, the sensor data 204 can be fused with the map data 210 or used to update the map data 210 in real time.

[0075] The autonomous system 200 may include a position estimation system 230 that can provide the autonomous platform with an understanding of its position and orientation in the environment. In some examples, the position estimation system 230 can support one or more other subsystems of the autonomous system 200 by providing, for example, an integrated local reference frame for performing recognition, planning, or control operations.

[0076] In some embodiments, the position estimation system 230 may determine the current position of the autonomous platform. The current position may include an absolute position (Global Position) (e.g., a position relative to a geographic reference anchor, etc.) or a relative position (e.g., a position relative to an object in the environment, etc.). The position estimation system 230 may generally include or interface with any device or circuit for analyzing the position or change in position of the autonomous platform (e.g., a ground-based autonomous vehicle, etc.). For example, the position estimation system 230 may determine the position by using one or more of the following: inertial sensors (e.g., inertial measurement units, etc.), satellite position estimation systems, radio receivers, networking devices (e.g., devices based on IP addresses, etc.), triangulation or proximity to network access points or other network components (e.g., cellular towers, Wi-Fi access points, etc.), or other suitable techniques. The position of the autonomous platform may be used by various subsystems of the autonomous system 200, or may be provided to a remote computing system (e.g., using a communication interface 206).

[0077] In some embodiments, the position estimation system 230 can register the relative positions of elements in the surrounding environment of the autonomous platform along with positions recorded in the map data 210. For example, the position estimation system 230 can process sensor data (e.g., LIDAR data, RADAR data, camera data, etc.) for alignment or other registration on a map of the surrounding environment (e.g., from the map data 210) to determine the position of the autonomous platform within that environment. Thus, in some embodiments, the autonomous platform can identify its position in the surrounding environment (e.g., across six axes) based on a lookup of the map data 210. In some embodiments, given an initial position, the position estimation system 230 can update the position of the autonomous platform by performing incremental realignment based on recorded or estimated deviations from the initial position. In some embodiments, the position can be registered directly within the map data 210.

[0078] In some embodiments, the map data 210 may include a large amount of data subdivided into geographic tiles so that a desired area of ​​the map stored in the map data 210 can be reconstructed from one or more tiles. For example, multiple tiles selected from the map data 210 may be stitched together by the autonomous system 200 based on locations obtained by the location estimation system 230 (e.g., multiple tiles selected in the vicinity of a location).

[0079] In some embodiments, the position estimation system 230 may determine the position (e.g., relative or absolute position) of one or more attachments or accessories relative to the autonomous platform. For example, the autonomous platform may be associated with a cargo platform, and the position estimation system 230 may provide the position of one or more points on the cargo platform. For example, the cargo platform may include trailers or other devices towed, or otherwise attached to or operated by the autonomous platform, and the position estimation system 230 may provide data describing the position (e.g., absolute, relative position) of the autonomous platform as well as the cargo platform. This information can be obtained by other autonomous driving systems to assist the operation of the autonomous platform.

[0080] The autonomous system 200 may include a recognition system 240 that enables the autonomous platform to detect, classify, and track objects and actors in its environment. Environmental features or objects recognized in the environment may be those that are within the field of view of sensor 202 or that are expected to be obstructed by sensor 202. This may include objects that are not moving or are not expected to move (static objects) and objects that are moving or are expected to move (dynamic objects / actors).

[0081] The recognition system 240 may determine one or more states (e.g., current or past states) of one or more objects in the surrounding environment of the autonomous platform. For example, a state may describe the object's current or past position (also called position) (e.g., in a given time, period, etc.), current or past speed / velocity, current or past acceleration, current or past direction of travel, current or past orientation, size / footprint (e.g., represented by boundary shape, object emphasis, etc.), classification (e.g., pedestrian class vs. vehicle class vs. bicycle class), associated uncertainty, or estimates of other state information. In some embodiments, the recognition system 240 may determine states using one or more algorithms or machine learning-based models configured to identify / classify objects based on input from sensors 202. The recognition system 240 may use different forms of sensor data 204 to generate representations of the environment that are processed by one or more algorithms or machine learning-based models. In some embodiments, the state of one or more identified or unidentified objects can be maintained and updated over time as the autonomous platform continuously recognizes and interacts with the objects (e.g., launches, makes concessions, etc., with or around the objects). In this way, the recognition system 240 can provide an understanding of the current state of the environment (e.g., the state including objects in the environment) known from records of the environment's previous state (e.g., the state including movement history of objects in the environment). This information can be useful when the autonomous platform plans its actions through the environment.

[0082] The autonomous system 200 may include a planning system 250 that can be configured to determine how the autonomous platform interacts with and moves within the environment. The planning system 250 may determine one or more action plans for the autonomous platform. An action plan may include one or more trajectories (e.g., action trajectories) that indicate the path the autonomous platform should follow. The trajectories may be of a specific length or time range. The length or time range may be defined by the computational planning range of the planning system 250. The action trajectories may be defined by one or more intermediate points (with associated coordinates). The intermediate points may be future locations of the autonomous platform. The action plans may be continuously generated, updated, and reviewed by the planning system 250.

[0083] The motion planning system 250 may determine a strategy for the autonomous platform. The strategy may be a series of individual decisions made by the autonomous platform (e.g., concessions to actors, counter-concessions to actors, merging, lane changes). The strategy may be selected from several potential strategies. The selected strategy may be the lowest-cost strategy determined by one or more cost functions. The cost functions may, for example, evaluate the probability of collision with other actors or objects.

[0084] The planning system 250 may determine a preferred trajectory for executing a strategy. For example, the planning system 250 may obtain one or more trajectories for executing one or more strategies. The planning system 250 may evaluate and rank the trajectories or strategies (e.g., using scores, costs, compensations, constraints, etc.). For example, the planning system 250 may provide an evaluation of candidate trajectories or strategies for the autonomous platform using predictive outputs that represent interactions (e.g., proximity, intersections, etc.) between the autonomous platform's trajectory and one or more objects. In some embodiments, the planning system 250 may use static costs to evaluate the autonomous platform's trajectory (e.g., "avoid lane boundaries," "minimize jerks," etc.). Additionally or alternatively, the planning system 250 may use dynamic costs to evaluate trajectories or strategies for the autonomous platform based on predictive results for the current operating scenario (e.g., predicted trajectories or strategies leading to interactions between actors, predicted trajectories or strategies leading to interactions between actors and the autonomous platform, etc.). The planning system 250 can rank trajectories based on one or more static costs, one or more dynamic costs, or a combination thereof. The planning system 250 can select an action plan (and corresponding trajectory) based on the ranking of multiple candidate trajectories. In some embodiments, the planning system 250 can select the highest-ranked candidate or the highest-ranked feasible candidate.

[0085] Next, the planning system 250 can verify the selected trajectory for one or more constraints before the trajectory is executed by the autonomous platform.

[0086] To assist in action planning decisions, the planning system 250 may be configured to perform predictive functions. The planning system 250 can predict future states of the environment. This may include predicting future states of other actors in the environment. In some embodiments, the planning system 250 can predict future states based on current or past states (e.g., developed or maintained by the perception system 240). In some embodiments, the future state may be or include the predicted trajectory (e.g., position over time) of an object in the environment, such as another actor. In some embodiments, one or more of the future states may include one or more probabilities associated with it (e.g., marginal probabilities, conditional probabilities). For example, one or more probabilities may include one or more probabilities conditioned on a strategy or trajectory option available to the autonomous platform. Additionally or alternatively, probabilities may include probabilities conditioned on a trajectory option available to one or more other actors.

[0087] In some embodiments, the planning system 250 may perform interactive predictions. The planning system 250 may understand how the predicted future state of the environment will be affected by the execution of one or more candidate action plans and determine an action plan for the autonomous platform. For example, referring again to Figure 1, the autonomous platform 110 may determine candidate action plans corresponding to a set of platform trajectories 112A-C corresponding to the first actor trajectories 122A-C of the first actor 120, the trajectory 132 of the second actor 130, and the trajectory 142 of the third actor 140 (for example, each trajectory correspondence is shown in a matching line style). For example, the autonomous platform 110 (for example, using the autonomous system 200) may predict that platform trajectory 112A, which moves the autonomous platform 110 more quickly in the area ahead of the first actor 120, may be associated with the first actor 120 slowing its forward speed according to the first actor trajectory 122A and making a more rapid concession to the autonomous platform 110. Additionally or alternatively, the autonomous platform 110 can predict that a platform trajectory 112B that smoothly moves the autonomous platform 110 into the area in front of the first actor 120 may be associated with the first actor 120 slightly decelerating in accordance with the first actor trajectory 122B and slowly making concessions to the autonomous platform 110. Additionally or alternatively, the autonomous platform 110 can predict that a platform trajectory 112C that is maintained in a state aligned parallel to the first actor 120 may be associated with the first actor 120 not making any concessions to the autonomous platform 110 in accordance with the first actor trajectory 122C. The planning system 250 can compare the predicted scenarios with a set of desired outcomes (for example, by scoring the scenarios based on cost or compensation) and select a motion plan (and associated trajectories) taking into account the interaction between the autonomous platform and the environment 100. In this way, for example, the autonomous platform 110 can interleave the prediction and motion planning functions.

[0088] To implement the selected action plan, the autonomous system 200 may include a control system 260 (e.g., a vehicle control system). Generally, the control system 260 may provide an interface between the autonomous system 200 and the platform control device 212 to implement the strategy and action plan generated by the planning system 250. For example, the control system 260 can implement the selected action plan / trajectory to control the behavior of the autonomous platform through the environment by following the selected trajectory (e.g., a trajectory that includes intermediate points). For example, the control system 260 can translate the action plan into commands to the appropriate platform control device 212 (e.g., acceleration control, brake control, steering control, etc.). For example, the control system 260 can translate the selected action plan into commands such as adjusting the steering component (e.g., steering angle) by a certain number of angles, applying a braking force of a certain size, or increasing / decreasing speed. In some embodiments, the control system 260 may communicate with the platform control unit 212 via a communication channel that includes, for example, one or more data buses (e.g., a controller area network (CAN)), an onboard diagnostic connector (e.g., OBD-II), or a combination of wired or wireless links. The platform control unit 212 may send or receive data, messages, signals, etc. to or from the autonomous system 200 (and vice versa) via the communication channel.

[0089] The autonomous system 200 may receive support signals from the remote support system 270 via the communication interface 206. The remote support system 270 may communicate with the autonomous system 200 via the network (for example, as a remote system 160 via network 170). In some embodiments, the autonomous system 200 may initiate a communication session with the remote support system 270. For example, the autonomous system 200 may initiate a session based on or in response to a trigger. In some embodiments, the trigger may be a warning, an error signal, a map function, a request, a location, traffic conditions, road conditions, etc.

[0090] After the session has started, the autonomous system 200 may provide context data to the remote assistance system 270. The context data may include sensor data 204 and state data of the autonomous platform. For example, the context data may include a live camera feed from the autonomous platform's camera and the current speed of the autonomous platform. The operator of the remote assistance system 270 (e.g., a human operator) can use the context data to select assistance signals. The assistance signals may provide values ​​or adjustments for various operating parameters or characteristics of the autonomous system 200. For example, assistance signals may include intermediate points (e.g., obstacle routing, lane changes, etc.), speed or acceleration profiles (e.g., speed limits, etc.), relative operation commands (e.g., escort formations, etc.), operation characteristics (e.g., use of auxiliary systems, reduction of energy processing modes, etc.), or other signals to assist the autonomous system 200.

[0091] The autonomous system 200 can use support signals input to one or more autonomous subsystems to perform autonomous functions. For example, the planning subsystem 250 may receive support signals as input for generating an action plan. For example, the support signals may include constraints for generating the action plan. Additionally or alternatively, the support signals may include costs or compensatory adjustments to influence the action plan by the planning subsystem 250. Additionally or alternatively, the support signals may be considered by the autonomous system 200 as suggested inputs to be taken into consideration in addition to other received data (e.g., sensor inputs).

[0092] The autonomous system 200 may be platform-independent, and the control system 260 may provide control commands to the platform control device 212 for various different platforms for autonomous mobility (e.g., multiple different autonomous platforms equipped with autonomous control systems). This may operate in various different environments and, in some embodiments, include various different types of autonomous vehicles from various different manufacturers / developers providing one or more vehicle services (e.g., sedans, vans, SUVs, trucks, electric vehicles, combustion-powered vehicles, etc.).

[0093] For example, referring to Figure 3a, the operating environment may include a dense environment 300. The autonomous platform may include an autonomous vehicle 310 controlled by the autonomous system 200. In some embodiments, the autonomous vehicle 310 may be configured for maneuverability in a dense environment, for example, having a set wheelbase or other specifications. In some embodiments, the autonomous vehicle 310 may be configured to transport cargo or passengers. In some embodiments, the autonomous vehicle 310 (e.g., a passenger van, shuttle, bus) may be configured to transport a large number of passengers. In some embodiments, the autonomous vehicle 310 may be configured to transport cargo such as large quantities of cargo (e.g., trucks, box vans, step vans, etc.) or smaller quantities of cargo (e.g., food, personal packaging, etc.).

[0094] Referring to Figure 3b, a selected overhead view 302 of the dense environment 300 overlapping an exemplary trip / service between the first position 304 and the second position 306 is shown. The exemplary trip / service can be assigned to an autonomous vehicle 320 by, for example, a remote computing system. The autonomous vehicle 320 may be, for example, the same type of vehicle as the autonomous vehicle 310. The exemplary trip / service may include transporting passengers or cargo between the first position 304 and the second position 306. In some embodiments, the exemplary trip / service may include traveling to or through one or more intermediate positions to load or unload passengers or cargo. In some embodiments, the exemplary trip / service may be pre-booked (for example, for regular service in a transport schedule). In some embodiments, the exemplary trip / service may be on-demand (for example, for the performance of or in response to requests for taxi, rideshare, vehicle call, courier, delivery service, etc.).

[0095] Referring to Figure 3c, in other examples, the operating environment may include an open-type mobile path environment 330. The autonomous platform may include an autonomous vehicle 350 controlled by the autonomous system 200. This may include an autonomous tractor for an autonomous truck. In some embodiments, the autonomous vehicle 350 may be configured for heavy load transport, such as long-distance, heavy load transport (e.g., transporting large quantities of cargo or other goods or passengers). For example, the autonomous vehicle 350 may include one or more cargo platform attachments, such as a trailer 352. Although shown as a towed attachment in Figure 3c, in some embodiments, one or more cargo platforms (e.g., box van, step van, etc.) may be integrated into the autonomous vehicle 350 (e.g., mounted on the chassis of the autonomous vehicle).

[0096] Referring to Figure 3d, a selected overhead view of the open travel path environment 330 is shown, including travel path 332, interchange 334, transfer hubs 336 and 338, access travel path 340, and locations 342 and 344. In some embodiments, an autonomous vehicle (e.g., autonomous vehicle 310 or autonomous vehicle 350) may be assigned an exemplary trip / service that traverses one or more travel paths 332 (selectively connected by interchange 334) to transport cargo between transfer hub 336 and transfer hub 338. For example, in some embodiments, the exemplary trip / service may include cargo delivery / transportation services such as cargo delivery / transportation services. The exemplary trip / service may be assigned by a remote computing system. In some embodiments, transfer hub 336 may be the origin of the cargo (e.g., a warehouse, wholesale store, facility, etc.), and transfer hub 338 may be the destination of the cargo (e.g., a retail store, etc.). However, in some embodiments, the transfer hub 336 may be an intermediate point along the final journey between each origin and each destination of the cargo. For example, the origin of the cargo may be located at position 342 along the access travel route 340. Thus, the cargo can be transported to the transfer hub 336 for staging (e.g., by a human-driven vehicle, an autonomous vehicle 310, etc.). At the transfer hub 336, various cargo can be grouped or staged for long-distance transport along the travel route 332.

[0097] In some embodiments of the exemplary trip / service, a group of staged cargo items may be loaded onto an autonomous vehicle (e.g., autonomous vehicle 350) for transport to one or more other transit hubs, such as transit hub 338. For example, although not shown, it should be understood that the open travel route environment 330 may include more transit hubs than transit hubs 336 and 338, and may include more travel routes 332 interconnected to more interchanges 334. A simplified map is shown here for clarity. In some embodiments, one or more cargo items to be transported to transit hub 338 may be distributed to one or more local destinations along a travel route 340 to access location 344 (e.g., by a human-driven vehicle, autonomous vehicle 310, etc.). In some embodiments, the exemplary trip / service may be booked in advance (e.g., for regular operation in a transport schedule, etc.). In some embodiments, the exemplary trip / service may be on-demand (e.g., to provide chartered passenger transport or cargo delivery services or upon request for such services).

[0098] To improve the performance of an autonomous platform, such as an autonomous vehicle (e.g., an autonomous vehicle 310 or 350) that is at least partially controlled using the autonomous system 200, the planning system 250 can implement verification techniques by exemplary embodiments of this disclosure.

[0099] Figure 4 is a block diagram of a verification system 400 according to some embodiments of the present disclosure. While Figure 4 shows an exemplary embodiment of the verification system 400 having various components, it should be understood that components may be rearranged, combined, omitted, etc., within the scope of the present disclosure and in accordance with the present disclosure.

[0100] The verification system 400 may take log data 402 as input. The log data 402 may include data recorded during an actual or simulated driving instance. The recorded data may include data collected by sensors mounted on one or more vehicles (e.g., autonomous vehicles, non-autonomous vehicles, etc.). The recorded data may include data collected from other sources (e.g., roadside cameras, aerial vehicles, etc.). The log data 402 from the simulated scenario may include probabilistic data, such as data extracted from a distribution aligned with multiple observations.

[0101] The verification system 400 can extract a reference scenario 404 from the log data. A reference scenario 404 may be part of log data 402 that describes a scene or event of interest that includes a reference object. The scene or event of interest may include an environment that includes objects, actors, etc. The reference object may be an object or actor that is the subject of study under verification. The reference object may be an actor that demonstrates behavior that can verify other systems. A reference scenario 404 may include driving scenarios. A reference scenario 404 can be selected based on the possibility of exploring events in exceptional situations and the system's response to them. For example, a reference scenario 404 may include a reference vehicle driving past a vehicle parked along the roadside, a reference vehicle merging into traffic, or a reference vehicle maintaining a trailing distance during steady-state cruising on a highway.

[0102] In reference scenario 404, the reference vehicle may perform reference action 410. For example, in a scenario in which the reference vehicle passes a vehicle parked along the roadside, reference action 410 may include the amount of deceleration (or reduction in acceleration) of the reference vehicle as it approaches the parked vehicle. In another example, in a scenario in which the reference vehicle maintains a steady-state cruise on a highway, reference action 410 may include the amount of distance maintained behind a preceding vehicle. In another example, in a scenario in which the reference vehicle merges (or is in a merging lane), reference action 410 may include the amount of acceleration or deceleration applied in response to the merge. Reference action 410 may be a defensive action performed to reduce the likelihood of a major event or to reduce the severity of a major event. For example, reference action 410 may be a defensive driving action.

[0103] The verification system 400 can verify the system under test (SUT) 412 by placing a simulated vehicle into a reference scenario 404 for comparison with a reference target. For example, considering the context of reference scenario 404, SUT operation 414 could represent an operation performed by the target vehicle implementing SUT 412 under the same circumstances that caused the reference target to perform reference operation 410 (e.g., an operation to "teleport" SUT 412 to the exact point in time when the reference target is operating on the log data, or an interpolation operation). In this way, for example, SUT operation 414—and its results—can be compared to reference operation 410 and its results.

[0104] SUT412 is or may include one or more operating systems for an autonomous vehicle. For example, system 412 may include one or more autonomous systems or one or more systems that operate to support autonomous systems. For example, SUT412 may include one or more parts of autonomous system 200, such as a position estimation subsystem 230, a perception subsystem 240, a planning subsystem 250, and a control subsystem 260. In some examples, SUT412 may include sensors 202, a communication interface 206, a remote assistance system 270, a platform control unit 212, etc. SUT412 may include one or more machine learning-based models.

[0105] The validation system 400 can evaluate changes in any part of the autonomous vehicle control system pipeline. Advantageously, the validation system 400 can facilitate evaluation in common reference frame comparisons with a reference. In this way, for example, tasks concerning changes to a complex system can be reduced overall by evaluating whether SUT412 behaves more favorably or unfavorably than the reference. For example, the velocity error of sensor 202 at a range of 200 meters may be greater than the velocity error of sensor 202 at a range of 1 meter, but the validation system 400 can help evaluate whether the ultimate impact on the overall performance of SUT412 actually results in significantly different behavior compared to the reference. Similarly, the validation system 400 can help evaluate whether a complex machine learning-based recognition model with a different machine learning-based model architecture has a significant impact on the overall system behavior compared to the reference.

[0106] Reference operation 410 and SUT operation 414 may include current or future state data describing the vehicle state for the vehicle in question. For example, the vehicle state may include position, velocity, acceleration, jerk, direction, and heading.

[0107] In some embodiments, the verification system 400 can directly compare the reference operation 410 with the SUT operation 414. For example, the verification system 400 can score the reference operation 410 and the SUT operation 414 based on various costs or rules of thumb, and then compare those scores. For example, the SUT operation 414 may include an operation plan for the vehicle being simulated, and the planned operation can be compared with a reference operation performed by the reference vehicle.

[0108] Additionally or alternatively, the verification system 400 may use the simulator 420 to unfold the reference scenario 404 over time, using both the reference action 410 and the SUT action 414 as different starting points or seed actions. For example, the simulator 420 may simulate what events might occur in the environment when the target vehicle performs the reference action 410 (for example, up to a given time range such as 1 second, 5 seconds, etc.). At various points in time, the simulator 420 may output environmental states 430 related to the reference action 410. The environmental states 430 may include state data related to the object and actor states of objects and actors in the environment over time.

[0109] Similarly, the simulator 420 can simulate what events might occur in the environment when the target vehicle performs the SUT operation 414. At various points in time, the simulator 420 can output environmental states 440 related to the SUT operation 414. The environmental states 440 may include state data related to the object and actor states of objects and actors in the environment over time.

[0110] Simulator 420 may include multiple types or hierarchical simulations. For example, Simulator 420 may consist of environmental data (e.g., initial state data describing the environment) and output planned SUT operation. Simulator 420 may simulate the operation of a system under test (e.g., one or more components of an autonomous vehicle computing system) in a simulated environment. For example, Simulator 420 may input simulated sensor data to a simulated autonomous vehicle control system to obtain simulated control decisions.

[0111] The simulator 420 can also be configured to perform lighter simulations. For example, the simulator 420 can compute simulations based on approximations. For example, the simulator 420 can compute simulations to obtain conservative estimates of stopping distance, deceleration speed, lane change speed, etc. For example, the simulator 420 can compute simulations to efficiently estimate the outer boundaries of the response for actor behavior (e.g., upper limits on stopping time or distance). For example, the simulator 420 can compute simulations using a simplified analytical formula that models the behavior of the subject vehicle (e.g., the behavior of a vehicle controlled using a SUT). For example, the behavior of the subject vehicle can be modeled as a one-dimensional, two-dimensional, or three-dimensional projectile trajectory. In this case, for example, the simulator 420 can estimate the SUT state 440 and reference state 430 in a cost-effective manner. The simulator 420 can compute SUT states and reference states for multiple actor behaviors. For example, given an external loop simulation stage (e.g., a simulation of a target vehicle driving in an environment at a given time point in time), the simulator 420 can repeatedly perform a lighter internal loop simulation to calculate the target vehicle's ability to handle a range of various hazards in the environment.

[0112] The Evaluator 450 may evaluate and selectively compare the environmental conditions 430 and 440. In some embodiments, the Evaluator 450 outputs only a relative evaluation of the SUT 412 compared to a reference. In some embodiments, the Evaluator 450 additionally or alternatively generates evaluations of the environmental conditions 430 and 440 across the entire corpus of log data 402.

[0113] Exemplary verification scenarios are shown in Figures 5a to 6c. For example, in Figure 5a, the vehicle being simulated, 500, can be simulated to travel in a road lane in an action 502 for execution. For example, action 502 may be a reference action 410 derived from log data 402. In Figure 5a, action 502 may include a coasting action, considering that actor 510 approaches a road merging area in an expected actor action 512 (e.g., a predictive action based on the current state of actor 510). Action 502 can represent defensive driving or defensive actions that a skilled human driver could take in the scenario.

[0114] In Figure 5b, at the point in time after the snapshot shown in Figure 5a, both the simulated target vehicle 500 and actor 510 have moved forward along the road. In this case, this forward movement may occur as simulated by the simulator 420. A hazard can be simulated at a later point in time. For example, the verification system 400 can simulate actor 510 performing actor action 514, which could be a sudden cut-in.

[0115] In Figure 5c, at the point after the snapshot shown in Figure 5b, both the simulated target vehicle 500 and actor 510 have moved further along the road. For example, such forward movement may be the result of a simulation by simulator 420. At a later point, the target vehicle 500 may perform a predetermined action 506 in response to actor action 514 or actor action 516. For example, the predetermined action 506 may include appropriate braking in response to actor 510's sudden merging.

[0116] The environmental states shown in Figures 5a to 5c may be environmental states output by the simulator 420. For example, the environmental state in Figure 5a may be the initial state of the environment. If operation 502 is a reference operation 410, the resulting state after Figure 5c may be an environmental state 430. The environmental state may include, for example, the distance between the target vehicle 500 and the actor 510, and data describing the braking operation 506, such as speed, acceleration, and jerk, related to the execution of the braking operation 506.

[0117] Figures 6a and 6c illustrate alternative scenarios in which the simulated vehicle 500 performs an initial action 602 instead of action 502. Action 602 may be an SUT action 414 (e.g., an action determined by the Motion Planner of the autonomous vehicle control system) output by the system 412 in response to one or more inputs describing the environment in Figure 6a. Action 602 may differ from action 502. For example, action 602 may be associated with a different velocity or acceleration than action 502.

[0118] In Figure 6b, at the point after the snapshot shown in Figure 6a, both the simulated target vehicle 500 and actor 510 are moving forward along the road. For example, such forward movement could be the result of a simulation by simulator 420. At a later point, the same hazard simulated in Figure 5b can be simulated here. For example, verification system 400 can simulate actor 510 performing actor action 614. Actor action 614 could be an abrupt interruption such as actor action 514.

[0119] In Figure 6c, at the point in time after the snapshot shown in Figure 6b, both the simulated target vehicle 500 and actor 510 have moved further along the road. For example, such forward movement may be the result of a simulation by simulator 420. At a later point in time, the target vehicle 500 may perform a predetermined action 606 in response to actor action 614 or actor action 616. For example, predetermined action 606 may include sudden braking in response to actor 510's abrupt merging. Predetermined action 606 may be the same as predetermined action 506.

[0120] The environmental states shown in Figures 6a to 6c may be environmental states output by the simulator 420. For example, the environmental state in Figure 6a may be the initial state of the environment. If operation 602 is SUT operation 414, the resulting state in Figure 6c may be environmental state 440. The environmental state may include, for example, the distance between the target vehicle 500 and the actor 510, and data describing the braking operation 606, such as speed, acceleration, and jerk, related to the execution of the braking operation 606. The verification system 400 can directly compare the resulting environmental state in Figure 5c with the resulting environmental state in Figure 6c to determine whether operation 602 yields results comparable to operation 502. In this way, the verification system 400 can directly measure a reference object (e.g., a skilled human driver) against the SUT (e.g., an autonomous vehicle control system).

[0121] In some embodiments, the simulations generated by the reference operation 410 and the SUT operation 414 can cover a variety of time scales. For example, in some embodiments, the simulation time scale may extend over a period sufficient to capture the interaction between the vehicle under study and its environment. In some embodiments, the simulation time scale may be relatively short, such as within the planning range of the motion planner for the SUT 412. For example, in some embodiments, the simulator 420 obtains results that are simulated directly based on the motion plan output by the SUT 412 itself.

[0122] In this way, for example, the verification system 400 can provide a paired comparison between the reference object and the SUT 412. The accuracy of the paired comparison can be configured, for example, to be based on the scenario level (e.g., comparing what the reference object does in a given scenario with what the SUT proposes in a given scenario) or the timestamp level (e.g., comparing the potential result of the reference object's operation 410 at a given moment with the potential result of the SUT operation 414 at the same moment).

[0123] Therefore, the verification system 400 can score the environmental conditions in Figures 5c and 6c to evaluate how well SUT412 performed compared to the reference. In general, the condition results can be scored according to any desired metric or rule of thumb. For example, the score may represent the severity of the result. Severity may relate to the potential intersection of the subject vehicle 500 with the boundary of an object or actor (e.g., bounding box, boundary line, etc.). For example, severity may include the occurrence of the intersection, the distance from the intersection (e.g., margin or buffer space), the contact rate or intersection rate, the degree of overlap, etc. In some embodiments, severity may include energy-based evaluation techniques (e.g., kinetic energy-based, etc.). Severity may also be based on inherent characteristics of the subject vehicle's own operation, such as acceleration, jerk, curvature, etc.

[0124] The score or severity value may include temporal characteristics. For example, if a series of actions being simulated proceeds over a time interval, the score can be calculated at each time step of the interval, at time steps sampled within the interval, at the end of the interval, etc. In some embodiments, the value may be averaged over the interval or otherwise combined.

[0125] The score or severity value may include features based on the context of a given scenario. For example, some merging scenarios are often expected to involve a longer period of longer backward distance after a shorter period of close backward distance. The calculation of the score can use the temporal context (e.g., the start of the merging interval) to ensure that excessively close but short merging starts are not underestimated, as they will have a lower severity in many subsequent time periods due to the longer backward distance. For example, context can be used to assign different weights to different timestamps or different aspects of cost or score.

[0126] Similarly, scores or severity levels may include other contextual features. For example, traffic priority status, merging progress, etc., may selectively influence the severity level of a given indicator, such as distance determination.

[0127] In this way, for example, the verification system 400 can comprehensively evaluate the actions proposed by the system under test. For example, the verification system 400 can compare not only the immediate action performed by the reference (e.g., reference action 410) under the same circumstances with the immediate corresponding SUT action 414 performed by SUT 412, but also the downstream results arising from different actions.

[0128] In particular, this comprehensive evaluation has the advantage of easily improving verification by focusing the evaluation on the more important deviations of SUT412. For example, Figures 7a and 8a show two scenarios in which the reference action 702 deviates from the SUT action 802. However, these deviations are relatively insignificant compared to the deviations shown in Figures 7b and 8b. In Figures 7b and 8b, the presence of actor 510 increases the likelihood of negative consequences arising from the deviation between the reference action 702 and the SUT action 802 (as shown, for example, in Figures 5a to 6c). In contrast, in Figures 7a and 8a, without the merging of actor 510, there is no opportunity for the same sudden braking event that occurred in Figure 6c to occur.

[0129] Another example involves a scenario in which actor 510 changes lanes at a distance ahead of the target vehicle 500. If actor 510 is sufficiently far from the target vehicle 500 and accelerating to move away from the target vehicle 500, the deviation between the reference action 702 and the SUT action 802 relative to the target vehicle (e.g., whether or not to apply braking force) may not be associated with a meaningful difference in the desirability of the resulting state of the environment. In contrast, if actor 510 is sufficiently close to the target vehicle 500, or if the distance between actor 510 and the target vehicle 500 is decreasing, the deviation between the reference action 702 and the SUT action 802 relative to the target vehicle (e.g., whether or not to apply braking force) may be associated with a meaningful difference in the desirability of the resulting state of the environment.

[0130] In this way, for example, by scoring the simulated results for deviations from the reference, the verification system 400 can effectively reduce the impact of deviations that do not cause negative results and increase the impact of deviations that are likely to lead to negative results. In this way, the verification system 400 can improve performance more efficiently by assisting in the prioritization of verification tasks and SUT improvements.

[0131] In some embodiments, the simulator 420 can simulate additional actor actions to evaluate how well the simulated target vehicle responds when starting from other initial or seed states. For example, the verification system 400 can simulate the target vehicle's response to other actor actions when starting from reference action 410 compared to starting from SUT action 414.

[0132] Figures 9a and 9b show simulated action-response pairs. For example, simulator 420 can introduce actor action 922-1 into the scenario and simulate the SUT response 924-1. Similarly, simulator 420 can introduce actor action 922-2 into the scenario and simulate the SUT response 924-2. Similarly, simulator 420 can introduce actor action 922-n into the scenario and simulate the SUT response 924-n.

[0133] Simulator 420 can start from the seed state of reference operation 410 and the seed state of SUT operation 414, respectively, and simulate responses 924-1, 924-2, and 924-n, respectively. Simulator 420 can then obtain the resulting environmental state for each of responses 924-1, 924-2, and 924-n, and evaluator 450 can score that state (for example, as described above with respect to Figures 5a to 6c). In this way, for example, verification system 400 can evaluate how well SUT 412 is prepared or defended to avoid a worse outcome than the reference in response to the operations 922-1, 922-2, 922-n, etc. that are invoked.

[0134] For example, referring to Figures 5b and 6b, actor actions 514 and 614 can be injected actions. Depending on the different seeding conditions (e.g., reference action 410 following Figure 5b, SUT action 414 following Figure 6b), the target vehicle 500 will encounter the injected actions in different ways. For example, because action 502 includes a coasting event to defend against an upcoming merge, the target vehicle 500 will encounter injected actor action 514 at a much greater distance behind actor 510. In contrast, because action 602 does not include a coasting event to defend against a merge, the target vehicle 500 will encounter injected event 614 at a small distance behind actor 510. Thus, the results are different. That is, action 506 (e.g., SUT 412's response to injected actor actions 514 / 516) is less serious than action 606 (e.g., SUT 412's response to injected actions 614 / 616). Thus, for example, the results may indicate that the reference results are less severe than the SUT results.

[0135] Exemplary pseudocode is provided for algorithm 1. [Table 1]

[0136] Simulator 420 can acquire input actions from various sources. Input actions can be generated or synthesized. For example, input actions can be synthesized or extracted from one or more probability distributions of feasible actions. The list of actions may include exceptional situation scenarios that are useful for validation. The list of actions may include low-probability actions that do not actually occur often (for example, a reduction in the sample size of the training data may present a scenario that is difficult for machine learning-based models to learn). The list of actions may include a set of predetermined benchmark actions that are useful for investigating the actions of SUT412.

[0137] Log data 402 can provide a basis for obtaining a list of actions. For example, in the case of an autonomous vehicle with a human operator (onboard or offboard), log data describing the vehicle operator's disengagements (e.g., disengaging all or part of the autonomous system) can illustrate scenarios in which the autonomous system did not operate as expected or in a desirable manner, and thus the scenarios may include actions of interest that are put into simulation for evaluation.

[0138] Furthermore, log data 402 can indicate situations in which the autonomous vehicle performed a suboptimal action. These situations can provide templates for generating actions to be introduced into the verification scenario. For example, if log data 402 indicates a suboptimal braking pattern (e.g., sudden braking) in a merging scenario where an actor interrupts early, the early actor's interrupt action forms part of the list of actions to be introduced into the verification scenario. In this way, for example, the verification system 400 can directly evaluate and score the overall effect of the changes to SUT412 on the score or severity of the target vehicle's response to the early interrupt.

[0139] The results of individual validation scenarios can be combined with other results to provide validation findings for SUT412. For example, in some situations, individual references may be irregular references (e.g., human drivers may exhibit various variations). Therefore, results from a wide range of validation scenarios can provide a more accurate assessment of SUT412.

[0140] In some embodiments, the average of performance metrics from all validation scenarios can provide a high level of evaluation of how SUT412 aligns with the references of the entire set. The deviation between the average reference result and the average SUT result can provide a validation metric.

[0141] In some embodiments, the evaluator 450 can use score / severity intervals (Bins) to calculate a histogram and classify and count individual verification results. In this way, for example, the deviation between the reference and the SUT can be examined in more detail based not only on the quantity of the deviation but also on the severity level at which more deviations occur.

[0142] In some embodiments, the evaluator 450 can divide or interval (Bin) the validation scenarios or results based on individual contexts and characteristics. The scores can then be averaged or otherwise combined for each interval or division to reduce noise in the evaluation signal and provide more granular insights into the situations in which the SUT differs from the reference set of the whole.

[0143] Figure 10 shows an exemplary decision tree 1000 for classifying validation results. The root 1002 may be a node in a larger tree. Based on the Boolean response to the root 1002, additional nodes 1004 and 1006 can further classify the validation scenarios. Intervals 1012, 1014, 1016, and 1018 may correspond to leaf nodes.

[0144] In some embodiments, the verification scenarios are segmented based on interpretable features corresponding to the metrics of interest (e.g., as shown in Figure 10). In this way, the classification can help identify areas of future performance improvement, areas of current improvement or regression, etc. For example, whether the vehicle under test has right-of-way may influence the classification of the behavior observed in the set of reference vehicles. For example, if a reference vehicle has right-of-way, it may exhibit less defensive behavior. Thus, segmenting segments 1012 and 1014 according to the right-of-way status can provide an interpretable understanding for calculating more granular performance metrics. For example, when evaluating the question "Does the AV defend as well as the reference vehicle when the AV does not have right-of-way?", it may involve calculating a performance metric comparing the reference behavior and the AV behavior in segment 1014. These granular performance metrics can be used for further segmentation and verification of the AV system under test.

[0145] However, in general, classifying validation scenarios based on their characteristics can facilitate the prediction of performance defects (e.g., excessive severity) based on those characteristics. These characteristics may be potential features learned by machine learning-based models. For example, a machine learning-based mixed model or other clustering model may be configured to describe the distribution of validation scenarios, identify groups of validation scenarios with similar characteristics, and perform clustering to compare a reference with SUT412.

[0146] The classification can be hierarchical. The desired subdivision of the hierarchical structure can be determined by analyzing the resulting distribution of leaf nodes or intervals. For example, the variance of references within an interval may indicate that multiple intervals are needed to better capture the variance and context dependency of SUT412 and reference behavior.

[0147] Figure 11 is a flowchart of Method 1100 for performing system verification according to the embodiments of this disclosure. One or more parts of Method 1100 can be implemented by a computing system including one or more computing devices, such as the computing system described with reference to other drawings (e.g., the autonomous platform 110, the vehicle computing system 180, the remote system 160, the system in Figure 14, etc.). Each individual part of Method 1100 can be performed by any (or any combination) of the one or more computing devices. Furthermore, one or more parts of Method 1100 can be implemented against the hardware components of the devices described herein (e.g., as in Figures 1, 2, 14, etc.). Figure 11 shows the elements to be performed in a particular order for illustrative and discussion purposes. Those skilled in the art will understand that elements of any of the methods discussed herein can be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure by using the disclosures provided herein. Figure 11 is described with reference to and not intended to limit elements / terms described in relation to other systems and drawings for illustrative purposes. One or more parts of method 1100 may be carried out additionally or by other systems.

[0148] In 1102, exemplary method 1100 includes the step of obtaining log data describing reference actions of a reference vehicle in an environment—reference actions that occur in the initial state of the environment. A reference action (e.g., reference action 410) can correspond to a defensive action (e.g., a defensive action against the occurrence of an event such as a low-probability event). A reference action 410 can represent the vehicle state of the reference vehicle for a particular point in time.

[0149] In 1104, exemplary method 1100 includes the step of using an operating system to determine planned actions for a vehicle to be simulated in an initial state of the environment. For example, based on a snapshot of the environment at the time a reference action 410 is performed, the operating system may be an SUT 412 that can (or can be used to) generate planned actions for the target vehicle 406 to be simulated. For example, the operating system may include at least one model of an autonomous vehicle control system configured to receive sensor data and control the movement of the autonomous vehicle. For example, the SUT 412 may include one or more parts of a recognition system that include recognition sensors (e.g., cameras, lidar, etc.) or recognition models (e.g., machine learning-based recognition models for object recognition, tracking, etc.). For example, the SUT 412 may include a motion planning model.

[0150] In 1106, an exemplary method 1100 includes the steps of (i) simulating the SUT state of the environment obtained by a simulated vehicle performing a planned action in an initial state of the environment, followed by an actor performing an actor action after the simulated vehicle performs the planned action, and (ii) simulating the reference state of the environment obtained by a simulated vehicle performing a reference action in an initial state of the environment, followed by an actor performing an actor action after the simulated vehicle performs the reference action. For example, Figures 6a-6c show an exemplary simulation of the SUT state of the environment (e.g., Figure 6c) obtained by a simulated vehicle performing a planned SUT action 602 (e.g., Figure 6a), followed by an actor performing an actor action 614 (e.g., Figure 6b) after the simulated vehicle performs the planned SUT action 602. For example, Figures 5a to 5c illustrate an exemplary simulation of the reference state of the environment (e.g., Figure 5c) obtained when the simulated vehicle performs a reference action 502 (e.g., Figure 5a) followed by the actor performing an actor action 614 (e.g., Figure 5b).

[0151] In 1108, exemplary method 1100 includes the step of determining a test score based on the SUT state of the environment and a reference score based on the reference state of the environment. For example, the score or severity may be determined based on the environment state 430 and the environment state 440. For example, exemplary method 1100 may include the step of evaluating the state of a vehicle simulated in the SUT state of the environment and the state of a vehicle simulated in the reference state of the environment using a cost function. The cost function may be determined based on the distance to the actor boundary. The cost function may be determined based on a severity measurement of the intersection with the actor boundary.

[0152] In 1110, exemplary method 1100 includes a step of evaluating the operating system based on test scores and reference scores. For example, the step of evaluating the operating system may include a step of measuring the difference in scores. In some embodiments, the results of one or more verification scenarios can be combined, and the scores can be evaluated for a group or interval of verification scenarios. For example, exemplary method 1100 may include a step of classifying the test scores and reference scores based on one or more characteristics of the initial state of the environment.

[0153] An exemplary method 1100 may include the step of determining one or more categories for improvement related to a sub-best test score. For example, the verification system 400 may provide multiple test scores related to verification scenarios of several different categories. The differences between the scores of different categories can identify the categories that need improvement. An exemplary method 1100 may include the step of adjusting one or more parameters of the operating system (e.g., one or more hardware configurations, one or more machine learning-based model parameters, etc.) corresponding to the behavior of the autonomous vehicle in one or more categories for improvement.

[0154] In some embodiments of the exemplary method, the actor action includes a hazard affecting the simulated vehicle. In some embodiments of the exemplary method, the actor action is selected from a plurality of actions having a probability less than a threshold probability. In some embodiments of the exemplary method, the exemplary method includes the step of determining a response to a hazard affecting the simulated vehicle.

[0155] In some embodiments of the exemplary method, the reference action may be a defensive driving action to prevent the occurrence of a hazard. For example, the hazard may be an early cut-in by an actor in a merging area. The defensive driving action may be a coasting action or a braking action.

[0156] Figure 12 shows an exemplary embodiment of 1100, which includes a multi-layer simulation environment. (For example, in 1104 to perform exemplary method 1100) A first simulation can process an environment (e.g., initial state data describing the environment) and output planned behavior. For example, in 1204, the exemplary embodiment may include the step of the first simulation using an operating system to determine planned behavior for a simulated vehicle in the initial state of the environment. The first simulation can simulate the behavior of a system under test (e.g., one or more components of an autonomous vehicle computing system) in the simulated environment. For example, the first simulation can input simulated sensor data to a simulated autonomous vehicle control system to obtain simulation-based control decisions.

[0157] Within the selective loop 1206, for individual actor actions of one or more actor actions by actors in the environment, an exemplary embodiment may include, in 1208, the step of obtaining from a second simulation: (i) the SUT state of the environment obtained by the individual actor actions that occur following the simulated vehicle performing a planned action in the initial state of the environment; and (ii) the reference state of the environment obtained by the individual actor actions that occur following the simulated vehicle performing a reference action in the initial state of the environment. The second simulation may be the same as or different from the first simulation. The second simulation may be implemented by the same simulator as the first simulation. The second simulation may be implemented by a different simulator than the first simulation. The second simulation may be calculated based on approximations. For example, the second simulation may include calculating conservative estimates for stopping distance, deceleration speed, lane change speed, etc. For example, the second simulation may be configured to efficiently estimate the outer boundary of the response to actor actions (e.g., upper limits on stopping time or distance). For example, the second simulation may involve computing a simplified analytical formula that models the movement of the target vehicle as the trajectory of a projectile. In this way, for example, the second simulation can estimate the SUT state and reference state in a cost-effective manner. The second simulation can compute the SUT state and reference state for several actor actions. For example, given a planned action and a reference action, the readiness of the target vehicle to respond to various actor actions can be evaluated by estimating the SUT state and reference state resulting from the second simulation.

[0158] Figure 13 shows a flowchart of a method 1300 for training one or more machine learning-based operational models according to aspects of this disclosure. For example, an operating system (e.g., SUT412) may include a machine learning-based operational model.

[0159] One or more parts of Method 1300 can be implemented by a computing system including one or more computing devices, such as the computing system described with reference to other drawings (e.g., the autonomous platform 110, the vehicle computing system 180, the remote system 160, the system in Figure 14, etc.). Each individual part of Method 1300 can be carried out by any (or any combination) of the one or more computing devices. Also, one or more parts of Method 1300 can be implemented on the hardware components of the devices described herein (e.g., as in Figures 1, 2, 14, etc.) to verify one or more systems or models. Figure 13 shows the elements to be performed in a particular order for illustrative and discussion purposes. Those skilled in the art will understand that elements of any of the methods discussed herein can be adjusted, rearranged, extended, omitted, combined, or modified in various ways without departing from the scope of this disclosure using the disclosures provided herein. Figure 13 is described with reference to elements / terms described in relation to other systems and drawings for illustrative purposes and is not intended to limit it. One or more parts of Method 1300 can be carried out additionally or by other systems.

[0160] In 1302, method 1300 may include the step of obtaining training data for training a machine learning-based operational model. The training data may include multiple training instances, such as reference plan data, such as labeled trajectories or strategies based on expert demonstrations.

[0161] Training data can be collected using one or more autonomous platforms (e.g., autonomous platform 110) or their sensors while the autonomous platform is in its environment. For example, training data can be collected using one or more autonomous vehicles (e.g., autonomous platform 110, autonomous vehicle 310, autonomous vehicle 350, etc.) or their sensors while the vehicles are operating along one or more travel paths. In some examples, training data can be collected using other sensors such as mobile device-based sensors, ground-based sensors, air-based sensors, satellite-based sensors, or any sensor interface configured to substantially acquire and / or record measurement data.

[0162] The training data may include multiple training sequences divided across multiple datasets (e.g., a training dataset, a validation dataset, or a test dataset). Each training sequence may include multiple pre-recorded recognition data points, point clouds, images, etc. In some embodiments, each sequence may include a LiDAR point cloud (e.g., collected using a LiDAR sensor on an autonomous platform) or images (e.g., collected using a mono or stereo imaging sensor). For example, in some embodiments, multiple images may be scaled for training and evaluation.

[0163] In 1304, method 1300 may include the step of selecting a training instance based at least partially on training data.

[0164] In 1306, method 1300 may include the step of inputting a training instance into a machine learning-based operational model.

[0165] In 1308, Method 1300 may include the step of generating one or more loss metrics and / or one or more Objective(s) for a machine learning-based operational model based on labels associated with at least some of the outputs and training instances of the machine learning-based operational model.

[0166] In 1310, method 1300 may include the step of modifying at least one parameter of at least a part of a machine learning-based operational model based on at least one of the loss metrics and / or at least one of the objectives. For example, a computing system can modify at least a part of a machine learning-based operational model based on at least one of the loss metrics and / or at least one of the objectives.

[0167] In some embodiments, machine learning-based operational models can be trained in an end-to-end manner. For example, in some embodiments, machine learning-based operational models can be fully differentiated.

[0168] After updating, the operational model or the operating system containing the operational model may be provided for verification (for example, by exemplary embodiments of Method 1100). In some embodiments, the verification system 400 can evaluate or verify the operating system to identify areas of underperformance compared to the entire set of reference objects. The verification system 400 may trigger retraining, discarding, etc., of the operating system based, for example, on the failure to meet verification thresholds in one or more areas (for example, preventing one or more hazards).

[0169] In some embodiments, the verification system 400 is implemented to provide a cost function for training the operating system to behave similarly to a reference (for example, for training one or more components of a machine learning-based behavior planning system). In some embodiments, the verification system 400 is implemented to impose a penalty on non-conforming behavior that does not conform to the reference behavior.

[0170] Figure 14 is a block diagram of an exemplary computing ecosystem 10 according to an exemplary embodiment of the present disclosure. The exemplary computing ecosystem 10 may include a first computing system 20 and a second computing system 40 that are communicably coupled over one or more networks 60. In some embodiments, the first computing system 20 or the second computing system 40 may implement one or more of the systems, operations, or functions described herein (e.g., a remote system 160, an onboard computing system 180, an autonomous system 200, etc.) to verify one or more systems or operating systems.

[0171] In some embodiments, the first computing system 20 may be included in an autonomous platform and may be used to perform functions of the autonomous platform as described herein. For example, the first computing system 20 may be onboard an autonomous vehicle and can implement an autonomous system for operating the autonomous vehicle autonomously. In some embodiments, the first computing system 20 may represent the entire onboard computing system or a part thereof (e.g., a position estimation system 230, a perception system 240, a planning system 250, a control system 260, or a combination thereof). In other embodiments, the first computing system 20 may not be onboard an autonomous platform. The first computing system 20 may include one or more separate physical computing devices 21.

[0172] The first computing system 20 (for example, its computing device 21) may include one or more processors 22 and memory 23. The one or more processors 22 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple operationally connected processors. The memory 23 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, and combinations thereof.

[0173] Memory 23 can store information accessible by one or more processors 22. For example, memory (e.g., one or more non-temporary computer-readable storage media, memory devices, etc.) can store data 24 that can be retrieved (e.g., received, accessed, written, manipulated, generated, generated, stored, pooled, downloaded, etc.). Data 24 may include, for example, sensor data, map data, data related to autonomous functions (e.g., data related to recognition, planning, or control functions), simulation data, or any data or information described herein. In some embodiments, the first computing system 20 may retrieve data from one or more memory devices located remotely from the first computing system 20.

[0174] Memory 23 can store computer-readable instructions 25 that can be executed by one or more processors 22. Instructions 25 can be implemented in software or hardware written in any suitable programming language. Additionally or alternatively, instructions 25 can be executed in logically or virtually isolated threads on the processors 22.

[0175] For example, memory 23 can store instructions 25 that can be executed by one or more processors (e.g., one or more processors 22, one or more other processors, etc.) to perform any of the operations, functions, or methods / processes (or parts thereof) described herein (e.g., in a computing device 21, a first computing system 20, or another system having processors that execute instructions). For example, an operation may include implementing system verification (e.g., as described herein).

[0176] In some embodiments, the first computing system 20 may store or include one or more models 26. In some embodiments, the model 26 may be one or more machine learning-based models (e.g., machine learning-based operating systems), or otherwise include them. For example, the model 26 may be a variety of machine learning-based models, such as regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear or nonlinear models. Examples of neural networks include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the first computing system 20 may include one or more models for implementing subsystems of an autonomous system 200, which may include a location estimation system 230, a recognition system 240, a planning system 250, or a control system 260.

[0177] In some embodiments, the first computing system 20 may acquire one or more models 26 that communicate with the second computing system 40 via a network 60 using a communication interface 27. For example, the first computing system 20 may store the models 26 (e.g., one or more machine learning-based models) in memory 23. The first computing system 20 can then use the models 26 or otherwise implement them (e.g., by a processor 22). For example, the first computing system 20 may implement the models 26 to estimate the position of an autonomous platform in an environment, recognize the environment of the autonomous platform or objects within it, plan one or more future states of the autonomous platform for movement through the environment, and control the autonomous platform to interact with the environment.

[0178] The second computing system 40 may include one or more computing devices 41. The second computing system 40 may include one or more processors 42 and memory 43. The one or more processors 42 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or multiple operationally connected processors. The memory 43 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, and combinations thereof.

[0179] Memory 43 can store information accessible by one or more processors 42. For example, memory 43 (e.g., one or more non-temporary computer-readable storage media, memory devices, etc.) can store retrieval data 44. The data 44 may include, for example, sensor data, model parameters, map data, simulation data, simulated environmental scenes, simulated sensor data, data related to vehicle trips / services, or any data or information described herein. In some embodiments, a second computing system 40 may retrieve data from one or more remote memory devices from the second computing system 40.

[0180] Furthermore, memory 43 can store computer-readable instructions 45 that can be executed by one or more processors 42. Instructions 45 may be software written in any suitable programming language, or they may be implemented in hardware. Additionally or alternatively, instructions 45 may be executed in logically or virtually isolated threads on the processors 42.

[0181] For example, memory 43 can store instructions 45 that can be executed (e.g., by one or more processors 42, one or more processors 22, one or more other processors, etc.) to perform any of the operations, functions, or methods / processes described herein (e.g., by one or more processors 42, one or more processors 22, one or more other processors, etc.) on a computing device 41 such as computing device 21 or first computing system 20, second computing system 40, or other system having a processor for executing instructions. This may include, for example, functions of the autonomous system 200 (e.g., position estimation, perception, planning, control, etc.) or other functions related to the autonomous platform (e.g., remote assistance, mapping, vehicle management, trip / service assignment and matching, etc.). This may also include, for example, verifying a machine learning-based operating system.

[0182] In some embodiments, the second computing system 40 may include one or more server computing devices. If the second computing system 40 includes multiple server computing devices, these server computing devices can operate using various computing architectures, such as sequential computing architectures, parallel computing architectures, or combinations thereof.

[0183] In addition to or alternative to Model 26 in the first computing system 20, the second computing system 40 may include one or more Models 46. For example, Model 46 may be one or more machine learning-based models (e.g., machine learning-based operating systems), or otherwise may include them. For example, Model 26 may be a variety of machine learning-based models, such as regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbor models, Bayesian networks, or other types of models including linear or nonlinear models, or otherwise may include them. Examples of neural networks include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the second computing system 40 may include one or more models of the autonomous system 200.

[0184] In some embodiments, the second computing system 40 or the first computing system 20 can train one or more machine learning-based models of model 26 or model 46 using one or more model trainers 47 and training data 48. The model trainers 47 can train either model 26 or model 46 using one or more training or learning algorithms. An example of a training technique is error backpropagation. In some embodiments, the model trainers 47 may perform supervised training techniques using labeled training data. In other embodiments, the model trainers 47 may perform unsupervised training techniques using unlabeled training data. In some embodiments, the training data 48 may include simulated training data (e.g., training data obtained from simulated scenarios, inputs, configurations, environments, etc.). In some embodiments, the second computing system 40 can implement a simulation to obtain the training data 48 or to implement the model trainers 47 for training or testing model 26 or model 46. For example, the model trainer 47 can be trained on one or more components of a machine learning-based model for the autonomous system 200 via an unsupervised training technique that uses an objective function (e.g., cost, compensation, heuristics, constraints, etc.). In some embodiments, the model trainer 47 may perform a number of generalization techniques to improve the generalization ability of the model during training. Generalization techniques include weight decay, dropouts, or other techniques.

[0185] For example, in some embodiments, the second computing system 40 may generate training data 48 in an exemplary manner of the present disclosure. For example, the second computing system 40 may generate training data 48. For example, the second computing system 40 may implement a method according to an exemplary manner of the present disclosure. The second computing system 40 may use the training data 48 to train the model 26. For example, in some embodiments, the first computing system 20 may include a computing system that is installed in or otherwise associated with a real or simulated autonomous vehicle. In some embodiments, the model 26 may include a perception or machine vision model that is installed in or distributed in a manner that is serviced for a real or simulated autonomous vehicle. Thus, for example, the second computing system 40 may provide a training pipeline for training the model 26.

[0186] The first computing system 20 and the second computing system 40 may each include communication interfaces 27 and 49, respectively. The communication interfaces 27 and 49 can be used to communicate with one or more other systems or devices, including systems or devices located remotely from the first computing system 20 or the second computing system 40. The communication interfaces 27 and 49 may include any circuits, components, software, etc., for communicating with one or more networks (e.g., network 60). In some embodiments, the communication interfaces 27 and 49 may include, for example, one or more communication controllers, receivers, transceivers, transmitters, ports, conductors, software for data communication, or hardware.

[0187] Network 60 may be any type of network or combination of networks that enables communication between devices. In some embodiments, the network may include one or more of the following: a short-range network, a wide-area network, the Internet, a security network, a cellular network, a mesh network, a peer-to-peer communication link, or some combination thereof, and may include any number of wired or wireless links. Communication over Network 60 can be achieved, for example, through a network interface using any type of protocol, protection scheme, encoding, format, packaging, etc.

[0188] Figure 14 shows an example computing ecosystem 10 that can be used to implement this disclosure. Other systems can also be used. For example, in some embodiments, the first computing system 20 may include a model trainer 47 and training data 48. In such embodiments, models 26, 46 can all be trained and used locally on the first computing system 20. In other examples, in some embodiments, the computing system 20 may not be connected to any other computing system. Also, components that are exemplified or described as being included in one of the computing systems 20 or 40 may instead be included in the other one of the computing systems 20 or 40.

[0189] Computing tasks discussed here as being performed on a remote computing device from an autonomous platform (e.g., an autonomous vehicle) may instead be performed on the autonomous platform (e.g., via the vehicle computing system of the autonomous vehicle), or vice versa. Such configurations can be implemented without departing from the scope of this disclosure. Computer-based systems allow for a wide variety of possible configurations, combinations, and divisions between or between components. Computer-implemented operations may be performed on a single component or across multiple components. Computer-implemented tasks or operations may be performed sequentially or in parallel. Data and instructions may be stored in a single memory device or multiple memory devices.

[0190] The aspects of this disclosure have been described in terms of their exemplary embodiments. Many other embodiments, variations, or modifications within the scope and spirit of the appended claims can be conceived by those skilled in the art through examination of this disclosure. Any features described in the following claims can be combined or rearranged in all possible ways. Thus, the scope of this disclosure is exemplary and not restrictive, and this disclosure does not preclude the inclusion of variations, modifications, or additions to the subject matter that would be readily apparent to those skilled in the art. Furthermore, terms are described herein using lists of exemplary elements linked by conjunctions such as “and,” “or,” and “but.” It should be understood that these conjunctions are provided for illustrative purposes only. For example, a list linked by a particular conjunction such as “or” may mean “at least one” or “any combination” of the exemplary elements enumerated therein, and unless otherwise specified, “or” is understood as “and / or.” Similarly, terms such as “based on” should be understood as “based at least partially.”

[0191] Those skilled in the art will understand that any element of the claims, actions, or processes discussed herein can be modified, rearranged, expanded, omitted, combined, or altered in various ways using the disclosures provided herein without departing from the scope of this disclosure. Some claims are described as letter references to claim elements for illustrative purposes, and are not limiting. Letter references do not imply a specific order of actions. For example, letter identifiers such as (a), (b), (c), ..., (i), (ii), (iii), ... can be used to describe actions. These identifiers are provided for the convenience of the reader and do not indicate a specific order of steps or actions. Actions indicated by list identifiers such as (a), (i) may occur before, after, or in parallel with other actions indicated by list identifiers such as (b), (ii).

Claims

1. A method for verifying a test operating system (SUT) for operating an autonomous vehicle, wherein the method is: (a) A step of obtaining log data that describes the reference action of a reference vehicle in the environment (Exemplary Action) - a reference action is an action that occurs in the initial state of the environment, (b) The steps of determining a planned operation for a vehicle to be simulated in the initial state of the environment using the operating system, (c) (i) The simulated vehicle performs the planned operation in the initial state of the environment, and the Actor performs the actor operation following the simulated vehicle performing the planned operation, thereby simulating the SUT (System Under Test) state of the environment and (ii) the simulated vehicle performs the reference operation in the initial state of the environment, and the Actor performs the actor operation following the simulated vehicle performing the reference operation, (d) A step of determining a test score based on the SUT state of the environment and a reference score based on the reference state of the environment, (e) A method comprising the step of evaluating the operating system based on the test score and the reference score.

2. The method according to claim 1, wherein the actor action includes a risk of affecting the simulated vehicle.

3. The method according to claim 1 or 2, wherein the actor action is selected from a plurality of actions having a probability lower than a threshold probability.

4. (c) The method according to any one of claims 1 to 3, further comprising the step of determining the response to the hazard affecting the simulated vehicle.

5. The method according to any one of claims 1 to 4, wherein the reference operation includes a defensive operation to prevent the occurrence of the hazard.

6. The method according to any one of claims 1 to 5, wherein the reference operation includes the vehicle state of a vehicle driven by a human.

7. The method according to any one of claims 1 to 6, wherein the operating system comprises at least one model of an autonomous vehicle control system configured to receive sensor data and control the movement of the autonomous vehicle.

8. The method according to any one of claims 1 to 7, wherein the operating system includes an operation planning model.

9. The method according to any one of claims 1 to 8, wherein the test score is based on the state of the simulated vehicle in the SUT state of the environment, and the reference score is based on the state of the simulated vehicle in the reference state of the environment.

10. (d) The method of claim 9, comprising the step of evaluating the state of the simulated vehicle in the SUT state of the environment and the state of the simulated vehicle in the reference state of the environment using a cost function.

11. The method according to claim 10, wherein the cost function is determined based on the distance to the actor boundary or a measure of the severity of the intersection with the actor boundary.

12. (e) The method according to any one of claims 1 to 11, further comprising the step of classifying the test score and the reference score based on one or more characteristics of the initial state of the environment.

13. (f) The step of determining one or more categories for improvement related to the Suboptimal test score, (g) The method of claim 12, comprising the step of adjusting one or more parameters of the operating system corresponding to the operation of the autonomous vehicle in one or more categories for improvement.

14. One or more non-temporary computer-readable media for storing instructions executable to cause one or more processors to perform an operation including the method according to any one of claims 1 to 13.

15. An autonomous vehicle control system for controlling an autonomous vehicle, wherein the autonomous vehicle control system is One or more processors, Includes one or more non-temporary computer-readable media that store instructions executable by one or more processors to cause the autonomous vehicle control system to control the operation of the autonomous vehicle using an operating system, The operating system is an autonomous vehicle control system verified using the method described in any one of claims 1 to 14.

Images