Methods and systems for remote support of autonomous agents

Abstract

To create improved and useful systems and methods for receiving remote assistance from remote operators. [Solution] The system for remote assistance of an autonomous agent may include, and / or interface with, any or all of, a sensor suite, a computing system, a communication interface, and / or any other suitable components. The system may further optionally include a set of infrastructure devices, a remote operator platform, and / or any other suitable components. The system functions to enable the exchange of information between the autonomous agent and the remote assist. Additionally or alternatively, the system may function to operate the autonomous agent (e.g., based on remote input indirectly received from a remote operator) and / or perform any other suitable functions.

Description

Technical Field

[0001] Cross - Reference to Related Applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 195,774, filed on Jun. 2, 2021, the entire disclosure of which is incorporated herein by reference.

Background Art

[0002]

[0002] The present invention generally relates to the field of autonomous vehicles, and more specifically, to novel and useful systems and methods for remote assistance of autonomous agents in the field of autonomous vehicles.

[0003]

[0003] In current systems and methods associated with autonomous vehicles, the usefulness of a remote operator to take over control of the vehicle in dangerous scenarios and / or otherwise difficult scenarios is recognized. However, in such cases, full communication between the vehicle and the remote operator is required - a requirement that is very complex, if not impossible, to establish reliably. In the case of systems and methods for autonomous vehicles that do not employ a remote operator, conventionally, either a human operator has to be on board, or the autonomous vehicle has to be trained to understand and react reliably even in the most rare scenarios.

[0004]

[0004] Therefore, in the field of autonomous vehicles, there is a need to create improved and useful systems and methods for receiving remote assistance from a remote operator.

Brief Description of the Drawings

[0005] [Figure 1]

[0005] It is a schematic diagram of a system for remote assistance of an autonomous agent. [Figure 2]

[0006] It is a schematic diagram of a method for remote assistance of an autonomous agent. [Figure 3A]

[0007] This diagram shows a schematic example of a modified method for remotely assisting autonomous agents. [Figure 3B]

[0007] A schematic diagram of a modified example of a method for remotely assisting an autonomous agent is shown. [Figure 3C]

[0007] A schematic diagram of a modified example of a method for remotely assisting an autonomous agent is shown. [Figure 4]

[0008] This diagram shows a schematic representation of a modified system for remote support of autonomous agents. [Figure 5]

[0009] This diagram shows a schematic example of a modified method for remotely assisting autonomous agents. [Figure 6]

[0010] This diagram shows a schematic example of a modified method for remotely assisting autonomous agents. [Figure 7A]

[0011] This shows a schematic example of control based on a remote operator-provided route in one modified version of this method. [Figure 7B]

[0011] This shows a schematic example of control based on a route provided by a remote operator in one modified example of the present method. [Modes for carrying out the invention]

[0006]

[0012] The following description of preferred embodiments of the present invention is not intended to limit the invention to these preferred embodiments, but rather to enable those skilled in the art to create and use the invention.

[0007] 1. Overview

[0013] A system 100 for remote assistance of an autonomous agent, an example of which is shown in Figure 1, may include and / or interface with any or all of the following components: a sensor suite 110, a computing system 120, a communication interface 130, and / or any other suitable components. The system may further optionally include a set of infrastructure devices 140, a remote operator platform 150, and / or any other suitable components. System 100 functions to enable the exchange of information between the autonomous agent and the remote assistance platform (also referred to herein as the remote operator platform). Additionally or alternatively, system 100 may function to operate the autonomous agent (e.g., based on remote input indirectly received from a remote operator) and / or perform any other suitable functions. Additionally or alternatively, the system may function to facilitate the execution of method 200.

[0008]

[0014] As an addition or alternative, the system includes U.S. Application No. 16 / 514,624, filed July 17, 2019, now issued as U.S. Patent No. 10,564,641; U.S. Application No. 16 / 505,372, filed July 8, 2019, now issued as U.S. Patent No. 10,614,709; U.S. Application No. 16 / 540,836, filed August 14, 2019; and U.S. Application No. 16 / 792, filed February 17, 2020. This may include components or all thereof described in U.S. Application No. 780; U.S. Application No. 17 / 365,538 filed July 1, 2021; U.S. Application No. 17 / 550,461 filed December 14, 2021; U.S. Application No. 17 / 554,619 filed December 17, 2021; and U.S. Application No. 17 / 712,757 filed April 4, 2022, and by this reference each of these applications in whole is incorporated herein.

[0009]

[0015] System 100 is preferably used to perform any or all of Method 200, but may be used, in addition or as an alternative, to perform any other methods and / or processes.

[0010]

[0016] As shown in Figure 2, the method 200 for remote assistance of an autonomous agent may include any or all of the following: step S210 receiving a first set of inputs associated with the autonomous agent; step S215 optionally determining a remote assistance event; step S220 providing a set of outputs to a remote assistance platform; step S230 receiving a second set of inputs from the remote assistance platform; step S240 determining an action for the agent based on the first set of inputs and / or the second set of inputs; and step S250 optionally implementing the action. As an addition or alternative, Method 200 is described in U.S. Application No. 16 / 514,624, filed July 17, 2019, now issued as U.S. Patent No. 10,564,641; U.S. Application No. 16 / 505,372, filed July 8, 2019, now issued as U.S. Patent No. 10,614,709; U.S. Application No. 16 / 540,836, filed August 14, 2019; U.S. Application No. 16 / 792,780, filed February 17, 2020; and U.S. Application No. 16 / 792,780, filed July 1, 2021. U.S. Application No. 17 / 365,538; U.S. Application No. 17 / 550,461 filed December 14, 2021; U.S. Application No. 17 / 554,619 filed December 17, 2021; and U.S. Application No. 17 / 712,757 filed April 4, 2022. This may include any or all of the processes described in any or all of these applications, or any other suitable processes performed in any suitable order, and by this reference each of these applications in its entirety is incorporated herein.

[0011]

[0017] Method 200 may be carried out using the system described above and / or any other suitable system.

[0012] 2. Advantages

[0018] Systems and methods for remote assistance of autonomous agents can offer several advantages over current systems and methods.

[0013]

[0019] Firstly, variations of this technology can offer the advantage of interfacing with a human remote operator to alleviate the burden of complete classification (e.g., in edge cases) by the autonomous agent's computing system. In specific examples, for instance, the human remote operator may provide remote inputs related to policies available for consideration by the autonomous agent's computing system. In specific examples, the remote operator is used to effectively extend the operational design domain of the autonomous agent (e.g., when the vehicle's onboard computer remains the final decision-maker) instead of directly operating the autonomous agent. This may serve to reduce the importance of establishing a completely low-latency communication system between the agent and the remote operator. Additionally or alternatively, the remote operator may participate in and / or be used in other ways in direct operation of the autonomous agent.

[0014]

[0020] Secondly, since there is no need to train the agent to respond to all scenarios, and remote input from a remote operator can be utilized in rare (e.g., edge case) scenarios, variations of this technology can offer the advantage of reducing / minimizing the amount of training required for autonomous agents.

[0015]

[0021] Thirdly, a variant of this technology can facilitate the autonomous vehicle to overcome or bypass an impasse (e.g., “pull away” the vehicle from a waiting state where the vehicle cannot move forward) along the target vehicle path. For example, in the variant, in order to avoid an obstacle along the target vehicle path, it temporarily crosses a solid white line lane, a yellow lane or a double yellow lane; enters a lane associated with oncoming traffic; and / or facilitates the provision of permission from a remote operator to permit a temporary suspension of nominal driving rules / procedures (e.g., as per the instructions of traffic personnel, in the case of edge case events, etc.).

[0016]

[0022] Fourthly, a variant of this technology can facilitate the notification-based and / or discrete (non-continuous) remote operation of the autonomous vehicle, which may enable one remote operator to easily operate multiple autonomous vehicles (e.g., a 1:N relationship between the remote operator and the autonomous vehicles, where N is greater than 1), thereby reducing the burden and / or labor costs associated with remote vehicle operation.

[0017]

[0023] Additionally or alternatively, the system and method can provide any other advantages.

[0018] 3. System

[0024] System 100 for remote assistance of an autonomous agent, an example of which is shown in Figure 1, may include and / or interface with any or all of the following components: a sensor suite 110, a computing system 120, a communication interface 130, and / or any other suitable components. The system may optionally further include a set of infrastructure devices 140, a remote operator platform 150, and / or any other suitable components. System 100 functions to enable the exchange of information between the autonomous agent and the remote assistance platform (hereinafter also referred to as the remote operator platform). Additionally or alternatively, System 100 may function to operate the autonomous agent (e.g., based on remote input indirectly received from a remote operator) and / or perform any other suitable functions. Additionally or alternatively, the system may function to facilitate the execution of Method 200.

[0019]

[0025] The system 100 for remote assistance of an autonomous agent may include and / or interface with an autonomous operating system and an autonomous agent 102. Additionally or alternatively, the system may include U.S. applications 16 / 514,624 filed July 17, 2019, now issued as U.S. Patent No. 10,564,641; U.S. applications 16 / 505,372 filed July 8, 2019, now issued as U.S. Patent No. 10,614,709; U.S. applications 16 / 540,836 filed August 14, 2019; and U.S. applications 16 / 79 filed February 17, 2020. This may include components or all of those described in U.S. Application No. 2,780; U.S. Application No. 17 / 365,538 filed July 1, 2021; U.S. Application No. 17 / 550,461 filed December 14, 2021; U.S. Application No. 17 / 554,619 filed December 17, 2021; and U.S. Application No. 17 / 712,757 filed April 4, 2022, and by this reference each of these applications in whole is incorporated herein.

[0020] 3.1 System Components

[0026] System 100 preferably includes and / or interfaces with (e.g., is integrated within) an autonomous vehicle (also referred to herein as an autonomous agent, agent, and / or self-agent). The autonomous agent is preferably an autonomous vehicle, more preferably a fully autonomous vehicle and / or a vehicle operable as a fully autonomous vehicle, but additionally or alternatively may be a semi-autonomous vehicle and / or any other vehicle.

[0021]

[0027] Additionally or alternatively, the autonomous agent may be a vehicle that switches between a semi-autonomous state and a fully autonomous state (or a fully manned state), and thus, the autonomous agent may have the attributes of both a semi-autonomous vehicle and a fully autonomous vehicle depending on the state of the autonomous agent.

[0022]

[0028] In a preferred variant, the autonomous vehicle is an automobile (e.g., a passenger car, a driverless car, a bus, a shuttle, a taxi, a shared vehicle, a truck, a semi-truck, etc.). Additionally or alternatively, the autonomous vehicle may include: a ship (e.g., a boat, a water taxi, etc.), an aircraft (e.g., an airplane, a helicopter, a drone, a VTOL aircraft, etc.), a ground vehicle (e.g., a motorcycle, a bicycle, a moped, a scooter, etc.), and / or any one or all of any other suitable vehicle and / or transport device, autonomous machine, autonomous device, autonomous robot, and / or any other suitable device.

[0023]

[0029] The system may include or interface with a sensor suite 110 (also referred to herein as the sensor system) which functions to collect input to a computing system that can be used to determine one or more trajectories of an autonomous agent, and / or to perform any other block of block S210 and / or the method. Additionally or alternatively, the sensor suite may function to enable the operation of the autonomous agent (e.g., autonomous driving, steering an autonomous vehicle along a trajectory), capture data about the circumstances surrounding the autonomous agent, capture data related to the operation of the autonomous agent, detect maintenance needs of the autonomous agent (e.g., through engine diagnostic sensors, external pressure sensor strips, sensor health sensors, etc.), detect cleanliness criteria inside the autonomous agent (e.g., internal cameras, ammonia sensors, methane sensors, alcohol vapor sensors), and / or to perform any other appropriate function.

[0024]

[0030] A sensor suite (also referred to herein as a sensor system) may include vehicle sensors mounted on an autonomous agent, such as inertial sensors (e.g., accelerometers, gyroscopes, magnetometers, IMUs, INSs, etc.), external antennas (e.g., GPS, cellular, Bluetooth, Wi-Fi, short-range wireless communication, etc.), diagnostic sensors (e.g., engine load, tire pressure, temperature sensors, etc.), vehicle motion sensors (e.g., inertial sensors, wheel speed sensors, encoders, resolvers, etc.), environmental sensors (e.g., cameras, time-of-flight sensors, temperature sensors, wind speed / direction sensors, barometers, etc.), guidance sensors (e.g., LiDAR, radar, sonar, cameras, etc.), computer vision (CV) sensors, cameras (e.g., stereo cameras, hyperspectral, multispectral, video cameras, wide-angle, CMOS, CCD, etc.), time-of-flight sensors (e.g., radar, LiDAR, sonar, etc.), and / or any or all of any other suitable sensors. The sensor suite preferably includes sensors mounted on the autonomous vehicle (e.g., radar sensors and / or LiDAR sensors and / or cameras coupled to the exterior of the agent, IMUs and / or encoders coupled to and / or located within the agent), but may also include, as an addition or alternative, sensors located remotely from the agent (e.g., sensors communicating with the agent as part of one or more infrastructure devices), and / or any suitable sensors located at any suitable location.

[0025]

[0031] However, the sensor suite may include any other suitable set of sensors and / or may be otherwise appropriately configured.

[0026]

[0032] The autonomous agent preferably includes and / or interfaces with a computing system 120, which functions to process information (e.g., sensor inputs) to determine one or more sets of trajectories for the vehicle. Additionally or alternatively, the computing system may function to perform any or all processes involved in any or all of any other processes involved in perceiving, predicting, positioning, planning, controlling and / or operating the autonomous agent.

[0027]

[0033] The computing system preferably includes an onboard computing system mounted on and deployed within an autonomous agent (e.g., integrated within the autonomous agent). Additionally or alternatively, the computing system may include any or all of the following: remote computing systems (e.g., cloud computing systems, remote computing communicating with the onboard computing system, alternatives to the onboard computing system, etc.), computing systems integrated into auxiliary devices (e.g., mobile devices, user devices, etc.), edge devices including mobile computing devices, and / or any other suitable computing systems and devices. In one modification, for example, the autonomous agent may be operable to communicate with a remote or distributed computing system, which may include a user device (e.g., a mobile phone, laptop, etc.), a remote server, a cloud server, or any other suitable local and / or distributed computing system remote from the vehicle. The remote computing system may be connected to one or more systems of the autonomous agent through one or more data connections (e.g., channels), but alternatively, it may communicate with the vehicle system in any suitable manner.

[0028]

[0034] The computing system preferably includes a processing system (e.g., a graphics processing unit or GPU, a central processing unit or CPU, or any suitable processing circuit) and memory, but may include any other suitable components as additions or substitutions. The memory may be short-term memory (e.g., volatile, non-volatile, random-access memory, or RAM, etc.) and / or long-term memory (e.g., flash memory, hard disk, etc.).

[0029]

[0035] In one modification, for example, the onboard computing system functions to interact with and / or control operablely any one or more of the identified components or modules described herein. In a preferred modification, for example, the onboard computing system executes computer instructions to implement a Multi-Policy Decision Module (MPDM) that implements an MPDM process. In a specific example, the processing system and memory work together to dynamically manage a set of policies available to an autonomous agent within the framework of a Multi-Policy Decision Framework, such as those described in U.S. Application No. 16 / 514,624 filed July 17, 2019 and / or U.S. Application No. 17 / 365,538 filed July 1, 2021, each of which is incorporated herein by this reference in its entirety. In addition or alternative, the processing system and memory and / or any other suitable components may be used for any other suitable function.

[0030]

[0036] In a specific example, the system is configured to interface with and / or implement a multi-policy decision-making process (e.g., a multi-policy decision-making task block in computer-readable media) of its agent and any related components (e.g., a computer, processor, software module, etc.), and the multi-policy decision-making module of the computing system (e.g., an onboard computing system) includes, for example, a simulator module (or a similar machine or system) (e.g., a simulator task block in computer-readable media) that functions to predict (e.g., estimate) the effects of the behavior policies of each configured environment agent (e.g., other vehicles in the agent's environment) and / or objects identified in the agent's operating environment (e.g., pedestrians). The simulation may be based on each agent's past actions or past behavior derived from the agent's current state (e.g., current hypothesis) and / or historical data buffer (preferably including data up to the present). The simulation may further consider a target path (also referred to herein as a nominal path) that describes the desired routing of the vehicle (e.g., along a fixed path, along an unfixed path, towards a destination) so that the simulation is performed according to this target path. The simulation may also provide data on the interaction (e.g., relative position, relative velocity, relative acceleration, etc.) between the predicted behavior policy of each environment agent and one or more potential behavior policies that can be implemented by the autonomous agent.Data from the simulation may be used to determine (e.g., calculate) any number of metrics, which may work individually and / or collectively to evaluate any or all of the following: the potential impact of the agent on any or all of the environment agents when implementing a particular policy; the risks of implementing a particular policy (e.g., collision risk); how far the implementation of a particular policy will advance the agent toward a particular goal; and / or any other metrics involved in the selection of policies for the agent to implement.

[0031]

[0037] The set of metrics may optionally include and / or be determined based on a set of simulations performed against the proposed policy, including the cost function (and / or loss function) associated with each proposed agent policy, and / or jointly (for example, by aggregating any or all of the sets of metrics described below). Additionally or alternatively, the sets of metrics described below may be determined and / or analyzed individually, other metrics may be determined, metrics may be aggregated in other appropriate ways, and / or metrics may be constructed in other ways. Based on these metrics and / or functions, the best policy may be selected from the set of policies by comparing the metrics and / or functions among various proposed policies (for example, selecting the policy with the smallest cost / loss function, selecting the policy that optimizes the objective function [e.g., maximizing, minimizing, etc.], selecting the policy that minimizes risk and / or maximizes progress toward the goal / destination, etc.).

[0032]

[0038] The computing system (e.g., onboard computing system) preferably controls the autonomous agent to determine the state of the autonomous agent and / or the state of objects in the autonomous agent's operating environment (e.g., other vehicles / agents, pedestrians, bicycles, etc.), and functions to process detection data from the autonomous agent's sensor system (also referred to herein as sensor suite) (e.g., computer vision system, LIDAR, flash LIDAR, wheel speed sensor, GPS, etc.) and / or other sensors (e.g., from infrastructure devices) and / or information sources (e.g., remote assistance platform). Based on the state of the autonomous agent and / or objects in the operating environment (e.g., real objects, virtual objects, etc.) and / or any other information / instructions (e.g., program instructions, learning instructions, etc.), the onboard computing system may generate behavioral policies through a multi-policy decision module, etc., and may select behavioral policies to be executed by the computing system to control the behavior of the autonomous agent (e.g., lane change, merge, maintain current lane, turn left, turn right, stop, decelerate, accelerate, stop at a signal, stop at a stop sign, yield, etc.).

[0033]

[0039] In the first set of modifications, the computing system includes an onboard general-purpose computer adapted for I / O communication with the vehicle control system and sensor system, but may also include any suitable computing device, either additionally or alternatively. The onboard computing system is preferably connected to the Internet via a wireless connection (e.g., via a cellular link or connection). Additionally or alternatively, the onboard computing system may be coupled to any number of wireless or wired communication systems. The system may further include a second computing system or set of computing systems associated with and / or the remote assistance platform (e.g., a cloud-based computing system), and the onboard computing system and the remote assistance computing system communicate as described below.

[0034]

[0040] Any other computing system may be used as an addition or alternative.

[0035]

[0041] System 100 preferably includes a communication interface 130 that communicates with a computing system, the communication interface 130 functioning to enable the computing system to receive information (e.g., from infrastructure devices, remote computing systems and / or remote servers, remote operator platforms, or other autonomous agents or other vehicles) and to transmit information from the computing system (e.g., to remote computing systems and / or remote servers, remote operator platforms, infrastructure devices, other autonomous agents or other vehicles). The communication interface preferably includes a wireless communication system (e.g., Wi-Fi, Bluetooth, cellular 3G, cellular 4G, cellular 5G, multi-input multi-output or MIMO, one or more radios, or any other suitable wireless communication system or protocol), but may also include, in addition or alternatively: any or all of the following: a wired communication system (e.g., modulated power line data transfer, Ethernet, or any other suitable wired data communication system or protocol), a data transfer bus (e.g., CAN, FlexRay), and / or any other suitable components). In a concrete example, a communication interface implements a communication protocol between the onboard computing system and the remote support platform (for example, the computing system of the remote support platform).

[0036]

[0042] System 100 may optionally include a set of infrastructure devices 140 (e.g., as shown in Figure 4), also referred to herein as roadside units, that function individually and / or collectively to observe one or more aspects and / or features of the environment and to collect observational data relating to one or more aspects and / or features of the environment. The set of infrastructure devices preferably communicates with the onboard computing system of the autonomous agent, but may also communicate with a remote assistance platform, any other components and / or any combination thereof, in addition or alternatively.

[0037]

[0043] In a preferred modification, the infrastructure device may additionally function to collect data associated with observations and transmit the collected data and / or processed derivatives thereof to the autonomous agent. Additionally or alternatively, the infrastructure device may also function to collect data and transmit it to a remote operator platform, which the remote operator may use to inform the remote operator of its decisions, such as whether to include behavioral policies in consideration by the autonomous agent's computing system and / or to exclude behavioral policies from consideration. In a specific example, for instance, the infrastructure device may enable viewing of the view around the corners of a vehicle, which the agent and / or operator and / or the agent's remote operator may use to enable the autonomous agent to consider a turning behavior policy (by seeing that the road is clear for turning).

[0038]

[0044] In the first modified example, for example, the infrastructure device transfers the collected observation data to an autonomous vehicle service and / or remote platform (e.g., implemented via a network of distributed computing systems), such as a remote assistance platform, which is operated to interactively communicate with and / or control one or more functions of an autonomous agent.

[0039]

[0045] The infrastructure device preferably includes devices located very close to and / or within short-range communication of the autonomous agent's operating position and may function to collect data on the circumstances surrounding the autonomous agent and the circumstances of areas adjacent to the autonomous agent's operating zone. In one embodiment, the roadside unit includes one or more off-vehicle detection devices, including flash LiDAR, thermal imaging devices (thermal cameras), still image or video capture devices (e.g., image cameras and / or video cameras), global positioning systems, radar systems, microwave systems, inertial measurement units (IMUs), and / or any other suitable detection devices or combinations of detection devices.

[0040]

[0046] The infrastructure device may optionally include computing functions via processing circuits and communication interfaces that enable the infrastructure device to communicate with: autonomous agent computing systems, remote computing systems, remote operator platforms, and / or any other suitable components or combinations of components, or any of them.

[0041]

[0047] The technical advantages achieved by implementing infrastructure devices may include the ability to observe situations beyond the observable range of the autonomous agent (e.g., around a corner, under a vertical road), which may then function in the curation of one or more behavioral policies available to the agent (e.g., and considered by the agent in a multi-policy decision-making [MPDM] module). For example, at a given time, observation of one or more aspects of a given environment may be performed by the autonomous agent, and observation of one or more different and / or overlapping aspects of the given environment may be performed from different viewpoints by one or more infrastructure devices placed and operating in the given environment. In such embodiments, the viewpoint of the infrastructure devices, including observational data from the infrastructure devices, may be extended to observational data from the viewpoint of the autonomous agent to generate a comprehensive viewpoint of the autonomous agent's operating environment and / or provide additional information to one or more remote operators of a remote operator platform. This enables improved prediction of the operating environment and improved determination of behavioral policies to be selected and / or executed by the autonomous agent for safe and independent operation within the operating environment (away from the human operator on board).

[0042]

[0048] In one variation, the autonomous agent may augment and / or fuse the data derived by the onboard sensor suite with additional observations from infrastructure devices (e.g., roadside units) to improve policy curation and / or trajectory determination by the autonomous agent.

[0043]

[0049] As an addition or alternative, infrastructure devices may detect and track any type or kind of object in and / or virtually inserted into the operating environment, such as using video cameras or radar. In one variation, for example, a video camera could function to provide detection of objects, such as a pedestrian attempting to cross a road, or a car attempting to turn left, a driver attempting to get out of a car with the door open, a cyclist traveling in a bicycle lane, and / or any other relevant information, as well as a semantic classification of the type of object and the possible intent of the object.

[0044]

[0050] In addition or alternatively, any or all of the infrastructure devices may include: traffic management devices (e.g., traffic sensors, traffic lights, pedestrian signals, etc.) operating in an environment in which they can function to communicate directly with an autonomous agent with respect to any or all of the data collected and / or detected by the infrastructure devices, data relating to the operational status of the infrastructure devices (e.g., red or green light), and / or any other information; and / or communicate directly with a remote operator platform, and / or communicate in any other suitable way. For example, a traffic light may be an infrastructure device in the environment surrounding an autonomous vehicle, which can function to communicate operational status information such as the color of the light projected by the traffic light, or other information such as the timing of changes in the light of the traffic light.

[0045]

[0051] Additionally or alternatively, information from traffic management devices may be used to trigger notifications to remote operators, or to prevent triggers (e.g., false positives) from being sent to remote operators. For example, if a vehicle is stopped for longer than a threshold time, which may indicate that it needs to be contacted by a remote operator, a display indicating that the vehicle is stopped at a red light based on the traffic management device may be used to prevent a request for assistance from being sent to the remote operator. Alternatively, if the vehicle is stopped for longer than a threshold time and the traffic light at which the vehicle is stopped is green, a request for assistance may be sent to the remote operator.

[0046]

[0052] The system may optionally include and / or interface with a vehicle control system which includes one or more controllers and / or control systems, and which include any suitable software and / or hardware components (e.g., a processor and a computer-readable memory device) used to generate control signals for controlling the autonomous agent in accordance with the autonomous agent's routing goals and selected behavioral policies and / or selected trajectories.

[0047]

[0053] In addition or alternatively, the vehicle control system may include and / or interface with any or all of the agent's configured electronic modules, including but not limited to, an electronic control unit [ECU], a telematics control unit [TCU], a transmission control module [TCM], an anti-lock braking system [ABS] control module, a body control module [BCM], and / or any other suitable control subsystems and / or modules.

[0048]

[0054] In a preferred modification, the vehicle control system includes, interfaces with, and / or implements the vehicle's drive-by-wire system. Additionally or alternatively, the vehicle may be operated according to the operation of one or more mechanical components, and / or implemented in other ways.

[0049]

[0055] In addition or alternatively, the system may include any or all of the following components: a sensor fusion system, a positioning system (e.g., including position sensors in the sensor system), a guidance system, and / or any suitable components. In one variation, for example, the sensor fusion system synthesizes and processes sensor data and, together with a multi-policy decision module, predicts the presence, location, classification, and / or path of objects, as well as the characteristics of the environment for an autonomous agent (real or virtual). In various embodiments, the sensor fusion system may function to incorporate data from multiple sensors and / or data sources, including, but not limited to, cameras, LiDAR, radar, infrastructure devices, remote data feeds (Internet-based data feeds), and / or any number of other types of sensors.

[0050]

[0056] The positioning system may process sensor data along with other data to determine the autonomous agent's position relative to the environment (e.g., local position on a map, precise position relative to road lanes, vehicle direction of travel, speed, etc.) and may function to determine what behavioral policies are available to the autonomous agent (e.g., as described later). The guidance system may process sensor data along with other data to determine the path the vehicle should follow.

[0051]

[0057] The system preferably includes and / or interfaces with a remote assistance (remote operator) platform 150, the remote assistance (remote operator) platform 150 being one or more remote remote operators and associated components (e.g., a communication interface with an autonomous agent, a computing system, an output device for displaying information from the autonomous agent and / or an infrastructure device to the remote operator, an input device for receiving instructions / commands from the remote operator, etc.). The remote assistance platform may function to provide output to one or more remote operators and to receive input from one or more remote operators, which may be used to determine a curated behavior policy for the vehicle. In a preferred variation, for example, remote input from a remote operator functions to effectively extend the operational design area (ODD) associated with the vehicle by approving and / or proposing a behavior policy for the vehicle. In a preferred specific example, the behavior policy approved and / or proposed by the remote operator is ultimately either selected or rejected by the computing system onboard the autonomous agent (e.g., the remote operator may not have direct command / control authority over the vehicle system—such as braking, steering, or trajectory execution). In a concrete example of implementing an MPDM module, for instance, a remote operator may effectively approve behavioral policies for consideration by the onboard computing system, which ultimately selects the policy to implement (and plans / executes vehicle control commands according to the policy decision). Therefore, in a preferred example, commands for directly controlling the agent are not received from the remote operator. This has the advantage that latency from the remote assistance platform to the onboard computing system becomes less of a concern, as the remote operator is not used for safety-critical tasks but rather to provide contextual information to the onboard computing system, which ultimately makes the driving decisions.For example, remote operator input (e.g., remote operator approval policy) may be received via a wireless connection (e.g., between the onboard computing system and the remote assistance platform) and / or implemented with a latency of more than 50 milliseconds, thereby facilitating remote operator intervention even in areas with insufficient wireless communication range / connectivity. However, the remote operation latency may be less than 50 milliseconds, and / or a human operator may be on board the autonomous agent to quickly control it as needed.

[0052]

[0058] Alternatively, any or all of the behavioral policies approved and / or proposed by the remote operator may be implemented automatically, or finally implemented (e.g., to override other behavioral policies), the remote operator may directly control the vehicle, and / or the remote operator's input may be implemented in another way.

[0053]

[0059] The remote operator of the remote assistance platform may, as an addition or alternative, be used to monitor the interior of the autonomous agent, such as viewing and / or communicating (e.g., via voice) with the rider / passenger inside the vehicle cabin. In one variation, the remote operator may be used, for example, to ensure that the rider is seated and / or seated before the vehicle moves, to communicate with the passenger to provide instructions to the passenger, and / or to perform any other functions.

[0054]

[0060] As described above, the system preferably includes and / or interfaces with a multi-policy decision-making (MPDM) module. In a preferred set of variations of the MPDM module, the MPDM module includes a simulator or similar machine or system which functions to estimate future (i.e., forward-looking in time) behavioral policies (operations or actions) for each of the autonomous agents identified in the operating environment of the autonomous agents (e.g., other vehicles on the road, cyclists, pedestrians, etc.), including potential behavioral policies that can be performed by the autonomous agents, as described in either or all of U.S. applications 15 / 923,577 filed March 16, 2018 and 16 / 514,624 filed July 17, 2019, each of which is incorporated by this reference. The simulation may be based on the current state of each agent (e.g., current hypothesis), the historical actions or behaviors of each agent derived from a historical data buffer (preferably including data up to the present), and / or any combination or other information. The simulation may provide data relating to the interaction (e.g., relative position, relative velocity, relative acceleration, etc.) between each agent's predicted behavior policy and one or more potential behavior policies that the autonomous agent may execute. The MPDM module may then select one of several behavior policies based on one or more default or dynamic selection criteria. The selection criteria may be based on any appropriate behavior policy selection elements that can be described before the autonomous agent is made to operate, or they may be dynamically based on one or more characteristics relating to the autonomous agent's operating environment or operating mode. For example, the selection criteria may be predetermined and / or set so that the autonomous agent selects the behavior policy that is most likely to be executed safely. In another example, if the operating environment of an autonomous vehicle includes an emergency, the selection criteria may be dynamic and the computing system may be set to select a behavior policy from a manageable set of behavior policies that require a (weighted) balance of operational efficiency and safety, etc.As an addition or alternative, a policy may be selected to limit potential behavioral policies for autonomous agent execution based on one or more default thresholds relating to the probability of execution by the autonomous agent. That is, in one embodiment, the MPDM module may theoretically generate hundreds, if not thousands, of simulations, resulting in hundreds or thousands of potential behavioral policies for autonomous agent execution under given circumstances. Accordingly, the MPDM module may function to identify only a subset of those generated behavioral policies according to default thresholds that identify one or more minimum probability values ​​for safely executing an action or operation by the autonomous agent, or one or more minimum probability values ​​for successfully executing an operation or action by the autonomous agent under given circumstances or real-time scenarios.

[0055]

[0061] Additionally or as an alternative, system 100 may include and / or interface with any other suitable components.

[0056] 4. Method

[0062] As shown in Figure 2, the method 200 for remote assistance of an autonomous agent may include any or all of the following: a step S210 of receiving a first set of inputs associated with the autonomous agent; an optional step S215 of determining a remote assistance event; a step S220 of providing a set of outputs to a remote assistance platform; a step S230 of receiving a second set of inputs from the remote assistance platform; a step S240 of determining an action for the agent based on the first set of inputs and / or the second set of inputs; and an optional step S250 of implementing the action. As an addition or alternative, Method 200 is incorporated herein by reference in its entirety by this reference, U.S. Application No. 16 / 514,624, filed July 17, 2019, now issued as U.S. Patent No. 10,564,641; U.S. Application No. 16 / 505,372, filed July 8, 2019, now issued as U.S. Patent No. 10,614,709; U.S. Application No. 16 / 540,836, filed August 14, 2019; and U.S. Application No. 16 / 540,836, filed February 17, 2020. This may include any or all of the processes described in any or all of the following: UNITED NATIONS No. 16 / 792,780; U.S. NATIONAL NATIONS No. 17 / 365,538 filed 1 July 2021; U.S. NATIONAL NATIONS No. 17 / 550,461 filed 14 December 2021; U.S. NATIONAL NATIONS No. 17 / 554,619 filed 17 December 2021; and U.S. NATIONAL NATIONS No. 17 / 712,757 filed 4 April 2022, or any other suitable processes performed in any suitable order.

[0057]

[0063] Method 200 is preferably performed using the system 100 described above, but may also be performed using any other suitable system, either additionally or as an alternative.

[0058]

[0064] Method 200 preferably functions to help an autonomous agent navigate uncertain and / or otherwise difficult environments, such as environments that the agent has not learned to classify. Additionally or alternatively, Method 200 may function to enable internal monitoring of the agent (e.g., for passenger safety) and / or perform any other functions.

[0059]

[0065] Method 200 is preferably implemented in the context of a ride-sharing use case involving the transportation of passengers, such as a shuttle service, an on-demand ride service, a robot taxi, and / or any other passenger use case. Additionally or alternatively, Method 200 may be implemented for transporting goods (e.g., delivering groceries) and / or may be used in any other use case.

[0060]

[0066] Method 200 and / or its sub-elements may be executed once, and / or repeatedly, recursively, iteratively, periodically, cyclically, sequentially, and / or at any other appropriate timing. For example, an autonomous agent may select a policy for each step of a selection cycle (e.g., at a predetermined frequency such as 10Hz, 15Hz, 20Hz, 25Hz, and above 25Hz) and autonomously control the vehicle based on the selected policy until a new policy is selected (e.g., in the next / subsequent step of the selection cycle). In variations, combinations / permutations of method elements may occur sequentially, simultaneously, synchronously, asynchronously, periodically, aperiodicly (e.g., on an event-driven basis), and / or at any other appropriate frequency / timing.

[0061] 4.1 Method - Step S210 to receive the first set of inputs associated with the agent

[0067] Method 200 preferably includes step S210, which receives a first set of inputs associated with an agent, and step S210 functions to receive information for performing any or all of the remaining processes of Method 200. In a preferred modification, for example, the first set of inputs received in S210 functions to collect information for making decisions in the onboard computing system and / or the remote assistance platform. In addition or alternative, S210 may perform any other suitable function.

[0062]

[0068] As an addition or alternative, method 200 may be performed without S210 and / or may include any other process.

[0063]

[0069] S210 is preferably executed first in method 200, and more preferably multiple times during the operation of the autonomous agent (e.g., continuously, at a predetermined frequency, in a set of predetermined intervals, at random intervals, in response to a trigger, periodically / periodically, once for each selection cycle of the MPDM and / or autonomous agent). Additionally or alternatively, S210 may be executed in response to any other process of method 200, in parallel with any other process of method 200, and / or at any other time. Further additionally or alternatively, method 200 may be executed without S210.

[0064]

[0070] The first set of inputs is preferably received at least partially from the autonomous agent's sensor system (e.g., as described above), such as any or all of the sensors described above (e.g., LiDAR sensors, RADAR sensors, cameras, microphones, diagnostic sensors, etc.). In addition or alternatively, inputs may be received from any suitable sensors (e.g., part of one or more infrastructure devices remote from the agent), other sources (e.g., online sources, databases, etc.), other agents and / or objects, and / or any other suitable sources.

[0065]

[0071] The set of inputs preferably includes a set of camera streams (also referred to herein as video streams) collected from a set of cameras mounted and arranged on the autonomous agent. Additionally or alternatively, the set of inputs may include camera streams collected from within the autonomous agent, information from a set of lidar sensors, information from a set of radar sensors, diagnostic information associated with one or more components of the autonomous agent (e.g., health information associated with sensors), and / or any other inputs.

[0066]

[0072] In the first modified example, S210 includes the step of receiving a set of video streams collected from a set of cameras mounted and arranged on the autonomous agent, the set of video streams representing the environment of the autonomous agent. Additional or alternative, the first set of inputs may include information from a set of lidar sensors, information from a set of radar sensors, a second set of video streams having a field of view inside the autonomous agent, information from infrastructure devices, and / or any other information.

[0067]

[0073] As an addition or alternative, S210 may include any or all of the following steps: receiving sensor measurements from vehicle sensors (e.g., a sensor suite); and obtaining stored data / information (e.g., from the memory and / or remote data storage of a computing system; such as previous states of the vehicle and / or previous environmental representations generated by the vehicle's perception / tracking); current environmental representations (e.g., generated by a perception / tracking system); current state of the vehicle (which may include, for example, the vehicle's position in an Earth coordinate system); current vehicle trajectory; target vehicle path (e.g., a default and / or received from a guidance / control system); and / or any other suitable inputs.

[0068]

[0074] In a modified version, S210 may optionally include a step of processing sensor inputs to generate an environmental representation and an estimate of the current vehicle state (e.g., location / position, speed, acceleration on various axes, throttle, steering angle, etc.) (e.g., using a perception and / or tracking system / method, such as, for example, described in whole by this reference in U.S. Application No. 17 / 554,619 filed December 17, 2021). Alternatively, the environmental representation and / or the current vehicle state and / or any other suitable (pre-processed) inputs may be received during S210 (e.g., pre-processed by an upstream element of a sensor data processing pipeline).

[0069]

[0075] In a concrete example, the input may include an environment representation that includes a set of dynamic objects and a set of static objects of the vehicle's environment, each uniquely labeled with an identifier and object parameters that may include a dynamic or static label, an object classification (e.g., car, pedestrian, bicycle, etc.), object dimensions, object movement parameters (e.g., position, velocity, acceleration, etc.), and / or any other appropriate information.

[0070]

[0076] However, any other suitable input may be received during S210.

[0071] 4.2 Method - Step S215 for determining a remote support event

[0077] Method 200 may optionally include a step of determining a remote assistance event S215 that can function to initiate remote assistance for a vehicle (for example, if the vehicle is stuck / expected to be stuck or otherwise unable to move forward). For example, remote assistance may be provided in a notification-driven, discretization-based, and / or discontinuous-based manner in response to the determination of a remote assistance event, thereby enabling a single remote operator to assist multiple vehicles.

[0072]

[0078] Remote assistance events may be determined based on sensor inputs received during S210, such as the vehicle state and / or environmental representation (e.g., and / or dynamic or static objects within it), predicted probabilities / scenarios, past vehicle states, policies, and / or action decisions (e.g., according to S240), and / or any other appropriate information. Remote assistance events may also be determined according to a set of one or more thresholds, such as a time threshold while the vehicle is stopped / stationary. Remote assistance events may be provided when the vehicle is stationary (e.g., stopped in a dead end), moving, traversing along a target vehicle path, traversing beyond a target vehicle path (or otherwise deviating from a target path or a path approved by the remote operator), and / or during any appropriate vehicle state or situation.

[0073]

[0079] In addition or alternatively, the set of thresholds may include other time thresholds (e.g., a vehicle being stationary in a specific situation, such as being in the center of a lane for more than a threshold time, such as 15 seconds), speed thresholds (e.g., a vehicle speed below a threshold speed), uncertainty thresholds (e.g., determined by a computing system), and / or any or all of any other thresholds or other triggers.

[0074]

[0080] As an addition or alternative, remote assistance events may be determined based on one or more probability and / or predictive scenarios. In one variation, for example, a remote assistance event may be determined based on the probability that the vehicle will get stuck (e.g., unable to easily merge into traffic, unable to move again within a predetermined time), and the probability may be an actual probability, a predicted probability (e.g., based on the volume of traffic ahead of the agent, based on environmental representation, based on a set of training models, etc.) and / or any combination thereof.

[0075]

[0081] Some or all of the triggers may optionally utilize information from infrastructure devices, such as infrastructure devices connected to and / or having a line of sight to traffic signals. In one variation, for example, if a vehicle has been stopped for at least a minimum time threshold, information from the infrastructure device may be used to determine whether the vehicle is stopped at a red light, in which case no notification is triggered to the remote operator.

[0076]

[0082] In the first modified example, a remote assistance event may be determined in response to the detection that the vehicle has been stopped for at least a threshold time (e.g., 1 minute, 2 minutes, 1 to 5 minutes, less than 1 minute, more than 5 minutes, etc.; in a dead end, etc.).

[0077]

[0083] In a second variation, a remote assistance event may be determined in response to the detection that the agent's environment cannot be classified and / or that the uncertainty associated with the classification exceeds a predetermined threshold. For example, a remote operator may be triggered in response to a high level of uncertainty in the onboard computing system, such as when an autonomous agent encounters a semi-truck reversing toward the autonomous agent.

[0078]

[0084] In the third variation, a remote assistance event may be determined in response to the failure of the fallback system to start and / or the failure of the onboard computer to update the policy.

[0079]

[0085] In the fourth modified example, a remote assistance event may be determined in response to a determination that the vehicle is unable to move forward along the target path (for example, an obstacle or obstruction is blocking the vehicle, an obstacle or obstruction is stopped / stationary along the target path, or an obstacle or obstruction is stopped / stationary in front of the vehicle).

[0080]

[0086] In the fifth variation, a remote assistance event may be determined in response to the determination and / or selection of a policy requiring remote permission (e.g., to deviate from nominal driving rules, to cross a double yellow line, to follow instructions from a construction worker).

[0081]

[0087] In the sixth variant, a remote assistance event may be determined based on a time threshold during which the vehicle is stopped at a dead end along the target vehicle path.

[0082]

[0088] In the seventh variant, the remote assistance event may be determined based on a probability threshold for the occurrence of a deadlock along the target vehicle path.

[0083]

[0089] In the eighth variant, the remote assistance event may be determined based on a prior trigger based on input in S230. For example, the remote operator may be prompted to check in the vehicle again at one or more points during the execution of the approved policy (or route) to reconfirm or continue the policy (e.g., if only a portion of the policy is approved at a time). Thus, such triggers may occur repeatedly, periodically, or based on secondary event triggers (e.g., reaching a waypoint, deviating from a route approved by the remote operator, etc.) and / or at any other appropriate time.

[0084]

[0090] However, remote assistance events may be appropriately determined in other ways. Alternatively, in some implementations, such as when a remote operator continuously monitors vehicle operations, remote assistance events may not need to be determined (for example, they may not need to be notified on an event-driven basis).

[0085] 4.2 Method - Step S220 of providing a set of outputs to the remote support platform

[0091] Method 200 preferably includes step S220, which provides a set of outputs to a remote assistance platform, the steps of which function to provide information that a remote operator can see and / or analyze to help an autonomous agent navigate its environment. In addition or alternatively, S220 may function to send warnings and / or notifications to a remote operator and / or perform any other functions.

[0086]

[0092] As an addition or alternative, method 200 may be performed without S220 and / or may include any other process.

[0087]

[0093] S220 is preferably executed in response to and based on S215 and / or S210, and more preferably multiple times during the operation of the autonomous agent (e.g., continuously, at a predetermined frequency, at a predetermined set of intervals, at random intervals, in response to triggers, etc.), such as in response to each instance of S215 (e.g., during multiple selection cycles after a remote assistance event, during a period of deviation from the target path, during a predetermined period, etc.). Additionally or alternatively, S220 may be executed in response to any other process of method 200, in parallel with any other process of method 200, and / or at any other time. Further additionally or alternatively, method 200 may be executed without S220 (e.g., during periods of operation in which no remote assistance events are detected).

[0088]

[0094] The remote operator associated with the remote support platform is preferably a human remote operator, but may also include and / or interface with a robotic operator, either as an addition or alternative.

[0089]

[0095] The set of outputs preferably includes any or all of the first set of inputs received in S210, for example, camera streams collected from a set of cameras that view the environment of the autonomous agent; camera streams collected from a set of cameras that view the interior of the autonomous agent; diagnostic information associated with the autonomous agent (e.g., sensor health); information from the onboard diagnostic ports of the autonomous agent (e.g., speed, acceleration, vehicle status); information collected from a set of infrastructure devices; and / or any or all of any other inputs.

[0090]

[0096] The set of outputs may also include, as an addition or alternative, outputs determined based on a first set of inputs (e.g., outputs derived from the first set of inputs, or outputs determined based on further processing of the first set of inputs), such as a set of visualizations provided to a remote operator. Visualizations may include, for example, proposed and / or predicted trajectories for the agent itself, proposed and / or predicted trajectories for other agents in the agent's environment, proposed maneuvers for the autonomous agent, and / or any other visualizations.

[0091]

[0097] The set of outputs may also include, as an addition or alternative, one or more notifications to the remote operator (e.g., messages, warnings, etc.), which may function to alert the remote operator in specific scenarios requiring input from the remote operator (e.g., the vehicle has stopped for 30 seconds, deceleration ahead is expected, etc.). The notifications may optionally include proposed policies (e.g., behavior, action, maneuver, trajectory, etc.) and / or multiple proposed policies for the autonomous agent, and the remote operator may approve and / or reject the proposed policies. Based on the remote operator's input (e.g., received during S230), one or more proposed policies may be included (e.g., if permitted / approved by the remote operator) or excluded for consideration by the computing system in determining the best policy for the autonomous agent (e.g., as part of the MPDM module, during the current or subsequent selection cycle).

[0092]

[0098] For example, in one modification, the computing system 120 (e.g., its MPDM) may determine a policy or policy proposal that is not autonomously selectable (e.g., obtained from a database) which may be provided as output to the remote operator in S220 (e.g., in response to S215, in response to a decision that the vehicle is stuck). A policy may be deemed "not autonomously selectable" as a result of a confidence score associated with that policy that falls below a default threshold (e.g., a minimum threshold) (e.g., confidence that the vehicle can implement the policy, confidence that the vehicle can implement the policy safely, etc.; a policy that is not autonomously selectable may be a policy that is nominally not selectable without the approval of the remote operator, but becomes selectable with the approval of the remote operator), a deviation of the policy from nominal road rules / conventions, the current status of the vehicle, and / or any other appropriate reason. In addition or alternatively, policy proposals provided as output during S220 may be selectable, and / or no policy proposals may be provided.

[0093]

[0099] The set of outputs is preferably communicated wirelessly to the remote operator platform 150 via the communication interface 130. Furthermore, it is even more preferable that the set of outputs is at least partially provided to a display associated with the set of remote operators. The remote support platform may, in addition or alternatively, include speakers, microphones and / or any other output devices. The remote support platform may further, in addition or alternatively, include any number of input devices configured to receive a second set of inputs (e.g., described later), such as touchscreens, buttons, joysticks, microphones (e.g., with voice detection) and / or any other input devices.

[0094]

[0100] In a modified version, the output set may optionally include a top-down (bird's-eye view) of the environmental representation, including estimated object parameters for the environmental representation (e.g., dimensions, classification, etc.), thereby facilitating the provision of waypoints by a remote operator (e.g., to bypass obstacles around a vehicle).

[0095]

[0101] However, any other suitable set of outputs may be provided to the remote assistance platform.

[0096] 4.3 Method - Step S230 to receive a second set of input from the remote support platform

[0102] Method 200 preferably includes step S230 of receiving a second set of inputs from a remote support platform, the step of which S230 functions to allow the autonomous agent to receive assistance from a remote operator in decision-making. In addition or alternatively, S230 may perform any other function.

[0097]

[0103] As an addition or alternative, method 200 may be performed without S230 and / or may include any other process.

[0098]

[0104] Preferably, S230 is executed in response to and based on S220, and optionally multiple times during the operation of the autonomous agent (e.g., consecutively, at a predetermined frequency, in a predetermined set of intervals, at random intervals, in response to a trigger, etc.). Additionally or alternatively, S230 may be executed in response to any other process of method 200, in parallel with any other process of method 200, and / or at any other time. Further additionally or alternatively, method 200 may be executed without S230.

[0099]

[0105] A second set of inputs is preferably received from a remote support platform and more preferably from one or more remote operators in response to viewing a set of outputs, but may be received from any other location and / or entity, additionally or alternatively. The second set of inputs is preferably received after or in response to a remote support event trigger, such as exceeding a threshold (e.g., downtime). Additionally or alternatively, the second set of inputs may be received without a trigger, in response to a different trigger, and / or at any other point in time.

[0100]

[0106] The second set of inputs preferably relates to policies and / or sets of policies available to the autonomous agent, where a policy is a behavior policy that defines specific behaviors (e.g., lane changes, lane keeping, stopping, speeding up, speeding down, etc.) and / or actions on the vehicle. Additionally or alternatively, a policy may include and / or define any or all of the following: vehicle actions, vehicle steering (e.g., crossing a set of lanes), vehicle trajectory (e.g., tracing, a set of waypoints, etc.), a set of parameters associated with implementing the behavior (e.g., vehicle speed, distance to be maintained relative to other objects and / or road geometry, vehicle position, etc.), and / or any or all of any other information associated with the agent.

[0101]

[0107] The set of policy inputs received by S230 may optionally include policies that the agent may not be able to independently select and / or consider selecting, such policies include, for example: policies corresponding to rare policies and / or edge cases; policies that violate traffic rules and / or customs (e.g., crossing lanes on a two-lane road); policies that move the agent outside of a prescribed route and / or fixed route; behaviors that a vehicle may normally be able to select but have different parameters and / or characteristics (e.g., different speeds, different distances maintained relative to other objects, inclusion of locations where a vehicle crosses lanes); and / or any other policies, either or all of them. Additionally or alternatively, policies may include policies that the agent can select and / or consider selecting without input from a remote operator.

[0102]

[0108] The second set of inputs may include: approval (e.g., permission) of specific proposed policies (e.g., crossing double lines on the road, driving onto the shoulder, deviating from a fixed route); rejection of specific proposed policies; addition of policies (e.g., outside the set that the vehicle itself can select); custom policies (e.g., based on a trajectory trace created by a remote operator); and / or any or all of any other inputs.

[0103]

[0109] In the first set of variations, for example, the second set of inputs includes policies (e.g., actions, behaviors, maneuvers, etc.) for consideration by the autonomous system (e.g., the onboard computing system). In the first set of specific examples, the second set of inputs includes instructions for the remote operator to approve the proposed policies. In the second set of specific examples, as an addition or alternative to the first set of inputs, the second set of inputs includes additional policies proposed by the remote operator, which may be selected from, for example, a set of non-standard policies. As an addition or alternative, the second set of inputs may include rejection of proposed policies. In specific examples, for example, a policy may be proposed to the remote operator that the remote operator identifies as suboptimal (e.g., inefficient, disruptive to passengers, vehicle deviating from a fixed route, etc.), dangerous (e.g., moving a vehicle into a construction area, moving a vehicle into an area with many pedestrians, etc.), or otherwise unideal. Thus, the remote operator may reject the proposed policy and, therefore, prevent the agent (e.g., the MPDM framework) from considering the proposed policy.

[0104]

[0110] In a second set of modifications, the second set of inputs may include custom maneuvers from the remote operator, such as a trace and / or a set of associated waypoints for a vehicle trajectory drawn (e.g., via a touch interface), and / or one created by the remote operator in another manner. In specific examples, the second set of inputs may include a trace or route approved by the remote operator in the form of a set of waypoints that bypass obstacles (e.g., dynamic or static obstacles) along the target vehicle path. In additional or alternative examples, the remote operator may provide a set of waypoints in response to determining that a proposed (non-autonomously selectable) policy is not optimal (e.g., unsafe, unsuccessful in overcoming a stuck vehicle, etc.).

[0105]

[0111] In this set of specific examples, the routes approved by this remote operator may be used as input to the MPDM module (e.g., along with the target route, instead of the target route) so that a policy is simulated and / or selected according to one or both of these routes (e.g., a policy is selected that helps steer the vehicle to avoid obstacles and return the vehicle to the target route).

[0106]

[0112] In the third set of modifications, the second set of inputs may include a dichotomy decision by the remote operator (e.g., accept / reject the proposed optimal policy). In the fourth set of modifications, the second set of inputs may include a command from the vehicle's remote operator to wait (e.g., remain stopped), the command may function to allow the vehicle to collect more sensor data and gain a better understanding of its environment before moving and / or selecting a new policy (e.g., a better understanding of how other objects are moving, a more reliable classification of object types, etc.). The wait command is preferably used in accordance with the MPDM module so that policies that move the vehicle are weighted and / or scored lower (e.g., less attractive to the vehicle's selection so that they are only selected if their weighted scores still outperform the scores of other policies). In specific examples, the remote operator may voluntarily provide a wait command (e.g., without S215). Alternatively, the wait behavior may be proposed as an autonomously non-selectable policy, included in a set of selectable policies, used to override vehicle decision-making, and / or used in other ways. Additionally or alternatively, a remote operator may provide and / or approve a “continue” policy to allow the vehicle to proceed after a wait command has been provided. For example, a wait command may be executed by default upon arrival at the vehicle’s destination, placing the vehicle and / or policy selection into a wait state and waiting for remote operator approval before the vehicle proceeds. Alternatively, the vehicle may be configured to autonomously terminate the wait command, or the wait command may be manually disabled by a human (e.g., by initiating the vehicle’s departure).

[0107]

[0113] As an addition or alternative, the second set of inputs may include any other information.

[0108] 4.4 Method - Step S240 of determining the agent's action based on the first set of inputs and / or the second set of inputs.

[0114] Method 200 preferably includes step S240, which determines an agent action based on a first set of inputs and / or a second set of inputs, wherein step S240 functions to perform an autonomous agent decision based on any or all of the above inputs.

[0109]

[0115] As an addition or alternative, method 200 may be performed without S240 and / or may include any other process.

[0110]

[0116] Preferably, S240 is executed multiple times at the discretion of the autonomous agent during its operation, in response to and based on S230 (e.g., sequentially, at a predetermined frequency, in a predetermined set of intervals, at random intervals, in response to a trigger, etc.). Additionally or alternatively, S240 may be executed in response to any other process of method 200, in parallel with any other process of method 200, and / or at any other time. Further additional or alternatively, method 200 may be executed without S240.

[0111]

[0117] S240 is preferably executed by the onboard computing system based on either or all of the first set of inputs and the second set of inputs. More preferably, S240 is executed according to a multi-policy decision module (e.g., one as described above), in which inputs received from the remote operator in S240 are used to determine from the MPDM module which policies the autonomous agent can select. In one variation, for example, if a proposed policy by the remote operator is approved, that proposed policy is included in a group of policies for the computing system to consider and select, and if the proposed policy is rejected, that policy is excluded from consideration by the computing system.

[0112]

[0118] In a modified example, S240 is preferably performed based on a target vehicle path, such as a predetermined vehicle route or path for the vehicle (e.g., determined before execution, determined in a previous selection cycle). As an example, the system and / or MPDM may be configured to prioritize a policy that strictly follows the target path (e.g., using a cost function or a set of selection criteria / heuristics; the system typically follows the target path broadly under nominal operation; etc.). Additionally or alternatively, if a remote operator provides a trace of the autonomous agent's trajectory in S240, S250 may include a step of planning a path for the vehicle based on that trace (and / or a set of associated waypoints) that deviates from the target vehicle path. For example, a remote operator may provide an autonomous agent with a set of waypoints in S240, and in S250, the computing system plans a route for the autonomous agent that does one or all of the following: follow waypoints (or prioritize a trajectory that follows waypoints of policy selection), reach the maximum number of waypoints, reach at least a default number of waypoints, reach the first waypoint, reach the final waypoint, and / or plan a route in another way based on waypoints. This may be included as a policy available to the agent if the vehicle is stuck (e.g., in a construction area), if the remote operator determines that the vehicle is stuck or will become stuck, and / or at any other time (e.g., this may be the only policy in an extreme edge case where there are no other viable policies available to the agent). Additionally or alternatively, the policy / action selection may consider a route approved by the remote operator, while retaining the authority to decide to deviate from a waypoint and / or return to the (previous) target route.For example, in response to receiving a route and / or waypoint approved by the remote operator during S230, the target vehicle route may be updated based on that route, and the steps of autonomously controlling the vehicle may include: selecting an updated policy from a set of selectable policies, the updated policy deviating from the route approved by the remote operator (and / or the target route; an example is shown in Figure 7B). Alternatively, the vehicle may precisely follow the route approved by the remote operator (and / or a portion thereof) and / or a portion of the target route.

[0113]

[0119] In an exemplary example, a remote operator may provide waypoints based on a top-down (overhead) view that includes, for example, the estimated object dimensions of an obstacle ahead of a vehicle (e.g., a heavy truck) (determined using a classifier in a computing system, where this is determined based on classified object types such as vehicle vs., truck vs. bus, etc.). However, the dimensions of the obstacle, particularly its length, may not be directly or easily observable via vehicle sensors (for example, a 72-foot Class 8 semi-truck may visually resemble a 26-foot mobile truck when viewed from behind). Therefore, the vehicle may deviate from the path approved by the remote operator (returning earlier or later to the previous target path) based on subsequent environmental observations (e.g., the vehicle begins to move around the object and, as a result, directly observes the dimensions of the obstacle as it collects more information from onboard sensors, etc.).

[0114]

[0120] In the second example, in S240, a policy is selected using a multi-policy decision module of an autonomous agent that defines a remote operator policy decision node and a set of autonomous policy decision nodes downstream of the remote operator decision node. For example, a policy may be added to a set of selectable policies based on a binary decision by a remote operator and then evaluated (subsequently) by the multi-policy decision module along with the set of selectable policies.

[0115]

[0121] In one set of variations, S240 may include the step of determining a set of selectable policies based on a first and second set of inputs (e.g., using MPDM). In one variation, the set of selectable policies may be determined from a default set based on vehicle position estimation and / or scenarios (e.g., environmental representation, vehicle state estimation, etc.), the scenarios being used to refine the set of selectable policies.

[0116]

[0122] As an addition or alternative, S240 may include any other process and / or may be appropriately executed in another manner.

[0117] 4.5 Method - Steps to implement the action S250

[0123] Method 200 may include a step S250 which implements an action, and step S250 functions to perform the action for an autonomous agent.

[0118]

[0124] As an addition or alternative, method 200 may be performed without S250 and / or may include any other process.

[0119]

[0125] Preferably, S250 is executed multiple times at the discretion of the operator during the operation of the autonomous agent, in response to and based on S240 (e.g., sequentially, at a predetermined frequency, in a predetermined set of intervals, at random intervals, in response to a trigger, etc.). Additionally or alternatively, S250 may be executed in response to any other process of method 200, in parallel with any other process of method 200, and / or at any other time. Further additional or alternatively, method 200 may be executed without S250.

[0120]

[0126] S250 preferably includes the step of implementing the selection policy determined in step S240, but may also include, additionally or alternatively, the steps of determining the trajectory and / or route plan of the autonomous agent based on the selection policy (e.g., in an operations planner), operating the control of the autonomous agent based on the selection policy, and / or any other or all of any other processes. As an example, S250 may include the steps of determining vehicle commands (e.g., to various systems and / or ECUs installed in the vehicle) and controlling the vehicle in accordance with those commands.

[0121]

[0127] In specific examples, a remote operator may not have direct authorization for the execution of a vehicle's operational plan and / or vehicle actions / policies. For example, a remote operator may not have the ability to directly intervene to adjust the steering angle, operate the throttle, apply the brakes, or otherwise perform a vehicle action. In one exemplary example, a remote operator may approve a policy of gradually moving to the left side of the lane to facilitate subsequent policy decisions (e.g., determining whether it is safe to overtake a wide tractor traveling at 5 mph on a single-lane highway). However, once the vehicle initiates this policy, it may observe oncoming vehicles and react quickly (e.g., by returning to the center of the lane) without the intervention of the remote operator or without the communication delay caused by the direct involvement of the remote operator.

[0122]

[0128] In a first variation of Method 200, the Method includes: step S210 receiving a first set of inputs associated with its own agent, the first set of inputs including sensor information collected on its own agent and optionally, information from a set of one or more infrastructure devices; step S220 providing a set of outputs to a remote support platform, the set of outputs including the first set of inputs and optionally, any or all of a set of one or more warnings and / or notifications; step S230 receiving a second set of inputs from the remote support platform, the second set of inputs including inputs related to a set of policies available to its own agent; step S240 optionally determining its own actions based on the first and second sets of inputs; and step S250 optionally implementing the actions. As an addition or alternative, Method 200 may include any other processes that are performed in any appropriate order.

[0123]

[0129] In a specific example (as shown in Figures 3A to 3C), method 200 includes the steps of: detecting that the path of its agent is being obstructed by an obstacle (e.g., based on exceeding a time threshold) such as an obstacle unfamiliar to the vehicle; transmitting this sensor information from the agent (and optionally one or more infrastructure devices) and a proposed maneuver to the agent to a remote assistance platform; receiving input from a remote operator regarding the proposed maneuver; if the remote operator acknowledges the proposed maneuver, adding the proposed maneuver to a set of policies for consideration by the onboard computing system (e.g., the MPDM module of the onboard computing system); if the remote operator rejects the proposed maneuver, ceasing to add the proposed maneuver to a set of policies for consideration by the onboard computing system; selecting the proposed maneuver from a set of available policies (e.g., by the MPDM module); and operating its agent based on the selected maneuver.

[0124]

[0130] In a second variation (for example, shown in Figure 5), method 200 may include the step of autonomously controlling the vehicle using the vehicle's onboard computing system, which in each selection step of the selection cycle includes: determining a vehicle state estimate based on a set of sensor inputs; determining a set of selectable policies based on the vehicle state estimate; selecting a policy from the set of selectable policies; determining a vehicle command based on the policy; controlling the vehicle in accordance with the vehicle command; while autonomously controlling the vehicle, determining whether an event trigger has been satisfied based on the vehicle state estimate; providing a set of outputs to the remote assistance platform based on whether the event trigger has been satisfied; and updating the set of selectable policies to include a remote operator approval policy determined by the remote assistance platform. For example, the remote operator approval policy may include a set of waypoints provided by a remote operator (e.g., bypassing obstacles in the vehicle environment), which the vehicle may choose to follow (e.g., temporarily deviating from the target path, changing / updating the target path, etc.).

[0125]

[0131] In a third variation (for example, an example is shown in Figure 6), method 200 may include: a step of an autonomous agent determining a first set of inputs; a step of determining, based on the first set of inputs, the satisfaction of event triggers associated with dead ends along the target path; a step of providing a set of outputs to a remote assistance platform based on the satisfaction of event triggers; a step of receiving a second set of inputs from the remote assistance platform; a step of determining a set of selectable policies based on the first and second sets of inputs; a step of the autonomous agent selecting a policy from the set of selectable policies based on the target path; and a step of autonomously controlling the vehicle based on the policy.

[0126]

[0132] An alternative embodiment implements the above method and / or processing module in a non-temporary computer-readable medium that stores computer-readable instructions. Instructions may be executed by a computer-executable component integrated with the computer-readable medium and / or processing system. The computer-readable medium may include any suitable computer-readable medium such as RAM, ROM, flash memory, EEPROM, optical devices (CD or DVD), hard drives, floppy drives, non-temporary computer-readable medium, or any suitable device. The computer-executable component may include a computing system and / or processing system (e.g., including one or more co-occurring or distributed, remote or local processors) connected to the non-temporary computer-readable medium such as a CPU, GPU, TPUS, microprocessor, or ASIC, but instructions may also be executed by any suitable dedicated hardware device, as an alternative or addition.

[0127]

[0133] For the sake of brevity, details have been omitted, but preferred embodiments include any combination and permutations of various system components and various method processes, the method processes may be executed sequentially or simultaneously in any suitable order.

[0128]

[0134] Embodiments of a system and / or method may include any combination and permutation of various system components and various method processes, and one or more instances of the methods and / or processes described herein may be performed asynchronously (e.g., sequentially), concurrently (e.g., simultaneously, in parallel, etc.), or in any other suitable order by and / or using one or more instances of the systems, elements, and / or entities described herein. The following system and / or method components and / or processes may be used in addition to, in place of, or in other ways integrated with, all or part of the systems and / or methods disclosed in the above-mentioned applications, and the entirety of these applications is incorporated by this reference.

[0129]

[0135] Those skilled in the art will understand from the above-mentioned detailed description and from the drawings and claims that modifications and changes can be made to preferred embodiments of the present invention without departing from the scope of the present invention as defined by the following claims.

Claims

1. The process includes the step of autonomously controlling the vehicle using the vehicle's onboard computing system, wherein each selection step in the selection cycle is performed as follows: A step of determining the vehicle state estimate based on a set of sensor inputs, The steps include determining a set of selectable policies based on the vehicle state estimation, The steps include selecting a policy from the aforementioned set of selectable policies, The steps include determining a vehicle order based on the aforementioned policy, A step of controlling the vehicle in accordance with the vehicle command, While the vehicle is being autonomously controlled, Based on the vehicle state estimation, the step of determining whether the event trigger is satisfied, Based on the satisfaction of the aforementioned event trigger, the steps include providing a set of outputs to the remote support platform, A method comprising the steps of updating the set of selectable policies to include the remote operator approval policy determined on the remote support platform.

2. The method according to claim 1, further comprising the step of running a set of multiple simulations, wherein each policy is autonomously selected from the set of selectable policies based on a set of results arising from the set of multiple simulations.

3. The method according to claim 2, further comprising the step of estimating a set of state estimates for a set of dynamic objects in the vehicle environment based on the set of sensor inputs, wherein each of the set of simulations is performed based on the set of state estimates and the vehicle state estimate.

4. The vehicle state estimation includes vehicle position estimation, the method further includes a step of determining a scenario, and the set of selectable policies is The steps include determining a set of policies from a default set based on the vehicle position estimation, The method according to claim 1, comprising the step of improving the set of policies based on the aforementioned scenario, as determined by:

5. The method according to claim 1, further comprising the step of determining an autonomously unselectable policy using the onboard computing system, wherein the set of outputs provided to the remote support platform includes the autonomously unselectable policy.

6. The method of claim 5, further comprising the steps of: providing the set of outputs to the remote support platform; receiving remote operator approval from the remote support platform for the non-autonomously selectable policies; and updating the set of selectable policies in response thereto.

7. The method according to claim 1, wherein the policy is selected based on a target vehicle route, and the method further comprises the step of updating the target vehicle route based on a remote operator approved route received from the remote support platform.

8. The method according to claim 7, wherein the remote operator approval route includes a set of waypoints that bypass obstacles.

9. The method according to claim 8, wherein the set of outputs provided to the remote support platform includes a set of estimated object parameters associated with the fault.

10. The method of claim 8, wherein, after the step of updating the target vehicle route based on a remote operator-approved route, the step of autonomously controlling the vehicle includes the step of selecting an update policy from a set of selectable policies, wherein the update policy deviates from the remote operator-approved route.

11. The method according to claim 10, wherein the update policy is associated with returning to the target vehicle route.

12. The method according to claim 1, wherein the event trigger corresponds to a time threshold for the vehicle that is stopped due to a dead end along the target vehicle path.

13. The method according to claim 1, wherein the event trigger includes a probability threshold for the occurrence of a deadlock along the target vehicle path.

14. The method according to claim 1, wherein the remote operator approval policy is received via a wireless connection between the onboard computing system and the remote support platform with a latency of more than 50 milliseconds.

15. A method for remotely controlling an autonomous agent, The autonomous agent has the step of determining the first set of inputs, A step of determining whether an event trigger associated with a dead end along the target path is satisfied, based on the first set of inputs, Based on the satisfaction of the aforementioned event trigger, the steps include providing a set of outputs to the remote support platform, The steps include receiving a second set of inputs from the remote support platform, A step of determining a set of selectable policies based on the first set of inputs and the second set of inputs, The autonomous agent selects a policy from the set of selectable policies based on the target path. A method comprising the step of autonomously controlling the vehicle based on the aforementioned policy.

16. The method of claim 15, further comprising the step of updating the target path based on a second set of inputs, prior to the step of selecting the policy.

17. The method according to claim 16, wherein the second set of inputs includes a set of waypoints.

18. The method according to claim 15, wherein the set of outputs provided to the remote support platform includes a proposed policy that deviates from the target path, the second set of inputs includes remote operator approval of the policy proposal, and the set of selectable policies includes the proposed policy.

19. The method according to claim 18, wherein the selection policy is different from the proposed policy.

20. The second set of inputs includes a remote operator approval policy, the first set of inputs includes location estimation and scenarios, and the set of selectable policies is The steps include determining a set of policies from a labeled map based on the aforementioned location estimation, The steps include extending the set of policies based on the remote operator approval policy, The method according to claim 15, determined by the step of improving the set of policies to generate the set of selectable policies based on the scenario.

21. The method according to claim 15, wherein the second set of inputs includes a binary decision by a remote operator.

22. The method according to claim 15, wherein the policy is selected using a multi-policy decision module of the autonomous agent that defines a set of remote operator policy decision nodes and autonomous policy decision nodes downstream of the remote operator decision nodes.

Images