Method and system for expanding the navigation design universe of autonomous agents - Patents.com

Abstract

A system for extending an operational design domain (ODD) of an autonomous agent includes a decision-making platform (equivalently referred to herein as a decision-making architecture). A method for extending an operational design domain (ODD) includes determining a decision-making architecture for a first domain and adapting the decision-making architecture to a second domain. Additionally or alternatively, the method 200 can include implementing the decision-making architecture S300 and / or any other process.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 290,531, filed December 16, 2021, and U.S. Provisional Application No. 63 / 316,108, filed March 3, 2022, each of which is incorporated by reference in its entirety.

[0002] The present invention relates generally to the field of autonomous vehicles, and more particularly to new and useful systems and methods for extending the operational design domain of autonomous agents in the field of autonomous vehicles. [Background technology]

[0003] Making safe and effective decisions in autonomous vehicles is a challenging task. This type of decision making requires an understanding of the current environment around the vehicle, the evolution of this environment into the future, and safe and continuous progress towards predefined driving goals. Every decision must be continuously constrained by both the driving rules of the road and human driving practices, which, combined with the enormous number of possible interactions, makes decision making a very complex and challenging problem for autonomous systems. Even if the decision making system can be properly provided, when scaling the domain of deployment, the decision making system in conventional systems generalizes poorly, which can lead to poor performance, reliability issues, and safety risks of the vehicle. The conventional approach to alleviate this problem is to recollect a large amount of labeled or partially labeled data with the same variance as the test data, and then train the machine learning model on the new data. However, many factors prevent easy access to such data, which leads to high annotation costs, increased time to market, and limited domain scaling, among other limitations.

[0004] Thus, there is a need in the autonomous vehicle field to create improved and useful systems and methods for expanding the navigation design space of autonomous agents in the autonomous vehicle field. [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 is a schematic diagram of a system for extending the operational design domain of autonomous agents. [Diagram 2] FIG. 2 is a schematic diagram of a method for extending the operational design domain of an autonomous agent. [Diagram 3] FIG. 3 is a diagram of a schematic variation of a method for extending the navigation design domain of an autonomous agent. [Figure 4] FIG. 4 is a diagram illustrating a variation of the decision-making platform. [Diagram 5] FIG. 5 is a diagram illustrating the variation of a set of models in a decision-making platform. [Figure 6] FIG. 6 shows a schematic variation of a method for expanding the operational design domain of an autonomous agent. [Figure 7] FIG. 7 illustrates a schematic example of a portion of a method for expanding the operational design domain of an autonomous agent. [Figure 8] FIG. 8 illustrates a variation of a method for expanding the operational design space of an autonomous agent. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The following description of preferred embodiments of the invention is not intended to limit the invention to those preferred embodiments, but rather to enable any person skilled in the art to make and use the invention.

[0007] 1. Overview 1, a system 100 for extending an operational design domain (ODD) of an autonomous agent includes a decision-making platform (equivalently referred to herein as a decision-making architecture). Additionally or alternatively, system 100 may include and / or interface with any or all of a set of computing subsystems (e.g., a set of computers, a set of software modules and / or task blocks implemented in the set of computers, etc.) and / or a processing subsystem (e.g., a set of processors, a set of processing modules and / or task blocks implemented in the set of processors), a set of sensors, a control subsystem, an actuation subsystem (e.g., a drive-by-wire subsystem), a communications subsystem (e.g., for communicating with other agents, remote computing subsystems, remote operator platforms, etc.), an autonomous agent, and / or any other suitable components. Additionally or alternatively, system 100 may include and / or interface with any or all of the components described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2020, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, and U.S. Application No. 17 / 962,459, filed October 7, 2022, each of which is incorporated by reference in its entirety.

[0008] 2, a method 200 for expanding an operational design domain (ODD) includes determining a decision-making architecture for a first domain S100 and adapting the decision-making architecture to a second domain S200. Additionally or alternatively, method 200 may include implementing the decision-making architecture S300 and / or any other process. Additionally or alternatively, method 200 may include and / or interface with any or all of the processes described in any or all of U.S. Application Nos. 17 / 116,810, filed December 9, 2020, 17 / 125,668, filed December 17, 2020, 17 / 127,599, filed December 18, 2020, and 17 / 962,459, filed October 7, 2022, each of which is incorporated by reference in its entirety, or any other suitable processes performed in any suitable order. Method 200 may be performed by system 100 described above and / or by any other suitable system.

[0009] 2.Profit Systems and methods for extending ODDs of autonomous agents can provide several advantages over current systems and methods.

[0010] In a first variant, the technique provides the benefit of reducing the amount of time and / or data required to train one or more of the sets of models in the decision-making architecture of an autonomous agent when expanding the operational design domain (ODD) associated with the autonomous agent. In a preferred variant, the method and / or system leverages data and training from a previous training of the decision-making architecture to adapt it for use in other scenarios (e.g., routes, contexts, actions, etc.) through transfer learning and / or domain adaptation, which allows the autonomous agent to continually accelerate the learning of new tasks based on knowledge learned from previously demonstrated tasks, thereby reducing the amount of demonstration, and therefore the time, required to generate a decision-making architecture that is adequately equipped to handle the new task(s).

[0011] In a second variant, additional to the first variant, the technique provides the benefit of preventing and / or minimizing the effects of negative transfer (e.g., leading to decision-making errors made by the vehicle) or other degradation of performance when adapting the decision-making architecture to a new domain. In a particular set of examples, implementing a compact latent space representation that extracts features of the vehicle's environment, where the compact latent space representation serves as input to a modular set of models in the decision-making architecture, enables an efficient and accurate task mapping process to be implemented from source domain tasks to target domain tasks, which the inventors have discovered prevents or minimizes the effects of negative transfer.

[0012] In a third variant, additional or alternative to the variants described above, the present technology provides the benefit of enabling optimal selection of a particular model to be used as a starting point for determining a new model for a new operational design domain, which can increase the effectiveness of transferring learning to the new model, improve the performance of the new model, reduce the amount of data and / or time required to generate the new model, and / or provide any other suitable benefit. In an example set, the optimal selection of the particular model is performed in a manual process (e.g., a rule-based process), which allows the particular model that is most suitable (e.g., most closely related to) to determine the new model to be used. Additionally or alternatively, the models of the vehicle's decision-making architecture are arranged in a modular, sequential arrangement (e.g., a first set of models is evaluated prior to and informs the selection of a second set of models), which allows only the most relevant models to be identified and used in determining the new set of models (e.g., those most closely related to the new context based on shared features, those most closely related to the new action based on shared features, etc.). This may be in contrast to conventional systems and methods that utilize end-to-end deep learning models, for example, which often result in negative transfer due to a lack of concrete and / or measurable similarity between the old model use cases (source model) and the new model use cases (target model).

[0013] In a fourth variant, additional or alternative to the variants described above, the technology provides for expanding the set of routes (e.g., fixed routes) that the vehicle is configured to traverse (e.g., autonomously traverse) (e.g., is capable of, is eligible to, is authorized to, etc.), expanding the types of routes (e.g., fixed routes, dynamically determined routes, etc.) that the vehicle is configured to traverse (e.g., route length, route location, starting point(s) of the route(s), destination(s) of the route(s), etc.), expanding the number and / or type(s) of contexts (e.g., scenarios) to which the vehicle can respond (e.g., type of road, number of lanes in the road, type of intersection, type of traffic sign, type of zone [e.g., school zone, residential zone, highway, freeway, etc.], etc.). expanding the number and / or type(s) of actions that the vehicles can perform (e.g., merging, slowing down, parking, loading, unloading, lane changing, etc.); expanding the number and / or type of trajectories that the vehicles can perform; expanding the number and / or type of driving conditions in which the vehicles can operate (e.g., weather conditions, lighting conditions, traffic conditions, pedestrian concentration, cyclist concentration, etc.); expanding the performance conditions in which the vehicles can operate reliably (e.g., sensor performance, number and / or type of sensors, sensor degradation levels, etc.); expanding the use cases in which the vehicles may be used (e.g., delivering goods, carrying passengers, dynamic route use cases, etc.); and / or otherwise expanding the operating design space of the vehicles.

[0014] Additionally or alternatively, the systems and methods may provide any other benefits.

[0015] 3. System 100 1, a system 100 for extending the operational design domain (ODD) of an autonomous agent (equivalently referred to herein as an autonomous vehicle, ego-agent, ego-vehicle, or vehicle) preferably includes a decision-making platform (equivalently referred to herein as a decision-making architecture). Additionally or alternatively, system 100 may include and / or interface with any or all of a set of computing subsystems (e.g., a set of computers, a set of software modules and / or task blocks implemented in the set of computers, etc.) and / or a processing subsystem (e.g., a set of processors, a set of processing modules and / or task blocks implemented in the set of processors), a set of sensors, a control subsystem, an actuation subsystem (e.g., a drive-by-wire subsystem), a communications subsystem (e.g., for communicating with other agents, remote computing subsystems, remote operator platforms, etc.), an autonomous agent, and / or any other suitable components. Additionally or alternatively, system 100 may include and / or interface with any or all of the components described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2020, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, and U.S. Application No. 17 / 962,459, filed October 7, 2022, each of which is incorporated by reference in its entirety.

[0016] System 100 includes and / or implements a decision-making architecture that functions to implement decision-making for an autonomous vehicle through any or all of high-level action and / or behavior determination, path planning, maneuver planning, trajectory planning, and / or any other decision-making process for operating an autonomous agent.

[0017] System 100 may additionally or alternatively function to control the autonomous agent, perform cognition for the autonomous agent, perform predictions for the autonomous agent, and / or otherwise enable the autonomous agent to operate (e.g., navigate safely, navigate optimally, etc.). Additionally or alternatively, system 100 may further function to enable updates to the decision-making architecture and / or any other processes.

[0018] At least a portion of the decision-making architecture used by the autonomous agent is preferably trained (equivalently referred to herein as learned). In a preferred variant, for example, the decision-making architecture includes one or more sets of trained models, such as, but not limited to, machine learning models, deep learning models (e.g., neural networks, deep neural networks, convolutional neural networks, etc.), and / or any combination. In a preferred variant (e.g., described below), the decision-making architecture includes one or more sets of trained micro-models (equivalently referred to herein as learning modules, learning models, models, etc.) that are selected and evaluated in a modular manner depending on the particular scenario (e.g., context) encountered by the autonomous agent and, optionally, outputs from previously evaluated micro-models. Variations of this decision-making architecture are described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2020, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, and U.S. Application No. 17 / 962,459, filed October 7, 2022, each of which is incorporated by reference in its entirety. Additionally or alternatively, the decision-making architecture may include and / or implement any other trained process.

[0019] The modular and / or sequential nature of the trained models can provide numerous benefits in efficiently and optimally expanding the vehicle's operational design space. In a preferred variant (e.g., as described further below), for example, each of the first set of models is associated with a specific context (and optionally with a specific fixed route), each of the second set of models is associated with a specific action (e.g., determined based on evaluating one or more of the first set of models), and the models selected for evaluation in the first set and the second set are selected by a rule-based process (e.g., a pre-determined mapping, a look-up table, etc.). In contrast to an end-to-end decision-making architecture, for example, models from the first set and the second set can be selected in a logical and explainable manner to determine a new model (e.g., in a task mapping process described below). For example, if a new route is being added and the new route includes a new context (e.g., scenario), the model from the first set that is most relevant to the new context may be selected and used (e.g., used exclusively) as the starting point (e.g., providing an initial set of weights) for the new context model. This can also prevent negative transfer effects (e.g., poor performance and / or unexpected outputs from the new model) that may occur when the source model (the model used to train the new model) is not explainably related to the use case of the new model.

[0020] Each of the models is preferably in the form of and / or includes a machine learning model, and more preferably in the form of one or more neural networks and / or network models (e.g., deep Q-learning networks, convolutional neural networks [CNN], inverse reinforcement learning [IRL] models, reinforcement learning [RL] models, imitation learning [IL] models, etc.), but may additionally or alternatively include any other suitable models, algorithms, decision trees, lookup tables, and / or other tools.

[0021] Each of the models is further preferably trained with inverse reinforcement learning, which serves to determine a reward function and / or an optimal driving policy for each of the context-aware learning modules. The output of this training is further preferably a compact fully connected network model representing the reward function and optimal policy for each learning module. Additionally or alternatively, the learning modules may be suitably trained and / or implemented otherwise (e.g., with reinforcement learning, etc.).

[0022] In a first set of variations, the decision-making architecture includes and / or defines a first subset of models that function to implement a first portion of the autonomous agent's decision-making (e.g., the deep decision networks shown in Figures 4 and 5) and a second subset of models that function to implement a second portion of the decision-making (e.g., the deep trajectory network shown in Figure 4). Additionally or alternatively, the decision-making architecture may include any other subset of models, a single subset of models, and / or any other combination of models.

[0023] Each of these subsets of models preferably includes multiple models (e.g., multiple deep decision networks in a first subset, multiple deep trajectory networks in a second subset), equivalently referred to herein as a modular set of models, and a single model from each of the subsets is preferably selected at each point in time during vehicle decision-making (e.g., at a predetermined frequency during operation of the autonomous vehicle, continuously during operation of the autonomous vehicle, etc.). Additionally or alternatively, any or all of the subsets may include a single model, the set of models may include any other subset of models, multiple models may be selected within any or all of the subsets, and / or the decision-making architecture may be otherwise configured and / or organized.

[0024] The subsets of models are further preferably related to an order of evaluation and / or other interfacing (e.g., communication) between the subsets, and outputs from one or more subsets are used in selecting a model from the subset to be evaluated next. In a set of variations, for example, an output(s) of a model selected in a first subset of models (e.g., a selected deep decision network) is used to select which model from a second subset of models is evaluated next (e.g., a selected deep trajectory network). Additionally or alternatively, the subsets of models may be related, unrelated, and / or evaluated differently as appropriate.

[0025] In a particular set of examples, a first subset of models includes a set of deep decision networks (DDNs), one or more (e.g., single, multiple, etc.) of which are selected based on the vehicle's current context and are responsible for making decisions regarding actions the autonomous agent should take during the current planning cycle (e.g., as described in U.S. Application No. 17 / 125,668, filed December 17, 2020, which is incorporated herein by reference in its entirety). A second subset of models includes a set of deep trajectory networks (DTNs) that are selected, optimized, and / or safely constrained based on specific actions. Based on the actions determined by the selected DDNs, a corresponding DTN is selected and used (e.g., with a localized view around the vehicle) to plan a safe, valid, and natural trajectory for the vehicle to follow.

[0026] The input to any or all of the set of models preferably includes environmental information relevant to the autonomous agent, thereby enabling the agent to utilize awareness of its environment in its decision-making. This awareness preferably includes information from the current time step as well as from previous time steps, but may alternatively include any other information. In a preferred variant, the environmental information is refined into a smaller latent space representation. This can simplify the learning process of any or all of the set of models, as well as enable efficient and logical mappings to be determined in S220 in accordance with updating the decision-making architecture through transfer learning.

[0027] The latent space representation is further preferably used to train and / or refine the training of any or all of the set of models. Additionally or alternatively, the latent space representation (e.g., features of the latent space representation) may be used to determine which models to use as source models in determining a new target model, the models may be trained on other data, the method may be implemented without the latent space representation, and / or the method may be implemented as otherwise suitable.

[0028] The latent space representation (equivalently referred to herein as an abstract space representation) is preferably configured to represent the autonomous agent's environment (or a targeted subset of the environment) with information and / or features that are comparable between different use cases (e.g., contexts, actions, routes, etc.), thereby allowing learning to be easily and efficiently transferred between models. This may be in contrast to and / or advantageous compared to, for example, models and / or implementations that use raw data (e.g., raw sensor data as input), since raw data is not easily consumable or transferable between models. In a preferred variant, for example, the latent space representation determines a set of features that define the environment, which are preferably abstract / extracted (e.g., determined using an algorithm, model, trained algorithm and / or model, etc.) and further preferably order-independent (e.g., allowing comparison between latent space representations for different contexts, actions, routes, etc.). In an example, for example, the latent space representation generates an order-independent representation for how many objects are in the autonomous agent's environment, what features (e.g., extracted features, cost values, or other metrics, etc.) are associated with the objects and / or environment, and / or any other information. In certain specific examples in which features of the latent space representation are extracted (e.g., not qualitatively meaningful), for some contexts and / or actions, curbs may be relevant to the vehicle's motion, but in other contexts and / or actions, similar obstacles to the vehicle's motion (e.g., barricades) exist, and the latent space representation extracts these objects such that learnings from navigating curbs can be effectively utilized and used in learning how to navigate barricades.

[0029] Additionally or alternatively, the latent space representation may include any or all of the following information, which is preferably standardized (e.g., normalized, processed to non-unit values, extracted, processed according to a machine learning algorithm, etc.) into the latent space representation format: vehicle position information (e.g., latitude, longitude, altitude, etc.), vehicle movement information (e.g., speed, acceleration, etc.), chassis information (e.g., fuel level, tire pressure, etc.), external input (e.g., a map), cognitive input (e.g., location(s) of other vehicles or other objects, movement information related to other vehicles or other objects, etc.), predicted information (e.g., predicted trajectories of other vehicles, predicted future positions or future movements of other vehicles, etc.), decisions from a previous decision-making process (e.g., selected context for the vehicle based on the vehicle's location relative to a map, etc.), and / or any other information.

[0030] A potential benefit of the addition and / or replacement of the latent space representation is that it allows the same basic architecture (e.g., structure) to be implemented in any or all of the models (e.g., all of the first set of models, all of the second set of models, all models, etc.), which allows learning to be more effectively and efficiently transferred between models. For example, all of the first set of models (e.g., deep decision networks, context-based models, etc.) and / or any or all of the second set of models can have the same model architecture (e.g., number of neural network layers, type of neural network, size of neural network, model input type and / or model format, etc.), which allows the method to transfer learning between models faster. Additionally or alternatively, any or all of the models can have different architectures and / or other differences.

[0031] In a particular set of examples, each of the first set of models has the same architecture (e.g., number of neural network layers, neural network type, etc.) despite being associated with different contexts, and the models differ in some or all of their weight values. Additionally or alternatively, each of the second set of models has the same architecture (e.g., number of neural network layers, neural network type, etc.) despite being associated with different actions, and the models differ in some or all of their weight values.

[0032] Additionally or alternatively, the models may be otherwise suitably designed and / or related.

[0033] The latent space representation preferably differs between the first set of models and the second set of models, and the latent space representation utilized by the second set of models configured to be implemented after the corresponding models of the first set is refined (e.g., localized, targeted, etc.) based on the output of the corresponding models of the first set. In a set of variations, for example, each of the first set of models (a set of deep decision networks) is associated with a particular context type (e.g., scenario, road shape, road features, etc.), and optionally within a particular fixed route, and each of these models uses the vehicle's current context (e.g., taken from a predefined set of labels in a map, determined based on the vehicle's location, etc.) and a latent space representation of its complete environment (e.g., all of the environment within the field of view of its sensor stack) to select an action for the vehicle, which action informs the selection of one of the second set of models (a set of deep trajectory networks). This selected model of the second set then uses a latent space representation that is informed (e.g., constrained) based on the selected action such that the representation of the vehicle's environment for this model is localized to only those areas that are important for the vehicle to know in order to perform actions through determining the trajectory the vehicle should follow (e.g., only the view in front of the vehicle if the vehicle is moving forward, only the view behind the vehicle if the vehicle is backing up, etc.).

[0034] Additionally or alternatively, the models may use the same latent space representation, different latent space representations, and / or any combination of latent space representations.

[0035] In a preferred set of variants (e.g., shown in FIG. 5), for example, any or all of the first subset of models and / or the second subset of models take as input any or all of the following information: information related to the set of detected dynamic objects (e.g., their representations) including any or all of their current positions, sizes, previous paths, and predicted paths into the future (alternatively, the system can perform self-prediction of dynamic object motion), information related to the set of all static objects and their current states, a map (e.g., defining a context for the vehicle based on the vehicle's position), routing information, and the current state of the autonomous agent (e.g., position, orientation, pose, etc.). Additionally or alternatively, any other information may be used to determine the latent space representation. The latent space representation is preferably determined using a model, more preferably a trained model (e.g., separate and distinct from the first and second subsets of models), which serves to output an effective latent space representation (e.g., providing order invariance for the objects serving as inputs). Additionally or alternatively, the latent space representation may be determined otherwise as appropriate.

[0036] In a particular set of examples, the static and dynamic objects returned by the system's perception module (along with their predicted future paths) serve as input to a neural network, which generates a latent space representation that provides order invariance for the objects. This data is then combined with the map, routing information, and vehicle state and used as input to a second network that represents the entire input space as the most effective latent space representation.

[0037] The decision-making architecture further preferably implements one or more programmed (e.g., rule-based, coded, hard-coded, etc.) processes, such as a programmed selection of trained models to be evaluated during decision-making (e.g., as shown in FIG. 4, as shown in FIG. 5, etc.). In a preferred variant, for example, the decision-making architecture (e.g., for motion planning, for trajectory generation, etc.) implements a programmed selection of its trained components (e.g., trained micro-models, trained micro-model task blocks, trained algorithms and / or models, etc.).

[0038] The combination of both trained and programmed features of an autonomous agent's decision-making architecture (e.g., as shown in FIG. 4, as shown in FIG. 5, etc.) can have numerous benefits, such as in minimizing and / or eliminating drawbacks incurred from exclusively implementing either a trained approach or a programmed approach. These drawbacks and / or complications can include, for example, unnatural decisions and / or movements performed by the autonomous agent (e.g., when using only a programmed movement planner), an exhaustive and specific list of scenarios to be programmed (e.g., hard-coded) (e.g., when using only a programmed movement planner), no safety guarantees and / or safety integration (e.g., resulting from implementing movement planning in an end-to-end trained manner), a requirement to capture all possible scenarios that the agent may encounter, and / or any other drawbacks.

[0039] In a preferred variant, for example, the decision-making architecture captures the flexibility of machine learning techniques while ensuring safety across ODDs associated with autonomous agents and a practical and efficient learning framework for scalability. In an example, for example, each of the set of models is a trained neural network, but model selection from the associated modular set is performed by a rule-based (e.g., programmed) process.

[0040] Alternatively, the decision-making architecture may be fully trained (e.g., including an end-to-end deep learning model), fully programmed, and / or distributed between being separately trained and programmed.

[0041] The decision-making architecture may optionally be configured for an operational design domain (ODD) associated with one or more particular use cases for the autonomous agent. In some variations, for example, the decision-making architecture is configured for a limited ODD in the form of a fixed-route architecture (e.g., for use in making deliveries with the autonomous agent, for use cases without passengers, etc.). Additionally or alternatively, the autonomous agent and associated decision-making architecture may be configured for any other use case.

[0042] In one set of examples, ODD may further relate to the transportation of goods (e.g., distribution, business-to-business [B2B] distribution, transportation of commercial goods, transportation of non-human objects, etc.), such as between any or all of a distribution center, a retail center (e.g., a store), a warehouse or goods storage location, a manufacturer, and / or any other location. In certain examples, for example, any or all of the fixed routes may involve loading and / or unloading locations (e.g., loading docks) associated with any or all of these locations.

[0043] Additionally or alternatively, the vehicle ODD(s) may be otherwise suitably configured.

[0044] Additionally or alternatively, the decision-making architecture may include any other components and / or features and / or be otherwise suitably configured.

[0045] In a first variant of system 100, the system includes a decision-making architecture developed for fixed-route ODD using a context-aware data-driven modular learning system (e.g., as described in U.S. Application No. 17 / 125,668, filed December 17, 2020, the entire contents of which are incorporated herein by reference).

[0046] In a particular example, the system allows for the use of deep learning micro-models optimized and trained for a particular route in a particular context for a particular action, which can be reused (e.g., through transfer learning, through retraining with a smaller dataset, through retraining with a more targeted dataset, reused, repurposed) for use on different routes in a similar context for a similar set of actions. The proposed system is further preferably designed to ensure that such transformations enable transfers that reduce target training time without producing negative transfers (NT), but may additionally or alternatively be configured differently. This can be done by designing a decision-making architecture system with multiple subsets of models whose intended functionality is restricted to specific explainable tasks, and inputs for such models are transformed into a latent feature space to enable a common space in which variance mismatch of different source domain data and target domain data can be minimized. A first subset of models uses the current context of the vehicle and, optionally, a full representation of the environment around the vehicle to select the action the vehicle should undertake. Vehicle actions may include, among others, "stop behind a vehicle," "yield to a vehicle," or "merge onto a road." A second subset of models is selected, optimized, and safely constrained based on the specific actions. This subset of models may optionally use a localized view (e.g., of an environment representation) around the vehicle to plan a safe, valid, and natural trajectory for the vehicle to follow.

[0047] Additionally or alternatively, system 100 may include any other components.

[0048] 4. Method 200 2, a method 200 for expanding an operational design domain (ODD) includes determining a decision-making architecture for a first domain S100 and adapting the decision-making architecture to a second domain S200. Additionally or alternatively, method 200 may include implementing the decision-making architecture S300 and / or any other process. Additionally or alternatively, method 200 may include and / or interface with any or all of the processes described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2020, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, and U.S. Application No. 17 / 962,459, filed October 7, 2022, each of which is incorporated by reference in its entirety, or any other suitable processes performed in any suitable order.

[0049] Method 200 functions to enable the autonomous agent's decision-making architecture to efficiently adapt to new scenarios (e.g., new contexts, actions, environments, etc.), such as when expanding the number of routes, the type of route, the tasks and / or contexts and / or actions within the route, the conditions associated with the route (e.g., daytime vs. nighttime, inclement weather, etc.), non-fixed route scenarios, and / or other features associated with expanding the autonomous agent's Operational Design Domain (ODD). This can enable any or all of minimizing and / or reducing the time required to adapt the decision-making architecture, minimizing and / or reducing the amount of data required to adapt the decision-making architecture, and / or any other results. Additionally or alternatively, method 200 can function to optimize the decision-making architecture for various domains, enable the decision-making architecture to be reliably and / or safely adapted to different domains (e.g., without negative transfer effects, etc.), and / or otherwise provide a robust yet adaptable decision-making architecture for use by the autonomous agent.

[0050] In examples, for example, the method may adapt the vehicle's decision-making architecture for use in any or all of: different contexts, different actions, different context-action pairs, different routes (e.g., same context and / or action on different routes, different context and / or action on different routes), different weather conditions, different traffic conditions, different hardware and / or vehicle types (e.g., different sensor types and / or sensor numbers and / or sensor placements), the occurrence of an anomaly (e.g., sensor degradation) in the vehicle's hardware and / or software (e.g., as indicated by an uncertainty value associated with a vehicle input exceeding a predetermined threshold), and / or any other use.

[0051] In a preferred variant, the method 200 functions to enable scaling of an agent's ODD through transfer learning while minimizing and / or preventing the adverse effects of domain scaling. For example, learning from demonstrations (examples and / or interactions) can be implemented to operate a vehicle that mimics a human driver. However, even for a single task (e.g., making a protected right turn), such an approach generally requires a large number of demonstrations. For scalability, such a system must learn many tasks and scenarios through demonstrations, a process that would be quite burdensome for system developers if each task were learned separately. The method 200 can optionally function to utilize transfer learning from previously learned scenarios and / or tasks, which allows the agent to continually accelerate the learning of new tasks based on knowledge learned from previously demonstrated tasks, reducing the amount of demonstrations, and therefore the time, required for learning. A further complication in implementing transfer learning is that its effectiveness is not guaranteed, and the following may add complexity: the learning tasks in the two domains are not related or similar, the source domain data and the target domain data distributions have differences (e.g., significant differences), and a suitable model may not be applicable to both domains. In these cases, negative transfer (NT) may occur, and introducing source domain data / knowledge undesirably reduces learning performance in the target domain. The inventors have discovered that a solution to the above problem is transfer learning (TL), or domain adaptation (DA), which attempts to utilize data or knowledge from a related domain (called the source domain) to enable learning in a new domain (called the target domain). In machine learning applications, TL can function to improve the ability of a model to generalize in a target domain, which typically has zero or very few labeled data.

[0052] Additionally or alternatively, the method 200 may perform any other function and / or provide any other benefit.

[0053] Method 200 is preferably performed and / or utilized by system 100 described above, but may additionally or alternatively be performed in accordance with any other suitable system(s).

[0054] Method 200 is preferably implemented at least in part in a computing and / or processing system on-board the agent, in a computing and / or processing system remote from the agent (e.g., in a cloud-based computing system and / or server), in another location, and / or in any combination of locations. In a preferred set of variations, for example, the decision-making architecture is determined and / or updated in the remote computing system and then transmitted to the agent, stored, and utilized on-board the agent. Additionally or alternatively, method 200 may be implemented in and / or by any other suitable components.

[0055] 4.1 Method—Determining a Decision-Making Architecture for a First Domain S100 Method 200 can include determining a decision-making architecture for the first domain, which serves to determine an initial decision-making architecture for the autonomous agent. Additionally or alternatively, S100 can serve to determine a decision-making architecture that can be efficiently and / or quickly adapted to expand into new domains for the autonomous agent, and / or can perform any other suitable functionality.

[0056] S100 is preferably performed initially in method 200 (e.g., prior to deployment of the autonomous agent, prior to performance of any or all remaining processes of the method, prior to S200, etc.), but may additionally or alternatively be performed multiple times during method 200 (e.g., when new inputs are received, when the decision-making architecture is constantly / continuously revised and / or updated, etc.), and / or at any other time. Alternatively, method 200 may be performed in the absence of S100.

[0057] Domain, as used herein, preferably refers to an operational design domain (ODD) associated with an autonomous agent, which specifies the operational domain in which the decision-making architecture is configured and / or designed to at least operate (e.g., operate safely, operate reliably, etc.). In variations involving fixed-route use cases, for example, the domain may include a fixed route along which the autonomous agent has been trained and / or validated to travel.

[0058] Additionally or alternatively, a region can include and / or specify any or all of a particular set of contexts (e.g., single-lane roads, multi-lane roads, parking lots, residential zones, school zones, highway roads, one-way roads, two-way roads, etc.), a particular set of actions and / or behaviors (e.g., within a particular context, independent of a particular context), environmental conditions (e.g., weather conditions, lighting conditions, timing conditions, traffic conditions, etc.), infrastructure conditions (e.g., road quality [e.g., smooth, potholed, asphalt, dirt, etc.], whether a road has adjacent sidewalks, etc.), and / or any other features or conditions (e.g., with a set of fixed routes, independent of a set of fixed routes, etc.). Additionally or alternatively, a region can include and / or point to any other information.

[0059] In some examples, for example, expanding the domain of an autonomous vehicle may include any or all of: adding new routes, adding new contexts, adding new actions for the vehicle to perform, changing the type of data used to train and / or update the training of any or all of the set of models (e.g., training on simulation data in addition to real-world sensor data), changing the driving habits of the vehicle(s) (e.g., adapting the vehicle to drive on the left side of the road instead of the right side), adding new driving conditions (e.g., weather conditions, traffic conditions, etc.) in which the vehicle can operate, adding new use cases in which the vehicle can operate (e.g., making deliveries now at a residence instead of going to a loading dock to make a delivery), and / or otherwise modifying or expanding the use of the vehicle.

[0060] In a preferred variation of method 200, the first region refers to an initial region, where the second region (e.g., as described below) extends on top of the first region (e.g., includes additional routes and / or contexts and / or actions, etc.) (e.g., includes the first region with additional scenarios for the autonomous agent). Additionally or alternatively, the second region may include a portion of the first region, be separate and / or unrelated to (e.g., does not overlap with) the first region, partially overlap with the first region, be of equal or lesser extent than the first region, and / or be otherwise configured.

[0061] In a set of examples, the first domain includes a predefined set of fixed routes on which a model of a decision-making architecture (e.g., DDN, DTN, etc.) was trained (e.g., comprehensively trained). This can include, for example, a set of predefined fixed routes from which data was collected and used to build (e.g., train, determine basic architectural features thereof, etc.) an initial set of models (e.g., an initial first subset of models, an initial second subset of models, etc.) for decision-making in an autonomous vehicle.

[0062] Additionally or alternatively, the first region may include and / or be defined based on any other information, such as, without limitation, a predetermined geographic area, a use case associated with the autonomous vehicle (e.g., delivering goods, transporting passengers, etc.), a set of road features (e.g., number of lanes, road surface, road type, etc.), zones / zoning types associated with the regions of roads in the first region (e.g., residential zones, commercial zones, etc.), and / or any other information.

[0063] Additionally or alternatively, S100 may include any other suitable processes (eg, those described below, in addition to or in place of those described below, etc.).

[0064] 4.11 Method—Collecting a First Set of Inputs S110 S100 may optionally include collecting S110 a first set of inputs (e.g., as shown in FIG. 3), which serves to receive data for determining (e.g., training) any or all of the decision-making architectures. Additionally or alternatively, the first set of inputs may be used for any or all of evaluating and / or validating any or all of the decision-making architectures, testing any or all of the decision-making architectures, performing task mapping between domains in S220, updating the decision-making architectures in S230, implementing the decision-making architectures in S300, and / or as otherwise suitable.

[0065] S110 is preferably performed initially, and more preferably multiple times (e.g., repeatedly, continuously, at a predetermined frequency, at a random set of intervals, etc.) during operation of the vehicle within a first region (e.g., having a first set of fixed routes), but may additionally or alternatively be performed at any suitable time during method 200.

[0066] The first set of inputs preferably relates to a first domain (e.g., as described above) associated with the autonomous agent, equivalently referred to herein as the source domain, but may additionally or alternatively relate to another domain (e.g., a second domain), a combination of domains, no domains (e.g., among others), and / or any other domain.

[0067] In a preferred set of variations, for example, the first set of inputs includes information (e.g., sensor data) collected while the autonomous vehicle traverses one or more of a predetermined set of fixed routes that define the first region.

[0068] The first set of inputs preferably includes data collected at a set of sensors (e.g., sensors onboard the autonomous agent, sensors in the autonomous agent's environment, sensors onboard the aggregated set of autonomous agents, sensors onboard the aggregated set of non-autonomous agents, etc.), such as data collected from any or all of cameras, radar sensors, lidar sensors, audio sensors, location sensors (e.g., GPS sensors), motion and / or orientation sensors (e.g., accelerometers, speedometers, gyroscopes, audio sensors (e.g., microphones), and / or any other sensors. Additionally or alternatively, the first set of inputs may include historical information (e.g., previously collected at one or more agents), information from one or more databases (e.g., maps, labeled maps, etc.), simulated data, and / or any other data.

[0069] The first set of inputs (and / or a portion of the first set of inputs) is preferably used as training data for the decision-making architecture, but may additionally or alternatively be used as test data (equivalently referred to herein as testing data), evaluation data (e.g., in performing hyper-parameter tuning of any or all of the set of models of the decision-making architecture), validation data, and / or any other data.

[0070] The first set of inputs may additionally or alternatively be used in defining and / or characterizing a domain associated with the autonomous agent (e.g., for use in perception, prediction, localization, etc.) used in implementing the decision-making architecture, and / or may be otherwise used and / or determined as appropriate.

[0071] Additionally or alternatively, S110 may include any other suitable processes.

[0072] 4.12 Methods - Training the Decision-Making Architecture S120 S100 preferably includes training a decision-making architecture S120 (e.g., as shown in FIG. 3), which serves to create an initial iteration of a decision-making architecture for operation of the autonomous agent (e.g., according to the first domain). Additionally or alternatively, S120 may serve to generate a decision-making architecture that can be efficiently adapted and improved in subsequent processes of method 200, and / or may perform any other functions.

[0073] S120 is preferably performed in response to and based on S110, but may additionally or alternatively be performed prior to S110, based on a portion of S110 (e.g., a subsequent iteration of S110 is used to refine and / or update and / or retrain the decision-making architecture), multiple times during method 200, and / or at any other time(s) during method 200.

[0074] The decision-making architecture trained in S120 is preferably determined for a first domain and based on a first set of inputs (e.g., a portion of the first set of inputs, all of the first set of inputs, etc.) Additionally or alternatively, the decision-making architecture may be trained for any other domain.

[0075] S120 preferably includes training any or all of the set of models of the decision-making architecture (e.g., determining an architecture for each of the set of models, determining learned model parameters for each of the set of models, etc.), but may additionally or alternatively include any or all of testing, evaluating (e.g., hyper-parameter tuning), and / or validating the models of the decision-making architecture.

[0076] Training preferably includes determining values ​​for any or all of the parameters associated with the decision-making architecture, such as parameters (e.g., weights) that define any or all of a set of micro-models (e.g., deep decision networks, deep trajectory networks, etc.) to be evaluated during decision-making of the autonomous agent. This preferably includes determining at least any or all of the learned model parameters (e.g., node weights, parameters that optimize a loss function associated with the model(s) and / or overall architecture, etc.) and, optionally, any or all of the architecture of each model (e.g., number of layers, organization of layers, etc.). Additionally or alternatively, training may include determining any or all of the parameters (e.g., constraints, weights, learning rates, etc.) that define the model architecture, equivalently referred to herein as hyperparameters (e.g., parameters that are not learned), such as through one or more hyperparameter tuning processes (e.g., grid search tuning, random search tuning, parameter sweep tuning, Bayesian optimization process, gradient-based optimization process, evolutionary optimization process, etc.). Additionally or alternatively, training may include testing, evaluating, optimizing, and / or otherwise determining any or all of the models of the decision-making architecture.

[0077] In some variations, any or all of the models in the decision-making architecture are optimized for one or more of a particular action, a particular context, and / or a particular route associated with the domain. In a hyper-optimization use case, for example, any or all of the micro-models (e.g., a first subset of models) in the decision-making architecture have parameters (e.g., weights) that are optimized for a particular action in a particular context for a particular route (e.g., a fixed route). Additionally or alternatively, any or all of the models may be optimized, not optimized, and / or any combination, possibly for any of this information.

[0078] At least a portion of the decision-making architecture is preferably trained with inverse reinforcement learning. Additionally or alternatively, any or all of the models may be trained with reinforcement learning, imitation learning, and / or any other type of training / learning. Additionally or alternatively, training any or all of the models may include and / or interface with any or all of the processes described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2020, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, and U.S. Application No. 17 / 962,459, filed October 7, 2022, each of which is incorporated by reference in its entirety.

[0079] In a preferred set of variations, S120 includes training each of a first subset of models (e.g., deep decision networks) and each of a second subset of models (e.g., deep trajectory networks), such as those described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2020, U.S. Application No. 17 / 125,668, filed December 17, 2020, and U.S. Application No. 17 / 127,599, filed December 18, 2020, each of which is incorporated by reference in its entirety.

[0080] In a particular example, S120 includes training each of the first subset of models based on a context associated with each of the first subset of models, and training each of the second subset of models based on one and / or more actions associated with each of the second subset of models.

[0081] S120 may additionally or alternatively include determining any or all of the programmed processes and / or features of a decision-making architecture (eg, as described above).

[0082] Additionally or alternatively, S100 may include any other suitable processes.

[0083] 4.2 Method—Determining a Decision-Making Architecture for a Second Domain S200 Method 200 includes determining a decision-making architecture for a second domain S200, which functions to adapt the decision-making architecture to the second domain, such as the expanded domain. Additionally or alternatively, S200 may function to enable the decision-making architecture to be adapted efficiently and / or with a minimal set of data, and / or may perform any other suitable functionality.

[0084] S200 is preferably performed in response to and / or based on S100, but may additionally or alternatively be performed multiple times, in response to other processes, in response to a trigger, and / or at any other time.

[0085] 4.21 Method - Collecting a second set of inputs S210 S200 preferably includes collecting S210 a second set of inputs (e.g., as shown in FIG. 3), which serves to receive information for performing S200 and / or any or all of the remaining processes of method 200, such as performing task mapping in S220, updating the decision-making architecture in S230, implementing the decision-making architecture in S300, and / or any other process. Additionally or alternatively, the second set of inputs may be used in any of the processes described above, and / or S210 may perform any other suitable function.

[0086] S210 may be performed prior to S220, in response to S220, prior to and / or in response to any other processes of method 200, during operation of one or more autonomous agents, independent of any or all of the processes of method 200, contemporaneously with (e.g., overlapping with, partially overlapping with, simultaneously with, etc.) S110, multiple times (e.g., continuously, at a predetermined frequency, at a set of random intervals, etc.), and / or at any other time.

[0087] The second set of inputs (equivalently referred to herein as auxiliary data) is preferably associated with a second region (equivalently referred to herein as a target region), which is further preferably an extension of and / or distinct from (e.g., not overlapping with, partially overlapping with, overlapping with, etc.) the first region (e.g., including additional context, including additional actions, including additional routes, including additional environmental conditions, any combination, etc.) Alternatively, the second set of inputs may be associated with the first region, multiple regions, and / or any other region.

[0088] The second set of inputs preferably includes the same types of data as the first set of inputs (e.g., as described above), such as sensor data from the same or similar sensors, simulation data from the same or similar simulation subsystems, and / or any other data, but may additionally or alternatively include different types of data, any other data types, and / or any combination of data.

[0089] In a preferred set of variations, the second set of inputs includes data relating to a context and / or action and / or set of routes and / or environmental conditions not included in the first domain. Additionally or alternatively, the second set of inputs may include data relating to the first domain (e.g., categorized within the first domain), data from a combination of domains, and / or any other data.

[0090] The second set of inputs further preferably includes data specific to (and collected from) the particular use case associated with the new model being developed in S200. For example, each new model is preferably refined (e.g., retrained, updated, etc.) in S230 based on data specific to the particular use for which the new model is specifically configured (e.g., a particular new context, a particular new action type, a particular new weather condition, a particular new fixed route, etc.). This may include, for example, for a new model of the first set of models (e.g., a deep decision network), collecting data from the new fixed route along the new fixed route and in the new context, which data is part of the second set of inputs and used to refine the weights of this particular new model. For a new model of the second set of models (e.g., a deep trajectory network), the data used to refine the model may be taken from the vehicle's sensors while the vehicle is performing the new action (e.g., in the new fixed route, in the particular context, etc.). This serves to enable the collection of a minimal, targeted set of data for refining the new model for that particular use.

[0091] Additionally or alternatively, S210 may include any other suitable process and / or be otherwise suitably performed.

[0092] 4.22 Method - Performing a mapping between a first domain and a second domain S220 S200 preferably includes performing a mapping (equivalently referred to herein as a task mapping) S220 between the first domain and the second domain (e.g., as shown in FIG. 3), which functions to enable the decision-making architecture determined in S110 to be adapted (e.g., efficiently adapted) to the second domain (e.g., without requiring training of a comprehensive and new decision-making architecture). S220 may additionally or alternatively leverage the decision-making architecture determined in S100 and function to prevent negative transference and / or other undesired outcomes that may result from improper updating of the set of models, and / or perform any other suitable functionality.

[0093] S220 may be performed at any suitable time(s) during method 200, such as in response to S210, prior to S210 (e.g., when a second set of inputs is collected in response to determining a second region and / or performing task mapping), prior to and / or in response to any other processes of the method, independent of other processes in the method, multiple times during the method (e.g., continuously, at a predetermined frequency, at a random set of intervals, for each new model being developed, etc.), during development of a new model, and / or at any other time.

[0094] A task herein refers to any feature and / or aggregation of features of an ODD associated with an autonomous agent, such as, but not limited to, a context, an action, a behavior, a route, an environmental condition, and / or any other feature. In a preferred set of variations (e.g., shown in FIG. 7), for example, a task may include a context-action pairing (e.g., a lane change action in a multi-lane residential context), with a second region reflecting the addition of any or all of a new context, a new action (e.g., overall), and / or a new action within that context. A mapping herein refers to a mapping between these tasks (e.g., a defined association between those tasks, a relationship between those tasks, etc.).

[0095] S220 preferably includes comparing tasks of the second domain with tasks of the first domain to determine a mapping (e.g., relationship, overlap, similarity, etc.) between tasks and / or domains (e.g., between the source domain and the target domain as shown in FIG. 6), where the mapping preferably serves to identify which tasks of the source domain are relevant to the target domain (e.g., can be utilized by the target domain, can be used in a modular manner to form target tasks in the target domain, etc.). This allows the decision-making architecture to be updated in an efficient and reliable manner (e.g., through transfer learning) (e.g., without negative transfer, without large amounts of data in the second domain, etc.) since the most relevant tasks of the source domain can be leveraged to efficiently update the decision-making architecture.

[0096] For a decision-making architecture that includes a first subset of models each trained for a particular context and a second subset of models each trained for a particular action (e.g., resulting from a model in the first subset of models), assuming a systematic categorization of the models with respect to context and action, a well-defined mapping between tasks (e.g., context-action pair tasks) can be used to update the decision-making architecture efficiently (e.g., in a minimum amount of time, with a minimum amount of data, etc.).

[0097] The system and / or method is preferably designed and / or configured to use multiple source tasks (e.g., multiple source models, multiple source models of the same set, etc.), any or all of which may be used in determining the mapping to the target task. In some variations, for example, all relevant experienced source tasks (e.g., determined based on shared features) are leveraged when learning a new target task. Alternatively, a subset of previously experienced tasks may be used. Which of these variations is implemented preferably depends on knowledge and / or assumptions regarding task distribution, and if the tasks are expected to be similar enough that all past experiences are determined and / or estimated to be useful, there may be no need to select a subset. On the other hand, if the distribution of tasks is multimodal, it may be suboptimal (e.g., inefficient, costly in time and / or computation and / or data, etc.) to transfer from all tasks, and a subset may be determined and utilized.

[0098] In a preferred implementation, S220 is performed for each new model (equivalently referred to herein as an updated model) being generated, such as for each new deep decision network being generated, for each new deep trajectory network being generated, and / or for any other model being generated. Additionally or alternatively, S220 may be performed for a set of multiple new models at a time (e.g., simultaneously, sequentially, etc.), for each new domain expansion (e.g., for each new route that is added, for each new action requirement for a vehicle, for each new context occurrence within one or more routes, for each new location of deployment [e.g., with a new route, new weather conditions, new traffic conditions, etc.], etc.), and / or at any other time(s).

[0099] In a set of variants, for example, for each new model being generated / developed, a mapping is determined, where the mapping indicates which of the prior set of models should be used to generate the new model. The mapping is preferably determined based on similarity between features of the prior set of models and features related to the purpose(s) and / or use(s) of the new model. The prior set of models (equivalently referred to herein as source models) can include any or all of multiple models, a single model, and / or any number of models. The prior set of models is preferably of the same set of models (e.g., all of the first set, all of the second set, etc.), and additionally or alternatively, the source models may come from multiple sets and / or types (e.g., a combination of the first set and the second set).

[0100] The mapping is preferably determined manually (e.g., according to a rule-based and / or programmed process, without a trained model, with human input / feedback, etc.), and / or otherwise determined, such as any or all of: using a programmed (e.g., manually determined) process, using a rule-based process, using a lookup table, using a decision tree, using a set of predetermined mapping assignments, using a set of similarity scores between models, using a set of heuristics, using any combination of processes, etc. This can serve to prevent and / or minimize the occurrence of negative transference, for example, because manual determination of the mapping can enable selection of the most relevant (e.g., directly comparable, explainably comparable, modularly related, etc.) models and relevant portions of the domain for use in determining a new model for the new domain.

[0101] Additionally or alternatively, any or all of the architecture of the decision-making platform can function to enable precise, relevant mappings to be made between models, such as, but not limited to, the modular nature of the models, the continuous evaluation of the models, and / or the volume of models. In some variations, the modular nature and large number of models (e.g., hundreds for a first domain, thousands for a first domain, etc.), each with high specificity (e.g., targeted to a particular context within a particular route, targeted to a particular action within a particular context and / or particular route, etc.), allows the most relevant models to be identified and used (e.g., used exclusively) in generating the new model.

[0102] Context features (e.g., number of lanes, type of road, specific objects in the context [e.g., specific traffic signs, traffic signals, intersections, shoulders, curbs, barricades, bike lanes, etc.]) may be used to generate a first set of new models, without limitation. Any number of features between the models (and / or associated regions or region portions) may be determined and / or used in determining the mapping, such as the presence of obstacles (e.g., obstacles in the context, e.g., obstacles in the context ...

[0103] Additionally or alternatively, any or all of the mappings may be determined using a trained model and / or algorithm and / or may be otherwise suitably determined.

[0104] Determining the mapping may additionally or alternatively leverage latent space representations, such as those described above, which may enable efficient, logical, and / or simple mappings to be determined between source and target domain tasks. This may be enabled, for example, by lower dimensional, meaningful and comparable metrics that the latent space representations can be refined from (e.g., from large amounts of detailed data received in the cognitive subsystem). For example, when considering transfer learning via dimensionality reduction, a low dimensional latent feature space (e.g., for inputs received / determined in the cognitive module) where the variances between the source and target domain data are the same or close to each other may provide a number of benefits. Onto this latent feature space, for example, data in the relevant domain may be projected, and a training process may be applied to train a model (e.g., for a first subset of models, a second subset of models, etc.). Thus, the latent feature space may effectively act as a bridge to transfer knowledge from the source domain to the target domain (e.g., and used to determine an efficient mapping). A dimensionality reduction process is further preferably implemented when determining the latent space, which minimizes the distance between the variances of data in different domains in the latent space. This framework can further function to exploit that some aspects of reward functions are often shared between different (but related) tasks, such as the negative reward that an autonomously driven vehicle may receive for stopping too close to an obstacle. Thus, variations of the method can optionally assume that reward functions "r(t)" for different tasks are related via a latent basis "L" of reward components. These components can be used to reconstruct the true reward function via a sparse combination of such components with task-specific coefficients s(t), using L as a mechanism for transfer.

[0105] In particular examples, a latent space representation of data relevant to a particular model (e.g., data used to train a source model, data collected for a new domain / new model, etc.) may be used to calculate a proximity metric and / or perform a proximity check to determine which source models are similar enough (e.g., similar enough to the purpose / use case / context / action / etc. of the new model) to be used in generating the new model.

[0106] In a first variation of S220, S220 includes determining a first set of new context-based models (e.g., a deep decision network that determines an action for a particular context), and generating the new context-based model includes determining which of the existing first set of models to use in S230 based on a shared set of inter-context features of the existing models and features of the new context. Additionally or alternatively, features of the action options considered and / or selected by the new model and / or the first set of models may be further considered and / or compared between the models. In a particular example, for example, a model configured to select an action of performing a right turn (and / or any other action) in a residential zone may be used to develop a model configured to select an action of performing a right turn (and / or any other action) in a highway context. In another particular example (e.g., shown in FIG. 7), one or more models for a first context of a single-lane residential road, one or more models for a second context of a one-way residential road, and one or more models for a third context of a parking lot are selected (e.g., mapped), and a new model for a fourth context of a multi-lane residential zone is developed due to similarities between the features of these contexts (e.g., residential zoning, associated speed limits [e.g., low speeds in both residential zones and parking lots], number of lanes, lane direction, etc.). Additionally or alternatively, similarities between available actions associated with the contexts may be utilized in determining the mapping.

[0107] In a second variation of S220, S220 includes determining a second set of new action-based models (e.g., a deep trajectory network that generates trajectories for a particular action), and generating the new action-based model includes determining which of the existing second set of models to use in S230 based on a shared set of features between the actions of the existing models and features of the new action. In a particular example, for example, an existing model for a right turn action may be used (e.g., together with other models, alone, etc.) to generate a model for a new action of changing lanes into the rightmost lane due to similarities in features between the actions and / or the purpose of the actions (e.g., refraining from riding up a curb close to the right lane, refraining from crossing a yellow line of the lane, refraining from crossing a set of solid lines of the lane, performing a smooth trajectory, etc.). In another particular example, for a lane keeping action in a new context and / or a new route, other models for lane keeping (e.g., in different contexts, in different routes, etc.) may be used in generating the new model, models for other actions may be used based on similarities in action features (e.g., left nudge, right nudge, etc.), and / or any other model or combination of models may be used to generate the new model. In another particular example, an existing model for implementing a creep action (e.g., a vehicle temporarily slows down / stalls to gather more data regarding what is potentially approaching the vehicle's path before proceeding) may be used to determine a model for a new action of merging (e.g., onto a highway, in a highway context, etc.), and although there are some differences between these actions and / or contexts (e.g., speeds may be different, environments may be different, etc.), the resulting movements (e.g., trajectories) may have shared features and / or goals (e.g., including a decelerating movement, including a gradual angled maneuver when the path is clear, changing lanes, etc.).

[0108] Additionally or alternatively, S220 may include any other process and / or be otherwise suitably implemented.

[0109] 4.23 Method - Updating the Decision-Making Architecture S230 S200 includes updating the decision-making architecture S230 (e.g., as shown in FIG. 3), which serves to extend the ODD associated with the autonomous agent. Additionally or alternatively, S230 can serve to enable the vehicle to navigate new contexts (e.g., an entirely new context, a previous context in a new route, etc.), perform new actions (e.g., an entirely new action, a previous action in a new context, a previous action in the same context in a new route, etc.), perform a new set of routes, and / or otherwise extend the operation and / or usability of the vehicle.

[0110] S230 is preferably performed by a transfer learning process (and / or domain adaptation) and based on any or all of the mappings determined in S220, and optionally by any or all of the training processes described in S120, but may additionally or alternatively be performed in response to any other process of method 200 and / or at any other time(s).

[0111] S230 is further preferably performed based on the decision-making architecture determined in S120, and thus the decision-making architecture determined in S230 starts with any or all of the general model architecture, the model architecture and parameters (e.g., hyperparameters, learned parameters, etc.), parameters, and / or any other information of the previous decision-making architecture.

[0112] The transfer learning process preferably involves determining and / or refining (e.g., retraining, updating, adjusting, etc.) a set of weights associated with the target model(s), but may additionally or alternatively involve determining and / or adjusting the model architecture (e.g., number of neural network layers, organization of neural network layers, number and / or placement of neural network nodes, type of neural network, number of neural networks, etc.) and / or determining any other information.

[0113] In a preferred set of variations (e.g., shown in FIG. 8), S230 starts with a common model architecture (e.g., a base model framework shared among all of the first set of models, a base model framework shared among all of the second set of models, a modified framework, etc.) to determine a new model; derive a set of source models based on the mapping performed in S220; and refine the weights of the derived set of models (e.g., using an algorithm, model, formula, mathematical operation [e.g., summing, averaging, calculating the median, calculating the minimum, calculating the maximum, etc.], finding a weighted average using manual tuning, through gradient descent-based optimization, etc.) to form an initial set of weights for the new model. The method includes aggregating the initial set of weights for the new model based on (e.g., according to approximate sampling from the posterior of the model weights for any individual context-action pair using finding a weighted average, an isotropic Gaussian approximation, and / or approximating the accuracy matrix using Fisher Information, etc.), and refining the initial set of weights for the new model based on training (e.g., retraining, updating, etc.) the new model with data specific to the new model (e.g., data collected [e.g., from sensors on the vehicle] within a particular route and context relevant to the model, data collected while the vehicle is performing a particular action, data collected from sensors with the same level of degradation as when the model was constructed, etc.). Additionally or alternatively, S230 may include any other suitable process.

[0114] Refining the initial set of weights (e.g., during training, retraining, etc.) preferably involves a target model evaluation and tuning process (e.g., providing the model with new data represented in the latent space representation and tuning the model while evaluating its performance), but may additionally or alternatively include any other process.

[0115] Additionally or alternatively, S230 may include a hyperparameter tuning process (eg, based on the second set of inputs, as described above), and additionally or alternatively, any other process.

[0116] S230 preferably takes less time and / or is performed with less data compared to S120, but may alternatively be performed at any other time and / or with any other data.

[0117] Additionally or alternatively, S230 may include any other process.

[0118] 4.3 Methodology - Implementing the Decision Making Architecture S300 Method 200 may optionally include implementing 300 a decision-making architecture, which functions to operate (e.g., manipulate) the autonomous agent in accordance with the decision-making architecture. S300 may additionally or alternatively function to enable operation of the vehicle in new and / or expanded areas (e.g., along a new set of fixed routes, in new contexts, while performing new actions, etc.).

[0119] The decision-making architecture implemented in S300 may include one determined in S100, one determined in S200 and / or any repeated iterations, a combination of architectures, and / or any other decision-making architecture.

[0120] 5. Variations In one variation of the method, such as for use in developing new deep decision networks (e.g., as described above), the method is used to enable a decision-making process learned through an inverse reinforcement learning (IRL) process in a first domain to be adapted for use in a second domain, where the decision-making is configured to generate (e.g., output) an optimal policy that outputs a particular action given a particular vehicle state of the vehicle. The policy preferably refers to (e.g., includes) a suggested action that the vehicle should take for every possible vehicle state.

[0121] In a particular example using inverse reinforcement learning where the goal is to obtain (e.g., generate) a reward function based on state-action pairs (e.g., from human driving data, from simulated data, from data collected on-board an autonomous vehicle, from a combination of data sources / types, etc.), data is first collected from a set of source domains including at least a first domain, and parameters (e.g., weights) of the reward function(s) are learned to represent training samples associated with the source domain data set. In a particular specific example, for example, the model is configured to determine a policy (e.g., at a given time) based on any or all of: a set of internal states (e.g., collectively defining a state space) generated in a latent space representation for the vehicle; a set of actions (e.g., turn right, turn left, change lanes, stop, yield, etc.) that the vehicle can take; a transformation (e.g., a matrix) representing the probability of transitioning from one state to another (e.g., a matrix including the probability of transitioning from one state to another, thereby representing potential modifications that may occur to the agent's state in response to performing an action); and a reward function for the vehicle (e.g., generating reward values ​​in response to the agent's state). Optionally, this reward function determination process can be followed by a reinforcement learning (RL) process (e.g., a training process) configured to generate an optimal policy as output.To use these learnings in a new domain, referred to as the second domain, the process is preferably as follows: collect data from the second domain and derive new deep decision network parameters (reward function) based on model parameters (e.g., weighted average parameters) aggregated from multiple contexts and actions (e.g., through manual tuning, gradient descent-based optimization, etc.) and / or parameters learned from the source domain using approximate sampling (e.g., isotropic Gaussian approximation, Fisher Information Matrix approximation, etc.) from parameters (e.g., posterior of model weights) of any context-action pair (e.g., from the deep decision network associated with the source domain). initializing the source domain parameters (e.g., weights) and further training these initial parameters based on data from the second domain; initializing a policy for RL training by creating a composite policy (e.g., additive, multiplicative, etc.), where the composite policy includes a weighted sum of variances from the source domain policies, and a function (e.g., a gating function) may be determined and / or used to calculate parameter values ​​(e.g., weight values ​​that determine the probability of activating each source domain for a given vehicle state); and conducting an RL re-training session using data from the second domain to obtain an optimal policy for the second domain.

[0122] Additionally or alternatively, variations and / or examples (and / or modified versions) of this may be applied to models other than deep decision networks (e.g., deep trajectory networks), the methods may be implemented otherwise (e.g., without IRL, with a training process other than IRL and / or RL, etc.), and / or the methods may be otherwise suitably performed.

[0123] Although omitted for the sake of brevity, preferred embodiments include any combinations and permutations of the various system components and various method processes, which may be performed in any suitable order, serially or concurrently.

[0124] Embodiments of the systems and / or methods may include any combination and permutation of the various system components and method processes, and one or more instances of the methods and / or processes described herein may be performed asynchronously (e.g., serially), contemporaneously (e.g., concurrently, in parallel, etc.), or in any other suitable order by and / or using one or more instances of the systems, elements, and / or entities described herein. The following system and / or method components and / or processes may be used in conjunction with, in addition to, instead of, or otherwise integrated with all or portions of the systems and / or methods disclosed in the above-mentioned applications, each of which is incorporated by this reference in its entirety.

[0125] Additional or alternative embodiments implement the above methods and / or processing modules in a non-public, transitory computer-readable medium that stores computer-readable instructions. The instructions may be executed by computer-executable components integrated with the computer-readable medium and / or processing system. The computer-readable medium may include any suitable computer-readable medium, such as RAM, ROM, flash memory, EEPROM, optical devices (CD or DVD), hard drives, floppy drives, non-public, transitory computer-readable medium, or any suitable device. The computer-executable components may include a computing system and / or processing system (e.g., including one or more co-located or distributed, remote or local processors) connected to the non-public, transitory computer-readable medium, such as a CPU, GPU, TPU, microprocessor, or ASIC, although the instructions may alternatively or additionally be executed by any suitable dedicated hardware device.

[0126] As those skilled in the art will recognize from the foregoing detailed description, and from the figures and claims, modifications and variations can be made to the preferred embodiments of the invention without departing from the scope of the invention, which is defined in the following claims.

Claims

1. 1. A method for augmenting operation of an autonomous vehicle, the method comprising: determining an initial set of models for operation of the autonomous vehicle along a first set of fixed routes, the initial set of models comprising: a first set of a plurality of models, each of the first set of a plurality of models configured to determine a set of actions for the autonomous vehicle based on a context associated with an environment of the autonomous vehicle; and a second set of models, each of the second set of models configured to determine a trajectory for the autonomous vehicle based on a set of outputs of at least one of the first set of models; and determining, operating the autonomous vehicle along a fixed route of the first set of fixed routes based on the first set of models and the second set of models; augmenting the initial set of models for operation of the autonomous vehicle along a second set of fixed routes, where augmenting the initial set of models includes: determining a third set of the plurality of models, determining a new set of context labels associated with said second set of fixed routes; For each context type associated with the new set of context labels, selecting, in a first manual process, a subset of the first set of models based on a first shared set of features between the context type and a set of contexts associated with the subset of the first set of models; aggregating, in an automated process, model weights associated with the subset of the first set of models to produce a first set of aggregated weights; refining the first set of aggregated weights based on a set of sensor data collected during traversal of the second set of fixed routes, thereby producing the third set of models; determining, determining a fourth set of a plurality of models, determining a new set of actions associated with said second set of fixed routes; and For each action type associated with the new set of actions, selecting, in a second manual process, a subset of the second set of models based on a second shared set of features between the action types and a set of actions associated with the subset of the second set of models; aggregating, in an automated process, model weights associated with the subset of the second set of models to produce a second set of aggregated weights; refining the second set of aggregated weights based on the set of sensor data, thereby producing the fourth set of models; and and extending operating the autonomous vehicle along a fixed route of the second set of fixed routes based on the third set of models and the fourth set of models; A method comprising:

2. Operating the autonomous vehicle along a fixed route of the first set of fixed routes based on the first set of models and the second set of models, at each of a set of a plurality of time points during traversal of the fixed route; Retrieving a context related to the location from a labeled map based on the location of the autonomous vehicle; mapping the context to a model in the first set; evaluating the models of the first set to generate an action; and mapping the actions to models in the second set; evaluating the models of the second set to generate a trajectory; and steering the autonomous vehicle according to the trajectory; The method of claim 1 , comprising:

3. Operating the autonomous vehicle along a fixed route of the second set of fixed routes based on the third set of models and the fourth set of models includes, at each of a set of a plurality of time points during traversal of the fixed route, retrieving a context related to the location from a labeled map based on the location of the autonomous vehicle, wherein the new set of context labels includes the context; mapping the context to a model in the third set; evaluating the models of the third set to generate an action; and mapping the actions to models in the fourth set; evaluating the models of the fourth set to generate a trajectory; and steering the autonomous vehicle according to the trajectory; The method of claim 2 , comprising:

4. 2. The method of claim 1 , wherein each of the first set of models, the second set of models, the third set of models, and the fourth set of models is a machine learning model.

5. 5. The method of claim 4, wherein each of the first set of models, the second set of models, the third set of models, and the fourth set of models comprises a deep neural network.

6. 2. The method of claim 1 , wherein the first set of models, the second set of models, the third set of models, and the fourth set of models each receive as an input an environment representation associated with the autonomous vehicle, and wherein the environment representations of the first set of models and the third set of models are more comprehensive than the environment representations of the second set of models and the fourth set of models.

7. 7. The method of claim 6, wherein the environment representations of the second set of models and the fourth set of models are refined based on a directionality of action selected for the autonomous vehicle.

8. The method of claim 7 , wherein the directionality comprises a heading of the autonomous vehicle defined by the action.

9. 8. The method of claim 7, wherein the environment representations of the first set of models, the second set of models, the third set of models, and the fourth set of models include latent space representations, the latent space representations defining a set of abstract features associated with an environment of the autonomous vehicle, the set of abstract features being determined based on sensor data collected onboard the autonomous vehicle.

10. The method of claim 1 , wherein the second set of fixed routes is separate and distinct from the first set of fixed routes.

11. The method of claim 1 , wherein the subset of the first set of models comprises a plurality of models.

12. The method of claim 11 , wherein the subset of the second set of models comprises a plurality of models.

13. The method of claim 1 , wherein the manual process comprises at least one of a rule-based process and a programmed process.

14. 2. The method of claim 1 , wherein each of the first set of fixed routes and the second set of fixed routes is associated with a delivery use case for the autonomous vehicle, and each of the first set of fixed routes and the second set of fixed routes is arranged between any or all of a set of distribution centers, a set of warehouses, and a set of retailers.

15. 15. The method of claim 14, wherein the new set of actions includes parking at a loading dock associated with one or more of a new distribution center of the set of distribution centers, a new warehouse of the set of warehouses, and a new retailer of the set of retailers.

16. The first shared set of features comprises: A shared context label, or - Set of road features The method of claim 1 , comprising at least one of:

17. The method of claim 16 , wherein the set of road features includes at least one of an intersection, a crosswalk, and a bicycle lane.

18. The method of claim 1 , wherein the second shared set of features includes a directionality of movement implemented in the action type and the set of actions.

19. The method of claim 1 , wherein the automated process includes an evaluation of an algorithm.

20. The method of claim 19 , wherein the algorithm is a trained algorithm.

21. 1. A method for augmenting operation of an autonomous vehicle, the method comprising: determining an initial set of models for movement of the vehicle along a first set of fixed routes, the initial set of models comprising: a first set of models; and a second set of a plurality of models, wherein models in the second set of models are evaluated based on outputs produced by one or more of the first set of models; and determining, operating the autonomous vehicle along a fixed route of the first set of fixed routes based on the first set of models and the second set of models; extending operation of the autonomous vehicle to a second set of fixed routes; determining a set of features associated with said second set of fixed routes; selecting a first subset of the first set of models, the first subset including a plurality of models, based on the set of features associated with the second set of fixed routes and a set of features associated with the first subset; selecting a second subset of the second set of models, the second subset including a plurality of models, based on the set of features associated with the second set of fixed routes and a set of features associated with the second subset; aggregating model weights from the first subset to produce a first set of aggregated weights; aggregating model weights from the second subset to produce a second set of aggregated weights; training a third set of models based on the first set of aggregated weights and a set of data collected during traversal of the second set of fixed routes; training a fourth set of models based on the second set of aggregated weights and the set of data; and extending operating the autonomous vehicle along fixed routes of the second set of fixed routes; A method comprising:

Images