Method and system for handling failures in an autonomous agent - Patents.com

Abstract

A system for addressing faults in an autonomous agent includes a driving subsystem, a control subsystem, a central computing subsystem, and an autonomous vehicle (AV) sensor subsystem. The system may optionally further include a power subsystem, a vehicle chassis subsystem, a communication subsystem, a distributed computing and / or processing subsystem, an auxiliary sensor subsystem, and / or any other components. A method for addressing faults may include any or all of detecting and responding to a fault and operating the vehicle. Additionally or alternatively, method 200 may include any other process.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 290,407, filed December 16, 2021, which is incorporated by reference in its entirety.

[0002] The present invention relates generally to the field of autonomous vehicles, and more particularly to a new and useful system and method for dealing with obstacles in an autonomous agent in the field of autonomous vehicles. [Brief description of the drawings]

[0003] [Figure 1] FIG. 1 is a schematic diagram of a system for dealing with failures in an autonomous agent. [Diagram 2] FIG. 2 is a schematic diagram of a method for dealing with failures in an autonomous agent. [Diagram 3] FIG. 3 is a schematic variation diagram of a drive-by-wire subsystem. [Figure 4] FIG. 4 is a diagram of a first schematic variant of the control subsystem. [Diagram 5] FIG. 5 is a diagram of a second schematic variation of the control subsystem. [Figure 6] FIG. 6 is a schematic variation diagram of an AV computing subsystem in communication with an AV sensor subsystem. [Figure 7] FIG. 7 illustrates a variation on reaching consensus on a failure response among subsystems of a system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The following description of preferred embodiments of the invention is not intended to limit the invention to those preferred embodiments, but rather to enable any person skilled in the art to make and use the invention.

[0005] 1. Overview 1, a system 100 for addressing faults in an autonomous agent includes a driving subsystem 110, a control subsystem 120, a central computing subsystem 130, and an autonomous vehicle (AV) sensor subsystem 140. System 100 may optionally include a power subsystem 150, a communications subsystem 160, a vehicle chassis subsystem 170, a distributed processing and / or computing subsystem, an auxiliary sensor subsystem, and / or any other components. Additionally or alternatively, system 100 may include and / or interface with any or all of the components described in U.S. Application No. 17 / 116,810, filed December 9, 2021, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, U.S. Application No. 17 / 962,459, filed October 7, 2022, and U.S. Application No. 18 / 075,778, filed December 6, 2022, each of which is incorporated herein by reference in its entirety.

[0006] 2, a method 200 for addressing a fault may include any or all of detecting and responding to a fault S100 and operating a vehicle S200. Additionally or alternatively, the method 200 may include any other process. Additionally or alternatively, method 200 may include and / or interface with any or all of the methods, processes, embodiments, and / or examples described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2021, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, U.S. Application No. 17 / 962,459, filed October 7, 2022, and U.S. Application No. 18 / 075,778, filed December 6, 2022, each of which is incorporated herein by reference in its entirety. Method 200 may be performed by the systems described above and / or any other suitable system.

[0007] 2. Profit A system and method for dealing with failures in an autonomous agent can provide several advantages over current systems and methods.

[0008] In a first variant, the technology provides the benefit of enabling an autonomous vehicle to be fail-operational in a manner that is safe and optimally targeted to specific faults through a multi-layered redundant architecture for the vehicle's hardware and software subsystems.

[0009] Additionally, distributed and / or sharded computing (e.g., of various compute, various capabilities and / or functions, etc.) within and / or across subsystems of a system can further contribute to providing this benefit and / or any other benefit. In an example, for example, any or all subsystems of a system may have associated (e.g., designated) computing capabilities (e.g., low-level computing via one or more processors, embedded controllers / computers, etc.), thereby enabling individual subsystems and / or individual components within the subsystems to detect (e.g., autonomously detect, independently detect, etc.) and assess failures, and implement subsystem-specific failure responses (e.g., without confirmation from a higher-level computer, upon confirmation from a higher-level computer, etc.). This can allow failures of different severity and / or resulting from different causes and sources to be optimally handled by the relevant subsystems, thereby saving time, computational resources, and allowing the most relevant failure responses to be initiated. In certain examples, for example, data sharding (e.g., distributing, partitioning, etc.) serves to separate data across an entire compute network into data shards, which allows processing of the data to be easily scalable with low latency.

[0010] In a second variant, additional or alternative to the first variant, the present technology provides the benefit of operating a vehicle autonomously, robustly, reliably, and safely, without reliance on a human operator inside the vehicle (and / or a remote operator remote from the vehicle), as enabled through an architecture that implements various redundancies in both hardware and software (these multiple types and sources of redundancy are equivalently referred to herein as a multi-layer architecture). In certain examples, in the event of a failure (e.g., of a hardware component, of a software system, etc.), the severity of the failure may be assessed and, in light of the failure, addressed by introducing a subsystem or module capable of performing a nominal or degraded functionality.

[0011] Additionally or alternatively, the technology may provide the benefit of enabling any or all of the following: dynamically escalating fail-operational behavior to fail-safe behavior (e.g., after a fail-operational action is triggered, while the fail-operational action is being executed, etc.); dynamically de-escalating fail-safe behavior to fail-operational behavior (e.g., after a fail-safe action is triggered, while the fail-safe action is being executed, etc.); and / or otherwise dynamically adjusting the vehicle's response / behavior (e.g., in response to new information, changed conditions, etc.).

[0012] Additionally, robustness, reliability, and rapid implementation of targeted fault responses may be enabled through a fixed route and / or otherwise limited operational design domain (ODD) associated with the vehicle's use case. In certain examples, for example, a vehicle's fixed route use case allows faults to be detected precisely and with a high level of accuracy and specificity, which in turn allows a precisely targeted fault-specific response to be reliably selected and initiated. In certain examples, for example, fault responses are programmed (e.g., hard-coded, explicitly programmed, etc.)

[0013] In a third variation, additional or alternative to the variations described above, the technique provides the benefit of intelligently arbitrating between various subsystems and various redundant components within each subsystem such that various inputs do not conflict. In certain instances, assessing and responding to a fault occurs on multiple subsystems, and thus appropriate / optimal logic may be used to implement various fallback and / or failsafe responses depending on the particular fault. This can serve to not require overly conservative fallback and / or failsafe behavior in all cases, but rather allow the behavior that is most appropriate and optimal for a particular fault scenario to be selected.

[0014] In a fourth variation, additional or alternative to the variations described above, the technique provides the benefit of working within the space constraints associated with a particular type of vehicle. In a particular example, the system can be incorporated into vehicles sized between passenger cars and semi-trucks, such as class 3-7 commercial trucks, which may not have the space available to simply duplicate all the hardware components to implement sufficient redundancy. Additionally or alternatively, the system and / or method provides the benefit of implementing a trade-off between handling redundancy exclusively through software and handling redundancy exclusively through hardware.

[0015] Additionally or alternatively, the systems and methods may provide any other benefits.

[0016] 3. System 100 1, a system 100 for addressing faults in an autonomous agent includes a driving subsystem 110 (equivalently referred to herein as an actuation subsystem), a control subsystem 120, a computing subsystem 130, and an autonomous vehicle (AV) sensor subsystem 140. System 100 may optionally further include a power subsystem 150, a communications subsystem 160, a vehicle chassis subsystem 170, a distributed computing and / or processing subsystem, an auxiliary sensor subsystem, and / or any other components. Additionally or alternatively, system 100 may include and / or interface with any or all of the components described in U.S. Application No. 17 / 116,810, filed December 9, 2021, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, U.S. Application No. 17 / 962,459, filed October 7, 2022, and U.S. Application No. 18 / 075,778, filed December 6, 2022, each of which is incorporated herein by reference in its entirety.

[0017] System 100 functions to achieve multiple types and locations of redundancy (e.g., hardware redundancy, software redundancy, etc.), equivalently referred to herein as multi-layered redundancy, as well as multiple types and layers of fault management to facilitate reliable, robust, and / or safe operation of the autonomous agents. Examples of types of faults (equivalently referred to herein as faults, errors, susceptibility to faults, etc.) that may be handled by this redundancy may include, but are not limited to, any or all of hardware faults (e.g., electrical faults, sensor faults, actuator faults, controller faults, etc.), software and / or logic and / or algorithmic faults (e.g., logic decision / execution faults, algorithmic decision / execution faults, function execution faults, code execution faults, etc.), hardware and software faults, and / or other faults. Faults may additionally or alternatively refer to the susceptibility of the vehicle to faults, such as when it encounters something unknown in the vehicle's environment. In certain examples, for example, a vehicle's susceptibility to a fault is determined according to an uncertainty estimation process (e.g., for data collected in a sensor subsystem) and / or out-of-distribution detection, such as, but not limited to, any or all of those described in U.S. Application No. 17 / 127,599, filed December 18, 2020, the entirety of which is incorporated herein by reference.

[0018] The system preferably includes at least redundancies managed by at least (at a minimum) the shared and distributed architecture of the system elements, but may additionally or alternatively include redundancies in the manner of operation of the system components, and / or any other redundancies. In a preferred variant, for example, multi-layer redundancies are implemented within and across subsystems of the system, such as, but not limited to, in any or all of redundant actuators of the driving subsystem (e.g., drive-by-wire [DBW] subsystem), redundant sensors in the driving subsystem, overlapping field of view (FOV) sensors in the autonomous vehicle (AV) sensor subsystem, independent and multi-modal AV sensing, redundant and / or independent computing devices, redundant and / or independent communication networks, multiple power sources, and / or any other redundancies. These redundant components are preferably implemented in a manner that then determines and / or implements logical scenarios that function to ensure that any subsystem and / or component failure can be tolerated with minimal risk to the vehicle or others on the road. Additionally or alternatively, the multi-layer redundancy architecture may be implemented in any other suitable manner(s).

[0019] In a preferred variant, redundancy is achieved by introducing physical elements into the architecture, such as duplicated power supplies, duplicated actuators, duplicated sensors, any combination, and / or any other duplicated components. This is combined with various logical scenarios to ensure that any subsystem or component failure can be tolerated through intelligent selective implementation of hardware with associated logic depending on where the failure occurs.

[0020] In particular examples, for example, redundancy may be achieved through any or all of: redundancy in power (e.g., power in the DBW subsystem), redundancy in actuation (e.g., mechanical, hydraulic, and electrical actuators in the DBW subsystem), redundancy in program execution (e.g., redundant controllers, redundant software, etc.), redundancy in communications (e.g., redundant communications buses, redundant Ethernet sources / switches, etc.), redundancy in sensor field of view (FOV) and multi-modal AV sensing (e.g., redundant lidar, redundant cameras, redundant radar, etc.), redundancy in AV compute, redundancy in performing MRC, redundant paths in comparison and / or voting for safety-critical software processes, various redundancies in software design, redundant storage of data, multiple communication network channels (e.g., including to a remote operator), multiple sensor modalities (e.g., lidar and cameras and radar providing 360-degree coverage around the vehicle along with high precision GPS), and / or any other redundancy.

[0021] System 100 further preferably functions to enable the autonomous agent to operate in a fail-operational state such that the vehicle can operate (e.g., with degraded capabilities depending on the type of failure) without reliance on a human operator (e.g., an in-vehicle human operator). Additionally or alternatively, system 100 can operate in a fail-safe state (where the system enters a safe operational mode with reduced capabilities), in a fail-soft state (where the system enters a degraded mode after a failure), and / or otherwise suitably operate with a human operator (e.g., an in-vehicle human operator, a remote teleoperator, etc.).

[0022] In a preferred set of variations, for example, the system and / or associated methods are designed with a fail-safe principle that is both proactively and reactively safe. For example, the system is preferably configured with software and hardware that self-corrects and self-heals in the presence of a fault, enabling an active dynamic recovery mechanism for the vehicle that proactively detects a fault and triggers a self-correction (e.g., in code execution, logic, hardware reset, etc.) to effect optimal (e.g., graceful, with minimal disruption to other vehicles, etc.) fault behavior.

[0023] Proactive safety, as used herein, preferably refers to a system being self-aware (e.g., capable of self-diagnosing) its faults and ready to handle such faults gracefully, safely, and efficiently (e.g., always in a low-latency manner, without requiring the involvement of higher-level systems, etc.). Custom tiered diagnostic systems may be used with built-in self-tests and built-in capabilities (e.g., through low-level distributed computing within a subsystem, through high-level computing capabilities in communication with the low-level computing of the subsystem, etc.) to proactively detect, for example, hardware issues, software issues, and / or vehicle platform issues (or potential issues). This ensures fail-safe redundancy at the system level and / or subsystem level and / or component level, as well as enhancing the effectiveness of the vehicle's overall safety system. Additionally, even in the absence of active human intervention (e.g., from a backup safety driver, from a remote operator, etc.), records may be maintained (e.g., dynamically maintained) and all errors (e.g., critical errors, medium severity errors, low severity errors, etc.) reported from vehicle validation testing, operation, and / or field deployment in order to continuously improve the system's fault response.

[0024] Reactive safety, as used herein, preferably refers to the ability of a system (and / or method) to respond to factors unrelated to the system itself (e.g., characterizing the vehicle's susceptibility to a fault) that may lead to a critical fault, such as a potential collision of the vehicle with another road user (e.g., based on the behavior of other road users due to uncertainties associated with perception sensors, etc.). Based on the potential severity of such a potential fault, an appropriate Minimum Risk Condition (MRC) or other fault response (e.g., a new trajectory) of the vehicle may be selected that minimizes the possibility of harm (e.g., risk of collision).

[0025] Proactive and reactive faults may optionally be systematically characterized in the system and / or method with respect to a common shared fault architecture such that faults occurring in any part of the system, subsystems and / or individual components may be considered with an appropriate fault response (e.g., selection and implementation of MRC). Alternatively, these different types of faults may be characterized differently and / or detected and responded to differently.

[0026] The system further preferably functions as and / or defines a fault management architecture that is distributed among any or all subsystems of the system (e.g., as described in an implementation of method 200 below), such that faults may be detected and / or responded to by any or all subsystems of the system (e.g., through distributed computing among the subsystems, through voting and arbitration within and / or between the subsystems, etc.).

[0027] The system preferably interfaces with (e.g., embedded within) an autonomous vehicle (equivalently referred to herein as an autonomous agent and / or ego agent), but may additionally or alternatively interface with any other suitable robotic system or other subsystem. In a preferred set of variations, the autonomous vehicle is a truck, more preferably a commercial truck between classes 3-7, but may additionally or alternatively include any other vehicle(s). In a particular example, the autonomous vehicle is a class 3-7 truck used for delivery of goods (e.g., business-to-business [B2B] delivery, delivery between retail centers and / or distribution centers and / or warehouses, etc.). Additionally or alternatively, the autonomous vehicle may be used for any suitable function (e.g., transporting passengers, ride sharing, etc.).

[0028] The autonomous vehicles are preferably used in conjunction with fixed route use cases, but may additionally or alternatively be used in non-fixed route use cases.

[0029] In a preferred set of variations, for example, the autonomous vehicle is configured for use in connection with short-range logistics applications, such as those using the autonomous vehicle to deliver goods between destinations. The autonomous vehicle preferably performs these deliveries according to a set of fixed routes, but may additionally or alternatively follow dynamic routes and / or be otherwise operated. In a particular example, for example, the autonomous vehicle may be used according to any or all of the use cases described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2021, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, U.S. Application No. 17 / 962,459, filed October 7, 2022, and U.S. Application No. 18 / 075,778, filed December 6, 2022, each of which is incorporated herein by reference in its entirety.

[0030] In an example, for example, the domain in which an autonomous vehicle operates is purposefully limited, thereby allowing validation and training associated with the system and method to be reduced (e.g., exponentially relative to conventional autonomous vehicles) due to this constrained domain. Also, the probability of unknowns or "edge cases" leading to failures may be substantially reduced relative to other conventional systems and methods, and / or the confidence with which such unknowns and / or edge cases may be identified may be substantially increased. Repeated driving on fixed routes may, for example, allow the technology to be highly optimized for these routes, thereby allowing deployed autonomous vehicles to go through several validation and verification stages that successfully optimize detection and response to specific failures.

[0031] Additionally or alternatively, the system may function and / or be suitably configured differently.

[0032] 3.1. System - Driving Subsystem 110 System 100 includes and / or interfaces with a driving subsystem 110 that functions to control the movement of the vehicle during normal operation, and more preferably during fail-operational and / or fail-safe conditions (e.g., as described below). For example, nominally, in the absence of faults, the autonomous agent controls the movement of the vehicle through the driving subsystem 110. However, in the presence of some faults, redundancy may be achieved by bypassing the driving subsystem for certain, ungraded, movement control actions that have significantly less computational overhead. Additionally or alternatively, any or all of the movement / motion of the vehicle may be controlled by any other subsystem and / or by any combination of subsystems in the absence of the driving subsystem.

[0033] The driving subsystem further preferably functions to detect (e.g., self-detect by low-level computing components as described below) and self-correct (e.g., self-recover, self-reset, etc.) at least a subset of fault types, such as, but not limited to, those associated with the driving subsystem components (e.g., vehicle braking components, vehicle throttle components, vehicle steering components, vehicle shifting components, etc.). Additionally or alternatively, the driving subsystem may function to detect any other fault types and / or events.

[0034] The driving subsystem is also preferably operable to initiate (e.g., trigger, implement, etc.) a fault response and / or any other action (e.g., associated with a component of the driving subsystem) in response to a detected fault (e.g., hard-coded to be handled by the driving subsystem), such as a fault detected by the driving subsystem and / or a fault detected by one or more other subsystems and involving a component of the driving subsystem.

[0035] Additionally or alternatively, the driving subsystem may perform any other suitable functionality.

[0036] The driving subsystem is preferably in the form of a drive-by-wire infrastructure implementing drive-by-wire (DBW) components. Alternative variations may additionally or alternatively include a cable-by-wire infrastructure and / or any other type of driving subsystem or combination of types of driving subsystems.

[0037] The driving subsystem may optionally be operable in a set of operating modes, such as an autonomous operating mode in which the driving subsystem executes vehicle movements when it receives commands from the high-level computing subsystem (e.g., as described below), a manual operating mode in which the driving subsystem acts as a gateway to relay driver inputs to actuators (e.g., a human operating mode, a teleoperated mode, etc.), and / or any other operating mode. Alternatively, the drive-by-wire subsystem may be operable in only a subset of these modes and / or in any other mode.

[0038] The driving subsystem preferably includes a set of actuators incorporated into the subsystems described below along with associated lower level computing components and / or diagnostic sensors (e.g., as described below) (e.g., for detecting actuator health, for detecting actuator outputs, for sensing actuator inputs, etc.), but may additionally or alternatively include any other components.

[0039] The driving subsystem preferably includes a steering subsystem (e.g., as shown in FIG. 3) that functions to steer (turn) the vehicle during autonomous and / or manual modes of operation. Additionally or alternatively, the actuation subsystem may be without a steer-by-wire subsystem.

[0040] The steering subsystem is preferably a steer-by-wire (SBW) subsystem, but may additionally or alternatively include a steer-by-cable subsystem and / or any other subsystem.

[0041] The steering subsystem (equivalently referred to herein as the steer subsystem) preferably includes a redundant set of actuators (e.g., of the same type, different types, hydraulic and electric, etc.), such as a set of actuators, and more preferably a set of motors (e.g., electric power assisted steering [EPAS] motors) that function to rotate one or more steering components (e.g., the steering column) of the vehicle, but may additionally or alternatively include any other actuators and / or types of actuators.

[0042] The steering actuator preferably includes and / or interfaces with processing and / or computing capabilities (e.g., through a low-level processor / computer, such as an embedded controller / computer, as described below), thereby enabling the actuator to detect and / or respond to failures, such as through built-in voting and arbitration functions that can serve to support a fail-operational architecture and to eliminate and / or minimize the risk of failures (e.g., any single point failures) that may disable and / or otherwise affect the system. The steering subsystem further preferably includes and / or interfaces with a communications subsystem, and more preferably a redundant communications subsystem (e.g., redundant communications networks, redundant controller area network [CAN] networks, etc.), that serves to enable commands to be communicated to the steering actuator (and, e.g., receive feedback from the steering actuator, communicate with each other, etc.), and to enable steering information to be communicated to any or all other subsystems and / or components of the system. The steering subsystem further preferably interfaces with a redundant power supply that functions to ensure that continuous power is provided to the steering subsystem (e.g., even if one of the communication buses [e.g., CAN bus] fails). The steering subsystem may optionally implement a mechanical backup mode that allows the vehicle to be steered by a safety operator in the event of a multiple point failure. Additionally or alternatively, the steering subsystem may be implemented in the absence of a mechanical backup mode and / or may be operable in any other mode and / or may include any other components.

[0043] The driving subsystem further preferably includes a braking subsystem (equivalently referred to herein as a brake subsystem) that functions to apply braking forces at the road wheels to stop the vehicle in autonomous and / or manual modes of operation. Additionally or alternatively, the driving subsystem may be without a braking subsystem and / or configured differently.

[0044] The braking subsystem is preferably a brake-by-wire (BBW) subsystem, but may additionally or alternatively include a brake-by-cable subsystem and / or any other subsystem.

[0045] The braking subsystem preferably includes a set of braking actuators (e.g., electro-pneumatic actuators, hydraulic actuators, engine actuators, brake actuators, EPB actuators, etc.), and more preferably a redundant set of braking actuators (e.g., of the same type, different types, etc.) that communicate over a redundant communication subsystem (e.g., a redundant communication network as described above). The braking actuators preferably include and / or interface with processing and / or computing capabilities (e.g., through a low level processor / computer such as an embedded controller / computer, as described below, etc.), thereby enabling the actuators to detect and / or respond to failures, such as through built-in voting and arbitration functions that can function to support a fail-operational architecture as well as to eliminate and / or minimize the risk of failures (e.g., any single point failures) that may disable and / or otherwise affect the system. In some variations, the brake "lines" that hold hydraulic and / or pneumatic pressure are designed with shutoff valves that not only isolate the lines from each other but also distribute pressure to the road wheels so that a failure in a single brake line does not lead to a complete loss of braking. The brake valves are preferably designed to fail open, and upon detection of a loss of pressure (and / or any other relevant failure), the valves open instantly and apply high braking forces to bring the vehicle to an immediate stop. Additionally or alternatively, the brakes may be configured differently. The braking subsystem further preferably interfaces with redundant power sources that function to ensure that continuous power is provided to the steering subsystem (e.g., even if one of the communication buses fails). The braking subsystem can optionally transition to braking through an electric parking brake (EPB) (equivalently referred to herein as an electric park brake) in the event of a multi-point failure.The braking subsystem may optionally and / or additionally utilize engine braking through appropriate transmission gear selection. In the event of a failure, a combination of engine braking and an electric parking brake (EPB) may provide redundant braking action. Additionally or alternatively, the braking subsystem may include any other components and / or operate differently.

[0046] The driving subsystem further preferably includes a throttling subsystem that functions to control longitudinal motion of the vehicle in the autonomous and / or manual modes of operation. Additionally or alternatively, the throttling subsystem may control other motions of the vehicle and / or may be otherwise suitably configured.

[0047] The throttling subsystem is preferably a throttle-by-wire (TBW) subsystem, but may additionally or alternatively include a throttle-by-cable subsystem and / or any other subsystem.

[0048] The throttling subsystem (equivalently referred to herein as the throttle subsystem) preferably includes a set of actuators (e.g., electrical actuators) that control the longitudinal motion of the vehicle by sending appropriate (e.g., optimal) acceleration torque demands. The throttle subsystem preferably interfaces with a redundant set of sensors (e.g., as described below) and an engine control module (ECM) associated with high performance (e.g., high reliability), but may additionally or alternatively interface with any other components (e.g., redundant communication buses, redundant power supplies, etc.).

[0049] The throttling subsystem preferably communicates over a redundant communication subsystem (e.g., the redundant communication network described above). The throttle actuators preferably include and / or interface with processing and / or computing capabilities (e.g., through a low-level processor / computer, such as an embedded controller / computer, as described below), thereby enabling the actuators to detect and / or respond to failures, such as through built-in voting and arbitration functions that can serve to support a fail-operational architecture and eliminate and / or minimize the risk of failures (e.g., any single point failures) that may disable and / or otherwise affect the system. The throttling subsystem further preferably interfaces with redundant power supplies that serve to ensure that continuous power is provided to the throttle actuators (e.g., even if one of the communication buses fails).

[0050] Additionally or alternatively, the throttling subsystem may be suitably configured differently.

[0051] The driving subsystem further preferably includes a shifting subsystem (equivalently referred to herein as a shift subsystem) that functions to facilitate gear selection for the autonomous vehicle such that the autonomous vehicle may be parked, driven forward, driven backward, driven in a low gear, and / or otherwise driven. Additionally or alternatively, the driving subsystem may be without a shifting subsystem and / or may be otherwise suitably configured.

[0052] The shifting subsystem is preferably a Shift-by-Wire (ShBW) subsystem, but may additionally or alternatively include a Shift-by-Cable subsystem and / or any other subsystem.

[0053] The shifting subsystem preferably includes redundant transmission controllers (equivalently referred to herein as shift controllers) coupled to redundant transmission actuators that shift gears on the transmission in autonomous and / or manual operating modes.

[0054] The shift controller preferably includes and / or interfaces with processing and / or computing capabilities (e.g., through a lower level processor / computer such as an embedded controller / computer, as described below), thereby enabling the shifting subsystem to detect and respond to faults, such as through built-in voting and arbitration functions, which can serve to support a fail-operational architecture and eliminate and / or minimize the risk of failures (e.g., any single point failure) that may disable and / or otherwise affect the system.

[0055] The shifting subsystem further preferably interfaces with redundant power supplies that function to ensure that continuous power is provided to the shifting subsystem (e.g., even if one of the communication buses fails). Optionally, the transmission shifter may be manually overridden in the event of a multi-point failure. Additionally or alternatively, the shifting subsystem may include any other components and / or operate otherwise.

[0056] Any or all of the steering, braking, throttling, and shifting subsystems are preferably in communication with each other (e.g., via a communication subsystem, via direct electrical and / or mechanical communication, etc.). Alternatively, the subsystems may be without communication with each other. The driving subsystem is further preferably configured for communication with any other subsystems (e.g., via a communication network) such that a fault can be communicated and / or collectively detected and / or responded to by multiple subsystems and / or multiple components (e.g., as described below). In a preferred variant, the communication subsystem itself handles redundancy by ensuring that multiple communication modes (e.g., CAN, Ethernet, etc.) are available and can be switched to provide redundancy in case of a fault. These are preferably automotive grade communication (e.g., automotive Ethernet) that includes physically robust components and provides high safety performance (e.g., stronger, more resistant to damage, able to withstand higher temperatures and / or physical perturbations, etc.), but may additionally or alternatively include and / or interface with any other components.

[0057] The driving subsystem (and / or any other components of the system) further preferably includes and / or interfaces with an auxiliary sensor subsystem, which includes a set of sensors (e.g., diagnostic sensors, health sensors, health monitoring sensors, etc.), and more preferably a redundant set of sensors (e.g., different types of sensors, same type of sensors, etc.), which function to monitor (e.g., to detect faults, to detect faults of a particular type, etc.) any or all components of the steering subsystem, braking subsystem, throttling subsystem, shifting subsystem, and / or any other components of the driving subsystem. The sensor subsystem further preferably functions to provide feedback from the set of actuators to other components of the system. In a preferred variant, for example, sensor values ​​are sent to a control subsystem (e.g., to one or more controllers of the control subsystem described below) so that the values ​​may be evaluated (e.g., compared to each other, compared to expected values, etc.) and / or used. These values ​​may additionally or alternatively be used (e.g., by a controller, by a low-level computing component, by a high-level computing subsystem, etc.) to determine whether a fault has occurred and, optionally, which component (e.g., which actuator) is the source of the fault. Driving subsystem inputs / outputs (I / O) and measured estimates (e.g., steering angle, brake or throttle rate, etc.) are further preferably evaluated for plausibility, reasonableness, latency / communication diagnostics, cross-sensor checks, output monitoring, kinematically feasible I / O, signal validity, command vs. response diagnostics, and / or any other factors. This due diligence at the low-level control can ensure that only the rarest of rare occurrences of faults can arise from the driving subsystem, and that these rare faults are significantly easier to triage and identify their root causes.Additionally or alternatively, the sensor subsystem may perform any other function.

[0058] The sensors associated with each actuator may be any or all of the same, different, or any combination.

[0059] In a preferred set of variations, the driving subsystem sensors are redundant for each function (e.g., steering, braking, throttling, and shifting) and measure and report the position of each actuator, such as steering angle position, throttle percentage torque, applied brake pedal percentage, and current gear. This feedback can then be optionally transmitted to other subsystems through a communications subsystem (e.g., a redundant communications network) and used by the driving subsystem to detect faults and / or otherwise used by any part of the system to detect and respond to faults. For example, based on the information provided by these sensors, the driving subsystem, and therefore the AV system, can know whether the trajectory set by the planner is being maintained (and / or whether a fault or potential fault has occurred).

[0060] Additionally or alternatively, the driving subsystem may include any other components.

[0061] 3.2. System - Control Subsystem 120 System 100 includes a control subsystem 120 that functions to perform decision-making (e.g., determining commands and sending commands to actuators) regarding how the vehicle should be operated. The control subsystem may further function to determine the health of any or all of the actuators in the driving subsystem such that if there is a failure in any actuator, the control subsystem may trigger utilization of a backup actuator (e.g., a hydraulic SBW actuator that is triggered if an electrical SBW actuator fails). Additionally or alternatively, the driving subsystem may trigger this utilization alone. Still additionally or alternatively, the control subsystem may be used in combination with any other subsystem and / or for any other suitable purpose.

[0062] In a preferred variation, control subsystem inputs / outputs (I / O) and measured estimates (e.g., transmission control range) may be evaluated for any or all of plausibility, reasonableness, latency / communication diagnostics, cross sensor checks, output monitoring, kinematically feasible I / O, signal validity, command vs. response diagnostics, and / or any other characteristics; this due diligence can serve to ensure that only the rarest of rare occurrences of faults can arise from the control subsystem, and that these rare faults are significantly easier to triage and have their root causes identified.

[0063] Additionally or alternatively, the control subsystem may be configured differently.

[0064] The control subsystem 120 is preferably in communication with the computing subsystem 130, and optionally in communication (e.g., via a communications subsystem) with any other subsystems, such as the driving subsystem 110, any other subsystems, and / or all subsystems. The control subsystem 120 may further be distributed among any number of subsystems of the system, such as through controllers (e.g., low-level controllers, embedded controllers, etc.) associated with any or all of the subsystem(s)' actuators or other components (e.g., actuators of the driving subsystem).

[0065] The control subsystem preferably includes a set of one or more controllers, which preferably include an electronic control unit (ECU), but may additionally or alternatively include any other controller (e.g., a microcontroller, a two-position controller, a proportional controller, an integral controller, a derivative controller, etc.), such as, but not limited to, an engine control module (ECM), a powertrain control module (PCM), a transmission control module (TCM), a brake control module (BCM), a central control module (CCM), a central timing module (CTM), a generic electronic module (GEM), a body control module, a suspension control module, and / or any other type of controller.

[0066] The controllers may be of the same kind / type, different kinds / types, associated with the same performance parameters, associated with different performance parameters, and / or include any combination of controllers. Additionally or alternatively, the control subsystem may include any other components (e.g., diagnostic sensors configured to detect / determine controller health, etc.). The controllers are preferably physically embedded within the vehicle network, but may additionally or alternatively be located / configured otherwise.

[0067] The control subsystem preferably includes multiple controllers that may function to provide redundancy (e.g., in case one controller fails), to allow comparison and / or checking of values ​​received by and / or produced in different controllers, and / or to perform any other function. In a preferred set of variations (e.g., as shown in FIG. 4, as shown in FIG. 5, etc.), the control subsystem includes a redundant controller, which includes a first set of one or more controllers (e.g., a first ECU) and a second set of one or more controllers (e.g., a second ECU). The first controller and the second controller are preferably of the same type, but may alternatively be of different types. Additionally or alternatively, the control subsystem may include more than one controller, a single controller, and / or any other components.

[0068] In the example set, for example, the first and second sets of controllers include ECUs configured with a relatively high level of computing power (eg, relative to a watchdog controller).

[0069] In some variations, for example, the control subsystem includes three or more redundant controllers of the same type (eg, three redundant controllers, four redundant controllers, etc.).

[0070] In some variations, for example, the control subsystem includes multiple controllers of the same type and then one or more controllers of a different type (eg, a watchdog controller as described below).

[0071] Any or all of the controllers may optionally include and / or interface with processing and / or computing capabilities to enable the controllers to calculate information, analyze information, compare information, and / or otherwise assist in detecting and / or responding to faults. In some variations, for example, any or all of the controllers include a built-in vehicle dynamics model for evaluation. Each controller further preferably has built-in voting and arbitration capabilities to support a fail-operational architecture and eliminate the risk of failures (e.g., single point failures, multiple point failures, failures occurring in the controller, failures occurring outside the control subsystem) that disable overall system functionality.

[0072] The controller preferably interfaces with redundant power supplies that function to ensure continuous power (e.g., even in the event of a failure on the communication bus). Additionally or alternatively, the DBW controller may include and / or interface with any other components.

[0073] The control subsystem may optionally further include a supervisory controller (equivalently referred to herein as a watchdog controller), which functions to monitor inputs received at (and / or outputs produced by) other controllers of the control subsystem (e.g., the first set of controllers described above, the second set of controllers described above, etc.). For example, if the other controllers are receiving and / or providing different information (e.g., providing different control commands to the watchdog controller), the watchdog controller may determine and / or trigger appropriate actions for the vehicle (e.g., stop, perform MRC, etc.). The watchdog controller may additionally or alternatively function to pass information from the central computing subsystem to the other controllers and / or perform any other functions.

[0074] The watchdog controller is preferably of a different type and / or associated with different performance and / or hardware parameters relative to the other controllers. In a preferred variant, for example, the watchdog controller is physically more robust, of higher safety performance (e.g., ASIL-D rated, automotive grade, safety critical, stronger, more resistant to damage, capable of withstanding higher temperatures and / or physical perturbations, etc.) and / or computationally less sophisticated (e.g., capable of less processing, more resistant to computational errors, etc.) relative to the other controllers, which can minimize the likelihood of a watchdog controller failing, allow for a single watchdog controller in the control subsystem, and / or otherwise provide any other benefit.

[0075] In a particular example, for example, the watchdog controller is a safety-critical, automotive-grade (e.g., Automotive Safety Integrity Level [ASIL] rated, ASIL-D rated, etc.) controller (e.g., ECU) with lower and / or more limited computational capabilities to other controllers in the control subsystem (e.g., passing information back and forth, comparing outputs from other controllers, etc.).

[0076] Alternatively, the watchdog controller may be of the same type and / or capabilities as the other controllers, have greater intelligence than the other controllers, and / or be configured differently.

[0077] In variations that include a watchdog controller, the control subsystem preferably includes a single watchdog controller, but may further include multiple watchdog controllers.

[0078] In a first variant (e.g., as shown in FIG. 4 ), a watchdog controller (e.g., safety-critical, ASIL D-rated ECU, automotive-grade ECU, etc.) with high reliability and availability (e.g., reliable communication, minimal processing tasks, etc.) is implemented as a separate ECU to the other ECUs and serves to perform safety-critical functions, such as performing fault monitoring, fault management, diagnostics, fallback actions (MRC), and ensuring the operation of the AV until a safe state. The watchdog ECU can further thwart attacks by continuously planning minimum-risk operations and / or other fault responses and also performing cybersecurity activities. The entire chain of events, from sensor inputs to the execution of control by the vehicle, is preferably under the supervisory control of the watchdog. Based on the severity level of the faults / failures, they can be raised accordingly to the watchdog controller, where its supervisory algorithms determine the course of corrective actions to be taken by the autonomous vehicle. To ensure continuous power, redundant power supplies can be implemented as needed.

[0079] Additionally or alternatively, any or all of these functions may be performed by other controllers, by a central computing subsystem, and / or otherwise.

[0080] In an example set (e.g., as shown in FIG. 4), the watchdog controller may function and / or otherwise be used to pass information from the central computing subsystem to the other controllers, monitor inputs being received at the other controllers (e.g., to detect that a fault exists if there is a mismatch in the inputs being sent to the other controllers), monitor outputs produced by the other controllers (e.g., to detect that a fault exists if there is a mismatch in the outputs being produced by the controllers), and select which controller to use in passing control commands to the driving subsystem. If the watchdog controller is unavailable (e.g., due to a fault), the other controllers may receive information directly from the central computing subsystem (e.g., in addition to triggering a fault response, instead of initiating a fault response, etc.). Additionally or alternatively, the other controllers may always receive information from the central computing subsystem and / or otherwise be used.

[0081] Alternatively, the control subsystem may be implemented in the absence of a watchdog controller.

[0082] In a second variant (e.g., as shown in FIG. 5), an architecture between multiple controllers (e.g., a first set of controllers and a second set of controllers) and multiple computers is created that allows fault diagnosis and arbitration behavior without the use of additional monitoring hardware (e.g., a watchdog controller). This architecture provides redundancy by having each controller compare its own output and / or input from the computer with the other controller(s) (e.g., paired components in a subset of two controllers). Effectively, each controller provides a check on its partner and on neighboring controllers of different functionality. Thus, any single point of failure is diagnosable and redundantly covered. To check output consistency, for example, each controller compares its output with the output of the other controller(s) to make sure that command timing and values ​​are consistent and / or within otherwise predetermined boundaries. If the output or input does not match, each controller is able to enact a fault response (e.g., a defined minimum risk condition (MRC), a pre-determined fault response, etc.). Thus, preferably, only one component is needed to trigger a fault response. The controller may optionally trigger a fault response without requiring confirmation and / or initiation from the high-level computing subsystem. Additionally or alternatively, the high-level computing subsystem may trigger the fault response and / or the controller may be otherwise suitably configured.

[0083] In an example set (e.g., as shown in FIG. 5), for example, a first set of controllers may receive information from each of a set of multiple computers and a second set of controllers may also receive information from each of a set of multiple computers, and the first set of controllers may compare the information it receives with the information received by the second set of controllers, thereby allowing a fault by one of the computers to be detected (e.g., if different controllers receive different information from the computers) and / or a fault by one of the controllers to be detected (e.g., if different controllers receive the same information from the computers but produce different outputs).

[0084] Additionally or alternatively, control subsystem 120 may include any other components and / or be otherwise suitably configured.

[0085] 3.3. System - Central Computing Subsystem 130 System 100 preferably includes a central computing subsystem 130 (equivalently referred to herein as an autonomous vehicle (AV) computing subsystem, a high-level computing subsystem, and / or a central computing subsystem) that functions to perform computing for decision-making and operation of the autonomous vehicle (e.g., perception tasks, prediction tasks, path planning, generating trajectories, etc.). The AV computing subsystem further preferably functions to detect at least a portion of a fault associated with the vehicle and / or the vehicle's susceptibility to the fault (e.g., due to environmental conditions). Additionally, the AV computing subsystem may function to determine (e.g., select) and initiate a fault response (e.g., a commensurate fault response, an optimal fault response, etc.) associated with the detected fault.

[0086] Additionally or alternatively, the AV computing subsystem may perform any other suitable functionality.

[0087] The AV Computing Subsystem is preferably configured to detect and / or respond to high-level faults associated with the vehicle (e.g., individually, collectively with other subsystems, etc.), such as faults and / or susceptibility to faults arising from the AV Sensor Subsystem and / or any other faults that require high-level computing for detection (e.g., evaluation with a set of trained models, processing with advanced compute algorithms, etc.). Additionally or alternatively, the AV Computing Subsystem may be configured to detect and / or respond to high criticality faults, medium criticality faults, low criticality faults, faults requiring direct intervention, faults not requiring intermediate intervention (e.g., enabling further monitoring of the fault, enabling more sophisticated fault response such as communication with a human operator and / or rerouting the vehicle, etc.), and / or any other faults or combinations of faults.

[0088] Data flowing across the system (e.g., central computing subsystem, central computing subsystem, and collectively subsystems, etc.) is preferably distributed and / or sharded. The result of this data distribution / partitioning is that the data is scalable, has low latency, and there is sufficient information distributed across the subsystems to enable independent decision making regarding operational tasks. Types of data include from various compute, various sensors, communication switches, and / or other capabilities and functions distributed across the subsystems. Furthermore, this information can also include the entire chain of events from sensor input to execution of control. Additionally or alternatively, any other data may be received at and / or processed by the AV computing subsystem.

[0089] Additionally, the AV computing subsystem may function to provide permissions that enable the subsystem to trigger a fault response.

[0090] The AV computing subsystem is further preferably configured to perform normal routine decision-making related to the operation of the vehicle (e.g., action selection, trajectory generation, etc.), such as, but not limited to, any or all of those described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2021, U.S. Application No. 17 / 125,668, filed December 17, 2020, and U.S. Application No. 17 / 127,599, filed December 18, 2020, each of which is incorporated by reference in its entirety into this specification.

[0091] The AV computing subsystem is preferably in communication (e.g., via a communications subsystem) with at least the control subsystem 120 and the AV sensor subsystem 140 (e.g., as shown in FIG. 6 ), but may additionally or alternatively be in communication with one or more subsystems but in communication with any other component (e.g., any or all of the other subsystems) and / or otherwise be capable of communicating with other parts of the system.

[0092] Additionally, the AV Computing Subsystem may be in communication with any or all of the controllers in the control subsystem, portions of the controllers, a particular subset of the controllers (e.g., a watchdog controller), no controllers, and / or any combination of the controllers.

[0093] The AV computing subsystem preferably includes a set of multiple computing devices, such as a set of two computers, but optionally more than two or more than two, which are preferably redundant in operation and may be of the same type, different types, or any combination.

[0094] In a preferred set of variations, for example, the computing subsystem includes multiple computers that are redundant in operation, each processing data from the AV sensor subsystem. The computers process sensory information (data) in a sharded / distributed manner to perceive the world and issue actuator control messages for the driving subsystem to satisfy. In information / data sharding, AV functions such as perception, prediction, trajectory planning, and control can optionally be handled independently by each computer or synchronously with the help of voting and arbitration functions built in to support a fail-operational architecture. Redundant power supplies are preferably used to ensure continuous power.

[0095] Additionally or alternatively, the computing device may be configured differently.

[0096] Additionally or alternatively, the AV computing subsystem may include a single computer, computers of different types and / or functions, and / or any other computing device.

[0097] The system may optionally further include and / or interface with a set of switches (e.g., Ethernet switches), such as high reliability, high availability switches capable of high speed data transfer between the AV sensors, the computing subsystem, and / or any other components. Each switch preferably has its own built-in diagnostic capabilities, but may additionally or alternatively be configured otherwise.

[0098] In a set of variations, the set of switches includes automotive-grade, switch-on-chip (SOC) networking switches that have built-in, fail-operational capabilities, forming a redundant architecture. Optionally, redundant power supplies are used to ensure continuous power.

[0099] Additionally or alternatively, computing subsystem 130 may include and / or interface with any other components.

[0100] 3.4 System - AV Sensor Subsystem 140 System 100 includes an AV sensor subsystem 140 that functions to receive information related to the vehicle's environment, which may be further used for any or all of processing by computing subsystem 130 (e.g., for autonomous vehicle decision-making, etc.), detection of a set of faults (e.g., by the computing subsystem, by other subsystems, etc.) and / or detection of potential faults (e.g., such that an appropriate MRC may be triggered), normal operation of the autonomous vehicle (e.g., navigation along a route, action selection, trajectory generation, etc.), and / or any other purposes.

[0101] The AV sensor subsystem is preferably in communication with the computing subsystem 130 (e.g., via the communications subsystem), but may additionally or alternatively be in communication with any other components of the system (e.g., any or all of the other subsystems).

[0102] The AV sensor subsystem preferably includes a set of sensors configured to collect environmental information relevant to the autonomous vehicle, such as, but not limited to, the presence and / or proximity of other objects (e.g., other vehicles, road signs, pedestrians, static objects, dynamic objects, etc.), type(s) of objects in the vehicle's environment (e.g., vehicles vs. pedestrians vs. inanimate objects, etc.), other characteristics of the vehicle's environment (e.g., road features, road geometry, road type, lane line locations and / or types, etc.), and / or any other information.

[0103] In a preferred variant, for example, the AV sensor subsystem is part of and / or defines a perception module (equivalently referred to herein as a perception subsystem) of the autonomous vehicle.

[0104] The sensors of the AV sensor subsystem may include, but are not limited to, any or all of: cameras (e.g., visual range, multispectral, hyperspectral, IR, stereo vision, etc.), light detection and ranging (lidar) sensors, radio detection and ranging (radar) sensors, orientation sensors (e.g., accelerometers, gyroscopes, altimeters), acoustic sensors (e.g., microphones), light sensors (e.g., photodiodes, etc.), temperature sensors, pressure sensors, flow sensors, vibration sensors, proximity sensors, chemical sensors, electromagnetic sensors, force sensors, sensors in communication with the OBD port or other original equipment manufacturer (OEM) systems, telematic sensors, and / or any other type of sensor(s).

[0105] In a preferred set of variations, the AV sensor subsystem 140 includes a first subset of sensors and a second subset of sensors that are redundant (e.g., of the same type, with the same sensing area, etc.) or at least partially redundant in terms of field of view (FOV) (e.g., having partially overlapping FOV). More preferably, through the use of multimodal sensors (e.g., multiple types of sensors), the dependency on any one sensor type is reduced. Data sharding / distribution can ensure that the same quality of AV sensor data all around the vehicle (redundant FOV coverage) is always available throughout the system. This information includes the entire sequence of events from sensor input to execution of control. In certain examples, at a minimum, under any failure condition, each set of sensors provides enough information to perform a minimum risk operation and bring the vehicle to a safe state. To ensure continuous power, preferably redundant power supplies are used. Additionally or alternatively, the sensors may be configured differently.

[0106] If all sensors fail, the computing subsystem 130 is preferably configured to perform decision-making accordingly (e.g., executing a predetermined failure response that may be implemented without sensor data).

[0107] 3.5. System - Power Subsystem 150 System 100 may optionally include and / or interface with a power subsystem 150 that functions to provide power to any or all of the components of system 100, such as, but not limited to, any or all of the components in subsystems 110, 120, 130, 140, 160, 170, and / or any other components.

[0108] Power subsystem 150 is preferably in communication with (e.g., electrically connected to, connected to by wired connections, operable to power, etc.) all components of the system that it is operable to power (e.g., all hardware components, all electronic components, etc.). Additionally or alternatively, the power subsystem may be in communication with any other components.

[0109] The power subsystem 150 preferably implements redundancy through at least the hardware components of the power subsystem, such as redundancy in power sources (e.g., multiple power sources, different types of power sources, multiple batteries, etc.), redundancy in electrical connections between the power subsystem and other components, any combination, and / or any other redundancy. Additionally or alternatively, the power subsystem may implement a safe electrical architecture for safety-critical functions (e.g., power net), where intelligent switching between software modules and / or hardware components, diagnostics of electrical components, and / or any other measures may be used to achieve fail-operational power for the vehicle. Additionally or alternatively, the power subsystem may implement and / or interface with components that implement software redundancies and / or any other redundancies.

[0110] Each power source preferably has a suitable power capacity to power the associated components of the system, and sufficient additional power capacity (e.g., power backup), so that one or more types of fault responses can be executed (e.g., before all power is lost, without the components executing the fault response losing power, etc.). Additionally or alternatively, a power source may be associated with any other power capacity.

[0111] A switching mechanism that changes the power source from one portion of the power subsystem to another (e.g., from one electrical bus to another) is preferably implemented, which provides the benefits of high reliability and availability. Additionally or alternatively, the power source(s) may be configured separately and associated with and / or configured separately from the switching mechanisms of any other subsystems.

[0112] In a preferred set of variations, power subsystem 150 includes multiple power sources, including at least a primary power source and a secondary power source, which function to power at least the operation and control subsystems, and optionally any other components of system 100.

[0113] In a first particular example, the multiple power sources (e.g., the first power source and the second power source) are of the same type (e.g., the same type of battery, rechargeable battery, non-rechargeable battery, etc.) and / or are associated with the same performance parameter. In a second particular example, the multiple power sources (e.g., the first power source and the second power source) are of different types and / or are associated with different performance parameters.

[0114] Additionally or alternatively, the power subsystem may be otherwise suitably configured and / or perform any other suitable functionality.

[0115] 3.6. System - Communications Subsystem 160 System 100 may optionally include and / or interface with a communications subsystem 160 that functions to establish communications, and more preferably redundant communications, between any or all of the components of system 100.

[0116] Thus, communications subsystem 160 is preferably in communication with (e.g., operable to provide communications capabilities to) and / or distributed among at least a portion of the subsystems of the system (e.g., all components of those subsystems, communications components of those subsystems, etc.), and more preferably in communication with and / or distributed among all subsystems of the system.

[0117] The communications subsystem further preferably enables (e.g., establishes) communication (e.g., information exchange) between and / or among any or all of the subsystems of the system, components within the subsystems (e.g., between components of the driving subsystem), and / or any other parts of the system.

[0118] Communications subsystems may additionally or alternatively be distributed among subsystems and / or components of the system, such as in the presence of communications switches between components of other subsystems.

[0119] The communications subsystem may be of any suitable network type (e.g., network protocol), such as automotive Ethernet, controller area network (CAN), local interconnect network (LIN), any other network type, and / or any combination of networks.

[0120] In a preferred set of variations, the communications subsystem includes a CAN gateway module that bridges different CAN networks of the driving subsystem and / or other subsystems of the vehicle while simultaneously incorporating the computing subsystem over Ethernet. Redundant power supplies may be used to ensure continuous power.

[0121] Additionally or alternatively, the communications subsystem 160 may be configured differently.

[0122] 3.7. Systems - Vehicle Chassis Subsystems 170 System 100 may optionally include and / or interface with a vehicle chassis subsystem 170 that functions to support and enable operation of the vehicle. Additionally or alternatively, vehicle chassis subsystem 170 may function to identify and / or respond to faults (e.g., involving vehicle chassis components, involving driving subsystem components, involving any other components, etc.) and / or perform any other suitable functionality.

[0123] The vehicle chassis subsystem 170 is preferably in communication with the driving subsystem (eg, via a communications subsystem), and optionally, additionally or alternatively, with any other subsystems and / or components.

[0124] In a preferred set of variations, vehicle chassis subsystem 170 includes and / or interfaces with any or all of: one or more emergency parking brake actuators, one or more powertrain transmission shift actuators, one or more anti-lock braking systems (ABS) traction control and / or stability control, one or more advanced driver assistance systems (ADAS), and / or any other components.

[0125] The vehicle chassis subsystem may additionally or alternatively include and / or interface with any number of distributed computing systems (e.g., as described below), auxiliary sensors (e.g., diagnostic sensors for use in fault detection), and / or any other components (e.g., for fault detection, for voting and arbitration, etc.).

[0126] 3.8. Systems - Distributed Processing and / or Computing Subsystems System 100 may optionally include distributed processing and / or computing subsystems that function to allow failures to be detected and, optionally, responded to by multiple (e.g., any, all, some, etc.) subsystems of the system and / or components of any of said subsystems. This may constitute a system for distributed fault management, thereby allowing fault responses to be targeted, rapidly initiated when necessary, and / or otherwise optimally implemented.

[0127] The distributed processing and / or computing subsystem is preferably located in components within one or more subsystems of the system, such as components of the driving subsystem (e.g., embedded controllers / computers coupled to each of the steering, braking, throttle, and shifting subsystems), components of the chassis subsystem, components of the control subsystem, components of the AV sensor subsystem, components of the communications subsystem, and / or any other subsystems, portions of subsystems, and / or any combination.

[0128] In a preferred variant, for example, data flowing across a central computing subsystem is distributed and / or sharded. The result of this data distribution / partitioning is that the data and / or data processing is scalable, has low latency, and there is sufficient information distributed across the subsystems to enable independent decision making regarding operational tasks. The types of information / data can include from various computing subsystems, various sensors, communication switches, and / or other capabilities and functions distributed across the subsystems. Furthermore, this information can also include the entire chain of events from sensor input to execution of control.

[0129] In one particular set of examples, for example, the subsystem includes an embedded computer (eg, an ECU) embedded within a component (eg, an actuator) of the subsystem.

[0130] The distributed processing and / or computing subsystems preferably include lower level (e.g., having lower computational power, less operational scope / responsibility, less computational responsibility, etc.) processors and / or computers relative to the central computing subsystem, but may additionally or alternatively include the same type of computers and / or processors and / or any combination of computers and / or processors. The processors and / or computers are preferably configured to detect faults associated with their respective subsystems and / or components (e.g., based on data from diagnostic support sensors associated with the respective components), but may additionally or alternatively function to initiate fault responses, participate in voting and / or arbitration (e.g., with other subsystems in selecting and / or initiating and / or approving fault responses, etc., as shown in FIG. 7), transmit data to other subsystems (e.g., the central computing subsystem), and / or participate in any other functionality.

[0131] The distributed processing and / or computing subsystems are preferably in communication with a central computing system, where the central computing subsystem uses processed data from these lower level computers to gain an understanding of what is happening throughout the system (e.g., monitor what is happening throughout the system). Additionally, the central computing subsystem can optionally aggregate and / or otherwise process information from these distributed computers, as well as initiate a response if errors and / or anomalies are detected.

[0132] The distributed processing and / or computing subsystems may optionally additionally or alternatively receive commands and / or other information (e.g., authorization to initiate a fault response) from a central computing subsystem. The distributed computers and / or processors are further preferably configured to trigger a subset of fault responses (e.g., a predetermined subset of fault responses specific to the associated component) on their own (e.g., without authorization from one or more other subsystems) such that some faults (e.g., high criticality faults, faults related only to that subsystem, faults most related to that subsystem, etc.) may be initiated with low latency and / or in a timely manner.

[0133] Additionally or alternatively, the distributed processing and / or computing subsystem may include any other components and / or be otherwise suitably configured.

[0134] 3.9 System - Auxiliary Sensor Subsystem The system may optionally include and / or interface with an auxiliary sensor subsystem that functions to detect errors and / or anomalies by any or all of the components of the system. The sensors of the auxiliary sensor subsystem are preferably distributed among any or all of the subsystems of the system, but may additionally or alternatively be located otherwise.

[0135] The auxiliary sensors preferably include diagnostic and / or health monitoring sensors, such as sensors that detect inputs received at and / or outputs produced by one or more actuators (e.g., of the driving subsystem) such that a fault may be detected if a mismatch occurs between the output value and a desired value (and / or if the output value is outside a predetermined range). Thus, the auxiliary sensors (e.g., in combination with a distributed processor and / or computer) can function to enable the individual subsystems to identify and respond to faults. Additionally or alternatively, the auxiliary sensors can include any other sensors.

[0136] Additionally or alternatively, the sensor subsystem may include any other sensor.

[0137] 4. Method 200 As shown in FIG. 2, method 200 may include any or all of detecting and responding to a fault S100 and operating the vehicle S200. Additionally or alternatively, method 200 may include any other process. Additionally or alternatively, method 200 may include and / or interface with any or all of the methods, processes, embodiments, and / or examples described in any or all of U.S. Application No. 17 / 116,810, filed December 9, 2021, U.S. Application No. 17 / 125,668, filed December 17, 2020, U.S. Application No. 17 / 127,599, filed December 18, 2020, U.S. Application No. 17 / 962,459, filed October 7, 2022, and U.S. Application No. 18 / 075,778, filed December 6, 2022, each of which is incorporated herein by reference in its entirety.

[0138] Method 200 is preferably implemented in accordance with system 100 described above, but may additionally or alternatively be implemented by any other suitable system(s).

[0139] The method 200 functions to detect and optimally respond to faults that may occur to and / or affect any or all of the system's components (e.g., as described above). Even in the case of a highly redundant system, there are some faults that may occur that reduce the performance of the AV system. In such cases, the vehicle may be optimally operated in a fail-safe or fail-operational manner that allows the vehicle to take the most appropriate response action based on the severity level and / or type and / or source of the fault so that the AV system reaches a safe state. A safe state herein refers to a reduced operation mode in which the vehicle is in a state of minimal (and / or minimized) risk to itself and other road users. This response action is preferably linked to an operation or action that may be accomplished by the DBW subsystem (herein equivalently referred to as the motion control subsystem and / or the motion control system) in response to the particular fault. Additionally or alternatively, a minimum risk condition (MRC) may be implemented such that a safe state may be guaranteed under all conditions. Online and offline diagnostics across individual elements of the system as well as a tiered fault management system, for example, are some of the ways in which MRC redundancy can be implemented.

[0140] Additionally or alternatively, the method 200 may perform any other functions.

[0141] 4.1 Method - Detecting and Responding to Faults S100 The method 200 may include detecting and responding to a fault S100, which serves to monitor the vehicle for faults and optimal responses based on a type and / or characteristics associated with the fault, which may serve to enable any or all of maintaining the vehicle's progress toward a destination, maintaining a maximum fail-operational status of the vehicle, maintaining and / or exceeding safety standards associated with the vehicle, and / or any other functions.

[0142] Faults are preferably checked continuously (eg, at a predetermined frequency) during operation of the vehicle, but may additionally or alternatively be detected at alternative intervals.

[0143] Faults are preferably checked in multiple subsystems (e.g., in parallel, sequentially according to prioritization, by distributed computing, etc.), but may additionally or alternatively be checked in any other subsystem.

[0144] In a preferred variant, S100 functions to prevent overburdening individual subsystems by utilizing a shared and distributed deployment / architecture for fault management.

[0145] S100 may include any or all of detecting a fault S110, characterizing the fault S120 (e.g., performing a fault analysis process, identifying a specific component / source of the fault, identifying whether there is backup hardware, etc.), triggering an action based on the fault S130 (e.g., triggering / activating backup hardware, determining and / or selecting logic for optimally operating the vehicle in a failed operational status), selecting and / or implementing a fault response (e.g., MRC, further monitoring, selecting a redundant component to use, etc.), and / or any other process.

[0146] The fault types detected in S110 are preferably nuanced and numerous (e.g., along domains of severity, type, criticality, etc.), and each fault type may have unique responses, overlapping responses, and / or any other suitable responses.

[0147] In such cases, the AV computer(s) may optionally be made aware of the detected failure (e.g., prior to initiation of a failure response by the distributed computer, after initiation of a failure response by the distributed computer, without initiation of a failure response by the distributed computer, etc.), and the AV computer may optionally take action (e.g., further action in addition to that by the sub-computer, action instead of that taken by the sub-computer, implementation of more sophisticated minimum risk conditions, etc.). Alternatively, the distributed computer(s) may take action themselves (e.g., in case of high criticality failures, in emergency cases where latency is very important, etc.), effectively bypassing the AV computer(s).

[0148] Other examples of fault detection may include, without limitation, individual component failure, collective component failure, detection of AV sensor data being out of distribution (e.g., out of range, outside predetermined limits, outside predetermined acceptable limits, outside acceptable latency, etc.), detection of the vehicle's environment being unknown and / or dangerous, and / or any other fault.

[0149] Fault responses may be any or all of the following: individually initiated (e.g., by a single subsystem and / or component), collectively initiated (e.g., through voting and / or arbitration and / or approval from multiple subsystems, through approval from an AV computing subsystem, etc.), and / or otherwise initiated.

[0150] Determining (e.g., selecting) the fault response and how it is initiated is preferably determined based on characteristics associated with the fault, such as the fault type, which components are associated with and / or affected by the fault, the severity and / or criticality of the fault, and / or any other characteristics and / or combinations of features.

[0151] For example, different tiers and / or types of responses may be triggered depending on the criticality of the detected fault. In an extremely critical fault (e.g., brakes get stuck), the subsystems preferably act in a low latency manner and may bring the vehicle to a stop without (and / or before) approval from other parts of the system. If the fault is non-critical and / or moderately critical, the subsystem that first detects the fault may send information to other subsystems, such as the AV Computing Subsystem, to enable the AV Computing Subsystem to decide on next steps (e.g., continue to monitor for faults, implement a high performance MRC, etc.).

[0152] In addition or as an alternative to criticality, the number of components and / or subsystems affected by a fault may be taken into consideration in determining and / or initiating a fault response (e.g., the driving subsystem may only determine and initiate a fault response for a fault affecting only the driving subsystem, the AV computing subsystem may determine and initiate a fault response for a fault affecting multiple subsystems, the subsystems may collectively determine and / or initiate a fault response, such as through a voting and / or arbitration and / or consensus process, for a fault affecting multiple subsystems, etc.) and / or the subsystems may otherwise function together and / or independently in determining and / or triggering an action in response to a fault.

[0153] The fault responses are preferably predetermined (e.g., hard-coded) and thus determine which components take actions, in what fault types the components take actions, which subsystems need to provide approval in implementing the action, and / or any other parts of the action implementation. This may be enabled, for example, through a limited domain associated with the vehicle (e.g., fixed route use case).

[0154] Additionally or alternatively, any or all of the responses may be dynamically determined (eg, with a trained model and / or algorithm).

[0155] Examples of fault responses can include, but are not limited to, any or all of: a warning sent within and / or outside the system (e.g., to a remote operator), selecting and / or activating another component (e.g., implementing a redundant component, etc.), sending a notification (e.g., to a remote operator, to a human operator, etc.), implementing MRC (e.g., new trajectory, stopping, slowing down, etc.), waiting and / or doing nothing (e.g., until more information is received, until a reliability value related to the fault exceeds a predetermined threshold, until a consensus is reached among a minimum number of subsystems, until the vehicle completes a move, etc.), and / or any other response. In preferred variations, for example, the fault responses are nuanced and specific such that they do not all result in the vehicle being stopped, thereby preventing overly conservative, prohibitive, and / or otherwise undesirable driving behavior.

[0156] In a first set of variations, for some faults, at least a portion of the subsystems (e.g., predetermined subsystems based on the fault type, all subsystems, etc.) are required to agree on initiating a fault response, thereby preventing any one subsystem from prematurely and / or erroneously initiating a fault response (e.g., when not required).

[0157] In a second set of variations, additional or alternative to the first variation, a single subsystem (e.g., the subsystem that first detects the fault, the subsystem that contains the component involved in the fault, etc.) may select and initiate a fault response (e.g., for high criticality faults, for faults that are predetermined to require direct intervention, etc.).

[0158] In a third set of variations, additionally or alternatively to those above, a subsystem that detects a fault contacts a particular set of subsystem receivers (e.g., as determined based on the fault type), and the set of subsystem receivers collectively determine and / or initiate a fault response. In a particular example, for example, if the steering subsystem experiences a fault (e.g., stalls), the braking and throttle subsystems collectively act together to move the vehicle (e.g., according to a predetermined MRC).

[0159] In a fourth set of variations, additionally or alternatively to those above, for at least some fault types, a single subsystem is responsible for taking action (e.g., based on which is closest [e.g., physically, electrically, etc.] to the source of the fault, based on which subsystem has the greatest effect on other subsystems, based on which subsystem has the least effect and / or least cascading effect on other subsystems, etc.). In a first set of examples, a subsystem is responsible for taking action, but only after confirmation / approval from a particular subsystem (e.g., AV computing subsystem, control subsystem, etc.). In a second set of examples, a subsystem is responsible for taking action without needing approval (e.g., for high criticality errors, for high severity errors, for low severity errors, etc.).

[0160] Additionally or alternatively, S100 may include any other suitable processes.

[0161] 4.2 Method - Operating a Vehicle S200 Method 200 may include operating the vehicle S200, which functions to optimally operate the vehicle according to the faults determined in S100. Additionally or alternatively, S200 may function to operate the vehicle in the absence of the faults (e.g., S200 is performed prior to S100, in the absence of S100, etc.) and / or perform any other functions.

[0162] S200 preferably includes operating the vehicle in accordance with the fault response (e.g., AV logic) determined in S100 (e.g., in a fail-operational status), but may additionally or alternatively include operating the vehicle in a fail-safe status, operating the vehicle in the absence of a detected fault, and / or operating the vehicle otherwise.

[0163] 5. Variations In a preferred variation, method 200 includes detecting an error or potential risk associated with the vehicle and / or any of the vehicle's subsystems, assessing the error or potential risk via any or all of the vehicle's computing and / or processing subsystems (e.g., distributed computing / processing subsystems, central computing subsystems, etc.), determining a categorization (e.g., classification, severity, etc.) of the error or potential risk, and triggering a suitable (e.g., predetermined) action based on the categorization.

[0164] In a first set of examples, the method 200 includes classifying an error or potential risk as being of high severity (e.g., having a severity above a predetermined threshold) and initiating a predetermined fail-safe action (e.g., a minimum risk maneuver that brings the vehicle to an immediate stop).

[0165] In certain examples, as additional information is collected, method 200 can include dynamically determining that de-escalation of a fail-safe action (e.g., to a fail-operational action) is appropriate for the vehicle (e.g., due to a decrease in the calculated severity of the error, due to the elimination of a potential risk when the vehicle's environment changes, etc.). In an illustrative example, for example, a fault is detected at time t=0, which leads to the vehicle selecting and triggering a fail-safe behavior (e.g., initiating a minimum risk maneuver in which the vehicle drives and parks on the side of the road), the vehicle attempts to address and correct the root cause of the fault (e.g., without human input, with remote operator input, etc.) (e.g., switching which actuators are used, further processing of sensory information, etc.), the vehicle successfully corrects the fault, such as by any or all of resetting a subsystem associated with the fault, naturally clearing the fault (e.g., when the vehicle's environment changes), built-in self-tests and / or self-diagnoses of the system's components, and / or any other action, and the vehicle enters a fail-operational (e.g., degraded capability) state, where a different action (e.g., a predetermined action) may be triggered in response to detecting this correction of the fault (e.g., driving to a depot associated with the vehicle).

[0166] In a second set of examples, method 200 includes classifying the error or potential risk as being of medium and / or low severity (e.g., having a severity below one or more predetermined thresholds) and initiating a predetermined fail-operational action in response (e.g., switching to a backup actuator).

[0167] In certain illustrative examples, as additional information is gathered, method 200 can include dynamically determining that escalation of fail-operational behavior (e.g., to a fail-safe action) is appropriate for the vehicle (e.g., by escalating the severity of the potential risk when the vehicle's environment changes, due to an increase in the calculated severity of the error, etc.). In an illustrative example, for example, the vehicle detects a fault at time t=0, which leads to the vehicle selecting and triggering a fail-operational behavior (e.g., continuing to operate along the route in a degraded mode, such as by utilizing one or more backup components and / or without one or more components), the vehicle attempts to address and correct the root cause of the fault (e.g., without human input, with remote operator input, etc.) (e.g., switching which actuators are used, further processing of sensory information, etc.), the vehicle is not successful in correcting the failure, determines that the fault is uncorrectable, and begins a progression toward an escalation of the fault response, and the vehicle initiates an escalation to a fail-safe action (e.g., making a safety stop on the side of the road).

[0168] Additionally or alternatively, method 200 may be implemented in a different manner.

[0169] Additionally or alternatively, method 200 may include any other suitable processes.

[0170] Although omitted for the sake of brevity, preferred embodiments include any combinations and permutations of the various system components and various method processes, which may be performed in any suitable order, serially or concurrently.

[0171] Embodiments of the systems and / or methods may include any combination and permutation of the various system components and method processes, and one or more instances of the methods and / or processes described herein may be performed asynchronously (e.g., serially), contemporaneously (e.g., concurrently, in parallel, etc.), or in any other suitable order by and / or using one or more instances of the systems, elements, and / or entities described herein. The following system and / or method components and / or processes may be used in conjunction with, in addition to, instead of, or otherwise integrated with all or portions of the systems and / or methods disclosed in the above-mentioned applications, each of which is incorporated by this reference in its entirety.

[0172] Additional or alternative embodiments implement the above methods and / or processing modules in a non-transitory computer-readable medium that stores computer-readable instructions. The instructions may be executed by computer-executable components integrated with the computer-readable medium and / or processing system. The computer-readable medium may include any suitable computer-readable medium, such as RAM, ROM, flash memory, EEPROM, optical devices (CD or DVD), hard drives, floppy drives, non-transitory computer-readable media, or any suitable device. The computer-executable components may include a computing system and / or processing system (e.g., including one or more co-located or distributed, remote or local processors) connected to the non-transitory computer-readable medium, such as a CPU, GPU, TPU, microprocessor, or ASIC, although the instructions may alternatively or additionally be executed by any suitable dedicated hardware device.

[0173] As those skilled in the art will recognize from the foregoing detailed description, and from the figures and claims, modifications and variations can be made to the preferred embodiments of the invention without departing from the scope of the invention, which is defined in the following claims.

Claims

1. 1. A system for targeted fault management for an autonomous vehicle, the system comprising: a first set of subsystems, an environmental sensor subsystem comprising a set of sensors configured to detect environmental information associated with the autonomous vehicle; a high-level computing subsystem comprising a set of computers configured to process the environmental information, to detect a first set of faults, and to implement a first set of fault responses in response to the first set of faults; a control subsystem configured to provide information from the high-level computing subsystem to a drive-by-wire subsystem, a first set of controllers comprising a plurality of controllers; a second set of controllers separate and distinct from the first set of controllers, each of the second set of controllers associated with higher hardware performance and lower computational performance relative to each of the first set of controllers; a control subsystem comprising: - the drive-by-wire subsystem comprising a set of actuators; ● Chassis subsystem; a set of low-level processing subsystems disposed within at least one of the environmental sensor subsystem, the drive-by-wire subsystem, the control subsystem, or a vehicle chassis subsystem, the set of low-level processing subsystems configured to detect a second set of faults and implement a second set of fault responses in response to the second set of faults; a first set of subsystems comprising: a second set of subsystems comprising a communication subsystem, the communication subsystem configured to enable communication between subsystems of the first set of subsystems; and A system comprising:

2. The system of claim 1 , wherein the first set of faults is distinct and separate from the second set of faults.

3. The system of claim 2 , wherein at least a portion of the first set of faults are detected based on a set of uncertainty values ​​associated with the environmental information collected at the set of sensors of the environmental sensor subsystem.

4. The system of claim 2 , wherein the first set of fault responses is distinct and separate from the second set of fault responses.

5. 5. The system of claim 4, wherein at least a portion of the first set of obstacle responses comprises navigating the autonomous vehicle in accordance with a minimum risk condition.

6. 5. The system of claim 4, wherein at least a portion of the first set of fault responses includes waiting until the autonomous vehicle completes a movement before triggering an action.

7. 5. The system of claim 4, wherein at least a portion of the second set of fault responses includes activation of one of the set of actuators of the drive-by-wire subsystem, the one of the set of actuators being previously inactive.

8. The system of claim 4 , wherein at least a portion of the second set of fault responses may be initiated without approval from the high-level computing subsystem.

9. The system of claim 8 , wherein the at least a portion of the second set of fault responses is associated with faults having a criticality value above a predetermined threshold.

10. 10. The system of claim 8, wherein at least a second portion of the second set of fault responses require consensus from at least a portion of the first set of subsystems prior to implementation.

11. The system of claim 10 , wherein the portion of the first set of subsystems is predetermined based on a fault type of a fault associated with each of the second portions of the second set of fault responses.

12. 2. The system of claim 1, wherein each of the second set of fault responses is determined based on a predetermined mapping of a first set of predetermined mappings from an associated fault of the second set of faults to the fault response.

13. 13. The system of claim 12, wherein each of the first set of fault responses is determined based on a predetermined mapping of a second set of predetermined mappings from associated faults of the first set of faults to the fault responses.

14. The system of claim 13 , wherein each of the first set of predetermined mappings and the second set of predetermined mappings is hard-coded.

15. 14. The system of claim 13, wherein each of the first set of predetermined mappings and the second set of predetermined mappings is enabled and determined based on a fixed route use case associated with the autonomous vehicle.

16. 2. The system of claim 1, wherein the second set of controllers is configured to monitor a set of outputs produced by each of the first set of controllers, and wherein the second set of controllers is configured to trigger a fault response of the second set of fault responses in response to detecting an inconsistency in the set of outputs.

17. 1. A method for targeted fault management for an autonomous vehicle, the method comprising: checking for a first set of faults with a high-level computing subsystem comprising a set of computers, the set of computers configured to process environmental information from an environmental sensor subsystem of the autonomous vehicle; checking for a second set of faults by a set of distributed processing subsystems disposed within at least one of a set of subsystems of the autonomous vehicle, the set of subsystems comprising: the environmental sensor subsystem; and ● A control subsystem; - a drive-by-wire subsystem having a set of actuators; ● Vehicle chassis subsystem and checking for a second set of faults comprising: in response to detecting one or more of the first set of faults, triggering one of a first set of fault responses in the high-level computing subsystem; in response to detecting one or more of the second set of faults, triggering one of a second set of fault responses, where at least a portion of the second set of fault responses is implemented in the absence of confirmation from the high-level computing subsystem; A method comprising:

18. 20. The method of claim 17, wherein the method is performed continuously during operation of the autonomous vehicle along a fixed route.

19. 20. The method of claim 17, wherein each of the set of distributed processing subsystems is associated with a lower computational performance capability for a computer of the high-level computing subsystem.

20. The control subsystem includes: a first set of controllers comprising a plurality of controllers; a second set of controllers separate and distinct from the first set of controllers, each of the second set of controllers associated with higher hardware performance and lower computational performance relative to each of the first set of controllers; 20. The method of claim 17 comprising:

Images