Hardware virtualization for fault management

Abstract

The invention relates to hardware virtualization for fault management. In various examples, systems and methods related to hardware virtualization for fault management are disclosed. Systems and methods of determining a task flow to perform tasks on a first partition and a second partition are disclosed. The processor may include one or more circuits. The one or more circuits may determine that a first task of the plurality of tasks satisfies criteria for execution in a redundant mode, determine a task flow for execution of the plurality of tasks, where a translator is assigned prior to execution of the first task, the translator causing the one or more circuits to be divided into a first partition and a second partition, and execute the plurality of tasks according to the task stream by executing a first instance of the first task on the first partition and executing a second instance of the first task on the second partition.

Description

Background Technology

[0001] Hardware parallelism can be used to detect failures associated with the deployment of applications, including those for autonomous or semi-autonomous machines. For example, multiple instances of an application can be executed redundantly, such as on different hardware elements and / or at different points in time, which can allow for the occurrence and detection of failures (e.g., random failures). However, variations in application characteristics can make managing failures in a manner that achieves application integrity. Furthermore, implementing redundancy may require exposing low-level redundancy features of the target hardware to the application and / or user layer, which increases the complexity of software design and impacts functional safety. Summary of the Invention

[0002] Embodiments of this disclosure relate to hardware virtualization for fault management. For example, systems and methods are disclosed that facilitate device virtualization to detect random faults. The system can achieve greater diagnostic coverage, for example, by promoting the use of spatial redundancy. The system can improve availability and programmability, for example, by providing the functionality of one or more converters, which can be implemented based on task-related user input and / or processing of task characteristics. The system can implement converters as fast converters to improve performance. The system can scale to multiple tasks and / or applications, and multiple redundant and / or non-redundant nodes, for example, based on processing application tasks into scalable task flows. The system can achieve higher hardware utilization based on higher task occupancy.

[0003] At least one aspect relates to one or more processors. In various embodiments, one or more processors may include one or more circuits, or may be one or more circuits. One or more circuits may: determine that a first task among a plurality of tasks meets the criteria for execution in a redundant mode; determine a task flow for executing the plurality of tasks, wherein a converter is assigned prior to the execution of the first task, the converter being configured to divide the one or more circuits into a first partition and a second partition; and execute the plurality of tasks according to the task flow by executing a first instance of the first task on the first partition and a second instance of the first task on the second partition.

[0004] In various implementations, the converter may be a first converter, and one or more circuits may: determine that a second task among the plurality of tasks depends on the first task and does not meet the criteria for execution in the redundancy mode; and assign a second converter to the task flow between the execution of the first task and the execution of the second task, the second converter being used to departition the one or more circuits.

[0005] In various implementations, one or more circuits may determine that the first task meets the criteria based at least on characteristics of the first task received from at least one of an application containing the first task or user input regarding the first task. One or more circuits may also define the task flow as a graph containing multiple nodes located between a first node for performing a non-redundant task and each of the following converters: (i) a second node coupled to the first node, the second node being used to perform the first instance of the first task; and (ii) a third node coupled to the first node, the third node being used to perform the second instance of the first task.

[0006] In various implementations, the task flow can instruct the one or more circuits to use hardware tools for partitioning the one or more circuits without exposing the hardware tools to the applications associated with the multiple tasks. In various implementations, the converter is a first converter, and the one or more circuits can assign a second converter to the task flow to switch the context from the first task to the execution of a second task to be redundantly performed.

[0007] In various implementations, one or more circuits can configure the first partition and the second partition as multiple contexts, graphics processing unit (GPU) partitions, or multiple instances of a multi-instance GPU (MIG) simultaneously. One or more processors may be included in at least one of the following systems: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system containing one or more virtual machines (VMs); a system implemented using robots; a system implemented using edge devices; a system for generating synthetic data; a system implementing one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; a system for performing operating system (OS) level virtualization (e.g., using containers) (including one or more deep learning models optimized for deployment, software for executing one or more deep learning models, and / or telemetry software for evaluating, monitoring, and / or performing health checks on the system); a system for deploying one or more microservices (e.g., inference microservices (e.g., NVIDIA NIM)); a system for performing conversational AI operations; a system for performing deep learning operations; a system for performing simulation operations; a system for performing collaborative content creation of 3D assets; a system for performing digital twin operations; a system for performing optical transmission simulation; a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0008] At least one aspect relates to a system. In various embodiments, the system may include one or more processing units and one or more memory units. In various embodiments, the one or more memory units may store instructions that, when executed by the one or more processing units, cause the one or more processing units to perform operations including: determining that a first task among a plurality of tasks meets a criterion for execution in a redundant mode; determining a task flow for executing the plurality of tasks, wherein a converter is assigned prior to the execution of the first task, the converter being used to divide the one or more tasks into a first partition and a second partition; and executing the plurality of tasks according to the task flow by executing a first instance of the first task on the first partition and a second instance of the first task on the second partition.

[0009] In various implementations, the converter is a first converter, and the operations performed by the one or more processing units include: determining that a second task among the plurality of tasks depends on the first task and does not meet the criteria for execution in the redundancy mode; and assigning a second converter to the task flow between the execution of the first task and the execution of the second task, the second converter being used to de-partition the one or more circuits.

[0010] In various implementations, the one or more processing units may determine that the first task meets the criteria based at least on characteristics of the first task received from at least one of an application containing the first task or user input regarding the first task. The one or more processing units may also define the task flow as a graph containing a plurality of nodes located between a first node for performing a non-redundant task and each of the following transponders: (i) a second node coupled to the first node, the second node being used to perform the first instance of the first task; and (ii) a third node coupled to the first node, the third node being used to perform the second instance of the first task.

[0011] In various implementations, the task flow may instruct the one or more processing units to use hardware tools for partitioning the one or more processing units without exposing the hardware tools to the application associated with the multiple tasks. In various implementations, the converter is a first converter, and the one or more processing units may assign a second converter to the task flow to switch the context from the first task to the execution of a second task to be redundantly executed.

[0012] In various implementations, one or more processing units may configure the first partition and the second partition as multiple contexts, graphics processing unit (GPU) partitions, or multiple instances of a multi-instance GPU (MIG) simultaneously. The system may be included in at least one of the following systems: a control system for autonomous or semi-autonomous machines; a perception system for autonomous or semi-autonomous machines; a system containing one or more virtual machines (VMs); a system implemented using robots; a system implemented using edge devices; a system for generating synthetic data; a system including one or more large language models (LLMs); a system implementing one or more visual language models (VLMs); a system implementing one or more multimodal language models; a system for performing operating system (OS) level virtualization (e.g., using containers) (which includes one or more deep learning models optimized for deployment, software for executing one or more deep learning models, and / or telemetry software for evaluating, monitoring, and / or performing health checks on the system); a system for deploying one or more microservices (e.g., inference microservices (e.g., NVIDIA NIM)); a system for performing conversational AI operations; a system for performing deep learning operations; a system for performing simulation operations; a system for performing collaborative content creation for 3D assets; a system for performing digital twin operations; a system for performing optical transmission simulation; a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0013] At least one aspect relates to a method. The method may include: determining that a first task among a plurality of tasks meets a criterion for execution in a redundant mode; determining a task flow for executing the plurality of tasks, wherein a converter is assigned prior to the execution of the first task, the converter causing the one or more circuits to be divided into a first partition and a second partition; and executing the plurality of tasks according to the task flow by executing a first instance of the first task on the first partition and a second instance of the first task on the second partition.

[0014] In various implementations, the converter is a first converter, and the method may further include: determining that a second task among the plurality of tasks depends on the first task and does not meet the criteria for execution in the redundancy mode; and assigning a second converter to the task flow between the execution of the first task and the execution of the second task, the second converter being used to departition the one or more circuits.

[0015] In various implementations, the task flow is defined as a graph, which may include multiple nodes, located as a converter between a first node for performing a non-redundant task and each of the following: (i) a second node coupled to the first node, the second node being for performing the first instance of the first task; and (ii) a third node coupled to the first node, the third node being for performing the second instance of the first task. In various implementations, the first and second partitions may be configured as multiple instances of a multi-context, graphics processing unit (GPU) partition, or multi-instance GPU (MIG) simultaneously.

[0016] The processors, systems, and / or methods described herein can be implemented by or include at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system implemented using a robot; a system for performing deep learning operations; a system implemented using an edge device; a system comprising one or more virtual machines (VMs); a system for performing simulation operations; a system for performing digital twin operations; a system for performing optical transmission simulation; a system for performing collaborative content creation for 3D assets; and a system for generating or presenting at least one of virtual reality, augmented reality, or mixed reality content. Systems; systems for performing conversational AI operations; systems including one or more large language models (LLMs); systems including one or more visual language models (VLMs); systems including one or more multimodal language models for generating synthetic data; systems for performing operating system (OS) level virtualization (e.g., using containers) (which include one or more deep learning models optimized for deployment, software for performing one or more deep learning models, and / or telemetry software for evaluating, monitoring, and / or performing health checks on the system); systems for deploying one or more microservices (e.g., inference microservices (e.g., NVIDIA NIM)); systems implemented at least partially in a data center; or systems implemented at least partially using cloud computing resources. Attached Figure Description

[0017] The system and method for hardware utilization for fault management will now be described in detail with reference to the accompanying drawings, wherein:

[0018] Figure 1 This is a block diagram of an example system for performing hardware utilization for fault management according to some embodiments of this disclosure;

[0019] Figure 2 It is based on some implementation schemes of this disclosure that may be provided by Figure 1 A schematic diagram illustrating an example of the task flow generated and executed by the system;

[0020] Figure 3 This is a flowchart illustrating an example of a fault management method according to some embodiments of this disclosure;

[0021] Figure 4A These are illustrations of example autonomous vehicles according to some embodiments of the present disclosure;

[0022] Figure 4B According to some embodiments of this disclosure Figure 4A Examples of camera positions and fields of view for autonomous vehicles;

[0023] Figure 4C According to some embodiments of this disclosure Figure 4A A block diagram of an example system architecture for an example autonomous vehicle;

[0024] Figure 4D Cloud-based servers and according to some embodiments of this disclosure Figure 4A A system diagram illustrating communication between autonomous vehicles;

[0025] Figure 5 This is a block diagram of an example computing device applicable to implementing some embodiments of this disclosure; and

[0026] Figure 6 This is a block diagram of an example data center applicable to implementing some embodiments of this disclosure. Detailed Implementation

[0027] Systems and methods related to hardware utilization for fault management are disclosed, such as those for executing applications in autonomous or semi-autonomous machines to facilitate the detectability of virtualized devices for random faults. Hardware parallelism can be used to detect faults, such as random faults. This can be useful for detecting certain faults in safety-related applications, such as those associated with autonomous or semi-autonomous machines. For example, to achieve a safety integrity level (e.g., Automotive Safety Integrity Level (ASIL)), it can be useful to utilize the built-in parallelism of hardware for components such as massively parallel systems (e.g., graphics processing units (GPUs)) and computing accelerators (e.g., deep learning accelerators). In some cases, parallelism is achieved using redundant task execution. However, various applications have different performance and utilization targets, as well as different sensitivities to random faults. Such applications may be expected to run and / or run concurrently on the same hardware devices. Therefore, managing random faults and achieving the target integrity level can be challenging. Furthermore, while hardware may have attributes and / or functions to facilitate redundancy, exposing such functions to applications and / or users can lead to increased complexity in software design and / or compromise integrity.

[0028] The systems and methods according to this disclosure can manage task execution on target hardware to facilitate greater diagnostic coverage of faults, for example, by selectively implementing temporal redundancy (e.g., sequential execution of redundant workloads) and / or spatial redundancy (e.g., executing workloads in different hardware units) based on one or more criteria for the workload (e.g., performance requirements; integrity levels). For example, the system can retrieve an indication of a task flow containing one or more processing tasks. The system can update the task flow based on one or more criteria, such as assigning one or more redundant operations between processing tasks in the task flow. This can include, for example, executing two tasks in a temporally or spatially redundant manner based on criteria.

[0029] The system can assign mode converters to points in a task flow to achieve redundancy, for example, by assigning mode converters based on standards. Mode converters can partition hardware into multiple parts (e.g., virtual parts) to achieve spatial redundancy (although keeping hardware in partitioned mode may impact the performance of non-safe tasks, this can result in greater diagnostic coverage). The system can use the first part to execute a first task and the second part to execute a second task. In response to detecting task completion, the system can switch the hardware to an unpartitioned mode (e.g., to allow full utilization of the hardware) to execute a third task. The system can detect one or more dependencies between tasks to determine a task flow (e.g., as a graph with nodes corresponding to the operational order of the task flow and the hardware resources to be used in the task flow). The system can generate task flows based on one or more standard modifications to achieve redundancy, but which are logically equivalent to the unmodified task flows.

[0030] Pattern converters can be exposed to applications and / or users at different levels of abstraction. For example, a pattern converter library can be provided for software configuration. The system can retrieve one or more criteria-based indications related to a task (e.g., based on user input and / or parsing of software code) and determine whether to assign a pattern converter to a task flow based on these indications.

[0031] The system can provide output representing runtime diagnostics of the performed task, such as indicating faults detected during task execution. The system can generate this output to indicate security and performance profiles (e.g., based on different redundancy usage scenarios), allowing users to select a profile for runtime operation.

[0032] Although this disclosure may be combined with exemplary autonomous or semi-autonomous vehicles or machines 400 (e.g., "vehicle 400", "self-vehicle 400", "machine 400" or "self-machine 400"), examples are combined with Figures 4A-4DThe invention is described herein (as described herein), but this is not intended to limit the invention. For example, the systems and methods described herein can be used (but are not limited to) non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, unmanned aerial vehicles, and / or other vehicle types. Furthermore, the systems and methods according to this disclosure can be used to monitor the health and / or malfunction conditions of multi-sensor fusion systems used in autonomous or semi-autonomous driving and active safety systems, as well as augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technical field where multi-sensor fusion systems can be used.

[0033] refer to Figure 1 , Figure 1 This is an example system for performing hardware exploitation for fault management according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to (or instead of) the arrangements and elements shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions described herein, as performed by entities, can be performed by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein can be used with… Figures 4A-4D The exemplary autonomous vehicle 400 in the example Figure 5 Exemplary computing device 500 and / or Figure 6 The components, features, and / or functions of the exemplary data center 600 in the example are similar to those of other components, features, and / or functions.

[0034] System 100 may include an application layer 104. Application layer 104 may include one or more applications. One or more applications may be installed on, for example, an autonomous or semi-autonomous vehicle or machine 400 or an example computing device 500, or may be implemented at least partially by one or more devices remote from the autonomous vehicle 400 and / or the example computing device 500. One or more applications may be configured to perform operations including, but not limited to, collecting data, visualizing data, monitoring, and implementing control. For example, one application may be configured to control the braking system of vehicle 400. Furthermore, applications installed on the autonomous vehicle 400 are expected to meet the Automotive Safety Integrity Level (ASIL) to ensure that the autonomous vehicle 400 maintains and remains within the ASIL range. In various implementations, users can add and / or remove applications from application layer 104.

[0035] To perform operations (e.g., on autonomous vehicle 400 and / or components of autonomous vehicle 400), one or more applications in application layer 104 may include multiple tasks. One or more applications may include application data (e.g., code). The application data may define multiple tasks. For example, a first portion of the application data may represent a first task among multiple tasks. Multiple tasks associated with an application for detecting obstacles and / or hazards around the vehicle may, for example, include collecting sensor data, generating bounding boxes, and / or identifying hazards. Each of the multiple tasks may include multiple instances (e.g., functions). For example, if the first task is to generate bounding boxes, the first instance of the first task may be to detect objects.

[0036] System 100 may include stream generator 108. Stream generator 108 may include one or more processors for performing operations such as processing application data from application layer 104 to generate a data structure representing a task flow for performing multiple tasks. The task flow may indicate at least one of the following: the order or redundancy of each of the multiple tasks (e.g., requirements for spatial and / or temporal redundancy in execution).

[0037] A task flow can be a graph containing multiple nodes. Flow generator 108 can generate a task flow by assigning at least one node to any one or more of a plurality of tasks. For example, a task flow may include nodes of the following types to represent tasks: mission nodes, non-redundant nodes, and redundant nodes. In various embodiments, flow generator 108 may assign multiple nodes to a given task, such as one or more of mission nodes and redundant nodes. For example, each of the plurality of tasks may include multiple nodes or be assigned to multiple nodes. Flow generator 108 may determine the node type to be assigned to one or more nodes for a given task based on input received from application layer 104 and / or the user. For example, flow generator 108 may fully parse application data to determine the purpose of a given task (e.g., perform a redirection operation), thereby determining the node type. These purposes may be determined by, for example (but not limited to), the codebase, characteristics, and / or functionality of the application data for the given task.

[0038] In various implementations, stream generator 108 can determine that a first task among a plurality of tasks meets the criteria for execution in a redundant mode, such as assigning the first task to one or more redundant nodes. For example, stream generator 108 can classify multiple portions of a plurality of tasks to assign them to mission nodes and non-redundant nodes. As described above, stream generator 108 can classify tasks into redundant (or non-redundant) operations based on user input or parsing at least one of the application code received from application layer 104. In response to determining that the first task contains at least one portion to be assigned to a mission node, stream generator 108 can determine that the criteria for execution in a redundant mode are met. In various implementations, a redundant mode means that the task flow contains redundant nodes and / or contains spatial redundancy.

[0039] In various implementations, the stream generator 108 can determine that at least one criterion is met based on the characteristics of a first task received from one or more applications. This characteristic may include user input. For example, if a user wants to achieve a higher ASIL level, the user can input the task to be performed in redundant mode. In various implementations, multiple tasks can meet ASIL by being redundant and having greater diagnostic coverage for faults within system 100. In various implementations, the characteristics are based on the functional assignment of the first task. For example, if the first task involves generating a graph, the characteristics can be determined based on a codebase associated with the first task (e.g., a graph codebase). In this case, the graph generating the first task can then be assigned non-redundant characteristics and / or graph characteristics.

[0040] In some implementations, within a vehicle environment, the flow generator 108 can determine which mission nodes to assign to perform safety-critical tasks (e.g., vehicle control) by sending commands to the vehicle's control system. In this case, the primary task associated with controlling the vehicle's brakes can be categorized as a mission node. Mission nodes can be redundant (e.g., executed on multiple hardware devices and / or multiple components of the hardware devices, with redundant nodes associated with the mission node).

[0041] In some implementations, the flow generator 108 can determine which redundant nodes to assign to appropriate mission nodes, for example, allowing spatial and / or temporal redundancy for fault detection. In some implementations, the redundant nodes can be implemented as backup systems to ensure continued vehicle operation in the event of a failure (e.g., a mission node failure) – such as a fault in the code or a hardware failure. For example, a task flow for an application involving steering would include redundant nodes to maintain steering control in the event of a mission node failure. Redundant nodes also allow for the isolation and resolution of faults without affecting system 100. Redundant nodes can be synchronized with mission nodes to ensure that, in the event of a failure, the redundant nodes can take over without any loss of data or functionality.

[0042] In some implementations, non-redundant nodes are not critical to the vehicle and can be performed without a backup system. An example of a non-redundant node could be a node that manages media playback (e.g., music). For instance, the first task in a series of tasks preceding both the mission node and the redundant node could be a non-redundant node.

[0043] To generate a task flow, stream generator 108 can evaluate one or more dependencies among multiple tasks. These dependencies can include control dependencies and data dependencies. Stream generator 108 can determine at least one of the following based on task dependencies: the order of multiple tasks or the parallelization of multiple tasks. Stream generator 108 can take, for example, application code as input and can modify the task flow of that code to output a modified task flow based on one or more criteria (e.g., ASIL) to achieve redundancy (e.g., adding redundant nodes) and / or be logically equivalent to the unmodified task flow. Stream generator 108 can also modify the task flow by considering user input and / or application performance requirements.

[0044] Stream generator 108 can generate task streams (e.g., task streams modified relative to the initial order of instructions or tasks indicated by application layer 104) with temporal redundancy so that redundant workloads can be executed sequentially. For example, one or more first processing units (e.g., GPUs) can sequentially execute multiple tasks of an application, completing the tasks of one node before moving to the next node.

[0045] Stream generator 108 can generate task flows with spatial redundancy. For example, stream generator 108 can assign tasks that need to be executed redundantly to different processing units. In some implementations, spatial redundancy can have higher diagnostic coverage and a lower failure rate than temporal redundancy. For example, spatial redundancy can have approximately 99% diagnostic coverage, compared to approximately 95% for temporal redundancy (although implementing spatial redundancy is not always useful, as it reduces resource availability for tasks such as non-critical tasks). Stream generator 108 can assign a first task among multiple tasks to execute with temporal redundancy and a second task to execute with spatial redundancy, based on one or more criteria (e.g., performance requirements, ASIL).

[0046] In various implementations, the task flow may include logical nodes. These logical nodes may include algorithms for examining task logic. For example, the logical node examines the task's logic (e.g., code) and may issue alerts to the user regarding systemic failures within the task and / or application data.

[0047] Stream generator 108 can assign one or more converters (e.g., mode converters) to a task flow. Stream generator 108 can assign the converter to change the hardware resource usage of target hardware 120. Stream generator 108 can assign one or more converters to a location in the task flow that meets the criteria for a redundancy mode after determining that a task meets the criteria for a redundancy mode. These converters can be used to guide the hardware management of target hardware 120, including, for example, switching the occupancy state and / or partition state of target hardware 120. These converters can be nodes that allow the task flow to switch between a default state (e.g., full use, unpartitioned mode of target hardware 120) and a partitioned mode (e.g., partitioned state of target hardware 120). For example, the converter can split target hardware 120 into a first partition and a second partition to run mission nodes and / or redundant nodes. The first partition can execute mission workloads (e.g., multiple mission nodes), and the second partition can execute corresponding redundant workloads (e.g., multiple redundant nodes synchronized with multiple mission nodes). These partitions can be virtual entities (e.g., MIGs) of physical devices (e.g., GPUs). The converter can be performed by, for example, a central processing unit (CPU) or a GPU system processor (GSP).

[0048] In some implementations, the converter includes a cross-instance synchronization mechanism that allows for rapid reconfiguration of partitions. For example, the converter can create synchronization objects for mission nodes and redundant nodes within a task flow and initiate a switchover mode (e.g., partitioned, unpartitioned) once the execution of the indicated node (task) has been completed.

[0049] Stream generator 108 can retrieve indications of one or more criteria (e.g., performance requirements, redundancy) related to a task. These indications can be received by parsing application code and / or from user input. For example, user input may include high ASIL thresholds for multiple tasks, resulting in more node redundancy in the task flow across multiple tasks. Based on these indications, stream generator 108 can assign one or more converters to the task flow. For example, converters can be assigned to the task flow before executing the first task among multiple tasks. In this case, the first task can be classified as executing in a redundant mode, so converters can be assigned before executing the first task to initiate the redundant mode.

[0050] For example, the converter can be located between a first node for executing non-redundant tasks and a second and third node. In this case, the second node can execute a first instance of the first task, and the third node can execute a second instance of the first task. The first instance can correspond to the mission node, and the second instance can correspond to the redundant node. The second node can be located in a first partition 116, and the third node can be located in a second partition 116. The first partition 116 and the second partition 116 (e.g., partitions, partition states) can be configured as multiple contexts, GPU partitions, or multiple instances of a multi-instance GPU (MIG) simultaneously (e.g., concurrently, in parallel). In this case, after the execution of the first node, the converter is triggered to switch from unpartitioned mode to partitioned mode, thereby executing the second and third nodes.

[0051] In various implementations, nodes in a task flow execute in one or more contexts (e.g., computation contexts), and one or more converters also cause one or more context switches. One or more contexts may include state information about the node's execution on the hardware kernel. Context switching can maximize resource utilization and allow the hardware to handle multiple tasks simultaneously, and in various implementations, also handle one or more applications.

[0052] In various implementations, one or more converters only indicate context switching. In various implementations, one or more converters only switch from partitioned mode to unpartitioned mode and vice versa. In various implementations, one or more converters switch both context and partitioned mode.

[0053] In various implementations, system 100 maintains one or more converters in a converter library provided for software configuration. The converter library may include, but is not limited to, converters for switching task flows, switching contexts, etc., which allows the user to change the task flows generated by stream generator 108.

[0054] In various implementations, one or more converters include a first converter and a second converter. The application includes a second converter after determining that a second task among multiple tasks depends on the first task and does not meet the criteria sufficient for execution in redundant mode (e.g., the second task is assigned to a non-redundant node). The second converter can then be placed between the execution of the first and second tasks, and the second task can cause hardware 120 to unpartition. In various implementations, it is determined that an instance of the first task does not meet the criterion, so a second converter can be placed between the execution of instances of the first task. In various implementations, one or more converters include a third converter placed between nodes in partitioned mode. In this case, the third converter switches the context of the node executing on a portion of the hardware.

[0055] Further reference Figure 1 System 100 may include hardware manager 112. Hardware manager 112 may perform operations such as processing task streams received from stream generator 108 to partition target hardware 120 according to the task streams. Multiple tasks may be executed by hardware manager 112. For example, hardware manager 112 may split multiple tasks into executions on various hardware components (e.g., MIG of GPU).

[0056] For example, when a task flow includes a converter, hardware manager 112 can assign nodes after the converter to one or more virtual entities (e.g., MIGs) and / or physical entities (e.g., streaming multiprocessors (SMCs)) of one or more hardware 120s. Hardware manager 112 assigns nodes to various entities of hardware 120 based on, for example, whether a node is a mission node or a non-redundant node. To prevent hardware manager 112 from being exposed to applications associated with multiple tasks, the task flow can instruct hardware manager 112 to partition the task flow using hardware (e.g., SMCs, MIGs).

[0057] Depending on whether the task flow includes one or more converters, hardware manager 112 can assign nodes to partition 116 to be processed and executed. Hardware manager 112 can assign one or more applications with one or more nodes to run concurrently (e.g., simultaneously, in parallel) on partition 116, thereby increasing the utilization of hardware 120. In various implementations, hardware manager 112 evaluates the task flow to determine whether the task flow is redundant or non-redundant. In the non-redundant case, hardware manager 112 can assign the task flow to continue running on the full (e.g., unpartitioned) hardware 120, while partitioning the redundant flow to run in parallel on partition 116.

[0058] In various implementations, hardware manager 112 sequentially reuses components of the same hardware (e.g., virtual entities and / or physical components) to redundantly perform tasks. For example, redundant nodes may run sequentially on the same components (e.g., MIG, SMC). In various implementations, hardware manager 112 mutates hardware 120 (e.g., creates its digital twin) to split it into one or more virtual devices, thereby performing tasks redundantly in parallel. In this case, hardware 120 can then perform multiple redundant tasks simultaneously. This can improve the detectability of persistent random failures. One or more virtual devices can then be mutated into a single or multiple virtual devices to perform non-redundant tasks.

[0059] In various implementations, hardware manager 112 can assign different types of hardware devices (e.g., GPUs, deep learning accelerators (DLAs)) to redundantly perform the same inference task for an inference task (e.g., object detection). For example, a first inference task can be run multiple times on different hardware devices to improve the detectability of faults in each hardware 120, which complements the fault detection capabilities of hardware 120.

[0060] In various implementations, hardware manager 112 collects activation patterns from hardware 120 by inserting diagnostic nodes into the task flow. Activation patterns describe how different components of hardware 120 (e.g., texture units, MIGs) are used during multiple tasks. Based on the collected activation patterns, hardware manager 112 can add additional runtime diagnostics to hardware 120. For example, if an activation pattern does not meet expectations and / or fails to meet activation pattern thresholds, hardware manager 112 adds additional diagnostic nodes to the task flow to assess whether a fault exists within hardware 120. Reports on different profiles of hardware 120 regarding security and performance can then be generated based on the runtime diagnostics received from the diagnostic nodes. The user can then select one of the different profiles for hardware 120 to run. For example, if the activation patterns indicate that hardware 120 prioritizes performance over security due to a lack of implementations such as converters, the user can adjust the profile accordingly.

[0061] In various implementations, user input may include a task flow description, dependencies between tasks, and / or at least one or all of the target hardware on which the task flow is executed. For example, the user can determine the configuration of the task flow and which tasks depend on each other. In this case, based on the task flow provided by the flow generator 108, the user can modify the task flow according to their preferences. Furthermore, the user can select on which target hardware 120 and / or components of target hardware 120 to execute multiple tasks and / or nodes. The user can provide input to the hardware manager 112 to execute the task flow accordingly.

[0062] In various implementations, the task flow executes multiple nodes on physical components of the hardware device. For example, if the hardware device contains multiple streaming multiprocessors, in a redundant mode, the task flow can execute multiple nodes in parallel, assigning each of the multiple nodes to a streaming multiprocessor of the hardware device.

[0063] Figure 2 Based on some implementation schemes disclosed herein, it can be made by Figure 1 A schematic diagram illustrating an example of the system-generated and executed task flow 200. Figure 2 In this context, task flow 200 begins with a non-redundant node in a first context C0. This non-redundant node runs on full hardware (e.g., unpartitioned mode) and can also be the first instance of the first task in a plurality of tasks. Task flow 200 includes a first converter 204, which is executed after the non-redundant node (the task) completes. The first converter 204 can serve as both a context converter and a partition configuration change (e.g., switching from unpartitioned mode to partitioned mode). Figure 2 As shown, M1 (e.g., the first mission node) and R1 (e.g., the first redundant node) are partitioned and can run on separate hardware components of the same device (e.g., the MIG of the GPU). The first mission node and the first redundant node reside in the second context C1 and the third context C2, respectively. The converter may also include synchronization objects, enabling the first mission node and the first redundant node to operate in parallel. This ensures that the first redundant node can take over in the event of a failure of the first mission node.

[0064] After the tasks of the first mission node and the first redundant node have been completed, a second converter 212 may be included. The second converter 212 can switch contexts. In response to the stream generator 108 determining that a redundant task (e.g., using a second mission node and a second redundant node) should be performed after the tasks of the first mission node and the first redundant node have been executed, the second converter 212 may be included.

[0065] like Figure 2 As seen in the diagram, the first mission node can then be moved to the second mission node M2, while the first redundant node can be moved to the second redundant node R2. The second mission node can execute on the same hardware and context as the first mission node, and the second redundant node can execute on the same hardware and context as the second mission node. In various embodiments, the converter 212 switches contexts, and the second redundant node and the second mission node execute on different contexts than the first redundant node and the first mission node.

[0066] In response to the completion of the tasks on the second mission node and the second redundant node, the third converter 208 switches the context and partition configuration (e.g., from partitioned mode to unpartitioned mode). The third converter 208 may be implemented in response to, for example, a task flow switching from mission nodes and redundant nodes to non-redundant nodes. The third converter 208 may instruct, for example, the hardware manager 112 to switch from partitioning and running on the first and second MIGs to running on the full GPU.

[0067] In various implementations, a task flow (e.g., task flow 200) may include two or more task nodes, redundant nodes, and / or non-redundant nodes in the task flow graph. In this case, two or more converters may be implemented to accommodate multiple instances of the nodes.

[0068] In various implementations, a task flow can include both temporal and spatial redundancy. For example, a task flow can begin with two or more nodes (e.g., non-redundant, redundant, or mission-based) that execute sequentially (e.g., temporally redundant) and can move to two or more nodes that execute in parallel on one or more components of the hardware (e.g., spatially redundant).

[0069] Now for reference Figure 3 Each block of the method 300 described herein contains a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. These methods can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by a standalone application, service, or managed service (standalone or in combination with other managed services), or a plug-in to another product, etc. Furthermore, method 300 is designed for… Figure 1 The system described herein is an example. However, these methods can be implemented by any other or alternative system or combination of systems, including but not limited to the system described herein.

[0070] At block 302 of method 300, a first task among multiple tasks is determined to meet the criteria for execution in a redundant mode. This criterion may be associated with whether the first task should be redundant (e.g., run on multiple nodes). A redundant task could be, for example, controlling a vehicle's braking system. For safety reasons, braking-related tasks would meet the criteria for redundant execution.

[0071] At box 304, a task flow for performing multiple tasks is determined. This is based on the number of tasks and whether each of the tasks meets the criterion. The task flow can be determined based on redundant and non-redundant tasks. An example task flow can be found in... Figure 2As can be seen, since the first task is determined to meet the criteria, a converter can be assigned before the first task is executed. The converter can switch the task flow from execution in a non-redundant mode (e.g., all-hardware) to execution in a redundant mode (e.g., running tasks on separate components of hardware). The converter can divide the hardware on which multiple tasks are being executed into a first partition and a second partition (e.g., partition 116) and enter redundant mode. Therefore, converters in the task flow can allow redundant tasks and / or mission tasks to run in parallel. In various implementations, the converter also switches contexts.

[0072] At box 306, multiple tasks are executed according to the task flow. Multiple tasks can be executed after the task flow is determined and a switch is implemented. Multiple tasks can be executed by executing a first instance of a first task on a first partition and a second instance of the first task on a second partition. For example, in a single partition (e.g., multiple MIGs on the same GPU), the first instance can be executed on a redundant node, and the second instance can run on a mission node. Multiple tasks can begin, for example, on a non-redundant node running on all GPUs, and then encounter a converter, causing the task flow to execute on partitioned hardware. In redundancy mode (e.g., partitioned hardware), tasks can run concurrently and be synchronized via converters, such that tasks in redundancy mode complete simultaneously when they encounter a second converter or a second non-redundant node. Tasks in redundancy mode can be configured to complete simultaneously without encountering a second converter or a second non-redundant node.

[0073] The systems and methods described herein can, and are not limited to, be used by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), manned and unmanned robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, ships, space shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, the systems and methods described herein can be used for a variety of purposes, including, but not limited to, machine control, machine motion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, safety and supervision, simulation and digital twins, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or participant simulation and / or digital twins, data center processing, conversational AI, optical transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or any other suitable application.

[0074] The disclosed embodiments can be included in a variety of different systems, such as automotive systems (e.g., control systems for autonomous or semi-autonomous machines, perception systems for autonomous or semi-autonomous machines), systems implemented using robots, aviation systems, medical systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using edge devices, systems containing one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for hosting real-time streaming applications, systems for presenting one or more of virtual reality content, augmented reality content, or mixed reality content, systems for performing optical transmission simulations, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0075] Example autonomous vehicles

[0076] Figure 4AThis is an illustration of an example autonomous vehicle 400 according to some embodiments of the present disclosure. The autonomous vehicle 400 (or, alternatively, referred to herein as “vehicle 400”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, ambulances, shuttles, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, boats, engineering vehicles, underwater vessels, robotic vehicles, drones, aircraft, vehicles coupled to trailers (e.g., semi-trailer trucks for transporting goods), and / or other types of vehicles (e.g., driverless and / or capable of accommodating one or more passengers). Autonomous vehicles are typically described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) in its "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201506, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 400 is capable of performing one or more functions that meet Level 3 through Level 5 of autonomous driving. Vehicle 400 is capable of performing one or more functions that meet Level 1 through Level 5 of automated driving. For example, depending on the embodiment, vehicle 400 may be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). As used herein, the term “autonomy” can include any and / or all types of autonomy for the 400 or other machines, such as full autonomy, high autonomy, conditional autonomy, partial autonomy, providing auxiliary autonomy, semi-autonomy, primary autonomy, or other designations. Vehicle 400 may include ASILs for the various systems further described herein.

[0077] Vehicle 400 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 400 may include a propulsion system 450, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 450 may be connected to the drivetrain of vehicle 400, which may include a transmission, to enable propulsion of vehicle 400. Propulsion system 450 may be controlled in response to receiving a signal from throttle / accelerator 452.

[0078] A steering system 454, which may include a steering wheel, can be used to steer the vehicle 400 (e.g., along a desired path or route) when the propulsion system 450 is operating (e.g., when the vehicle is in motion). The steering system 454 may receive signals from the steering actuator 456. For fully automatic (level 5) functions, the steering wheel may be optional.

[0079] The brake sensor system 446 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 448 and / or the brake sensor.

[0080] It can include one or more System-on-Chip (SoC) 404 ( Figure 4C One or more controllers 436, including and / or one or more GPUs, may provide signals (e.g., signals representing commands) to one or more components and / or systems of vehicle 400. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 448, to operate steering system 454 via one or more steering actuators 456, and to operate propulsion system 450 via one or more throttles / accelerators 452. One or more controllers 436 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 400. One or more controllers 436 may include a first controller 436 for autonomous driving functions, a second controller 436 for functional safety functions, a third controller 436 for artificial intelligence functions (e.g., computer vision), a fourth controller 436 for infotainment functions, a fifth controller 436 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 436 can handle two or more of the functions described above, and two or more controllers 436 can handle a single function, and / or any combination thereof.

[0081] One or more controllers 436 may provide signals for controlling one or more components and / or systems of vehicle 400 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, Global Navigation Satellite System (“GNSS”) sensors 458 (e.g., Global Positioning System sensors), RADAR sensors 460, ultrasonic sensors 462, LIDAR sensors 464, inertial measurement unit (IMU) sensors 466 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 796, stereo cameras 468, wide-angle cameras 470 (e.g., fisheye cameras), infrared cameras 472, surround cameras 474 (e.g., 360-degree cameras), long-range and / or medium-range cameras 498, speed sensors 444 (e.g., for measuring the rate of vehicle 400), vibration sensors 442, steering sensors 440, braking sensors (e.g., as part of braking sensor system 446), and / or other sensor types.

[0082] One or more of the controllers 436 may receive inputs (e.g., represented by input data) from the instrument cluster 432 of the vehicle 400 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 434, an auditory signaling device, a speaker, and / or via other components of the vehicle 400. These outputs may include information such as vehicle speed, rate, time, map data (e.g., [missing information]). Figure 4C Information such as a high-resolution (“HD”) map 422, location data (e.g., the location of vehicle 400, such as its position on the map), orientation, the location of other vehicles (e.g., occupying a grid), and information about objects and their states perceived by controller 436, etc. For example, HMI display 434 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0083] Vehicle 400 further includes a network interface 424, which can communicate via one or more networks using one or more wireless antennas 426 and / or a modem. For example, network interface 424 may be able to communicate via Long Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile Communications (“GSM”), IMT-CDMA Multicarrier (“CDMA2000”), etc. One or more wireless antennas 426 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (“LPWAN”) such as LoRaWAN, SigFox, etc.

[0084] To meet the ASIL requirements of various applications, vehicle 400 can use system 100 to manage fault detection and efficiently process and execute the applications of the various systems of vehicle 400.

[0085] Figure 4B For use in accordance with some embodiments of this disclosure Figure 4A This is an example of the camera position and field of view of an autonomous vehicle 400. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 400.

[0086] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 400. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity.

[0087] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0088] One or more of the cameras can be mounted in mounting components such as custom-designed (3D-printed) components to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0089] A camera with a field of view that includes the environment in front of the vehicle 400 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 436 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (“LDW”), Autonomous Cruise Control (“ACC”), and / or other functions such as traffic sign recognition.

[0090] A variety of cameras can be used in front-facing configurations, including, for example, monocular camera platforms including complementary metal-oxide-semiconductor (“CMOS”) color imagers. Another example could be a wide-angle camera 470, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although Figure 4B The diagram shows only one wide-angle camera, but any number (including zero) of wide-angle cameras 470 can be present on vehicle 400. Furthermore, any number of remote cameras 498 (e.g., long-view stereo camera pairs) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. Remote cameras 498 can also be used for object detection and classification, as well as basic object tracking.

[0091] Any number of stereo cameras 468 may also be included in the front-mounted configuration. In at least one embodiment, one or more stereo cameras 468 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (“FPGA”) with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 468 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 468 may be used in addition to those described herein or alternatively.

[0092] Cameras with a field of view including the side portion of the vehicle 400 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 474 (e.g., ... Figure 4B The four surround cameras 474 shown can be mounted on the vehicle 400. The surround cameras 474 can include wide-angle cameras 470, fisheye cameras, 360-degree cameras, and / or the like. For example, four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 474 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround-view camera.

[0093] A camera with a field of view that includes the environment behind the vehicle 400 (e.g., a rear-view camera) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 498, stereo camera 468, infrared camera 472, etc.).

[0094] Figure 4C For use in accordance with some embodiments of this disclosure Figure 4AThe example autonomous vehicle 400 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented via hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0095] Figure 4C Each component, feature, and system in vehicle 400 is illustrated as being connected via bus 402. Bus 402 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 400 used to assist in the control of various features and functions of vehicle 400, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CANID). The CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0096] Although bus 402 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 402 is represented by a single line, this is not intended to be limiting. For example, any number of buses 402 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 402 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 402 may be used for a collision avoidance function, and a second bus 402 may be used for drive control. In any example, each bus 402 may communicate with any component of vehicle 400, and two or more buses 402 may communicate with the same component. In some examples, each SoC 404, each controller 436, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of vehicle 400) and may be connected to a common bus such as a CAN bus.

[0097] Vehicle 400 may include one or more controllers 436, such as those described herein. Figure 4A The controllers described. Controller 436 can be used for a wide variety of functions. Controller 436 can be coupled to any other different components and systems of vehicle 400 and can be used for the control of vehicle 400, artificial intelligence of vehicle 400, infotainment and / or the like for vehicle 400.

[0098] Vehicle 400 may include one or more System-on-Chip (SoC) 404. SoC 404 may include a CPU 406, GPU 408, processor 410, cache 412, accelerator 414, data storage 416, and / or other components and features not shown. SoC 404 can be used to control vehicle 400 across a wide variety of platforms and systems. For example, one or more SoCs 404 may be combined with an HD map 422 in a system (e.g., the system of vehicle 400), the HD map being transmitted via a network interface 424 from one or more servers (e.g., [server name missing]). Figure 4D One or more servers (478) receive map refresh and / or updates.

[0099] CPU 406 may include a CPU cluster or CPU complex (or, alternatively, referred to herein as "CCPLEX"). CPU 406 may include multiple cores and / or L2 cache. For example, in some embodiments, CPU 406 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 406 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., 2MB L2 cache). CPU 406 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of CPU 406 can be active at any given time.

[0100] CPU 406 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. CPU 406 can further implement enhanced algorithms for managing power states, wherein allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0101] GPU 408 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). GPU 408 may be programmable and efficient for parallel workloads. In some examples, GPU 408 may use an enhanced tensor instruction set. GPU 408 may include one or more streaming microprocessors, wherein each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, GPU 408 may include at least eight streaming microprocessors. GPU 408 may use a computation application programming interface (API). Furthermore, GPU 408 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0102] In automotive and embedded applications, the GPU 408 can be power-optimized for optimal performance. For example, the GPU 408 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 408 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, dispatch units, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to provide efficient execution of workloads by leveraging the mixture of computation and addressing computation. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors can include a combination of L1 data cache and shared memory units to improve performance while simplifying programming.

[0103] The GPU 408 may include, in some examples, a High Bandwidth Memory (HBM) and / or a 16GB HBM2 memory subsystem providing a peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, Synchronous Graphics Random Access Memory (SGRAM), such as Generation 5 Graphics Double Data Rate Synchronous Random Access Memory (GDDR5), may be used.

[0104] The GPU 408 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 408 to directly access the CPU 406 page tables. In such examples, when the GPU 408 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 406. In response, the CPU 406 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 408. In this way, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 406 and the GPU 408, thereby simplifying GPU 408 programming and porting applications to the GPU 408.

[0105] In addition, the GPU 408 may include access counters that track how frequently the GPU 408 accesses the memory of other processors. These access counters help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0106] SoC 404 may include any number of caches 412, including those described herein. For example, cache 412 may include an L3 cache available to both CPU 406 and GPU 408 (e.g., it connects both CPU 406 and GPU 408). Cache 412 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4 MB or more, but a smaller cache size may also be used.

[0107] SoC 404 may include one or more arithmetic logic units (ALUs) that can be used to perform processing of any of a variety of tasks or operations relating to vehicle 400, such as processing a DNN. Additionally, SoC 404 may include a floating-point unit (FPU) or other mathematical coprocessor or digital coprocessor type for performing mathematical operations within the system. For example, SoC 104 may include one or more FPUs integrated as execution units within CPU 406 and / or GPU 408.

[0108] SoC 404 may include one or more accelerators 414 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 404 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to supplement GPU 408 and offload some tasks from GPU 408 (e.g., freeing up more cycles of GPU 408 to perform other tasks). As an example, accelerator 414 can be used for targeted workloads (e.g., perceptrons, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0109] Accelerator 414 (e.g., a hardware acceleration cluster) may include a Deep Learning Accelerator (DLA). The DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than a general-purpose GPU and significantly outperform CPUs. The TPU can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0110] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0111] The DLA can perform any function of the GPU 408, and by using inference accelerators, for example, a designer can make either the DLA or the GPU 408 target any function. For example, a designer can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 408 and / or other accelerators 414.

[0112] Accelerator 414 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0113] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0114] DMA enables PVA components to access system memory independently of the CPU 406. DMA can support any number of features used to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0115] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0116] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. Consequently, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a hardware-accelerated cluster, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error-correcting code (ECC) memory to enhance overall system security.

[0117] Accelerator 414 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 414. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks, accessible by both the PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network that interconnects the PVA and DLA to memory.

[0118] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0119] In some examples, SoC 404 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0120] Accelerators 414 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are efficient in object detection and integer mathematical operations.

[0121] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3–5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0122] In some examples, PVA can be used to perform intensive optical flow, processing raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide processed RADAR. In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0123] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network to regress the confidence value. This neural network can take at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimates (e.g., from another subsystem), inertial measurement unit (IMU) sensor 466 outputs related to vehicle orientation and distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 464 or RADAR sensor 460), etc.

[0124] SoC 404 may include one or more data stores 416 (e.g., memory). Data stores 416 may be on-chip memory of SoC 404, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, data stores 416 may be large enough to store multiple instances of the neural network. Data stores 412 may include L2 or L3 caches 412. References to data stores 416 may include references to memory associated with PVA, DLA, and / or other accelerators 414 as described herein.

[0125] SoC 404 may include one or more processors 410 (e.g., embedded processors). Processor 410 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 404 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 404 thermal and temperature sensor management, and / or SoC 404 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 404 may use the ring oscillator to detect the temperature of CPU 406, GPU 408, and / or accelerator 414. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 404 into a lower power state and / or place vehicle 400 into a driver-safe parking mode (e.g., safely stop vehicle 400).

[0126] Processor 410 may further include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces, as well as a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0127] The processor 410 may further include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0128] Processor 410 may further include a secure cluster engine, which includes a dedicated processor subsystem for handling security management of automotive applications. The secure cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0129] The processor 410 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0130] The processor 410 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0131] Processor 410 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 470, the surround camera 474, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0132] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0133] The video image compositer can also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 408 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 408 is powered on and active, performing 3D rendering, the video image compositer can be used to offload the GPU 408 to improve performance and responsiveness.

[0134] SoC 404 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions for receiving video and input from a camera. SoC 404 may further include an input / output controller that can be software-controlled and can be used to receive I / O signals not assigned to a specific role.

[0135] SoC 404 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 404 can be used to process data from cameras and sensors (e.g., LIDAR sensor 464, RADAR sensor 460, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 402 (e.g., vehicle 400 speed, steering wheel position, etc.), and data from GNSS sensor 458 (connected via Ethernet or CAN bus). SoC 404 may further include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine, and which can be used to free up CPU 406 from routine data management tasks.

[0136] SoC 404 can be an end-to-end platform with a flexible architecture spanning Levels 3-5 of automation, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools, to deliver a flexible and reliable driving software stack. SoC 404 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with CPU 406, GPU 408, and data storage 416, accelerator 414 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0137] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0138] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 420) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could further include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0139] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions" along with a light can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network that informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, informing the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 408.

[0140] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 400. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 404 provides security against theft and / or carjacking.

[0141] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 796 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 404 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to recognize the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 458. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 462, the control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0142] The vehicle may include a CPU 418 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 404 via a high-speed interconnect (e.g., PCIe). The CPU 418 may include, for example, an x86 processor. The CPU 418 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 404, and / or monitoring the status and health of the controller 436 and / or the infotainment SoC 430.

[0143] Vehicle 400 may include a GPU 420 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 404 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 420 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based on inputs from sensors of vehicle 400 (e.g., sensor data).

[0144] Vehicle 400 may further include a network interface 424, which may include one or more wireless antennas 426 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 424 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 478 and / or other network devices), with other vehicles, and / or with computing devices (e.g., a passenger's client device). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 400 with information about vehicles approaching vehicle 400 (e.g., vehicles in front, to the side, and / or behind vehicle 400). This functionality can be part of vehicle 400's cooperative adaptive cruise control function.

[0145] Network interface 424 may include a SoC that provides modulation and demodulation functions and enables controller 436 to communicate over a wireless network. Network interface 424 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0146] Vehicle 400 may further include data storage 428, which may include off-chip (e.g., off-chip SoC 404) storage devices. Data storage 428 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0147] Vehicle 400 may further include a GNSS sensor 458. The GNSS sensor 458 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used for auxiliary mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 458 can be used, including, for example, but not limited to, GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0148] Vehicle 400 may further include a RADAR sensor 460. The RADAR sensor 460 can be used by vehicle 400 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 460 can use CAN and / or bus 402 (e.g., to transmit data generated by the RADAR sensor 460) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 460 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0149] The RADAR sensor 460 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, the long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. The RADAR sensor 460 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assistance and forward collision warning. The long-range RADAR sensor can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 400's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 400's lane.

[0150] As an example, a mid-range RADAR system can include a range of up to 460m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 450 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0151] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0152] Vehicle 400 may further include ultrasonic sensors 462. Ultrasonic sensors 462, which may be positioned at the front, rear, and / or sides of vehicle 400, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 462 can be used, and different ultrasonic sensors 462 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 462 can operate at functional safety level ASIL B.

[0153] Vehicle 400 may include a LIDAR sensor 464. The LIDAR sensor 464 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 464 may be of functional safety level ASIL B. In some examples, vehicle 400 may include multiple LIDAR sensors 464 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0154] In some examples, the LiDAR sensor 464 may be able to provide a list of objects and their distances within a 360-degree field of view. Commercially available LiDAR sensors 464 may have an advertising range of, for example, approximately 400m, with an accuracy of 2cm-3cm, and support for 400Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 464 may be used. In such examples, the LiDAR sensor 464 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 400. In such examples, the LiDAR sensor 464 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. Front-mounted LiDAR sensors 464 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0155] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. A flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-browsing LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 464 is less susceptible to motion blur, vibration, and / or shock.

[0156] The vehicle may further include an IMU sensor 466. In some examples, the IMU sensor 466 may be located at the center of the rear axle of the vehicle 400. The IMU sensor 466 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 466 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 466 may include an accelerometer, a gyroscope, and a magnetometer.

[0157] In some embodiments, the IMU sensor 466 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 466 can enable the vehicle 400 to estimate heading by directly observing and correlating velocity changes from GPS to the IMU sensor 466 without input from a magnetic sensor. In some examples, the IMU sensor 466 and the GNSS sensor 458 can be combined into a single integrated unit.

[0158] The vehicle may include a microphone 496 placed in and / or around the vehicle 400. Among other things, the microphone 496 may be used for emergency vehicle detection and identification.

[0159] The vehicle may further include any number of camera types, including stereo camera 468, wide-angle camera 470, infrared camera 472, surround camera 474, long-range and / or mid-range camera 498, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 400. The types of cameras used depend on the embodiment and the requirements of the vehicle 400, and any combination of camera types can be used to provide the necessary coverage around the vehicle 400. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described herein with respect to... Figure 4A and Figure 4B It was described in more detail.

[0160] Vehicle 400 may further include vibration sensor 442. Vibration sensor 442 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 442 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between a power drive shaft and a free-rotating shaft).

[0161] Vehicle 400 may include ADAS system 438. In some examples, ADAS system 438 may include SoC. ADAS system 438 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0162] The ACC system can use a RADAR sensor 460, a LIDAR sensor 464, and / or a camera. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 400 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 400 to change lanes. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0163] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or network connection (e.g., via the Internet) through network interface 424 and / or wireless antenna 426. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 400 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both I2V and V2V information sources. Given information about vehicles ahead of vehicle 400, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0164] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0165] An AEB (Autonomous Emergency Braking) system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0166] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses a lane marking. When the driver indicates intentional lane departure, the LDW system is deactivated by activating a turn signal. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0167] The LKA system is a variation of the LDW system. If vehicle 400 begins to leave the lane, the LKA system provides corrective steering input or braking to vehicle 400.

[0168] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses turn signals. The BSW system may use one or more rear-facing cameras and / or one or more RADAR sensors 460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically connected to driver feedback devices (such as displays, speakers, and / or vibration components).

[0169] RCTW systems can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of a rear-view camera while the vehicle is reversing at 400 degrees. Some RCTW systems include AEB (Autonomous Emergency Braking) to ensure the application of the vehicle's brakes to avoid a collision. RCTW systems may use one or more rear-view RADAR sensors 460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0170] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as ADAS systems alert the driver and allow them to determine whether a safe condition truly exists and take appropriate action. However, in an autonomous vehicle 400, in the event of conflicting results, the vehicle 400 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 436 or the second controller 436). For example, in some embodiments, ADAS system 438 may be a backup and / or auxiliary computer used to provide perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and diverse software on hardware components to detect faults in perception and dynamic driving tasks. Outputs from ADAS system 438 may be provided to a supervisory MCU. If outputs from the main computer and auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0171] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0172] The supervisory MCU can be configured to run a neural network trained and configured to determine the conditions under which the auxiliary computer provides a false alarm based on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include components of SoC 404 and / or be included as components of SoC 404.

[0173] In other examples, ADAS system 438 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0174] In some examples, the output of ADAS system 438 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 438 issues a forward collision warning because an object is immediately in front, the perception block can use this information when identifying the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0175] Vehicle 400 may further include an infotainment SoC 430 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 430 may include a combination of hardware and software that can be used to provide vehicle 400 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.) and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 430 may be a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 434, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 430 may further be used to provide information (e.g., visual and / or auditory) to the vehicle's users, such as information from ADAS system 438, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0176] The infotainment SoC 430 may include GPU functionality. The infotainment SoC 430 can communicate with other devices, systems, and / or components of the vehicle 400 via bus 402 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 430 may be coupled to a supervisory MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 436 (e.g., the primary and / or backup computer of the vehicle 400). In such an example, the infotainment SoC 430 may place the vehicle 400 into a driver-safe parking mode as described herein.

[0177] Vehicle 400 may further include an instrument cluster 432 (e.g., a digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). The instrument cluster 432 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 432 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 430 and the instrument cluster 432. In other words, the instrument cluster 432 may be included as part of the infotainment SoC 430, or vice versa.

[0178] Figure 4D For cloud-based servers and according to some embodiments of this disclosure Figure 4A This is a system diagram illustrating communication between example autonomous vehicles 400. System 476 may include server 478, network 490, and vehicles including vehicle 400. Server 478 may include multiple GPUs 484(A)-484(H) (collectively referred to herein as GPU 484), PCIe switches 482(A)-482(H) (collectively referred to herein as PCIe switch 482), and / or CPUs 480(A)-480(B) (collectively referred to herein as CPU 480). GPUs 484, CPUs 480, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 486, such as, but not limited to, NVLink interface 488 developed by NVIDIA. In some examples, GPUs 484 are connected via NVLink and / or NVSwitch SoCs, and GPUs 484 and PCIe switches 482 are connected via PCIe interconnects. Although eight GPUs 484, two CPUs 480, and two PCIe switches are shown in the diagram, this is not intended to be limiting. Depending on the embodiment, each of the servers 478 may include any number of GPUs 484, CPUs 480, and / or PCIe switches. For example, each of the servers 478 may include eight, sixteen, thirty-two, and / or more GPUs 484.

[0179] Server 478 can receive image data from vehicles via network 490, representing images of unexpected or altered road conditions such as recently commenced roadworks. Server 478 can also transmit neural network 492, updated neural network 492, and / or map information 494, including information about traffic and road conditions, to vehicles via network 490. Updates to map information 494 may include updates to HD map 422, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 492, updated neural network 492, and / or map information 494 may have been generated from new training and / or data received from any number of vehicles in the environment, and / or based on experience gained from training performed at a data center (e.g., using server 478 and / or other servers).

[0180] Server 478 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component analysis and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to the vehicle via network 490), and / or the machine learning model can be used by server 478 to remotely monitor the vehicle.

[0181] In some examples, server 478 can receive data from vehicles and apply that data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 478 may include a deep learning supercomputer powered by GPU 484 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 478 may include a deep learning infrastructure in a data center that uses only CPU power.

[0182] The deep learning infrastructure of server 478 may be capable of rapid, real-time inference and can be used to assess and verify the health status of the processor, software, and / or associated hardware in vehicle 400. For example, the deep learning infrastructure may receive periodic updates from vehicle 400, such as image sequences and / or objects located in those image sequences by vehicle 400 (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 400. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 400 has malfunctioned, then server 478 may transmit a signal to vehicle 400 instructing the vehicle 400's fail-safe computer to take control, notify passengers, and complete a safe stopping operation.

[0183] For inference, server 478 may include GPU 484 and one or more programmable inference accelerators (such as NVIDIA's TensorRT 3). The combination of a GPU-powered server and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0184] Example computing device

[0185] Figure 5 The block diagram is provided for an example computing device 500 suitable for implementing some embodiments of the present disclosure. The computing device 500 may include an interconnect system 502 directly or indirectly coupled to the following devices: memory 504, one or more central processing units (CPUs) 506, one or more graphics processing units (GPUs) 508, a communication interface 510, input / output (I / O) ports 512, input / output components 514, a power supply 516, one or more presentation components 518 (e.g., a display), and one or more logic units 520. In at least one embodiment, the computing device 500 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 508 may include one or more vGPUs, one or more CPUs 506 may include one or more vCPUs, and / or one or more logic units 520 may include one or more virtual logic units. Therefore, computing device 500 may include discrete components (e.g., a complete GPU dedicated to computing device 500), virtual components (e.g., a portion of the GPU dedicated to computing device 500), or a combination thereof. For example, non-partitioned mode of system 100 may be executed on discrete components, while partitioned mode of system 100 may be executed on virtual components.

[0186] although Figure 5 The various boxes are shown connected via an interconnect system 502 with wiring, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 518, such as a display device, can be considered an I / O component 514 (e.g., if the display is a touchscreen). As another example, CPU 506 and / or GPU 508 may include memory (e.g., memory 504 may represent a storage device other than the memory of GPU 508, CPU 506, and / or other components). In other words, Figure 5 The computing devices mentioned are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all of these are considered within the same category. Figure 5 Within the scope of computing devices.

[0187] Interconnect system 502 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 502 may include one or more link or bus types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Fast (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 506 may be directly connected to memory 504. Furthermore, CPU 506 may be directly connected to GPU 508. In cases where there is a direct or point-to-point connection between components, interconnect system 502 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required in computing device 500.

[0188] Memory 504 may include any medium of a wide variety of computer-readable media. Computer-readable media can be any available medium that can be accessed by computing device 500. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. For example and without limitation, computer-readable media may include computer storage media and communication media.

[0189] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media, implemented in any way or by any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 504 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by computing device 500. As used herein, computer storage media does not include the signal itself.

[0190] Computer storage media may contain computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and include any information transport medium. The term "modulated data signal" can refer to a signal whose characteristics are set or altered in a manner that encodes information into that signal. For example and without limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0191] CPU 506 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 500 to perform one or more of the methods and / or processes described herein. Each of CPU 506 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a large number of software threads simultaneously. CPU 506 may include any type of processor and may include different types of processors depending on the type of computing device 500 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 500, the processor may be an advanced RISC mechanism (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, computing device 500 may also include one or more CPUs 506.

[0192] In addition to or replacing CPU 506, GPU 508 may also be configured to execute at least some computer-readable instructions to control one or more components of computing device 500 to perform one or more of the methods and / or processes described herein. One or more GPUs 508 may be integrated GPUs (e.g., having one or more CPUs 506) and / or one or more GPUs 508 may be discrete GPUs. In embodiments, one or more GPUs 508 may be coprocessors of one or more CPUs 506. Computing device 500 may use GPU 508 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPU 508 may be used for general-purpose computing on a GPU (GPGPU). GPU 508 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU 508 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received via a host interface from CPU 506). GPU 508 may include graphics memory, such as display memory, for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 504. GPU 508 may include two or more GPUs operating in parallel (e.g., via links). The links may connect the GPUs directly (e.g., using NVLINK) or via a switch (e.g., using NVSwitch). When combined, each GPU 508 may generate different portions of pixel data or GPGPU data for different outputs (e.g., the first GPU for the first image, the second GPU for the second image). Each GPU may include its own memory or may share memory with other GPUs.

[0193] In addition to or replacing CPU 506 and / or GPU 508, logic unit 520 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 500 to perform one or more methods and / or processes described herein. In embodiments, CPU 506, GPU 508, and / or logic unit 520 may execute any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 520 may be part of and / or integrated into one or more CPUs 506 and / or one or more GPUs 508, and / or one or more logic units 520 may be discrete components of CPU 506 and / or GPU 508 or otherwise external thereto. In embodiments, one or more logic units 520 may be processors of one or more CPUs 506 and / or one or more GPUs 508. For example, hardware manager 112 may utilize GPU 508 to execute applications and multiple tasks in a task flow.

[0194] Examples of logic unit 520 include one or more processing cores and / or components thereof, such as data processing unit (DPU), tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU)), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, etc.

[0195] The communication interface 510 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 500 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The communication interface 510 may include components and functions that enable communication via any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication via Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 520 and / or the communication interface 510 may include one or more data processing units (DPUs) to directly transmit data received via a network and / or via interconnect system 502 to one or more GPUs 508 (e.g., memory within GPU 508).

[0196] I / O port 512 enables computing device 500 to be logically coupled to other devices, including I / O component 514, presentation component 518, and / or other components, some of which may be built into (e.g., integrated into) computing device 500. Illustrative I / O component 514 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, browsers, printers, wireless devices, and so on. I / O component 514 can provide a Natural User Interface (NUI) for processing user-generated air gestures, voice, or other physiological input. In some instances, the input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 500 (as described in more detail below). Computing device 500 may include depth cameras such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof for gesture detection and recognition. In addition, computing device 500 may include an accelerometer or gyroscope that enables motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 500 to render immersive augmented reality or virtual reality.

[0197] Power supply 516 may include a hard-wired power supply, a battery power supply, or a combination thereof. Power supply 516 may supply power to computing device 500 so that the components of computing device 500 can operate.

[0198] The presentation component 518 may include a display (such as a monitor, touch screen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 518 may receive data from other components (such as GPU 508, CPU 506, DPU, etc.) and output that data (e.g., as images, videos, sounds, etc.).

[0199] Example Data Center

[0200] Figure 6 An example data center 600 is illustrated, which can be used in at least one embodiment of this disclosure. Data center 600 may include a data center infrastructure layer 610, a framework layer 620, a software layer 630, and an application layer 640. Application layer 640 may be application layer 104.

[0201] like Figure 6As shown, the data center infrastructure layer 610 may include a resource coordinator 612, grouped computing resources 614, and node computing resources (“nodes CR”) 616(1)-616(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CR 616(1)-616(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state drives or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and cooling modules, etc. In some embodiments, one or more nodes CR 616(1)-616(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR616(1)-616(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more of nodes CR616(1)-616(N) may correspond to virtual machines (VMs).

[0202] In at least one embodiment, the grouped computing resources 614 may include individual groups (not shown) of nodes CR616 housed in one or more racks, or a plurality of racks (also not shown) housed in data centers in various geographic locations. Individual groups of nodes CR616 within the grouped computing resources 614 may include computing, networking, memory, or storage resources that can be configured or allocated to support groups of one or more workloads. In at least one embodiment, several nodes CR616, including CPUs, GPUs, DPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0203] Resource coordinator 612 can be configured or otherwise controlled to control one or more nodes CR616(1)-616(N) and / or groups of computing resources 614. In at least one embodiment, resource coordinator 612 may include a Software Design Infrastructure (SDI) management entity for data center 600. Resource coordinator 612 may include hardware, software, or some combination thereof.

[0204] In at least one embodiment, such as Figure 6As shown, framework layer 620 may include a job scheduler 633, a configuration manager 634, a resource manager 636, and a distributed file system 638. Framework layer 620 may include a framework of software 632 supporting software layer 630 and / or one or more applications 642 of application layer 640. Software 632 or application 642 may respectively include web-based service software or applications, such as service software or applications provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 620 may be, but is not limited to, a free and open-source software web application framework, such as Apache Spark, which can utilize distributed file system 638 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark"). In at least one embodiment, the job scheduler 633 may include a Spark driver for facilitating the scheduling of workloads supported by various layers of data center 600. In at least one embodiment, the configuration manager 634 may be able to configure different layers, such as software layer 630 and framework layer 620 including Spark and a distributed file system 638 for supporting large-scale data processing. The resource manager 636 is able to manage cluster or grouped computing resources mapped to or allocated for supporting distributed file system 638 and job scheduler 633. In at least one embodiment, cluster or grouped computing resources may include grouped computing resources 614 at data center infrastructure layer 610. The resource manager 636 may coordinate with resource coordinator 612 to manage these mapped or allocated computing resources.

[0205] In at least one embodiment, the software 632 included in the software layer 630 may include software used by at least a portion of the nodes CR616(1)-616(N), the grouped computing resources 614, and / or the distributed file system 638 of the framework layer 620. One or more types of software may include, but are not limited to, Internet web page search software, email virus browsing software, database software, and streaming video content software.

[0206] In at least one embodiment, one or more applications 642 included in application layer 640 may include one or more types of applications used by at least a portion of nodes CR616(1)-616(N), grouped computing resources 614, and / or the distributed file system 638 of framework layer 620. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments. Applications 642 in application layer 642 may execute and control the systems of vehicle 400 and contain different ASILs.

[0207] In at least one embodiment, any of the configuration manager 634, resource manager 636, and resource coordinator 612 can perform any number and type of self-modification actions based on any amount and type of data acquired in any technically feasible manner. Self-modification actions can mitigate potentially poor configuration decisions by data center operators of data center 600 and can prevent underutilization and / or skewed portions of the data center.

[0208] Data center 600 may include tools, services, software, or other resources for training one or more machine learning models or using one or more machine learning models to predict or infer information according to one or more embodiments described herein. For example, a machine learning model can be trained by calculating weight parameters based on a neural network architecture using the software and computing resources described above regarding data center 600. In at least one embodiment, by using weight parameters calculated through one or more training techniques, information can be inferred or predicted using trained machine learning models corresponding to one or more neural networks, such as, but not limited to, those described herein, using the resources described above regarding data center 600.

[0209] In at least one embodiment, the data center 600 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured as a service to allow a user to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services.

[0210] Example network environment

[0211] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may... Figure 5 This is implemented on one or more instances of computing device 500—for example, each device may include similar components, features, and / or functions of computing device 500. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 600, examples of which are described herein. Figure 6 To describe in more detail.

[0212] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks, or networks within multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0213] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the server functionality described herein can be implemented on any number of client devices.

[0214] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting software at the software layer and / or one or more applications at the application layer. The software or applications may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software network application framework, such as one that can use a distributed file system for large-scale data processing (e.g., "big data").

[0215] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). If the connection to a user (e.g., a client device) is relatively close to an edge server, the core server can assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0216] Client devices may include those described in this article. Figure 5 The example computing device 500 described includes at least some components, features, and functions. By way of example and not limitation, the client device can be a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, aircraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming device or system, entertainment system, in-vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0217] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0218] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" could include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0219] This document describes in detail the subject matter of this disclosure to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have envisioned that the claimed subject matter may also be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. One or more processors, comprising: One or more circuits are used for: Determine the first task among multiple tasks to meet the criteria used for execution in redundant mode; A task flow for performing the plurality of tasks is determined, wherein a converter is assigned prior to the execution of the first task, the converter being used to divide the one or more circuits into a first partition and a second partition; as well as The plurality of tasks are executed according to the task flow by executing a first instance of the first task on the first partition and executing a second instance of the first task on the second partition.

2. The processor of claim 1 or more, wherein the converter is a first converter, and the one or more circuits are used for: Determining that the second task among the plurality of tasks depends on the first task and does not meet the criteria for execution in the redundancy mode; and A second converter is assigned to the task flow between the execution of the first task and the execution of the second task, the second converter being used to departition the one or more circuits.

3. The processor of claim 1, wherein the one or more circuits are configured to determine, at least based on characteristics of the first task received from at least one of an application including the first task or user input regarding the first task, that the first task meets the criterion.

4. The processor of claim 1, wherein the one or more circuits are configured to determine the task flow as a graph, the graph comprising a plurality of nodes, and a converter between a first node for performing a non-redundant task and each of the following: (i) a second node coupled to the first node, the second node being configured to perform a first instance of the first task; and (ii) a third node coupled to the first node, the third node being configured to perform a second instance of the first task.

5. The processor of claim 1, wherein the task flow instructs the one or more circuits to use hardware tools for partitioning the one or more circuits without exposing the hardware tools to an application associated with the plurality of tasks.

6. The processor of claim 1, wherein the converter is a first converter, and the one or more circuits are configured to assign a second converter to the task flow to switch the context from the first task to the execution of a second task to be redundantly executed.

7. The processor of claim 1, wherein the one or more circuits are configured to configure the first partition and the second partition as multiple contexts, graphics processing unit (GPU) partitions, or multiple instances of a multi-instance GPU (MIG) simultaneously.

8. The processor of claim 1 or more, wherein the processor is included in at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system containing one or more virtual machines (VMs); Systems implemented using robots; Systems implemented using edge devices; A system for generating synthetic data; A system that includes one or more large language models (LLMs); A system that includes one or more visual language models (VLMs); A system that includes one or more multimodal language models; A system that performs OS-level virtualization, including at least one of the following: one or more deep learning models, software for executing the one or more deep learning models, or telemetry software for evaluating, monitoring, or performing health checks on the system. A system that deploys one or more microservices; A system for deploying one or more inference microservices; A system for performing conversational AI operations; A system used to perform deep learning operations; A system used to perform simulation operations; A system for collaborative content creation of 3D assets; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

9. A system comprising: One or more processors are used to perform operations including the following: Determine the first task among multiple tasks to meet the criteria used for execution in redundant mode; A task flow for performing the plurality of tasks is determined, wherein a converter is assigned prior to the execution of the first task, the converter being used to divide the one or more circuits into a first partition and a second partition; as well as The plurality of tasks are executed according to the task flow by executing a first instance of the first task on the first partition and executing a second instance of the first task on the second partition.

10. The system of claim 9, wherein the converter is a first converter, and the one or more processors are configured to perform operations further comprising: Determining that the second task among the plurality of tasks depends on the first task and does not meet the criteria for execution in the redundancy mode; and A second converter is assigned to the task flow between the execution of the first task and the execution of the second task, the second converter being used to departition the one or more circuits.

11. The system of claim 9, wherein the one or more processors are configured to determine, based at least on characteristics of the first task received from at least one of an application including the first task or user input regarding the first task, that the first task satisfies the criterion.

12. The system of claim 9, wherein the one or more processors are configured to determine the task flow as a graph, the graph comprising a plurality of nodes, and a converter between a first node for performing a non-redundant task and each of the following: (i) a second node coupled to the first node, the second node being configured to perform a first instance of the first task; and (ii) a third node coupled to the first node, the third node being configured to perform a second instance of the first task.

13. The system of claim 9, wherein the task flow instructs the one or more processing units to use hardware tools for partitioning the one or more processors without exposing the hardware tools to an application associated with the plurality of tasks.

14. The system of claim 9, wherein the converter is a first converter, and the one or more processors are configured to assign a second converter to the task flow to switch the context from the first task to the execution of a second task to be redundantly executed.

15. The system of claim 9, wherein the one or more processors are configured to configure the first partition and the second partition as multiple contexts, graphics processing unit (GPU) partitions, or multiple instances of a multi-instance GPU (MIG) simultaneously.

16. The system of claim 9, wherein the system comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system containing one or more virtual machines (VMs); Systems implemented using robots; Systems implemented using edge devices; A system for generating synthetic data; A system that includes one or more large language models (LLMs); A system that includes one or more visual language models (VLMs); A system that includes one or more multimodal language models; A system for performing OS-level virtualization of an operating system includes at least one of the following: one or more deep learning models, software for executing the one or more deep learning models, or telemetry software for evaluating, monitoring, or performing health checks on the system. A system that deploys one or more microservices; A system for deploying one or more inference microservices; A system for performing conversational AI operations; A system used to perform deep learning operations; A system used to perform simulation operations; A system for collaborative content creation of 3D assets; Systems used to perform digital twin operations; A system for performing optical transmission simulation; A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

17. A method comprising: Determine the first task among multiple tasks to meet the criteria used for execution in redundant mode; A task flow for performing the plurality of tasks is determined, wherein a converter is assigned prior to the execution of the first task, the converter being used to divide the one or more circuits into a first partition and a second partition; as well as The plurality of tasks are executed according to the task flow by executing a first instance of the first task on the first partition and executing a second instance of the first task on the second partition.

18. The method of claim 17, wherein the converter is a first converter, the method further comprising: The second task among the plurality of tasks is determined to depend on the first task and does not meet the criteria for execution in the redundancy mode; as well as A second converter is assigned to the task flow between the execution of the first task and the execution of the second task, the second converter being used to departition the one or more circuits.

19. The method of claim 17, wherein the task flow is defined as a graph comprising a plurality of nodes, and a converter located between a first node for performing a non-redundant task and each of the following: (i) a second node coupled to the first node for performing a first instance of the first task; and (ii) a third node coupled to the first node for performing a second instance of the first task.

20. The method of claim 17, wherein the first partition and the second partition are configured as multiple contexts, graphics processing unit (GPU) partitions, or multiple instances of a multi-instance GPU (MIG) simultaneously.

Images