EFFICIENT EXECUTION OF DEPENDENT TASKS

Abstract

Circuits and techniques for the efficient execution of dependent tasks are disclosed. These techniques include retrieving a consumer task from memory that depends on a producer task executed by a processing device, and retrieving a first other task from memory. The techniques further include, in response to a first execution state of the producer task satisfying a first execution criterion at a first time, making the first other task available for execution by the processing device. The techniques further include, in response to a second execution state of the producer task satisfying a second execution criterion at a second time, making the consumer task available for execution by the processing device.

Description

TECHNICAL AREA

[0001] At least one embodiment relates to scheduling tasks for execution by a processor, and in particular to scheduling tasks for execution that depend on one or more other tasks. BACKGROUND

[0002] Processors (e.g., central processing units (CPUs), graphics processing units (GPUs), data processing units (DPUs), parallel processing units, accelerator engines, etc.) of a processing device can execute tasks from multiple processes concurrently. Tasks can have an assigned priority and can be executed based on that priority. Before execution, tasks can be loaded from memory and stored in a scheduling table, which may have limited space (e.g., due to hardware limitations). An in-memory task descriptor can be used to manage tasks (e.g., to be able to pause a running task and resume the next task selected based on priority) while they are in the scheduling table.The in-memory task descriptor can contain a task state associated with a task, such as its dependency tracking information, resource requirements, the number of threads (e.g., thread blocks) to be executed, and a current state (e.g., raster state) of the task. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram of an exemplary computer system for the efficient execution of dependent tasks according to at least one embodiment. Fig. Figure 2 is an exemplary diagram of the processing unit occupancy over time according to at least one embodiment. Fig. Figure 3 is a flowchart of an exemplary procedure for holding the execution of a consumer task for the efficient execution of dependent tasks according to at least one embodiment. Fig. Figure 4 is a flowchart of an exemplary procedure for lowering the priority of a consumer task for the efficient execution of dependent tasks according to at least one embodiment. Fig. Figure 5 is a flowchart of an exemplary procedure for efficient cleanup operations of producer tasks according to at least one embodiment. Fig. Figure 6 is a block diagram illustrating an exemplary computer system according to at least one embodiment of the present disclosure. Fig. 7A illustrates inference and / or training logic according to at least one embodiment of the present disclosure. Fig. 7B illustrates inference and / or training logic according to at least one embodiment. Fig. Figure 8 illustrates the training and use of a neural network according to at least one embodiment. Fig. Figure 9 is an exemplary data flow diagram for an advanced computing pipeline according to at least one embodiment. Fig. Figure 10 is a system diagram for an exemplary system for training, adapting, instantiating and deploying machine learning models in an advanced computing pipeline according to at least one embodiment. DETAILED DESCRIPTION

[0003] A processor scheduler can receive (e.g., load) multiple tasks from memory into a scheduling table. These tasks can then be dispatched from the scheduling table to one or more processors for execution. Before dispatching a task from the scheduling table to a processor, the processor can be queried to determine its resource availability. For example, a task might have five threads to be executed. After querying the processor, it can be determined that the processor has resources available to execute three threads. Therefore, three of the task's five threads can be dispatched to the processor for execution. Once these threads have completed, the processor can be queried again to determine its updated resource availability.If sufficient resources are available to execute the remaining 2 threads of the task, the 2 threads can be sent to the processor for execution.

[0004] In some cases, one task can depend on another. For example, a first task (such as a producer task) can generate data that another task (such as a consumer task) uses during execution. In some cases, a task can have more than one dependent task. In some cases, a task can depend on more than one producer task. If a consumer task is submitted to a processor (such as a central processing unit (CPU), graphics processing unit (GPU), data processing unit (DPU), parallel processing unit, accelerator engine, etc.) for execution before the data from one or more producer tasks is available, the processor may have to wait until the data is available, resulting in inefficient processor use and wasted resources.

[0005] If a consumer task is provided to a processing device for execution (e.g., by a processor or a processing unit of the processing device) after the data of one or more producer tasks are available, latency in querying the processor before providing the consumer task can reduce the processor's throughput.

[0006] In some cases, tasks are dispatched for execution by one or more processors based on an assigned task priority. For example, tasks with a higher priority may be dispatched for execution before tasks with a lower priority (e.g., made available to a processor for execution). In some cases, a first task with a higher priority than a second task will be dispatched for execution before the second task, even if the second task was loaded into the scheduling table before the first.

[0007] If a high-priority task is a consumer task and waits for its one or more producer tasks to finish before being sent out for execution, one or more lower-priority tasks may be delayed. For example, a first task may have a high priority and may be waiting for data from a producer task. A second task may have a lower priority than the first task and may be ready to execute (e.g., it may not depend on data from another task, it may depend on data from another task, and the data may already be available, etc.). However, the second task cannot be sent out for execution until the first task (e.g., the higher-priority task) has been sent out, resulting in reduced processor throughput.

[0008] In some cases, a group of tasks can share a common barrier dependency. For example, a first task and a second task can be producer tasks for the same third task (e.g., a consumer task that depends on data from the first task and on data from the second task) and can share a common barrier dependency. At the end of the first task's execution, memory used by the first task can be flushed (e.g., "memflush"). At the end of the second task's execution, the memory used by the second task can be flushed. Once the third task (e.g., the consumer task) begins execution, one or more caches used during the execution of the first task and one or more caches used during the execution of the second task can become invalid.Processor throughput can be reduced by performing memory flushes after each producer task and invalidating caches at the beginning of each consumer task, instead of performing memory flushes once for all producer tasks in the group when all producer tasks reach the common barrier, and performing cache invalidations once for all consumer tasks before the first consumer task starts executing.

[0009] Aspects of the present disclosure address the aforementioned and other shortcomings by providing systems and techniques that enable the efficient execution of dependent tasks. In particular, the techniques dispatch a consumer task for execution by a processor when the producer task(s) are nearly complete, thereby reducing the time the processor spends waiting for data from the producer task(s), while simultaneously reducing the overhead associated with querying the processor and dispatching the consumer task to the processor for execution.

[0010] In some embodiments, the techniques can lower the priority of a task in the scheduling table if it is waiting for data from a producer task(s). For example, a first task can be loaded into the scheduling table and may have a high priority (e.g., priority 2 within a first priority range of 0–63, where a lower number represents a higher priority). If the data on which the first task depends is not yet ready, the priority of the first task can be lowered (e.g., changed from 2 within the first priority range to 66 within a second range of lowered priorities 64–127). After the priority of the first task has been lowered, a second task with a lower priority than the original task of the first task (e.g.,A second task with a priority of 5 is made available for execution by a processor, thereby increasing the processor's throughput. Once the one or more producer tasks have finished preparing the data on which the first task depends, the priority of the first task can be restored (e.g., from 66 back to 2), and the first task can be made available for execution by a processor based on its original priority compared to the priorities of the other tasks in the scheduling table.

[0011] In some embodiments, the techniques can perform cleanup operations for a group of producer tasks once, when all tasks reach a barrier common to the tasks in the group, instead of performing cleanup operations for each task at the end of each producer task and at the beginning of each consumer task. For example, a first producer task and a second producer task may have a common barrier dependency. After the first task has finished executing, no cleanup operations can be performed. After the second task has finished executing, cleanup operations can be performed for both the first and second tasks. In some embodiments, the cleanup operations involve flushing memory used during task execution and / or invalidating caches used during task execution.By performing cleanup operations at the end of a group of producer tasks instead of at the end of each individual producer task and at the beginning of each consumer task, processor throughput can be increased because fewer cycles are spent performing cleanup operations.

[0012] Advantages of the disclosed embodiments over the existing technology include, but are not limited to, an increased task execution throughput of a processor.

[0013] Fig. Figure 1 is a block diagram of an exemplary computer system 100 for the efficient execution of dependent tasks according to at least one embodiment. The system 100 may include a central processing unit (CPU) 102 and a parallel processing device 106. In some embodiments, the CPU 102 and the parallel processing device 106 may be located in another system (e.g., the computer system 600 of Fig. 6) be included. For example, the system 100 may be included in a desktop computer, a server, a laptop, a mobile device, and / or the like. In some embodiments, the parallel processing device 106 may be used to perform machine learning and / or artificial intelligence (AI) tasks. For example, the parallel processing device 106 may be used to train AI models, perform inferences using trained AI models, and / or the like.

[0014] The CPU 102 can include a parallel processing device driver 104 for establishing an interface with the parallel processing device 106. The CPU 102 can send one or more task descriptors to the parallel processing device 106 for execution via the parallel processing device driver 104. In some embodiments, the parallel processing device driver 104 can check task descriptors as they are being received by the CPU 102.

[0015] The parallel processing device 106 can include a front end 108, a memory 110, a scheduler 114, a dependency-efficient scheduling table 116, and one or more processing units 118 (e.g., parallel processing units). The front end 108 can interface with the CPU 102 via the parallel processing device driver 104. The front end 108 can receive one or more task descriptors from the CPU 102, which are to be executed by the parallel processing device 106 (e.g., by the processing units 118). In some embodiments, the task descriptors are initially stored by the CPU 102 (or the parallel processing device driver 104) in the memory 110 of the parallel processing device 106.The frontend 108 can read the task descriptors from memory 110, make all necessary modifications to the task descriptor, and then store the modified task descriptor back in memory 110.

[0016] For example, memory 110 can contain task descriptors 112a and 112b. Task descriptor 112a can contain dependency information and task state information related to the task. The dependency information can include information about synchronization points (e.g., barriers) related to the task descriptor, one or more references to task descriptors on which this task descriptor depends, and / or one or more references to task descriptors that depend on this task descriptor. In some embodiments, the task state information can include a raster state of the task and a shader state of the task. The task state information can be used to track the execution progress of the task.For example, execution information for the task descriptor can include whether the task descriptor is currently running ("in-flight"), the number of task descriptor threads that have been executed, the number of threads that still need to be executed, whether cleanup operations have been performed on the task descriptor, and / or similar information. In some embodiments, a task descriptor is considered "in-flight" if at least one thread of the task descriptor is running. In some embodiments, the execution information can be used if the task needs to be interrupted or prematurely terminated during execution and later resumed without losing its state.

[0017] In some embodiments, the task descriptor 112a may include a task priority. For example, the task priority may be represented as a value within a range (e.g., 0-63, 0-127, etc.). In some embodiments, a low value means that the task has a high priority and can be made available for execution before tasks with a lower priority (e.g., higher priority value). In other embodiments, a low value means that the task has a low priority and can be made available for execution after tasks with a higher priority (e.g., higher priority value).

[0018] Similarly, task descriptor 112b can contain dependency information, task state information related to the task, and a task priority value. For example, task descriptor 112a can have a priority value of 2 and can contain a reference to task descriptor 112b, which may depend on data from task descriptor 112a (e.g., data generated by task descriptor 112a, data modified by task descriptor 112a, etc.). Task descriptor 112b can have a priority of 0 (e.g., the highest priority) and can contain a reference to the task descriptor 112a on which it depends. In other words, task descriptor 112a can be a producer task, and task descriptor 112b can be a consumer task that depends on task descriptor 112a.

[0019] If task descriptor 112a and task descriptor 112b have both been loaded into the dependency-efficient scheduling table 116, even though task descriptor 112b has a higher priority than task descriptor 112a, task descriptor 112a can be made available for execution (e.g., by processing units 118) before task descriptor 112b, because task descriptor 112b depends on data from task descriptor 112a.

[0020] The scheduler 114 can read task descriptors from memory 110 and load them into the dependency-efficient scheduling table 116. The dependency-efficient scheduling table 116 can contain one or more task descriptors (e.g., task descriptor 112c, task descriptor 112d, etc.), each of which can contain dependency information, task state information related to the task, and / or a task priority value, as described above. The dependency-efficient scheduling tables 116 can start task descriptors for execution by one or more processing units 118. During the execution of a task descriptor, the state of the task descriptor can be updated in the dependency-efficient scheduling table 116. For example, during execution, execution information of the task descriptor (e.g.,the task descriptor 112c), the raster state, the shader state and / or the like are updated.

[0021] To achieve high processing unit utilization (e.g., high utilization of processing units 118) and high processing unit throughput, the task descriptors for execution can be selected from the dependency-efficient scheduling table 116 based on one or more properties of the task descriptors, such as the dependency information of the task descriptors and their priority values.

[0022] As discussed above, before sending a task descriptor from the dependency-efficient scheduling table 116 to a processor (e.g., the processing units 118) for execution, the processor can be queried to determine the processor's resource availability ("state synchronization"). For example, a task might have 5 threads to be executed. After querying a processor, it can be determined that the processor has resources available to execute 3 threads. Thus, 3 of the task's 5 threads can be sent to the processor for execution. After these threads have completed, the processor can be queried again to determine a new resource availability. If sufficient resources are available to execute the remaining 2 threads of the task, these 2 threads can be sent to the processor for execution.

[0023] Performing state synchronization can require many computation cycles and can introduce latency, reducing the throughput of a processing device (e.g., parallel processing device 106). The latency with respect to state synchronization of a particular task descriptor can be "hidden" if state synchronization is performed while one or more other task descriptors are executing.

[0024] After selecting a task descriptor to be executed and performing state synchronization, one or more threads of the task descriptor can be made available to the processing units 118 for execution. The state of the task descriptor can be updated in the dependency-efficient scheduling table 116 during execution. For example, after performing state synchronization for the task descriptor, the task descriptor's state can be updated to indicate that state synchronization has been performed. After making one or more threads of the task descriptor available for execution, the task descriptor's state can be updated to indicate the number of threads made available for execution and / or to indicate the number of threads that still need to be made available for execution.The state of the task descriptor can be updated periodically based on the execution of the task descriptor's threads.

[0025] In some embodiments, if a task descriptor depends on data from another task descriptor and this data is not yet ready, state synchronization for a task descriptor can be performed without making threads of the task descriptor available to the processing units for execution. For example, the dependency-efficient scheduling table 116 can include task descriptors 112c and 112d. Task descriptor 112c can be a producer task, and task descriptor 112d can be a consumer task that depends on task descriptor 112c. State synchronization can be performed for task descriptor 112c, and one or more threads of task descriptor 112c can be made available to the processing units 118 for execution.While the threads of task descriptor 112c are executing, state synchronization for task descriptor 112d can be performed, thereby "hiding" the latency associated with the state synchronization of task descriptor 112d. If one or more threads of task descriptor 112c have not yet started executing when the state synchronization of task descriptor 112d ends, the dependency-efficient scheduling table 116 can wait to make any threads of task descriptor 112d available for execution, since the processing unit would have to wait idle until the data of task descriptor 112c is available, resulting in inefficient processing unit utilization. If the dependency-efficient scheduling table 116 has another, independent task descriptor (e.g.,If a task descriptor that does not depend on the data of task descriptor 112c) such as task descriptor 112a is included, state synchronization can be performed for the independent task descriptor and one or more threads of the independent task descriptor can be made available for execution while task descriptor 112d waits for all threads of task descriptor 112c to finish execution.

[0026] In some embodiments, producer task(s) can be loaded from memory (e.g., memory 110) into the scheduling table (e.g., the dependency-efficient scheduling table 116), and the corresponding consumer task(s) may not be in the scheduling table. Since the scheduling table may have limited space, it may be advantageous to wait to load the consumer task(s) from memory until the producer task(s) are executed. In some embodiments, the consumer task(s) can be loaded from memory into the scheduling table in response to the execution of the producer task(s) satisfying an execution criterion. For example, the consumer task(s) can be loaded from memory into the scheduling table in response to the last thread of the producer task(s) being made available for execution (e.g., when a first execution criterion is met).In some cases, the consumer task(s) can be loaded from memory into the scheduling table in response to a "pre-out" instruction being executed during the execution of a thread of the producer task(s).

[0027] In some embodiments, one or more threads of task descriptor 112d can be made available for execution before the data of task descriptor 112c is available, thus reducing the idle time of the processing unit executing the one or more threads. For example, one or more threads of task descriptor 112d can be made available for execution as soon as all threads of task descriptor 112c have finished execution (e.g., when a second execution criterion is met). While the one or more threads of task descriptor 112d are beginning execution, one or more cleanup operations can be performed on task descriptor 112c.The cleanup operations may include flushing memory used during the execution of task descriptor 112c ("memflush") and / or invalidating caches used during the execution of task descriptor 112c. In some cases, the task descriptor 112c data on which task descriptor 112d depends may not be available until after the memflush has been performed.

[0028] The processing units 118, which execute the one or more threads of task descriptor 112d, can perform operations that do not depend on the data of task descriptor 112c while task descriptor 112c is undergoing a memflush. For example, the processing units 118 can load (e.g., fetch) instructions and / or constants that will be used during the execution of task descriptor 112d. The processing units 118 may be idle as soon as they reach instructions that depend on the data of task descriptor 112c. After task descriptor 112c's memflush is complete, the data that task descriptor 112d is waiting for may be available, and the processing units 118 can resume execution of task descriptor 112d.

[0029] By starting the execution of the consumer task (e.g., task descriptor 112d) while the memflush of the producer task (e.g., task descriptor 112c) is being performed, at least part of the memflush latency can be “hidden”.

[0030] In some embodiments, the dependency-efficient scheduling table 116 selects task descriptors for execution based on an assigned priority value. For example, task descriptors with a higher priority may be made available for execution before task descriptors with a lower priority. In some cases, a first task descriptor (e.g., task descriptor 112d) with a higher priority than a second task descriptor (e.g., task descriptor 112c) is made available for execution before the second task descriptor, even if the second task descriptor was loaded into the dependency-efficient scheduling table 116 before the first.

[0031] If a high-priority task is a consumer task and waits for its one or more producer tasks to finish before being made available for execution, as described above, one or more lower-priority tasks may be delayed. To avoid the delay and the associated reduced processing device throughput, the dependency-efficient scheduling table 116 can lower the priority value of a consumer task while its one or more producer tasks are "in-flight" to allow other task descriptors to complete before the consumer task is ready for execution. For example, an initial task may be loaded into the dependency-efficient scheduling table 116 and may have a high priority (e.g., priority 2 within an initial priority range of 0–63, where a lower number represents a higher priority).If the data on which the first task depends is not yet ready, or if the producer task is still executing, the priority of the first task can be lowered (e.g., changed from 2 within the first priority range to 66 within a second range of lowered priorities 64-127). After the priority of the first task has been lowered, a second task with a lower priority than the original task (e.g., a second task with a priority of 5) can be made available for execution by a processor, thereby increasing processor throughput. Once the one or more producer tasks finish executing or complete preparing the data on which the first task depends, the priority of the first task can be restored (e.g.,from 66 back to 2), and the first task can be made available for execution by a processor based on its original priority compared to the priorities of the other tasks in the dependency-efficient scheduling table 116.

[0032] In some cases, a group of tasks may share a common barrier dependency. For example, a first task and a second task may be producer tasks for the same third task (e.g., a consumer task that depends on data from the first task and on data from the second task) and may share a common barrier dependency. To avoid the delays associated with performing cleanup operations at the end of each producer task, cleanup operations for a group of tasks can be performed once, when all tasks reach a barrier common to the group, instead of performing cleanup operations for each task at the end of each task.

[0033] For example, a first producer task and a second producer task might share a common barrier dependency. After the first task finishes execution, no cleanup operations can be performed. After the second task finishes execution, cleanup operations can be performed for both the first and second tasks. If the second task were to finish execution first, no cleanup operations could be performed for the second task until the first task finished execution. In some embodiments, the cleanup operations involve flushing memory used during task execution and / or invalidating caches used during task execution.By performing cleanup operations at the end of a group of producer tasks instead of at the end of each individual producer task and at the beginning of each consumer task, processor throughput can be increased because fewer cycles are spent performing cleanup operations.

[0034] Fig. Figure 2 is an exemplary diagram 200 of the processing unit occupancy over time of a processing unit of a parallel processing device configured for the efficient execution of dependent tasks, according to at least one embodiment. The diagram 200 represents a processing unit with an occupancy percentage of approximately 37%, represented by the average occupancy line 216, and includes elements representing the execution of multiple tasks: Task 0 202, Task 1 206, Task 2 210, and Task 3 214. In the Fig. In the two included examples, task 0 202 can be a producer task and task 1 206 can be a consumer task. Task 1 206 can be a producer task and task 2 210 can be a consumer task. Task 2 210 can be a producer task and task 3 214 can be a consumer task.

[0035] When the task's threads are made available to the processing unit for execution, the processing unit's utilization increases (as shown by the ramp-up to the left of each of tasks 0 202, task 1 206, task 2 210, and task 3 214). When individual threads of the task finish execution, the processing unit's utilization decreases (as shown by the decreasing steps to the right of each of tasks 0 202, task 1 206, task 2 210, and task 3 214).

[0036] Diagram 200 contains elements representing "memflush" operations as diagonally shaded rectangles: memflush 204, memflush 208, and memflush 212. As described above and illustrated in Diagram 200, threads of a consumer task can be made available for execution once the producer task has finished and before the memflush associated with the producer task has finished. For example, the left side of Task 1 206 overlaps memflush 204, the left side of Task 2 210 overlaps memflush 208, and the left side of Task 3 214 overlaps memflush 212. Once the threads are made available to the processing unit, initialization operations can be performed before the memflush finishes. For example, the processing unit can load (e.g., fetch) instructions and / or constants that will be used during task execution.Once the initialization operations are complete, the processing unit can remain idle until the memory flush is finished. Once the memory flush is complete, task execution can resume.

[0037] Once task 1 206 has finished executing, its corresponding memflush 208 can begin, and the threads of task 2 210 can be made available for execution. Once task 2 210 has finished executing, its corresponding memflush 212 can begin, and the threads of task 3 214 can be made available for execution.

[0038] Although not explicitly shown in Diagram 200, it is understood that the threads of a consumer task corresponding to a producer task can be made available to the processing unit for execution immediately after the producer task has finished executing, because the consumer task descriptor has already been loaded into the scheduling table and state synchronization has already occurred, as discussed above. The consumer task descriptor may have been held in the scheduling table until its corresponding producer task has finished executing. In some embodiments, the consumer task descriptor was loaded into the scheduling table when the last thread of the producer task began executing.In some embodiments, the priority of the consumer task descriptor was lowered while the producer task was executing and restored when the producer task finished executing.

[0039] Fig. Figure 2 also includes an example diagram 220 of the processing unit utilization over time of a processing unit of a parallel processing device not configured for the efficient execution of dependent tasks. Diagram 220 represents a processing unit with a utilization percentage of approximately 26%, represented by the average utilization line 236, and includes elements representing the execution of multiple tasks: Task 0 222, Task 1 226, Task 2 230, and Task 3 234. In the example shown in diagram 220, Task 0 222 may be a producer task and may have Task 1 226 as a consumer task. Task 1 226 may be a producer task and may have Task 2 230 as a consumer task. Task 2 230 may be a producer task and may have Task 3 234 as a consumer task.

[0040] When the task's threads are made available to the processing unit for execution, the processing unit's utilization increases (as shown by the ramp-up to the left of each of tasks 0 222, 1 226, 230, and 3 234). When individual threads of the task finish execution, the processing unit's utilization decreases (as shown by the decreasing steps to the right of each of tasks 0 222, 1 226, 230, and 3 234).

[0041] Diagram 220 contains elements that represent "memflush" operations as rectangles with diagonal shading: memflush 224, memflush 228, and memflush 232. Unlike the tasks in Diagram 200, the tasks in Diagram 220 cannot benefit from the techniques revealed here. As shown in Diagram 220, the corresponding memflush 224 begins as soon as task 0 222 has finished executing. At the end of memflush 224, the processing unit allocation is 0 for a period of time, represented by Gap 238. During Gap 238, the task descriptor of task 1 226 (e.g., the consumer task corresponding to producer task 0 222) can be loaded from memory into a scheduling table and / or state synchronization can be performed for the task descriptor.Once the task descriptor is loaded into the scheduling table and state synchronization has been performed, threads of task 1 226 can be made available to the processing unit for execution, and execution can begin. This pattern repeats for the remaining tasks of diagram 220.

[0042] Once task 1 226 has finished executing, its corresponding memflush 228 begins. At the end of memflush 228, the processing unit's allocation during the gap 240 is 0. The task descriptor for task 2 230 can be loaded from memory, state synchronization can be performed, then the threads of task 2 230 can be made available for execution, and execution can begin.

[0043] Once task 2 230 has finished executing, its corresponding memflush 232 begins. At the end of memflush 232, the utilization of the processing units during 242 is 0. The task descriptor for task 3 234 can be loaded from memory, state synchronization can be performed, then the threads of task 3 234 can be made available for execution, and execution can begin.

[0044] Due to the gaps in diagram 220 and because consumer task threads are not provided when the producer task finishes execution, the average occupancy percentage of diagram 220 is reduced compared to that of diagram 200.

[0045] Fig. Figure 3 is a flowchart of an exemplary method 300 for holding the execution of a consumer task for the efficient execution of dependent tasks according to at least one embodiment. Fig. Figure 4 is a flowchart of an exemplary method 400 for lowering the priority of a consumer task for the efficient execution of dependent tasks according to at least one embodiment. Fig. Figure 5 is a flowchart of an exemplary procedure 500 for efficient cleanup operations of producer tasks according to at least one embodiment.

[0046] Methods 300, 400, and / or 500 can be performed using one or more processing units (e.g., CPUs, GPUs, accelerators, physical processing units (PPUs), data processing units (DPUs), parallel processing units, etc.) that include (or can communicate with) one or more storage devices. In at least one embodiment, methods 300, 400, and / or 500 can be performed using one or more processing devices. In at least one embodiment, methods 300, 400, and / or 500 can be performed using a processing circuit arrangement. In at least one embodiment, methods 300, 400, and / or 500 can be performed using a circuit arrangement of the parallel processing device 106. Fig. 1. In at least one embodiment, methods 300, 400 and / or 500 can be carried out using a dependency-efficient scheduling table 116 of the parallel processing device 106. Fig. 1 will be carried out.

[0047] In at least one embodiment, processing units performing one of methods 300, 400, and / or 500 can execute instructions stored on a non-volatile, computer-readable storage medium. In at least one embodiment, one of methods 300, 400, and / or 500 can be performed using multiple processing threads (e.g., CPU threads and / or GPU threads), with individual threads performing one or more individual functions, routines, subroutines, or operations of the method. In at least one embodiment, processing threads implementing one of methods 300, 400, and / or 500 can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, processing threads implementing one of methods 300, 400, and / or 500 can be executed asynchronously.Various operations of procedures 300, 400 and / or 500 can be compared to those in . Fig. 3, Fig. 4 and Fig. The operations shown in Figure 5 can be performed in a different order. Some operations of one of methods 300, 400, and / or 500 can be performed simultaneously with other operations. In at least one embodiment, one or more operations that are described in Figure 5 can be performed simultaneously with other operations. Fig. 3, Fig. 4 and / or Fig. The steps shown in point 5 are not always carried out.

[0048] With reference to Fig. 3. Processing units executing method 300 can retrieve a consumer task from memory at block 302, which depends on a producer task executed by a processing device (e.g., a processor or a processing unit of a processing device). In some embodiments, the consumer task further depends on another producer task. In some embodiments, the producer task generates data to be consumed by more than one consumer task.

[0049] At block 304, processing units can retrieve a first other task from memory. At block 306, processing units can determine an initial execution state of the generating task. For example, processing units can query a processor for the current execution state of the generating task. In some cases, processing units can receive a report (e.g., one or more signals or values) including an execution state of the generating task from the processor.

[0050] If the initial execution state indicates that not all threads of the producer task have started execution, processing units at decision block 308 can return to block 306 and determine the initial execution state of the producer task again.

[0051] When the initial execution state indicates that all threads of the producer task have begun execution, processing units at block 310 can make the first other task available for execution by the processing device. As discussed above, if one or more threads of the producer task have yet to begin execution, the consumer task can be held in the scheduling table and cannot be made available to a processor for execution. Another task that is ready for execution (for example, a task with no dependencies or a task whose producer task(s) has already finished) can be made available for execution before the consumer task.

[0052] At block 312, processing units can determine a second execution state of the producer task. If the second execution state does not meet a second execution criterion, processing units can return to block 312 at decision block 314 and determine the second execution state of the producer task again.

[0053] If the second execution state satisfies the second execution criterion, processing units at block 316 can make the consumer task available for execution by the processing device. In some embodiments, the second execution criterion is based on the threads of the producer task completing their execution. For example, if all threads of the producer task have completed their execution, the second execution criterion may be satisfied.

[0054] In some embodiments, the provision of the consumer task for execution by the processing device further responds to the availability of the processing device's resources, which must meet a resource criterion. For example, the processing device's state can be queried to determine its available resources. The consumer task may include one or more resource requests. If the processing device's available resources meet the one or more resource requests of the consumer task, the consumer task (or one or more threads of the consumer task) can be provided to the processing device for execution.

[0055] In some embodiments, processing units can retrieve the consumer task from memory, which depends on the producer task executed by the processing device in response to the first execution state of the producer task satisfying the first execution criterion. In some embodiments, the first execution criterion is based on the last thread of the producer task being made available for execution. For example, if the last thread of the producer task has started execution, the first execution criterion may be satisfied. In some embodiments, the first execution criterion is based on the execution of a "pre-output" instruction during the execution of the producer task.

[0056] With reference to Fig. In some embodiments, procedure 400 is performed after (or in conjunction with) procedure 300. At block 402, processing units performing procedure 400 can determine a first execution state of the producer task. If the first execution state indicates that not all threads of the producer task have started execution, processing units at decision block 404 can return to block 402 and determine the first execution state of the producer task again. In some embodiments, block 402 and decision block 404 correspond to block 306 and decision block 308 of Fig. 3.

[0057] If the initial execution state indicates that all threads of the producer task have begun execution, processing units can lower the priority of the consumer task at block 406. At block 408, processing units can retrieve a second, different task with a corresponding priority from memory. At block 410, processing units can determine, based on the priorities of at least the consumer task and the second, different task, a next task for execution by the processing device (e.g., the priority of the consumer task and the priority of the second, different task). At block 412, processing units can make the next task available for execution by the processing device.

[0058] In some embodiments, processing units at block 414 can determine a third execution state of the producer task. If the third execution state does not satisfy a third execution criterion, processing units at decision block 416 can return to block 414 and determine the third execution state of the producer task again. In some embodiments, the third execution state indicates whether the producer task's memory has been flushed (e.g., whether a "memflush" was performed for the producer task). For example, if the third execution state indicates that a memflush was performed for the producer task, the third execution criterion may be satisfied.

[0059] If the third execution state satisfies the third execution criterion, processing units at block 418 can restore the consumer task priority to an original priority.

[0060] With reference to Fig. In block 502, processing units can provide a first producer task for execution by a processing device. In block 504, processing units can provide a second producer task for execution by a processing device. In some embodiments, the first producer task and the second producer task share a common barrier dependency.

[0061] At block 506, processing units can determine the execution state of the first producer task. If the execution state of the first producer task indicates that the first producer task has not yet completed its execution, processing units can return to block 506 and determine the execution state of the first producer task again.

[0062] At block 510, processing units can determine the execution state of the second producer task. If the execution state of the second producer task indicates that the second producer task has not yet completed its execution, processing units can return to block 510 and determine the execution state of the second producer task again.

[0063] Once both the first producer task and the second producer task have completed execution, processing units at block 514 can cause one or more cleanup operations to be performed for both the first and second producer tasks. At block 516, processing units can provide a first consumer task for execution by the processing device. In some embodiments, the consumer task depends on the first and second producer tasks.

[0064] In some embodiments, the one or more cleanup operations include at least one of emptying a first memory relating to the first producer task and a second memory relating to the second producer task, or invalidating first caches relating to the first producer task and second caches relating to the second producer task.

[0065] In some embodiments, causing one or more cleanup operations to be performed involves providing a completion trace task for execution by the processing device. The completion trace task may depend on the first producer task and the second producer task. The consumer task may depend on the completion trace task. In some embodiments, the completion trace task acts as a link between the producer tasks (e.g., the first producer task and the second producer task) and the consumer task. When the completion trace task is executed, it may terminate execution immediately, and a consumer task may be provided for execution.

[0066] In some embodiments, a closing trace task descriptor is added to the scheduling table when a producer task is loaded from memory and the corresponding consumer task is not yet in the scheduling table. For example, an initial producer task can be loaded from memory into the scheduling table. The initial producer task can generate data that is consumed by an initial consumer task. If the initial consumer task is not yet in the scheduling table, a closing trace task descriptor can be added to the scheduling table, and the dependency information of the producer task descriptor can be updated to point to the closing trace task descriptor instead of the initial consumer task descriptor.Once the first consumer task descriptor is loaded from memory into the scheduling table, the dependency information of the closing trace task descriptor can be updated to point to the first consumer task descriptor. The dependency information of the first consumer task descriptor can also be updated to point to the closing trace task descriptor instead of the first producer task descriptor.

[0067] Once the first producer task has finished executing, the closing trace task can be ready to execute because the task it depends on has finished. The closing trace task can then be made ready to execute and can finish executing immediately. At this point, the first consumer task can be ready to execute because the task it depends on (the closing trace task) has finished executing.

[0068] In some embodiments, processing units can, in response to the completion of one or more cleanup operations, provide a second consumer task for execution by the processing device. The second consumer task may depend on the first producer task and the second producer task.

[0069] In some embodiments, the consumer task generates data that is consumed by another task. In other words, the consumer task can be a producer task for another consumer task.

[0070] Fig. Figure 6 is a block diagram illustrating an exemplary computer system according to at least one embodiment of the present disclosure. In some embodiments, computer system 600 can replace system 100. Fig. 1. The Computer System 600 can operate in the capacity of a server or endpoint machine in an endpoint-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a television, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web application, a server, a network router, a switch or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) that specify actions to be performed by that machine. Furthermore, although only a single machine is illustrated, the term "machine" is also to be understood as including any collection of machines that, individually or collectively, execute a set (or multiple sets) of instructions to perform one or more of the methodologies discussed herein.

[0071] The exemplary computer system 600 includes a processing device (processor) 602, a main memory 604 (e.g., read-only memory (ROM), flash memory, dynamic random-access memory (DRAM), such as synchronous DRAM (SDRAM), dual data rate (DDR-SDRAM), or DRAM (RDRAM), etc.), a static memory 606 (e.g., flash memory, static random-access memory (SRAM), etc.), and a data storage device 616, which communicate with each other via a bus 628.

[0072] The processor (processing device) 602 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like, and may include processing logic 622. In particular, the processor 602 may be a complex instruction set (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. The processor 602 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processor 602 is configured to execute instructions 626 (e.g.,(to generate threat indicator alerts) to perform the operations discussed here.

[0073] The computer system 600 may further include a network interface device 608. The computer system 600 may also include a video display unit 610 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an input device 612 (e.g., a keyboard and an alphanumeric keypad, a motion-sensing input device, a touchscreen), a cursor control device 614 (e.g., a mouse), and a signal generation device 618 (e.g., a loudspeaker). In some embodiments, the computer system 600 may not include a video display unit 610, an input device 612, and / or a cursor control device 614 (e.g., in a headless configuration).

[0074] The data storage device 616 may include a non-volatile, machine-readable storage medium 624 (also called a computer-readable storage medium) on which one or more sets of instructions 626 (e.g., for the efficient execution of dependent tasks) are stored, embodying one or more of the methodologies or functions described herein. The instructions 626 may also reside wholly or at least partially within the main memory 604 and / or within the processor 602 during its execution by the computer system 600, the main memory 604 and the processor 602 also being machine-readable storage media. Furthermore, the instructions may be transmitted or received over a network 620 via the network interface device 608.

[0075] In one implementation, the instructions 626 include instructions for the efficient execution of dependent tasks. Although the computer-readable storage medium 624 (machine-readable storage medium) is shown in an exemplary implementation as a single medium, the terms "computer-readable storage medium" and "machine-readable storage medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store the one or more sets of instructions.The terms “computer-readable storage medium” and “machine-readable storage medium” shall also be understood to include any medium capable of storing, encoding, or carrying a set of instructions for execution by the machine, and which causes the machine to carry out one or more of the methodologies of the present disclosure. Accordingly, the terms “computer-readable storage medium” and “machine-readable storage medium” shall be understood to include, but are not limited to, solid-state storage media, optical media, and magnetic media. INFERENCE AND TRAINING LOGIC

[0076] Fig. 7A illustrates inference and / or training logic 715, which is used to perform inference and / or training operations associated with one or more embodiments.

[0077] In at least one embodiment, the inference and / or training logic 715 can, without limitation, include code and / or data storage 701 for storing forward and / or output weights and / or input / output data and / or other parameters for configuring neurons or layers of a neural network that are trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the training logic 715 can include graph code or other software for controlling timing and / or sequencing (or be coupled to code and / or data storage 701 that stores this), wherein weight and / or other parameter information is to be loaded to configure processing units, including logic units, integer and / or floating-point units (collectively, arithmetic logic units (ALUs) or simply circuits).In at least one embodiment, code, such as graph code, loads weight or other parameter information into processor ALUs based on a neural network architecture to which that code corresponds. In at least one embodiment, the code and / or data store 701 stores weight parameters and / or input / output data of each layer of a neural network, which are trained or used in conjunction with one or more embodiments during the forward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments. In at least one embodiment, any portion of the code and / or data store 701 can be contained in another on-chip or off-chip data store, including the L1, L2, or L3 cache or system memory of a processor.

[0078] In at least one embodiment, any section of the code and / or data memory 701 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 701 can be cache memory, dynamic directly addressable memory (“DRAM”), static directly addressable memory (“SRAM”), non-volatile memory (e.g., flash memory), or other memory.In at least one embodiment, the choice of whether the code and / or data memory 701 is, for example, internal or external to a processor, or comprises DRAM, SRAM, Flash, or another type of memory, may depend on available on-chip versus off-chip memory, latency requirements of training and / or inference functions being performed, batch size of data used in inference and / or training of a neural network, or a combination of these factors.

[0079] In at least one embodiment, the inference and / or training logic 715 may, without limitation, include a code and / or data store 705 for storing backward and / or output weights and / or input / output data corresponding to neurons or layers of a neural network that are trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the code and / or data store 705 stores weight parameters and / or input / output data of each layer of a neural network that are trained or used in conjunction with one or more embodiments during the backward propagation of input / output data and / or weight parameters during training and / or inference using aspects of one or more embodiments.In at least one embodiment, the training logic 715 may include diagram code or other software for controlling the timing and / or sequence (or be coupled with code and / or data storage 705 which stores this), wherein weight and / or other parameter information is to be loaded to configure processing units, including logic units, integer and / or floating-point units (collectively arithmetic logic units (ALUs)).

[0080] In at least one embodiment, code, such as graph code, causes weight or other parameter information to be loaded into processor ALUs based on a neural network architecture to which that code corresponds. In at least one embodiment, any portion of the code and / or data memory 705 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor. In at least one embodiment, any portion of the code and / or data memory 705 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 705 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory.In at least one embodiment, the choice of whether the code and / or data memory 705 is, for example, internal or external to a processor, or comprises DRAM, SRAM, Flash, or another type of memory, may depend on available on-chip versus off-chip memory, latency requirements of training and / or inference functions being performed, batch size of data used in inference and / or training of a neural network, or a combination of these factors.

[0081] In at least one embodiment, the code and / or code and / or data memory 701 and the code and / or data memory 705 can be separate memory structures. In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be a combined memory structure. In at least one embodiment, the code and / or data memory 701 and the code and / or data memory 705 can be partially combined and partially separate. In at least one embodiment, any portion of the code and / or data memory 701 and the code and / or data memory 705 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.

[0082] In at least one embodiment, the inference and / or training logic 715 may, without limitation, include one or more arithmetic logic unit(s) (“ALU(s)”) 710, including integer and / or floating-point units, to perform logical and / or mathematical operations that are at least partially based on or indicated by training and / or inference code (e.g., graph code), wherein a result thereof may generate activations (e.g., output values ​​of layers or neurons within a neural network) that are stored in an activation memory 720, which are functions of input / output and / or weight parameter data that are stored in code and / or data memory 701 and / or code and / or data memory 705.In at least one embodiment, activations stored in activation memory 720 are generated according to linear algebraic and / or matrix-based mathematics performed by ALU(s) 710 in response to the execution of instructions or other code, wherein weight values ​​stored in code and / or data memory 705 and / or code and / or data memory 701 are used as operands together with other values, such as deviation values, gradient information, pulse values, or other parameters or hyperparameters, some or all of which may be stored in code and / or data memory 705 or code and / or code and / or data memory 701 or another on-chip or off-chip memory.

[0083] In at least one embodiment, ALU(s) 710 are contained within one or more processors or other hardware logic devices or circuits, whereas in another embodiment, ALU(s) 710 may be external to a processor or other hardware logic device or circuit that uses them (e.g., a coprocessor). In at least one embodiment, ALU(s) 710 may be contained within the execution units of a processor or otherwise in a bank of ALUs that the execution units of a processor can access either within the same processor or distributed across different processors of different types (e.g., central processing units, graphics processing units, fixed-function units, etc.).In at least one embodiment, code and / or data memory 701, code and / or data memory 705, and activation memory 720 can share a processor or other hardware logic device or circuit, whereas in another embodiment, they can be located in different processors or other hardware logic devices or circuits, or a combination of the same and different processors or other hardware logic devices or circuits. In at least one embodiment, any portion of the activation memory 720 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.Furthermore, inference and / or training code may be stored with other code that a processor or other hardware logic or circuitry can access and that is retrieved and / or processed using the fetch, decode, schedule, execute, retract and / or other logical circuitry of a processor.

[0084] In at least one embodiment, the activation memory 720 can be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the activation memory 720 can be located wholly or partially inside or external to one or more processors or other logic circuits. In at least one embodiment, the choice of whether the activation memory 720 is, for example, internal or external to a processor, or whether it comprises DRAM, SRAM, flash memory, or another type of memory, can depend on available on-chip versus off-chip memory, the latency requirements of training and / or inference functions being performed, the batch size of data used in the inference and / or training of a neural network, or a combination of these factors.

[0085] In at least one embodiment, the inference and / or training logic 715, which is described in Fig. Figure 7A illustrates that the inference and / or training logic 715, which is illustrated in Figure 7A, can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as a Google TensorFlow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., “Lake Crest”). In at least one embodiment, ... Fig. 7A illustrates how they can be used in conjunction with hardware of a central processing unit (“CPU”), hardware of a graphics processing unit (“GPU”) or other hardware, such as field-programmable gate arrays (“FPGAs”).

[0086] Fig. Figure 7B illustrates inference and / or training logic 715 according to at least one embodiment. In at least one embodiment, the inference and / or training logic 715 can, without limitation, include hardware logic in which computing resources are dedicated or otherwise used exclusively in connection with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the inference and / or training logic 715, which is described in Fig. Figure 7B illustrates this, in conjunction with an application-specific integrated circuit (ASIC), such as a Google TensorFlow® processing unit, a Graphcore™ inference processing unit (IPU), or an Intel Corp. Nervana® processor (e.g., “Lake Crest”). In at least one embodiment, the inference and / or training logic 715, which is illustrated in Figure 7B, can be used ... Fig. Figure 7B illustrates the use of the inference and / or training logic 715 in conjunction with hardware of a central processing unit (CPU), hardware of a graphics processing unit (GPU), or other hardware, such as field-programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 715 includes, without limitation, code and / or data memory 701 and code and / or data memory 705, which can be used to store code (e.g., graph code), weight values, and / or other information, including deviation values, gradient information, pulse values, and / or other parameter or hyperparameter information. In at least one embodiment, which is illustrated in Fig. As illustrated in Figure 7B, each of the code and / or data memory 701 and code and / or data memory 705 is assigned to a dedicated computing resource, such as computing hardware 702 and computing hardware 706, respectively. In at least one embodiment, each of the computing hardware 702 and computing hardware 706 comprises one or more ALUs that perform mathematical functions, such as linear algebraic functions, only on information stored in code and / or data memory 701 and code and / or data memory 705, respectively, with the result being stored in activation memory 720.

[0087] In at least one embodiment, each of the code and / or data storage units 701 and 705 and corresponding computing hardware 702 and 706, respectively, corresponds to different layers of a neural network, such that the resulting activation from one memory / computing pair 701 / 702 of code and / or data storage unit 701 and computing hardware 702 is provided as an input to the next memory / computing pair 705 / 706 of code and / or data storage unit 705 and computing hardware 706, in order to reflect a conceptual organization of a neural network. In at least one embodiment, each of the memory / computing pairs 701 / 702 and 705 / 706 can correspond to more than one layer of a neural network. In at least one embodiment, additional memory / computing pairs (not shown) may be included after or in parallel to memory / computing pairs 701 / 702 and 705 / 706 in inference and / or training logic 715. TRAINING AND USE OF A NEURAL NETWORK

[0088] Fig. Figure 8 illustrates the training and deployment of a deep neural network according to at least one embodiment. In at least one embodiment, the untrained neural network 806 is trained using a training dataset 802. In at least one embodiment, the training framework 804 is a PyTorch framework, whereas in other embodiments, the training framework 804 is a TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or another training framework. In at least one embodiment, the training framework 804 trains an untrained neural network 806 and enables it to be trained using processing resources described herein to generate a trained neural network 808. In at least one embodiment, weights can be selected randomly or by pretraining using a deep belief network.In at least one embodiment, the training can be carried out in a supervised, partially supervised or unsupervised manner.

[0089] In at least one embodiment, the untrained neural network 806 is trained using supervised learning, wherein the training dataset 802 contains an input paired with a desired output for an input, or wherein the training dataset 802 contains an input paired with a known output, and an output of the neural network 806 is manually graded. In at least one embodiment, the untrained neural network 806 is trained in a supervised manner and processes inputs from the training dataset 802 and compares resulting outputs with a set of expected or desired outputs. In at least one embodiment, errors are then propagated back by the untrained neural network 806. In at least one embodiment, the training framework 804 adjusts weights that control the untrained neural network 806.In at least one embodiment, the training framework 804 includes tools to monitor how well the untrained neural network 806 converges to a model, such as the trained neural network 808, that is capable of generating correct answers, as in the result 814, based on input data, such as a new dataset 812. In at least one embodiment, the training framework 804 repeatedly trains the untrained neural network 806 while adjusting weights to refine an output of the untrained neural network 806 using a loss function and an adjustment algorithm, such as stochastic gradient descent. In at least one embodiment, the training framework 804 trains the untrained neural network 806 until the untrained neural network 806 achieves a desired accuracy.In at least one embodiment, the trained neural network 808 can then be used to implement any number of machine learning operations.

[0090] In at least one embodiment, the untrained neural network 806 is trained using unsupervised learning, wherein the untrained neural network 806 attempts to train itself using unlabeled data. In at least one embodiment, the training dataset 802 for unsupervised learning contains input data without associated output data or "ground truth" data. In at least one embodiment, the untrained neural network 806 can learn groupings within the training dataset 802 and can determine how individual inputs relate to the untrained dataset 802. In at least one embodiment, unsupervised training can be used to generate a self-organizing map in the trained neural network 808 that is capable of performing operations useful in reducing the dimensionality of the new dataset 812.In at least one embodiment, unsupervised training can also be used to perform anomaly detection, enabling the identification of data points in the new data set 812 that deviate from normal patterns of the new data set 812.

[0091] In at least one embodiment, semi-supervised learning can be used, which is a technique in which the training dataset 802 contains a mixture of labeled and unlabeled data. In at least one embodiment, the training framework 804 can be used to perform incremental learning, such as through transferred learning techniques. In at least one embodiment, incremental learning allows the trained neural network 808 to adapt to the new dataset 812 without forgetting knowledge that was embedded in the trained neural network 808 during the initial training.

[0092] With reference to Fig. 9 is Fig. 9 An exemplary data flow diagram for a process 900 for generating and deploying a processing and inference pipeline according to at least one embodiment. In at least one embodiment, the process 900 can be used to perform game name recognition analysis and inference on user feedback data at one or more facilities 902, such as a data center.

[0093] In at least one embodiment, the process 900 can be executed within a training system 904 and / or a deployment system 906. In at least one embodiment, the training system 904 can be used to perform training, deployment, and an embodiment of machine learning models (e.g., neural networks, object detection algorithms, computer vision algorithms, etc.) for use in the deployment system 906. In at least one embodiment, the deployment system 906 can be configured to offload processing and computing resources to a distributed computing environment to reduce infrastructure requirements at the facility 902. In at least one embodiment, the deployment system 906 can provide a streamlined platform for selecting, customizing, and implementing virtual tools for use with computing devices at the facility 902.In at least one embodiment, virtual instruments can include software-defined applications for performing one or more processing operations on feedback data. In at least one embodiment, one or more applications in a pipeline can use or call services (e.g., inference, visualization, computation, AI, etc.) of the Deployment System 906 during application execution.

[0094] In at least one embodiment, some applications used in advanced processing and inference pipelines can use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, machine learning models can be trained at the facility 902 using feedback data 908 (such as imaging data) stored at the facility 902, or feedback data 908 from another facility or facilities, or a combination thereof. In at least one embodiment, the training system 904 can be used to provide applications, services, and / or other resources for generating functional, deployable machine learning models for the deployment system 906.

[0095] In at least one embodiment, a model register 924 can be supported by object storage, which can support versioning and object metadata. In at least one embodiment, object storage can be provided, for example, via cloud storage (e.g., a cloud 1026). Fig. 10), which is compatible with an application programming interface (API), can be accessed from within a cloud platform. In at least one embodiment, machine learning models within the model register 924 can be uploaded, listed, modified, or deleted by developers or partners of a system interacting with an API. In at least one embodiment, an API can provide access to procedures that allow users with appropriate credentials to associate models with applications, so that models can be executed as part of the execution of containerized instantiations of applications.

[0096] In at least one embodiment, one or more training pipelines 1004 ( Fig. 10) include a scenario in which the facility 902 trains its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, feedback data 908 can be received from various channels, such as forums, web forms, or the like. In at least one embodiment, once feedback data 908 is received, AI-assisted annotation 910 can be used to assist in generating annotations that correspond to feedback data 908 to be used as ground-truth data for a machine learning model. In at least one embodiment, AI-assisted annotation 910 can include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that can be trained to generate annotations that correspond to certain types of feedback data 908 (e.g.,of certain devices) and / or certain types of anomalies in the feedback data 908 correspond. In at least one embodiment, AI-assisted annotations 910 can then be used directly or can be adapted or fine-tuned using an annotation tool to generate ground-truth data. In at least one embodiment, tagged data 912 can be used as ground-truth data for training a machine learning model in some examples. In at least one embodiment, AI-assisted annotations 910, tagged data 912, or a combination thereof can be used as ground-truth data for training a machine learning model, e.g., via model training 914 in . Fig. 9 and / or Fig. 10. In at least one embodiment, a trained machine learning model can be designated as an output model 916 and can be used by the deployment system 906 as described herein.

[0097] In at least one embodiment, one or more training pipelines 1004 ( Fig. 10) include a scenario in which the facility 902 requires a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 906, but the facility 902 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model can be selected from the model register 924. In at least one embodiment, the model register 924 can contain machine learning models that are trained to perform a variety of different inference tasks on imaging data. In at least one embodiment, machine learning models in the model register 924 can have been trained on imaging data from facilities other than the facility 902 (e.g., facilities that are remotely located).In at least one embodiment, machine learning models can be trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data, which can be a form of feedback data 908, from a specific location, training can take place at that location or at least in a manner that protects the confidentiality of imaging data or limits the transfer of imaging data outside of premises (e.g., to comply with HIPAA regulations, privacy regulations, etc.). In at least one embodiment, once a model is trained—or partially trained—at a location, a machine learning model can be added to the model register 924.In at least one embodiment, a machine learning model can then be retrained or updated on any number of other facilities, and a retrained or updated model can be made available in the model register 924. In at least one embodiment, a machine learning model can then be selected from the model register 924—and designated as output model(s) 916—and can be used in the deployment system 906 to perform one or more processing tasks for one or more applications of a deployment system.

[0098] In at least one embodiment, one or more training pipelines 1004 ( Fig. 10) in a scenario involving the facility 902, which requires a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 906, but the facility 902 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from the model register 924 may not be fine-tuned or optimized for feedback data 908 generated at the facility 902 due to differences in the population, genetic variations, robustness of training data used to train a machine learning model, diversity in anomalies of training data, and / or other problems with training data.In at least one embodiment, AI-assisted annotation 910 can be used to assist in generating annotations corresponding to feedback data 908, which are to be used as ground-truth data for retraining or updating a machine learning model. In at least one embodiment, labeled data 912 can be used as ground-truth data for training a machine learning model. In at least one embodiment, retraining or updating a machine learning model can be referred to as model training 914. In at least one embodiment, the model training 914 can include data—for example, AI-assisted annotations 910, labeled data 912, or a combination thereof—that can be used as ground-truth data for retraining or updating a machine learning model.

[0099] In at least one embodiment, the deployment system 906 can include software 918, service 920, hardware 922, and / or other components, features, and functionality. In at least one embodiment, the deployment system 906 can include a software "stack" such that software 918 can be built on top of service 920 and use service 920 to perform some or all of the processing tasks, and service 920 and software 918 can be built on top of hardware 922 and use hardware 922 to perform the processing, storage, and / or other computing tasks of the deployment system 906.

[0100] In at least one embodiment, Software 918 can include any number of different containers, each container capable of executing an instantiation of an application. In at least one embodiment, each application can perform one or more processing tasks in an advanced processing and inference pipeline (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.). In at least one embodiment, there can be any number of containers for each type of computing device that can perform a data processing task with respect to feedback data 908 (or other data types, such as those described herein).In at least one embodiment, an advanced processing and inference pipeline can be defined based on selections of various containers desired or required for processing feedback data 908, in addition to containers that receive and configure imaging data for use by each container and / or for use by the facility 902 after processing by a pipeline (e.g., to convert outputs back into a usable data type for storage and display at the facility 902). In at least one embodiment, a combination of containers within software 918 (e.g., forming a pipeline) can be referred to as a virtual instrument (as described in more detail herein), and a virtual instrument can utilize service 920 and hardware 922 to perform some or all of the processing tasks of applications instantiated in containers.

[0101] In at least one embodiment, data can be preprocessed as part of the data processing pipeline to prepare it for processing by one or more applications. In at least one embodiment, postprocessing can be performed on the output of one or more inference tasks or other processing tasks of a pipeline to prepare output data for a subsequent application and / or to prepare output data for transmission and / or use by a user (e.g., in response to an inference request). In at least one embodiment, inference tasks can be performed by one or more machine learning models, such as trained or deployed neural networks, which may include output model(s) 916 of the training system 904.

[0102] In at least one embodiment, tasks of the data processing pipeline can be encapsulated in one or more containers, each representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications can be published in a private (e.g., access-restricted) area of ​​a container register (described in more detail here), and trained or deployed models can be stored in model register 924 and associated with one or more applications. In at least one embodiment, images of applications (e.g.,Container images) are available in a container registry, and once they have been selected by a user from a container registry for use in a pipeline, an image can be used to create a container for instantiating an application for use by a user system.

[0103] In at least one embodiment, developers can develop, publish, and store applications (e.g., as containers) for performing processing and / or inference on supplied data. In at least one embodiment, development, publication, and / or storage can be performed using a software development kit (SDK) associated with a system (e.g., to ensure that an application and / or a developed container is compliant or compatible with a system). In at least one embodiment, an application under development can be tested locally (e.g., on a first facility, on data from a first facility) using an SDK that includes at least some of the services 920 as a system (e.g., System 1000 of Fig. 10) can support. In at least one embodiment, once it has been validated by System 1000 (e.g., with regard to accuracy, etc.), an application may be available in a container registry for selection and / or embodiment by a user (e.g., a hospital, clinic, laboratory, healthcare provider, etc.) to perform one or more processing tasks relating to data at a facility (e.g., a second facility) of a user.

[0104] In at least one embodiment, developers can then distribute applications or containers through a network for access and use by users of a system (e.g., System 1000 from Fig. 10) share. In at least one embodiment, completed and validated applications or containers can be stored in a container register, and associated machine learning models can be stored in model register 924. In at least one embodiment, a requesting entity providing an inference or image processing request can search a container register and / or model register 924 for an application, container, dataset, machine learning model, etc., select a desired combination of elements to include in a data processing pipeline, and submit a processing request. In at least one embodiment, a request can include input data necessary to execute a request and / or a selection of application(s) and / or machine learning models to be executed when processing a request.In at least one embodiment, a request can then be passed to one or more components of the deployment system 906 (e.g., a cloud) to perform processing in a data processing pipeline. In at least one embodiment, the processing by the deployment system 906 can involve referencing selected elements (e.g., applications, containers, models, etc.) from a container register and / or model register 924. In at least one embodiment, results, once generated by a pipeline, can be returned to a user for reference (e.g., for viewing in a viewing application suite running on a local workstation or a local terminal).

[0105] In at least one embodiment, Service 920 can be used to assist in processing or executing applications or containers in pipelines. In at least one embodiment, Service 920 can include compute services, collaborative content creation services, simulation services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, Service 920 can provide functionality common to one or more applications in Software 918, such that the functionality can be abstracted to a service that can be called or used by applications. In at least one embodiment, the functionality provided by Service 920 can run dynamically and more efficiently, while also scaling well by enabling applications to process data in parallel, for example, using a parallel computing platform 1030. Fig. 10) In at least one embodiment, Service 920 can be shared between and among different applications, instead of each application sharing the same functionality offered by a Service 920 having to have its own instance of Service 920. In at least one embodiment, services can include an inference server or inference engine that can be used to perform detection or segmentation tasks as non-restrictive examples. In at least one embodiment, a model training service can be included that can provide capabilities for training and / or retraining machine learning models.

[0106] In at least one embodiment, where a service 920 includes an AI service (e.g., an inference service), one or more machine learning models associated with an anomaly detection application (e.g., tumors, growth abnormalities, scarring, etc.) can be executed by calling (e.g., as an API call) an inference service (e.g., an inference server) to execute one or more machine learning models or processing thereof as part of the application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application can call an inference service to execute machine learning models to perform one or more processing operations associated with segmentation tasks.In at least one embodiment, software 918 implementing an advanced processing and inference pipeline can be streamlined, since each application can call the same inference service to perform one or more inference tasks.

[0107] In at least one embodiment, the hardware can include 922 GPUs, CPUs, graphics cards, an AI / deep learning system (e.g., an AI supercomputer such as the DGX). TM-NVIDIA supercomputer system), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 922 can be used to provide efficient, purpose-built support for software 918 and service 920 in the deployment system 906. In at least one embodiment, the use of GPU processing for local processing (e.g., at the facility 902), within an AI / deep learning system, in a cloud system, and / or in other processing components of the deployment system 906 can be implemented to improve the efficiency, accuracy, and effectiveness of game name recognition.

[0108] In at least one embodiment, software 918 and / or service 920 can be optimized for GPU processing with respect to deep learning, machine learning and / or high-performance computing, simulation and visual computing, as non-limiting examples. In at least one embodiment, at least a part of the computing environment of the deployment system 906 and / or the training system 904 can be located in a data center or one or more supercomputers or high-performance computing systems with GPU-optimized software (e.g., a hardware and software combination of the DGX). TM-Systems from NVIDIA). In at least one embodiment, Hardware 922 can include any number of GPUs that can be called upon to perform parallel data processing as described herein. In at least one embodiment, the cloud platform can further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computational tasks. In at least one embodiment, the cloud platform (e.g., NGC) can include TM from NVIDIA) using one or more AI / deep learning supercomputers and / or GPU-optimized software (e.g., as on the DGX TM(provided by NVIDIA systems) can be run as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform can integrate an application container clustering system or orchestration system (e.g., Kubernetes) across multiple GPUs to enable seamless scaling and load balancing.

[0109] Fig. Figure 10 is a system diagram for an exemplary system 1000 for generating and implementing an inference pipeline according to at least one embodiment. In at least one embodiment, the system 1000 can be used to perform the process 900 of Fig. 9 and / or other processes, including advanced processing and inference pipelines. In at least one embodiment, the system 1000 may include the training system 904 and the deployment system 906. In at least one embodiment, the training system 904 and the deployment system 906 may be implemented using software 918, services 920, and / or hardware 922, as described herein.

[0110] In at least one embodiment, System 1000 (e.g., Training System 904 and / or Deployment System 906) can be implemented in a cloud computing environment (e.g., using Cloud 1026). In at least one embodiment, System 1000 can be implemented locally with respect to a facility or as a combination of both cloud and local computing resources. In at least one embodiment, access to APIs in Cloud 1026 can be restricted to authorized users through security measures or protocols. In at least one embodiment, a security protocol can include web tokens that can be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and carry appropriate authorization.In at least one embodiment, APIs of virtual instruments (described here) or other instantiations of System 1000 can be restricted to a set of public Internet service providers (ISPs) that have been vetted or authorized for interaction.

[0111] In at least one embodiment, various components of System 1000 can communicate with each other using any of a variety of different network types, including local area networks (LANs) and / or wide area networks (WANs), via wired and / or wireless communication protocols. In at least one embodiment, communication between devices and components of System 1000 (e.g., for transmitting inference requests, receiving results of inference requests, etc.) can be carried out via a data bus or data buses, wireless data protocols (e.g., Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0112] In at least one embodiment, the training system 904 can execute training pipelines 1004, which are described here in relation to Fig. 9 are similar to those described. In at least one embodiment, in which one or more machine learning models are to be used in a deployment pipeline 1010 by the deployment system 906, one or more training pipelines 1004 can be used to train or retrain one or more (e.g., pre-trained) models and / or to implement one or more pre-trained models 1006 (e.g., without the need for retraining or updating). In at least one embodiment, output model(s) 916 can be generated as a result of a training pipeline 1004.In at least one embodiment, one or more training pipelines 1004 may include any number of processing steps, AI-assisted annotation 910, tagging or annotating feedback data 908 to generate tagged data 912, model selection from a model register, model training 914, training, retraining, or updating models, and / or other processing steps. In at least one embodiment, the DICOM adapter 1002a may be used to access DICOM data. In at least one embodiment, different training pipeline(s) 1004 may be used for different machine learning models used by the deployment system 906. In at least one embodiment, one or more training pipelines 1004 may be used, each with respect to . Fig. Similar to the first example described in 9, for a first machine learning model, one or more training pipelines 1004 can be used, which are similar to Fig. The second example described in 9 is similar, can be used for a second machine learning model, and can include one or more training pipelines 1004, which are related to Fig.Similar to the third example described in 9, a third machine learning model can be used. In at least one embodiment, any combination of tasks within the training system 904 can be used, depending on the requirements of each machine learning model. In at least one embodiment, one or more machine learning models can already be trained and ready for use, so that machine learning models may not require processing by the training system 904 and can be implemented by the deployment system 906.

[0113] In at least one embodiment, one or more output models 916 and / or pretrained models 1006 may include any type of machine learning model, depending on the embodiment. In at least one embodiment, and without limitation, machine learning models used by the system 1000 may include one or more machine learning models employing linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbor (Knn), K-means clustering, random forest, dimensionality reduction algorithms, gradient enhancement algorithms, neural networks (e.g., autocoders, convolutional networks, recurrent networks, perceptrons, long / short term memory (LSTM), Bi-LSTM, Hopfield, Boltzmann, deep belief, unfolding networks, generative adversarial, fluid state machines, etc.), and / or other types of machine learning models.

[0114] In at least one embodiment, one or more training pipelines can include AI-assisted annotation. In at least one embodiment, annotated data (e.g., traditional annotation) can be generated by any number of techniques. In at least one embodiment, in some examples, markings or other annotations can be generated within a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a marking program, another type of program suitable for generating annotations or markings for ground truth, and / or can be hand-drawn. In at least one embodiment, ground truth data can be synthetically produced (e.g., generated from computer models or renderings), physically produced (e.g., designed and produced from real-world data), or machine-automated (e.g.,using feature analysis and learning to extract features from data and then generate tags), be annotated by a human (e.g., tagger or annotation expert, defines the location of tags) and / or a combination thereof. In at least one embodiment, for each instance of feedback data 908 (or other data type used by machine learning models), corresponding ground-truth data generated by the training system 904 may be available. In at least one embodiment, AI-assisted annotation may be performed as part of deployment pipeline(s) 1010; either in addition to or instead of AI-assisted annotation included in the training pipeline(s) 1004. In at least one embodiment, the system 1000 may include a multi-layered platform comprising a software layer (e.g.,Software 918) may include diagnostic applications (or other application types) that can perform one or more medical imaging and diagnostic functions.

[0115] In at least one embodiment, a software layer can be implemented as a secure, encrypted, and / or authenticated API through which applications or containers can be called from an external environment (or environments), e.g., facility 902. In at least one embodiment, applications can then call or execute one or more services 920 to perform computational, AI, or visualization tasks associated with the respective applications, and the software 918 and / or services 920 can utilize hardware 922 to perform processing tasks effectively and efficiently.

[0116] In at least one embodiment, the deployment system 906 can execute deployment pipelines 1010. In at least one embodiment, one or more deployment pipelines 1010 can include any number of applications that can be applied sequentially, non-sequentially, or otherwise to feedback data (and / or other data types), including AI-assisted annotation, as described above. In at least one embodiment, as described herein, one or more deployment pipelines 1010 for a single device can be designated as a virtual instrument for that device. In at least one embodiment, there can be more than one deployment pipeline 1010 for a single device, depending on information desired from data generated by that device.

[0117] In at least one embodiment, the applications available for the deployment pipeline(s) 1010 can include any application that can be used to perform processing tasks on feedback data or other data from devices. In at least one embodiment, since different applications can share common image operations, a data extension library (e.g., as one of the services 920) can be used in some embodiments to accelerate these operations. In at least one embodiment, to avoid bottlenecks of conventional processing approaches that rely on CPU processing, the parallel computing platform 1030 can be used to GPU-accelerate these processing tasks.

[0118] In at least one embodiment, the deployment system 906 may include a user interface (UI) 1014 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications for inclusion in one or more deployment pipelines 1010, to arrange applications, to modify or change applications or parameters or constructs thereof, to use and interact with one or more deployment pipelines 1010 during setup and / or deployment, and / or to otherwise interact with the deployment system 906. In at least one embodiment, although not illustrated with respect to the training system 904, the UI 1014 (or another user interface) may be used to select models for use in the deployment system 906, to select models for training or retraining in the training system 904, and / or to otherwise interact with the training system 904.

[0119] In at least one embodiment, the pipeline manager 1012 can be used in addition to an application orchestration system 1028 to manage the interaction between applications or containers of deployment pipeline(s) 1010 and services 920 and / or hardware 922. In at least one embodiment, the pipeline manager 1012 can be configured to facilitate application-to-application, application-to-service 920, and / or application-or-service-to-hardware 922 interactions. While this is illustrated as being contained in software 918 in at least one embodiment, this is not intended to be limiting, and in some examples, the pipeline manager 1012 can be contained in services 920. In at least one embodiment, the application orchestration system 1028 (e.g., Kubernetes, Docker, etc.) can be configured to facilitate application-to-application, application-to-service 920, and / or application-to-service or hardware 922 interactions.) include a container orchestration system that can group applications as logical units for coordination, management, scaling, and deployment into containers. In at least one embodiment, each application can be run by assigning applications from deployment pipeline(s) 1010 (e.g., a reconstruction application, a segmentation application, etc.) to individual containers in a self-contained environment (e.g., at a kernel level) to increase speed and efficiency.

[0120] In at least one embodiment, each application and / or container (or each image thereof) can be developed, modified, and deployed individually (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separate from the first user or developer). This allows users to focus on and attend to the tasks of a single application and / or container without being hindered by tasks of other application(s) or container(s). In at least one embodiment, communication and collaboration between different containers or applications can be supported by the pipeline manager 1012 and the application orchestration system 1028.In at least one embodiment, as long as an expected input and / or output of each container or application is known to a system (e.g., based on constructs of applications or containers), the application orchestration system 1028 and / or the pipeline manager 1012 can facilitate communication and resource sharing between and between each of the applications or containers. In at least one embodiment, since one or more of the applications or containers in the deployment pipeline(s) 1010 can share the same services and resources, the application orchestration system 1028 can orchestrate, load balance, and determine the sharing of services or resources between and between different applications or containers.In at least one embodiment, a scheduler can be used to track resource requests from applications or containers, the current or planned use of these resources, and resource availability. In at least one embodiment, the scheduler can thus allocate resources to different applications and distribute resources between and between applications with respect to system requirements and availability. In some examples, the scheduler (and / or another component of the application orchestration system 1028) can determine resource availability and distribution based on constraints imposed on a system (e.g., user constraints), such as quality of service (QoS), the urgency of data output (e.g., to determine whether real-time or delayed processing should be performed), and so on.

[0121] In at least one embodiment, services 920, which are used and shared by applications or containers in the deployment system 906, can include compute service(s) 1016, collaborative content generation service(s) 1017, AI service(s) 1018, simulation service(s) 1019, visualization service(s) 1020, and / or other service types. In at least one embodiment, applications can call (e.g., execute) one or more of the services 920 to perform processing operations for an application. In at least one embodiment, compute service(s) 1016 can be used by applications to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, compute service(s) 1016 can be used for parallel processing (e.g.,using a parallel computing platform 1030) for processing data by one or more applications and / or performing one or more tasks of a single application substantially simultaneously. In at least one embodiment, the parallel computing platform 1030 (e.g., CUDA) can be used. ®from NVIDIA) enable general-purpose computing on GPUs (GPGPU) (e.g., GPUs / Graphics 1022). In at least one embodiment, a software layer of the Parallel Computing Platform 1030 can provide access to virtual instruction sets and parallel computational elements of GPUs for executing computation cores. In at least one embodiment, the Parallel Computing Platform 1030 can include memory, and in some embodiments, memory can be shared between and among multiple containers and / or between and among different processing tasks within a single container. In at least one embodiment, inter-process communication (IPC) calls can be generated for multiple containers and / or for multiple processes within a container to use the same data from a shared segment of the memory of the Parallel Computing Platform 1030 (e.g.,where several different stages of an application or multiple applications process the same information). In at least one embodiment, instead of creating a copy of data and moving data to different locations in memory (e.g., a read / write operation), the same data can be used in the same location of memory for any number of processing tasks (e.g., at the same time, at different times, etc.). In at least one embodiment, since data is used to generate new data as a result of processing, this information can be stored in a new location of data and shared between different applications. In at least one embodiment, the location of data and a location of updated or modified data can be part of a definition of how a payload is understood within containers.

[0122] In at least one embodiment, AI service(s) 1018 can be used to perform inference services for executing machine learning model(s) assigned to applications (e.g., tasked with performing one or more processing tasks of an application). In at least one embodiment, AI service(s) 1018 can utilize AI system(s) 1024 to execute machine learning model(s) (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, applications of deployment pipeline(s) 1010 can use one or more of output model(s) 916 from training system 904 and / or other application models to perform inference on imaging data (e.g., DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.).For example, the DICOM adapter 1002b can be used to access DICOM data. In at least one embodiment, two or more examples of inferences using the application orchestration system 1028 (e.g., a scheduler) can be available. In at least one embodiment, a first category can include a high-priority / low-latency path that can reach higher-level service agreements, such as for performing an inference on urgent requests during an emergency or for a radiologist during diagnosis. In at least one embodiment, a second category can include a standard-priority path that can be used for requests that may not be urgent or where analysis can be performed at a later time. In at least one embodiment, the application orchestration system 1028 can access resources (e.g.,Distribute services 920 and / or hardware 922) based on priority paths for different inference tasks of AI service(s) 1018.

[0123] In at least one embodiment, a shared memory can be mounted on AI service(s) 1018 within the system 1000. In at least one embodiment, the shared memory can function as a cache (or another type of storage device) and can be used to process inference requests from applications. In at least one embodiment, when an inference request is submitted, the request can be received by a set of API instances of the deployment system 906, and one or more instances can be selected (e.g., for best fit, load balancing, etc.) to process the request.In at least one embodiment, to process a request, a request can be entered into a database; a machine learning model can be located in the model register 924 if it is not already in a cache; a validation step can ensure that a suitable machine learning model is loaded into a cache (e.g., shared memory); and / or a copy of a model can be stored in a cache. In at least one embodiment, the scheduler (e.g., of the pipeline manager 1012) can be used to start an application referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, if an inference server is not already started to execute a model, an inference server can be started.In at least one embodiment, any number of inference servers can be started per model. In at least one embodiment, models can be cached in a pull model where inference servers are clustered, whenever load balancing is advantageous. In at least one embodiment, inference servers can be statically loaded onto corresponding distributed servers.

[0124] In at least one embodiment, inference can be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server can be associated with a model (and optionally with multiple versions of a model). In at least one embodiment, if an instance of an inference server does not exist when a request to perform inferences on a model is received, a new instance can be loaded. In at least one embodiment, a model can be passed to an inference server when it is started, so that the same container can be used to serve different models as long as the inference server is running as a separate instance.

[0125] In at least one embodiment, an inference request for a given application can be received during application execution, and a container (e.g., hosting an instance of an inference server) can be loaded (if not already loaded), and a start procedure can be invoked. In at least one embodiment, preprocessing logic within a container can load, decode, and / or perform any additional preprocessing on incoming data (e.g., using a CPU(s) and / or GPU(s)). In at least one embodiment, once data is prepared for inference, a container can perform inferences on demand. In at least one embodiment, this can involve a single inference call on a single image (e.g., a handheld X-ray) or it can require inferences on hundreds of images (e.g., a breast CT scan).In at least one embodiment, an application can summarize results prior to completion, which, without limitation, can include a single confidence score, pixel-level segmentation, voxel-level segmentation, generating a visualization, or generating text to summarize results. In at least one embodiment, different models or applications can be assigned different priorities. For example, some models can have a real-time priority (turnaround time less than one minute), while others can have a lower priority (e.g., turnaround time less than 10 minutes). In at least one embodiment, model execution times can be measured by a requesting institution or entity and can include partner network throughput time as well as execution on an inference service.

[0126] In at least one embodiment, the transmission of requests between Services 920 and inference applications can be hidden behind a software development kit (SDK) and robust transport can be provided through a queue. In at least one embodiment, a request is queued via an API for a single application / tenant ID combination, and an SDK pulls a request from the queue and delivers it to an application. In at least one embodiment, a queue name can be provided in an environment from which an SDK retrieves the request. In at least one embodiment, asynchronous communication through a queue can be beneficial because it allows any instance of an application to begin work as it becomes available.In at least one embodiment, results can be passed back through a queue to ensure that no data is lost. In at least one embodiment, queues can also provide the ability to segment work, as highest-priority work can go to a queue with the most instances of an associated application, while lowest-priority work can go to a queue with a single associated instance, processing tasks in a received order. In at least one embodiment, an application can run on a GPU-accelerated instance created in Cloud 1026, and an inference service can perform inferences on a GPU.

[0127] In at least one embodiment, visualization service(s) 1020 can be used to generate visualizations for viewing outputs from applications and / or deployment pipeline(s) 1010. In at least one embodiment, GPUs / graphics 1022 can be used by visualization service(s) 1020 to generate visualizations. In at least one embodiment, rendering effects, such as ray tracing or other light transport simulation techniques, can be implemented by visualization service(s) 1020 to generate higher-quality visualizations. In at least one embodiment, visualizations can include, without limitation, 2D image renderings, 3D volume renderings, 3D volume reconstruction, 2D tomography slices, virtual reality displays, augmented reality displays, etc. In at least one embodiment, virtualized environments can be used to provide a virtual interactive display or environment (e.g.,to create a virtual environment for interaction by users of a system (e.g., doctors, nurses, radiologists, etc.). In at least one embodiment, visualization service(s) 1020 may include an internal visualizer, kinetics, and / or other rendering or image processing capabilities or functionality (e.g., ray tracing, rasterization, internal optics, etc.).

[0128] In at least one embodiment, Hardware 922 can include GPUs / graphics 1022, AI system(s) 1024, cloud 1026 and / or any other hardware used to run the training system 904 and / or the deployment system 906. In at least one embodiment, GPUs / graphics 1022 (e.g., TESLA) can include ® - and / or QUADRO ®NVIDIA GPUs) include any number of GPUs that can be used to perform processing tasks of Computing Service(s) 1016, Collaborative Content Creation Service(s) 1017, AI Service(s) 1018, Simulation Service(s) 1019, Visualization Service(s) 1020, other services, and / or any features or functionality of Software 918. For example, with respect to AI Service(s) 1018, GPUs / Graphics 1022 can be used to perform preprocessing on imaging data (or other data types used by machine learning models), postprocessing on outputs of machine learning models, and / or to perform inference (e.g., to run machine learning models). In at least one embodiment, Cloud 1026, AI system(s) 1024 and / or other components of the system 1000 can use GPUs / graphics 1022.In at least one embodiment, Cloud 1026 can include a GPU-optimized platform for deep learning tasks. In at least one embodiment, AI System(s) 1024 can utilize GPUs, and Cloud 1026—or at least a section tasked with deep learning or inference—can run using one or more AI System(s) 1024. Although Hardware 922 is illustrated as discrete components, this is not intended to be restrictive, and any components of Hardware 922 can be combined with or utilized by any other components of Hardware 922.

[0129] In at least one embodiment, AI system(s) 1024 may include a dedicated computing system (e.g., a supercomputer or an HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, AI system(s) 1024 (e.g., DGX) may include TM from NVIDIA) include GPU-optimized software (e.g., a software stack) that can be run using a variety of GPUs / graphics 1022, in addition to CPUs, RAM, storage, and / or other components, features, or functionality. In at least one embodiment, one or more AI system(s) 1024 can be deployed in the cloud 1026 (e.g., in a data center) to perform some or all of the AI-based processing tasks of the system 1000.

[0130] In at least one embodiment, Cloud 1026 can utilize a GPU-accelerated infrastructure (e.g., NGC). TMfrom NVIDIA) that can provide a GPU-optimized platform for performing processing tasks of the System 1000. In at least one embodiment, Cloud 1026 can include one or more AI System(s) 1024 for performing one or more AI-based tasks of the System 1000 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, Cloud 1026 can be integrated with an Application Orchestration System 1028 that utilizes multiple GPUs to enable seamless scaling and load balancing between and within applications and Services 920. In at least one embodiment, Cloud 1026 can be tasked with running at least some of the Services 920 of the System 1000, including Computing Service(s) 1016, AI Service(s) 1018, and / or Visualization Service(s) 1020, as described herein. In at least one embodiment, Cloud 1026 can perform small and large stack inference (e.g.Running NVIDIA's TensorRT™), an accelerated parallel computing platform 1030 (e.g. CUDA). ® from NVIDIA), run the Application Orchestration System 1028 (e.g., Kubernetes), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher-quality kinetics), and / or provide other functionality for System 1000. In at least one embodiment, the Parallel Computing Platform 1030 may include an API.

[0131] In at least one embodiment, Cloud 1026 can include a registry, such as a deep learning container registry, in an effort to maintain patient confidentiality (e.g., where patient data or datasets are to be used off-site). In at least one embodiment, a registry can store containers for instantiations of applications that can perform preprocessing, postprocessing, or other processing tasks on patient data. In at least one embodiment, Cloud 1026 can receive data containing patient data as well as sensor data in containers, perform requested processing only on sensor data in these containers, and then deliver a resulting output and / or visualizations to suitable parties and / or devices (e.g.,Medical devices in rooms used for visualization or diagnosis) can transmit data without extracting, storing, or otherwise accessing patient data. In at least one embodiment, Cloud 1026 can process data containing patient and sensor data in containers to process the output and / or visualizations. In at least one embodiment, the confidentiality of patient data is maintained in accordance with HIPAA and / or other data protection regulations.

[0132] Other variations are within the spirit of the present disclosure. While disclosed techniques are receptive to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described in detail above. It is understood, however, that the disclosure is not intended to be limited to specific disclosed forms or shapes, but rather, on the contrary, to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure, as defined in the attached claims.

[0133] The use of the terms "a," "an," and "the" and similar references in the context of describing disclosed embodiments (particularly in the context of the following claims) is to be interpreted as covering both the singular and the plural unless otherwise specified herein or the context clearly contradicts this, and not as a definition of a term. The terms "comprising," "having," "including," and "containing" are to be interpreted as open terms (meaning "including but not limited to") unless otherwise specified. "Connected," when not modified and referring to physical connections, is to be interpreted as partially or completely contained in, attached to, or connected to one another, even if something is in between.The recitation of value ranges is intended here merely as a short-lived procedure for individually referring to each separate value falling within the range, unless otherwise specified herein, and each separate value is included in the specification as if it were being individually recited here. In at least one embodiment, the use of the term "sentence" (e.g., "a set of elements") or "subsentence," unless otherwise specified or the context contradicts, is to be interpreted as a non-empty collection comprising one or more elements. Furthermore, unless otherwise specified or the context contradicts, the term "subsentence" of a corresponding sentence does not necessarily denote a proper subsentence of the corresponding sentence, but subsentence and corresponding sentence may be the same.

[0134] Subjunctive language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," unless explicitly stated otherwise or the context clearly contradicts it, is otherwise understood with context as it is generally used to indicate that an element, term, etc., can be either A, B, C, or any non-empty subset of a sentence containing A, B, and C. For example, in illustrative examples of a three-element sentence, subjunctive phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any one of the following: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such subjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of A, at least one of B, and at least one of C, respectively.Additionally, unless otherwise stated or the context contradicts, the term "multiple" indicates a plural state (e.g., "a multitude of elements" indicates multiple elements). In at least one embodiment, a number of elements in a multitude is at least two, but may be more if either explicitly stated or indicated by context. Furthermore, unless otherwise stated or the context clearly contradicts, the phrase "based on" means "at least partly based on" or "at least partially based on" and not "exclusively based on".

[0135] Operations of the processes described herein may be performed in any suitable order unless otherwise specified herein or the context clearly contradicts this. In at least one embodiment, a process such as the processes described herein (or variations and / or combinations thereof) is carried out under the control of one or more computer systems configured with executable instructions and is implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors by hardware or combinations thereof. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that are executable by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-volatile computer-readable storage medium that excludes volatile signals (e.g., a propagating transient electrical or electromagnetic transmission) but includes a non-volatile data storage circuit arrangement (e.g., buffers, cache, and queues) within transceivers of volatile signals. In at least one embodiment, code (e.g., executable code or source code) is stored on a set of one or more non-volatile computer-readable storage media on which executable instructions (or other memory for storing executable instructions) are stored. When executed (i.e., as a result of their execution) by one or more processors of a computer system, these instructions cause the computer system to perform the operations described herein.In at least one embodiment, the set of non-volatile, computer-readable storage media comprises several non-volatile, computer-readable storage media, and one or more individual non-volatile storage media of several non-volatile, computer-readable storage media lack all the code, while several non-volatile, computer-readable storage media collectively store all the code. In at least one embodiment, executable instructions are executed such that different instructions are executed by different processors—for example, a non-volatile, computer-readable storage medium stores instructions, and a central processing unit (“CPU”) executes some of the instructions, while a graphics processing unit (“GPU”) executes other instructions.In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0136] Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that individually or jointly perform operations of the processes described herein, and such computer systems are configured with applicable hardware and / or software that enables the performance of operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, it is a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs the operations described herein and such that a single device does not perform all operations.

[0137] The use of any and all of the examples or exemplary language provided herein (e.g., "like") is intended only to better illustrate embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless otherwise claimed. No language in the specification should be interpreted as indicating any unclaimed element as essential for the practical implementation of the disclosure.

[0138] All references, including publications, patent applications and patents cited herein, are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated as incorporated by reference and set forth herein in its entirety.

[0139] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It is understood that these terms may not be intended as synonyms. Rather, in certain examples, "connected" or "coupled" may be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" may also mean that two or more elements are not in direct contact with each other but nevertheless cooperate or interact.

[0140] Unless expressly stated otherwise, in some embodiments, terms such as "processing", "calculating", "calculating", "determining" or the like are understood to refer throughout the specification to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical, such as electronic, quantities within the registers and / or memory of the computing system into other data similarly represented as physical quantities within the memory, registers or other such information storage, transmission or display devices of the computing system.

[0141] Similarly, the term "processor" can refer to any device or section of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As non-restrictive examples, "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used here, "software" processes can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each process can refer to multiple processes for executing instructions sequentially or in parallel, continuously or intermittently.In at least one embodiment, the terms “system” and “method” are used interchangeably here, provided that a system can embody one or more methods and methods can be considered as a system.

[0142] This document may refer to the acquisition, capture, reception, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, a process of acquiring, capturing, receiving, or inputting analog and digital data can be achieved in a variety of ways, such as receiving data as a parameter of a function call or a call to an application programming interface. In at least one embodiment, processes of acquiring, capturing, receiving, or inputting analog or digital data can be achieved by transmitting data over a serial or parallel interface.In at least one embodiment, processes of obtaining, acquiring, receiving, or inputting analog or digital data can be achieved by transmitting data over a computer network from a providing entity to an acquiring entity. In at least one embodiment, reference can also be made to providing, outputting, transmitting, sending, or displaying analog or digital data. In various examples, processes of providing, outputting, transmitting, sending, or displaying analog or digital data can be achieved by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0143] Although the descriptions presented here are exemplary embodiments of the described techniques, other architectures may be used to implement the described functionality and are intended to be within the scope of this disclosure. While specific distributions of responsibilities may be defined above for the purpose of description, various functions and responsibilities may also be distributed and subdivided in different ways depending on the circumstances.

[0144] Although the subject matter was described in a language specific to structural features and / or methodological actions, it is further understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms of implementing the claims.

Claims

[1] Procedure, encompassing: Retrieving a consumer task from a memory that depends on a producer task performed by a processing device; Retrieving a first, different task from memory; In response to the fact that a first execution state of the producer task fulfills a first execution criterion at a first time point, the processing device provides the first other task for execution; and In response to the fact that a second execution state of the producer task fulfills a second execution criterion at a second time, the consumer task is made available for execution by the processing device. [2] Method according to claim 1, wherein the retrieval of the consumer task from the memory, which depends on the producer task executed by the processing device, takes place in response to the fact that the first execution state of the producer task satisfies the first execution criterion at the first time point. [3] Method according to claim 1 or 2, further comprising: In response to the fact that the first execution state of the producer task fulfills the first execution criterion at the first time point, a lower priority is set for the consumer task; Retrieving a second, different task with a corresponding priority from memory; Determine, among at least the consumer task and the second other task, a next task to be performed by the processing device based on the priorities; and Providing the next task for execution by the processing device. [4] The method of claim 3, further comprising: In response to a third execution state of the producer task fulfilling a third execution criterion at a third time, the priority of the consumer task is restored to an original priority. [5] Method according to any of the preceding claims, wherein the producer task generates data that is to be consumed by more than one consumer task. [6] Method according to any of the preceding claims, wherein the consumer task further depends on another producer task. [7] Method according to one of the preceding claims, wherein the provision of the consumer task for execution by the processing device further responds to a resource availability of the processing device that satisfies a resource criterion. [8] Circuit coupled to a parallel processing unit and a memory, wherein the circuit: retrieves a consumer task from memory that depends on a producer task executed by the parallel processing unit; retrieves a first other task from memory; in response to the fact that a first execution state of the producer task fulfills a first execution criterion at a first time point, the first other task is made available for execution by the parallel processing unit; and In response to the fact that a second execution state of the producer task fulfills a second execution criterion at a second time point, the consumer task is made available for execution by the parallel processing unit. [9] Circuit according to claim 8, wherein the retrieval of the consumer task from memory, which depends on the producer task executed by the parallel processing unit, occurs in response to the fact that the first execution state of the producer task satisfies the first execution criterion at the first time point. [10] Circuit according to claim 8 or 9, wherein the circuit further: as a reaction to the fact that the first execution state of the producer task fulfills the first execution criterion at the first time point, a priority of the consumer task is lowered; retrieves a second, different task with a corresponding priority from memory; From at least the consumer task and the second other task, a next task is determined for execution by the parallel processing unit based on the priorities; and provides the next task for execution by the parallel processing unit. [11] Circuit according to claim 10, wherein the circuit further: In response to a third execution state of the producer task fulfilling a third execution criterion at a third time point, the priority of the consumer task is restored to an original priority. [12] Circuit according to one of claims 8 to 11, wherein the generator task generates data which is to be consumed by more than one consumer task. [13] Circuit according to one of claims 8 to 12, wherein the consumer task further depends on another producer task. [14] Circuit according to one of claims 8 to 13, wherein the provision of the consumer task for execution by the parallel processing unit further responds to a resource availability of the parallel processing unit that satisfies a resource criterion. [15] Procedures, including: Providing an initial producer task for execution by a processing device; Providing a second producer task for execution by the processing device; in response to the completion of the first producer task and the second producer task, cause one or more cleanup operations to be performed for both the first producer task and the second producer task; and in response to the completion of one or more cleanup operations, providing a first consumer task for execution by the processing device, wherein the first consumer task depends on the first producer task and on the second producer task. [16] Method according to claim 15, wherein the one or more cleanup operations comprise at least one of the following for both the first producer task and the second producer task: Emptying a first memory relating to the first producer task and a second memory relating to the second producer task; or Invalidate first caches relating to the first producer task and second caches relating to the second producer task. [17] Method according to claim 15 or 16, wherein causing the one or more cleanup operations to be carried out comprises providing a completion tracking task for execution by the processing device, wherein the completion tracking task depends on the first producer task and on the second producer task. [18] Method according to any one of claims 15 to 17, wherein the first producer task and the second producer task have a common barrier dependency. [19] Method according to any one of claims 15 to 18, further comprising: In response to the completion of one or more cleanup operations, a second consumer task is provided for execution by the processing device, wherein the second consumer task depends on the first producer task and on the second producer task. [20] Method according to any one of claims 15 to 19, wherein the first consumer task generates data which is consumed by another task.

Images