IMPROVED TASKS EXECUTION

Abstract

Systems and methods for improving task execution throughput by a processor are disclosed. The methods include executing a first task with associated processor configuration requirements by a processing unit. The methods further include receiving a second and a third task to be executed by the processing unit. The second and third tasks each have associated processor configuration requirements. The methods further include determining a first next task to be executed by the processing unit based on a comparison of the processing unit's current processor configuration with the processor configuration requirements of the second and third tasks. The processing unit's current processor configuration is based on the processor configuration requirements associated with the first task.The procedures also include providing the processing unit with the first next task for execution.

Description

TECHNICAL AREA

[0001] At least one embodiment relates to the distribution of tasks for execution by a processor, and in particular the distribution of tasks for execution in a sequence that differs from the sequence in which the tasks were received. BACKGROUND

[0002] Processors (e.g., central processing units (CPUs), graphics processing units (GPUs)) can execute tasks from multiple processes simultaneously. Tasks can have associated priorities and can be executed based on these priorities. Before execution, tasks can be loaded from memory and stored in a scheduling table, which may have limited space (e.g., due to hardware limitations). An in-memory task descriptor can be used to track tasks (e.g., to be able to pause a running task and resume the next task selected for execution based on priority) while they are in the scheduling table.The in-memory task descriptor can contain a task state associated with a task, such as processor configuration requirements, resource requirements, a program / shader associated with the task, and / or the like. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a block diagram of an exemplary system for improved processor throughput according to at least one embodiment. Fig. Figure 2A is a block diagram of an exemplary efficient distribution planning table according to at least one embodiment. Fig. Figure 2B is a block diagram of an exemplary efficient distribution planning table according to at least one embodiment. Fig. Figure 3 is a flowchart of an exemplary procedure for efficient distribution of a task to execution by a processor according to at least one embodiment. Fig. Figure 4 is a block diagram illustrating an exemplary computer system according to at least one embodiment of the present disclosure. Fig. Figure 5A illustrates an inference and / or training logic according to at least one embodiment of the present disclosure. Fig. 5B illustrates an inference and / or training logic according to at least one embodiment. Fig. Figure 6 illustrates the training and deployment of a neural network according to at least one embodiment. Fig. Figure 7 is an exemplary data flow diagram for an advanced computing pipeline according to at least one embodiment. Fig. Figure 8 is a system diagram for an exemplary system for training, adapting, instantiating and deploying machine learning models in an advanced computing pipeline according to at least one embodiment. DETAILED DESCRIPTION

[0003] Tasks are often executed in the order they are received. For example, a processor scheduler might receive (e.g., load) multiple tasks from memory into a scheduling table. These tasks can then be distributed for execution by one or more processors in the order they were loaded into the table. As discussed above, each task may have its own processor configuration requirements, resource requirements, and / or associated program / shader (e.g., a shader program or script).

[0004] Processor configuration requirements can include one or more properties of a processor that can be set to enable the processor to perform a task. In some implementations, the processor configuration requirements may include a Level 1 (L1) cache memory configuration. For example, the processor configuration requirements may specify how much tagged (or untagged) L1 memory is required to perform a particular task.

[0005] To execute an initial task with initial processor configuration requirements, a processor can be reconfigured to meet those requirements. For example, the processor's memory can be modified to match the memory configuration specified in the processor configuration requirements. After the initial task has completed, if a second task has subsequent processor configuration requirements that differ from the initial requirements, the processor can be reconfigured again to meet those second requirements. Frequent processor reconfigurations can reduce the processor's throughput.

[0006] Additionally, before distributing a task from the scheduling table to a processor for execution, the processor must be queried to determine its resource availability. For example, a task might have 5 threads (or thread blocks or cooperative thread arrays (CTAs)) to execute. After querying the processor, it can be determined that the processor has resources to execute 3 threads (or thread blocks or CTAs). Thus, 3 of the task's 5 threads can be distributed to the processor for execution. After these threads have completed, the processor can be queried again to determine its updated resource availability. If sufficient resources are available to execute the remaining 2 threads of the task, these 2 threads can be distributed to the processor for execution.Frequent processor queries to determine resource availability can also reduce processor throughput.

[0007] Aspects of the present disclosure address the aforementioned and other shortcomings by providing systems and methods that execute tasks in a different order than the order in which the tasks were received. Specifically, the methods distribute tasks for execution based on the processor configuration, resource requirements, and / or the program / shader (e.g., shader program or script) associated with the task to be executed. To improve the throughput of a processor's execution, tasks from a scheduling table can be distributed in such a way as to minimize processor reconfiguration and queries of processor resource availability.For example, if a first, second, and third task are loaded into a scheduling table in that order, and the first and third tasks have the same processor configuration requirements, the third task can be scheduled after the first task has finished executing, thus avoiding the need to reconfigure the processor between the first and third tasks. After the third task has finished executing, the processor can then be reconfigured to execute the second task.

[0008] In another example, a first and a second task might be loaded into a scheduling table, and the tasks might have the same processor configuration requirements. A processor might currently be executing a task that has the same processor configuration requirements as both the first and second tasks in the scheduling table. Before executing the current task, the processor might have been queried to determine its resource availability. If the processor has resource availability for 5 threads and the currently executing task uses only 3 threads, 2 threads from another task with resource requirements matching those of the currently executing task can be allocated for execution without querying the processor again. In some cases, a "thread" can represent a group of threads, such as a thread block or a Cooperative Thread Assembly (CTA).

[0009] Continuing with the previous example, the second task in the scheduling table may have resource requirements that match the currently executing task, while the first task in the scheduling table may have different resource requirements. Two threads of the second task can be distributed for execution by the processor without querying the processor for resource availability. After the second task has finished executing, the processor can be queried again for resource availability before the first task from the scheduling table is executed.

[0010] If the scheduling table contains tasks with the same processor configuration and resource requirements as the currently executing task, a next task can be determined based on the task's program / shader and the program / shader of the currently executing task. For example, if a task in the scheduling table has the same program / shader as the currently executing task, the task from the scheduling table can be distributed to the processor next to take advantage of instruction cache matches, further reducing the time required to execute the task and increasing processor throughput.

[0011] Advantages of the disclosed embodiments over the existing technology include, but are not limited to, an increased task execution throughput of a processor.

[0012] Fig. Figure 1 is a block diagram of an exemplary system 100 for improved processor throughput according to at least one embodiment. The system 100 may include a central processing unit (CPU) 102 and a parallel processing device 106 (e.g., a graphics processing unit (GPU)). In some embodiments, the CPU 102 and the parallel processing device 106 may be located in a different system (e.g., the computer system 400 of Fig. 4) be included. For example, the System 100 may be included in a desktop computer, a server, a laptop, a mobile device, and / or the like. In some embodiments, the parallel processing device 106 may be used to perform machine learning and / or artificial intelligence (AI) tasks. For example, the parallel processing device 106 may be used to train AI models, to perform inference using trained AI models, and / or the like.

[0013] The CPU 102 can include a GPU driver 104 for establishing an interface with the parallel processing device 106. The CPU 102 can send one or more task descriptors to the parallel processing device 106 for execution via the GPU driver 104. Each task descriptor can contain information relating to its execution, such as processor configuration requirements, resource requirements, a program / shader of the task descriptor, and / or the like.

[0014] Processor configuration requirements can include one or more processor properties that can be set to enable the processor to perform the task corresponding to the task descriptor. In some implementations, the processor configuration requirements of a task descriptor can include a Level 1 cache memory configuration (L1 cache memory configuration). For example, the processor configuration requirements can specify how much tagged (or untagged) L1 memory is required to perform the task corresponding to the task descriptor.

[0015] Resource requests can include one or more processor resources required for execution by the task corresponding to the task descriptor. In some embodiments, a task descriptor's resource requests can include one or more resources corresponding to a thread block size of the task corresponding to the task descriptor, a number of registers required by the task, or a number of barriers required by the task.

[0016] The task descriptor program / shader can contain one or more addresses that the processor can access during task descriptor execution. For example, the task descriptor program / shader can contain the first address of a range of instructions that the processor can execute during task descriptor execution.

[0017] The parallel processing device 106 can include a front end 108, a memory 110, a scheduler 114, a scheduling table 116 for efficient distribution, and one or more processing units 118. The processing units 118 can include one or more processing units for performing parallel processing (e.g., parallel processing units). The front end 108 can interface with the CPU 102 via the GPU driver 104. The front end 108 can receive one or more task descriptors from the CPU 102, which are to be executed by the parallel processing device 106 (e.g., by the processing units 118), and store them in the memory 110.

[0018] For example, memory 110 can contain a task descriptor 112a and a task descriptor 112b. Task descriptors 112a and 112b can each contain processor configuration requirements, resource requirements, and / or a program / shader. Task descriptor 112a can have the same (or different) processor configuration requirements, resource requirements, and / or a program / shader as task descriptor 112b.

[0019] Scheduler 114 can read task descriptors from memory 110 and load them into scheduling table 116 for efficient distribution. Scheduling table 116 for efficient distribution may have limited memory (e.g., due to hardware limitations). Task descriptors can be stored in scheduling table 116 for efficient distribution after being loaded from memory 110 and during execution by processing units 118.

[0020] The efficient distribution scheduling table 116 can contain one or more task descriptors (e.g., a task descriptor 112c, a task descriptor 112d, etc.). The efficient distribution scheduling table 116 can initiate task descriptors for execution by one or more processing units 118. During the execution of a task descriptor, its state can be updated in the efficient distribution scheduling table 116. For example, when a task descriptor is executed, a task state (e.g., a raster state, a shader state, etc.) of the task descriptor can be updated.

[0021] Providing task descriptors from scheduling table 116 for efficient distribution to processing units 118 can be done quickly and with minimal overhead (e.g., without accessing memory 110). To maintain high utilization of processing units 118 and thus increase their overall throughput, task descriptors for execution can be selected from scheduling table 116 based on one or more properties of the task descriptors and the current state of one or more processing units 118.

[0022] As discussed above, the processor configuration requirements of a task descriptor include properties of a processor (e.g., a processing unit) that can be set before the task descriptor can be executed. Before a task descriptor with an initial set of processor configuration requirements can be executed, the processor can be configured to meet those requirements. Then the task descriptor can be executed. Other task descriptors with the same processor configuration requirements can be executed concurrently on the processor.

[0023] To run a different task descriptor with a second set of processor configuration requirements, the processor can be idled (e.g., all running processes on the processor can be terminated so that the processor enters a sleep state) and reconfigured to meet the second set of processor configuration requirements.

[0024] To avoid the overhead of emptying and reconfiguring the processor before executing a subsequent task descriptor, task descriptors can be selected in scheduling table 116 for efficient distribution that match the processor's current configuration. If a task descriptor has processor configuration requirements that match a processor's current configuration, that task descriptor can be selected next for execution, even if one or more task descriptors were loaded into scheduling table 116 before it.

[0025] If there are multiple task descriptors with the same processor configuration requirements, the task descriptor's resource requirements can be an additional criterion used to select the next task to be made available to processing units 118 for execution. For example, task descriptors with resource requirements that match those of a currently running task can be selected as the next task to execute, even if one or more task descriptors prior to that task descriptor were loaded into scheduling table 116 for efficient distribution.

[0026] Before a task descriptor can be distributed to a processor for execution, the processor must be queried to determine if it has sufficient resources to run the task descriptor. The processor can respond to the query with a number of available resources. If sufficient resources are available to run a task descriptor, the task descriptor can be provided to the processor for execution. In some cases, the task descriptor does not use all of the processor's available resources during execution. For example, the processor might respond to the query by indicating that 8 resources (such as registers, barriers, thread block slots, etc.) are available. The task descriptor might only use 3 of these resources, leaving 5 additional resources available.If another task descriptor in scheduling table 116 has the same resource requirements as the currently executing task (and the same processor configuration requirements, as discussed above), this next task descriptor can be allocated to the processor for execution without requiring an additional query, since it is known that 5 resources are available and the next task descriptor requires fewer than that. Selecting the next task descriptors based on their common resource requirements eliminates the round-trip time associated with querying a processor, resulting in lower task descriptor allocation latencies and higher processor throughput.

[0027] In some embodiments, a next task descriptor can be selected based on the shared use of a program / shader with a currently executing task. For example, if a processor is executing a first task descriptor with a first program / shader, and the efficient distribution scheduling table 116 includes a second task descriptor with a second program / shader and a third task descriptor with a program / shader identical to the first, the third task descriptor can be identified as the next task descriptor to be provided to the executing processor.Since the first task descriptor and the third task descriptor share a program / shader, the processor's instruction cache is more likely to encounter cache hits during the execution of the third task descriptor, resulting in reduced task descriptor execution time and higher processor throughput.

[0028] In some embodiments, a processing device (e.g., the parallel processing device 106) can have more than one processor (e.g., the processing units 118). Each processor can have its own configuration and resource availability. In some embodiments, tasks from the efficient distribution scheduling table 116 with a first processor configuration can be executed by a first processor, and tasks from the efficient distribution scheduling table 116 with a second processor configuration can be executed by a second processor. In some embodiments, a first processor and a second processor can have the same processor configuration, and each of the processors can be queried to determine its current resource availability.A task descriptor can be selected for execution from scheduling table 116 for efficient distribution based on the current resource availability of one of the processors. If none of the processors in the processing device has a processor configuration that meets the processor configuration requirements of the task descriptors in scheduling table 116 for efficient distribution, one of the processors in the processing device can be emptied and reconfigured while the other processors continue executing the task descriptors.

[0029] In some embodiments, task descriptors are loaded from memory 110 as soon as they have completed their execution and been removed from the planning table 116 for efficient distribution. In some embodiments, task descriptors are loaded from memory 110 when the planning table 116 for efficient distribution reaches a occupancy threshold (e.g., 90% full, 80% full, 50% full, 10% full, etc.). In some embodiments, task descriptors are loaded from memory 110 when the planning table 116 for efficient distribution is empty.

[0030] Fig. Figure 2A is a block diagram of an exemplary planning table 202 for efficient distribution according to at least one embodiment. The planning table 202 for efficient distribution can include one or more task descriptors (e.g., a task descriptor 204a, a task descriptor 204b, a task descriptor 204c, a task descriptor 204d, a task descriptor 204e, a task descriptor 204f, etc.). Each task descriptor can include processor configuration requirements (e.g., 206a, 206b, 206c, 206d, 206e, 206f, etc.), resource requirements (e.g., 208a, 208b, 208c, 208d, 208e, 208f, etc.), and / or a program / shader (e.g., 210a, 210b, 210c, 210d, 210e, 210f, etc.). The task descriptors in the efficient distribution scheduling table 202 are labeled Task 1 through Task 6. Fig. In 2A, the task descriptor identifiers are in ascending order from top to bottom, which can indicate that task 1 was first loaded from memory into the planning table 202 for efficient distribution, then task 2, then task 3, and so on.

[0031] Elements of the same type in Fig. 2A, which have the same hatching (e.g., diagonal lines, dashed lines, dots, checkerboard patterns, etc.), can have the same values. For example, Fig. 2A Represent task descriptors with two different processor configuration requirements. A first group of task descriptors (e.g., 204a, 204c, 204e, and 204f) may have a first set of processor configuration requirements represented by diagonal hatching. A second group of task descriptors (e.g., 204b and 204d) may have a second set of processor configuration requirements represented by hatched hatching.

[0032] Fig. 2A can also represent task descriptors with four different resource requirements. A first group of task descriptors (e.g., 204a, 204c, and 204e) can have a first set of resource requirements represented by diagonal hatching. A second group of task descriptors (e.g., 204b) can have a second set of resource requirements represented by hatched crosshatching. A third group of task descriptors (e.g., 204d) can have a third set of resource requirements represented by dot hatching. A fourth group of task descriptors (e.g., 204f) can have a fourth set of resource requirements represented by a checkerboard pattern.

[0033] Fig. 2A can also represent task descriptors with three different programs / shaders. A first group of task descriptors (e.g., 204a, 204c, and 204e) can have a first program / shader represented by diagonal hatching. A second group of task descriptors (e.g., 204b) can have a second program / shader represented by hatched patterns. A third group of task descriptors (e.g., 204d and 204f) can have a third program / shader represented by dot hatching.

[0034] In some embodiments, the processor configuration requirements and / or resource requirements are stored as one or more integer and / or bitstream representations. For example, a first set of processor configuration requirements may be represented by a first integer and / or bitstream representation, while a second set of processor configuration requirements may be represented by a second integer and / or bitstream representation. The resource requirements may be similar. In some embodiments, the resource requirements include a variety of integer and / or bitstream representations. For example, the resource requirements may include a first integer value representing a number of registers required for task execution and a second integer value representing a number of barriers required for task execution.

[0035] Fig. Figure 2B is a block diagram of an exemplary planning table 202 for efficient distribution according to at least one embodiment. Fig. 2B can use the same task descriptors as Fig. 2A. Fig. In 2B, however, the task descriptors were rearranged to show an efficient top-to-bottom execution order. In some embodiments, the first task descriptor to be executed is the first task descriptor loaded into memory. In other embodiments, the first task descriptor to be executed can be selected randomly (or by another method) from the task descriptors in scheduling table 202 for efficient distribution.

[0036] For example, a first task descriptor to be executed by a processor could be the first task descriptor in the efficient distribution scheduling table 202 (e.g., task descriptor 204a). Before execution, the processor can be flushed and reconfigured to meet the processor configuration requirements of the first task descriptor (e.g., processor configuration requirements 206a). The processor can also be queried to ensure that it has sufficient available resources to meet the resource requirements of the first task descriptor (e.g., resource requirements 208a).

[0037] After task descriptor 204a has been provided to the processor for execution, the next task descriptor to be executed can be determined. Task descriptors with the same processor configuration requirements as task descriptor 204a can be identified: task descriptor 204c, task descriptor 204e, and task descriptor 204f. From these task descriptors, those with the same resource requirements as task descriptor 204a can be identified: task descriptor 204c and task descriptor 204e. From these task descriptors, those with the same program / shader as task descriptor 204a can be identified: task descriptor 204c and task descriptor 204e. The next task descriptor can be selected from this group (e.g., randomly, based on the order in which they were loaded from memory, etc.).In this example, the task descriptor 204c can be provided to the processor for execution next.

[0038] Since task descriptor 204a and task descriptor 204c have the same processor configuration requirements, they can be executed concurrently on the processor. Because task descriptor 204a and task descriptor 204c have the same resource requirements, the amount of resources available to execute task descriptor 204c can be determined without querying the processor. If the determined amount of resources available to execute task descriptor 204c is sufficient, task descriptor 204c can be provided to the processor for execution without querying the processor.If the identified amount of resources available to execute task descriptor 204c is insufficient to execute task descriptor 204c, the processor can be queried to obtain its current resource availability, and then task descriptor 204c can be executed as soon as resources are available.

[0039] After task descriptor 204c has been provided to the processor for execution, the next task descriptor to be executed can be determined. Task descriptors with the same processor configuration requirements as task descriptor 204c can be identified: task descriptor 204e and task descriptor 204f. From these, task descriptors with the same resource requirements as task descriptor 204c can be identified: task descriptor 204e. Since task descriptor 204e is the only task descriptor in this group, it can be selected as the next task descriptor to be executed, regardless of its program / shader.

[0040] Since task descriptor 204c and task descriptor 204e have the same processor configuration requirements, the task descriptors can be run concurrently on the processor as long as the processor has available resources.

[0041] Once task descriptor 204e has been provided to the processor for execution, the next task descriptor to be executed can be determined. Task descriptors with the same processor configuration requirements as task descriptor 204e can be identified: task descriptor 204f. Since task descriptor 204f is the only task descriptor in this group, it can be selected as the next task descriptor to be executed, regardless of its resource requirements or program / shader.

[0042] Since task descriptor 204e and task descriptor 204f do not have the same resource requirements, the processor can be queried to determine its resource availability before task descriptor 204f is deployed for execution. If the processor has sufficient available resources, task descriptor 204f can be deployed for execution.

[0043] After task descriptor 204f has been provided to the processor for execution, the next task descriptor to be executed can be determined. Since no other task descriptors have processor configuration requirements matching task descriptor 204f, task descriptors with different processor configuration requirements can be determined: task descriptors 204d and 204b. In some embodiments, task descriptors with the most frequent processor configuration requirements in scheduling table 202 are given priority for efficient distribution.For example, if 5 task descriptors remain in scheduling table 202 for efficient distribution, and 3 of them have a first set of processor configuration requirements and 2 of them have a second set of processor configuration requirements, one task descriptor from the group of 3 can be selected as the next task descriptor to be made available for execution. In some embodiments, the next task descriptor can be selected randomly.

[0044] In this example, task descriptor 204d is selected as the next task descriptor to be made available for execution. Because task descriptor 204d has different processor configuration requirements than task descriptor 204f, the processor can be idled (for example, wait until task descriptor 204f has finished executing so that the processor is idle) and reconfigured from the previous processor configuration (for example, based on 206a, 206c, 206e, and 206f) to the new processor configuration (for example, based on 206d). The processor can then be queried to determine its available resources. After the processor has been idled, reconfigured, and queried, task descriptor 204d can be made available to the processor for execution.

[0045] After task descriptor 204d has been provided to the processor for execution, the next task descriptor to be executed can be determined using a similar procedure as before. In this example, task descriptor 204b can be selected as the next task descriptor to be provided to the processor because it is the last task descriptor in the scheduling table 202 for efficient distribution.

[0046] Since task descriptor 204b has the same processor configuration requirements as task descriptor 204d, it can be executed concurrently with task descriptor 204d as long as resources are available. However, since the resource requirements of task descriptor 204b differ from those of task descriptor 204d, the processor can be queried to determine resource availability before task descriptor 204b is made available for execution.

[0047] Fig. Figure 3 is a flowchart of an exemplary method 300 for an efficient distribution of tasks for execution by a processor according to at least one embodiment.

[0048] Method 300 can be performed using one or more processing units (e.g., CPUs, GPUs, accelerators, parallel processing units, physical processing units (PPUs), data processing units (DPUs), etc.) which may include (or communicate with) one or more storage devices. In at least one embodiment, Method 300 can be performed using one or more processing devices. In at least one embodiment, Method 300 can be performed using processing units of System 100. Fig. 1. In at least one embodiment, the method 300 can be carried out by the parallel processing device 106 of Fig. 1. In at least one embodiment, processing units executing Method 300 can execute instructions stored on a non-volatile, computer-readable storage medium. In at least one embodiment, Method 300 can be executed using multiple processing threads (e.g., CPU threads and / or GPU threads), with individual threads executing one or more individual functions, routines, subroutines, or operations of the Method. In at least one embodiment, processing threads implementing Method 300 can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, processing threads implementing Method 300 can be executed asynchronously. Various operations of Method 300 can be performed in comparison to those described in Fig. The sequence shown in 3 can be performed in a different order. Some operations of method 300 can be performed simultaneously with other operations. In at least one embodiment, one or more operations that are shown in Fig. The steps shown in point 3 are not always carried out.

[0049] In Block 302, processing units executing Procedure 300 can perform a first task with associated processor configuration requirements. In some embodiments, the first task's processor configuration requirements may include a Level 1 cache memory configuration (L1 cache memory configuration).

[0050] In block 304, the processing units can receive a second and a third task to be executed by the processing unit. The second and third tasks can each have associated processor configuration requirements.

[0051] In block 306, the processing units can determine a first next task to be executed by the processing unit based on a comparison of the processing unit's current processor configuration with the processor configuration requirements of the second and third tasks. The processing unit's current processor configuration can be based on the processor configuration requirements associated with the first task (e.g., the task currently being executed by the processing unit).

[0052] In some embodiments, the second task and the third task each have these associated resource requirements. In some embodiments, the resource requirements of the second task include at least one of the thread block size of the second task, a number of registers required by the second task, or a number of barriers required by the second task.

[0053] Determining the next task to be executed by the processing unit can also be based on the current resource availability of the processing unit and the resource requirements of the second and third tasks. For example, the current resource availability of the processing unit may have been determined before the first task was executed. If the first task does not use all of the processing unit's available resources, the next task can be made available to the processing unit for execution without further querying, as long as the resource requirements of the first task are the same as those of the next task (e.g., the second or third task).

[0054] In some embodiments, the second and third tasks each have an associated program / shader. Determining the next task to be executed by the processing unit can also be based on the current program / shader of the processing unit and the programs / shaders of the second and third tasks. For example, if the third task and the first task (e.g., the task currently executing on the processing unit) use the same program / shader, the third task can be selected as the next task over the second task to improve task execution performance (e.g., by increasing instruction cache hits).

[0055] At block 308, the processing units can provide the first next task of the processing unit for execution.

[0056] In some embodiments, the second task is received before the third, and the third task is the next first task. Thus, the tasks can be made available to the processing unit for execution in an order that differs from the order in which the tasks were received.

[0057] In some embodiments, the processing units executing method 300 can receive a fourth task at block 310, which the processing unit is to execute. The fourth task may have associated processor configuration requirements. At block 312, at least under the second and fourth tasks, the processing units can determine a second next task to be executed by the processing unit based on a comparison of the processor configuration requirements of the third task (e.g., the task currently being executed by the processing unit) with the processor configuration requirements of the second and fourth tasks. At block 314, the processing units can provide the second next task to the processing unit for execution.

[0058] Fig. Figure 4 is a block diagram illustrating an exemplary computer system according to at least one embodiment of the present disclosure. In some embodiments, computer system 400 can replace system 100. Fig. 1. The Computer System 400 can operate as a server or endpoint in an endpoint-server network environment, or as a peer device in a peer-to-peer (or distributed) network environment. The machine can be a television, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a web application, a server, a network router, a switch or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) specifying actions to be performed by that machine. Furthermore, although only a single machine is illustrated, the term "machine" is also to be understood as including any collection of machines that, individually or collectively, execute a set (or multiple sets) of instructions to perform one or more of the procedures discussed herein.

[0059] The exemplary computer system 400 includes a processing device (a processor) 402, a main memory 404 (e.g., a read-only memory (ROM), a flash memory, a dynamic random-access memory (DRAM), such as a synchronous DRAM (SDRAM), a dual data rate (DDR-SDRAM), or a DRAM (RDRAM), etc.), a static memory 406 (e.g., a flash memory, a static random-access memory (SRAM), etc.), and a data storage device 416, which communicate with each other via a bus 428.

[0060] The processor (processing device) 402 represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like, and may include processing logic 422. Specifically, the processor 402 may be a complex instruction set (CISC) microprocessor, a reduced instruction set (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. The processor 402 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. The processor 402 is configured to execute instructions 426 (e.g.,(to generate threat indicator alerts) to perform the operations discussed here.

[0061] The computer system 400 may further include a network interface device 408. The computer system 400 may also include a video display unit 410 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an input device 412 (e.g., a keyboard and an alphanumeric keypad, a motion-sensing input device, a touchscreen), a cursor control device 414 (e.g., a mouse), and a signal generation device 418 (e.g., a loudspeaker). In some embodiments, the computer system 400 may not include a video display unit 410, an input device 412, and / or a cursor control device 414 (e.g., in a headless configuration).

[0062] The data storage device 416 may include a non-volatile, machine-readable storage medium 424 (also called a computer-readable storage medium) on which one or more sets of instructions 426 (e.g., for the efficient distribution of tasks for execution by a processor) are stored, embodying one or more of the procedures or functions described herein. The instructions 426 may also reside wholly or at least partially within the main memory 404 and / or within the processor 402 during execution by the computer system 400, the main memory 404 and the processor 402 also constituting machine-readable storage media. Furthermore, the instructions may be transmitted or received over a network 420 via the network interface device 408.

[0063] In one implementation, the instructions 426 include instructions for the efficient distribution of tasks for execution by a processor. Although the computer-readable storage medium 424 (machine-readable storage medium) is shown in an exemplary implementation as a single medium, the terms "computer-readable storage medium" and "machine-readable storage medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store the one or more sets of instructions.The terms “computer-readable storage medium” and “machine-readable storage medium” shall also be understood to include any medium capable of storing, encoding, or carrying a set of instructions for execution by the machine, and which causes the machine to execute one or more of the methods of the present disclosure. Accordingly, the terms “computer-readable storage medium” and “machine-readable storage medium” shall be understood to include, but are not limited to, solid-state storage media, optical media, and magnetic media. INFERENCE AND TRAINING LOGIC

[0064] Fig. Figure 5A shows an inference and / or training logic 515 that is used to perform inference and / or training operations associated with one or more embodiments.

[0065] In at least one embodiment, the inference and / or training logic 515 may, without limitation, include a code and / or data store 501 to store forward and / or output weights and / or input / output data and / or other parameters to configure neurons or layers of a neural network that is trained and / or used for inference in aspects of one or more embodiments.In at least one embodiment, the training logic 515 may include graph code or other software (or be coupled to a code and / or data memory 501 that stores the graph code or other software) to store graph code or other software that controls the timing and / or order in which information about weights and / or other parameters is loaded to configure processing units, including logic units, integer and / or floating-point units (collectively referred to as arithmetic logic units (ALUs) or simple circuits). In at least one embodiment, code, such as graph code, based on a neural network architecture to which the code corresponds, loads weights or other parameter information into processor ALUs.In at least one embodiment, the code and / or data storage 501 stores weighting parameters and / or input / output data of each layer of a neural network that is trained during the forward propagation of input / output data and / or weighting parameters during training and / or inference using aspects of one or more embodiments, or used in conjunction with one or more embodiments. In at least one embodiment, each part of the code and / or data storage 501 may be encompassed by another on-chip or off-chip data storage, including the L1, L2, or L3 cache of a processor or system memory.

[0066] In at least one embodiment, each part of the code and / or data memory 501 can be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 501 can be a cache memory, a dynamic randomly addressable memory (“DRAM”), a static randomly addressable memory (“SRAM”), a non-volatile memory (e.g., flash memory), or another type of memory.In at least one embodiment, the decision as to whether the code and / or data storage 501 is internal or external to a processor, or comprises DRAM, SRAM, Flash or another type of memory, may depend on whether memory is available on or off the chip, on the latency requirements of the training and / or inference functions being performed, on the size of the data stacks used in inferencing and / or training a neural network, or on a combination of these factors.

[0067] In at least one embodiment, the inference and / or training logic 515 may, without limitation, comprise a code and / or data store 505 to store backward and / or output weighting and / or input / output data corresponding to neurons or layers of a neural network that is trained and / or used for inference in aspects of one or more embodiments. In at least one embodiment, the code and / or data store 505 stores weighting parameters and / or input / output data of each layer of a neural network that is trained or used in conjunction with aspects of one or more embodiments during backward propagation of input / output data and / or weighting parameters during training and / or inference.In at least one embodiment, the training logic 515 may include a graph code or other software (or be coupled with a code and / or data memory 505 that stores the graph code or other software) to control a timing and / or sequence in which the information about weights and / or other parameters is loaded to configure processing units including logic units, integer and / or floating-point units (collectively referred to as arithmetic logic units (ALUs)).

[0068] In at least one embodiment, a code, such as a graph code, based on a neural network architecture to which the code corresponds, causes weight or other parameter information to be loaded into processor ALUs. In at least one embodiment, any part of the code and / or data memory 505 may include another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor. In at least one embodiment, any part of the code and / or data memory 505 may be located internally or externally in one or more processors or other hardware logic devices or circuits. In at least one embodiment, the code and / or data memory 505 may be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other memory.In at least one embodiment, the choice of whether the code and / or data storage 505 is internal or external to a processor, or whether it comprises, for example, DRAM, SRAM, Flash or another type of memory, may depend on the available memory on or off the chip, on the latency requirements of the training and / or inference functions being performed, on the batch size of the data used in inferencing and / or training a neural network, or on a combination of these factors.

[0069] In at least one embodiment, the code and / or data memory 501 and the code and / or data memory 505 can be separate memory structures. In at least one embodiment, the code and / or data memory 501 and the code and / or data memory 505 can be a combined memory structure. In at least one embodiment, the code and / or data memory 501 and the code and / or data memory 505 can be partially combined and partially separate. In at least one embodiment, each part of the code and / or data memory 501 and the code and / or data memory 505 can be encompassed by another on-chip or off-chip data memory, including the L1, L2, or L3 cache of a processor or system memory.

[0070] In at least one embodiment, the inference and / or training logic 515 may, without limitation, comprise one or more arithmetic logic unit(s) (“ALU(s)”) 510, including integer and / or floating-point units, to perform logical and / or mathematical operations that can be performed at least partially on a training and / or inference code (e.g., graph code), the result of which can generate activations (e.g., output values ​​of layers or neurons within a neural network) stored in an activation memory 520, which are functions of input / output and / or weighting parameter data stored in code and / or data memory 501 and / or code and / or data memory 505.In at least one embodiment, activations stored in an activation memory 520 are generated according to linear algebraic and / or matrix-based mathematics, which are executed by ALU(s) 510 in response to execution instructions or other code, wherein weight values ​​stored in the code and / or data memory 505 and / or the code and / or data memory 501 are used as operands together with other values, such as default values, gradient information, pulse values ​​or other parameters or hyperparameters, any or all of which may be stored in the code and / or data memory 505 or the code and / or data memory 501 or in another memory on or off the chip.

[0071] In at least one embodiment, ALU(s) 510 are contained within one or more processors or other hardware logic devices or circuits, while in another embodiment, ALU(s) 510 may be external to a processor or other hardware logic device or circuit that uses them (e.g., a coprocessor). In at least one embodiment, the ALU(s) 510 may be contained within the execution units of a processor or otherwise in a group of ALUs that the execution units of a processor can access, either within the same processor or distributed across different processors of different types (e.g., central processing units, graphics processing units, fixed functional units, etc.).In at least one embodiment, the code and / or data memory 501, the code and / or data memory 505, and the activation memory 520 can share a processor or other hardware logic device or circuit, while in another embodiment they can be located in different processors or other hardware logic devices or circuits, or in a combination of identical and different processors or other hardware logic devices or circuits. In at least one embodiment, any part of the activation memory 520 can be contained in another on-chip or off-chip data memory, including the L1, L2, or L3 cache or system memory of a processor.Furthermore, the code for inference and / or training can be stored together with other code that a processor or other hardware logic or circuitry can access and that is fetched and / or processed using the fetch, decode, scheduling, execution, elimination, and / or other logic circuitry of a processor.

[0072] In at least one embodiment, the activation memory 520 can be a cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or another type of memory. In at least one embodiment, the activation memory 520 can be located wholly or partially inside or outside one or more processors or other logic circuits. In at least one embodiment, the decision as to whether the activation memory 520 is located, for example, inside or outside a processor, or whether it comprises DRAM, SRAM, flash memory, or another type of memory, can depend on the available on-chip or off-chip memory, the latency requirements of the training and / or inference functions being performed, the batch size of the data used in inferencing and / or training a neural network, or a combination of these factors.

[0073] In at least one embodiment, the in Fig. The inference and / or training logic 515 shown in Figure 5A can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as Google’s TensorFlow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or an Intel Corp. Nervana® processor (e.g., “Lake Crest”). In at least one embodiment, the logic shown in Figure 5A can be used in conjunction with an application-specific integrated circuit (“ASIC”), such as Google’s TensorFlow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or an Intel Corp. Nervana® Processor (e.g., “Lake Crest”). Fig. 5A The inference and / or training logic 515 shown can be used in conjunction with hardware of the central processing unit (“CPU”), the graphics processing unit (“GPU”) or other hardware, such as field-programmable gate arrays (“FPGAs”).

[0074] Fig. Figure 5B shows the inference and / or training logic 515 according to at least one embodiment. In at least one embodiment, the inference and / or training logic 515 can, without limitation, comprise hardware logic in which computing resources are dedicated or otherwise used exclusively in conjunction with weight values ​​or other information corresponding to one or more layers of neurons within a neural network. In at least one embodiment, the Fig. The inference and / or training logic 515 shown in Figure 5B can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's TensorFlow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or an Intel Corp. Nervana® processor (e.g., "Lake Crest"). In at least one embodiment, the logic shown in Figure 5B can be used in conjunction with an application-specific integrated circuit (ASIC), such as Google's TensorFlow® Processing Unit, a Graphcore™ Inference Processing Unit (IPU), or an Intel Corp. Nervana® Processor (e.g., "Lake Crest"). Fig. The inference and / or training logic 515 shown in Figure 5B is used in conjunction with central processing unit (CPU) hardware, graphics processing unit (GPU) hardware, or other hardware, such as field-programmable gate arrays (FPGAs). In at least one embodiment, the inference and / or training logic 515 comprises, without limitation, a code and / or data memory 501 and a code and / or data memory 505, which can be used to store code (e.g., graph code), weight values, and / or other information, including default values, gradient information, pulse values, and / or other parameter or hyperparameter information. In at least one embodiment, the inference and / or training logic 515 comprises, without limitation, a code and / or data memory 501 and a code and / or data memory 505, which can be used to store code (e.g., graph code), weight values, and / or other information, including default values, gradient information, pulse values, and / or other parameter or hyperparameter information. Fig. As shown in Figure 5B, each code and / or data store 501 and each code and / or data store 505 is associated with a dedicated computing resource, such as computing hardware 502 and computing hardware 506, respectively. In at least one embodiment, each of the computing hardware 502 and computing hardware 506 comprises one or more ALUs that execute mathematical functions, such as linear algebraic functions, only on information stored in the code and / or data store 501 and in the code and / or data store 505, respectively, with the result being stored in the activation memory 520.

[0075] In at least one embodiment, each of the code and / or data stores 501 and 505 and the corresponding computing hardware 502 and 506, respectively, corresponds to different layers of a neural network, such that the resulting activation from a "memory / computing pair 501 / 502" consisting of code and / or data store 501 and computing hardware 502 is provided as input for a next "memory / computing pair 505 / 506" consisting of code and / or data store 505 and computing hardware 506, in order to reflect a conceptual organization of a neural network. In at least one embodiment, each of the memory / computing pairs 501 / 502 and 505 / 506 can correspond to more than one layer of the neural network. In at least one embodiment, additional memory / computing pairs (not shown) can be included sequentially or in parallel with the memory / computing pairs 501 / 502 and 505 / 506 in the inference and / or training logic 515. TRAINING AND USE OF A NEURAL NETWORK

[0076] Fig. Figure 6 shows, according to at least one embodiment, the training and use of a deep neural network. In at least one embodiment, the untrained neural network 606 is trained using a training dataset 602. In at least one embodiment, the training framework 604 is a PyTorch framework, while in other embodiments, the training framework 604 is a TensorFlow, Boost, Caffe, Microsoft Cognitive Toolkit / CNTK, MXNet, Chainer, Keras, Deeplearning4j, or other training framework. In at least one embodiment, the training framework 604 trains an untrained neural network 606 and enables it to be trained using the processing resources described herein to generate a trained neural network 608.In at least one embodiment, the weights can be selected randomly or by pre-training using a deep belief network. In at least one embodiment, the training can be supervised, partially supervised, or unsupervised.

[0077] In at least one embodiment, an untrained neural network 606 is trained using supervised learning, wherein the training dataset 602 comprises an input paired with a desired output for that input, or wherein the training dataset 602 comprises an input paired with a known output, and an output of the neural network 606 is manually evaluated. In at least one embodiment, the untrained neural network 606 is trained in a supervised manner and processes inputs from the training dataset 602 and compares the resulting outputs with a set of expected or desired outputs. In at least one embodiment, the errors are then propagated back by the untrained neural network 606. In at least one embodiment, the training framework 604 adjusts the weights that control the untrained neural network 606.In at least one embodiment, the training framework 604 includes tools to monitor how well the untrained neural network 606 converges to a model, such as the trained neural network 608, that is capable of generating correct answers based on input data, such as a new dataset 612, as shown in the result 614. In at least one embodiment, the training framework 604 repeatedly trains the untrained neural network 606 while adjusting the weights to refine an output of the untrained neural network 606 using a loss function and an adjustment algorithm, such as stochastic gradient descent. In at least one embodiment, the training framework 604 trains the untrained neural network 606 until the untrained neural network 606 achieves a desired accuracy.In at least one embodiment, the trained neural network 608 can then be used to implement any number of machine learning operations.

[0078] In at least one embodiment, the untrained neural network 606 is trained using unsupervised learning, wherein the untrained neural network 606 attempts to train itself using unlabeled data. In at least one embodiment, the unsupervised training dataset 602 comprises input data without associated execution data or "ground truth" data. In at least one embodiment, the untrained neural network 606 can learn groupings within the training dataset 602 and determine how individual inputs relate to the untrained dataset 602. In at least one embodiment, unsupervised training can be used to generate a self-organizing map in a trained neural network 608, which is capable of performing operations useful in reducing the dimensionality of a new dataset 612.In at least one embodiment, unsupervised training can also be used to perform anomaly detection, which makes it possible to identify data points in a new data set 612 that deviate from normal patterns of the new data set 612.

[0079] In at least one embodiment, semi-supervised learning can be used, which is a method in which the training dataset 602 comprises a mixture of labeled and unlabeled data. In at least one embodiment, the training framework 604 can be used to perform incremental learning, for example, by transferred learning methods. In at least one embodiment, the incremental learning enables a trained neural network 608 to adapt to a new dataset 612 without forgetting the knowledge imparted to the trained neural network during the initial training.

[0080] With reference to Fig. 7 is Fig. 7 An exemplary data flow diagram for a method 700 for generating and deploying an image processing and inference pipeline according to at least one embodiment. In at least one embodiment, the method 700 can be used to perform game name recognition analysis and inference based on user feedback data in one or more facilities 702, for example, a data center.

[0081] In at least one embodiment, the method 700 can be performed within a training system 704 and / or a deployment system 706. In at least one embodiment, the training system 704 can be used to perform the training, deployment, and implementation of machine learning models (e.g., neural networks, object detection algorithms, machine vision algorithms, etc.) for use in the deployment system 706. In at least one embodiment, the deployment system 706 can be configured to offload processing and computing resources to a distributed computing environment to reduce the infrastructure requirements in the facility 702. In at least one embodiment, the deployment system 706 can provide a streamlined platform for selecting, customizing, and implementing virtual tools for use with computing devices in the facility 702.In at least one embodiment, virtual instruments can include software-defined applications for performing one or more processing operations on feedback data. In at least one embodiment, one or more applications in a pipeline can use or call services (e.g., inference, visualization, computation, AI, etc.) of the Deployment System 706 during application execution.

[0082] In at least one embodiment, some of the applications used in advanced processing and inference pipelines can use machine learning models or other AI to perform one or more processing steps. In at least one embodiment, machine learning models can be trained in the facility 702 using the feedback data 708 (such as imaging data) stored in the facility 702, and they can be trained using feedback data 708 from another facility or facilities, or a combination thereof. In at least one embodiment, the training system 704 can be used to provide applications, services, and / or other resources for generating functional, deployable machine learning models for the deployment system 706.

[0083] In at least one embodiment, a model registration database 724 can be supported by object storage that can support versioning and object metadata. In at least one embodiment, access to the object storage can be provided, for example, by a cloud storage system (e.g., a cloud 826). Fig. 8) A compatible application programming interface (API) from within a cloud platform. In at least one embodiment, machine learning models within the model registration database 724 can be uploaded, listed, modified, or deleted by developers or partners of a system that interacts with an API. In at least one embodiment, an API can provide access to procedures that allow users with appropriate credentials to associate models with applications so that models can be executed as part of the execution of containerized instantiations of applications.

[0084] In at least one embodiment, (a) training pipeline(s) 804 ( Fig. 8) include a scenario in which the facility 702 trains its own machine learning model or has an existing machine learning model that needs to be optimized or updated. In at least one embodiment, feedback data 708 can be received from various channels, such as forums, web forms, or the like. In at least one embodiment, once the feedback data 708 is received, the AI-powered annotation 710 can be used to assist in generating annotations that correspond to the feedback data 708, which is to be used as ground-truth data for a machine learning model. In at least one embodiment, the AI-powered annotation 710 can include one or more machine learning models (e.g., convolutional neural networks (CNNs)) that can be trained to generate annotations that correspond to certain types of feedback data 708 (e.g.,of certain devices) and / or certain types of anomalies in the feedback data 708 correspond. In at least one embodiment, the AI-assisted annotations 710 can then be used directly or adjusted or fine-tuned using an annotation tool to generate ground-truth data. In at least one embodiment, data 712, characterized in some examples, can be used as ground-truth data for training a machine learning model. In at least one embodiment, the AI-assisted annotations 710, characterized data 712, or a combination thereof can be used as ground-truth data for training a machine learning model, e.g., via model training 714 in . Fig. 7 and / or Fig. 8. In at least one embodiment, a trained machine learning model can be designated as output model 716 and used by the deployment system 706 as described herein.

[0085] In at least one embodiment, the training pipeline(s) 804 ( Fig. 8) include a scenario in which the facility 702 requires a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 706, but the facility 702 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, an existing machine learning model can be selected from the model registration database 724. In at least one embodiment, the model registration database 724 can contain machine learning models that are trained to perform a variety of different inference tasks on imaging data.In at least one embodiment, the machine learning models in the model registration database 724 may have been trained on imaging data from facilities other than the facility 702 (e.g., facilities located at a different location). In at least one embodiment, the machine learning models may have been trained on imaging data from one location, two locations, or any number of locations. In at least one embodiment, when training on imaging data, which may be a form of feedback data 708, the training may take place from a specific location at that location or at least in a manner that protects the confidentiality of the imaging data or restricts the transmission of the imaging data outside the premises (e.g., to comply with HIPAA regulations, data protection regulations, etc.).In at least one embodiment, a machine learning model, once trained—or partially trained—at one location, can be added to the model registration database 724. In at least one embodiment, a machine learning model can then be retrained or updated at any number of other locations, and a retrained or updated model can be made available in the model registration database 724. In at least one embodiment, a machine learning model can then be selected from the model registration database 724—and designated as output model(s) 716—and used in the deployment system 706 to perform one or more processing tasks for one or more applications of a deployment system.

[0086] In at least one embodiment, the training pipeline(s) 804 ( Fig. 8) in a scenario where the facility 702 requires a machine learning model for use in performing one or more processing tasks for one or more applications in the deployment system 706, but the facility 702 may not currently have such a machine learning model (or may not have a model that is optimized, efficient, or effective for such purposes). In at least one embodiment, a machine learning model selected from the model registration database 724 might not be fine-tuned or optimized for the feedback data 708 generated in the facility 702 due to differences in populations, genetic variations, the robustness of the training data used to train a machine learning model, the diversity of anomalies in the training data, and / or other problems with the training data.In at least one embodiment, the AI-assisted annotation 710 can be used to assist in generating annotations corresponding to the feedback data 708, which is to be used as ground-truth data for retraining or updating a machine learning model. In at least one embodiment, the labeled data 712 can be used as ground-truth data for training a machine learning model. In at least one embodiment, the retraining or updating of a machine learning model can be referred to as model training 714. In at least one embodiment, the model training 714 can include data—e.g., AI-assisted annotations 710, labeled data 712, or a combination thereof—that can be used as ground-truth data for retraining or updating a machine learning model.

[0087] In at least one embodiment, the deployment system 706 can include software 718, a service 720, hardware 722, and / or other components, features, and functionalities. In at least one embodiment, the deployment system 706 can include a software "stack" such that the software 718 can be built on top of the service 720 and use the service 720 to perform some or all of the processing tasks, and the service 720 and the software 718 can be built on top of the hardware 722 and use the hardware 722 to perform processing, storage, and / or other computing tasks of the deployment system 706.

[0088] In at least one embodiment, the software 718 can include any number of distinct containers, each container capable of executing an instantiation of an application. In at least one embodiment, each application can perform one or more processing tasks in a more advanced processing and inference pipeline (e.g., inference, object detection, feature detection, segmentation, image enhancement, calibration, etc.). In at least one embodiment, any number of containers can be provided for each type of computing device, capable of performing a data processing task with respect to the feedback data 708 (or other data types, such as those described herein).In at least one embodiment, an advanced processing and inference pipeline can be defined based on selections of different containers that are desired or required for processing the feedback data 708, in addition to containers that receive and configure imaging data for use by each container and / or for use by the device 702 after processing by a pipeline (e.g., for converting outputs back into a usable data type for storage and display in the device 702). In at least one embodiment, a combination of containers within the software 718 (e.g., forming a pipeline) can be referred to as a virtual instrument (as described in more detail herein), and a virtual instrument can utilize the service 720 and hardware 722 to perform some or all of the processing tasks of applications instantiated in containers.

[0089] In at least one embodiment, data can be preprocessed as part of a data processing pipeline to prepare it for processing by one or more applications. In at least one embodiment, postprocessing can be performed on an output of one or more inference tasks or other processing tasks of a pipeline to prepare an output file for a subsequent application and / or to prepare output data for transmission and / or use by a user (e.g., in response to an inference request). In at least one embodiment, inference tasks can be performed by one or more machine learning models, such as trained or provisioned neural networks, which may include output models 716 of the training system 704.

[0090] In at least one embodiment, the tasks of the data processing pipeline can be encapsulated in a container(s), each representing a discrete, fully functional instantiation of an application and a virtualized computing environment capable of referencing machine learning models. In at least one embodiment, containers or applications can be published in a private (e.g., access-restricted) area of ​​a container registration database (described in more detail herein), and trained or deployed models can be stored in the model registration database 724 and associated with one or more applications. In at least one embodiment, images of applications (e.g.,Container images) are available in a container registry database, and once a user has selected an image from a container registry database for use in a pipeline, it can be used to create a container for instantiating an application for use by a user's system.

[0091] In at least one embodiment, developers can develop, publish, and store applications (e.g., as containers) for processing and / or inferring input data. In at least one embodiment, development, publication, and / or storage can be performed using a software development kit (SDK) associated with a system (e.g., to ensure that a developed application and / or container is compliant or compatible with a system). In at least one embodiment, an application under development can be tested locally (e.g., in a first facility, on data from a first facility) using an SDK that includes at least some of the services 720 as a system (e.g., System 800 from Fig. 8) can support. In at least one embodiment, once an application has been validated by the System 800 (e.g., regarding accuracy, etc.), it can be made available in a container registry database for selection and / or embodiment by a user (e.g., a hospital, clinic, laboratory, healthcare provider, etc.) to perform one or more processing tasks relating to data in a facility (e.g., a second facility) of a user.

[0092] In at least one embodiment, developers can then make applications or containers accessible through a network for users of a system (e.g., System 800). Fig. 8) share. In at least one embodiment, completed and validated applications or containers can be stored in a container registration database, and associated machine learning models can be stored in the model registration database 724. In at least one embodiment, a requesting entity—providing an inference or image processing request—can search a container registration database and / or model registration database 724 for an application, container, dataset, machine learning model, etc., select a desired combination of elements to include in the data processing pipeline, and submit a processing request.In at least one embodiment, a request may include input data necessary to execute a request and / or a selection of application(s) and / or machine learning models to be executed during the processing of a request. In at least one embodiment, a request may then be passed to one or more components of the deployment system 706 (e.g., a cloud) to perform the data processing pipeline. In at least one embodiment, the processing by the deployment system 706 may include referencing selected elements (e.g., applications, containers, models, etc.) from a container registration database and / or model registration database 724. In at least one embodiment, once results have been generated by a pipeline, the results may be returned to a user as a reference (e.g.,for viewing in a view application suite running on a local workstation or local terminal on the premises).

[0093] In at least one embodiment, Service 720 can be used to support the processing or execution of applications or containers in pipelines. In at least one embodiment, Service 720 can include computational services, content co-creation services, simulation services, artificial intelligence (AI) services, visualization services, and / or other service types. In at least one embodiment, Service 720 can provide functionality that is shared by one or more applications in Software 718, such that the functionality can be abstracted into a service that can be called or used by applications. In at least one embodiment, the functionality provided by Service 720 can run dynamically and more efficiently, while also being highly scalable by allowing applications to process data in parallel, e.g.,using a parallel computing platform 830 (. Fig. 8) Instead of requiring each application sharing the same functionality offered by a Service 720 to have a corresponding instance of Service 720, Service 720 can be shared by different applications in at least one embodiment. In at least one embodiment, the services can include an inference server or inference engine that can be used as non-restrictive examples for performing detection or segmentation tasks. In at least one embodiment, a model training service can be included that can provide the ability to train and / or retrain machine learning models.

[0094] In at least one embodiment, where a Service 720 includes an AI service (e.g., an inference service), one or more machine learning models associated with an anomaly detection application (e.g., tumors, growth abnormalities, scarring, etc.) can be executed by calling an inference service (e.g., an inference server) (e.g., as an API call) to execute the machine learning model(s) or their processing as part of the application execution. In at least one embodiment, where another application includes one or more machine learning models for segmentation tasks, an application can call an inference service to execute machine learning models to perform one or more processing operations associated with segmentation tasks.In at least one embodiment, the software 718, which implements an advanced processing and inference pipeline, can be streamlined, since each application can call the same inference service to perform one or more inference tasks.

[0095] In at least one embodiment, the hardware can include 722 GPUs, CPUs, data processing units (DPUs), an AI / deep learning system (e.g., an AI supercomputer such as the DGX). TM-NVIDIA supercomputer system), a cloud platform, or a combination thereof. In at least one embodiment, different types of hardware 722 can be used to provide efficient, purpose-built support for software 718 and the service 720 in the deployment system 706. In at least one embodiment, the use of GPU processing for local processing (e.g., in the facility 702), within an AI / deep learning system, in a cloud system, and / or in other processing components of the deployment system 706 can be implemented to improve the efficiency, accuracy, and effectiveness of game name recognition.

[0096] In at least one embodiment, the software 718 and / or the service 720 can be optimized as non-limiting examples of GPU processing with respect to deep learning, machine learning, and / or high-performance computing, simulation, and visual computing. In at least one embodiment, at least a portion of the computing environment of the deployment system 706 and / or the training system 704 can be located in a data center on one or more supercomputers or high-performance computing systems with GPU-optimized software (e.g., a hardware and software combination of the DGX). TM-Systems from NVIDIA). In at least one embodiment, the hardware 722 can include any number of GPUs that can be called upon for parallel processing of data as described herein. In at least one embodiment, the cloud platform can further include GPU processing for GPU-optimized execution of deep learning tasks, machine learning tasks, or other computational tasks. In at least one embodiment, the cloud platform (e.g., NGC) can include TM from NVIDIA) using AI / deep learning supercomputer(s) and / or GPU-optimized software (e.g., as on DGX) TM(provided by NVIDIA systems) as a hardware abstraction and scaling platform. In at least one embodiment, the cloud platform can integrate an application container clustering or orchestration system (e.g., Kubernetes) across multiple GPUs to enable seamless scaling and load balancing.

[0097] Fig. Figure 8 is a system diagram for an exemplary system 800 for generating and deploying a deployment pipeline according to at least one embodiment. In at least one embodiment, the system 800 can be used to perform the method 700 from Fig. 7 and / or other processes, including advanced processing and inference pipelines. In at least one embodiment, the system 800 may include the training system 704 and the deployment system 706. In at least one embodiment, the training system 704 and the deployment system 706 may be implemented using software 718, services 720, and / or hardware 722, as described herein.

[0098] In at least one embodiment, System 800 (e.g., Training System 704 and / or Deployment System 706) can be implemented in a cloud computing environment (e.g., using Cloud 826). In at least one embodiment, System 800 can be implemented locally with respect to a facility or as a combination of both cloud and local computing resources. In at least one embodiment, access to the APIs in Cloud 826 can be restricted to authorized users by imposed security measures or protocols. In at least one embodiment, a security protocol can include web tokens that can be signed by an authentication service (e.g., AuthN, AuthZ, Gluecon, etc.) and carry appropriate authorization.In at least one embodiment, APIs of virtual instruments (described herein) or other instantiations of System 800 may be restricted to a set of public Inter-Service Providers (ISPs) that have been security-vetted or authorized to interact.

[0099] In at least one embodiment, various components of System 800 can communicate with each other using any variety of different network types, including, but not limited to, local area networks (LANs) and / or wide area networks (WANs), via wired and / or wireless communication protocols. In at least one embodiment, communication between devices and components of System 800 (e.g., for transmitting inference requests, receiving results of inference requests, etc.) can be conducted via a data bus or data buses, wireless data protocols (e.g., Wi-Fi), wired data protocols (e.g., Ethernet), etc.

[0100] In at least one embodiment, the training system 704 can execute training pipelines 804 similar to those described herein with respect to Fig. 7 are described. In at least one embodiment, in which one or more machine learning models are to be used in a deployment pipeline 810 by the deployment system 706, a training pipeline 804 can be used to train or retrain one or more (e.g., pre-trained) models and / or to implement one or more of the pre-trained models 806 (e.g., without the need for retraining or updating). In at least one embodiment, output model(s) 716 can be generated as a result of the training pipeline 804.In at least one embodiment, the training pipeline(s) 804 may include any number of processing steps, AI-assisted annotation 710, labeling or annotation of feedback data 708 to generate labeled data 712, model selection from a model registration database, model training 714, training, retraining, or updating of models, and / or other processing steps. In at least one embodiment, a DICOM adapter 802a may be used to acquire DICOM data. In at least one embodiment, different training pipelines 804 may be used for different machine learning models used by the deployment system 706. In at least one embodiment, a training pipeline 804 similar to a first example described in relation to can be used for a first machine learning model. Fig. As described in section 7, a second machine learning model (a) training pipeline(s) 804 may be used, which is similar to a second example, which is related to Fig. 7 is described, and for a third machine learning model, (a) training pipeline(s) 804 are used, which is similar to a third example, which is in relation to Fig. 7 is described. In at least one embodiment, any combination of tasks can be used within the training system 704, depending on what is required for each respective machine learning model. In at least one embodiment, one or more of the machine learning models can already be trained and ready for use, so that the machine learning models may not require any processing by the training system 704 and can be implemented by the deployment system 706.

[0101] In at least one embodiment, the output model(s) 716 and / or the pre-trained model(s) 806 may, depending on the embodiment, include any type of machine learning model(s). In at least one embodiment and without limitation, the machine learning models used by System 800 may include machine learning models using linear regression, logistic regression, decision trees, support vector machines (SVM), naive Bayes classifier, k-nearest neighbor (Knn), k-means clustering, random forest, dimensionality reduction algorithms, gradient enhancement algorithms, neural networks (e.g., autocoders, convolutional, recurrent, perceptrons, long / short term memory (LSTM), Bi-LSTM, Hopfield, Boltzmann, deep belief, unfolding, generating adversary, fluid state machine, etc.).) and / or other types of machine learning models.

[0102] In at least one embodiment, the training pipeline(s) 804 may include AI-assisted annotation. In at least one embodiment, labeled data 712 (e.g., conventional annotation) may be generated by any number of techniques. In at least one embodiment, labels or other annotations may be generated within a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of program suitable for generating annotations or labels for ground truth, and / or, in some examples, drawn by hand. In at least one embodiment, the ground truth data may be synthetically produced (e.g., generated from computer models or renderings), actually produced (e.g., designed and produced from real-world data), or automatically generated (e.g.,using feature analysis and learning to extract features from data and then generate labels), annotated by humans (e.g., a labeler or annotation expert defines the location of labels), and / or a combination thereof. In at least one embodiment, for each instance of the feedback data 708 (or other data type used by machine learning models), corresponding ground-truth data generated by the training system 704 may be available. In at least one embodiment, the AI-assisted annotation may be performed as part of the deployment pipeline(s) 810, either in addition to or instead of the AI-assisted annotation included in the training pipeline(s) 804. In at least one embodiment, the system 800 may include a multi-layered platform comprising a software layer (e.g.,Software 718) may include diagnostic applications (or other application types) that can perform one or more medical imaging and diagnostic functions.

[0103] In at least one embodiment, a software layer can be implemented as a secure, encrypted, and / or authenticated API through which applications or containers from an external environment(s), e.g., facility 702, can be selected (e.g., called). In at least one embodiment, applications can then call or execute one or more services 720 to perform computational, AI, or visualization tasks associated with the respective applications, and the software 718 and / or the services 720 can utilize the hardware 722 to perform processing tasks effectively and efficiently.

[0104] In at least one embodiment, the deployment system 706 can execute deployment pipelines 810. In at least one embodiment, the deployment pipeline(s) 810 can include any number of applications that can be applied sequentially, non-sequentially, or otherwise to feedback data (and / or other data types), including AI-assisted annotation as described above. In at least one embodiment as described herein, the deployment pipeline(s) 810 for a single device can be referred to as a virtual instrument for a device. In at least one embodiment, more than one deployment pipeline 810 can exist for a single device, depending on the information desired from data generated by a device.

[0105] In at least one embodiment, applications available for the deployment pipeline(s) 810 can include any application that can be used to perform processing tasks on feedback data or other data from devices. Because different applications in at least one embodiment can share image operations, a data extension library (e.g., as one of the services 720) can be used in some embodiments to accelerate these operations. In at least one embodiment, to avoid bottlenecks of conventional processing approaches that rely on CPU processing, the parallel computing platform 830 can be used to GPU-accelerate this processing task.

[0106] In at least one embodiment, the deployment system 706 may include a user interface (UI) 814 (e.g., a graphical user interface, a web interface, etc.) that can be used to select applications for inclusion in the deployment pipeline(s) 810, to order applications, to modify or change applications or parameters or constructs thereof, to use and interact with the deployment pipeline(s) 810 during setup and / or deployment, and / or to otherwise interact with the deployment system 706. In at least one embodiment, although not illustrated with respect to the training system 704, the UI 814 (or another user interface) may be used to select models for use in the deployment system 706, to select models for training or retraining in the training system 704, and / or to otherwise interact with the training system 704.

[0107] In at least one embodiment, the pipeline manager 812 can be used in addition to an application orchestration system 828 to manage the interaction between the applications or containers of the deployment pipeline(s) 810 and the services 720 and / or the hardware 722. In at least one embodiment, the pipeline manager 812 can be configured to facilitate application-to-application, application-to-service 720, and / or application-or-service-to-hardware 722 interactions. Although it is included in the software 718 for illustrative purposes, this is not intended to be restrictive in at least one embodiment, and in some examples, the pipeline manager 812 can be included in the services 720. In at least one embodiment, the application orchestration system 828 (e.g., Kubernetes, Docker, etc.) can be configured to facilitate application-to-service, application-to-service 720, and / or application-to-service 722 interactions.) include a container orchestration system that can group applications into containers as logical units for coordination, management, scaling, and deployment. In at least one embodiment, by associating applications from the deployment pipeline(s) 810 (e.g., a reconstruction application, a segmentation application, etc.) with individual containers, each application can be run in a self-contained environment (e.g., at the kernel level) to increase speed and efficiency.

[0108] In at least one embodiment, each application and / or container (or an image thereof) can be developed, modified, and deployed individually (e.g., a first user or developer can develop, modify, and deploy a first application, and a second user or developer can develop, modify, and deploy a second application separately from the first user or developer). This allows users to focus on and concentrate on the task of a single application and / or container(s) without being hindered by tasks of other application(s) or container(s). In at least one embodiment, communication and cooperation between different containers or applications can be supported by the pipeline manager 812 and the application orchestration system 828.In at least one embodiment, as long as an expected input and / or output of each container or application is known to a system (e.g., based on constructs of applications or containers), the application orchestration system 828 and / or the pipeline manager 812 can facilitate communication among and between each of the applications or containers, as well as the sharing of resources among and between them. Since one or more applications or containers in the deployment pipeline(s) 810 can share the same services and resources, the application orchestration system 828, in at least one embodiment, can orchestrate, distribute, and determine the sharing of services or resources between and among different applications or containers.In at least one embodiment, a scheduler can be used to track the resource requirements of applications or containers, the current or planned use of these resources, and resource availability. In at least one embodiment, the scheduler can thus allocate resources to different applications and distribute resources between and among applications with respect to system demand and availability. In some examples, the scheduler (and / or another component of the Application Orchestration System 828) can determine resource availability and distribution based on constraints imposed on a system (e.g., user constraints), such as Quality of Service (QoS), urgency of data output needs (e.g., to determine whether real-time or delayed processing should be performed), and so on.

[0109] In at least one embodiment, the services 720, which are utilized and shared by applications or containers in the deployment system 706, may include the computing service(s) 816, a content co-creation service 817, the AI ​​service(s) 818, the visualization service(s) 820, and / or other service types. In at least one embodiment, applications may call (e.g., execute) one or more of the services 720 to perform processing operations for an application. In at least one embodiment, the computing service(s) 816 may be utilized by applications to perform supercomputing or other high-performance computing (HPC) tasks. In at least one embodiment, the computing service(s) 816 may be utilized to perform parallel processing (e.g.,using a parallel computing platform 830) for processing data by one or more applications and / or performing one or more tasks of a single application substantially simultaneously. In at least one embodiment, the parallel computing platform 830 (e.g., CUDA) can be used. ®from NVIDIA) enable general-purpose computing on GPUs (GPGPU) (e.g., GPUs / Graphics 822). In at least one embodiment, a software layer of the Parallel Computing Platform 830 can provide access to virtual instruction sets and parallel processing elements of GPUs for executing computation kernels. In at least one embodiment, the Parallel Computing Platform 830 can include memory, and in some embodiments, memory can be shared between and among multiple containers and / or between and among different processing tasks within a single container. In at least one embodiment, calls for inter-process communication (IPC) can be generated for multiple containers and / or for multiple processes or procedures within a container to use the same data from a shared memory segment of the Parallel Computing Platform 830 (e.g.,(e.g., when multiple different stages of an application or multiple applications process the same information). In at least one embodiment, the same data can be used in the same memory location for any number of processing tasks (e.g., at the same time, at different times, etc.) instead of creating a copy of the data and moving the data to different locations in memory (e.g., a read / write operation). In at least one embodiment, since data is used to generate new data as a result of processing, this information can be stored about a new data location and shared by different applications. In at least one embodiment, a data location and a location of updated or modified data can be part of a definition of how payload data is to be understood within containers.

[0110] In at least one embodiment, the AI ​​service(s) 818 can be used to perform inference services for executing machine learning models associated with applications (e.g., tasked with performing one or more processing tasks of an application). In at least one embodiment, the AI ​​service(s) 818 can utilize the AI ​​system(s) 824 to execute machine learning models (e.g., neural networks such as CNNs) for segmentation, reconstruction, object detection, feature detection, classification, and / or other inference tasks. In at least one embodiment, the applications of the deployment pipeline(s) 810 can use one or more of the output model(s) 716 from the training system 704 and / or other models of the applications to perform inference on imaging data (e.g. DICOM data, RIS data, CIS data, REST-compliant data, RPC data, raw data, etc.).For example, the DICOM adapter 802b can be used to acquire DICOM data. In at least one embodiment, two or more examples of inference using the application orchestration system 828 are provided. (e.g., a scheduler) may be available. In at least one embodiment, a first category may include a high-priority / low-latency path that can achieve higher Service Level Agreements, such as for performing inference on urgent requests during an emergency or for a radiologist during diagnosis. In at least one embodiment, a second category may include a standard-priority path that can be used for requests that are not urgent or where the analysis can be performed at a later time. In at least one embodiment, the application orchestration system 828 may distribute resources (e.g., services 720 and / or hardware 722) based on priority paths for different inference tasks of the AI ​​service(s) 818.

[0111] In at least one embodiment, the shared data storage can be connected to the AI ​​service(s) 818 within the system 800. In at least one embodiment, the shared data storage can operate as a cache (or other storage device type) and be used to process inference requests from applications. In at least one embodiment, when an inference request is sent, a request can be received by a set of API instances of the deployment system 706, and one or more instances can be selected (e.g., for best fit, load balancing, etc.) to process a request.In at least one embodiment, to process a request, a request can be entered into a database; a machine learning model can be located from the model registration database 724 if it is not already in a cache; a validation step can ensure that a suitable machine learning model is loaded into a cache (e.g., a shared data store); and / or a copy of a model can be saved in a cache. In at least one embodiment, the scheduler (e.g., of the pipeline manager 812) can be used to start an application referenced in a request if an application is not already running or if there are not enough instances of an application. In at least one embodiment, an inference server can be started if an inference server is not already running to execute a model.In at least one embodiment, any number of inference servers can be started per model. In at least one embodiment, models can be cached in a pull model where inference servers are clustered, whenever load balancing is advantageous. In at least one embodiment, inference servers can be statically loaded onto corresponding distributed servers.

[0112] In at least one embodiment, inference can be performed using an inference server running in a container. In at least one embodiment, an instance of an inference server can be associated with a model (and optionally a plurality of versions of a model). In at least one embodiment, if an instance of an inference server does not exist when a request to perform inference on a model is received, a new instance can be loaded. In at least one embodiment, a model can be passed to an inference server when it is started, so that the same container can be used to serve different models, as long as the inference server is running as a separate instance.

[0113] In at least one embodiment, an inference request for a given application can be received during application execution, and a container (which, for example, hosts an instance of an inference server) can be loaded (if it is not already loaded), and a start procedure can be invoked. In at least one embodiment, preprocessing logic in a container can load incoming data, decode it, and / or perform any additional preprocessing on it (e.g., using CPU(s) and / or GPU(s)). In at least one embodiment, once the data is prepared for inference, a container can perform the inference on the data as needed. In at least one embodiment, this can involve a single inference call on a single image (e.g., a hand X-ray) or require inference on hundreds of images (e.g., a breast CT scan).In at least one embodiment, an application can summarize results before completion, which may include, without limitation, a single confidence score, pixel-level segmentation, voxel-level segmentation, the generation of a visualization, or the generation of text summarizing findings. In at least one embodiment, different models or applications can be assigned different priorities. For example, some models may have a real-time priority (turnaround time less than one minute), while others may have a lower priority (e.g., turnaround time less than 10 minutes). In at least one embodiment, model execution times can be measured by the requesting institution or entity and may include the time spent traversing the partner network as well as execution on an inference service.

[0114] In at least one embodiment, the transmission of requests between the 720 services and the inference applications can be hidden behind a software development kit (SDK), and robust transport can be provided through a queue. In at least one embodiment, a request is placed in a queue via an API for a single application / tenant ID combination, and an SDK retrieves a request from the queue and returns it to an application. In at least one embodiment, a queue name can be provided in an environment from which an SDK retrieves the request. In at least one embodiment, asynchronous communication through a queue can be beneficial because it allows any instance of an application to begin work as soon as it becomes available.In at least one embodiment, results can be returned via a queue to ensure no data is lost. In at least one embodiment, queues can also provide the ability to segment work, as highest-priority work can go to a queue to which most instances of an application are connected, while lowest-priority work can go to a queue to which only a single instance is connected, processing tasks in the order they are received. In at least one embodiment, an application can run on a GPU-accelerated instance created in Cloud 826, and an inference service can perform the inference on a GPU.

[0115] In at least one embodiment, the visualization service(s) 820 can be used to generate visualizations for viewing outputs from applications and / or deployment pipeline(s) 810. In at least one embodiment, the GPUs / graphics 822 can be used by the visualization service(s) 820 to generate visualizations. In at least one embodiment, rendering effects, such as ray tracing or other light transport simulation methods, can be implemented by the visualization service(s) 820 to generate higher-quality visualizations. In at least one embodiment, visualizations can include, without limitation, 2D image renderings, 3D volume renderings, 3D volume reconstructions, 2D tomography slices, virtual reality displays, augmented reality displays, etc.In at least one embodiment, virtualized environments can be used to create a virtual interactive display or environment (e.g., a virtual environment) for interaction by users of a system (e.g., physicians, nurses, radiologists, etc.). In at least one embodiment, the visualization service(s) 820 can include an internal visualizer, kinematics, and / or other rendering or image processing capabilities or functions (e.g., ray tracing, rasterization, internal optics, etc.).

[0116] In at least one embodiment, the hardware 722 may include GPUs / graphics 822, the AI ​​system(s) 824, the cloud 826, and / or any other hardware used to run the training system 704 and / or the deployment system 706. In at least one embodiment, the GPUs / graphics 822 (e.g., TESLA) ® - and / or QUADRO ®NVIDIA GPUs) may include any number of GPUs that can be used to perform processing tasks by the Computing Service(s) 816, Content Co-creation Service(s) 817, AI Service(s) 818, Simulation Service(s), Visualization Service(s) 820, other services, and / or any features or functions of Software 718. With respect to AI Service(s) 818, the GPUs / Graphics 822 may, for example, be used to perform preprocessing on imaging data (or other data types used by machine learning models), postprocessing on outputs of the machine learning models, and / or to perform inference (e.g., to run machine learning models). In at least one embodiment, the Cloud 826, the AI ​​system(s) 824 and / or other components of the System 800 can use the GPUs / graphics 822.In at least one embodiment, the Cloud 826 can include a GPU-optimized platform for deep learning tasks. In at least one embodiment, the AI ​​system(s) 824 can utilize GPUs, and the Cloud 826—or at least a section tasked with deep learning or inference—can run using one or more AI systems 824. Although the hardware 722 is shown to consist of discrete components for illustrative purposes, this is not intended to be restrictive, and any components of the hardware 722 can be combined with or utilized by any other components of the hardware 722.

[0117] In at least one embodiment, the AI ​​system(s) 824 may include a specially designed computing system (e.g., a supercomputer or an HPC) configured for inference, deep learning, machine learning, and / or other artificial intelligence tasks. In at least one embodiment, the AI ​​system(s) 824 (e.g., DGX) may include TM from NVIDIA) include GPU-optimized software (e.g., a software stack) that can be run using a variety of GPUs / graphics 822 in addition to CPUs, RAM, data storage, and / or other components, features, or functions. In at least one embodiment, one or more AI systems 824 may be deployed in the cloud 826 (e.g., in a data center) to perform some or all of the AI-based processing tasks of the system 800.

[0118] In at least one embodiment, the Cloud 826 can utilize a GPU-accelerated infrastructure (e.g., NGC). TMfrom NVIDIA) that can provide a GPU-optimized platform for performing processing tasks of the System 800. In at least one embodiment, the Cloud 826 can include an AI system(s) 824 for performing one or more AI-based tasks of the System 800 (e.g., as a hardware abstraction and scaling platform). In at least one embodiment, the Cloud 826 can be integrated with the application orchestration system 828, which utilizes multiple GPUs to enable seamless scaling and load balancing between and among the applications and services 720. In at least one embodiment, the Cloud 826 can be tasked with performing at least some of the services 720 of the System 800, including the compute service(s) 816, the AI ​​service(s) 818, and / or the visualization service(s) 820, as described herein.In at least one embodiment, Cloud 826 can perform inference with small and large batches (e.g., using TensorRT). TM from NVIDIA), provide an accelerated parallel computing platform 830 (e.g., NVIDIA's CUDA), run an application orchestration system 828 (e.g., Kubernetes), provide a graphics rendering API and platform (e.g., for ray tracing, 2D graphics, 3D graphics, and / or other rendering techniques to produce higher-quality kinematics), and / or provide other functions for the System 800. In at least one embodiment, the parallel computing platform 830 may include an API.

[0119] In at least one embodiment, the Cloud 826 can include a registry database—such as a deep learning container registry database—in an effort to maintain the confidentiality of patient data (e.g., when patient data or records are to be used off-site). In at least one embodiment, a registry database can store containers for instantiating applications that can perform preprocessing, postprocessing, or other processing tasks on patient data. In at least one embodiment, the Cloud 826 can receive data containing patient data and sensor data in containers, perform the requested processing only on the sensor data in these containers, and then forward any resulting output and / or visualizations to appropriate parties and / or devices (e.g.,medical devices in the premises used for visualization or diagnosis) without requiring the extraction, storage, or other access to patient data. In at least one embodiment, patient data confidentiality is maintained in accordance with HIPAA and / or other data protection regulations.

[0120] Other variations are within the spirit of the present disclosure. Thus, while various modifications and alternative constructions can be made with respect to the disclosed methods, certain illustrated embodiments are shown in the drawings and have been described in detail above. However, it is understood that the intention is not to limit the disclosure to the specific disclosed form(s), but rather, on the contrary, to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of the disclosure as defined in the attached claims.

[0121] The use of the terms "a," "an," "the," and similar referents in the context of describing disclosed embodiments (particularly in the context of the following claims) is to be interpreted as covering both the singular and the plural unless otherwise specified herein or the context clearly contradicts this, and not as defining an expression. The terms "comprising," "having," "including," and "containing" are to be interpreted as open expressions (meaning "including but not limited to") unless otherwise noted. When used unmodified and referring to physical connections, "connected" is to be interpreted as partially or completely contained within, attached to, or joined to one another, even if an element is interposed.The mention of value ranges herein is intended merely as a short procedure for individually referring to each separate value falling within the range, unless otherwise stated herein, and each separate value is included in the description as if it were individually reproduced herein. In at least one embodiment, the use of the terms "set" or "set" (e.g., "a set or a set of items") or "subset" is to be interpreted as a non-empty collection comprising one or more elements, unless otherwise noted or the context contradicts. Furthermore, unless otherwise stated or the context contradicts, the term "subset" of a corresponding set does not necessarily denote a proper subset of the corresponding set, but the subset and the corresponding set may be the same.

[0122] Unless specifically stated otherwise or the context clearly contradicts it, connective language, such as phrases of the form "at least one of A, B, and C" or "at least one of A, B, and C," is otherwise to be understood in the context in which it is generally used to indicate that an object, expression, etc., can be either A, B, or C, or any non-empty subset of the sentence containing A, B, and C. For example, in the illustrative example of a sentence containing three elements, the connective phrases "at least one of A, B, and C" and "at least one of A, B, and C" refer to any one of the following: {A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such connective language is generally not intended to imply that certain statements require the presence of at least one of A, at least one of B, and at least one of C.Unless otherwise noted or the context indicates otherwise, the term "plurality" signifies a state of plurality (e.g., "a plurality of items" indicates multiple items). In at least one embodiment, the number of items in a plurality is at least two, but it may be more if either explicitly stated or indicated by the context. Unless otherwise stated or evident from the context, the phrase "based on" means "at least partly based on" or "at least on the basis of" and not "exclusively based on".

[0123] The operations of methods described herein may be performed in any suitable order unless otherwise specified herein or the context clearly precludes such order. In at least one embodiment, a method such as the methods described herein (or variations and / or combinations thereof) is performed under the control of one or more computer systems configured with executable instructions, and it is implemented by hardware or combinations thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors. In at least one embodiment, code is stored on a computer-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that can be executed by one or more processors.In at least one embodiment, a computer-readable storage medium is a non-transient computer-readable storage medium that excludes transitory signals (e.g., a propagating transient electrical or electromagnetic transmission) but includes non-transient data storage circuits (e.g., buffers, caches, and queues) within transient signal senders / receivers. In at least one embodiment, the code (e.g., executable code or source code) is stored on a set of one or more non-transient computer-readable storage media containing executable instructions (or other storage for executable instructions) which, when executed (i.e., as a result of execution) by one or more processors of a computer system, cause the computer system to perform the operations described herein.In at least one embodiment, a set of nontransitory computer-readable storage media comprises multiple nontransitory computer-readable storage media. One or more of the individual nontransitory storage media lack the entire code, while multiple nontransitory computer-readable storage media together store the entire code. In at least one embodiment, the executable instructions are executed such that different instructions are executed by different processors—for example, a nontransitory computer-readable storage medium stores instructions, and a main central processing unit (“CPU”) executes some of the instructions, while a graphics processing unit (“GPU”) executes other instructions.In at least one embodiment, different components of a computer system have separate processors, and different processors execute different subsets of instructions.

[0124] Accordingly, in at least one embodiment, computer systems are configured to implement one or more services that, individually or collectively, perform operations of the methods described herein, and such computer systems are equipped with suitable hardware and / or software that enables the execution of the operations. Furthermore, a computer system implementing at least one embodiment of the present disclosure is a single device, and in another embodiment, it is a distributed computer system comprising several devices that operate differently, such that the distributed computer system performs the operations described herein and such that a single device does not perform all of the operations.

[0125] The use of any and all examples or illustrative language (“such as”) provided herein is intended solely to better clarify the embodiments of the disclosure and does not constitute a limitation of the scope of the disclosure unless claimed otherwise. No wording in the description should be interpreted as indicating any unclaimed element as essential to the implementation of the disclosure.

[0126] Any references, including publications, patent applications and patents mentioned herein, are hereby incorporated by reference to the same extent as if each reference were individually and specifically indicated as being included by reference and set forth herein in their entirety.

[0127] In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. It is understood that these terms are not intended to be synonymous. Rather, in specific examples, "connected" or "coupled" can be used to indicate that two or more elements are in direct or indirect physical or electrical contact with each other. "Coupled" can also mean that two or more elements are not in direct contact with each other, but nevertheless interact or work together.

[0128] Unless expressly stated otherwise, in some embodiments, terms such as "processing", "calculating", "calculating", "determining" or the like throughout the description are understood to refer to actions and / or procedures of a computer or computing system or similar electronic computing device that manipulate and / or convert data represented as physical, e.g. electronic, quantities in the registers and / or memories of the computing system into other data represented in a similar manner as physical quantities in the memories, registers or other such information storage, transmission or display devices of the computing system.

[0129] Similarly, the term "processor" can refer to any device or section of a device that processes electronic data from registers and / or memory and converts that electronic data into other electronic data that can be stored in registers and / or memory. As non-restrictive examples, the "processor" can be a CPU or a GPU. A "computing platform" can include one or more processors. As used herein, "software" procedures can include, for example, software and / or hardware entities that perform work over time, such as tasks, threads, and intelligent agents. Furthermore, each procedure or process can refer to multiple procedures or processes for executing instructions sequentially or in parallel, continuously or intermittently.In at least one embodiment, the terms “system” and “method” are used interchangeably herein insofar as a system can embody one or more methods and the methods can be regarded as a system.

[0130] This document may refer to the acquisition, capture, reception, or input of analog or digital data into a subsystem, computer system, or computer-implemented machine. In at least one embodiment, the method of acquiring, capturing, receiving, or inputting analog and digital data can be achieved in a variety of ways, such as by receiving data as parameters of a function call or an application programming interface call. In at least one embodiment, methods of acquiring, capturing, receiving, or inputting analog or digital data can be achieved by transmitting data via a serial or parallel interface.In at least one embodiment, methods for obtaining, acquiring, receiving, or inputting analog or digital data can be carried out by transmitting data over a computer network from the providing entity to the acquiring entity. In at least one embodiment, references can also be made to providing, outputting, transmitting, sending, or displaying analog or digital data. In various examples, a method for providing, outputting, transmitting, sending, or displaying analog or digital data can be carried out by transmitting data as an input or output parameter of a function call, a parameter of an application programming interface, or an interprocess communication mechanism.

[0131] Although the description presented here presents exemplary embodiments of the described methods, other architectures may also be used to implement the described functionality, and these are intended to be within the scope of this disclosure. Furthermore, although specific distributions of responsibilities may be defined above for the purpose of description, various functions and responsibilities may be distributed and divided differently depending on the circumstances.

[0132] Although the subject matter has been further described in language specific to structural features and / or process steps, it is understood that the subject matter claimed in the appended claims is not necessarily limited to the specific features or steps described. Rather, specific features and steps are disclosed as exemplary ways of implementing the claims.

Claims

[1] Procedure encompassing: Execution, by a processing unit, of a first task with associated processor configuration requirements; Receiving a second task and a third task to be executed by the processing unit, wherein the second task and the third task each have associated processor configuration requirements; Determining a first next task to be performed by the processing unit based on a comparison of the processing unit's current processor configuration with the processor configuration requirements of the second and third tasks, wherein the processing unit's current processor configuration is based on the processor configuration requirements associated with the first task; and Providing the first next task to the processing unit for execution. [2] The method of claim 1, wherein the second problem is received before the third problem; and wherein the first next problem is the third problem. [3] The method of claim 2, further comprising: Receiving a fourth task to be executed by the processing unit, wherein the fourth task has associated processor configuration requirements; Determine at least among the second and fourth tasks a second next task to be performed by the processing unit, based on a comparison of the processor configuration requirements of the third task with the processor configuration requirements of the second and fourth tasks; and Providing the processing unit with the second next task for execution. [4] Method according to any of the preceding claims, wherein the second task and the third task each have associated resource requirements and wherein the determination of the next task to be performed by the processing unit is further based on the current resource availability of the processing unit and the resource requirements of the second task and the third task. [5] The method of claim 4, wherein the resource requirements of the second task comprise at least one of: a thread block size of the second task; a number of registers required by the second task; or a number of barriers required by the second task. [6] Method according to any of the preceding claims, wherein the second task and the third task each comprise an associated program / shader, and wherein determining the next task to be performed by the processing unit is further based on a current program / shader of the processing unit and the program / shader of the second task and the third task. [7] Method according to any of the preceding claims, wherein the processor configuration requirements of the second task include a Level 1 cache memory configuration, L1 cache memory configuration. [8] System encompassing: a parallel processing unit that performs an initial task with associated processor configuration requirements; and a circuit coupled to the parallel processing unit to: to receive a second task and a third task to be executed by the parallel processing unit, wherein the second task and The third task each has its own associated processor configuration requirements; to determine a first next task to be performed by the parallel processing unit based on a comparison of a current processor configuration of the parallel processing unit with the processor configuration requirements of the second and third tasks, wherein the current processor configuration of the parallel processing unit is based on the processor configuration requirements associated with the first task; and to provide the next task for the parallel processing unit to execute. [9] System according to claim 8, wherein the second task is received before the third task; and wherein the first next task is the third task. [10] System according to claim 9, wherein the circuit is further configured to: to receive a fourth task to be executed by the parallel processing unit, wherein the fourth task has associated processor configuration requirements; at least under the second task and the fourth task, a second next task to be executed by the parallel processing unit, to determine based on a comparison of the processor configuration requirements of the third task with the processor configuration requirements of the second and fourth tasks; and to provide the second next task for the parallel processing unit to execute. [11] System according to one of claims 8 to 10, wherein the second task and the third task each have associated resource requirements, and wherein the determination of the next task to be performed by the parallel processing unit is further based on the current resource availability of the parallel processing unit and the resource requirements of the second task and the third task. [12] System according to claim 11, wherein the resource requirements of the second task comprise at least one of: a thread block size of the second task; a number of registers required by the second task; or a number of barriers required by the second task. [13] System according to any one of claims 8 to 12, wherein the second task and the third task each comprise an associated program / shader, and wherein the determination of the next task to be performed by the parallel processing unit is further based on a current program / shader of the parallel processing unit and the program / shader of the second task and the third task. [14] System according to any one of claims 8 to 13, wherein the processor configuration requirements of the second task include a Level 1 cache memory configuration, L1 cache memory configuration. [15] System encompassing: a first processor that performs a first task with associated processor configuration requirements; and a second processor coupled with the first processor to: to receive a second task and a third task to be executed by the first processor, the second task and the third task each having associated processor configuration requirements; a first next task to be performed by the first processor, to determine based on a comparison of the current processor configuration of the first processor with the processor configuration requirements of the second and third tasks, wherein the current processor configuration of the first processor is based on the processor configuration requirements associated with the first task; and to provide the first next task to the first processor for execution. [16] System according to claim 15, wherein the second task is received before the third task; and wherein the first next task is the third task. [17] System according to claim 16, wherein the second processor is further configured to: to receive a fourth task to be executed by the first processor, wherein the fourth task has associated processor configuration requirements; at least under the second and fourth tasks, a second next task to be executed by the first processor, based on a comparison of the processor configuration requirements of the third task with the processor configuration requirements of the second task and to determine the fourth task; and to provide the second next task to the second processor. [18] System according to any one of claims 15 to 17, wherein the second task and the third task each have associated resource requirements, and wherein the determination of the next task to be performed by the first processor is further based on the current resource availability of the first processor and the resource requirements of the second task and the third task. [19] System according to claim 18, wherein the resource requirements of the second task comprise at least one of: a thread block size of the second task; a number of registers required by the second task; or a number of barriers required by the second task. [20] System according to any one of claims 15 to 19, wherein the second task and the third task each have an associated program / shader, and wherein determining the next task to be performed by the first processor is further based on a current program / shader of the first processor and the program / shader of the second task and the third task.

Images