Memory bandwidth limitation for virtual machines

Abstract

Systems and methods for limiting memory bandwidth accessed by virtual machines (VMs) are disclosed. A technique for dynamically limiting the memory access rate of a virtual computer processing unit (vCPU) that controls a VM (tenant) assigned to the VM. When memory is shared by multiple VMs in a cloud computing environment, one VM increasing its memory access rate can cause another VM to suffer from memory access starvation. This behavior violates the principle of VM isolation in cloud computing. In contrast to traditional systems, a software solution for dynamically limiting vCPUs can be implemented in a hypervisor, thus portable across CPU families, and does not require specialized server-level CPU features or limiting system configurations.

Description

[0001] Claiming priority

[0002] This application claims priority to U.S. Patent Application No. 17 / 015,377 (Attorney’s File No. 513160 / 19-PU-0451-US01), filed September 9, 2020, entitled “Memory Bandwidth Throttling for Virtual Machines,” the entire contents of which are incorporated herein by reference. Background Technology

[0003] In cloud computing, virtual machines (VMs) share server resources, including central processing unit (CPU) cores, storage devices, I / O devices, GPUs, and system memory. A fundamental principle of multi-tenancy in cloud computing is the efficient use of resources and the provision of isolation between each VM (tenant) and its neighbors. However, when system memory is shared by multiple VMs in a cloud computing environment, one VM increasing its memory access rate can cause another VM to suffer from insufficient memory access. This behavior undermines the principle of VM isolation in cloud computing.

[0004] Some (physical) CPU hardware implementations support allocating memory access bandwidth at the individual CPU core level. Hardware-based counters and hardware-based limiting techniques for monitoring operate at the physical CPU core level. However, many CPUs, especially cheaper ones, lack sufficient memory access bandwidth allocation capabilities. Typically, a CPU lacks information about which virtual computer processing unit (vCPU) belonging to one of the VMs is running a task on the CPU. VMs and vCPUs can be configured so that a given CPU core executes tasks scheduled for a single, given vCPU. In this way, a correlation can be established between HW counters and vCPU behavior. However, restricting the system to require each CPU core to be limited to use by one or more vCPUs allocated to a single VM may be overly restrictive. These issues and / or other problems related to the prior art need to be addressed. Summary of the Invention

[0005] Embodiments of this disclosure relate to memory bandwidth limiting for virtual machines. Systems and methods for dynamically limiting vCPUs allocated to virtual machines (tenants) to control the memory access rate of the VMs are disclosed. When memory is shared by multiple VMs in a cloud computing environment, one VM increasing its memory access rate may cause another VM to suffer from insufficient memory access. Compared to traditional systems, software solutions for dynamically limiting vCPUs can be implemented in the hypervisor, thus being portable across CPU families and requiring no dedicated server-level CPU functionality or limiting system configuration.

[0006] In one embodiment, a first value corresponding to the execution of a first VM among a plurality of virtual machines (VMs) is calculated based on a memory access metric. The plurality of VMs are configured to share the processing resources of a computing system according to time slices allocated among the VMs. The first value exceeds a threshold, and based on the first value, an idle duration is determined for a first virtual CPU resource allocated to the first VM that is placed in an idle state during a first set of time slices allocated to the first VM. During the execution of programming instructions for the first VM, the first virtual CPU resource is placed in an idle state for the idle duration of at least one time slice in the first set of time slices.

[0007] In one embodiment, an idle duration is applied within each of the N time slices in the first set of time slices. In one embodiment, a duty cycle based on the idle duration is applied across the N time slices included in the first set of time slices. In one embodiment, the first time slice is divided into M intervals, and the idle duration is applied within each of the M intervals. In one embodiment, the memory access metric includes the last-level cache miss rate. In one embodiment, the memory access metric includes at least one of the following: memory access rate, average memory access rate, or memory access count during a defined time interval. In one embodiment, performance data is tracked based on the memory access metric for each of the plurality of VMs. In one embodiment, performance data is sampled at least once in each time slice of the first set of time slices to produce a first value.

[0008] In one embodiment, after at least one time slice, a decrease in the first value is detected, reducing the idle duration of the first VM, and the first virtual CPU resource is placed idle for the reduced idle duration during subsequent time slices following at least one time slice in the first set of time slices. In another embodiment, the first value is equal to or below a threshold, the idle duration of the first VM is zero, and the first virtual CPU resource is active during subsequent time slices in the first set of time slices.

[0009] In one embodiment, the first virtual CPU resources include a plurality of virtual CPUs allocated to a first VM. In one embodiment, a second value is determined based on a second memory access metric of a second VM among the plurality of VMs exceeding a second threshold, and a second idle duration is determined based on the second value. During the second idle duration, the second virtual CPU resources allocated to the second VM are limited by placing the second virtual CPUs in an idle state during a second set of time slices allocated to the second VM. During the execution of programming instructions for the second VM, the second virtual CPU resources are placed in an idle state during the second idle duration of at least one time slice in the second set of time slices. Attached Figure Description

[0010] The system and method for limiting the memory bandwidth of a virtual machine are described in detail below with reference to the accompanying drawings, wherein:

[0011] Figure 1A The illustration shows a block diagram of an example cloud computing environment suitable for implementing some embodiments of the present disclosure;

[0012] Figure 1B The illustration shows a graph of memory access rates during a time slice according to one embodiment;

[0013] Figure 1C The illustrations depict some embodiments suitable for implementing this disclosure. Figure 1A The block diagram of the example scheduler shown below;

[0014] Figure 1D The illustration shows a flowchart of a method for performing memory bandwidth limiting on a VM according to an embodiment;

[0015] Figure 2A The illustration shows example duty cycles of time slices according to some embodiments of the present disclosure;

[0016] Figure 2B An example diagram illustrating a limited memory access rate according to some embodiments of the present disclosure is shown;

[0017] Figure 2C The illustration shows example duty cycles for cross-time-slice applications according to some embodiments of the present disclosure;

[0018] Figure 2D The illustration shows a flowchart of another method for performing memory bandwidth limiting on a VM according to an embodiment;

[0019] Figure 3A Examples of segmented time slices according to some embodiments of the present disclosure are illustrated;

[0020] Figure 3B Another example diagram illustrating the restricted memory access rate according to some embodiments of the present disclosure is shown;

[0021] Figure 3C Another example diagram illustrating the restricted memory access rate according to some embodiments of the present disclosure is shown;

[0022] Figure 4 The illustration shows example parallel processing units suitable for implementing some embodiments of this disclosure;

[0023] Figure 5A For the purpose of implementing some embodiments of this disclosure, the use of Figure 4 A conceptual diagram of the processing system implemented by the PPU;

[0024] Figure 5B The illustration shows an exemplary system in which various architectures and / or functions of various prior embodiments can be implemented;

[0025] Figure 6A For the purpose of implementing some embodiments of this disclosure, the use of Figure 4 A conceptual diagram of the graphics processing pipeline implemented by the PPU;

[0026] Figure 6B An exemplary game streaming system suitable for implementing some embodiments of this disclosure is illustrated. Detailed Implementation

[0027] Systems and methods for dynamically limiting or rate-limiting vCPUs allocated to virtual machines (tenants) in a multi-tenant system to control the memory access rate of each VM are disclosed. Dynamic limiting is implemented in software to adaptively reduce memory access shortages of one or more VMs in a cloud computing environment while preserving VM isolation principles for shared resources. In one embodiment, the software solution for dynamically limiting vCPUs is implemented within a hypervisor, thus being portable across CPU families and requiring no dedicated server-level CPU functionality or limiting system configuration. A scheduler in the hypervisor allocates time slices to vCPUs for processing. When performance data (e.g., measured performance values) based on a vCPU's memory access metric exceeds a threshold, the scheduler limits the vCPU's memory access bandwidth by forcing the vCPU to be idle for at least a portion of each time slice or for the entirety of at least one time slice. When the performance data based on the memory access metric no longer exceeds the threshold, the scheduler stops limiting the vCPU. In one embodiment, the memory access metric is related to the VM's memory access bandwidth requirements.

[0028] Figure 1A The illustration shows a block diagram of an example cloud computing environment suitable for implementing some embodiments of the present disclosure. It should be understood that such and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components and in any suitable combination and location. The various functions described herein as being performed by entities can be implemented by hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. Moreover, those skilled in the art will understand that any system performing the operation of cloud computing environment 100 is within the scope and spirit of the embodiments of the present disclosure.

[0029] like Figure 1AAs shown, the hardware 105 of the cloud computing environment 100 includes at least computing resources such as CPU 110 and system memory 115. Other types of shared and / or dedicated resources (not shown) may also be included in the cloud computing environment 100.

[0030] The memory bandwidth for accessing system memory 115 is shared by VMs 140, each VM 140 being allocated at least one vCPU 135 (virtual processor). A vCPU 135 represents computing resources specifically used by the VM 140 allocated the vCPU 135. In one embodiment, each vCPU 135 is a physical core within CPU 110. In another embodiment, each vCPU 135 includes one or more physical cores within CPU 110. In yet another embodiment, each vCPU 135 includes a portion of the physical cores within CPU 110.

[0031] Time slices 148 are allocated to each VM 140 by scheduler 130 within hypervisor / host operating system 125. Scheduling 145 indicates the allocation of time slices 148 to different VMs 140. For example, time slice 148-1 is a set of time slices allocated to VM 140-1. In one embodiment, each time slice 148 corresponds to a different time interval, and each of the VMs 140 is sequentially allocated a time slice 148 before being allocated another time slice 148, such that the time slices 148 allocated to a particular VM 140 are distributed over time. Over time, each VM 140 is allocated a set of time slices 148 (e.g., time slices 148-1, 148-2, etc.). In one embodiment, time slices are allocated according to a time-division multiplexing scheme to provide each VM 140 with exclusive use of computing resources.

[0032] In another embodiment, time slices 148 are distributed to VMs 140 according to a defined allocation, such that one VM 140 can be allocated N times the time slice of another VM 140. In one embodiment, time slices 148 are of equal duration. For example, 500ms is divided into five time slices 148, and the scheduler 130 allocates a 100ms time slice to each of the five VMs 140. In another embodiment, the duration of time slices 148 can vary and be specified for each VM 140.

[0033] During the allocated time slice 148, VM 140 can use the processing resources of the computing system, specifically CPU 110 and system memory 115. Allocating a dedicated time slice 148 to each VM 140 provides an isolation mechanism for VMs 140 to share processing resources within hardware 105. The hypervisor / host operating system 125 interfaces between VMs 140 and hardware 105, including CPU 110 and system memory 115.

[0034] The hypervisor / host operating system 125 is a software layer that can be configured to also perform operating system functions and run directly on the hardware 105. Alternatively, the hypervisor / host operating system 125 can be configured as a host operating system running directly on the hardware 105 and a hypervisor as a software layer running on the host operating system. The hypervisor / host operating system 125 creates and manages multiple VMs 140 to support the virtualization of the hardware 105.

[0035] Now, further illustrative information is provided regarding the various optional architectures and features that can be used to implement the aforementioned framework according to the user's wishes. It should be strongly noted that the following information is presented for illustrative purposes and should not be construed as restrictive in any way. Optionally, any of the following features may be combined, with or without excluding the other features described.

[0036] Figure 1B Figure 120 illustrates the memory access rate 142 during time slice 148 according to an embodiment. The memory access rate 142 allocated to VM 140 during time slice 148 varies within time slice 148. In one embodiment, a single threshold 143 is defined for all VMs 140. In another embodiment, separate thresholds may be defined for one or more VMs 140. When the memory access rate of VM 140 exceeds the threshold 143, scheduler 130 dynamically limits VM 140 by idling the vCPU 135 allocated to VM 140 for at least a portion of a set of time slices 148 allocated to VM 140. Figure 1B As shown, the memory access rate 142 of VM 140 exceeds the threshold 143 four times during time slice 148. If the memory access rate 142 remains above the threshold 143, the limit amount can be increased. In one embodiment, the limit amount is determined based on the amount by which the memory access rate 142 exceeds the threshold 143 (i.e., the difference between the memory access rate 142 and the threshold 143). This amount can be an instantaneous maximum value, accumulate over time, or be calculated using a function.

[0037] Figure 1C The diagram shows Figure 1AThe block diagram shown is suitable for implementing some embodiments of the present disclosure of a scheduler 130. Scheduler 130 includes a monitor 132 and a time-slice controller 136. Monitor 132 samples performance data 134 for each VM 140 at least once during a monitoring interval and provides a value 138 to scheduler 130. The monitoring interval may include one or more time slices 148 or only a portion of time slices 148. In one embodiment, the monitoring interval is predetermined or programmable. In one embodiment, monitor 132 samples performance data 134 every 10 milliseconds to generate value 138.

[0038] In one embodiment, performance data 134 is sampled once by monitor 132 during each monitoring time interval to generate value 138. In another embodiment, a moving average of performance data 134 is calculated by monitor 132 and provided as value 138, wherein performance data 134 is sampled at least once per time slice 148. Compared to sampling once during each monitoring time interval, a moving average can produce better constraint quality over time, but has a longer response time.

[0039] In one embodiment, performance data 134 includes the Last Level Cache Miss (LLCM) rate, corresponding to the memory access rate. The LLCCM rate represents the number of memory accesses per unit time that are not satisfied at any cache level, thus causing system memory access. In one embodiment, the LLCCM rate allocated to each vCPU 110 of VM 140 is tracked and aggregated to determine the overall LLCCM rate of VM 140. Typically, when the overall LLCCM rate of each VM 140 is less than or equal to a threshold 143, each VM 140 is able to access system memory without negatively impacting other VMs 140.

[0040] In one embodiment, the last-level cache (LLC) is the highest-level cache shared by all cores in one or more CPUs 110. In one embodiment, the LLC is a Level 2 (L2) cache. In other embodiments, the LLC is a Level 3 (L3) or Level 1 (L1) cache. At the end of each monitoring interval, any VM 140 with a memory access rate exceeding a threshold is considered "noisy" and is throttled by the time-slice controller 136. In one embodiment, the scheduler 130 maintains multiple thresholds, up to one per VM 140. In one embodiment, the thresholds may be constant, predetermined, or programmed.

[0041] Scheduler 130 can use various dynamic limiting techniques to idle vCPU 135 allocated to noisy VM 140. When vCPU 135 is limited by forcing vCPU 135 into an idle state or memory access rate limit, vCPU 135 suspends code execution of VM 140 and retains its execution state. In one embodiment, limiting stops when the limited VM 140 has a memory access rate 142 that does not exceed a threshold 143.

[0042] Figure 1D A flowchart illustrating a method 150 for performing memory bandwidth limiting on a VM 140 according to an embodiment is shown. Each block of the method 150 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be implemented by a processor executing instructions stored in memory. The method can also be implemented as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or hosting service (independently or in conjunction with another hosting service) or a plug-in to another product, to name a few. Furthermore, method 150 is provided by way of example for... Figure 1A The system described herein is described in this way. However, in addition or alternatively, the method may be performed by any kind of system, or any combination of systems (including, but not limited to, those systems described herein). Furthermore, those skilled in the art will understand that any system performing method 150 is within the scope and spirit of the embodiments of this disclosure.

[0043] In step 155, scheduler 130 calculates a first value corresponding to the execution of a first VM 140 among a plurality of VMs 140 based on memory access metrics, wherein the plurality of VMs 140 are configured to share the processing resources of the computing system according to time slices 148 allocated among the plurality of VMs 140. In one embodiment, the computing system includes hardware 105. In another embodiment, the computing system includes hardware 105 and a hypervisor / host operating system 125.

[0044] In one embodiment, when a first VM 140 is allowed to run a workload with a higher system memory access rate than the workloads running on other VMs 140, the first VM 140 can reduce the system memory access latency of the other VMs 140. In a steady state, all VMs 140 can have different rates driving system memory access. When the first VM 140 begins to increase its rate, less system memory bandwidth becomes available for the other VMs 140, potentially increasing the system memory access latency of the other VMs 140. In cloud streaming scenarios, reduced system memory bandwidth or increased system memory access latency may manifest as blurring and blocky artifacts in video frames, a decrease in the number of frames rendered per second, and / or stuttering during gameplay running on other VMs 140.

[0045] In step 158, scheduler 130 detects that a first value exceeds threshold 143. In step 160, based on the first value, scheduler 130 determines the idle duration for which the resources of the first vCPU 135 allocated to the first VM 140 are restricted by placing the first vCPU 135 in an idle state during a first set of time slices 148 allocated to the first VM 140. In one embodiment, the idle duration is applied within at least one time slice. In one embodiment, the first VM 140 includes VM 140-1 and the first set of time slices 148 includes time slice 148-1. In one embodiment, all vCPUs 135 allocated to the first VM 140 are restricted by placing them in an idle state during the first set of time slices 148 allocated to the first VM 140. In one embodiment, scheduler 130 includes a neural network trained to predict the optimal duration for keeping vCPUs idle based on the configuration of vCPUs or VMs in the system and one or more memory access metrics, in order to provide sufficient access to memory for workloads using the same shared resource pool.

[0046] In step 165, during the execution of programming instructions for the first VM 140-1, the first vCPU 135 is placed in an idle state for the idle duration of at least one time slice in the first set of time slices 148-1. In one embodiment, the idle duration is applied to more than one time slice in the first set of time slices 148-1. In one embodiment, the duty cycle is determined based on the idle duration of time slice 148-1. In one embodiment, the duty cycle is defined as a percentage of the time slice. In one embodiment, the scheduler 130 places the vCPU in an idle state during the idle duration and enables the vCPU to process after the idle duration. The duty cycle is applied to at least one time slice in time slice 148-1. In one embodiment, the idle duration is applied within each of the N time slices included in time slice 148-1, where N is an integer greater than 1. In one embodiment, the idle duration is determined by the duty cycle applied across the N time slices within time slice 148-1. In one embodiment, each time slice in time slice 148-1 is subdivided into M intervals, and the idle duration is applied to each of the M intervals, where M is an integer greater than 1.

[0047] Figure 2A The illustration shows an example duty cycle of time slice 248 according to some embodiments of the present disclosure. The duty cycle is determined based on the idle duration. Figure 2A As shown, the duty cycle is 60%, and time slice 248 is subdivided into active partition 205 (40% of time slice 248) and idle partition 210 (60% of time slice 248).

[0048] Figure 2B An example diagram illustrating the limitation of memory access rate 252 according to some embodiments of the present disclosure is shown. Figure 1B The memory access rate 142 shown can be limited by the scheduler 130 to produce a limited memory access rate 252. Although only a single time slice 248 is shown, the scheduler 130 allocates multiple time slices 248 to each VM 140. The idle duration 220 is determined by the scheduler 130 and used to control the duty cycle.

[0049] A 40% duty cycle can be applied within each time slice 248 to generate an idle duration 220 for the noisy VM 140. The vCPU 110 allocated to the noisy VM 140 is active for 40% of the period corresponding to the active partition 205 in each time slice 248. The remaining 60% (60%) of the period corresponding to the idle partition 210 in each time slice 248 is idle. The total memory access rate 252 of time slice 248 is reduced compared to the memory access rate 142. Figure 2BAs shown, the restricted memory access rate 252 exceeds the threshold 255 during active partition 205 and does not exceed the threshold 255 during idle partition 210.

[0050] Figure 2C The illustration shows example duty cycles applied across time slices according to some embodiments of the present disclosure. Instead of applying a duty cycle within each time slice 248, a duty cycle can be applied over a period of N time slices 248, such that vCPU 110 allocated to the noisy VM 140 is idle for 60% of the N time slices 248 and vCPU 110 allocated to the noisy VM 140 is active for 40% of the N time slices 248. Figure 2C As shown, N=5 and vCPU110 assigned to the noisy VM 140 is idle for three of the five time slices 248. The idle duration 225 is equal to the idle duration 220 scaled by N.

[0051] Figure 2D A flowchart illustrating another method 230 for performing memory bandwidth limiting on VM 140 according to an embodiment is shown. Method 230 includes steps 155, 158, 160, and 165 of method 150. Steps 155, 158, 160, and 165 are performed as previously described. In step 232, scheduler 130 calculates a first value to update the first value after limiting. In step 235, scheduler 130 determines whether performance data or performance values ​​exceed corresponding thresholds based on one or more metrics.

[0052] If, in step 235, scheduler 130 determines that the performance data or performance value exceeds a threshold, then in step 240, scheduler 130 determines an idle duration. Scheduler 130 may increase the idle duration to control the memory access bandwidth of the first VM 140. In one embodiment, scheduler 130 keeps the idle duration constant until the performance data or performance value decreases to be equal to or below the threshold. In another embodiment, scheduler 130 increases the idle duration until the performance data or performance value decreases below the threshold. In one embodiment, the idle duration may vary for each subsequent time slice 248 allocated to the first VM 140, incrementally increasing or decreasing the duty cycle based on the memory access rate measured for a previous time slice 248. The increments may be equal intervals (e.g., 100%, 90%, 80%, ... 10%) or non-linear (e.g., 100%, 63%, 40%, 25%, 16%, 10%). In step 240, scheduler 130 applies the idle duration to control the memory access bandwidth of the first VM 140.

[0053] If, in step 235, the scheduler 130 determines that the performance data or performance value does not exceed a threshold, then in step 245, the scheduler 130 sets the idle duration of the first VM 140 to zero. In one embodiment, the monitor 132 measures the response time from the start of limiting for the noisy VM 140 until the measured performance data or performance value falls below the threshold. In another embodiment, the monitor 132 measures the response time from the first detection of a noisy VM 140 (where the measured performance data or performance value exceeds the threshold) until the measured performance data or performance value falls below the threshold in response to the limit. The time-slice controller 136 can use the response time to determine the idle duration.

[0054] Figure 3A An example of a time slice 348 subdivided into multiple intervals 305 according to some embodiments of the present disclosure is illustrated. In one embodiment, the time slice 348 is divided into M intervals 305-1 to 305-5, and a duty cycle is applied to each interval 305 within the time slice 348. Each smaller interval 305 is the time slice duration divided by M. The value of M can be predetermined, constant, or programmed.

[0055] Figure 3B Another example diagram illustrating the limitation of memory access rate 352 according to some embodiments of the present disclosure is shown. Dividing time slice 348 into M intervals results in a reduced allocation 350 of M smaller idle durations. As... Figure 3B As shown, M=5 and a 50% duty cycle is applied to each of the five intervals 305-1 to 305-5 in time slice 348. For Figure 2A The time slice 248 shown has a duty cycle of 40% and M = 1. In another embodiment, the duty cycle is applied over a period of time (N time slices) allocated to the noisy VM 140, so that the vCPU 110 allocated to the noisy VM 140 is active for N*duty cycle time slices and idle for the remaining time slices within that period.

[0056] Figure 3C Another example diagram illustrating the limiting of memory access rate 362 according to some embodiments of the present disclosure is shown. Time slice 348 is divided into M smaller idle time intervals, wherein a duty cycle is applied to each idle time interval. Figure 3C As shown, M=5 and a 40% duty cycle is applied to each of the five intervals 305-1 to 305-5 in time slice 348 to produce a reduced allocation 360. Compared to the restricted memory access rate 352, the restricted memory access rate 362 is further reduced as expected due to the increased idle duration.

[0057] Applying the duty cycle to each interval within a time slice provides fine-grained constraints compared to applying the duty cycle to a time period comprising N time slices. Applying the duty cycle within each time slice provides a constraint frequency between fine-grained interval-based constraints and coarse-grained time period-based constraints.

[0058] Scheduler 130 provides a software solution for real-time monitoring of performance data 134, such as memory access rates, and dynamically limits noisy VMs 140 based on performance data values ​​138. Scheduler 130 can implement various limiting techniques to provide fine-grained or coarse-grained limits on time slices allocated to noisy VMs 140. Idle duration is determined and applied as a duty cycle within each time slice or across multiple time slices. Compared to traditional systems, the software solution for dynamic limiting is portable across CPU families and does not require specialized server-level CPU functionality or limiting system configurations. Memory access limiting can be used to reduce memory access shortages and maintain VM isolation principles in cloud computing.

[0059] Parallel processing architecture

[0060] Figure 4 The illustration shows a parallel processing unit (PPU) 400 according to one embodiment. The PPU 400 can be used to implement the CPU 110 and / or scheduler 130 of the cloud computing system 100.

[0061] In one embodiment, PPU 400 is a multi-threaded processor implemented on one or more integrated circuit devices. PPU 400 is a latency-hiding architecture designed to process many threads in parallel. A thread (e.g., an execution thread) is an instantiation of an instruction set configured to be executed by PPU 400. In one embodiment, PPU 400 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device. In other embodiments, PPU 400 may be used to perform general-purpose computing. While an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that such a processor is illustrated for illustrative purposes only, and any processor may be employed to complement and / or replace this processor.

[0062] One or more PPU 400s can be configured to accelerate thousands of high-performance computing (HPC), data center, cloud computing, and machine learning applications. PPU 400s can be configured to accelerate numerous deep learning systems and applications used in autonomous vehicles, simulations, computational graphics such as ray or path tracing, deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations, among others.

[0063] like Figure 4 As shown, PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a job allocation unit 425, a hub 430, a crossbar (Xbar) 470, one or more general purpose processing clusters (GPCs) 450, and one or more memory partitioning units 480. PPU 400 can be connected to a host processor or other PPU 400 via one or more high-speed NVLink 410 interconnects. PPU 400 can be connected to a host processor or other peripheral devices via interconnect 402. PPU 400 can also be connected to local memory 404, which includes multiple memory devices. In one embodiment, local memory may include multiple dynamic random access memory (DRAM) devices. The DRAM devices may be configured as a high-bandwidth memory (HBM) subsystem, wherein multiple DRAM dies are stacked within each device.

[0064] The NVLink 410 interconnect enables the system to expand and include one or more PPUs 400 in conjunction with one or more CPUs, supporting cache coherency between the PPUs 400 and the CPU, as well as the CPU controller. Data and / or commands can be sent from or from the NVLink 410 to other units of the PPU 400 via hub 430, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 5A A more detailed description of the NVLink 410.

[0065] I / O unit 405 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 402. I / O unit 405 may communicate directly with the host processor via interconnect 402, or via one or more intermediate devices such as memory bridges. In one embodiment, I / O unit 405 may communicate with one or more other processors, such as one or more PPUs 400, via interconnect 402. In one embodiment, I / O unit 405 implements a Peripheral Component Interconnect High Speed ​​(PCIe) interface for communication via a PCIe bus, and interconnect 402 is a PCIe bus. In alternative embodiments, I / O unit 405 may implement other types of known interfaces for communication with external devices.

[0066] I / O unit 405 decodes data packets received via interconnect 402. In one embodiment, the data packets represent commands configured to cause PPU 400 to perform various operations. I / O unit 405 transmits the decoded commands to various other units of PPU 400 that these commands may specify. For example, some commands may be transmitted to front-end unit 415. Other commands may be transmitted to hub 430 or other units of PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 405 is configured to route communication between and among the various logical units of PPU 400.

[0067] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. The workload may include instructions and data to be processed by those instructions. The buffer is an area of ​​memory accessible (e.g., read / write) by both the host processor and the PPU 400. For example, I / O unit 405 may be configured to access a buffer in system memory connected to interconnect 402 via a memory request transmitted through interconnect 402. In one embodiment, the host processor writes a command stream to the buffer and then transmits a pointer to the start of the command stream back to the PPU 400. Front-end unit 415 receives pointers to one or more command streams. Front-end unit 415 manages the one or more streams, reads commands from these streams, and forwards the commands to the respective units of the PPU 400.

[0068] Front-end unit 415 is coupled to scheduler unit 420, which configures various GPCs 450 to process tasks defined by the one or more streams. Scheduler unit 420 is configured to track status information related to the various tasks managed by scheduler unit 420. Status can indicate which GPC 450 a task is assigned to, whether the task is active or inactive, the priority associated with the task, etc. Scheduler unit 420 manages the execution of multiple tasks on the one or more GPCs 450.

[0069] Scheduler unit 420 is coupled to job allocation unit 425, which is configured to dispatch tasks for execution on GPC 450. Job allocation unit 425 can track several scheduled tasks received from scheduler unit 420. In one embodiment, job allocation unit 425 manages a pending task pool and an active task pool for each GPC 450. When GPC 450 completes the execution of a task, the task is evicted from the active task pool of GPC 450, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 450. If an active task on GPC 450 is idle, for example, while waiting for data dependencies to be resolved, then the active task can be evicted from GPC 450 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 450.

[0070] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API), enabling one or more applications executing on the host processor to be scheduled for operations to be performed on the PPU 400. In one embodiment, multiple computing applications are executed concurrently by the PPU 400, and the PPU 400 provides isolation, Quality of Service (QoS), and independent address spaces for the multiple computing applications. Applications may generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 400. The driver kernel outputs the tasks to one or more streams being processed by the PPU 400. Each task may include one or more associated thread groups, referred to herein as warps. In one embodiment, a warp includes 32 associated threads that can execute in parallel. Cooperative threads may refer to multiple threads that include instructions for performing tasks and can exchange data via shared memory. These tasks may be assigned to one or more processing units within the GPC 450, and instructions are scheduled for execution by at least one warp.

[0071] The work allocation unit 425 communicates with one or more GPCs 450 via an XBar 470. The XBar 470 is an interconnect network that couples a plurality of units of the PPU 400 to other units of the PPU 400. For example, the XBar 470 can be configured to couple the work allocation unit 425 to a specific GPC 450. Although not explicitly shown, one or more other units of the PPU 400 may also be connected to the XBar 470 via a hub 430.

[0072] Tasks are managed by scheduler unit 420 and dispatched to GPC 450 by job allocation unit 425. GPC 450 is configured to process tasks and generate results. Results can be consumed by other tasks within GPC 450, routed to different GPC 450 via XBar 470, or stored in memory 404. Results can be written to memory 404 via memory partitioning unit 480, which implements a memory interface for reading data from and writing data to memory 404. Results can be transferred to another PPU 400 or CPU via NVLink 410. In one embodiment, PPU 400 includes U number of memory partitioning units 480, equal to the number of individual and distinct memory devices coupled to memory 404 of PPU 400. Each GPC 450 may include a memory management unit to provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit provides one or more translation back buffers (TLBs) for performing virtual address to physical address translation in memory 404.

[0073] In one embodiment, memory partitioning unit 480 includes a raster operation (ROP) unit, a secondary (L2) cache, and a memory interface coupled to memory 404. The memory interface can implement 32-bit, 64-bit, 128-bit, or 1024-bit data buses for high-speed data transfer. PPU 400 can connect to up to Y memory devices, such as high-bandwidth memory stacks or graphics dual data rate, version 5, synchronous dynamic random access memory, or other types of persistent storage devices. In one embodiment, the memory interface implements an HBM2 memory interface, and Y equals half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 400, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y equals 4, wherein each HBM2 stack includes two 129-bit channels per die, for a total of eight channels, and the data bus width is 1024 bits.

[0074] In one embodiment, memory 404 supports single error correction double detection (SECDED) error correction codes (ECC) to protect data. ECC provides a high level of reliability for computing applications sensitive to data corruption. Reliability is particularly important where the PPU 400 is handling very large datasets and / or in large-scale cluster computing environments where applications run for extended periods.

[0075] In one embodiment, PPU 400 implements a multi-level memory hierarchy. In one embodiment, memory partitioning unit 480 supports unified memory that provides a single, unified virtual address space for the CPU and PPU 400 memory, allowing data sharing between virtual memory systems. In one embodiment, the frequency with which PPU 400 accesses memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 400 that is accessing these pages more frequently. In one embodiment, NVLink 410 supports address translation services, allowing PPU 400 to directly access the CPU's page table and providing PPU 400 with full access to the CPU's memory.

[0076] In one embodiment, the replication engine transfers data between multiple PPUs 400 or between a PPU 400 and a CPU. The replication engine can generate page faults for addresses that are not mapped to a page table. The memory partitioning unit 480 can then repair the page faults, mapping these addresses to the page table, after which the replication engine can perform the transfer. In conventional systems, memory is fixed (e.g., non-pageable) for multiple replication engine operations across multiple processors, significantly reducing available storage. In the event of a hardware page fault, addresses can be passed to the replication engine without concern for whether memory pages reside, and the replication process is transparent.

[0077] Data from memory 404 or other system memory can be fetched by memory partitioning unit 480 and stored in an on-chip L2 cache 460 shared among the various GPCs 450. As shown, each memory partitioning unit 480 includes a portion of the L2 cache associated with the corresponding memory 404. Low-level caches can then be implemented in various units within the GPC 450. For example, each processing unit within the GPC 450 can implement a Level 1 (L1) cache. The L1 cache is a private memory dedicated to a specific processing unit. The L2 cache 460 is coupled to memory interface 470 and XBar 470, and data from the L2 cache can be fetched and stored in each of the L1 caches for processing.

[0078] In one embodiment, the processing unit within each GPC 450 implements a SIMD (Single Instruction Multiple Data) architecture, where each thread in a thread group (e.g., a thread bundle) is configured to process a different data set based on the same set of instructions. All threads in the thread group execute the same instructions. In another embodiment, the processing unit implements a SIMT (Single Instruction Multiple Thread) architecture, where each thread in the thread group is configured to process a different data set based on the same set of instructions, but individual threads in the thread group are allowed to diverge during execution. In one embodiment, maintaining a program counter, call stack, and execution state for each thread bundle allows for concurrency between thread bundles and serial execution within a thread bundle when threads diverge. In another embodiment, maintaining a program counter, call stack, and execution state for each individual thread allows for equal concurrency among all threads within and between thread bundles. When maintaining an execution state for each individual thread, threads executing the same instructions can aggregate and execute in parallel for maximum efficiency.

[0079] Cooperative groups are a programming model for organizing groups of communicating threads. They allow developers to express the granularity at which threads are communicating, enabling richer and more efficient expressions of parallel decomposition. The Cooperative Startup API supports synchronization between blocks of threads executing parallel algorithms. Conventional programming models provide a single, simple construct for synchronizing cooperative threads: a barrier across all threads in a thread block (e.g., the `syncthreads()` function). However, programmers often prefer to define thread groups smaller than thread blocks in the form of a collective group-wide function interface, and synchronize within the defined group to allow for greater performance, design flexibility, and software reuse.

[0080] Collaboration groups enable programmers to explicitly define thread groups (as small as a single thread) at the sub-block and multi-block granularity, and perform collective operations such as synchronization on threads within the collaboration group. This programming model supports clean composition across software boundaries, allowing libraries and utility functions to be safely synchronized within their local contexts without having to make assumptions about aggregation. The collaboration group primitive allows for the implementation of new cooperative parallelism patterns, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire mesh of thread blocks.

[0081] Each processing unit comprises a large number (e.g., 128, etc.) of different processing cores (e.g., functional units), which may be fully pipelined, single-precision, double-precision, and / or mixed-precision, and include floating-point arithmetic logic units and integer arithmetic logic units. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the core comprises 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0082] Tensor cores are configured to perform matrix operations. Specifically, tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations used for neural network training and inference. In one embodiment, each tensor core operates on a 4x4 matrix and performs matrix multiplication and accumulation operations, D = A × B + C, where A, B, C, and D are 4x4 matrices.

[0083] In one embodiment, matrix multiplication inputs A and B are 16-bit floating-point matrices, while accumulation matrices C and D can be either 16-bit or 32-bit floating-point matrices. Tensor Core utilizes 32-bit floating-point accumulation on the 16-bit floating-point input data. 16-bit floating-point multiplication requires 64 operations and results in a full-precision product, which is then accumulated with other intermediate multiplications using 32-bit floating-point addition for 4x4x4 matrix multiplication. In practice, Tensor Core is used to perform operations on much larger two-dimensional or higher-dimensional matrices composed of these smaller elements. APIs such as the CUDA9 C++ API expose specialized matrix loading, matrix multiplication and accumulation, and matrix storage operations for efficient use with Tensor Core from CUDA-C++ programs. At the CUDA level, the thread bundle-level interface takes a 16x16 matrix spanning all 32 threads of the thread bundle.

[0084] Each processing unit may also include M Special Function Units (SFUs) that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, an SFU may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, an SFU may include a texture unit configured to perform texture map filtering operations. In one embodiment, a texture unit is configured to load texture maps (e.g., a 2D texture array) from memory 404 and sample these texture maps to produce sampled texture values ​​for use by a shader program executed by the processing unit. In one embodiment, the texture maps are stored in shared memory that may include or contain an L1 cache. The texture units use mip maps (e.g., texture maps with varying levels of detail) to implement texture operations such as filtering. In one embodiment, each processing unit includes two texture units.

[0085] Each processing unit also includes N Load Memory Units (LSUs) that implement load and store operations between shared memory and the register file. Each processing unit includes an interconnect network that connects each of the cores to the register file and connects the LSUs to the register file and the shared memory. In one embodiment, the interconnect network is a cross switch that can be configured to connect any of the cores to any register in the register file and connect the LSUs to memory locations in the register file and the shared memory.

[0086] Shared memory is an on-chip memory array that allows data storage and communication between processing units and between threads within a processing unit. In one embodiment, the shared memory includes 128KB of storage capacity and is located on the path from each of the processing units to memory partition unit 480. The shared memory can be used for caching reads and writes. One or more of the shared memory, L1 cache, L2 cache, and memory 404 serve as a backup cache.

[0087] Combining data caching and shared memory functionality into a single memory block provides optimal overall performance for both types of memory access. This capacity can be used as a cache by programs that do not use shared memory. For example, if shared memory is configured to use half its capacity, then texture and load / store operations can use the remaining capacity. Integration into shared memory allows it to function as a high-throughput conduit for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0088] When configured for general-purpose parallel computing, a simpler configuration can be used compared to graphics computing. Specifically, bypassing fixed-function graphics processing units (GPUs) creates a much simpler programming model. In this general-purpose parallel computing configuration, the work allocation unit 425 directly dispatches and assigns thread blocks to processing units within the GPC 450. Threads execute the same program using a unique thread ID during computation to ensure that each thread uses the executor program and the processing unit performing the computation, the shared memory for communication between threads, and the LSU (Local Subsystem for Memory) for reading and writing global memory via the shared memory and memory partitioning unit 480 to generate unique results. When configured for general-purpose parallel computing, processing units can also write commands, which the scheduler unit 420 can use to start new work on the processing unit.

[0089] Each of the PPUs 430 may include one or more processing cores and / or components thereof, such as a Tensor Core (TC), a Tensor Processing Unit (TPU), a Pixel Vision Core (PVC), a Vision Processing Unit (VPU), a Graphics Processing Cluster (GPC), a Texture Processing Cluster (TPC), a Streaming Multiprocessor (SM), a Tree Traversal Unit (TTU), an Artificial Intelligence Accelerator (AIA), a Deep Learning Accelerator (DLA), an Arithmetic Logic Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating Point Unit (FPU), Input / Output (I / O) elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect High Speed ​​(PCIe) elements and / or the like, and / or be configured to perform its functions.

[0090] The PPU 400 may be included in desktop computers, laptop computers, tablet computers, servers, supercomputers, smartphones (e.g., wireless, handheld devices), personal digital assistants (PDAs), digital cameras, vehicles, head-mounted displays, handheld electronic devices, and the like. In one embodiment, the PPU 400 is implemented on a single semiconductor substrate. In another embodiment, the PPU 400 is included in a system-on-a-chip (SoC) along with one or more other devices such as an additional PPU 400, memory 404, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), and the like.

[0091] In one embodiment, the PPU 400 may be included on a graphics card that includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on a desktop computer's motherboard. In yet another embodiment, the PPU 400 may be an integrated graphics processing unit (iGPU) or a parallel processor included in a chipset on the motherboard.

[0092] Exemplary computing system

[0093] As developers expose and leverage more parallelism in applications such as artificial intelligence computing, systems with multiple GPUs and CPUs are being used across various industries. High-performance GPU-accelerated systems with tens to thousands of compute nodes are being deployed in data centers, research facilities, and supercomputers to solve increasingly complex problems. With the increasing number of processing devices within high-performance systems, communication and data transmission mechanisms need to be scaled to support the increased bandwidth.

[0094] Figure 5A According to one embodiment, the use Figure 4 A conceptual diagram of a processing system 500 implemented by a PPU 400. An exemplary system 565 can be configured to be implemented in... Figure 1D and 2DMethods 150 and / or 230 are shown in the diagram. The processing system 500 includes a CPU 530, a switch 510, multiple PPUs 400, and various memories 404.

[0095] The NVLink 410 provides a high-speed communication link between each of the PPUs 400. Although Figure 5A The diagram illustrates a specific number of NVLink 410 and interconnect 402 connections, but the number of connections to each PPU 400 and CPU 530 can vary. Switch 510 forms an interface between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLink 410 can reside on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols to interface between various connections and / or links.

[0096] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and CPU 530, and switch 510 forms an interface between interconnect 402 and each PPU 400. PPU 400, memory 404, and interconnect 402 may reside on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 and CPU 530, and switch 510 uses NVLink 410 to form an interface between each PPU 400 to provide one or more high-speed communication links between PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between PPU 400 and CPU 530 via switch 510. In yet another embodiment (not shown), interconnect 402 directly provides one or more communication links between each PPU 400. One or more of the NVLink 410 high-speed communication links can be implemented as physical NVLink interconnects or on-chip or die-on interconnects using the same protocol as the NVLink 410.

[0097] In the context of this specification, a single semiconductor platform can refer to a single semiconductor-based integrated circuit fabricated on a bare die or chip. It should be noted that the term "single semiconductor platform" can also refer to a multi-chip module with increased connectivity, simulating on-chip operation and representing a significant improvement over conventional bus implementations. Of course, the various circuits or devices can also be located individually within the semiconductor platform or in various combinations thereof, as desired by the user. Alternatively, the parallel processing module 525 can be implemented as a circuit board substrate, and each PPU 400 and / or memory 404 can be a packaged device. In one embodiment, the CPU 530, switch 510, and parallel processing module 525 reside on a single semiconductor platform.

[0098] In one embodiment, the signaling rate of each NVLink 410 is 20-25 gigabits per second, and each PPU400 includes six NVLink 410 interfaces (e.g., Figure 5B As shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 gigabits per second in each direction, and the six links provide 400 gigabits per second. NVLink 410 can be used as follows: Figure 5A It is used exclusively for PPU-to-PPU communication, or for a combination of PPU-to-PPU and PPU-to-CPU when the CPU 530 also includes one or more NVLink 410 interfaces.

[0099] In one embodiment, NVLink 410 allows direct load / store / atomic access from CPU 530 to memory 404 of each PPU 400. In one embodiment, NVLink 410 supports coherent operation, allowing data read from memory 404 to be stored in the cache hierarchy of CPU 530, reducing cache access latency of CPU 530. In one embodiment, NVLink 410 includes support for Address Translation Service (ATS), allowing PPU 400 to directly access page tables within CPU 530. One or more of NVLink 410 may also be configured to operate in low-power mode.

[0100] Figure 5B An exemplary system 565 is illustrated, in which various architectures and / or functions of various prior embodiments can be implemented. The exemplary system 565 can be configured to implement, respectively... Figure 1D and Figure 2D Methods 150 and / or 230 are shown in the diagram.

[0101] As shown in the figure, a system 565 is provided, which includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 may directly or indirectly couple one or more of the following devices: main memory 540, network interface 535, CPU 530, display device 545, input device 560, switch 510, and parallel processing system 525. The communication bus 575 may be implemented using any suitable protocol and may represent one or more links or buses, such as address bus, data bus, control bus, or combinations thereof. The communication bus 575 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect High Speed ​​(PCIe) bus, HyperTransport, and / or another type of bus or link. In some embodiments, direct connections exist between components. As an example, CPU 530 may be directly connected to main memory 540. Furthermore, CPU 530 may be directly connected to parallel processing system 525. In cases where there is a direct or point-to-point connection between components, the communication bus 575 may include a PCIe link that implements the connection. In these examples, the PCI bus need not be included in the system 565.

[0102] Although using lines Figure 5B The different blocks are shown connected via a communication bus 575, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component such as a display device 545 can be considered an I / O component, such as an input device 560 (e.g., if the display is a touchscreen). As another example, the CPU 530 and / or the parallel processing system 525 may include memory (e.g., main memory 540 may represent storage devices other than the parallel processing system 525, the CPU 530, and / or other components). In other words, Figure 5B The term "computing device" is merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as all of these are expected to fall under [the relevant category]. Figure 5B Within the scope of computing devices.

[0103] System 565 also includes main memory 540. Control logic (software) and data are stored in main memory 540, which can take the form of a variety of computer-readable media. Computer-readable media can be any available medium that can be accessed by system 565. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. For example and without limitation, computer-readable media can include computer storage media and communication media.

[0104] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, main memory 540 may store computer-readable instructions such as an operating system (e.g., representing programs and / or program elements). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic cartridges, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that may be used to store desired information and that can be accessed by system 565. When used herein, computer storage media does not include the signal itself.

[0105] Computer storage media may contain computer-readable instructions, data structures, program modules, or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information transport medium. The term "modulated data signal" may refer to a signal whose characteristics are set or altered in a manner that encodes information into that signal. For example and without limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0106] When executed, the computer program enables system 565 to perform various functions. CPU 530 may be configured to execute at least some of the computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Each of CPUs 530 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing numerous software threads simultaneously. Depending on the type of system 565 implemented, CPU 530 may include any type of processor and may include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of system 565, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, system 565 may include one or more CPUs 530.

[0107] In addition to or alternatively to CPU 530, parallel processing module 525 may be configured to execute at least some of the computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Parallel processing module 525 may be used by system 565 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, parallel processing module 525 may be used for general-purpose computing on a GPU (GPGPU). In embodiments, CPU 530 and / or parallel processing module 525 may execute any combination of the methods, processes, and / or portions thereof, discretely or jointly.

[0108] System 565 also includes input device 560, parallel processing system 525, and display device 545. Display device 545 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. Display device 545 may receive data from other components (e.g., parallel processing system 525, CPU 530, etc.) and output that data (e.g., images, video, sound, etc.).

[0109] Network interface 535 enables system 565 to be logically coupled to other devices, including input device 560, display device 545, and / or other components, some of which may be embedded (e.g., integrated into) system 565. Illustrative input device 560 includes microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. Input device 560 can provide a natural user interface (NUI) that processes user-generated air gestures, voice, or other physiological input. In some instances, input can be transmitted to appropriate network elements for further processing. NUI can implement voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head-eye tracking, and touch recognition associated with the display of system 565 (described in more detail below). System 565 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. In addition, system 565 may include an accelerometer or gyroscope that allows motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by system 565 to render immersive augmented reality or virtual reality.

[0110] Furthermore, system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) via network interface 535 for communication purposes. System 565 can be included in a distributed network and / or cloud computing environment.

[0111] Network interface 535 may include one or more receivers, transmitters, and / or transceivers that enable system 565 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communications. Network interface 535 may include components and functions that allow communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0112] System 565 may also include an auxiliary storage device (not shown). The auxiliary storage device 610 includes, for example, a hard disk drive and / or a removable storage drive representing a floppy disk drive, magnetic tape drive, compact disc drive, digital versatile disc (DVD) drive, recording device, Universal Serial Bus (USB) flash memory. The removable storage drive reads from and / or writes to the removable storage unit in a well-known manner. System 565 may also include a hard-wired power supply, a battery power supply, or a combination thereof (not shown). This power supply can supply power to System 565 to enable the components of System 565 to operate.

[0113] Each of the aforementioned modules and / or devices may even reside on a single semiconductor platform to form system 565. Alternatively, various different modules may be placed individually or located in various combinations of semiconductor platforms as desired by the user. Although various different embodiments have been described above, it should be understood that they are given by way of example only and without limitation. Therefore, the breadth and scope of preferred embodiments should not be limited to any of the exemplary embodiments described above, but should be defined only by the following claims and their equivalents.

[0114] Machine Learning

[0115] Deep neural networks (DNNs) developed on processors such as the PPU 400 have been used in a wide variety of use cases, from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technique that models the neural learning process of the human brain, which continuously learns, becomes smarter, and delivers more accurate results faster over time. Just as a child is initially taught by adults to correctly identify and classify various shapes, eventually becoming able to identify shapes without any guidance, a deep learning or neural learning system needs to be trained in object recognition and classification so that it becomes smarter and more efficient at identifying basic objects, occluded objects, and so on, while also attaching context to objects.

[0116] At its simplest level, neurons in the human brain receive various inputs, assigning a level of importance to each of these inputs, and the output is passed to other neurons to make a response. Artificial neurons, or perceptrons, are the most basic model of neural networks. In one example, a perceptron can receive one or more inputs representing various features of objects that the perceptron is being trained to recognize and classify, and each of these features is assigned a weight based on its importance in defining the shape of the object.

[0117] Deep neural network (DNN) models consist of multiple layers of numerous connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) that can be trained on massive amounts of input data to solve complex problems quickly and with high accuracy. In one example, the first layer of a DNN model breaks down an input image of a car into different segments and searches for basic patterns such as lines and angles. The second layer assembles these lines to find higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final few layers generate labels for the input image that identify the model of a specific car brand.

[0118] Once trained, a DNN can be deployed and used to identify and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include identifying handwritten digits on a check deposited into an ATM, identifying images of friends in a photograph, delivering movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human language in real time.

[0119] During training, data flows through the DNN in the forward propagation phase until a prediction is produced indicating the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and the weights are adjusted for each feature during the backpropagation phase until the DNN correctly labels the input as well as other inputs in the training dataset. Training complex neural networks requires significant parallel computing power, including floating-point multiplication and addition supported by a PPU400. Inference is less computationally intensive than training and is a latency-sensitive process where the trained neural network is applied to new inputs it has not seen before for tasks such as image classification, sentiment detection, label recommendation, language recognition and translation, and typically infers new information.

[0120] Neural networks rely heavily on matrix operations, and complex, multi-layered networks require massive amounts of floating-point performance and bandwidth for both efficiency and speed. Leveraging thousands of processing cores optimized for matrix operations and delivering tens to hundreds of TFLOPS of performance, the PPU 400 is a computing platform capable of providing the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0121] Furthermore, images generated using one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for recognizing real-world objects and environments. Such images can include driveways, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or scenes of real-world environments. Such images can be used to train, test, or certify DNNs used in machines or robots to manipulate, process, or modify real-world physical objects. Additionally, such images can be used to train, test, or certify DNNs used in autonomous vehicles to navigate and move vehicles in the real world. Furthermore, images generated using one or more of the techniques disclosed herein can be used to communicate information to users of such machines, robots, and vehicles.

[0122] Graphics processing pipeline

[0123] In one embodiment, PPU 400 includes a graphics processing unit (GPU). PPU 400 is configured to receive commands specifying a shader program for processing graphics data. Graphics data can be defined as a set of primitives such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying the number of vertices used for that primitive (e.g., in a model-space coordinate system) and attributes associated with each vertex of that primitive. PPU 400 can be configured to process graphics primitives to generate framebuffers (e.g., pixel data for each of the pixels in a display).

[0124] The application writes model data (such as attributes and vertex sets) for the scene into memory such as system memory or memory 404. The model data defines each of the objects that may be visible on the display. The application then makes API calls to the driver kernel, requesting that the model data be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations that process the model data. These commands may reference different shader programs to be implemented on the SM 440 of the PPU 400, including one or more vertex shaders, shell shaders, domain shaders, geometry shaders, and pixel shaders. For example, one or more SM 440s may be configured to execute a vertex shader program that processes a number of vertices defined by the model data. In one embodiment, these different SM 440s may be configured to execute different shader programs concurrently. For example, a first subset of SM 440s may be configured to execute vertex shader programs, while a second subset of SM 440s may be configured to execute pixel shader programs. A first subset of the SM 440 processes vertex data to produce processed vertex data and writes it to L2 cache 460 and / or memory 404. After the processed vertex data is rasterized (e.g., transformed from 3D data to 2D data in screen space) to produce fragment data, a second subset of the SM 440 executes pixel shaders to produce processed fragment data, which is then mixed with other processed fragment data and written to the framebuffer in memory 404. The vertex shader and pixel shader programs can execute concurrently, pipelinedly processing different data from the same scene until all model data for that scene has been rendered to the framebuffer. The contents of the framebuffer are then transferred to the display controller for display on the display device.

[0125] Figure 6A According to one embodiment, by Figure 4 A conceptual diagram of a graphics processing pipeline 600 implemented by a PPU 400. The graphics processing pipeline 600 is an abstract flowchart of the processing steps implemented to generate 2D computer-generated images from 3D geometric data. It is well known that pipeline architectures can more efficiently perform long-latency operations by dividing operations into multiple stages, where the output of each stage is coupled to the input of the next successive stage. Therefore, the graphics processing pipeline 600 receives input data 601 passed from one stage of the graphics processing pipeline 600 to the next stage to generate output data 602. In one embodiment, the graphics processing pipeline 600 may represent a system... The graphics processing pipeline is defined by the API. As an option, the graphics processing pipeline 600 can be implemented within the context of the functionality and architecture of previous figures and / or one or more subsequent figures.

[0126] like Figure 6A As shown, the graphics processing pipeline 600 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 610, a vertex shading stage 620, a primitive assembly stage 630, a geometry shading stage 640, a viewport scale, cull, and clip (VSCC) stage 650, a rasterization stage 660, a fragment shading stage 670, and a raster operation stage 680. In one embodiment, input data 601 includes commands that configure processing units to implement the stages of the graphics processing pipeline 600 and configure geometric primitives (e.g., points, lines, triangles, quadrilaterals, triangular strips, or sectors, etc.) to be processed by these stages. Output data 602 may include pixel data (e.g., color data) that is copied to a framebuffer or other type of surface data structure in memory.

[0127] The data assembly stage 610 receives input data 601, which specifies vertex data for higher-order surfaces, primitives, etc. The data assembly stage 610 collects vertex data from a temporary storage device or queue, for example, by receiving a command from the host processor including a pointer to a buffer in memory and reading vertex data from that buffer. The vertex data is then transferred to the vertex shading stage 620 for processing.

[0128] The vertex shading stage 620 processes vertex data by performing a set of operations (e.g., a vertex shader or procedure) once for each vertex. A vertex can be specified, for example, as a 4-coordinate vector (e.g., color, texture coordinates, surface normals, etc.) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normals, etc.).<x,y,z,w> The vertex shading stage 620 can manipulate individual vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 620 performs operations on the vertex coordinates or other vertex attributes associated with a vertex. Such operations typically include lighting operations (e.g., modifying the vertex's color attributes) and transformation operations (e.g., modifying the vertex's coordinate space). For example, a vertex can be specified using coordinates in object coordinate space, which is transformed by multiplying the coordinates by a matrix that converts the coordinates from object coordinate space to world space or normalized-device-coordinate (NCD) space. The vertex shading stage 620 generates transformed vertex data that is then passed to the primitive assembly stage 630.

[0129] Primitive assembly stage 630 collects the vertices output by vertex shading stage 620 and groups the vertices into geometric primitives for processing by geometry shading stage 640. For example, primitive assembly stage 630 can be configured to group every three consecutive vertices into geometric primitives (e.g., triangles) for transmission to geometry shading stage 640. In some embodiments, specific vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). Primitive assembly stage 630 transmits geometric primitives (e.g., sets of associated vertices) to geometry shading stage 640.

[0130] The geometry shading stage 640 processes geometry primitives by performing a set of operations (e.g., geometry shaders or procedures) on them. Tessellation operations can generate one or more geometry primitives from each geometry primitive. In other words, the geometry shading stage 640 can subdivide each geometry primitive into a finer mesh of two or more geometry primitives for processing by the rest of the graphics processing pipeline 600. The geometry shading stage 640 then passes the geometry primitives to the viewport SCC stage 650.

[0131] In one embodiment, the graphics processing pipeline 600 can operate within a streaming multiprocessor, and the vertex shading stage 620, primitive assembly stage 630, geometry shading stage 640, fragment shading stage 670, and / or associated hardware / software can sequentially perform processing operations. In one embodiment, the viewport SCC stage 650 can utilize data once the sequential processing operations are complete. In one embodiment, the metadata processed by one or more stages in the graphics processing pipeline 600 can be written to a cache (e.g., L1 cache, vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 650 can access the data in the cache. In one embodiment, the viewport SCC stage 650 and the rasterization stage 660 are implemented as a fixed-function circuit system.

[0132] In the Viewport SCC stage 650, viewport scaling, culling, and clipping of geometry are performed. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of the viewer viewing the scene and defines the viewing frustum that surrounds the objects in the scene. The viewing frustum may include the viewing plane, the back plane, and four clipping planes. Any geometry completely outside the viewing frustum can be culled (e.g., discarded) because it will not contribute to the final rendered scene. Any geometry partially inside and partially outside the viewing frustum can be clipped (e.g., transformed into new geometry enclosed within the viewing frustum). Furthermore, each geometry can be scaled based on the depth of the viewing frustum. All potentially visible geometry is then passed to the rasterization stage 660.

[0133] Rasterization stage 660 converts 3D geometric primitives into 2D fragments (e.g., fragments usable for display, etc.). Rasterization stage 660 can be configured to use the vertices of the geometric primitives to establish a set of planar equations from which various properties can be interpolated. Rasterization stage 660 can also compute an overlay mask for multiple pixels, indicating whether one or more sample locations of the pixels occlude the geometric primitives. In one embodiment, a z-test can also be performed to determine whether the geometric primitives are occluded by other geometric primitives that have already been rasterized. Rasterization stage 660 generates fragment data (e.g., interpolated vertex properties associated with specific sample locations of each occluded pixel), which is then passed to fragment shading stage 670.

[0134] Fragment shading stage 670 processes fragment data by performing a set of operations (e.g., fragment shader or procedure) on each fragment. Fragment shading stage 670 may generate pixel data (e.g., color values) for the fragment by performing lighting operations or sampling a texture map using interpolated texture coordinates of the fragment. The pixel data generated by fragment shading stage 670 is then passed to raster operation stage 680.

[0135] The rasterization stage 680 can perform various operations on the pixel data, such as performing alpha tests, stencil tests, and mixing the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the rasterization stage 680 has completed processing the pixel data (e.g., output data 602), the pixel data can be written to a rendering target, such as a frame buffer, color buffer, etc.

[0136] It should be understood that, in addition to or in place of one or more of the aforementioned stages, one or more additional stages may be included in the graphics processing pipeline 600. Various implementations of the abstract graphics processing pipeline can implement different stages. Furthermore, in some embodiments, one or more of the aforementioned stages may be excluded from the graphics processing pipeline (such as geometry shading stage 640). Other types of graphics processing pipelines are contemplated within the scope of this disclosure. Additionally, any stage of the graphics processing pipeline 600 may be implemented by one or more dedicated hardware units within a graphics processor, such as the PPU 400. Other stages of the graphics processing pipeline 600 may be implemented by programmable hardware units, such as the SM440 of the PPU 400.

[0137] The graphics processing pipeline 600 can be implemented via an application executed by a host processor such as a CPU. In one embodiment, the device driver can implement an application programming interface (API), which defines various functions that can be utilized by the application to generate graphics data for display. The device driver is a software program that includes multiple instructions controlling the operation of the PPU 400. The API provides an abstraction for the programmer, allowing the programmer to utilize dedicated graphics hardware such as the PPU 400 to generate graphics data without requiring the programmer to utilize the specific instruction set of the PPU 400. The application can include API calls to the device driver routed to the PPU 400. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform operations at least in part by initiating operations on the PPU 400 using the input / output interface between the CPU and the PPU 400. In one embodiment, the device driver is configured to utilize the hardware of the PPU 400 to implement the graphics processing pipeline 600.

[0138] Various programs can be executed within the PPU 400 to implement different stages of the graphics processing pipeline 600. For example, a device driver can launch a kernel on the PPU 400 to execute the vertex shading stage 620 on one or more SM 440s. The device driver (or the initial kernel executed by the PPU 400) can also launch other kernels on the PPU 400 to execute other stages of the graphics processing pipeline 600, such as the geometry shading stage 640 and the fragment shading stage 670. Additionally, some stages of the graphics processing pipeline 600 can be implemented on fixed-function hardware such as a rasterizer or data assembler implemented within the PPU 400. It should be understood that the results from one kernel can be processed by one or more intermediate fixed-function hardware units before being processed by subsequent kernels on the SM 440s.

[0139] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device may be directly coupled to the system or processor that generates or renders the image. In other embodiments, the display device may be indirectly coupled to the system or processor, for example, via a network. Examples of such networks include the Internet, mobile telecommunications networks, Wi-Fi networks, and any other wired and / or wireless networking systems. When the display device is indirectly coupled, images generated by the system or processor can be streamed to the display device over the network. Such streaming allows, for example, video games or other applications that render images to execute in a server or data center, and the rendered images to be transmitted and displayed on one or more user devices (e.g., computers, video game consoles, smartphones, other mobile devices, etc.) physically separate from the server or data center. Therefore, the techniques disclosed herein can be applied to enhance streamed images and services that stream images, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0140] Example game streaming system

[0141] Figure 6B This is an example system schematic diagram of a game streaming system 605 according to some embodiments of the present disclosure. The game streaming system 605 can be used with... Figure 1A The cloud computing environment 100 is used in combination or as a replacement Figure 1A The cloud computing environment 100 is used. In particular, the game server 603 can be replaced by the cloud computing environment 100.

[0142] Figure 6B Including game server 603 (which may include and Figure 5A Example processing system 500 and / or Figure 5B(Similar components, features and / or functions to exemplary system 565), client 604 (which may include similar ... Figure 5A Example processing system 500 and / or Figure 5B The exemplary system 565 has similar components, features, and / or functions to the network 606 (which may be similar to the network described herein). In some embodiments of this disclosure, system 605 may be implemented.

[0143] In system 605, for a game session, client device 604 can simply receive input data in response to input from input device 626, send the input data to game server 603, receive encoded display data from game server 603, and display the display data on monitor 624. In this way, computationally intensive computation and processing are offloaded to game server 603 (e.g., rendering of the game session's graphics output, especially ray or path tracing, is performed by the GPU 615 of game server 603). In other words, the game session is streamed from game server 603 to client device 604, thereby reducing the demands on client device 604 for graphics processing and rendering.

[0144] For example, regarding the instantiation of a game session, client device 604 can display frames of the game session on display 624 based on display data received from game server 603. Client device 604 can receive input from one of input devices 626 and generate input data in response. Client device 604 can send the input data to game server 603 via communication interface 621 and through network 606 (e.g., the Internet), and game server 603 can receive the input data via communication interface 618. CPU 608 can receive the input data, process the input data, and send the data to GPU 615, which causes GPU 615 to generate a rendering of the game session. For example, the input data can represent the movement of a user character in the game, such as firing a weapon, reloading, passing a ball, turning a vehicle, etc. Rendering component 612 can render the game session (e.g., representing the result of the input data), and rendering capture component 614 can capture the rendering of the game session as display data (e.g., image data as frames of the captured game session rendering). The rendering of a game session may include lighting and / or shadow effects computed using one or more parallel processing units of the game server 603 (e.g., a GPU, which may further employ the use of one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques).

[0145] The encoder 616 can then encode the display data to generate encoded display data, which can be sent to the client device 604 via the network 606 through the communication interface 618. The client device 604 can receive the encoded display data via the communication interface 621, and the decoder 622 can decode the encoded display data to generate display data. The client device 604 can then display the display data via the display 624.

[0146] Example network environment

[0147] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 5A Processing system 500 and / or Figure 5B Implemented on one or more instances of the exemplary system 565—for example, each device may include similar components, features, and / or functions of the processing system 500 and / or the exemplary system 565.

[0148] Components of a network environment can communicate with each other through one or more networks, which can be wired, wireless, or a combination of both. A network can include multiple networks, or a network of networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. In cases where the network includes wireless telecommunications networks, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0149] A compatible network environment may include one or more peer-to-peer network environments—in which case the server may not be included in the network environment—and one or more client-server network environments—in which case the network environment may include one or more servers. In a peer-to-peer network environment, the functionality described herein regarding the server can be implemented on any number of client devices.

[0150] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. A cloud-based network environment may include a framework layer, job scheduler, resource manager, and distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework of software and / or application layers supporting the software layer and / or application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or application (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs). The framework layer may be, but is not limited to, a type of free and open-source software network application framework that can be used for large-scale data processing (e.g., "big data") using a distributed file system.

[0151] A cloud-based network environment can provide cloud computing and / or cloud storage, performing any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers in a state, region, country, globally, etc.). The core server can assign at least a portion of the functionality to the edge server if the connection to the user (e.g., client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0152] Client devices may include Figure 5A Exemplary processing system 500 and / or Figure 5B The exemplary system 565 includes at least some of its components, features, and functions. By way of example and not limitation, the client device may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, aircraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, in-vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0153] It should be noted that the techniques described herein can be contained in executable instructions stored in a computer-readable medium for use by or in conjunction with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will appreciate that, for some embodiments, various different types of computer-readable media may be included for storing data. When used herein, “computer-readable medium” includes one or more of any suitable media for storing executable instructions of a computer program, such that an instruction execution machine, system, apparatus, or device can read (or retrieve) the instructions from the computer-readable medium and execute those instructions to implement the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable compact discs (CDs), portable digital video discs (DVDs), and the like.

[0154] It should be understood that the arrangement of components shown in the accompanying drawings is for illustrative purposes, and other arrangements are possible. For example, one or more of the elements described herein may be implemented wholly or partially as electronic hardware components. Other elements may be implemented in software, hardware, or a combination of software and hardware. Moreover, some or all of these other elements may be combined, some may be omitted entirely, and additional components may be added while still achieving the functionality described herein. Therefore, the subject matter described herein can be implemented in many different variations, and all such variations are contemplated to be within the scope of the claims.

[0155] To facilitate understanding of the topics described herein, many aspects are described in sequence of actions. Those skilled in the art will recognize that various actions can be performed by dedicated circuitry or circuit systems, by program instructions executed by one or more processors, or by a combination of both. The description of any sequence of actions herein is not intended to imply that a particular order in which the actions described for execution must be followed. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context.

[0156] In the context of describing the subject matter (especially in the context of the claims below), the use of the terms “a,” “an,” “this,” and similar designations should be interpreted to cover both the singular and plural, unless otherwise specified herein or obviously contradicted by the context. The use of the term “at least one” (e.g., at least one of A and B) followed by a list of one or more items should be interpreted to mean one item selected from the listed items (A or B), or any combination of two or more of the listed items (A and B), unless otherwise specified herein or obviously contradicted by the context. Furthermore, the foregoing description is for illustrative purposes only and not for limiting purposes, as the scope of protection sought is defined by the claims set forth thereafter with their equivalents. The use of any and all example or exemplary language provided herein (e.g., “such as”) is intended merely to better illustrate the subject matter and does not constitute a limitation on the scope of the subject matter, unless otherwise stated. The use of “based on,” and other similar phrases indicating conditions leading to the result, in both the claims and the written description, is not intended to exclude any other conditions leading to that result. The language in the description should not be interpreted as indicating that any unclaimed element is essential for the implementation of the claimed invention.

Claims

1. A computer-implemented method, comprising: A first value corresponding to the execution of a first VM among a plurality of virtual machines (VMs) is calculated based on memory access metrics, the plurality of VMs being configured to share the processing resources of a computing system according to time slices allocated among the plurality of VMs; The first value was detected to exceed the threshold; Based on the first value, determine the idle duration of the first virtual CPU resource allocated to the first VM during the first set of time slices allocated to the first VM, wherein the first virtual CPU resource includes at least one virtual CPU allocated to the first VM. as well as During the execution of the programming instructions of the first VM, the first virtual CPU resource is placed in the idle state for the idle duration for at least one time slice in the first set of time slices.

2. The computer-implemented method of claim 1, wherein the idle duration is applied in each of the N time slices within the first set of time slices.

3. The computer-implemented method of claim 1, wherein a duty cycle based on the idle duration is applied across the N time slices included in the first set of time slices.

4. The computer-implemented method of claim 1, wherein the first time slice is divided into M intervals, and the idle duration is applied in each of the M intervals.

5. The computer-implemented method of claim 1, wherein the memory access metric includes the last-level cache miss rate.

6. The computer-implemented method of claim 1, wherein the memory access metric includes at least one of the following: memory access rate, average memory access rate, or memory access count during a defined time interval.

7. The computer-implemented method as described in claim 1, further comprising: For each of the plurality of VMs, performance data is tracked based on the memory access metric.

8. The computer-implemented method of claim 1, wherein each time slice in the first set of time slices samples the performance data at least once to produce the first value.

9. The computer-implemented method as described in claim 1, further comprising: After at least one time slice, it was detected that the first value had decreased; Reduce the idle duration of the first VM; as well as The first virtual CPU resource is placed in the idle state during a reduced idle duration in subsequent time slices after the at least one time slice in the first set of time slices.

10. The computer-implemented method of claim 1, further comprising: The first value is detected to be equal to or lower than the threshold. Reduce the idle duration of the first VM to zero; as well as This ensures that the first virtual CPU resource remains active during subsequent time slices in the first set of time slices.

11. The computer-implemented method of claim 1, wherein the first virtual CPU resource includes a plurality of virtual CPUs allocated to the first VM.

12. The computer-implemented method as described in claim 1, further comprising: The second VM among the plurality of VMs is detected to have a second value according to a second memory access metric that exceeds a second threshold. Based on the second value, it is determined that a second idle duration of the second virtual CPU resources, including at least one virtual CPU allocated to the second VM, is limited by placing the second virtual CPU resources allocated to the second VM in the idle state during the second set of time slices allocated to the second VM; and During the execution of the programming instructions of the second VM, the second virtual CPU resource is placed in the idle state for the second idle duration for at least one time slice in the second set of time slices.

13. The computer-implemented method of claim 12, wherein the second threshold is different from the threshold.

14. The computer-implemented method of claim 1, wherein the steps of detecting, determining, and limiting are performed using one or more computing resources in a cloud computing environment.

15. The computer-implemented method of claim 1, wherein the step of determining the idle duration is performed by a neural network.

16. The computer-implemented method of claim 1, wherein the computing system is used to train, test, or prove a neural network used in a machine, robot, or autonomous vehicle.

17. A system comprising: A computing system that includes processing resources shared by multiple virtual machines (VMs); as well as The management program is configured as follows: A first value corresponding to the execution of a first VM among the plurality of VMs is calculated based on memory access metrics, the plurality of VMs being configured by the hypervisor to share processing resources based on time slices allocated among the plurality of VMs; The first value was detected to exceed the threshold; Based on the first value, determine the idle duration of the first virtual CPU resource allocated to the first VM during the first set of time slices allocated to the first VM, wherein the first virtual CPU resource includes at least one virtual CPU allocated to the first VM. as well as During the execution of the code of the first VM, the first virtual CPU resource is placed in the idle state for the idle duration for at least one time slice in the first set of time slices.

18. The system of claim 17, wherein the management procedure is further configured to apply the idle duration in each of the N time slices within the first set of time slices.

19. The system of claim 17, wherein a duty cycle based on the idle duration is applied across the N time slices included in the first set of time slices.

20. A non-transitory computer-readable medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: A first value corresponding to the execution of a first VM among a plurality of virtual machines (VMs) is calculated based on memory access metrics, the plurality of VMs being configured to share the processing resources of a computing system according to time slices allocated among the plurality of VMs; The first value was detected to exceed the threshold; Based on the first value, determine the idle duration of the first virtual CPU resource allocated to the first VM during the first set of time slices allocated to the first VM, wherein the first virtual CPU resource includes at least one virtual CPU allocated to the first VM. as well as During the execution of the code of the first VM, the first virtual CPU resource is placed in the idle state for the idle duration for at least one time slice in the first set of time slices.

Images