Measuring and detecting idle processing periods and identifying their root causes in cloud-based streaming applications.

Abstract

This invention discloses methods for measuring and detecting idle processing cycle periods and determining their root causes in cloud-based streaming applications. A technique for analyzing data to detect problems within a cloud-based service is disclosed. A host computing device in a data center launches virtual machines, at least some of which run pipeline stacks for a streaming service. The virtual machines in the host computing device generate event data including timestamps. Metadata generated by the pipeline stack during each streaming session is analyzed to identify dead zones in the corresponding host computing device, and the event data is processed to identify potential root causes of the corresponding dead zones. The event data can be generated by the virtual machines hosting the streaming service or by different virtual machines on the same host computing device. The distribution of events for each event type relative to the identified dead zones is determined, and the operation of the host computing device can be adjusted based on this distribution.

Description

Background Technology

[0001] Many service providers allow millions of users to connect their client devices to hosted cloud-based services. Examples of cloud-based services include video streaming services, and more recently, game streaming services. Unlike traditional video game applications that run locally on console systems or computing devices, game applications in cloud-based game streaming services are hosted on server computers, and the video content frames generated by the game application are transmitted over the network to the client devices.

[0002] Streaming services are typically implemented using multiple virtual machines (VMs) running on one or more hosts across one or more data centers. Each VM can implement one or more stages (i.e., pipelines) of a software stack, where each stage performs some processing on the data stream. For example, in a GeForce... In a game streaming service like GFN (Game Forwarding Network), the pipeline can include stages such as capturing the rendered output (i.e., frames) of a VM-hosted game, encoding the frames into a bitstream, and adding forward error correction to the bitstream. The bitstream is then transmitted over the network to client devices connected to the service.

[0003] Each VM runs on a physical machine called a host. Each host can run multiple VMs substantially in parallel on multiple core / processor contexts or in a time-division duplex manner. For example, each host refers to a computing device (e.g., a server) that includes 16 CPU cores, 32GB of random access memory (RAM), and two graphics processing units (GPUs). In one scenario, a host can run two virtual machines in parallel, each with access to one GPU and eight CPU cores, plus approximately 16GB of memory. In another scenario, a host can execute four or more VMs, where each VM has access to one or more CPU cores and a portion of GPU resources, which are scheduled according to time slices and / or priorities associated with each VM.

[0004] When hosts are multiplexed in this way, physical resources can become very limited. The complexity of managing performance in a highly virtualized environment is accompanied by inherent performance issues. For example, a VM might generate a large memory access request that occupies host memory resources for a considerable period (e.g., 10 seconds or 100 milliseconds) because a page fault can trigger a load request to persistent storage media, such as a hard drive array or storage area network (SAN) hosted elsewhere in the data center. In some cases, the hardware resources used by one VM can cause execution latency for different VMs on the host, as the VM waits for host resources to become available. In services with real-time requirements, this latency can cause significant problems on the client side, such as frames failing to reach the expected frame rate, and these issues and / or other problems related to existing technologies need to be addressed. Summary of the Invention

[0005] Embodiments of this disclosure relate to systems and methods for identifying dead zones (defined as periods of time during which the software stack is idle and does not produce any output at any stage of the pipeline) on a host computing device.

[0006] According to a first aspect of this disclosure, a computer-implemented method is disclosed for identifying dead zones on a host computing device and determining the root cause of the dead zones. The method includes the steps of identifying one or more dead zones associated with at least one of a plurality of streaming sessions, wherein the at least one streaming session corresponds to at least a portion of a software stack executed on a virtual machine on the host computing device; for each dead zone identified for a particular streaming session, identifying events occurring near the dead zone in event data generated by one or more additional virtual machines on the host computing device; for each of a plurality of event types, determining an event distribution of the event types relative to the identified dead zones; and adjusting the operation of the host computing device based on the event distribution of the plurality of event types.

[0007] In some embodiments of the first aspect, the software stack includes multiple stages of a game streaming service. The stages include at least one of a frame capture stage, an encoding stage, and a launch stage.

[0008] In some embodiments of the first aspect, the host computing device executes a second virtual machine configured to execute a game application. The game application is configured to generate image frames, which are processed by a software stack to send a bitstream to a client device configured to display the image frames to a user of a game streaming service. The client device is configured to send input from the client device to the game application to render subsequent frames of the game application.

[0009] In some embodiments of the first aspect, identifying events occurring near dead zones in event data generated by one or more additional virtual machines of the host computing device includes, for each event in the event data, calculating the difference between a timestamp associated with the event and a timestamp associated with each identified dead zone; and for a particular dead zone, marking each event having an absolute value of the difference less than a threshold as associated with the dead zone.

[0010] In some embodiments of the first aspect, determining the event distribution relative to the corresponding dead time includes generating a histogram by counting the number of occurrences of each event type within multiple time periods around the corresponding dead time.

[0011] In some embodiments of the first aspect, adjusting the host computing device based on the event distribution of multiple event types includes determining that the event distribution of a specific event type includes the number of events in a specific time period, the number of events in the specific time period exceeding the average number of events in each time period by a threshold amount; and identifying operations based on the specific event type.

[0012] In some embodiments of the first aspect, adjusting the operation of the host computing device includes migrating a second virtual machine on the host computing device to a second host computing device.

[0013] In some embodiments of the first aspect, adjusting the operation of the host computing device includes migrating data corresponding to an event type from a first storage device to a second storage device.

[0014] In some embodiments of the first aspect, identifying a dead zone associated with at least one of a plurality of streaming sessions includes sorting one or more timestamps contained in the metadata of at least one streaming session into a series; calculating the difference between each timestamp in the series and a previous timestamp; and comparing each difference with a threshold, wherein any difference greater than the threshold indicates an identified dead zone.

[0015] According to a second aspect of the invention, a system is disclosed comprising at least one host computing device and an analytics engine. Each host computing device is configured to launch at least one virtual machine, which is configured to process the content of one or more streaming sessions. The at least one streaming session corresponds to a software stack executed on the virtual machine of the host computing device, which is configured to generate content for display by a corresponding client device. The analytics engine is configured to identify dead zones associated with at least one of the multiple streaming sessions; for each dead zone identified for a specific streaming session of a specific host computing device, identify events occurring near the dead zone in event data generated by one or more additional virtual machines of the specific host computing device; for each of a plurality of event types, determine an event distribution of the event types relative to the identified dead zones; and adjust the operation of the at least one host computing device based on the event distribution of the plurality of event types.

[0016] In some embodiments of the second aspect, the software stack includes multiple stages of a game streaming service, including at least one of a frame capture stage, an encoding stage, and a transmission stage.

[0017] In some embodiments of the second aspect, the host computing device executes a second virtual machine configured to execute a game application. The game application is configured to generate image frames, which are processed by a software stack to send a bitstream to a client device configured to display the image frames to a user of a game streaming service. The client device is configured to send input from the client device to the game application to render subsequent frames of the game application.

[0018] In some embodiments of the second aspect, identifying events occurring near dead zones in event data generated by one or more additional virtual machines of the host computing device includes, for each event in the event data, calculating the difference between a timestamp associated with the event and a timestamp associated with each identified dead zone; and for a particular dead zone, marking each event having an absolute value of the difference less than a threshold as associated with the dead zone.

[0019] In some embodiments of the second aspect, determining the event distribution relative to the corresponding dead time includes generating a histogram by counting the number of occurrences of each event type within multiple time periods around the corresponding dead time.

[0020] In some embodiments of the second aspect, adjusting the host computing device based on the event distribution of multiple event types includes determining that the event distribution of a specific event type includes the number of events in a specific time period, the number of events in the specific time period exceeding the average number of events in each time period by a threshold amount; and identifying operations based on the specific event type.

[0021] In some embodiments of the second aspect, adjusting the operation of the host computing device includes migrating a second virtual machine on the host computing device to a second host computing device.

[0022] In some embodiments of the second aspect, adjusting the operation of the host computing device includes migrating data corresponding to the event type from a first storage device to a second storage device.

[0023] In some embodiments of the second aspect, identifying a dead zone associated with at least one of a plurality of streaming sessions includes sorting one or more timestamps contained in the metadata of at least one streaming session into a series; calculating the difference between each timestamp in the series and a previous timestamp; and comparing each difference with a threshold, wherein any difference greater than the threshold indicates an identified dead zone.

[0024] According to a third aspect of the invention, a non-transitory computer-readable medium storing instructions is disclosed. When the instructions are executed by a processor, the processor performs the following steps: for at least one of a plurality of streaming sessions, identifying a dead zone associated with the streaming session, wherein the at least one streaming session corresponds to a software stack executed on a virtual machine on a host computing device; for each dead zone identified for a particular streaming session, identifying events occurring near the dead zone in event data generated by one or more additional virtual machines of the host computing device; for each of a plurality of event types, determining an event distribution of the event types relative to the identified dead zones; and adjusting the operation of the host computing device based on the event distribution of the plurality of event types.

[0025] In some embodiments of the third aspect, the software stack includes multiple stages of a game streaming service, including at least one of a frame capture stage, an encoding stage, and a transmission stage. Attached Figure Description

[0026] The following describes in detail, with reference to the accompanying drawings, existing systems and methods for measuring and detecting dead zones on host computing devices, wherein:

[0027] Figure 1 A pipeline stack for cloud-based services is shown according to some embodiments.

[0028] Figure 2A A flowchart of a method for identifying events associated with the occurrence of dead zones in a host computing device, according to some embodiments, is shown.

[0029] Figure 2B The following are examples of implementations. Figure 2A A system of methods.

[0030] Figure 2CA system for detecting dead zones and identifying root causes, according to some embodiments, is shown.

[0031] Figure 3 A visualization of a heatmap for determining the distribution of events relative to a corresponding dead zone, according to some embodiments, is shown.

[0032] Figure 4 Example parallel processing units suitable for implementing some embodiments of this disclosure are shown.

[0033] Figure 5A Is using Figure 4 The conceptual diagram of the processing system implemented by the PPU is suitable for implementing some embodiments of the present invention.

[0034] Figure 5B An exemplary system is shown in which various architectures and / or functions of various previous embodiments can be implemented.

[0035] Figure 5C Components of an exemplary system that can be used to train and utilize machine learning in at least one embodiment are shown.

[0036] Figure 6A It is by Figure 4 The conceptual diagram of the graphics processing pipeline implemented by the PPU is suitable for implementing some embodiments of the present invention.

[0037] Figure 6B An exemplary game streaming system suitable for implementing some embodiments of this disclosure is shown.

[0038] Figures 7A-7D The difference between remaining latency and dead time on a host device according to some embodiments is shown. Detailed Implementation

[0039] Systems and methods related to the measurement and detection of dead zones experienced by host computing devices—when the software stack is idle and produces no output at any stage of the application processing pipeline—are disclosed. The techniques described herein are capable of detecting dead zones and identifying their root causes, thereby enabling manipulation of cloud-based services to improve user experience. In a specific example, the interactions of multiple virtual machines running on various host devices can be analyzed to identify specific virtual machines whose interactions may negatively impact the quality of streaming services running on different virtual machines on the same host device. Once the root cause of the dead zone is identified, corrective actions can be taken manually or automatically to improve the service, such as changing the priority of various applications or migrating certain virtual machines to different host devices within the same data center.

[0040] Traditionally, analyzing complex systems (such as data center resources) to identify potential intermittent problems related to asynchronous resource contention is a challenging task. Gathering enough data to identify the problem and finding ways to quantify objective criteria to pinpoint its root cause can be difficult. This is especially true when virtual machines for unrelated applications can start on the same host at different times. Automated systems for collecting and analyzing such data can significantly enhance developers' ability to address underlying problems and can easily help transform live-on-demand services (such as game streaming) from virtually unusable to enjoyable. The technologies and solutions presented in this article are described in more detail below.

[0041] Figure 1 A pipeline stack 100 for a cloud-based service is illustrated according to some embodiments. The pipeline stack 100 may include multiple stages. Each stage includes one or more components, and the output of one stage is received at the input of subsequent stages. Although the pipeline stack 100 is described for a real-time game streaming service, other embodiments may use different sets of stages in the pipeline stack 100 to implement different types of services (e.g., video streaming, audio streaming, productivity applications, etc.).

[0042] like Figure 1 As shown, the pipeline stack 100 for a game streaming service includes multiple stages. The server-side 102 of the pipeline stack 100 implements these stages on one or more server devices, including a frame capture stage 110, an encoding stage 120, a forward error correction (FEC) stage 130, and a transmission stage 140. The pipeline stack 100 also includes a network stage 150, which includes transmitting data packets from the server-side 102 to the client 106 via a network 104. The client 106 of the pipeline stack 100 implements these stages on one or more client devices, including a decoding stage 160, a display stage 170, and a client resynchronization (re-synchronization) stage 180.

[0043] It should be understood that, in order to illustrate the various techniques described in this article, Figure 1 An exemplary pipeline stack 100 for a game streaming service is described herein. In other embodiments, the specific arrangement of stages in the pipeline stack 100 may differ, for example by including Figure 1 Additional stages not shown or omitted Figure 1 The phases shown are as follows. For example, FEC phase 130 can be omitted if FEC is not implemented by a service. In addition, some implementations can combine aspects of two or more phases into a single phase (e.g., FEC and transport can be combined into a single phase of pipeline stack 100).

[0044] Figure 1Each stage shown may include one or more components. For example, encoding stage 120 may include a queue component that asynchronously receives frames from frame capture stage 110 while the previous frame is being encoded by the encoder component. Encoding stage 120 may also include a send queue component that asynchronously receives encoded frames from the encoder component and sends the encoded frames to FEC stage 130 when the encoded frames are available and FEC stage 130 is ready. As will be discussed in more detail, the metadata of the content processed by the various stages of pipeline stack 100 may be updated by each stage or by each component within that stage.

[0045] In one embodiment, the content comprises one or more frames, and metadata is generated for each frame. Each component or stage updates the metadata by adding a timestamp to the frame's metadata after processing it. The metadata is then forwarded to the next component / stage along with the processed content.

[0046] At the end of pipeline stack 100, the client device transmits metadata back to the server device for storage and / or analysis. The metadata of a frame includes multiple timestamps, each corresponding to a specific stage or component of a stage completing processing of the frame. In some embodiments, the client device is configured to collect metadata for multiple frames (e.g., 60 seconds of video at 30-60 frames per second) within a data structure (e.g., a file) before sending the data structure to the server device. In such embodiments, the server device collects files for multiple sessions, each session being equal to, for example, several seconds of video. It will be understood that the size of the sessions can be adjusted to match the desired size of the collected metadata, for example, by collecting 30 to 60 minutes of metadata within a session. In some embodiments, the client device collects metadata in a circular buffer such that the metadata collected for a session is only the metadata of the last n frames in the session. For example, a circular buffer (e.g., metadata for 36,000 frames) with a size suitable for collecting 10 minutes of data at 60 frames per second can continuously collect metadata during a session, overwriting the oldest metadata when the session length exceeds 10 minutes. At the end of the session (e.g., when the game application terminates), the client device can encode the contents of the circular buffer and send it to the server device. It should be understood that the session is likely to terminate prematurely when the user experiences issues that degrade the user experience; therefore, the last m minutes of the session (corresponding to n frames) are usually sufficient to highlight the cause of the degraded user experience.

[0047] In a specific use case, a particular host might run a game on a first VM and a software stack on a second VM for streaming the game's generated frames. The streaming service's software stack might not require many compute resources, but it might need consistent access to a small number of compute cycles per hour (e.g., 16.6 ms) to maintain a frame rate of 60 Hz on client devices. If the game running on the first VM experiences issues with the host, typically due to heavy workloads on one or more of the host's resources, the streaming software stack might experience latency due to a lack of available resources.

[0048] As is understandable, a dead zone differs from the latency introduced at a particular stage of the pipeline, which pauses subsequent stages of the pipeline. Some inherent latency due to variable workloads is inherently hidden in the pipeline architecture, which is to be expected. Therefore, if the encoding stage of the pipeline takes longer than average to encode image frames into a bitstream (e.g., 24 ms), other stages of the pipeline can continue processing the bitstream of previous frames, which can allow some latency to be hidden as subsequent stages make up for some or all of the lost time. On the other hand, a dead zone represents a different problem, where all stages of the pipeline, or at least several stages running on a particular host's VM, stop abruptly due to a lack of available resources on the host.

[0049] Figures 7A-7C The difference between remaining latency and dead time on a host device according to some embodiments is shown. Figure 7A As shown, a timing diagram 710 illustrates multiple frames processed by three stages on the server side 102 of the pipeline stack 100. The server side 102 includes a frame capture stage 110, an encoding stage 120, and a transmission stage 140, with the FEC stage 130 omitted (i.e., not implemented). Each block in timing diagram 710 represents the completion of processing for a specific frame in that stage. Clearly, timing diagram 710 does not show dead zones in that particular host, as all three stages consistently complete processing of the corresponding frame approximately every 16.6 ms, corresponding to a 60 Hz refresh rate.

[0050] like Figure 7B As shown, a timing diagram 720 illustrates multiple frames processed by three stages on the server side 102 of the pipeline stack 100. In this case, the encoding stage 120 experiences some latency between frames 2 and 3. This, in turn, causes the transmission stage 140 to halt while waiting for the third frame from the encoding stage 120. However, this does not represent a dead zone, as the frame capture stage 110 is still processing the fourth frame during the latency of the encoding stage 120. As long as some stages are still executing on the host device, the latency may be caused by other issues unrelated to the host's resource availability.

[0051] like Figure 7C As shown, a timing diagram 730 illustrates multiple frames processed by three stages on the server side 102 of the pipeline stack 100. In this case, the frame capture stage 110 appears to stop after the second frame, causing pipeline starvation. Then, the encoding 120 and sending stages 140 stop after processing the second frame. However, there is a significant gap between the end of processing the second frame by the sending stage 140 and the end of processing the third frame by the frame capture stage 110. This delay could be caused by a potential dead zone, although it could also simply be due to a problem encountered by the game application that delayed the generation of the third frame. Therefore, there is no guarantee that this delay is related to a dead zone.

[0052] like Figure 7D The diagram shows a timing diagram 740 illustrating multiple frames processed by three stages on the server side 102 of the pipeline stack 100. In this case, none of the three stages of the pipeline stack 100 is active for a period of approximately 50 ms. The processing of the third frame is completed by the frame capture stage 110 at the beginning of the 50 ms waiting period, but the encoding stage 120 does not complete the processing of the third frame until after a significant delay. Similarly, the processing of the second frame is completed by the encoding stage 120 at the beginning of the 50 ms waiting period, but the transmission stage 140 does not complete the processing of the second frame until after a significant delay. Assuming that the stages of the pipeline stack 100 are all delayed simultaneously, and that the delay is clearly not caused by a lack of input to a stage, this indicates a dead zone that could be caused by a problem with the host device.

[0053] Now back Figure 1 A naive approach to measuring dead zones can use independent threads (e.g., running within the VM of pipeline stack 100 or in a separate VM on the host computing device) that generate timestamps periodically (e.g., every millisecond). When the thread is within the VM of pipeline stack 100, the timestamp will be delayed when the host computing device fails to execute the thread on time. Measuring the large increments between timestamps generated in this way will produce information about the dead zone. However, this approach only captures instances where the hypervisor has not scheduled the VM, and not instances where the VM is running but deadlocked, waiting for a response from some resource (e.g., memory resources, network resources, etc.). When the thread is on a different VM, the timestamp will only measure when the hypervisor has not scheduled that particular VM, even if the VM of pipeline stack 100 is running well. Therefore, this method of generating timestamps to identify dead zones is not ideal.

[0054] Conversely, in a preferred embodiment, it should be understood that dead zones associated with the streaming session can be identified by calculating the residual metric for each frame at each of the multiple stages of the server-side 102 of the pipeline stack 100. The residual metric for a given stage of the pipeline can be calculated as follows:

[0055]

[0056] Equation 1 measures the difference between the time between processing two frames (e.g., frame (i) and frame (i-1)) and the expected inter-frame time based on the target frame rate (TargetFPS) (in frames per second). The residual metric is measured in milliseconds.

[0057] In one embodiment, large residuals in a streaming session can be identified by comparing each residual of each frame with a threshold. It should be understood that residuals for frames can be computed for different timestamps throughout the entire pipeline stack 100. For example, the three stages of pipeline stack 100—frame capture stage 110, encoding stage 120, and transmission stage 140—can generate more than 20 timestamps during the processing of a single frame, with each stage generating at least one timestamp. Therefore, for each of the 20 timestamps generated for that frame, compared to the previous frame, each frame can be associated with 20 distinct residual values. In one embodiment, dead zones can be identified by comparing residual metrics above a threshold for each of the multiple stages of pipeline stack 100 to find overlaps between large residuals of different frames processed simultaneously by the pipeline. Techniques for identifying dead zones will be described in more detail below.

[0058] Figure 2A A flowchart of a method 200 for identifying events associated with the occurrence of a dead zone in a host computing device, according to some embodiments, is shown. The method 200 of FIG. 2 can be implemented by hardware, software, or some combination thereof. In an embodiment, method 200 is implemented by a server device including at least one of a central processing unit, a parallel processing unit, or any other type of processor capable of executing instructions that cause the server device to perform the following operations.

[0059] In step 202, for each of the multiple streaming sessions, a dead zone associated with the streaming session is identified. In an embodiment, the streaming session is executed at least in part by a pipeline stack running on a virtual machine on the host computing device. The pipeline stack generates metadata that includes timestamps associated with each stage of the pipeline stack during the streaming session. It should be understood that although the streaming session sends frames to the client device at a rate of approximately one frame every 16.6 ms, multiple timestamps corresponding to different stages and different frames may exist within a given 16.6 ms time slice of the metadata. For example, in a simplified version of the pipeline stack 100 that includes a frame capture stage 110, an encoding stage 120, and a transmission stage 140, different stages may process three or more frames simultaneously, and each stage may generate one or more timestamps during the processing of a single frame. Therefore, even if the transmission stage 140 is delayed for any reason, the frame capture stage 110 and the encoding stage 120 may still generate timestamps within a given 16.6 ms period corresponding to the same or different frames being processed by the transmission stage 140.

[0060] In one embodiment, timestamps collected in the metadata of the streaming session are tagged with the identifier (i.e., frame number) of the frame corresponding to that timestamp. The timestamps of all frames in the streaming session are then concatenated into a series ordered by timestamp values, such that each timestamp represents a different event completed by pipeline stack 100 over a period of time. However, timestamps of different frames can be interleaved in the sequence because multiple frames are processed simultaneously by different stages of the pipeline. For each timestamp in the sequence, the difference between the current timestamp and the previous timestamp in the sequence is calculated. This difference represents the latency associated with the frame at the current timestamp. It should be understood that this difference is not a time metric between the processing of a frame by a previous stage in pipeline stack 100 and the processing of a frame by a subsequent stage in pipeline stack 100, but merely a time metric between any frame being processed by any stage of pipeline stack 100 and the processing of that frame by a specific stage in pipeline stack 100 represented by the current timestamp. Therefore, even if, for example, the processing of a particular frame is delayed in the sending phase 140, the difference can still be small if the frame capture phase 110 or encoding phase 120 of the pipeline stack 100 is actively processing later frames output by the game application.

[0061] In one embodiment, identifying dead zones involves sorting the differences associated with each unique frame in the frame sequence of the streaming session and determining for each frame whether the maximum difference is greater than a threshold. In the case of a desired frame rate of 16.6 ms, the threshold could be, for example, 33.3 ms. Any frame with a difference greater than the threshold can then be marked as having a dead zone in the host.

[0062] In step 204, for each dead zone identified for a specific streaming session, events in event data generated by one or more attached virtual machines on the host computing device are identified as occurring near the dead zone. In one embodiment, event data for multiple VMs executing on the host computing device is collected. While event data may be associated with VMs of other streaming sessions, it is not limited to other streaming sessions. For example, event data may be associated with a VM running a game application or a completely independent process (e.g., a web application, a standalone cloud-based service, etc.). Any VM that can generate event data for one or more event types and include a timestamp indicating when the event occurred is within the scope of this disclosure.

[0063] Similarly, each host computing device can implement multiple different virtual machines, each virtual machine can run different streaming sessions on different client devices, and multiple hosts can run different virtual machines for different streaming sessions. Furthermore, other virtual machines can run concurrently on the host to execute, for example, game applications or other tools or applications, such as tools for network management. Each streaming session can be configured to record timestamps of different events, such as, but not limited to, VM start events, client connection events, shader cache load events, game launch events, and session end events. The event data reflects the timestamp of each event that occurs within a specific streaming session. It will be understood that events in a specific streaming session X are not related to the same events in different streaming sessions Y, because two clients can connect to the streaming service at different times, and the execution of the same game can diverge based on highly variable user input. Therefore, events such as the game launch event in session X may be temporally close to different events such as the session end event in session Y.

[0064] In one embodiment, for each dead zone identified for a specific streaming session, events in the event data are filtered to include those events located on the specific host (i.e., events occurring within a different VM but on the same host as the specific streaming session) that occur within a time window close to the dead zone. It should be understood that events on other hosts may not affect the lack of compute resources within the specific host, and therefore these events are filtered out. One exception is when network traffic generated by other hosts might cause problems with network resources on the specific host, for example, causing latency in the pipeline stack due to clients not receiving data packets. Another exception is if two or more hosts attempt to load data from network storage resources outside each host. However, correlating all events on all hosts with dead zones on a specific host is a more difficult problem to solve because, for example, fluctuations in network traffic can be highly random, making it less likely to identify dead zones by studying the distribution of event types associated with their occurrence on a specific host.

[0065] Among the filtered events, events that occur within a specific timeframe of the dead zone (e.g., up to M seconds before or after the dead zone) are marked as triggering events (i.e., events that may lead to the dead zone problem). In one embodiment, identifying events occurring near the dead zone in event data generated by one or more attached virtual machines of a host computing device includes: for each event in the filtered event data (e.g., an event associated with a specific host computing device and occurring within a specific timeframe), calculating the difference between the timestamp associated with the event and the timestamp associated with each identified dead zone; and for a specific dead zone, marking each event having an absolute value of the difference less than a threshold as related to the dead zone.

[0066] In step 206, for each of the multiple event types, the event distribution relative to the identified dead zone is determined. In one embodiment, the location distribution of triggering events for a specific event type relative to the corresponding dead zone can be plotted to identify events that may cause a dead zone in the host. The location of an event can be an event time relative to the dead zone time, such as identified based on the dead zone timestamp and the event timestamp. For example, if the triggering event occurs 3 seconds before the dead zone, the location of the triggering event is defined as -3 seconds. Or, if the triggering event occurs 5 seconds after the dead zone, the location of the triggering event is defined as +5 seconds.

[0067] It should be understood that if events in different VMs cause dead zones in other streaming sessions on the host, the event distribution corresponding to that event type will show the event concentration at a specific time relative to the dead zones recorded in one or more streaming sessions detected on the same host, relative to the dead zones occurring on the host. However, if the event is unrelated to the cause of the dead zone, the event distribution for that event type will be random or approximately uniform relative to the occurrence of dead zones on the host. For example, if a game launched for a streaming session causes a large amount of game data to be loaded into memory L seconds after game launch, then the distribution of game launch events occurring in other streaming sessions is likely to occur L seconds before the dead zones occur in different streaming sessions running on the same host.

[0068] It's understandable that triggering events can occur before or after the dead zone, because the timestamp for the event's generation can be generated before or after the root cause of the dead zone. For example, if the game launch event is generated by the game application after game data is loaded into host memory, the event's timestamp might appear after the dead zone, since the dead zone is related to insufficient memory resources on the host, and the data is loaded from a remote storage location into host RAM. Alternatively, if the dead zone occurs sometime after game launch during the execution of the game application, the dead zone might appear after the game launch event. This distribution provides some clues as to whether the occurrence of a dead zone is caused by an event in a different VM that occurs at a relatively stable time before or after a given event.

[0069] In one embodiment, the distribution can be determined by creating a histogram for each event type. Whenever a triggering event is identified for a specific dead zone, the location of the event relative to the dead zone is determined, and the time period of the histogram (e.g., intervals, slices, etc.) is incremented to show that events of that event type occurred within a specific time period relative to the detected dead zone. It should be understood that the distribution and histogram can be correlated with any number of dead zones occurring on multiple different hosts and at different times, and are not merely a measurement of events occurring on the same host within a single dead zone. Events occurring near a single dead zone on a single host are often insufficient to correlate data with determining the root cause of that dead zone; therefore, events are tracked across multiple hosts and dead zones identified in a large number of different streaming sessions to find enough data points to accurately represent the event distribution for a specific event type.

[0070] In one embodiment, the histogram is generated using a time window approximately 1 second wide, consisting of a time window of about 30 seconds before and after the dead zone. In other words, the histogram comprises at least 60 time periods or segments. In other embodiments, the size of the time window can be increased or decreased, and the width of each time period can also be increased or decreased. For example, a histogram can be created with a time period 100 milliseconds wide before and after the dead zone and a 10-second time window. The histogram can be included in the heatmap because the value of each time period is directly related to the number of events occurring at that location relative to the corresponding dead zone. If the event type is not related to the occurrence of the dead zone, the locations of the events should be random and roughly uniformly distributed across a set of time periods in the histogram (e.g., uniformly distributed). However, if the event type is related to the occurrence of the dead zone, the locations of the events should be highly concentrated in a small subset of the histogram time periods. This is described in more detail below. Figure 3 A visualization of a heatmap showing multiple histograms including multiple event types is presented.

[0071] In step 208, the operation of the host computing device is adjusted based on the event distribution of multiple event types. In one embodiment, the host computing device, in response to determining the event distribution of a specific event type, includes the number of events in a specific time period that exceeds the average number of events in each time period by a threshold amount, where the time period refers to a histogram. In one embodiment, the operation being adjusted is identified based on the specific event type. For example, a mapping table can associate a specific event type with the type of operation to be performed and different operations based on the shape of the distribution (e.g., spikes appearing in the histogram relative to corresponding dead zones).

[0072] In one embodiment, host computing devices can be adapted by migrating virtual machines from one host to another. For example, if a particular VM is determined to be the root cause of dead zones in a host device, that VM and the application running in it can be moved to a different host so that streaming services on one host are not affected by applications running in VMs on different hosts. In some embodiments, a VM running a game application can be moved from a VM running at least a portion of the streaming service of pipeline stack 100 to a separate host. It should be understood that in some cases, moving a VM from one host to another may be impractical, for example, when transferring image frame data from one host to another, it would result in excessive latency between the frame being rendered and the frame being displayed on the client device.

[0073] In other embodiments, host computing devices can be tuned by changing the operating parameters of a specific VM. For example, a game application can be reconfigured to use different settings (e.g., resolution, frame rate, alias settings, etc.). In other embodiments, storage resources can be adjusted, for example, by moving some data (e.g., high-priority data) from slower storage media (e.g., SAN, HDD) to faster storage media (e.g., SSD) connected to the host. For example, if a game launch event causes a dead zone, changing the data priority of a specific game to move it from a first storage location on an HDD to a second storage location on an SSD can reduce or mitigate the frequency of dead zones in the host. When different game applications can run on different VMs on the same host, adjusting the priority of a game application may affect the storage location of the game application.

[0074] It should be understood that some host operations can be automatically adjusted in response to analysis of distributions associated with different event types. However, other situations may not be easily adjusted, and in some cases, the server device can send notifications (i.e., reports) to the network manager. In some embodiments, the notification may include heatmaps generated by the server, which can be manually analyzed and used by designers to adjust code in certain applications. By changing the design of certain applications, these applications can utilize certain resources of the host computing device more efficiently, resulting in a reduction in the frequency of dead zones occurring in the host.

[0075] Figure 2B The following are examples of implementations. Figure 2A The system of method 200 includes multiple server devices 260 located in a data center 250. The server devices 260 are connected to one or more network devices 270, such as network switches and routers. The network devices 270 enable the server devices 260 to communicate within the data center and enable the server devices 260 to send data packets to multiple client devices 280 via network 104.

[0076] In some embodiments, each of the plurality of server devices 260 is a host running one or more virtual machines (VMs). A particular server device 160 may run a streaming service in a first VM and a game application in a second VM. The game application may generate frames for display on a corresponding client device 280, and the streaming service may encode the frames into a bitstream, which is sent to the client device 280 via network 104. The game application may receive input from the client device 280 for rendering subsequent frames for display.

[0077] In other embodiments, the streaming service can be executed in a VM on the first server device 260, and the game application can be executed in a VM on the second server device 260. Frames rendered by the game application are transmitted from the second server device 260 to the first server device 260, allowing the streaming service to encode the bitstream of the frames and send it to the client device 280.

[0078] Figure 2C A system for detecting dead zones and identifying root causes, according to some embodiments, is illustrated. The system includes an analytics engine 290. In one embodiment, the analytics engine 290 is a set of instructions configured to be executed by a server device. The analytics engine 290 may be hosted on a virtual machine in the same host computing device as the pipeline stack 100 of the cloud-based service, or alternatively, it may be hosted on a different host computing device or even a client device.

[0079] In one embodiment, server device 260 and / or client device 280 send metadata generated by pipeline stack 100 to analytics engine 290. The metadata may be stored in database 292. In some embodiments, analytics engine 290 processes the metadata after receiving it for a specific streaming session to identify dead zones (if any) within that streaming session. Analytics engine 290 does not store metadata for the entire session, but only information used to identify dead zones. For example, for each identified dead zone, analytics engine may store an identifier for the streaming session, a timestamp indicating the start of the dead zone, and the duration or length of the dead zone. Analytics engine 290 may also store any other relevant information associated with the dead zone, such as the frame identifier of the last frame processed by pipeline stack 100 before the start of the dead zone and the identifier of the stage of pipeline stack 100 that processed that frame.

[0080] It should be understood that metadata of a streaming session can be transmitted from server device 260 to analytics engine 290, including metadata received by server device 260 from client device 280, or metadata received directly from client device 280, without having to be routed to server device 260.

[0081] The analytics engine 290 also receives event data from multiple server devices 260 stored in the database 294. In an embodiment, the analytics engine 290 implements an application programming interface (API), which allows any processing performed in the server devices 260 to generate event data and send it to the analytics engine 290 for storage and / or processing. Therefore, any application that builds functionality for generating event data into the software used for that application will generate the event data stored in the database 294.

[0082] In an embodiment, the event data includes at least one of the following: an event timestamp, an identifier indicating the event type, a virtual machine identifier, a host computing device identifier, a data center identifier, and / or any other useful information related to the event.

[0083] Once the analytics engine 290 has the relevant metadata and event data, it is configured to process the data to perform the method 200 described above. In some embodiments, the analytics engine 290 is capable of making changes to parameters of the server device 260, the pipeline stack 100, and / or any other applications hosted in virtual machines on the host computing device.

[0084] Figure 3 A visualization of a heatmap 300 for determining the distribution of events relative to corresponding dead zones, according to some embodiments, is shown. Heatmap 300 refers to a data structure that includes histogram data of one or more event types. Figure 3As shown, heatmap 300 encodes data from multiple histograms for game launch events, attached application image events, completed maximum (Max) events, event streaming events, and event chair launch events. These event types correspond to specific events generated in one or more different VMs running on different hosts. It should be understood that developers of various software can define any number of events, and Figure 3 The examples of event types disclosed herein are for illustrative purposes only.

[0085] In one embodiment, each histogram includes multiple buckets corresponding to time periods relative to the dead zone. Each bucket is colored according to the relative frequency of events occurring within the corresponding dead zone time period. In one embodiment, each bucket corresponds to a time period of one second, and there are 30 buckets after the dead zone time, represented by a white line in the middle of the histogram. The horizontal axis represents the timestamp offset (in seconds), which is the difference between the event timestamp and the corresponding dead zone timestamp. Color scales associated with the corresponding relative frequencies are displayed on the right side of the visualization of heatmap 300, and various histograms for different event types are stacked in the vertical dimension. In other embodiments, each bucket may represent a different time period (e.g., 5 seconds or 3 seconds), and the length of time represented by the histogram may be increased or decreased (e.g., 60 seconds before and after the dead zone). Furthermore, the visualization of heatmap 300 can be modified to display histograms for a single event type or any number of event types (e.g., 30 event types).

[0086] In some embodiments, the network manager may configure a session analysis tool to periodically generate heatmaps 300 and specify any selected combinations of event types to be included in a particular heatmap 300. In embodiments, the server device is configured to periodically analyze event data and generate one or more heatmaps 300 corresponding to different subsets of event types. The heatmaps 300 are then used to automatically or manually adjust network operations. For example, in one embodiment, heatmaps 300 may be generated, and visualizations of heatmaps 300 are automatically and periodically emailed to the network manager for review. As another example, heatmaps 300 are generated, and then operations for reconfiguring services or the network (e.g., migrating VMs, provisioning new hosts, etc.) are performed based on the distribution in the heatmaps 300.

[0087] like Figure 3As shown, the visualization of heatmap 300 can quickly indicate whether a particular event type is likely the root cause of a dead zone. For example, the attached application image event type suggests that these events are likely the root cause of dead zones and occur before the dead zone, but these events vary significantly in time relative to the dead zone. Conversely, event streaming events are highly correlated with dead zones that are very close in time to the event time. In contrast, the completed maximum event type appears to show a more even distribution of events relative to the corresponding dead zone, suggesting that the completed maximum event is unlikely to be the root cause of the corresponding dead zone.

[0088] Based on the user's wishes, further illustrative information regarding various optional architectures and functionalities that can be used to implement the aforementioned framework will now be provided. It should be noted that the following information is for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be selectively combined with or do not exclude the other features described.

[0089] It should be understood that the system and component arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groups, etc.) may be used in addition to those shown, or alternative arrangements and elements may be used instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components, or in combination with other components, and can be implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Moreover, those skilled in the art will understand that any system performing the operations described herein is within the scope and spirit of the embodiments of the present invention.

[0090] Parallel processing architecture

[0091] Figure 4 A parallel processing unit (PPU) 400 according to an embodiment is shown. The PPU 400 can be used to at least partially implement one or more stages of the pipeline stack 100. For example, the PPU 400 can be used to implement one or more of an encoding stage 120, a decoding stage 160, a rendering engine for a game application server, an analysis engine 290, etc., performed by a host device.

[0092] In one embodiment, PPU 400 is a multi-threaded processor implemented on one or more integrated circuit devices. PPU 400 is a latency-hiding architecture designed for parallel processing of multiple threads. A thread (e.g., an execution thread) is an instantiation of a set of instructions configured to be executed by PPU 400. In one embodiment, PPU 400 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device. In other embodiments, PPU 400 can be used to perform general-purpose computing. While an exemplary parallel processor is provided herein for illustrative purposes, it should be strongly noted that the processor is for illustrative purposes only and any processor may be used to supplement and / or replace the processor described herein.

[0093] One or more PPU 400s can be configured to accelerate thousands of high-performance computing (HPC), data center, cloud computing, and machine learning applications. PPU 400s can be configured to accelerate numerous deep learning systems and applications for autonomous vehicles, simulations, computational graphics (e.g., ray or path tracking), deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations.

[0094] like Figure 4 As shown, PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a job allocation unit 425, a hub 430, a crossbar (Xbar) 470, one or more general purpose processing clusters (GPCs) 450, and one or more memory partitioning units 480. PPU 400 can be connected to a host processor or other PPU 400 via one or more high-speed NVLink 410 interconnects. PPU 400 can be connected to a host processor or other peripheral devices via interconnect 402. PPU 400 can also be connected to local memory 404, which includes multiple storage devices. In one embodiment, local memory may include multiple dynamic random access memory (DRAM) devices. The DRAM devices may be configured as a high-bandwidth memory (HBM) subsystem, with multiple DRAM dies stacked within each device.

[0095] The NVLink 410 interconnect enables the system to expand and include one or more PPUs 400 in combination with one or more CPUs, supporting cache coherency between the PPUs 400 and the CPU, as well as CPU master control. Data and / or commands can be sent from or from the NVLink 410 to other units of the PPU 400 via hub 430, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 5B A more detailed description of the NVLink 410.

[0096] I / O unit 405 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 402. I / O unit 405 may communicate directly with the host processor via interconnect 402 or via one or more intermediate devices such as a memory bridge. In one embodiment, I / O unit 405 may communicate with one or more other processors, such as one or more PPUs 400, via interconnect 402. In one embodiment, I / O unit 405 implements a Peripheral Component Interconnect Pass-Through (PCIe) interface for communication via a PCIe bus, and interconnect 402 is a PCIe bus. In alternative embodiments, I / O unit 405 may implement other types of well-known interfaces for communication with external devices.

[0097] I / O unit 405 decodes packets received via interconnect 402. In one embodiment, the packets represent commands configured to cause PPU 400 to perform various operations. I / O unit 405 sends the decoded commands to various other units of PPU 400, as specified by the commands. For example, some commands may be sent to front-end unit 415. Other commands may be sent to hub 430 or other units of PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 405 is configured to route communication between various logical units of PPU 400.

[0098] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. A workload may include multiple instructions and data processed by those instructions. A buffer is an area in memory that can be accessed (e.g., read / written) by both the host processor and the PPU 400. For example, I / O unit 405 may be configured to access a buffer in system memory connected to interconnect 402 via a memory request sent via interconnect 402. In one embodiment, the host processor writes a command stream to the buffer and then sends a pointer to the start of the command stream to the PPU 400. Front-end unit 415 receives pointers to one or more command streams. Front-end unit 415 manages one or more streams, reads commands from the streams, and forwards the commands to the respective units of the PPU 400.

[0099] Front-end unit 415 is coupled to scheduler unit 420, which configures various GPCs 450 to handle tasks defined by one or more flows. Scheduler unit 420 is configured to track status information associated with the various tasks managed by scheduler unit 420. This status may indicate which GPC 450 a task is assigned to, whether the task is active or inactive, the priority associated with the task, and so on. Scheduler unit 420 manages the execution of multiple tasks on one or more GPCs 450.

[0100] Scheduler unit 420 is coupled to job allocation unit 425, which is configured to schedule tasks for execution on GPCs 450. Job allocation unit 425 can track multiple scheduled tasks received from scheduler unit 420. In one embodiment, job allocation unit 425 manages a pending task pool and an active task pool for each GPC 450. When a GPC 450 completes the execution of a task, the task is removed from the active task pool of the GPC 450, and one of other tasks is selected and scheduled from the pending task pool for execution on the GPC 450. If an active task is idle on the GPC 450, for example, while waiting to resolve data dependencies, the active task can be removed from the GPC 450 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on the GPC 450.

[0101] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications executing on the host processor to schedule operations to be performed on the PPU 400. In one embodiment, multiple computing applications are executed concurrently by the PPU 400, and the PPU 400 provides isolation, Quality of Service (QoS), and independent address spaces for the multiple computing applications. Applications can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks for the PPU 400 to execute. The driver kernel outputs the tasks to one or more streams being processed by the PPU 400. Each task can include one or more sets of associated threads, referred to herein as warps. In one embodiment, a warp includes 32 associated threads that can execute in parallel. Cooperative threads can refer to multiple threads that include instructions for executing tasks and can exchange data via shared memory. Tasks can be assigned to one or more processing units within the GPC 450, and instructions are scheduled to be executed by at least one warp.

[0102] The work allocation unit 425 communicates with one or more GPCs 450 via XBar 470. XBar 470 is an interconnect network that couples a plurality of units of PPU 400 to other units of PPU 400. For example, XBar 470 can be configured to couple the work allocation unit 425 to a specific GPC 450. Although not explicitly shown, one or more other units of PPU 400 may also be connected to XBar 470 via hub 430.

[0103] Tasks are managed by scheduler unit 420 and scheduled to GPC 450 by job allocation unit 425. GPC 450 is configured to process tasks and generate results. Results can be used by other tasks within GPC 450, routed to different GPC 450 via XBar 470, or stored in memory 404. Results can be written to memory 404 via memory partitioning unit 480, which implements a memory interface for reading data from and writing data to memory 404. Results can be transferred to another PPU 400 or CPU via NVLink 410. In one embodiment, PPU 400 includes U number of memory partitioning units 480, which is equal to the number of independent and distinct memory devices of memory 404 coupled to PPU 400. Each GPC 450 may include a memory management unit to provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit provides one or more translation lookup buffers (TLBs) for translating virtual addresses into physical addresses in memory 404.

[0104] In one embodiment, memory partitioning unit 480 includes a raster operation (ROP) unit, a secondary (L2) cache, and a memory interface coupled to memory 404. The memory interface can implement 32, 64, 128, or 1024-bit data buses for high-speed data transfer. PPU 400 can connect to up to Y memory devices, such as high-bandwidth memory stacks or graphics dual data rate, version 5, synchronous dynamic random access memory, or other types of persistent memory. In one embodiment, the memory interface implements an HBM2 memory interface, where Y equals half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 400, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies, Y equals 4, each HBM2 stack includes two 128-bit channels, and each die has a total of eight channels with a 1024-bit data bus width.

[0105] In one embodiment, the memory 404 supports single error correction double detection (SECDED) error correction codes (ECC) to protect data. ECC provides higher reliability for computing applications sensitive to data corruption. Reliability is especially important in large-scale cluster computing environments because the PPU400 handles very large datasets and / or long-running applications.

[0106] In one embodiment, the PPU 400 implements a multi-level memory hierarchy. In one embodiment, memory partitioning unit 480 supports a unified memory to provide a single, unified virtual address space for the CPU and PPU 400 memory, thereby enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 400 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 400 that accesses pages more frequently. In one embodiment, NVLink 410 supports an address translation service, allowing the PPU 400 to directly access the CPU's page tables and providing the PPU 400 with full access to CPU memory.

[0107] In one embodiment, the replication engine transfers data between multiple PPUs 400 or between a PPU 400 and a CPU. The replication engine can generate page faults for addresses not mapped to page tables. The memory partitioning unit 480 can then service the page faults, mapping the addresses to page tables, after which the replication engine can perform the transfer. In traditional systems, memory is pinned (e.g., non-pageable) for multiple replication engine operations across multiple processors, significantly reducing available memory. Hardware page faults allow addresses to be passed to the replication engine without concern for whether memory pages reside, and the replication process is transparent.

[0108] Data from memory 404 or other system memory can be fetched by memory partitioning unit 480 and stored in L2 cache 460, which is located on-chip and shared among the various GPCs 450. As shown, each memory partitioning unit 480 includes a portion of the L2 cache associated with the corresponding memory 404. Lower-level caches can then be implemented in various units within the GPC 450. For example, each processing unit within the GPC 450 can implement a Level 1 (L1) cache. The L1 cache is dedicated memory for a specific processing unit. L2 cache 460 is coupled to memory interface 470 and XBar 470, and data from the L2 cache can be fetched and stored in each L1 cache for processing.

[0109] In one embodiment, the processing unit within each GPC 450 implements a SIMD (Single Instruction Multiple Data) architecture, where each thread in a group of threads (e.g., a warp) is configured to process a different dataset based on the same set of instructions. All threads in the thread group execute the same instructions. In another embodiment, the processing unit implements a SIMT (Single Instruction, Multiple Threads) architecture, where each thread in the thread group is configured to process a different dataset based on the same set of instructions, but individual threads in the thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, allowing concurrency between warps and serial execution within warps when threads within a warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby achieving equal concurrency among all threads, within warps, and between warps. When an execution state is maintained for each individual thread, threads executing the same instructions can aggregate and execute in parallel for maximum efficiency.

[0110] Cooperative groups are a programming model for organizing groups of communicating threads. They allow developers to represent the granularity of thread communication, enabling richer and more efficient parallel decomposition. The cooperative startup API supports synchronization between thread blocks to execute parallel algorithms. Traditional programming models provide a simple construct for synchronizing cooperative threads: barriers between all threads in a thread block (e.g., the `syncthreads()` function). However, programmers often prefer to define thread groups at a smaller granularity than thread blocks and synchronize within the defined groups for higher performance, design flexibility, and software reuse through a collective group-wide functional interface.

[0111] Collaboration groups enable programmers to explicitly define thread groups at both the sub-block (e.g., down to a single thread) and multi-block granularity, and perform collective operations, such as synchronization, on threads within the collaboration group. This programming model supports clean composition across software boundaries, allowing libraries and utility functions to be safely synchronized in their local contexts without assumptions about convergence. Collaboration group primitives support novel cooperative parallelism models, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire mesh of thread blocks.

[0112] Each processing unit comprises a large number (e.g., 128) of different processing cores (e.g., functional units), which can be fully pipelined, single-precision, double-precision, and / or mixed-precision, and include floating-point arithmetic logic units and integer arithmetic logic units. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the core comprises 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0113] Tensor cores are configured to perform matrix operations. Specifically, tensor kernels are configured to perform deep learning matrix algorithms, such as GEMM (matrix multiplication) used for convolution operations during neural network training and inference. In one embodiment, each tensor kernel operates on a 4x4 matrix and performs matrix multiplication and accumulation operations D = A × B + C, where A, B, C, and D are 4x4 matrices.

[0114] In one embodiment, matrix multiplication inputs A and B can be integer, fixed-point, or floating-point matrices, while accumulation matrices C and D can be integer, fixed-point, or floating-point matrices with equal or higher bit widths. In one embodiment, the tensor kernel operates on one-, four-, or eight-bit integer input data with a 32-bit integer accumulation. An 8-bit integer matrix multiplication requires 1024 operations and produces a full-precision product, which is then accumulated using 32-bit integer addition with other intermediate multiplications from an 8x16 matrix multiplication. In one embodiment, the tensor kernel operates on 16-bit floating-point input data with a 32-bit floating-point accumulation. A 16-bit floating-point multiplication requires 64 operations, resulting in a full-precision product, which is then accumulated using 32-bit floating-point addition with other intermediate multiplications from a 4x4 matrix multiplication. In practice, tensor kernels are used to perform larger two-dimensional or higher-dimensional matrix operations, which are composed of these smaller elements. APIs, such as the CUDA 9 C++ API, expose specialized matrix loading, matrix multiplication, and accumulation operations, as well as matrix storage operations, to efficiently utilize tensor kernels in CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that a 16x16 matrix spans all 32 threads of the warp.

[0115] Each processing unit may also include M Special Function Units (SFUs) that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, an SFU may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, an SFU may include a texture unit configured to perform texture mapping filtering operations. In one embodiment, a texture unit is configured to load a texture map (e.g., a 2D array of textures) from memory 404 and sample the texture map to produce sampled texture values ​​for use in a shader program executed by the processing unit. In one embodiment, the texture map is stored in shared memory, which may include or include a level-one cache. The texture unit implements texture operations, such as filtering operations using mip maps (e.g., texture maps at different levels of detail). In one embodiment, each processing unit includes two texture units.

[0116] Each processing unit also includes N Load Memory Units (LSUs) for performing load and store operations between shared memory and the register file. Each processing unit includes an interconnect network that connects each core to the register file and the LSUs to the register file's shared memory. In one embodiment, the interconnect network is a crossbar switch that can be configured to connect any core to any register in the register file and the LSUs to memory locations in both the register file and shared memory.

[0117] Shared memory is an on-chip memory array that allows data storage and communication between processing units and between threads within a processing unit. In one embodiment, shared memory includes 128KB of storage capacity and is located in the path from each processing unit to memory partition unit 480. Shared memory can be used to cache read and write operations. One or more of shared memory, L1 cache, L2 cache, and memory 404 are backup memory.

[0118] Combining data caching and shared memory functionality into a single memory block provides optimal overall performance for both types of memory access. This capacity can be used as a cache for programs that do not utilize shared memory. For example, if shared memory is configured to use half its capacity, texture and load / store operations can use the remaining capacity. This integration within shared memory enables it to function as a high-throughput pipeline for streaming data, providing high-bandwidth and low-latency access to frequently reused data.

[0119] When configured for general-purpose parallel computing, a simpler configuration than graphics processing can be used. In particular, by bypassing fixed-function graphics processing units, a much simpler programming model is created. In the general-purpose parallel computing configuration, the work allocation unit 425 directly allocates and assigns thread blocks to processing units within the GPCs 450. Threads execute the same program, using unique thread IDs in the computation to ensure each thread produces unique results. The processing units execute the program and perform computations, shared memory facilitates communication between threads, and the LSU reads and writes to global memory via shared memory and memory partitioning unit 480. When configured for general-purpose parallel computing, the processing units can also write commands to the scheduler unit 420 that can be used to start new jobs on the processing units.

[0120] Each PPU 400 may include and / or be configured to perform the functions of one or more processing cores and / or components thereof, such as a tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), ray tracing (RT) core, vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessors (SMs), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) elements, peripheral component interconnect (PCI) or peripheral component interconnect pass-through (PCIe) elements, etc.

[0121] The PPU 400 may be included in desktop computers, laptops, tablets, servers, supercomputers, smartphones (e.g., wireless handheld devices), personal digital assistants (PDAs), digital cameras, vehicles, head-mounted displays, handheld electronic devices, etc. In one embodiment, the PPU 400 is contained on a single semiconductor substrate. In another embodiment, the PPU 400 is contained in a system-on-a-chip (SoC) and one or more other devices, such as an additional PPU 400, memory 404, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.

[0122] In one embodiment, PPU 400 may be included on a graphics card that includes one or more storage devices. The graphics card may be configured to interface with a PCIe slot on a desktop computer motherboard. In another embodiment, PPU 400 may be an integrated graphics processing unit (iGPU) or a parallel processor included in a motherboard chipset. In another embodiment, PPU 400 may be implemented in reconfigurable hardware. In yet another embodiment, a portion of PPU 400 may be implemented in reconfigurable hardware.

[0123] Demonstration computing system

[0124] Systems with multiple GPUs and CPUs are used across various industries as developers expose and leverage greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens of thousands to thousands of compute nodes are deployed in data centers, research facilities, and supercomputers to tackle larger problems. As the number of processing devices in high-performance systems increases, communication and data transmission mechanisms need to be scaled to support the increased bandwidth.

[0125] Figure 5A According to the embodiment, using Figure 4 A conceptual diagram of a processing system 500 implemented by a PPU 400. An exemplary system 565 may be configured to implement the method 200 shown in FIG2. The processing system 500 includes a CPU 530, a switch 510, multiple PPUs 400, and corresponding memory 404.

[0126] The NVLink 410 provides a high-speed communication link between each PPU 400. Although in Figure 5BA specific number of NVLink 410 and interconnect 402 connections are shown, but the number of connections to each PPU 400 and CPU 530 can vary. Switch 510 interfaces between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLinks 410 may reside on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols to interface between various different connections and / or links.

[0127] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and CPU 530, and an interface of switch 510 between interconnect 402 and each PPU 400. The PPU 400, memory 404, and interconnect 402 may reside on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 and CPU 530, and switch 510 uses NVLink 410 to provide an interface between each PPU 400 to provide one or more high-speed communication links between PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between PPUs 400 and CPU 530 via switch 510. In yet another embodiment (not shown), interconnect 402 directly provides one or more communication links between each PPU 400. One or more of the NVLink 410 high-speed communication links can be implemented as physical NVLink interconnects or on-chip or on-chip interconnects using the same protocol as the NVLink 410.

[0128] In the context of this description, a single semiconductor platform can refer to a single, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform can also refer to a multi-chip module with increased connectivity, simulating on-chip operation and representing a substantial improvement over implementations using conventional buses. Of course, various circuits or devices can also be positioned individually or in various combinations of semiconductor platforms, depending on the user's expectations. Alternatively, the parallel processing module 525 can be implemented as a circuit board substrate, and each of the PPU 400 and / or memory 404 can be a packaged device. In one embodiment, the CPU 530, switch 510, and parallel processing module 525 reside on a single semiconductor platform.

[0129] In one embodiment, each NVLink 410 has a signaling rate of 20 to 25 gigabits per second, and each PPU 400 includes six NVLink 410 interfaces (e.g., Figure 5AAs shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 gigabits per second in each direction, and the six links provide a data transfer rate of 400 gigabits per second. When the CPU 530 also includes one or more NVLink 410 interfaces, the NVLink 410 can be dedicated to applications such as... Figure 5A The example shows PPU-to-PPU communication, or some combination of PPU-to-PPU and PPU-to-CPU communication.

[0130] In one embodiment, NVLink 410 allows direct load / store / atomic access from CPU 530 to memory 404 of each PPU. In one embodiment, NVLink 410 supports coherent operation, allowing data read from memory 404 to be stored in the cache hierarchy of CPU 530, thereby reducing cache access latency of CPU 530. In one embodiment, NVLink 410 includes support for Address Translation Service (ATS), allowing PPU 400 to directly access page tables within CPU 530. One or more of NVLink 410 may also be configured to operate in a low-power mode.

[0131] Figure 5B An exemplary system 565 is illustrated, in which various architectures and / or functions of various prior embodiments can be implemented. The exemplary system 565 can be configured to implement the method 200 shown in FIG2.

[0132] As shown, system 565 includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 may directly or indirectly couple one or more of the following devices: main memory 540, network interface 535, CPU 530, display device 545, input device 560, switch 510, and parallel processing system 525. The communication bus 575 may be implemented using any suitable protocol and may represent one or more links or buses, such as address bus, data bus, control bus, or combinations thereof. The communication bus 575 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Pass-Through (PCIe) bus, HyperTransport, and / or other types of buses or links. In some embodiments, there are direct connections between components. For example, CPU 530 may be directly connected to main memory 540. Furthermore, CPU 530 may be directly connected to parallel processing system 525. In cases where there is a direct or point-to-point connection between components, the communication bus 575 may include a PCIe link that performs the connection. In these examples, the PCI bus does not need to be included in system 565.

[0133] although Figure 5C The various modules are shown as being connected to the line via communication bus 575, but this is not intended to be limiting, but merely for clarity. For example, in some embodiments, a presentation component such as display device 545 can be considered as an I / O component such as input device 560 (e.g., if the display is a touchscreen). As another example, CPU 530 and / or parallel processing system 525 may include memory (e.g., main memory 540 may represent a storage device in addition to parallel processing system 525, CPU 530, and / or other components). In other words, Figure 5C The computing devices mentioned are merely illustrative. No distinction is made between "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types, as is the case in [the context of the previous sentence]. Figure 5C As envisioned within the scope of computing devices.

[0134] System 565 also includes main memory 540. Control logic (software) and data are stored in main memory 540, which may take the form of various computer-readable media. Computer-readable media can be any available medium accessible by system 565. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0135] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, main memory 540 may store computer-readable instructions (e.g., instructions representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and is accessible by system 565. As used herein, computer storage media itself does not include signals.

[0136] Computer storage media may contain computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals, such as carrier waves or other transmission mechanisms, and may include any information delivery medium. The term "modulated data signal" may refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media may include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included within the scope of computer-readable media.

[0137] When executing a computer program, system 565 is enabled to perform various functions. CPU 530 may be configured to execute at least some of computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Each CPU 530 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing multiple software threads simultaneously. CPU 530 may include any type of processor and may include different types of processors depending on the type of system 565 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of system 565, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (e.g., math coprocessors), system 565 may also include one or more CPUs 530.

[0138] In addition to, or selected from, CPU 530, parallel processing module 525 may be configured to execute at least some computer-readable instructions to control one or more components of system 565 to perform one or more methods and / or processes described herein. Parallel processing module 525 may be used by system 565 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, parallel processing module 525 may be used for general-purpose computing on a GPU (GPGPU). In embodiments, CPU 530 and / or parallel processing module 525 may execute any combination of methods, processes, and / or portions thereof discretely or jointly.

[0139] System 565 also includes input device 560, parallel processing system 525, and display device 545. Display device 545 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. Display device 545 may receive data from other components (e.g., parallel processing system 525, CPU 530, etc.) and output data (e.g., as images, video, sound, etc.).

[0140] Network interface 535 enables logical coupling of system 565 with other devices, including input device 560, display device 545, and / or other components, some of which may be built into (e.g., integrated into) system 565. Illustrative input device 560 includes microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. Input device 560 can provide a natural user interface (NUI) that processes air gestures, voice, or other user-generated physiological input. In some cases, input can be transmitted to appropriate network elements for further processing. NUI can enable any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (described in more detail below) associated with the display of system 565. System 565 may include depth cameras, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. In addition, system 565 may include an accelerometer or gyroscope capable of detecting motion (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by system 565 to present immersive augmented reality or virtual reality.

[0141] Furthermore, for communication purposes, system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) via network interface 535. System 565 can be included in a distributed network and / or cloud computing environment.

[0142] Network interface 535 may include one or more receivers, transmitters, and / or transceivers, enabling system 565 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communications. Network interface 535 may include components and functions to support communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0143] System 565 may also include auxiliary storage (not shown). Auxiliary storage 610 includes, for example, hard disk drives and / or removable storage drives, representing floppy disk drives, magnetic tape drives, optical disk drives, digital versatile disk (DVD) drives, recording devices, universal serial bus (USB) flash memory. The removable storage drive reads and / or writes to the removable storage unit in a well-known manner. System 565 may also include a hard-wired power supply, battery power supply, or a combination thereof (not shown). The power supply provides power to system 565 to enable its components to operate.

[0144] Each of the above modules and / or devices may even reside on a single semiconductor platform to form system 565. Alternatively, various modules may be positioned individually or in various combinations of semiconductor platforms, depending on the user's preference. While various embodiments have been described above, it should be understood that they are presented as examples only and not as limitations. Therefore, the breadth and scope of the preferred embodiments should not be limited by any of the exemplary embodiments described above, but should be defined only by the following claims and their equivalents.

[0145] Example network environment

[0146] A network environment suitable for implementing embodiments of the present invention may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 5A Processing system 500 and / or Figure 5B Implemented on one or more instances of the exemplary system 565—for example, each device may include similar components, features, and / or functions of the processing system 500 and / or the exemplary system 565.

[0147] Components of a network environment can communicate with each other via wired, wireless, or a combination of both. This network may include multiple networks, or networks of networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0148] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the server functionality described herein can be implemented on any number of client devices.

[0149] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, and combinations thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include web-based service software or applications. In embodiments, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software web application framework, for example, one that can use a distributed file system for large-scale data processing (e.g., "big data").

[0150] A cloud-based network environment can provide cloud computing and / or cloud storage to perform any combination (or one or more portions thereof) of the computing and / or data storage functions described herein. Any of these various functions can be distributed from a central or core server (e.g., servers in one or more data centers) to multiple locations, which may be located in a state, region, country, globally, etc. If the connection to the user (e.g., client device) is relatively close to the edge server, the core server may assign at least a portion of the functionality to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0151] Client devices may include Figure 5B Example processing system 500 and / or Figure 5CThe example system 565 includes at least some components, features, and functions. As an example and not a limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, device, consumer electronics device, workstation, edge device, any combination of these defined devices, or any other suitable device.

[0152] Machine Learning

[0153] Deep neural networks (DNNs) developed on processors such as the PPU 400 have been used for a wide range of purposes, from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technique that simulates the neural learning process of the human brain; it can continuously learn, become smarter, and provide more accurate results faster over time. Just as adults teach children to correctly identify and classify various shapes, eventually enabling them to recognize shapes without any guidance, deep learning or neural learning systems need to be trained on object recognition and classification because they are more intelligent and efficient at recognizing basic objects, occluded objects, and assigning context to objects.

[0154] At its simplest level, neurons in the human brain examine various inputs they receive, assign importance levels to each input, and pass the output to other neurons to take action. Artificial neurons, or perceptrons, are the most basic model of neural networks. In one example, a perceptron can receive one or more inputs representing various features of an object that the perceptron is being trained to recognize and classify, and assign a weight to each of these features based on their importance in defining the object's shape.

[0155] Deep neural network (DNN) models consist of multiple layers with numerous connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) and can be trained on large amounts of input data to solve complex problems quickly and with high accuracy. In one example, the first layer of a DNN model decomposes an input image of a car into different parts and looks for basic patterns such as lines and angles. The second layer combines the lines to find higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the vehicle type, and the final few layers generate labels for the input image, identifying the specific car brand and model.

[0156] Once trained, a DNN can be deployed and used in a process called inference to identify and classify objects or patterns. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten digits on a check deposited into an ATM, recognizing an image of a friend in a photograph, providing movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians and road hazards in self-driving cars, or translating human speech in real time.

[0157] During training, data flows through the DNN in the forward propagation phase until a prediction is generated that indicates the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and the weights of each feature are adjusted in the backpropagation phase until the DNN correctly labels the inputs in the training dataset and other inputs. Training complex neural networks requires significant parallel computing power, including floating-point multiplication and addition supported by a PPU 400. Inference is less computationally intensive than training, a latency-sensitive process in which the trained neural network is applied to new inputs it has never seen before to classify images, detect emotions, recognize suggestions, recognize and translate speech, and typically infer new information.

[0158] Neural networks rely heavily on matrix mathematical operations, and complex multi-layered networks require significant floating-point performance and bandwidth to improve efficiency and speed. The PPU 400, with thousands of processing cores optimized for matrix mathematical operations and delivering tens to hundreds of TFLOPS of performance, is a computing platform capable of providing the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0159] Furthermore, images generated using one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for recognizing objects and environments in the real world. Such images can include scenes of roads, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or real-world environments. Such images can be used to train, test, or certify DNNs in machines or robots for manipulating, processing, or modifying physical objects in the real world. Additionally, such images can be used to train, test, or certify DNNs in autonomous vehicles for navigating and moving vehicles in the real world. Furthermore, images generated using one or more of the techniques disclosed herein can be used to convey information to users of such machines, robots, and vehicles.

[0160] Figure 5CComponents of an exemplary system 555, which can be used to train and utilize machine learning according to at least one embodiment, are illustrated. As will be discussed, various components may be provided by various combinations of computing devices and resources, or by a single computing system controlled by a single entity or multiple entities. Furthermore, aspects may be triggered, initiated, or requested by different entities. In at least one embodiment, training of the neural network may be instructed by a provider associated with provider environment 506, while in at least one embodiment, training may be requested by a customer or other user accessing the provider environment via client device 502 or other such resources. In at least one embodiment, training data (or data to be analyzed by the trained neural network) may be provided by a provider, user, or third-party content provider 524. In at least one embodiment, client device 502 may be a vehicle or object navigating on behalf of a user, for example, it may submit requests and / or receive instructions to assist the device in navigation.

[0161] In at least one embodiment, the request can be submitted via at least one network 504 for receipt by a provider environment 506. In at least one embodiment, the client device can be any suitable electronic and / or computing device enabling a user to generate and send such requests, such as, but not limited to, desktop computers, laptop computers, computer servers, smartphones, tablet computers, game consoles (portable or otherwise), computer processors, computing logic, and set-top boxes. Network 504 can include any suitable network for transmitting requests or other such data, such as the Internet, intranet, Ethernet, cellular network, local area network (LAN), wide area network (WAN), personal area network (PAN), self-organizing network with direct wireless connection between peers, etc.

[0162] In at least one embodiment, a request may be received at interface layer 508, which in this example may forward data to training and inference manager 532. Training and inference manager 532 may be a system or service including hardware and software for managing request and service-related data or content. In at least one embodiment, training and inference manager 532 may receive a request to train a neural network and may provide data for the request to training module 512. In at least one embodiment, if the request is not specified, training module 512 may select an appropriate model or neural network to use and may train the model using relevant training data. In at least one embodiment, training data may be a batch of data stored in training data repository 514, received from client device 502, or obtained from third-party provider 524. In at least one embodiment, training module 512 may be responsible for training the data. The neural network may be any suitable network, such as a recurrent neural network (RNN) or convolutional neural network (CNN). Once the neural network is trained and successfully evaluated, the trained neural network may be stored in model repository 516, which may store different models or networks for users, applications, or services. In at least one embodiment, there may be multiple models for a single application or entity, which can be used based on many different factors.

[0163] In at least one embodiment, at a subsequent point in time, a request for content (e.g., path determination) or data, at least in part determined or influenced by a trained neural network, may be received from client device 502 (or another such device). This request may include, for example, input data to be processed using the neural network to obtain one or more inference or other output values, classifications, or predictions. Alternatively, in at least one embodiment, the input data may be received by interface layer 508 and directed to inference module 518, although different systems or services may also be used. In at least one embodiment, if inference module 518 is not already locally stored, it may obtain an appropriate training network, such as a trained deep neural network (DNN) as discussed herein, from model repository 516. Inference module 518 may provide data as input to the training network, which may then generate one or more inferences as outputs. This may, for example, include classification of instances of the input data. In at least one embodiment, the inference may then be sent to client device 502 for display or other communication with the user. In at least one embodiment, the user's context data may also be stored in a user context data repository 522, which may include data about the user that can be used as input to the network when generating inference or determining data to be returned to the user after obtaining an instance. In at least one embodiment, related data, which may include at least some input or inference data, may also be stored in a local database 534 for processing future requests. In at least one embodiment, the user may use account information or other information to access resources or functions of the provider environment. In at least one embodiment, user data may also be collected and used to further train the model to provide more accurate inference for future requests, if permitted and available. In at least one embodiment, requests from a machine learning application 526 executing on a client device 502 may be received through a user interface, and results may be displayed through the same interface. The client device may include resources for generating requests and processing results or responses, such as a processor 528 and a memory 562, and at least one data storage element 552 for storing data for the machine learning application 526.

[0164] In at least one embodiment, processor 528 (or the processor of training module 512 or inference module 518) will be a central processing unit (CPU). However, as mentioned above, resources in such an environment can utilize GPUs to process data for at least some types of requests. GPUs (such as the PPU300) have thousands of cores and are designed to handle large parallel workloads, thus becoming a popular tool in deep learning for training neural networks and generating predictions. While using GPUs for offline building allows for faster training of larger, more complex models, offline prediction generation means that on-demand input features cannot be used, or predictions must be generated for all features and stored in a lookup table to serve real-time requests. If the deep learning framework supports CPU mode and the model is small and simple enough to perform feedforwards on the CPU with reasonable latency, then services on CPU instances can host the model. In this case, training can be done offline on the GPU, and inference can be done in real-time on the CPU. If the CPU approach is not feasible, then services can run on GPU instances. However, because GPUs and CPUs have different performance and cost characteristics, running services that offload runtime algorithms to GPUs may require a different design than CPU-based services.

[0165] In at least one embodiment, video data may be provided from client device 502 for enhancement in provider environment 506. In at least one embodiment, video data may be processed on client device 502 for enhancement. In at least one embodiment, video data may be streamed from third-party content provider 524 and enhanced by third-party content provider 524, provider environment 506, or client device 502. In at least one embodiment, video data may be provided from client device 502 for use as training data in provider environment 506.

[0166] In at least one embodiment, supervised and / or unsupervised training may be performed by client device 502 and / or provider environment 506. In at least one embodiment, a set of training data 514 (e.g., classification or labeled data) is provided as input to be used as training data. In one embodiment, training dataset 514 may be used in a generative adversarial training configuration to train a generator neural network.

[0167] In at least one embodiment, the training data may include images of at least one human subject, avatar, or character to which the neural network will be trained. In at least one embodiment, the training data may include instances of at least one type of object for which the neural network is to be trained, and information identifying that type of object. In at least one embodiment, the training data may include a set of images, each image including a representation of an object type, wherein each image also includes or is associated with a label, metadata, classification, or other information fragment identifying the object type represented in the respective image. Various other types of data may also be used as training data, and may also include text data, audio data, video data, etc. In at least one embodiment, training data 514 is provided as training input to training module 512. In at least one embodiment, training module 512 may be a system or service including hardware and software, such as one or more computing devices executing a training application for training a neural network (or other model or algorithm, etc.). In at least one embodiment, training module 512 receives instructions or requests indicating the type of model to be used for training; in at least one embodiment, the model may be any suitable statistical model, network, or algorithm for such purposes, such as including artificial neural networks, deep learning algorithms, learning classifiers, Bayesian networks, etc. In at least one embodiment, training module 512 may select an initial model or other untrained model from an appropriate repository 516 and train the model using training data 514 to generate a trained model (e.g., a trained deep neural network) that can be used to classify similar types of data or generate other such inference. In at least one embodiment where training data is not used, an appropriate initial model can still be selected for training the input data of each training module 512.

[0168] In at least one embodiment, the model can be trained in a variety of different ways, depending in part on the type of model chosen. In at least one embodiment, a set of training data can be provided to the machine learning algorithm, wherein the model is a model artifact created by the training process. In at least one embodiment, each instance of the training data contains the correct answer (e.g., a classification), which may be referred to as the target or target attribute. In at least one embodiment, the learning algorithm looks for patterns in the training data that map input data attributes to targets, the answers to be predicted, and outputs a machine learning model that captures these patterns. In at least one embodiment, the machine learning model can then be used to obtain predictions for new data without a specified target.

[0169] In at least one embodiment, the training and inference manager 532 can select from a set of machine learning models, including binary classification, multi-class classification, generative, and regression models. In at least one embodiment, the type of model to be used may depend at least in part on the type of target to be predicted.

[0170] Graphics processing pipeline

[0171] In one embodiment, PPU 400 includes a graphics processing unit (GPU). PPU 400 is configured to receive commands specifying a shader program for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying the number of vertices of the primitive (e.g., in model space coordinates) and attributes associated with each vertex of the primitive. PPU 400 can be configured to process graphics primitives to generate frame buffers (e.g., pixel data for each pixel of a display).

[0172] The application writes model data of the scene (e.g., a set of vertices and attributes) to memory, such as system memory or memory 404. The model data defines each object that may be visible on the display. The application then makes API calls to the driver kernel, requesting the rendering and display of the model data. The driver kernel reads the model data and writes commands to one or more streams to perform operations that process the model data. These commands may reference different shader programs to be implemented on processing units within the PPU 400, including one or more vertex shaders, shell shaders, domain shaders, geometry shaders, and pixel shaders. For example, one or more processing units may be configured to execute a vertex shader program that processes multiple vertices defined by the model data. In one embodiment, different processing units may be configured to execute different shader programs simultaneously. For example, a first subset of processing units may be configured to execute a vertex shader program, while a second subset of processing units may be configured to execute a pixel shader program. The first subset of processing units processes the vertex data to produce processed vertex data and writes the processed vertex data to L2 cache 460 and / or memory 404. After the processed vertex data is rasterized (e.g., converted from 3D to 2D in screen space) to produce fragment data, a second subset of the processing units executes pixel shaders to produce processed fragment data, which is then mixed with other processed fragment data and written to the frame buffer in memory 404. The vertex shader and pixel shader programs can execute in parallel, pipelinedly processing different data from the same scene until all model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then sent to the display controller for display on the display device.

[0173] Figure 6A According to the embodiments, by Figure 4A conceptual diagram of a graphics processing pipeline 600 implemented by a PPU 400. The graphics processing pipeline 600 is an abstract flowchart of the processing steps for generating a two-dimensional computer-generated image from three-dimensional geometric data. It is well known that pipeline architectures can more efficiently perform long-latency operations by dividing operations into multiple stages, where the output of each stage is coupled to the input of the next successive stage. Therefore, the graphics processing pipeline 600 receives input data 601 from one stage of the graphics processing pipeline 600 to the next stage to generate output data 602. In one embodiment, the graphics processing pipeline 600 may represent a process... The graphics processing pipeline is defined by the API. Alternatively, the graphics processing pipeline 600 can be implemented within the context of the functionality and architecture of the previous and / or any subsequent figures.

[0174] like Figure 6A As shown, the graphics processing pipeline 600 includes a multi-stage pipeline architecture. These stages include, but are not limited to, a data assembly stage 610, a vertex shading stage 620, a primitive assembly stage 630, a geometry shading stage 640, a viewport scaling, hidden surface removal, and clipping (VSCC) stage 650, a rasterization stage 660, a fragment shading stage 670, and a raster operation stage 680. In one embodiment, input data 601 includes commands configuring the processing unit to implement the various stages of the graphics processing pipeline 600 and the geometric primitives (e.g., points, lines, triangles, quadrilaterals, triangle strips, or sectors, etc.) to be processed by these stages. Output data 602 may include pixel data (e.g., color data) copied into other types of surface data structures in a frame buffer or memory.

[0175] The data assembly stage 610 receives input data 601 specifying vertex data for higher-order surfaces, primitives, etc. The data assembly stage 610 collects vertex data in temporary memory or a queue, for example, by receiving a command from the host processor including a pointer to a buffer in memory and reading vertex data from the buffer. The vertex data is then sent to the vertex shading stage 620 for processing.

[0176] The vertex shading stage 620 processes vertex data by performing a set of operations (e.g., a vertex shader or procedure) once for each vertex. A vertex can be specified, for example, as a 4-coordinate vector associated with one or more vertex attributes (e.g., color, texture coordinates, surface normals, etc.).<x、y、z、w> The vertex shading stage 620 can manipulate individual vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 620 performs operations on vertex coordinates or other vertex attributes associated with a vertex. Such operations typically include lighting operations (e.g., modifying the vertex's color attributes) and transformation operations (e.g., modifying the vertex's coordinate space). For example, a vertex can be specified using coordinates in object coordinate space, and these coordinates can be transformed by multiplying them by a matrix that converts the coordinates from object coordinate space to world space or normalized device coordinate (NCD) space. The vertex shading stage 620 generates transformed vertex data that is transferred to the primitive assembly stage 630.

[0177] The primitive assembly stage 630 collects the vertices output by the vertex shader stage 620 and groups the vertices into geometric primitives for processing by the geometry shader stage 640. For example, the primitive assembly stage 630 can be configured to group every three consecutive vertices into geometric primitives (e.g., triangles) for transmission to the geometry shader stage 640. In some embodiments, specific vertices may be used for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip may share two vertices). The primitive assembly stage 630 sends geometric primitives (e.g., sets of associated vertices) to the geometry shader stage 640.

[0178] The geometry shading stage 640 processes geometric primitives by performing a set of operations (e.g., geometry shaders or procedures) on them. Subdivision operations can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 640 can subdivide each geometric primitive into a finer mesh of two or more geometric primitives for the rest of the graphics processing pipeline 600 to process. The geometry shading stage 640 then sends the geometric primitives to the viewport stage 650.

[0179] In one embodiment, the graphics processing pipeline 600 can operate within a streaming multiprocessor, with vertex shading stage 620, primitive assembly stage 630, geometry shading stage 640, fragment shading stage 670, and / or associated hardware / software sequentially executing processing operations. Once the sequential processing operations are complete, in one embodiment, the viewport SCC stage 650 can utilize the data. In one embodiment, the primitive data processed by one or more stages of the graphics processing pipeline 600 can be written to a cache (e.g., a level 1 cache, a vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 650 can access the data in the cache. In one embodiment, the viewport SCC stage 650 and rasterization stage 660 are implemented as fixed-function circuitry.

[0180] In the Viewport SCC stage 650, viewport scaling, culling, and clipping of geometry primitives are performed. Each surface rendered is associated with an abstract camera position. The camera position represents the position of the viewer looking at the scene and defines the view frustum that surrounds the scene objects. The view frustum may include a view plane, a back plane, and four clipping planes. Any geometry primitives located entirely outside the view frustum may be culled (e.g., discarded) because they do not contribute to the final rendered scene. Any geometry primitives partially inside and partially outside the view frustum can be clipped (e.g., converted into new geometry primitives enclosed within the view frustum). Furthermore, each geometry primitive may be scaled according to the depth of the view frustum. All potentially visible geometry primitives are then transferred to the rasterization stage 660.

[0181] Rasterization stage 660 converts 3D geometric primitives into 2D fragments (e.g., fragments usable for display, etc.). Rasterization stage 660 can be configured to construct a set of planar equations using the vertices of geometric primitives, from which various properties can be interpolated. Rasterization stage 660 can also compute an overlay mask of multiple pixels, which indicates one or more sampled locations of the geometric primitives at pixel intercepts. In one embodiment, a z-test can also be performed to determine whether the geometric primitives are occluded by other geometric primitives that have already been rasterized. Rasterization stage 660 generates fragment data (e.g., interpolated vertex properties associated with specific sample locations of each overlay pixel), which is then transferred to fragment shading stage 670.

[0182] Fragment shading stage 670 processes fragment data by performing a set of operations (e.g., fragment shader or procedure) on each fragment. Fragment shading stage 670 may generate pixel data (e.g., color values) for a fragment, for example, by performing lighting operations or sampling texture mapping using interpolated texture coordinates of the fragment. The pixel data generated by fragment shading stage 670 is then transferred to raster operation stage 680.

[0183] The rasterization stage 680 can perform various operations on the pixel data, such as performing alpha testing, stencil testing, and mixing the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the rasterization stage 680 has finished processing the pixel data (e.g., output data 602), the pixel data can be written to a rendering target such as a frame buffer, color buffer, etc.

[0184] It should be understood that the graphics processing pipeline 600 may include one or more additional stages as a supplement to or replacement of the aforementioned one or more stages. Various implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, the aforementioned one or more stages may be included, but in some embodiments, they are excluded from the graphics processing pipeline (e.g., geometry shading stage 640). Other types of graphics processing pipelines are considered to be within the scope of this disclosure. Additionally, any stage of the graphics processing pipeline 600 may be implemented by one or more dedicated hardware units within a graphics processor (such as PPU 400). Other stages of the graphics processing pipeline 600 may be implemented by programmable hardware units (e.g., processing units within PPU 400).

[0185] The graphics processing pipeline 600 can be implemented by an application executed by a host processor (such as a CPU). In one embodiment, the device driver can implement an application programming interface (API) that defines various functions that the application can use to generate graphics data for display. The device driver is a software program that includes multiple instructions controlling the operation of the PPU 400. The API provides an abstraction for the programmer, allowing the programmer to utilize dedicated graphics hardware (such as the PPU 400) to generate graphics data without the programmer utilizing the specific instruction set for the PPU 400. The application can include API calls routed to the device driver of the PPU 400. The device driver interprets the API calls and performs various operations in response to them. In some cases, the device driver can perform operations by executing instructions on the CPU. In other instances, the device driver can perform operations, at least partially, by initiating operations on the PPU 400 using the input / output interface between the CPU and the PPU 400. In one embodiment, the device driver is configured to utilize the hardware of the PPU 400 to implement the graphics processing pipeline 600.

[0186] To implement the various stages of the graphics processing pipeline 600, various programs can be executed within the PPU 400. For example, a device driver can launch a kernel on the PPU 400 to execute the vertex shading stage 620 on one or more processing units. The device driver (or the initial kernel executed by the PPU 400) can also launch other kernels on the PPU 400 to execute other stages of the graphics processing pipeline 600, such as the geometry shading stage 640 and the fragment shading stage 670. Furthermore, some stages of the graphics processing pipeline 600 can be implemented on fixed-function hardware, such as a rasterizer or data assembler implemented within the PPU 400. It should be understood that the results from one kernel can be processed by one or more intervening fixed-function hardware units before subsequent kernel processing on the processing unit.

[0187] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device may be directly coupled to the system or processor that generates or renders the image. In other embodiments, the display device may be indirectly coupled to the system or processor, for example, via a network. Examples of such networks include the Internet, mobile telecommunications networks, Wi-Fi networks, and any other wired and / or wireless network systems. When the display device is indirectly coupled, the image generated by the system or processor can be streamed to the display device over the network. For example, such streaming allows video games or other applications that render images to be executed in a server, data center, or cloud-based computing environment, and allows the transmission and display of rendered images on one or more user devices (e.g., computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separate from the server or data center. Therefore, the techniques disclosed herein can be used to enhance streaming images and services such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0188] Example game streaming system

[0189] Figure 6B This is an example system diagram of a game streaming system 605 according to some embodiments of the present disclosure. Figure 6B Including game server 603 (which may include...) Figure 5A Exemplary processing system 500 and / or Figure 5B (Similar components, features, and / or functions to the exemplary system 565), client device 604 (which may include components, features, and / or functions similar to those in the exemplary system 565), and client device 604 (which may include components, features, and / or functions similar to those in the exemplary system 565). Figure 5A Exemplary processing system 500 and / or Figure 5B The exemplary system 565 may contain similar components, features, and / or functions, and network 606 (which may be similar to the network described herein). In some embodiments of this disclosure, system 605 may be implemented.

[0190] In system 605, for a game session, client device 604 can simply receive input data in response to input from an input device, send the input data to game server 603, and receive encoded display data from the game server. In this way, computationally intensive computation and processing are offloaded to game server 603 (e.g., rendering of the game session's graphics output—especially ray or path tracing—is performed by the GPU of game server 603). In other words, the game session is streamed from game server 603 to client device 604, thereby reducing the requirements of client device 604 for graphics processing and rendering.

[0191] For example, regarding the instantiation of a game session, client device 604 can display frames of the game session on display 624 based on display data received from game server 603. Client device 604 can receive input from one of the input devices. Client device 604 can send input data to game server 603 via communication interface 621 and network 606 (e.g., the Internet), and game server 603 can receive input data via communication interface 618, process the input data, and transmit the data to the GPU to generate the rendering of the game session. For example, the input data can represent the movement of a user character in the game, firing weapons, reloading, passing the ball, turning vehicles, etc. Rendering component 612 can render the game session (e.g., the result representing the input data), and rendering capture component 614 can capture the rendering of the game session as display data (e.g., as image data capturing the rendered frames of the game session). The rendering of a game session may include ray or path-traced lighting and / or shadow effects, computed using one or more parallel processing units (such as GPUs) of a game server. The GPUs may further utilize one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques. An encoder 616 can then encode the display data to generate encoded display data, which can be sent over a network to a client device 604. The client device 604 can receive the encoded display data via a communication interface 621, and a decoder 622 can decode the encoded display data to generate display data. The client device 604 can then display the display data via a display 624.

[0192] Note that the techniques described herein may be embodied in executable instructions stored in a computer-readable medium for use or in conjunction with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will understand that, for some embodiments, various types of computer-readable media may be included for storing data. As used herein, “computer-readable medium” includes one or more of any suitable media for storing executable instructions of a computer program, such that an instruction execution machine, system, apparatus, or device can read (or retrieve) those instructions from the computer-readable medium and execute the instructions for performing the embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer floppy disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable optical discs (CDs), portable digital video discs (DVDs), etc.

[0193] It should be understood that the arrangement of components shown in the accompanying drawings is for illustrative purposes only, and other arrangements are possible. For example, one or more elements described herein may be implemented wholly or partially as electronic hardware components. Other elements may be implemented in software, hardware, or a combination of software and hardware. Furthermore, some or all of these other elements may be combined, some elements may be omitted entirely, and additional components may be added while still achieving the functionality described herein. Therefore, the subject matter described herein can be embodied in many different variations, and all such variations are contemplated within the scope of the claims.

[0194] To facilitate understanding of the subject matter described herein, many aspects are described in order of action. Those skilled in the art will recognize that various actions can be performed by dedicated circuitry or circuits, program instructions executed by one or more processors, or a combination of both. The description of any sequence of actions herein does not imply that a specific order for performing that sequence must be followed. Unless otherwise stated herein or there is a clear contradiction in the context, all methods described herein can be performed in any suitable order.

[0195] In the context of describing the subject matter (particularly in the context of the following claims), the use of the terms “a” and “an”, as well as “The” and similar references, should be interpreted to cover both the singular and plural, unless otherwise stated herein or obviously contradicted by the context. The use of the term “at least one” followed by a list of one or more items (e.g., “at least one of a and B”) should be interpreted to refer to one item selected from the listed items (a or B) or any combination of two or more listed items (a and B), unless otherwise stated herein or obviously contradicted by the context. Furthermore, the foregoing description is for illustrative purposes only and not for limiting purposes, as the scope of protection sought is defined by the claims described below and any equivalents thereof. The use of any and all example or exemplary language provided herein (e.g., “for example”) is intended only to better illustrate the subject matter and does not constitute a limitation on the scope of the subject matter unless otherwise required. The use of the terms “based on” and other similar phrases in the claims and written description to indicate conditions that produce the result does not imply the cancellation of any other conditions that produce that result. No language in the specification should be construed as indicating that any element not protected by the claims is essential for the implementation of the invention protected by the claims.

Claims

1. A computer-implemented method, comprising: Identify one or more dead zones associated with at least one of a plurality of streaming sessions, wherein the at least one streaming session corresponds to at least a portion of a software stack executed on a virtual machine on a host computing device; For each dead zone identified for a specific streaming session, identify events occurring near the dead zone in event data generated by one or more attached virtual machines of the host computing device; For each of the multiple event types, determine the event distribution of said event type relative to the identified dead zone; and The operation of the host computing device is adjusted based on the event distribution of the multiple event types.

2. The computer-implemented method of claim 1, wherein the software stack includes multiple stages of a game streaming service, the game streaming service including at least one of a frame capture stage, an encoding stage, and a transmission stage.

3. The computer-implemented method according to claim 2, wherein the host computing device executes a second virtual machine configured to execute a game application, wherein, The game application is configured to generate image frames, which are processed by the software stack to send a bitstream to a client device, the client device being configured to display the image frames to a user of the game streaming service, and wherein the client device is configured to send input from the client device to the game application to render subsequent frames of the game application.

4. The computer-implemented method of claim 1, wherein identifying events occurring near the dead zone in the event data generated by one or more additional virtual machines of the host computing device comprises: For each event in the event data, calculate the difference between the timestamp associated with the event and the timestamp associated with each identified dead zone; and For a given dead zone, mark each event in which the absolute value of the difference is less than a threshold associated with the dead zone.

5. The computer-implemented method of claim 1, wherein determining the event distribution of the event type relative to the time of the corresponding dead zone comprises: A histogram is generated by counting the number of occurrences of each event type within multiple time periods of a time window surrounding the corresponding dead zone.

6. The computer-implemented method of claim 5, wherein adjusting the operation of the host computing device based on the event distribution of the plurality of event types comprises: The event distribution for determining a specific event type includes the number of events in a specific time period, wherein the number of events exceeds the average number of events in each time period by a threshold amount; and The operation is identified based on the specific event type.

7. The computer-implemented method of claim 1, wherein the operation of adjusting the host computing device includes migrating a second virtual machine on the host computing device to a second host computing device.

8. The computer-implemented method of claim 1, wherein adjusting the operation of the host computing device includes migrating data corresponding to the event type from a first storage device to a second storage device.

9. The computer-implemented method of claim 1, wherein identifying one or more dead zones associated with at least one of the plurality of streaming sessions comprises: The timestamps included in the metadata used for the at least one streaming session are sorted into a series; Calculate the difference between each timestamp in the series and the previous timestamp; and Each difference is compared to a threshold, where any difference greater than the threshold indicates an identified dead zone.

10. A system comprising: At least one host computing device, each host computing device being configured to launch at least one virtual machine, the virtual machine being configured to process the content of one or more streaming sessions, wherein each streaming session corresponds to a software stack executed on the virtual machine of the host computing device, the host computing device being configured to generate content for display by a corresponding client device; and The analytics engine is configured as follows: Identify dead zones associated with at least one of multiple streaming sessions; For each dead zone identified for a specific streaming session of a specific host computing device, identify events occurring near the dead zone in event data generated by one or more additional virtual machines of the specific host computing device; For each of the multiple event types, determine the event distribution of said event type relative to the identified dead zone; and The operation of the at least one host computing device is adjusted based on the event distribution of the multiple event types.

11. The system of claim 10, wherein the software stack includes multiple stages of a game streaming service, the game streaming service including at least one of a frame capture stage, an encoding stage, and a transmission stage.

12. The system of claim 11, wherein the host computing device executes a second virtual machine configured to execute a game application, wherein the game application is configured to generate image frames, the image frames being processed by the software stack to send a bitstream to a client device, the client device being configured to display the image frames to a user of the game streaming service, and wherein, The client device is configured to send input from the client device to the game application to render subsequent frames of the game application.

13. The system of claim 10, wherein identifying events occurring near the dead zone in the event data generated by one or more additional virtual machines of the host computing device comprises: For each event in the event data, calculate the difference between the timestamp associated with the event and the timestamp associated with each identified dead zone; and For a given dead zone, mark each event in which the absolute value of the difference is less than a threshold associated with the dead zone.

14. The system of claim 10, wherein determining the event distribution of the event type relative to the time of the corresponding dead zone comprises: A histogram is generated by counting the number of occurrences of each event type within multiple time periods of a time window surrounding the corresponding dead zone.

15. The system of claim 14, wherein adjusting the host computing device based on the event distribution of the plurality of event types comprises: The event distribution for determining a specific event type includes the number of events in a specific time period, wherein the number of events exceeds the average number of events in each time period by a threshold amount; and The operation is identified based on the specific event type.

16. The system of claim 10, wherein the operation of adjusting the host computing device includes migrating a second virtual machine on the host computing device to a second host computing device.

17. The system of claim 10, wherein adjusting the operation of the host computing device includes migrating data corresponding to the event type from a first storage device to a second storage device.

18. The system of claim 10, wherein identifying one or more dead zones associated with at least one of the plurality of streaming sessions comprises: The timestamps included in the metadata used for the at least one streaming session are sorted into a series; Calculate the difference between each timestamp in the series and the previous timestamp; and Each difference is compared to a threshold, where any difference greater than the threshold indicates an identified dead zone.

19. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a step comprising the following steps: For at least one of a plurality of streaming sessions, identify one or more dead zones associated with the at least one streaming session, wherein each streaming session corresponds to a software stack executed on a virtual machine on a host computing device; For each dead zone identified for a specific streaming session, identify events occurring near the dead zone in event data generated by one or more attached virtual machines of the host computing device; For each of the multiple event types, determine the event distribution of said event type relative to the identified dead zone; and The operation of the host computing device is adjusted based on the event distribution of the multiple event types.

20. The non-transitory computer-readable medium of claim 19, wherein the software stack comprises multiple stages of a game streaming service, the game streaming service comprising at least one of a frame capture stage, an encoding stage, and a transmission stage.

Images