Evaluating Qualitative Streaming Experience Using Session Performance Metadata

Abstract

Disclosed herein is a method for evaluating qualitative streaming experiences using session performance metadata. A pipeline of a streaming service may be adapted to collect metadata, such as timestamps, from various components of the pipeline. The metadata may then be analyzed to compute an objective quality metric for each streaming session using weighted scores derived from the metadata for a number of different components, including but not limited to lags, latency, and / or picture quality. The quality metric is designed to have a high correlation with subjective quality measurements by users of the streaming service, but to provide a dense data sample compared to the typical sparse responses collected from user feedback (e.g., user surveys). The objective quality metric may be used to quickly adjust streaming service parameters, either manually or automatically, to improve the quality of the streaming service due to, for example, changes in streaming content.

Description

Technical Field

[0001] The present disclosure relates to data analytics. More specifically, the embodiments set forth below describe techniques for evaluating qualitative metrics of a streaming experience associated with a streaming session based on performance metadata captured by a pipeline. Background Art

[0002] Cloud-based services have been developed for a variety of applications, such as network storage, virtual computing resources, and the like. Popular video-on-demand (VOD) streaming services deliver videos to client devices (e.g., televisions or tablet computers). Streaming video presents certain challenges that require a minimum quality of service (QoS) to ensure an adequate user experience. For example, low network bandwidth or high network congestion can cause the video to be delayed, resulting in repeated buffering of the video on the client side, which prevents a seamless viewing experience.

[0003] New cloud-based services are being developed for streaming video games and / or virtual reality / augmented reality graphics to client devices over a network. These applications receive feedback (e.g., controller feedback, inertial feedback, etc.) from the client side to function. Frames for displaying graphical output on a display device or head-mounted display (HMD) can be rendered on a server and transmitted over the network to the client device for display. Feedback from the client device is transmitted to the server device via the network. This feedback signal is used by the server device to adjust subsequent frames delivered to the client device.

[0004] Managers of these cloud-based services are constantly striving to improve the experience for their users. Network congestion, the configuration of the rendering pipeline, the encoding or decoding of images, etc. can all contribute to user-perceived latency issues. For example, a long delay (also known as "lag") between an input to a controller at a client device and viewing a response to that input on the client device's display can degrade the user's experience. Delays can be caused by network congestion from the client to the server, long latency in the rendering pipeline, long encoding procedures, network congestion from the server to the client, or long decoding procedures.

[0005] However, defining and measuring the quality of a streaming session for a user can be difficult. Different users can have very different perceptions of the same experience. For example, a user with a higher-performance client device may perceive the experience as worse than a user with a lower-performance client device simply because the user with the higher-performance device may expect better quality of the displayed image due to the hardware utilized by the client device. Similarly, users in different regions of the world may have different expectations simply based on the public infrastructure available in their area.

[0006] A common method for evaluating the quality of a streaming session is to request user feedback at the end of the session. For example, a client device may prompt the user to enter a rating (e.g., a star rating from 1 to 5 stars) that indicates the user's subjective level of satisfaction with the streaming session. However, there are problems with collecting subjective feedback in this way. First, feedback may be collected sparsely (e.g., weekly or every 10 sessions, etc.) so that the user does not become annoyed by repeated feedback prompts. Second, the subjective feedback provided by the user may not be the most accurate. Some users may simply select 5 stars, regardless of their experience, to quickly dismiss the feedback prompt. As another example, even if a particular user had a bad experience due to, for example, network congestion, the user may still rate the session highly if the user has a poor internet connection in their location and is accustomed to poor network performance.

[0007] It is desirable to develop objective measures of the quality of streaming sessions that are highly correlated with subjective quality measures provided by sparse user feedback. Objective measures can be population-dense (e.g., provided for each streaming session and for all streaming sessions) and remove bias from individual end users, which can vary significantly from user to user. Therefore, new techniques or methods are needed for assessing consumer satisfaction with pipelines for streaming services. Summary of the Invention

[0008] A method, computer-readable medium, and system for evaluating a qualitative streaming experience using session performance metadata are disclosed. A streaming service pipeline can be adapted to collect metadata, such as timestamps, from various components of the pipeline. The metadata can then be analyzed to compute a quality metric for each streaming session using weighted scores derived from the metadata for a number of different components, including, but not limited to, jank, latency, and / or picture quality.

[0009] In a first aspect of the present disclosure, a system for evaluating the quality of a streaming session provided by a streaming service is described. The system includes one or more processors configured to: receive metadata for a plurality of frames included in the streaming session; calculate, for each of the plurality of frames, a score and a weight for each of a plurality of components derived from the metadata; calculate a quality metric for the streaming session based on the scores and weights for the plurality of components; and adjust at least one component of a pipeline implemented by the streaming service based on the quality metric.

[0010] In some embodiments, the quality metric is calculated as the sum of a picture quality score and a combined stutter-latency score. The combined stutter-latency score may be calculated by: calculating, for each of the plurality of frames, a rolling weighted score for each of at least one latency component and at least one stutter component based on a first sliding window; and calculating, for each of the at least one latency component and at least one stutter component, a rolling mean of a combined weighted sum of the rolling weighted scores, wherein the rolling mean of the combined weighted sum is based on a second sliding window.

[0011] In some embodiments, the combined jam-latency score is calculated as the difference between a total weighted mean calculated based on the first set of parameters and a standard deviation based on the second set of parameters.

[0012] In some embodiments, the picture quality score is calculated as a difference between a mean of the plurality of picture quality scores for the plurality of frames and a product of a coefficient and a standard deviation of the plurality of picture quality scores for the plurality of frames.

[0013] In some embodiments, adjusting the at least one component of the pipeline implemented by the streaming service based on the quality metric comprises adjusting at least one of a quantization parameter, a target bitrate and encoding parameters, or a target resolution of the frames generated by the pipeline.

[0014] In some embodiments, the one or more processors are further configured to generate a visual representation of the quality metric of the streaming session.

[0015] In some embodiments, the multiple components include at least one of the following components: a first jitter component based on a delta between a first timestamp corresponding to a frame rendered by an application and a second timestamp corresponding to the frame being captured by the first stage of the pipeline; a second jitter component based on a delta between a first timestamp corresponding to presenting the frame on a client device and a second timestamp corresponding to presenting a previous frame on the client device; a first latency component based on a round-trip delay of the frame; and a second latency component based on a stack latency of the frame.

[0016] In a second aspect of the present invention, a method for evaluating a quality metric for a streaming session is described. The method comprises the steps of receiving metadata for a plurality of frames included in the streaming session, calculating, for each of the plurality of frames, a score and a weight for each of a plurality of components derived from the metadata; and calculating a quality metric for the streaming session based on the scores and weights for the plurality of components. The metadata comprises information related to the number of stages in a pipeline implemented by a streaming service, wherein the pipeline comprises at least one stage implemented by a server device and at least one additional stage implemented by a client device connected to the server device via a network.

[0017] In some embodiments, the method further comprises the step of generating a visual representation of the quality metric for one or more streaming sessions.Alternatively, the method comprises the step of adjusting at least one component of the pipeline based on the quality metric.

[0018] In a third aspect of the present invention, a non-transitory computer-readable medium stores computer instructions that, when executed by one or more processors, cause the one or more processors to perform the method of the second aspect.

[0019] It will be appreciated that features or embodiments of one aspect may be combined with features of embodiments of another aspect of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A pipelined stack for cloud-based services is shown in accordance with some embodiments.

[0021] Figure 2 A flow chart illustrating a method for evaluating a quality metric for a session according to some embodiments is shown.

[0022] Figure 3 A parallel processing unit according to one embodiment is shown.

[0023] Figure 4A According to one embodiment, Figure 3 A general-purpose processing cluster within a parallel processing unit.

[0024] Figure 4B According to one embodiment, Figure 3 A memory partitioning unit of a parallel processing unit.

[0025] Figure 5A According to one embodiment, Figure 4A Streaming multiprocessor.

[0026] Figure 5B According to one embodiment, the Figure 3Conceptual diagram of the processing system implemented by the PPU.

[0027] Figure 5C An exemplary system is shown that can implement the various architecture and / or functionality of various previous embodiments.

[0028] Figure 6A According to one embodiment, Figure 3 Conceptual diagram of the graphics processing pipeline implemented by the PPU.

[0029] Figure 6B is an example system diagram of a game streaming system according to some embodiments.

[0030] Figure 7 Logic for calculating a rolling weighted mean metric for components derived from streaming session metadata is shown in accordance with some embodiments.

[0031] Figure 8A and Figure 8B A pair of transfer functions for the stutter component of a quality metric is shown, according to some embodiments.

[0032] Figure 9A and Figure 9B A pair of transfer functions for the delay component of a quality metric is shown in accordance with some embodiments.

[0033] Figure 10 Logic is shown for computing a combined score representing the quality of a streaming session from multiple per-frame component scores, in accordance with some embodiments.

[0034] Figure 11 Logic for generating quality metrics for a streaming session is shown in accordance with some embodiments.

[0035] Figure 12 A transfer function mapping a target bitrate to a picture quality score according to some embodiments is shown. DETAILED DESCRIPTION

[0036] The pipeline for cloud-based services is configured to track the progress of content being processed by the pipeline by updating metadata corresponding to the content. Each stage of the pipeline implemented by the server device can insert a timestamp into the metadata when processing of the content by that stage is complete. Upon receiving the content and metadata, the client device can update the metadata of any additional stages of the pipeline implemented by the client device and then transmit the updated metadata back to the server device.

[0037] A server device may analyze metadata of a streaming session in order to calculate an objective quality metric for the streaming session. It will be appreciated that conventional methods of characterizing a streaming session are somewhat unreliable or intrusive to end users, and therefore, a new method for objectively calculating a quality metric based on metadata collected by a pipeline for a streaming session is needed. The techniques disclosed herein for calculating a quality metric are based on various components of the metadata that are selected so as to correlate the objective quality metric with subjective quality scores provided by multiple end users, such as may be determined by statistical analysis of multiple configurations or sets of different components. The resulting objective quality metric has a good correlation with the subjective quality metric provided by the end users, but does not require any feedback from the end users because all input variables for the quality metric can be derived from the metadata automatically collected by the pipeline.

[0038] The use of this objective quality metric can help administrators or designers of cloud-based services (such as game streaming platforms) adjust the components of the pipeline to improve the experience of all users of the service. For example, a game streaming pipeline can be configured to deliver videos corresponding to more than one type of game application. When a new game is added to a streaming service, it may be necessary to adjust the pipeline to match the characteristics of the new game. The pipeline can be adjusted to reduce delay by changing the frame rate or quantization parameters of the encoding stage of the pipeline to improve the experience and consider the increased rendering time of the new game application. In the absence of any feedback, it may be difficult for the administrator to determine what parameters should be adjusted and whether the new game application performs better or worse than the previous game application. Information provided by quality metrics (such as the quality metrics described below) and based on objective data collected by the streaming service on both the server side and the client side can greatly improve the user experience of the platform. In some cases, quality metrics can also be used to automatically adjust the parameters of the pipeline to improve the functionality of the pipeline, thereby reducing the burden on network administrators.

[0039] Figure 1 A pipelined stack 100 for a cloud-based service according to some embodiments is shown. The cloud-based service is implemented as a pipelined stack 100 of multiple stages. Each stage includes one or more components, and the output of one stage is received at the input of the subsequent stage, except for the last stage. Figure 1As shown, a pipelined stack 100 for a cloud-based game streaming service includes multiple stages. The server side 102 implements multiple stages on one or more server devices, including a frame capture stage 110, an encoding stage 120, a forward error correction (FEC) stage 130, and a transmission stage 140. The pipelined stack 100 also includes a network stage 150, which includes the transmission of data packets from the server side 102 to the client side 106 over the network 104. The network stage 150 is executed on one or more network devices (such as network routers, switches, access points, etc.). The client side 106 implements multiple stages on one or more client devices, including a decoding stage 160, a rendering stage 170, and a presentation stage 180.

[0040] It will be appreciated that for purposes of illustrating the various techniques described herein, Figure 1 An exemplary pipelined stack 100 for a game streaming service is depicted in FIG. In other embodiments, the specific arrangement of the stages in the pipelined stack 100 may be different, for example by including Figure 1 Additional stages not shown or omitted Figure 1 For example, when a service does not implement FEC, FEC stage 130 may be omitted. Additionally, some implementations may combine aspects of two or more stages into a single stage (e.g., FEC and transport may be combined into a single stage of pipelined stack 100).

[0041] In addition, although Figure 1 It is not explicitly shown in the figure, but the initial stage of the pipelined stack 100 (frame capture stage 110) interacts with one or more applications (such as a gaming application) that generate video frames to be streamed to the client device. The gaming application can receive input from the client device and render each subsequent frame of video based on the input. The video frames can be stored in a memory accessible to the frame capture stage 110, which monitors the memory or receives a signal from the gaming application each time a new video frame is ready to be streamed to the client. It will be understood that in some embodiments, the first stage of the pipelined stack 100 may be adapted to include a gaming application that repeatedly generates new frames that are transmitted to the second encoding stage 120 of the pipelined stack 100.

[0042] Figure 1Each stage shown in can include one or more components. For example, the encoding stage 120 can include a queue component that asynchronously receives frames from the frame capture stage 110 while the previous frame is being encoded by the encoder component. The encoding stage 120 can also include a send queue component that asynchronously receives encoded frames from the encoder component and transmits the encoded frames to the FEC stage 130 when the encoded frames are available and the FEC stage 130 is ready. As will be discussed in more detail, metadata for content processed by the stages of the pipelined stack 100 can be updated by each stage, or alternatively, by each component within a stage, such that multiple timestamps are appended to the metadata by at least one stage of the pipelined stack 100.

[0043] In an embodiment, the content includes one or more frames of rendered video, and metadata is generated for each frame. As used herein, a frame may refer to a data structure that encodes image data for display on a display device. The frame may include an array of pixel values, where each pixel value includes one or more channels of color information. For example, a pixel value may include a red channel, a blue channel, and a green channel that indicate the color of the pixel via 8 or 10 bits. Of course, in other embodiments, the pixel values may be encoded in a different format, such as by encoding luminance and chrominance information in separate channels. In some embodiments, each component or stage of the pipelined stack 100 may be configured to update the metadata corresponding to the processed frame by adding a timestamp to the metadata when that stage / component has completed processing the frame. The metadata is then forwarded to the next component / stage along with the processed content. Furthermore, it will be appreciated that transmitting a video frame to a client device may refer to transmitting one or more data packets to the client device, where the frame data may be included in the payload of the one or more data packets.

[0044] In the context of a gaming application executed by a cloud-based service, where rendered frames of the game are delivered to a client device via a network, the metadata for each frame may include timestamps indicating when the frame was rendered by the gaming application on the server device, when the frame was captured (e.g., loaded from memory used by the gaming application) via the frame capture stage 110, when encoding of the frame was completed via the encoding stage 120, when the frame was transmitted over the network via the server device's network interface at the transmission stage 140, when the client device received the frame via the network stage 150, when the client device has decoded the frame via the decoding stage 160, when the decoded frame was processed to generate an image for display on the client device via the rendering stage 170, or when the image was presented on the display device 180, etc. In some cases, the metadata may include other timestamps for each frame, such as timestamps associated with input (e.g., keystroke information, mouse event information, or controller information) from the client side 106 used to render the frame by the gaming application on the server side 102 prior to frame processing by the pipelined stack 100. Each of these timestamps for a frame can be combined with corresponding timestamps for other frames (e.g., previous or subsequent frames) to generate delta values representing the original values of the various components of the quality metric. As used herein, a delta value refers to the difference between two timestamps. In some cases, the delta value can be shifted based on the expected time between two subsequent frames, which is based on the target frame rate. Thus, if the difference between the two timestamps is less than the expected time between frames, the delta value can be negative, and if the difference between the two timestamps is greater than the expected time between frames, the delta value can be positive. The expected time between frames refers to the average time it takes to display each frame given the target frame rate. For example, for a 60 Hz frame rate, the expected time between frames is approximately 16.6 ms.

[0045] At the end of the pipelined stack 100, the client device transmits the metadata back to the server device for storage and / or analysis. In one embodiment, the metadata for the frame returned to the server device includes multiple timestamps, each timestamp corresponding to a time indicating the time at which processing of the frame was completed by a particular stage or component of the stage. In some embodiments, the client device is configured to collect metadata for multiple frames (e.g., 60 seconds of video at 30-60 frames per second) in a data structure (e.g., a file) before transmitting the data structure (e.g., a file) to the server device. In such an embodiment, the server device collects files for multiple sessions, each session being, for example, the number of seconds of video. It will be appreciated that the size of the session can be adjusted to match the desired size of the collected metadata, such as by collecting 30 to 60 minutes worth of metadata in the session. In some embodiments, the client device collects metadata in a circular buffer so that the metadata collected for a session is only metadata for the last n frames in the session. For example, a circular buffer sized to collect 10 minutes of data at 60 frames per second (e.g., equivalent to 36,000 frames of metadata) can continuously collect metadata during a session, overwriting the oldest metadata when the session extends beyond 10 minutes in length. At the end of the session (e.g., when the gaming application terminates), the client device can encode the contents of the circular buffer and transmit it to the server device. It will be appreciated that when a user experiences an issue that degrades the user experience, the session will likely terminate early, and as such, the last m minutes of the session (corresponding to n frames) are typically sufficient to highlight the cause of the issue that degraded the user experience. In other embodiments, the circular buffer can exist on the server side of the pipeline. The client device can be configured to send metadata for each frame back to the server device, which is then stored in the circular buffer. At the end of the session, the server device can process the contents of the circular buffer. In other embodiments, the metadata added by each stage of the pipeline can be stripped off by the server device and temporarily stored there, so that the metadata is not forwarded to the client device along with the content. Then, as content is processed on the client side of the pipeline, new metadata is generated at the stages of the pipeline at the client device, and when the content is processed at the client device, the metadata is encoded and transmitted to the server device. The metadata from the client device is then combined with the corresponding metadata from the server device and stored in a circular buffer at the server device. It will be appreciated that any technique for collecting metadata at the server device (including having each stage individually transmit its corresponding metadata back to the server device) is within the scope of the present disclosure.

[0046] In an embodiment, the development of a quality metric relies on three different objective measurements. The first measurement relates to picture quality based on a target bitrate for the video stream. The parameter "bits per pixel" can be calculated by dividing the target bitrate (e.g., bits per second) by the product of the height of the frame in pixels, the width of the frame in pixels, and the frame rate (e.g., frames per second). In other embodiments, picture quality can be based on other objective measurements, such as the resolution and quantization parameters of the encoded frames.

[0047] The second measurement includes two components related to latency. The first component is related to round-trip delay (RTD), and the second component is related to stack latency. RTD measures the delay across the network and includes the delay from the server to the client and the delay from the client to the server. In one embodiment, when a frame (e.g., a data packet of a frame) is received from the server device, a first timestamp of the client device clock is captured (e.g., a timestamp attached to the metadata when the network stage 150 is completed), and a second timestamp of the client device clock is captured when the frame is presented by the client device (e.g., on a display attached to or included in the client device) (e.g., a timestamp attached to the metadata when the presentation stage 180 is completed). The difference between the second timestamp and the first timestamp is added to a third timestamp attached to the frame metadata, which represents the time associated with the server device clock (e.g., the time when the frame is transmitted to the client device at the end of the transmission stage 140), which is returned to the server device. RTD can be calculated by sampling the server device clock when receiving the metadata and subtracting the sum of the third timestamps from the difference between the second timestamp and the first timestamp. In other words, RTD represents the round-trip delay corresponding to the network delay to and from the server device relative to the clock maintained in the server device, but excludes processing delays at the client device (e.g., the difference between the first timestamp and the second timestamp). Stack latency is the difference between the time when input (e.g., keystroke, mouse input, controller input, etc.) is captured on the client side and the time when the frame containing the input is displayed on the client device. Stack latency can be generally referred to as a lag measurement. In an embodiment, the client device captures a timestamp corresponding to the capture of the input. Using a short-term history of the rendering rate of the application, the next frame generated by the application executed by the server can be evaluated so as to associate the timestamp corresponding to the input capture event at the client with the frame rendered by the server. The server then waits to receive a timestamp from the client indicating the time when the client presents the frame to the user, and the difference between the timestamp associated with the input capture and the timestamp associated with the frame rendering represents the stack latency. It will be understood that input is asynchronous with frame rendering, and therefore, not every frame will have a stack latency value.

[0048] In some embodiments, a forward filling technique can be used to replace any value of a component that is determined to be a special Not-a-Number (NaN) value. For example, not every frame may be associated with an input, and therefore, only some frames may have a corresponding stack delay value. Other frames may simply fill this value with a special NaN value. In practice, the stack delay component of one frame may be the same for the next, so that, as indicated by the user's perception, the sequence of subsequent frames is forward filled with the last good component value of the previous frame until the point where metadata allows a new valid component value to be calculated for the subsequent frame. This forward filling technique is useful for ensuring that every frame in a streaming session is associated with a valid value, even if metadata for the frame is unavailable. For example, remote input may be sampled at 1000 Hz at the client device, which may lead to memory storage overflow issues if timestamps are stored at this high sampling rate. In some embodiments, the sampling rate used to monitor remote input can be lower (e.g., 10 Hz or 100 ms per sample) to generate timestamps in order to reduce storage overload. Because multiple frames may correspond to a 100ms window for a single timestamp instance associated with remote input, the timestamp within the window can be forwarded to each frame within the window. This concept can be applied to any metadata (e.g., timestamps) generated asynchronously from the content's frame rate.

[0049] In some embodiments, certain frames can be discarded by the pipeline. For example, a frame that arrives at a client device before the client device has displayed the previous frame may cause the client device to discard the previous frame and instead present the most recent frame in the next frame presentation period. In such a case, any component that relies on the timestamp for frame presentation (which may not be present in the metadata of the discarded frame) can be filled forward using the techniques discussed above. In some cases, frames may also be discarded at the server device, such as when the frame capture stage 110 is ready to present multiple frames in memory when reading in a new frame from a game application. The original value of the forward filling component can be useful for completing sparse metadata.

[0050] The third measurement includes two components related to stuttering. The first component is related to stuttering by the gaming application and can be referred to as frame capture-related stuttering. In other words, the first component is a measure of how often the server device fails to render frames on time based on the expected frame rate. In one embodiment, stuttering refers to latency above a threshold. For example, if a frame is expected to be rendered every 16.6ms (corresponding to 60 frames per second), then stuttering can be defined whenever a frame is rendered at least 24ms after the previous frame (calculated as the difference between the timestamps associated with the completion of rendering of the current and previous frames). The second component is related to stuttering at the rendering stage of the pipeline and can be referred to as frame rendering-related stuttering. In other words, the second component is a measure of how often the client device fails to present a new frame at the expected frame transition time. It will be appreciated that when the client fails to receive a frame in the next rendering period, the previous frame can be displayed again during the next rendering period, resulting in potentially perceptible stuttering, where motion appears to stop to the viewer.

[0051] In one embodiment, the raw values of each of the components discussed above are calculated based on the metadata of each frame. The raw values are then converted using a transfer function that scales the raw values to the desired scale (e.g., 0-100). The transfer function does not have to be linear. Weights are also assigned based on a fixed function (depending on the raw values), and the weights can correspond to different scales (e.g., 1 to 10, 1 to 200, etc.). It should be understood that the transfer function and weight function can be different for each component. For example, the transfer function for the stack delay component may be different from the transfer function for the frame capture freeze component.

[0052] Subjective ratings of quality can depend on the frequency of events as well as the severity of the events. For example, a single severe event (such as a 3 second delay) may be perceived as better than frequently occurring 300 millisecond delays because the user may perceive the single event as affecting only a small portion of the session, while more frequently occurring events may cause the user to view the entire session negatively. To address this dependence of quality on the frequency of events, each component is averaged over a sliding window so that the score of the component at any given frame depends on the scores of the components for multiple frames before that frame. This will more accurately reflect the overall experience of an individual who will remember a negative event over time that will affect their perception of the session even after the event has passed.

[0053] In an embodiment, the raw value of each component is multiplied by a corresponding weight, and a rolling weighted mean is calculated for each component for each frame by averaging the weighted scores that fall within a sliding window. Since the weighted scores within the sliding window change for each new frame, the rolling weighted mean is essentially a weighted moving mean over a specific time period corresponding to X number of frames within the sliding window.

[0054] The rolling weighted means of the individual components are then combined to calculate a per-frame aggregate value based on the multiple components. In an embodiment, the combined weighted score is calculated as the sum of the partial products of the multiple components. The combined weighted score is a per-frame value based on the converted scores (e.g., the original values processed by the transfer function) and the weights from the multiple components. The partial product of a particular component is calculated as the product of the weight and the smaller (e.g., the minimum) of the rolling weighted mean and the converted score. In other embodiments, the partial product is simply the product of the weight and the rolling weighted mean. It will be appreciated that the weights for a given frame are reused when calculating the partial products for that component when calculating the combined weighted score. By applying the weights to the rolling weighted mean, an emphasis on negative events during that particular frame can be highlighted, even if the raw score for that component during that frame was initially silenced by the sliding window calculation.

[0055] The combined weighted score is a per-frame value, and using the second sliding window, a rolling mean of the combined weighted score and a rolling standard deviation of the combined weighted score can be calculated. The rolling mean of the combined weighted score is essentially a moving average over a specific time period corresponding to Y number of frames within the second sliding window, and the rolling standard deviation is essentially a measure of the variation of the combined weighted score over the specific time period. In one or more embodiments, both the rolling mean and the rolling standard deviation of the combined weighted score can be per-frame values.

[0056] A combined value of the quality metric can then be calculated based on the rolling mean and rolling standard deviation. In some embodiments, the quality metric (sometimes referred to as the streaming experience score (SES)) is the sum of the freeze-latency score and the picture quality score. In one embodiment, the quality metric can also be shifted and / or normalized to rank the experience on a set scale (e.g., 0 to 100). In one embodiment, the freeze-latency score is calculated by summing the base score and the score offset. The base score is calculated by generating a rolling mean of the combined weighted score and a rolling standard deviation of the combined weighted score using a first configuration of the first and second sliding windows (e.g., a first parameter set). An exemplary configuration of the first and second sliding windows is X=600 frames and Y=3600 frames (e.g., the second sliding window is larger than the first sliding window). The per-frame streaming experience freeze-latency score is then calculated by taking the difference between the rolling mean of the combined weighted score and the product of the coefficient (Z) and the rolling standard deviation of the combined weighted score. In the exemplary configuration, Z is set to 6.0 to calculate the base score. The fractional offset is calculated in a similar manner to the base fraction, but using a different configuration (e.g., a second set of parameters) for the first and second sliding windows and coefficients. An exemplary configuration of the first and second sliding windows is X=3600 frames and Y=600 frames (e.g., the first sliding window is larger than the second sliding window), and Z is set to 4.0 to calculate the fractional offset.

[0057] It will be appreciated that the base score and score offset can be calculated per frame. However, in some embodiments, the jam-latency score can be an aggregate value per session. As such, the jam-latency score can be calculated as the difference between the aggregate weighted mean of the base scores (per frame) and the standard deviation of the score offset, where the weights of the aggregate weighted mean are derived from the remote input frequency. As used herein, remote input frequency refers to a measurement of the percentage of time corresponding to active input (e.g., key events, mouse events, controller button or joystick events, etc.).

[0058] In an embodiment, the picture quality score is derived using an algorithm separate from the algorithm described above for the stutter and delay components. In such an embodiment, the picture quality score is calculated based on a raw score of picture quality, which is a per-session value related to a target bits per pixel bitrate for the video stream. In some cases, the target bitrate can change dynamically during a streaming session. In such cases, the picture quality score can be calculated as the average of the target bitrates during the session (e.g., by summing the target bitrates for each frame and then dividing by the number of frames).

[0059] The quality metric is then calculated by summing the picture quality score and the stall-delay score. The raw quality metric score can then be normalized and shifted so that the score falls within a desired range (e.g., 0-100) that is easily evaluated by an administrator or developer.

[0060] It will be appreciated that changes to a streaming service (such as adjusting encoding / decoding parameters, or adjusting the target bitrate or resolution of video content) can be quickly evaluated to determine the impact the change may have on the end user. Similarly, the pipeline can be configured to automatically adjust parameters in an attempt to improve quality metric scores. For example, when a new video game is added to be delivered by a streaming service, the pipeline can be automatically configured to adjust the parameters of various stages of the pipeline in an attempt to improve the quality of the streaming service to accommodate the new rendering portion of the pipeline. For example, the new game may require longer rendering times, which may cause severe stuttering when attempting to stream 60 frames per second. As the quality metric of the streaming session associated with the new game decreases, the server device may reduce the frame rate from 60fps to 30fps. Alternatively, the encoder's quantization parameters can be changed to attempt to reduce the target bitrate of the video stream, thereby reducing network bandwidth. Administrators or developers can also view graphs of the quality metrics of multiple streams and make decisions about how to manually adjust the pipeline (e.g., by increasing the number of clusters available to process certain stages of the pipeline, etc.).

[0061] Figure 2A flow chart of a method 200 for evaluating a quality metric for a session, according to some embodiments, is shown. Method 200 is described in the context of software executed by one or more processors. In some embodiments, method 200 is at least partially executable by instructions executed by a server device. In other embodiments, method 200 is executable by hardware or some combination of hardware and software.

[0062] At step 202, metadata for a streaming session is received at a server device. The metadata includes information about a plurality of frames included in the streaming session. The information may include a number of timestamps indicating the completion of processing at a plurality of points in the pipeline (e.g., completion of each stage in the pipeline).

[0063] At step 204, a score and weight are calculated for each of the plurality of components derived from the metadata. The score and weight are calculated for each of the plurality of frames included in the streaming session. In some embodiments, the score is derived from the raw value by applying a transfer function that converts the raw value into a score. The raw value may be calculated as a delta value, which is the difference between the timestamp of the current frame and the corresponding timestamp of the previous frame. In some embodiments, the raw value is shifted by an expected delay based on the frame rate, such that the raw value is positive or negative. In some embodiments, the weight is derived from a function that depends on the raw value.

[0064] At step 206, a quality metric for the streaming session is calculated based on the scores and weights of the multiple components. In some embodiments, the quality metric is calculated as the sum of a picture quality score and a combined stall-latency score. The combined stall-latency score is calculated by: calculating, for each of the plurality of frames, a rolling weighted score for each of at least one delay component and at least one stall component based on a first sliding window; and calculating a rolling mean of a combined weighted sum of the rolling weighted scores for each of the at least one delay component and at least one stall component, wherein the rolling mean of the combined weighted sum is based on a second sliding window. The picture quality score is calculated as the difference between a mean of the plurality of picture quality scores for the plurality of frames and a product of a coefficient and a standard deviation of the plurality of picture quality scores for the plurality of frames. In some embodiments, the quality metric is normalized and / or shifted.

[0065] At step 208, at least one component of the pipeline is adjusted based on the quality metric. The adjustment may include adjusting at least one of a quantization parameter, a target bitrate, an encoding parameter, or a target resolution of the frames generated by the pipeline. In some embodiments, a parameter such as a quantization parameter is adjusted when the quality metric of one or more streaming sessions is below a threshold. For example, an average of the quality metrics of multiple sessions may be calculated and compared to a threshold to determine whether to adjust the parameter. In some embodiments, a component of the pipeline may also be adjusted if the quality metric of one or more streaming sessions is above a threshold. For example, if the quality metric is very high (i.e., indicating high user satisfaction), certain parameters may be adjusted to, for example, increase the resolution, bitrate, frame rate, etc. of the streaming session while attempting to avoid negative effects on perceived quality due to, for example, freezes or delays caused by network congestion.

[0066] It will be appreciated that the stages of the pipelined stack 100 and / or the steps of the method 200 are implemented by at least one of the following: one or more server devices, one or more network devices, and / or client devices. Each of the server devices, network devices, and client devices includes at least a processor, a memory, and a network interface. The processor may include a conventional programmable processor, such as a central processing unit. Alternatively, the processor may include a reduced instruction set computer (RISC), such as an ARM processor or a system on a chip (SoC), which includes one or more CPU cores and one or more GPU cores on a single integrated circuit or multiple integrated circuits included in a package-on-package (PoP) device. In some embodiments, particularly for stages where the task is to perform frame rendering operations or other types of operations that can benefit from a parallel architecture (such as a single instruction, multiple data (SIMD), or a single instruction, multiple thread (SIMT) processor architecture), the processor may include a parallel processing unit (PPU) or a tensor processor. In one or more other embodiments, the one or more server devices may be implemented at least in part as an instantiation of a virtual machine using one or more virtual components, which may include but are not limited to one or more of a virtual CPU, a virtual GPU, and virtual memory.

[0067] More illustrative information about various optional architectures and features that can implement the aforementioned framework will now be described, depending on the user's desires. It should be noted that the following information is provided for illustrative purposes only and should not be construed as limiting in any way. The optional features below may optionally be incorporated with or without excluding the other features described.

[0068] Parallel processing architecture

[0069] Figure 3A parallel processing unit (PPU) 300 is shown according to one embodiment. In one embodiment, the PPU 300 is a multi-threaded processor implemented on one or more integrated circuit devices. The PPU 300 is a latency-hiding architecture designed for processing many threads in parallel. A thread (i.e., an execution thread) is an instance of an instruction set configured to be executed by the PPU 300. In one embodiment, the PPU 300 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device (such as a liquid crystal display (LCD) device). In other embodiments, the PPU 300 can be used to perform general-purpose computations. Although an exemplary parallel processor is provided herein for illustrative purposes, it should be specifically noted that the processor is described for illustrative purposes only and any processor may be used in addition to and / or in place of the processor.

[0070] One or more PPUs 300 can be configured to accelerate thousands of high-performance computing (HPC), data center, and machine learning applications. A PPU 300 can be configured to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision speech, image, and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, other simulation platforms for testing, validating, or certifying autonomous vehicle robotics applications involving ray tracing technology as described herein, and personalized user recommendations, among others.

[0071] like Figure 3 As shown, the PPU 300 includes an input / output (I / O) unit 305, a front-end unit 315, a scheduler unit 320, a work distribution unit 325, a hub 330, a crossbar switch (Xbar) 370, one or more general processing clusters (GPCs) 350, and one or more memory partitioning units 380. The PPU 300 can be connected to a host processor or other PPUs 300 via one or more high-speed NVLink 310 interconnects. The PPU 300 can be connected to a host processor or other peripheral devices via interconnect 302. The PPU 300 can also be connected to a local memory 304 including multiple memory devices. In one embodiment, the local memory can include multiple dynamic random access memory (DRAM) devices. The DRAM devices can be configured as a high-bandwidth memory (HBM) subsystem, in which multiple DRAM dies are stacked within each device.

[0072] The NVLink 310 interconnect enables the system to scale and include one or more PPUs 300 in conjunction with one or more CPUs, supporting cache coherency between the PPU 300 and the CPU, and CPU mastering. Data and / or commands can be sent by the NVLink 310 through the hub 330 to or from other units of the PPU 300, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). Figure 5B NVLink 310 is described in more detail.

[0073] I / O unit 305 is configured to send and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 302. I / O unit 305 can communicate with the host processor directly via interconnect 302, or through one or more intermediary devices (such as a memory bridge). In one embodiment, I / O unit 305 can communicate with one or more other processors (e.g., one or more PPUs 300) via interconnect 302. In one embodiment, I / O unit 305 implements a Peripheral Component Interconnect Express (PCIe) interface for communicating over a PCIe bus, and interconnect 302 is a PCIe bus. In alternative embodiments, I / O unit 305 can implement other types of known interfaces for communicating with external devices.

[0074] I / O unit 305 decodes data packets received via interconnect 302. In one embodiment, the data packets represent commands configured to cause PPU 300 to perform various operations. I / O unit 305 sends the decoded commands to various other units of PPU 300 as specified by the commands. For example, some commands may be sent to front-end unit 315. Other commands may be sent to hub 330 or other units of PPU 300, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 305 is configured to route communications between and among the various logical units of PPU 300.

[0075] In one embodiment, a program executed by a host processor encodes a command stream in a buffer that provides a workload to the PPU 300 for processing. The workload may include many instructions and data to be processed by those instructions. A buffer is an area of memory that is accessible (e.g., read / write) by both the host processor and the PPU 300. For example, the I / O unit 305 may be configured to access a buffer in system memory connected to the interconnect 302 via a memory request transmitted over the interconnect 302. In one embodiment, the host processor writes a command stream into the buffer and then sends a pointer to the start of the command stream to the PPU 300. The front end unit 315 receives pointers to one or more command streams. The front end unit 315 manages the one or more streams, reads commands from the streams, and forwards the commands to the various units of the PPU 300.

[0076] Front-end unit 315 is coupled to scheduler unit 320, which configures various GPCs 350 to process tasks defined by one or more streams. Scheduler unit 320 is configured to track state information related to the various tasks managed by scheduler unit 320. The state may indicate which GPC 350 a task is assigned to, whether the task is active or inactive, the priority associated with the task, and the like. Scheduler unit 320 manages the execution of multiple tasks on one or more GPCs 350.

[0077] Scheduler unit 320 is coupled to work distribution unit 325, which is configured to dispatch tasks for execution on GPCs 350. Work distribution unit 325 can track a number of scheduled tasks received from scheduler unit 320. In one embodiment, work distribution unit 325 manages a pending task pool and an active task pool for each GPC 350. The pending task pool can include a number of time slots (e.g., 32 time slots) containing tasks assigned to be processed by a particular GPC 350. The active task pool can include a number of time slots (e.g., 4 time slots) for tasks being actively processed by GPC 350. When a GPC 350 completes execution of a task, the task is evicted from the active task pool of GPC 350, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 350. If an active task on GPC 350 has become idle, such as while waiting for a data dependency to be resolved, the active task may be evicted from GPC 350 and returned to the pending task pool, while another task in the pending task pool is selected and scheduled for execution on GPC 350 .

[0078] Work distribution unit 325 communicates with one or more GPCs 350 via XBar (crossbar) 370. XBar 370 is an interconnect network that couples many units of PPU 300 to other units of PPU 300. For example, XBar 370 can be configured to couple work distribution unit 325 to a specific GPC 350. Although not explicitly shown, one or more other units of PPU 300 can also be connected to XBar 370 via hub 330.

[0079] Tasks are managed by the scheduler unit 320 and dispatched to GPCs by the work distribution unit 325

[0080] 350. A GPC 350 is configured to process tasks and generate results. The results can be consumed by other tasks within the GPC 350, routed to a different GPC 350 via XBar 370, or stored in memory 304. The results can be written to memory 304 via a memory partition unit 380, which implements a memory interface for reading data from and writing data to memory 304. The results can be sent to another PPU 300 or CPU via NVLink 310. In one embodiment, a PPU 300 includes a number U of memory partition units 380, which is equal to the number of independent and distinct memory devices coupled to the memory 304 of the PPU 300. This will be discussed below in conjunction with Figure 4B The memory partition unit 380 is described in more detail.

[0081] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications to be executed on the host processor to schedule operations for execution on the PPU 300. In one embodiment, multiple computing applications are executed simultaneously by the PPU 300, and the PPU 300 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. The application can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks to be executed by the PPU 300. The driver kernel outputs the tasks to one or more streams being processed by the PPU 300. Each task can include one or more related groups of threads, referred to herein as warps. In one embodiment, a warp includes 32 related threads that can execute in parallel. Collaborating threads can refer to multiple threads that include instructions to perform tasks and that can exchange data through shared memory. In combination Figure 5A Describes threads and cooperative threads in more detail.

[0082] Figure 4A According to one embodiment, Figure 3PPU 300 GPC 350. Figure 4A As shown, each GPC 350 includes multiple hardware units for processing tasks. In one embodiment, each GPC 350 includes a pipeline manager 410, a pre-raster operation unit (PROP),

[0083] 415, raster engine 425, work distribution crossbar switch (WDX) 480, memory management unit (MMU) 490 and one or more data processing clusters (DPC) 420. It should be understood that Figure 4A The GPC 350 may include instead Figure 4A Other hardware units or other than the units shown in Figure 4A Other hardware units besides those shown in .

[0084] In one embodiment, the operation of GPC 350 is controlled by pipeline manager 410. Pipeline manager 410 manages the configuration of one or more DPCs 420 for processing tasks assigned to GPC 350. In one embodiment, pipeline manager 410 can configure at least one of one or more DPCs 420 to implement at least a portion of a graphics rendering pipeline. For example, DPC 420 can be configured to execute vertex shading programs on programmable streaming multiprocessors (SMs) 440. Pipeline manager 410 can also be configured to route packets received from work distribution unit 325 to appropriate logic units within GPC 350. For example, some packets can be routed to fixed-function hardware units within PROP 415 and / or raster engine 425, while other packets can be routed to DPC 420 for processing by primitive engine 435 or SM 440. In one embodiment, pipeline manager 410 can configure at least one of one or more DPCs 420 to implement a neural network model and / or a compute pipeline.

[0085] PROP unit 415 is configured to route data generated by raster engine 425 and DPC 420 to the raster operations (ROP) unit, in conjunction with Figure 4B The PROP unit 415 may also be configured to perform optimizations for color blending, organize pixel data, perform address translation, and the like.

[0086] The raster engine 425 includes several fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster engine 425 includes a setup engine, a coarse raster engine, a culling engine, a clipping engine, a fine raster engine, and a tile aggregation engine. The setup engine receives the transformed vertices and generates plane equations associated with the geometric primitives defined by the vertices. The plane equations are sent to the coarse raster engine to generate coverage information for the primitives (e.g., the x, y coverage mask of the tile). The output of the coarse raster engine is sent to the culling engine, where fragments associated with primitives that fail the z-test are culled, and to the clipping engine, where fragments outside the viewing frustum are clipped. Those fragments remaining after clipping and culling can be passed to the fine raster engine to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. The output of the raster engine 425 includes, for example, fragments to be processed by the fragment shader implemented in the DPC 420.

[0087] Each DPC 420 included in the GPC 350 includes an M Pipeline Controller (MPC)

[0088] 430, a primitive engine 435, and one or more SMs 440. The MPC 430 controls the operation of the DPC 420 and routes data packets received from the pipeline manager 410 to the appropriate units in the DPC 420. For example, data packets associated with vertices may be routed to the primitive engine 435, which is configured to fetch vertex attributes associated with the vertices from the memory 304. Conversely, data packets associated with shading programs may be sent to the SM 440.

[0089] SM 440 includes a programmable streaming processor configured to process tasks represented by multiple threads. Each SM 440 is multi-threaded and configured to simultaneously execute multiple threads (e.g., 32 threads) from a particular thread group. In one embodiment, SM 440 implements a SIMD (single instruction, multiple data) architecture, in which each thread in a thread group (e.g., a warp) is configured to process a different data set based on the same instruction set. All threads in a thread group execute the same instructions. In another embodiment, SM 440 implements a SIMT (single instruction, multiple thread) architecture, in which each thread in a thread group is configured to process a different data set based on the same instruction set, but in which individual threads in a thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each warp, enabling concurrency between warps and serial execution within a warp when threads within the warp diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby achieving equal concurrency between all threads within and between warps. When execution state is maintained for each individual thread, threads executing the same instruction can be converged and executed in parallel for maximum efficiency. Figure 5A SM 440 is described in more detail.

[0090] MMU 490 provides an interface between GPC 350 and memory partition unit 380. MMU 490 can provide virtual to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, MMU 490 provides one or more translation lookaside buffers (TLBs) for performing translations from virtual addresses to physical addresses in memory 304.

[0091] Figure 4B According to one embodiment, Figure 3 The memory partition unit 380 of the PPU 300. Figure 4B As shown, the memory partition unit 380 includes a raster operations (ROP) unit 450, a level 2 (L2) cache 460, and a memory interface 470. The memory interface 470 is coupled to the memory 304. The memory interface 470 can implement a 32-, 64-, 128-, or 1024-bit data bus for high-speed data transfer. In one embodiment, the PPU 300 incorporates U memory interfaces 470, one for each pair of memory partition units 380, where each pair of memory partition units 380 is connected to a corresponding memory device of the memory 304. For example, the PPU 300 can be connected to up to Y memory devices, such as a high-bandwidth memory stack or graphics double data rate version 5 synchronous dynamic random access memory or other types of persistent memory.

[0092] In one embodiment, memory interface 470 implements an HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack is located on the same physical package as PPU 300, providing significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies and Y is equal to 4, where the HBM2 stack includes two 128-bit channels per die, for a total of 8 channels and a data bus width of 1024 bits.

[0093] In one embodiment, memory 304 supports single-error correction, double-error detection (SECDED) error correction code (ECC) to protect data. ECC provides increased reliability for computing applications sensitive to data corruption. Reliability is particularly important in large cluster computing environments where PPU 300 processes very large data sets and / or long-running applications.

[0094] In one embodiment, the PPU 300 implements a multi-level memory hierarchy. In one embodiment, the memory partitioning unit 380 supports unified memory to provide a single, unified virtual address space for the CPU and PPU 300 memory, enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 300 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 300 where the pages are accessed more frequently. In one embodiment, NVLink 310 supports address translation services that allow the PPU 300 to directly access the CPU's page tables and provide full access to the CPU's memory by the PPU 300.

[0095] In one embodiment, the copy engine transfers data between multiple PPUs 300 or between a PPU 300 and a CPU. The copy engine can generate a page fault for an address that is not mapped to a page table. The memory partition unit 380 can then service the page fault, map the address into a page table, and then the copy engine can perform the transfer. In conventional systems, multiple copy engines operate on fixed memory (e.g., non-pageable) between multiple processors, which significantly reduces the available memory. Due to hardware page faults, addresses can be passed to the copy engine without worrying about whether the memory page is resident, and the copy process is transparent.

[0096] Data from memory 304 or other system memory can be retrieved by memory partition unit 380 and stored in L2 cache 460, which is located on-chip and shared between various GPCs 350. As shown, each memory partition unit 380 includes a portion of L2 cache 460 associated with the corresponding memory 304. Lower-level caches can then be implemented in multiple units within a GPC 350. For example, each SM 440 can implement a level 1 (L1) cache. The L1 cache is a dedicated memory dedicated to a particular SM 440. Data from the L2 cache 460 can be retrieved and stored in each L1 cache for processing in the functional units of the SM 440. The L2 cache 460 is coupled to the memory interface 470 and the XBar 370.

[0097] The ROP unit 450 performs graphics raster operations related to pixel color such as color compression, pixel blending, etc. The ROP unit 450 also implements depth testing in conjunction with the raster engine 425, receiving the depth of the sample position associated with the pixel fragment from the culling engine of the raster engine 425. The depth of the sample position associated with the fragment is tested against the corresponding depth in the depth buffer. If the fragment passes the depth test for the sample position, the ROP unit 450 updates the depth buffer and sends the result of the depth test to the raster engine 425. It will be understood that the number of memory partition units 380 can be different than the number of GPCs 350, and therefore each ROP unit 450 can be coupled to each GPC 350. The ROP unit 450 tracks packets received from different GPCs 350 and determines to which GPC 350 the results generated by the ROP unit 450 are routed via the Xbar 370. Although in Figure 4B In the embodiment shown, ROP unit 450 is included within memory partition unit 380, but in other embodiments, ROP unit 450 may be external to memory partition unit 380. For example, ROP unit 450 may reside in GPC 350 or another unit.

[0098] Figure 5A According to one embodiment, Figure 4A Streaming multiprocessor 440. Figure 5A As shown, SM 440 includes an instruction cache 505, one or more scheduler units 510, a register file 520, one or more processing cores 550, one or more special function units (SFUs) 552, one or more load / store units (LSUs) 554, an interconnect network 580, and a shared memory / L1 cache 570.

[0099] As described above, the work distribution unit 325 schedules tasks for execution on the GPCs 350 of the PPU 300. Tasks are assigned to specific DPCs 420 within the GPC 350 and, if the task is associated with a shader program, may be assigned to an SM 440. The scheduler unit 510 receives tasks from the work distribution unit 325 and manages the scheduling of instructions for one or more thread blocks assigned to the SM 440. The scheduler unit 510 schedules thread blocks for execution as warps of parallel threads, where each thread block is assigned at least one warp. In one embodiment, each warp executes 32 threads. The scheduler unit 510 can manage multiple different thread blocks, assign warps to different thread blocks, and then dispatch instructions from multiple different cooperative groups to various functional units (i.e., cores 550, SFUs 552, and LSUs 554) during each clock cycle.

[0100] Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity at which threads are communicating, enabling the expression of richer and more efficient decompositions of parallelism. The cooperative launch API supports synchronization between thread blocks to execute parallel algorithms. Conventional programming models provide a single, simple construct for synchronizing cooperating threads: a barrier across all threads of a thread block (e.g., the syncthreads() function). However, programmers often want to define thread groups at a granularity smaller than the thread block granularity and synchronize within the defined group, enabling higher performance, design flexibility, and software reuse in the form of a collective group-wide function interface.

[0101] Cooperative Groups enable programmers to explicitly define thread groups at sub-block (e.g., as small as a single thread) and multi-block granularity and perform collective operations, such as synchronization, on threads in a cooperative group. The programming model supports clean composition across software boundaries so that libraries and utility functions can safely synchronize in their local environment without making assumptions about convergence. Cooperative Group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire grid of thread blocks.

[0102] The dispatch unit 515 is configured to deliver instructions to one or more functional units. In this embodiment, the scheduler unit 510 includes two dispatch units 515, which enable scheduling of two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 510 may include a single dispatch unit 515 or additional dispatch units 515.

[0103] Each SM 440 includes a register file 520 that provides a set of registers for the functional units of the SM 440. In one embodiment, the register file 520 is divided between each functional unit so that each functional unit is allocated a dedicated portion of the register file 520. In another embodiment, the register file 520 is divided between the different warps executed by the SM 440. The register file 520 provides temporary storage for operands connected to the data paths of the functional units.

[0104] Each SM 440 includes L processing cores 550. In one embodiment, the SM 440 includes a large number (e.g., 128, etc.) of different processing cores 550. Each core 550 may include a fully pipelined, single-precision, double-precision, and / or mixed-precision processing unit, including a floating-point arithmetic logic unit (FLU) and an integer arithmetic logic unit (ALU). In one embodiment, the FLU implements the IEEE 754-2008 standard for floating-point operations. In one embodiment, the core 550 includes 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0105] Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are included in core 550. Specifically, the tensor cores are configured to perform deep learning matrix operations, such as convolution operations for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs a matrix multiplication and accumulation operation D=A×B+C, where A, B, C, and D are 4×4 matrices.

[0106] In one embodiment, the matrix multiplication inputs A and B are 16-bit floating point matrices, while the accumulation matrices C and D can be 16-bit floating point or 32-bit floating point matrices. The tensor cores operate on 16-bit floating point input data as well as 32-bit floating point accumulations. The 16-bit floating point multiplication requires 64 operations to produce a full-precision product, which is then accumulated using 32-bit floating point additions with other intermediate products of the 4×4×4 matrix multiplication. In practice, tensor cores are used to perform larger two-dimensional or higher-dimensional matrix operations built from these smaller elements. APIs (such as the CUDA 9 C++ API) expose specialized matrix load, matrix multiplication and accumulation, and matrix store operations to efficiently use tensor cores from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes that the 16×16 size matrix spans all 32 threads of the warp.

[0107] Each SM 440 also includes M SFUs 552 that perform special functions (e.g., attribute evaluation, reciprocal square root, etc.). In one embodiment, the SFUs 552 may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 552 may include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture unit is configured to load a texture map (e.g., a 2D array of texels) from memory 304 and sample the texture map to generate sampled texture values for use in shader programs executed by the SM 440. In one embodiment, the texture map is stored in shared memory / L1 cache 570. The texture unit implements texture operations, such as filtering operations using mip maps (i.e., texture maps at different levels of detail). In one embodiment, each SM 440 includes two texture units.

[0108] Each SM 440 also includes N LSUs 554, which implement load and store operations between the shared memory / L1 cache 570 and the register file 520. Each SM 440 includes an interconnect network 580 that connects each functional unit to the register file 520 and the LSUs 554 to the register file 520 and the shared memory / L1 cache 570. In one embodiment, the interconnect network 580 is a crossbar switch that can be configured to connect any functional unit to any register in the register file 520 and to connect the LSUs 554 to memory locations in the register file and the shared memory / L1 cache 570.

[0109] Shared memory / L1 cache 570 is an on-chip memory array that allows data storage and communication between SM 440 and primitive engine 435, as well as between threads in SM 440. In one embodiment, shared memory / L1 cache 570 includes 128KB of storage capacity and is in the path from SM 440 to memory partition unit 380. Shared memory / L1 cache 570 can be used to cache reads and writes. One or more of shared memory / L1 cache 570, L2 cache 460, and memory 304 are backing stores.

[0110] Combining data cache and shared memory functionality into a single memory block provides the best overall performance for both types of memory access. This capacity can be used by programs as a cache that does not utilize the shared memory. For example, if the shared memory is configured to use half of its capacity, texture and load / store operations can use the remaining capacity. Integration within shared memory / L1 cache 570 enables shared memory / L1 cache 570 to function as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0111] When configured for general parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, Figure 3 The fixed-function graphics processing unit is shown bypassed, creating a simpler programming model. In a general-purpose parallel computing configuration, work distribution unit 325 assigns and distributes thread blocks directly to DPC 420. The threads in the block execute the same program, using unique thread IDs in computations to ensure each thread produces unique results, using SM 440 to execute the program and perform computations, using shared memory / L1 cache 570 to communicate between threads, and using LSU 554 to read and write global memory through shared memory / L1 cache 570 and memory partition unit 380. When configured for general-purpose parallel computing, SM 440 can also write commands that scheduler unit 320 can use to start new work on DPC 420.

[0112] The PPU 300 may be included in a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a virtual machine (VM), or other shared computing resources. In one embodiment, the PPU 300 is included on a single semiconductor substrate. In another embodiment, the PPU 300 is included on a system-on-chip (SoC) along with one or more other devices (such as an additional PPU 300, a memory 304, a reduced instruction set computer (RISC) CPU, a memory management unit (MMU), a digital-to-analog converter (DAC), etc.).

[0113] In one embodiment, PPU 300 may be included on a graphics card that includes one or more memory devices. The graphics card may be configured to interface with a PCIe slot on a desktop computer's motherboard. In another embodiment, PPU 300 may be an integrated graphics processing unit (iGPU) or parallel processor included in a chipset on the motherboard.

[0114] Exemplary Computing System

[0115] Systems with multiple GPUs and CPUs are being used across various industries as developers expose and exploit greater parallelism in applications such as artificial intelligence computing. High-performance GPU-accelerated systems with tens to thousands of computing nodes are deployed in data centers, research institutions, and supercomputers to solve larger problems. As the number of processing devices within high-performance systems increases, communication and data transmission mechanisms need to scale to support this increased bandwidth.

[0116] Figure 5B According to one embodiment, the Figure 3 The processing system 500 includes a CPU 530, a switch 511, and multiple PPUs 300 and corresponding memories 304. NVLink 310 provides a high-speed communication link between each PPU 300. Figure 5B A specific number of NVLink 310 and interconnect 302 connections are shown, but the number of connections connected to each PPU 300 and CPU 530 can vary. Switch 511 interfaces between interconnect 302 and CPU 530. PPU 300, memory 304, and NVLink 310 can be located on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 511 supports two or more protocols that interface between various different connections and / or links.

[0117] In another embodiment (not shown), NVLink 310 provides one or more high-speed communication links between each PPU 300 and the CPU 530, and a switch 511 interfaces between the interconnect 302 and each PPU 300. The PPUs 300, memory 304, and interconnect 302 may be located on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), the interconnect 302 provides one or more communication links between each PPU 300 and the CPU 530, and the switch 511 interfaces between each PPU 300 using NVLink 310 to provide one or more high-speed communication links between the PPUs 300. In another embodiment (not shown), NVLink 310 provides one or more high-speed communication links between the PPUs 300 and the CPU 530 through the switch 511. In yet another embodiment (not shown), the interconnect 302 provides one or more communication links directly between each PPU 300. One or more NVLink 310 high-speed communication links may be implemented as a physical NVLink interconnect or as an on-chip or on-die interconnect using the same protocol as NVLink 310 .

[0118] In the context of this specification, a single semiconductor platform may refer to a unique, single semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform may also refer to a multi-chip module with increased connectivity that emulates on-chip operation and is substantially improved by utilizing conventional bus implementations. Of course, various circuits or devices may also be placed separately or in various combinations of semiconductor platforms, depending on the needs of the user. Alternatively, the parallel processing module 525 may be implemented as a circuit board substrate, and each of the PPU 300 and / or memory 304 may be a packaged device. In one embodiment, the CPU 530, switch 511, and parallel processing module 525 are located on a single semiconductor platform.

[0119] In one embodiment, the signaling rate of each NVLink 310 is 20 to 25 Gbit / s, and each PPU 300 includes six NVLink 310 interfaces (e.g., Figure 5B As shown, each PPU 300 includes five NVLink 310 interfaces. Each NVLink 310 provides a data transfer rate of 25 Gbit / s in each direction, with six links providing 300 Gbit / s. When the CPU 530 also includes one or more NVLink 310 interfaces, the NVLink 310 can be used exclusively for Figure 5B PPU to PPU communication shown, or some combination of PPU to PPU and PPU to CPU.

[0120] In one embodiment, NVLink 310 allows direct load / store / atomic access from the CPU 530 to the memory 304 of each PPU 300. In one embodiment, NVLink 310 supports coherency operations, allowing data read from memory 304 to be stored in the cache hierarchy of the CPU 530, reducing cache access latency for the CPU 530. In one embodiment, NVLink 310 includes support for Address Translation Services (ATS), allowing the PPU 300 to directly access page tables within the CPU 530. One or more NVLinks 310 can also be configured to operate in a low-power mode.

[0121] Figure 5CAn exemplary system 565 is shown in which various architectures and / or functions of various previous embodiments can be implemented. As shown, a system 565 is provided that includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 can be implemented using any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI-Express, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or one or more point-to-point communication protocols. The system 565 also includes a main memory 540. Control logic (software) and data are stored in the main memory 540, which can take the form of random access memory (RAM).

[0122] System 565 also includes an input device 560, a parallel processing module 525, and a display device 545, such as a conventional CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), plasma display, etc. User input can be received from input device 560 (e.g., a keyboard, mouse, touchpad, microphone, etc.). Each of the aforementioned modules and / or devices can even be located on a single semiconductor platform to form system 565. Alternatively, the modules can be located separately or in various combinations of semiconductor platforms, depending on the user's needs.

[0123] Furthermore, system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) through network interface 535 for communication purposes.

[0124] System 565 may also include auxiliary storage (not shown). Auxiliary storage includes, for example, a hard drive and / or a removable storage drive, representative of a floppy disk drive, a tape drive, an optical drive, a digital versatile disk (DVD) drive, a recording device, and a universal serial bus (USB) flash memory. The removable storage drive reads from and / or writes to the removable storage unit in a well-known manner.

[0125] Computer programs or computer control logic algorithms may be stored in the main memory 540 and / or the secondary storage. These computer programs, when executed, enable the system 565 to perform various functions. The memory 540, storage, and / or any other storage are possible examples of computer-readable media.

[0126] The architecture and / or functionality of the various preceding figures can be implemented in the context of a general-purpose computer system, a circuit board system, a game console system dedicated for entertainment purposes, a dedicated system, and / or any other desired system. For example, system 565 can take the form of a desktop computer, a laptop computer, a tablet computer, a server, a supercomputer, a smartphone (e.g., wireless, handheld device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted display, a handheld electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system, and / or any other type of logic.

[0127] Although various embodiments have been described above, it should be understood that they are presented by way of example only, not limitation. Therefore, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

[0128] Machine Learning

[0129] Deep neural networks (DNNs) developed on processors such as the PPU 300 are already being used in a variety of use cases: from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain, constantly learning, getting smarter, and delivering more accurate results faster over time. A child, initially taught by an adult to correctly identify and classify various shapes, eventually becomes able to recognize shapes without any tutoring. Similarly, deep learning or neural learning systems need to be trained in object recognition and classification in order to become smarter and more efficient at recognizing basic objects, occluded objects, and assigning context to objects.

[0130] At the simplest level, neurons in the human brain examine the various inputs they receive, assign a level of importance to each of these inputs, and pass outputs to other neurons for processing. An artificial neuron, or perceptron, is the most basic model of a neural network. In one example, a perceptron can receive one or more inputs representing various features of the object it is being trained to recognize and classify, and each of these features is assigned a certain weight based on its importance in defining the object's shape.

[0131] Deep neural network (DNN) models consist of multiple layers of connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.), which can be trained with large amounts of input data to solve complex problems quickly and accurately. In one example, the first layer of a DNN model breaks down an input image of a car into its components and looks for basic patterns (such as lines and angles). The second layer assembles the lines to find higher-level patterns, such as wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the final layers generate labels for the input image, identifying the model of a specific car brand.

[0132] Once trained, a DNN can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten numbers on a check deposited at an ATM, identifying images of friends in photos, providing movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human speech in real time.

[0133] During training, data flows through the DNN in a forward propagation phase until a prediction is produced, which indicates the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and the weights are adjusted for each feature during a backward propagation phase until the DNN correctly labels that input and other inputs in the training dataset. Training complex neural networks requires a large amount of parallel computing performance, including floating-point multiplications and additions supported by the PPU 300. Inference is a less computationally intensive and latency-sensitive process than training, in which a trained neural network is applied to new inputs it has not seen before to classify images, translate speech, and generally reason about new information.

[0134] Neural networks rely heavily on matrix math operations, and complex, multi-layer networks require significant floating-point performance and bandwidth for efficiency and speed. With thousands of processing cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS of performance, the PPU 300 is a computing platform capable of delivering the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0135] Furthermore, images generated using one or more of the techniques disclosed herein can be used to train, test, or validate DNNs used to recognize objects and environments in the real world. Such images can include scenes of roads, factories, buildings, urban environments, rural environments, people, animals, and any other physical objects or real-world environments. Such images can be used to train, test, or validate DNNs employed in machines or robots to manipulate, process, or modify physical objects in the real world. Furthermore, such images can be used to train, test, or validate DNNs employed in autonomous vehicles to navigate and move the vehicles in the real world. Furthermore, images generated using one or more of the techniques disclosed herein can be used to convey information to users of such machines, robots, and vehicles.

[0136] Graphics processing pipeline

[0137] In one embodiment, the PPU 300 includes a graphics processing unit (GPU). The PPU 300 is configured to receive commands specifying a shader for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying a plurality of vertices of the primitive (e.g., in a model space coordinate system) and attributes associated with each vertex of the primitive. The PPU 300 can be configured to process the primitives to generate a frame buffer (e.g., pixel data for each of the pixels of a display).

[0138] An application writes model data for a scene (e.g., a collection of vertices and attributes) to memory (such as system memory or memory 304). The model data defines each of the objects that may be visible on the display. The application then makes an API call to the driver kernel, requesting the model data to be rendered and displayed. The driver kernel reads the model data and writes commands to one or more streams to perform operations to process the model data. These commands may reference different shading programs to be implemented on the SMs 440 of the PPU 300, including one or more of vertex shading, hull shading, domain shading, geometry shading, and pixel shading. For example, one or more of the SMs 440 may be configured to execute a vertex shading program that processes multiple vertices defined by the model data. In one embodiment, different SMs 440 may be configured to execute different shading programs simultaneously. For example, a first subset of SMs 440 may be configured to execute a vertex shading program, while a second subset of SMs 440 may be configured to execute a pixel shading program. The first subset of SMs 440 processes the vertex data to generate processed vertex data and writes the processed vertex data to the L2 cache 460 and / or memory 304. After the processed vertex data is rasterized (e.g., converted from three-dimensional data to two-dimensional data in screen space) to generate fragment data, a second subset of SMs 440 performs pixel shading to generate processed fragment data, which is then blended with other processed fragment data and written to a frame buffer in memory 304. Vertex shading programs and pixel shading programs can be executed simultaneously, processing different data from the same scene in a pipelined manner, until all model data for the scene has been rendered to the frame buffer. The contents of the frame buffer are then transmitted to a display controller for display on a display device.

[0139] Figure 6A According to one embodiment, Figure 3 300 . The graphics processing pipeline 600 is an abstract flow chart of the processing steps implemented to generate a 2D computer-generated image from 3D geometric data. As is well known, pipeline architectures can perform long-latency operations more efficiently by breaking the operations into multiple stages, where the output of each stage is coupled to the input of the next consecutive stage. Thus, the graphics processing pipeline 600 receives input data 601 that is passed from one stage of the graphics processing pipeline 600 to the next stage to generate output data 602. In one embodiment, the graphics processing pipeline 600 may represent a graphics processing pipeline composed of API-defined graphics processing pipeline. Alternatively, graphics processing pipeline 600 can be implemented within the functional and architectural context of the previous figures and / or one or more of any subsequent figures.

[0140] like Figure 6AAs shown, graphics processing pipeline 600 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 610, a vertex shading stage 620, a primitive assembly stage 630, a geometry shading stage 640, a viewport scale, cull, and clip (VSCC) stage 650, a rasterization stage 660, a fragment shading stage 670, and a raster operation stage 680. In one embodiment, input data 601 includes commands that configure a processing unit to implement the stages of graphics processing pipeline 600 and configure geometric primitives (e.g., points, lines, triangles, quads, triangle strips, or fans, etc.) to be processed by these stages. Output data 602 may include pixel data (i.e., color data), which is copied to a frame buffer or other type of surface data structure in memory.

[0141] The data assembly stage 610 receives input data 601, which specifies vertex data for high-level surfaces, primitives, etc. The data assembly stage 610 collects the vertex data in temporary storage or queues, such as by receiving a command from the host processor that includes a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then passed to the vertex shading stage 620 for processing.

[0142] The vertex shading stage 620 processes vertex data by executing a set of operations (e.g., a vertex shader or program) on each vertex at a time. A vertex may be specified, for example, as a 4-coordinate vector (e.g., ) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normal, etc.).<x,y,z,w> ). The vertex shading stage 620 can manipulate various vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 620 performs operations on the vertex coordinates or other vertex attributes associated with the vertex. These operations typically include lighting operations (e.g., modifying the color attribute of a vertex) and transformation operations (e.g., modifying the coordinate space of a vertex). For example, a vertex can be specified using coordinates in an object coordinate space, which is transformed by multiplying the coordinates by a matrix that converts the coordinates from the object coordinate space to world space or normalized-device-coordinate (NCD) space. The vertex shading stage 620 generates transformed vertex data that is passed to the primitive assembly stage 630.

[0143] The primitive assembly stage 630 collects the vertices output by the vertex shading stage 620 and groups the vertices into geometric primitives for processing by the geometry shading stage 640. For example, the primitive assembly stage 630 can be configured to group every three consecutive vertices into geometric primitives (e.g., triangles) for transmission to the geometry shading stage 640. In some embodiments, particular vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). The primitive assembly stage 630 transmits the geometric primitives (e.g., a collection of associated vertices) to the geometry shading stage 640.

[0144] The geometry shading stage 640 processes geometric primitives by executing a set of operations (e.g., geometry shaders or programs) on the geometric primitives. A tessellation operation can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading stage 640 can subdivide each geometric primitive into a finer mesh of two or more geometric primitives for processing by the rest of the graphics processing pipeline 600. The geometry shading stage 640 passes the geometric primitives to the viewport SCC stage 650.

[0145] In one embodiment, the graphics processing pipeline 600 may operate within a streaming multiprocessor and vertex shading stage 620, primitive assembly stage 630, geometry shading stage 640, fragment shading stage 670, and / or hardware / software associated therewith, and may perform processing operations sequentially. Once the sequential processing operations are completed, in one embodiment, the viewport SCC stage 650 may utilize the data. In one embodiment, primitive data processed by one or more stages in the graphics processing pipeline 600 may be written to a cache (e.g., an L1 cache, a vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 650 may access the data in the cache. In one embodiment, the viewport SCC stage 650 and the rasterization stage 660 are implemented as fixed function circuits.

[0146] The viewport SCC stage 650 performs viewport scaling, culling, and clipping of geometric primitives. Each surface being rendered is associated with an abstract camera position. The camera position represents the position of the viewer viewing the scene and defines a viewing cone that surrounds the objects of the scene. The viewing cone can include a viewing plane, a back plane, and four clipping planes. Any geometric primitives that are completely outside the viewing cone can be culled (e.g., discarded) because they will not contribute to the final rendered scene. Any geometric primitives that are partially inside the viewing cone and partially outside the viewing cone can be clipped (e.g., converted to new geometric primitives that are enclosed within the viewing cone). In addition, each geometric primitive can be scaled based on the depth of the viewing cone. All potentially visible geometric primitives are then transferred to the rasterization stage 660.

[0147] The rasterization stage 660 converts 3D geometric primitives into 2D fragments (e.g., capable of being used for display, etc.). The rasterization stage 660 can be configured to use the vertices of the geometric primitives to set a set of plane equations from which various attributes can be interpolated. The rasterization stage 660 can also calculate a coverage mask for multiple pixels, which indicates whether one or more sample positions of the pixel intercept the geometric primitive. In one embodiment, a z test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have already been rasterized. The rasterization stage 660 generates fragment data (e.g., interpolated vertex attributes associated with a specific sample position for each covered pixel), which is passed to the fragment shading stage 670.

[0148] The fragment shading stage 670 processes the fragment data by executing a set of operations (e.g., a fragment shader or program) on each of the fragments. The fragment shading stage 670 can generate pixel data (e.g., color values) for the fragment, such as by performing lighting operations or sampling a texture map using the fragment's interpolated texture coordinates. The fragment shading stage 670 generates pixel data, which is sent to the raster operations stage 680.

[0149] The raster operations stage 680 may perform various operations on the pixel data, such as performing alpha tests, stencil tests, and blending the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 680 has completed processing the pixel data (e.g., output data 602), the pixel data may be written to a render target, such as a frame buffer, a color buffer, etc.

[0150] It should be appreciated that one or more additional stages may be included in the graphics processing pipeline 600 in addition to or in place of one or more of the above-described stages. Various implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments, one or more of the above-described stages may be excluded from the graphics processing pipeline (such as the geometry shading stage 640). Other types of graphics processing pipelines are contemplated within the scope of the present disclosure. Furthermore, any stage of the graphics processing pipeline 600 may be implemented by one or more dedicated hardware units within a graphics processor (such as the PPU 300). Other stages of the graphics processing pipeline 600 may be implemented by programmable hardware units (such as the SM 440 of the PPU 300).

[0151] The graphics processing pipeline 600 can be implemented via an application program executed by a host processor (such as a CPU). In one embodiment, a device driver can implement an application programming interface (API) that defines various functions that can be utilized by the application program to generate graphics data for display. A device driver is a software program that includes multiple instructions that control the operation of the PPU 300. The API provides an abstraction for programmers, allowing them to utilize specialized graphics hardware (such as the PPU 300) to generate graphics data without requiring them to utilize the specific instruction set of the PPU 300. An application program can include API calls that are routed to the device driver of the PPU 300. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other cases, the device driver can perform operations at least in part by initiating operations on the PPU 300 using an input / output interface between the CPU and the PPU 300. In one embodiment, the device driver is configured to implement the graphics processing pipeline 600 using the hardware of the PPU 300.

[0152] Various programs may be executed within the PPU 300 to implement the various stages of the graphics processing pipeline 600. For example, a device driver may launch a kernel on the PPU 300 to execute the vertex shading stage 620 on one SM 440 (or multiple SMs 440). The device driver (or the initial kernel executed by the PPU 400) may also launch other kernels on the PPU 400 to execute other stages of the graphics processing pipeline 600, such as the geometry shading stage 640 and the fragment shading stage 670. In addition, some of the stages of the graphics processing pipeline 600 may be implemented on fixed unit hardware, such as a rasterizer or data assembler implemented within the PPU 400. It should be appreciated that the results from one kernel may be processed by one or more intermediate fixed-function hardware units before being processed by subsequent kernels on the SM 440.

[0153] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device can be directly coupled to a system or processor that generates or renders the image. In other embodiments, the display device can be indirectly coupled to the system or processor, for example, via a network. Examples of such networks include the Internet, mobile telecommunications networks, Wi-Fi networks, and any other wired and / or wireless networking systems. When the display device is indirectly coupled, the image generated by the system or processor can be streamed to the display device via the network. For example, such streaming allows, for example, a video game or other application that renders an image to be executed on a server or in a data center, and the rendered image is transmitted and displayed on one or more user devices (such as computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separated from the server or data center. Therefore, the techniques disclosed herein can be applied to enhanced streaming images and services that enhance streaming images, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0154] Rendering via ray tracing

[0155] While rendering tasks are typically accomplished using raster-based rendering techniques, some algorithms for performing rendering utilize a light transport simulation known as ray tracing. Ray tracing algorithms simulate the interaction of light with virtual objects in a scene in order to calculate the color of a pixel of an image. For each pixel of an image defined by the viewport and the position of a virtual camera, a ray is cast into the scene. By casting a new ray from the point at which a ray intersects a virtual object, effects such as diffuse and specular reflection, refraction, and absorption can be simulated at each intersection of a ray with a virtual object. As a ray interacts with a light source, the path of the light from the virtual camera to the light source can be traced to determine the color of the light that passes through the pixel corresponding to the ray. The aggregate color from multiple rays passing through a pixel can be calculated to generate the color of a pixel of an image.

[0156] In some embodiments, the PPU 300 can be used to render images according to various ray tracing techniques. For example, ray casting can be implemented for multiple rays in parallel by executing multiple threads on one or more SMs 440. Each thread compares the parameters of the ray with the representation of the virtual objects in the scene to determine whether the ray intersects with the virtual objects. In some embodiments, the representation of the virtual objects includes a bounding volume hierarchy that is used to efficiently determine whether the ray intersects with any virtual objects in the scene. The ray that is determined to intersect with the virtual object can generate one or more new rays corresponding to a second plurality of threads executed by the SM 440 to simulate lighting effects. When the ray intersects with the light source, the chain of rays from the virtual camera to the light source is identified as a light path, and the color of the pixel can be calculated for the light path.

[0157] Ray tracing algorithms can be used to produce realistic computer-generated images. Such algorithms have been used to produce animated features in movies or television programs, simulated images for commercial applications (e.g., images included in manuals or catalogs), and the like. Although ray tracing has traditionally been too complex for real-time applications, advances in processing power and underlying algorithms have made real-time ray tracing practical in some applications.

[0158] Game streaming system

[0159] Figure 6B is an example system diagram of a game streaming system 605 according to some embodiments of the present disclosure. Figure 6B Includes one or more game servers 603 (which may include Figure 5B The example processing system 500 and / or Figure 5C ), one or more client devices 604 (which may include components, features, and / or functionality similar to the exemplary system 565 of Figure 5B The example processing system 500 and / or Figure 5C 5 ) and one or more networks 606 (which may be similar to one or more networks described herein). In some embodiments of the present disclosure, system 605 may be implemented.

[0160] In system 605, for a game session, one or more client devices 604 can receive input data only in response to input to one or more input devices, transmit the input data to one or more game servers 603, receive encoded display data from the one or more game servers 603, and display the display data on a display 624. In this way, more computationally intensive calculations and processing are offloaded to one or more game servers 603 (e.g., rendering—specifically, ray or path tracing—for one or more GPUs of one or more game servers 603 to perform graphical output for the game session). In other words, the game session is streamed from one or more game servers 603 to one or more client devices 604, thereby reducing the graphics processing and rendering requirements of one or more client devices 604.

[0161] For example, with respect to instantiation of a game session, client device 604 may display a frame of the game session on display 624 based on display data received from game server 603. Client device 604 may receive input from one of one or more input devices and generate input data in response. Client device 604 may transmit the input data to one or more game servers 603 via communication interface 621 and over one or more networks 606 (e.g., the Internet), and one or more game servers 603 may receive the input data via communication interface 618. The CPU may receive the input data, process the input data, and transmit the data to the GPU, causing the GPU to generate a rendering of the game session. For example, the input data may represent movement of a user's character in the game, firing a weapon, reloading, passing a ball, turning a vehicle, etc. Rendering component 612 may render the game session (e.g., representing the results of the input data), and rendering capture component 614 may capture the rendering of the game session as display data (e.g., capturing image data of a rendered frame of the game session). Rendering of the game session may include ray or path tracing lighting and / or shadow effects calculated using one or more parallel processing units (such as GPUs), which may further employ one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques on one or more game servers 603. The encoder 616 may then encode the display data to generate encoded display data, and the encoded display data may be transmitted to the client device 604 via the network 606 via the communication interface 618. The client device 604 may receive the encoded display data via the communication interface 621, and the decoder 622 may decode the encoded display data to generate display data. The client device 604 may then display the display data via the display 624. It will be appreciated that one or more stages of the pipelined stack 100 may be implemented by the game server 603 and / or the client device 604.

[0162] Evaluating stream quality

[0163] Figure 7 Logic for calculating a rolling weighted mean metric for components derived from streaming session metadata is shown in accordance with some embodiments. As described above, the metadata for a streaming session includes various timestamps associated with different stages or components of the pipelined stack 100. Component scores 732 and weights 714 associated with component scores 732 may be generated by a module 700 configured to process metadata. As used herein, module 700 may be implemented in hardware, software, or a combination of hardware and software. In an embodiment, module 700 is implemented in software executed by one or more processors of a server device. For example, module 700 may be implemented as a set of instructions stored in a memory and executed by one or more processors.

[0164] In one embodiment, the raw value 702 includes a delta calculated based on two or more timestamps read from the metadata. The raw value 702 is calculated for each frame of the streaming session, and the raw value 702 for a given frame can be derived based on the timestamp associated with the frame and the timestamps associated with one or more individual frames. For example, components may include, but are not limited to, capture delta, render delta, RTD, and stack delay.

[0165] like Figure 7 , per-frame raw values 702 for a particular component are received by a component module 700. In one embodiment, the raw values 702 are stored in an array stored in a memory, the array being generated using a processor configured to process metadata for each frame in a streaming session to calculate the raw values 702. The processor generates two or more arrays corresponding to two or more components of a quality metric (e.g., one or more stutter components and one or more delay components).

[0166] The raw values are then processed using a pair of transfer functions 710 to generate scores 712 and weights 714 . Figure 8A and Figure 8B A pair of transfer functions for the stutter component of a quality metric according to some embodiments is shown. The transfer functions are provided for illustrative purposes only and should not be construed as limiting the embodiments disclosed herein. The specific transfer functions implemented by the embodiments may be selected based on statistical analysis of a large amount of streaming session metadata and accurately selected or filtered subjective quality metrics to maximize the correlation between the objective quality metric and the subjective quality metric.

[0167] like Figure 8A As depicted in FIG, a transfer function 802 for a score of a stutter component of a quality metric, such as a first stutter component that may also be referred to as stutter associated with frame presentation, is shown in the figure, which maps a raw value 702 given in milliseconds to a dimensionless score 712 within a specified range. In this case, score 712 is mapped to a range between 0 and 100, but in other embodiments, the range may be selected to be a range of arbitrary values and may include negative as well as positive values. Thus, for each raw value 702 of a stutter component associated with a frame of a streaming session, score 712 is generated by mapping the raw value 702 to a corresponding value given by the transfer function.

[0168] Figure 8AThe transfer function 802 shown in FIG can be described as follows. If the raw value 702 is less than a minimum threshold (referred to as Delta_Min), the score 712 is equal to the maximum score (e.g., 100). If the raw value 702 is greater than the minimum threshold, the score 712 is equal to the maximum score in the linear region (e.g., Score_Linear_Max), which may be less than the maximum score (e.g., 50), multiplied by the ratio of the maximum score in the linear region to the difference between the maximum and minimum values of the range of the linear region (e.g., Delta_Max and Delta_Min), multiplied by the difference between the raw value 702 and Delta_Min. The Delta_Max value can be set equal to the total delta (e.g., the difference between the calculated timestamp difference and the expected frame rate), above which all raw values are assigned a score of 0, which in this example is set to 116 ms. Delta_Min can be set equal to a multiplier of the frame rate (e.g., 1.5*1000 / {frames_per_second}).

[0169] like Figure 8B , a second transfer function 804 of the weights of the jamming component of the quality metric is shown on a graph that maps the raw scores 702 to dimension weights in a specified range. As shown herein, the weights 714 are also given on a scale between 0 and 100, but a different scale may be selected for the weight transfer function (e.g., a scale between 0.0 and 1.0, a scale between 0 and 200, etc.).

[0170] Figure 9A and Figure 9B A pair of transfer functions for the delay component of a quality metric according to some embodiments is shown. For example, the delay component can be RTD. It will be appreciated that the transfer function for the delay component is different from the transfer function for the stutter component shown above. By implementing the use of different transfer functions, the raw delay value can be interpreted as having different scores and weights that reflect the relative differences in the impact of various components on the final quality metric.

[0171] like Figure 9A, a transfer function 902 of the fraction of the delay component of the quality metric is shown on a graph that maps raw values 702 given in milliseconds to dimensionless fractions 712 within a specified range. The transfer function 902 maps raw values 702 in the linear region between Delta_Min and Delta_Max of 0 and 100, respectively, to fractions 712. Above Delta_Max, the fraction 712 is given as 0; otherwise, the fraction ranges between 100 and 0 in a linear fashion from Delta_Min to Delta_Max. Although Delta_Min is set equal to the minimum possible delta value (0 ms) in this example, in some embodiments, Delta_Min may be set to a value, for example, between 16-32 ms, such that any raw value 702 below this threshold is considered equal to the maximum fraction.

[0172] like Figure 9B As depicted in FIG, transfer function 904 uses a linearly increasing function to map raw values 702 to weights 714. In an embodiment, the weights are set equal to 1 / 10 of the raw values up to a maximum threshold (e.g., 200). In some embodiments, weights 714 may also be set to a minimum value below a minimum threshold (e.g., a weight equal to 1.0 for raw values below 10 ms).

[0173] It will be appreciated that different raw values 702 may be assigned the same score 712, but different weights 714, such that the contribution of the component to the quality metric is adjusted based on a combination of two different transfer functions and is not limited to a single transfer function. For example, raw values of 200ms and 1000ms may both be considered well outside the normal range, such that they are both assigned a score equal to the minimum score. However, by applying different weights, the raw value of 1000ms may contribute more to the quality metric than the raw value of 200ms.

[0174] return Figure 7Scores 712 and corresponding weights 714 are provided to a sliding window unit 720, which applies a sliding window to calculate a rolling weighted mean of scores 712 based on corresponding weights 714. It will be appreciated that higher scores correspond to good performance, while lower scores correspond to poor performance. To characterize a streaming session, there should be a way to aggregate scores over time (e.g., aggregate scores across multiple frames included in the streaming session). One goal is to spread severe drops in scores across multiple frames. For example, intuitively, a sudden but severe freeze will typically be perceived by a user as affecting more than a single frame, which may have experienced the problem. The user's perception is that the multiple frames surrounding the bad frame are also bad, because the user cannot immediately fully recover from the event when the next frame is correctly displayed. In other words, negative experiences will tend to contribute to our subjective perception: other experiences immediately following a negative experience will also be more negative than they would be in isolation. Another goal is that more frequent but less severe drops in scores should be weighted differently than isolated severe drops.

[0175] In one embodiment, the size of the sliding window is X frames (e.g., X=30, 600, etc.) For each frame, a rolling weighted mean of a particular component is calculated by summing the weighted scores 712 of the current frame and X-1 previous frames included in the sliding window, where each score is multiplied by a corresponding weight 714 before summing, and the weighted sum is normalized by the sum of the weights 714 within the sliding window to generate a rolling weighted mean (RWM) 722.

[0176] Finally, a final component quality score 732 is calculated as a per-frame value by selecting the minimum of the score 712 or the RWM 722 for each frame via selection logic 730 .

[0177] In some embodiments, the component module 700 can be configured to sequentially calculate multiple per-frame component scores 732 and corresponding weights 714 for two or more different components based on each component's different transfer functions 710. In other embodiments, multiple component modules 700 can operate in parallel to calculate the per-frame component scores 732 and corresponding weights 714 for two or more components in parallel.

[0178] Figure 10Logic for calculating a combined score representing the quality of a streaming session based on multiple per-frame component scores is illustrated in accordance with some embodiments. Component data for multiple components (e.g., per-frame component scores 732 and corresponding weights 714) is received from one or more modules 700. Module 1000 processes the component data to generate a combined score 1032. In one embodiment, combined score 1032 represents at least a portion of a quality metric for a streaming session based on stutter and delay components. As used herein, module 1000 may be implemented in hardware, software, or a combination of hardware and software. In one embodiment, module 1000 is implemented in software executed by one or more processors of a server device. For example, module 1000 may be implemented as a set of instructions stored in a memory and executed by one or more processors.

[0179] In an embodiment, the combination logic 1010 receives at least four sets of component data from one or more modules 700. The component data may include, but is not limited to, component scores 732 and corresponding weights 714 for frame capture and presentation jank, as well as RTD and stack latency. The combination logic 1010 calculates a per-frame weighted mean (referred to as a combined weighted score (CWS) 1012) by summing the partial products of each component, which are calculated by multiplying the component scores 732 by the corresponding weights 714 and then normalizing the sum by dividing by the sum of the weights 714 over all components.

[0180] CWS 1012 is a per-frame value of component scores 732 based on multiple components related to jank and latency. It will be appreciated that jank and latency components have been selected for one implementation of the quality metric; however, in other embodiments, different components may be included in the quality metric in addition to or in place of the jank / latency components disclosed herein.

[0181] The CWS 1012 is then processed by distribution logic 1020, which calculates parameters for the distribution of the CWS 1012 values based on a second sliding window. In one embodiment, the size of the second sliding window is Y frames (e.g., Y=30, 600, etc.). It will be appreciated that the size of the second sliding window may differ from the size of the first sliding window implemented in module 700. For each frame, a rolling mean 1022 and a rolling standard deviation (STD) 1024 of the CWS 1012 values within the second sliding window are calculated. The rolling mean 1022 is calculated by summing the CWS 1012 values and dividing by Y. The rolling STD 1024 is then calculated by taking the square root of the sum of the squared differences between the CWS 1012 values and the rolling mean 1022.

[0182] In one embodiment, the rolling mean 1022 may be used as the combined score 1032. However, as Figure 10As shown, in other embodiments, confidence unit 1030 processes rolling mean 1022 and rolling STD 1024. Confidence unit 1030 is configured to subtract a plurality of rolling STDs 1024 from rolling mean 1022 to generate a combined score 1032. In an embodiment, the multiple is 6.0, but other multiples (e.g., 1.5, 3.0, etc.) are contemplated for use with lower confidence levels. It will be appreciated that confidence unit 1030 is used to ensure that combined score 1032 reflects the low range of the distribution of CWS 1012 values included in the sliding window, such that the majority of values in the distribution (e.g., based on the plurality of STDs) are higher than combined score 1032, and in a manner that adjusts the average quality score within the second sliding window based on the variance of the calculated quality scores within a specified time period. Thus, for two different samples of CWS values 1012 within the sliding window, combined score 1032 will be lower for the sample set with the higher variance, even when the two sets of samples have the same rolling mean 1022.

[0183] It will be appreciated that the combined score 1032 is still calculated as a per-frame value, even though each combined score 1032 is a reflection of the raw value 702 of metadata from multiple frames in the neighborhood of a given frame according to both the first sliding window and the second sliding window implemented in modules 700 and 1000. While a graph of the combined score 1032 over the course of a streaming session is useful for visualizing how variables in terms of objective quality are set for a particular streaming session, it is often more useful to characterize a streaming session in terms of the session as a whole rather than just on a frame-by-frame basis. As a result, the stall / delay combined score can be combined with the per-session picture quality score to calculate a quality metric for the streaming session.

[0184] Figure 11 Logic for generating quality metrics for a streaming session according to some embodiments is shown. Figure 11 As shown, the quality metric 1132 is based on metadata. In an embodiment, the streaming experience engine 1130 calculates the quality metric by combining both the picture quality component and the freeze / delay component. The freeze / delay component is generated by using module 1000 to calculate a base score based on two different parameter sets. In a first instance of module 1000-1, the first parameter set 1002 configures the first sliding window to use parameter X=600 and the second sliding window to use parameter Y=3600, which correspond to sliding windows of 10 seconds and 1 minute at a frame rate of 60 Hz, respectively. In a second instance of module 1000-2, the second parameter set 1004 configures the first sliding window to use parameter X=3600 and the second sliding window to use parameter Y=600.

[0185] The stutter / delay quality engine 1110 then receives two sets of combined scores 1032 corresponding to different parameter sets. These per-frame combined scores are then aggregated to produce an aggregate mean and an aggregate standard deviation (STD). The combined scores 1032 from the first parameter set 1102 are aggregated by calculating a weighted mean of the per-frame combined scores 1032 to generate the aggregate mean, where the weights are selected based on the remote input frequency. The remote input frequency refers to a measure of how often the client device registers input (e.g., key presses, mouse events, etc.), and therefore, the aggregate mean is weighted based on the amount of input activity.

[0186] In one embodiment, the remote input signal is collected by the client device according to a 10 Hz sampling rate. If any input was received in the previous 100 ms, the sample of the remote input signal is active (e.g., 1 or logic high), and conversely, if no input was received in the previous 100 ms, the sample of the remote input is inactive (e.g., 0 or logic low). It will be appreciated that multiple frames may overlap with a single remote input sample (e.g., due to differences in sampling frequency and frame rate), and that the weight of inactive remote input signals may be non-zero so that the aggregate mean is not based solely on a base fraction of active remote input periods.

[0187] In some embodiments, the weights may be based on a sliding window of remote input signals. For example, a sliding window of 100 samples (e.g., 10 seconds) is used to calculate the frequency of the remote input signal within the sliding window. The remote input frequency is then the ratio of active samples to the total samples in the sliding window, which in the example provided herein using 100 samples at a 10 Hz sampling frequency would be an integer value between 0 and 100. In one embodiment, the minimum weight is 1.0, so that even if no input was received in the previous 10 seconds, the weight is set to 1.0 rather than 0.0.

[0188] The aggregate mean provides a base score for the stutter / delay quality component. The base score is an offset, where the fractional offset value is calculated as the aggregate STD of the combined scores 1032 based on the second parameter set 1104. It should be understood that the aggregate STD is not simply the standard deviation of the combined scores 1032 used to calculate the aggregate mean of the base score, as the aggregate STD is based on the combined scores 1032 generated using different parameters for the sliding window size. The aggregate STD is calculated by taking the square root of the sum of the squared differences between the combined scores 1032 and the mean of the combined scores 1032 using the combined scores 1032 based on the second parameter set 1104. The stutter / delay component of the quality metric is then generated by subtracting the fractional offset (e.g., the aggregate STD) from the base score (e.g., the aggregate mean).

[0189] The picture component of the quality metric is calculated differently from the stutter / delay component because the raw values related to picture quality are not expected to change much during a streaming session. In a first embodiment, the picture quality score is derived simply by mapping the target bitrate per pixel for the streaming session, which is constant throughout the session, to a fractional value using a transfer function 1202. Figure 12 A transfer function is shown that maps a target bit rate (ie, original value 1106) to a picture quality score 1112 according to some embodiments. The transfer function 1202 represents a curve fit to the following set of points:

[0190] Table 1

[0191]

[0192]

[0193] The fraction may be limited to a maximum fraction of 100 (ie, bpps exceeding the maximum target bitrate of 1.608 are all set to a fraction of 100).

[0194] Now return to Figure 11 In some embodiments, the target bitrate may occasionally change during the course of a streaming session. For example, network conditions may be monitored by the server device (e.g., based on returned metadata), and the server device may adjust the target bitrate based on the network conditions to try and help improve the experience of the streaming session. In such embodiments, the target bitrate, and therefore the score for picture quality, is not constant across the entire streaming session. As a result, the picture quality engine 1120 may be configured to calculate a base score and a score offset for picture quality. The base score is generated by calculating the mean of the picture quality scores over the course of the session, and the score offset is generated by calculating the STD of the picture quality scores over the course of the session. The picture quality component of the quality metric is then calculated by subtracting a multiple of the score offset (e.g., 1.5, etc.) from the base score.

[0195] Finally, the streaming experience engine calculates a quality metric 1132 by summing the picture quality component of the quality metric generated by the picture quality engine 1120 with the stutter / latency component of the quality metric generated by the stutter / latency quality engine 1110. In some embodiments, the quality metric 1032 is also shifted and / or normalized (e.g., by adding a base quality fraction of 50 to the unshifted quality metric or by multiplying the unshifted quality metric by 0.5 before shifting).

[0196] It will be appreciated that the techniques described above are not limited to streaming video games or VR / AR applications. Anything that requires a pipelined stack implemented in a client-server framework and requires some type of client-side feedback is contemplated as being within the scope of this disclosure. For example, video teleconferencing applications and / or business productivity software streaming as a service are all within the scope of this disclosure.

[0197] It should be noted that the techniques described herein may be embodied in executable instructions stored in a computer-readable medium for use by or in conjunction with a processor-based instruction execution machine, system, device, or apparatus. Those skilled in the art will appreciate that for some embodiments, different types of computer-readable media may be included for storing data. As used herein, "computer-readable medium" includes one or more of any suitable media for storing executable instructions of a computer program, such that an instruction execution machine, system, device, or apparatus can read (or obtain) instructions from the computer-readable medium and execute instructions for implementing the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable compact disks (CDs), portable digital video disks (DVDs), and the like.

[0198] It should be understood that the arrangement of the components shown in the drawings is for illustrative purposes, and other arrangements are also possible. For example, one or more of the elements described herein may be implemented as an electronic hardware assembly in whole or in part. Other elements may be implemented with software, hardware, or a combination of software and hardware. In addition, some or all of these other elements may be combined, some elements may be omitted entirely, and additional components may be added while still implementing the functions described herein. Thus, the subject matter described herein may be embodied in many different variations, and all such variations are contemplated to be within the scope of the claims.

[0199] In order to promote understanding of the subject matter described herein, many aspects are described with respect to action sequences. Those skilled in the art will recognize that various actions can be performed by dedicated circuits or circuits, by program instructions executed by one or more processors, or by a combination of the two. The description of any action sequence herein is not intended to imply that the described particular order for executing the sequence must be followed. Unless otherwise indicated herein or the context clearly contradicts, all methods described herein can be performed in any suitable order.

[0200] In the context of describing a subject matter (particularly in the context of the following claims), the use of the terms "a" and "an" and "the" and similar references should be interpreted to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. The use of the term "at least one" followed by a list of one or more items (e.g., "at least one of A and B") should be interpreted to mean one item selected from the listed items (A or B) or any combination of two or more items in the listed items (A and B), unless otherwise indicated herein or clearly contradicted by the context. In addition, the foregoing description is for illustrative purposes only and not for limiting purposes, as the scope of protection sought is defined by the set forth claims and any equivalents thereof. The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the subject matter and does not limit the scope of the subject matter unless otherwise required. The use of the term "based on" and other similar phrases indicating conditions that cause a result in the claims and written description is not intended to exclude any other conditions that cause the result. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention as claimed.

Claims

1. A system for evaluating the quality of a streaming session provided by a streaming service, the system comprising: One or more processors configured to: receiving metadata for a plurality of frames included in the streaming session; calculating, for each frame of the plurality of frames, a score and a weight for each component of a plurality of components derived from the metadata; calculating a quality metric for the streaming session based on the scores and weights of the plurality of components; as well as adjusting at least one component of a pipeline implemented by the streaming service based on the quality metric; wherein the plurality of components include at least one of the following components: a first jitter component based on a delta between a first timestamp corresponding to a frame rendered by the application and a second timestamp corresponding to the frame captured by the first stage of the pipeline; - a second jitter component based on a delta between a first timestamp corresponding to presentation of the frame on the client device and a second timestamp corresponding to presentation of a previous frame on the client device; - a first delay component based on the round-trip delay of the frame; as well as - A second delay component based on the stack delay of the frame.

2. The system of claim 1, wherein the quality metric is calculated as the sum of a picture quality score and a combined stall-latency score.

3. The system of claim 2 , wherein the combined jank-latency score is calculated by: For each of the plurality of frames, calculating a rolling weighted score for each of at least one delay component and at least one jamming component based on a first sliding window; and For each of the at least one delay component and the at least one jam component, a rolling mean of a combined weighted sum of the rolling weighted scores is calculated, wherein the rolling mean of the combined weighted sum is based on a second sliding window.

4. The system of claim 3, wherein the combined jam-latency score is calculated as the difference between a total weighted mean calculated based on a first set of parameters and a standard deviation based on a second set of parameters. 5 . The system of claim 2 , wherein the picture quality score is calculated as a difference between a mean of a plurality of picture quality scores for the plurality of frames and a product of a coefficient and a standard deviation of the plurality of picture quality scores for the plurality of frames.

6. The system of claim 1 , wherein adjusting the at least one component of the pipeline implemented by the streaming service based on the quality metric comprises: At least one of a quantization parameter, a target bit rate and encoding parameters, or a target resolution of the frame generated by the pipeline is adjusted. 7 . The system of claim 1 , wherein the one or more processors are further configured to generate a visual representation of the quality metric of the streaming session.

8. A method for evaluating a quality metric of a streaming session, the method comprising: receiving metadata for a plurality of frames included in the streaming session, wherein the metadata includes information related to a number of stages in a pipeline implemented by the streaming service, and wherein the pipeline includes at least one stage implemented by a server device and at least one additional stage implemented by a client device connected to the server device via a network; calculating, for each frame of the plurality of frames, a score and a weight for each component of a plurality of components derived from the metadata; as well as calculating a quality metric for the streaming session based on the scores and weights of the plurality of components; wherein the plurality of components include at least one of the following components: a first jitter component based on a delta between a first timestamp corresponding to a frame rendered by the application and a second timestamp corresponding to the frame captured by the first stage of the pipeline; - a second jitter component based on a delta between a first timestamp corresponding to presentation of the frame on the client device and a second timestamp corresponding to presentation of a previous frame on the client device; - a first delay component based on the round-trip delay of the frame; as well as - A second delay component based on the stack delay of the frame.

9. The method according to claim 8, further comprising: generating a visual representation of the quality metric for one or more streaming sessions; or At least one component of the pipeline is adjusted based on the quality metric.

10. The method of claim 8, wherein the quality metric is calculated as the sum of a picture quality score and a combined stall-latency score.

11. The method of claim 10, wherein the combined jam-latency score is calculated by: For each of the plurality of frames, calculating a rolling weighted score for each of at least one delay component and at least one jamming component based on a first sliding window; and For each of the at least one delay component and the at least one jam component, a rolling mean of a combined weighted sum of the rolling weighted scores is calculated, wherein the rolling mean of the combined weighted sum is based on a second sliding window.

12. The method of claim 11, wherein the combined jam-latency score is calculated as the difference between a total weighted mean calculated based on a first set of parameters and a standard deviation based on a second set of parameters. 13 . The method of claim 10 , wherein the picture quality score is calculated as a difference between a mean of a plurality of picture quality scores for the plurality of frames and a product of a coefficient and a standard deviation of the plurality of picture quality scores for the plurality of frames.

14. The method according to claim 8, further comprising: At least one of a quantization parameter, a target bitrate and encoding parameters, or a target resolution of the frames generated by the pipeline is adjusted in response to the quality metric.

15. A non-transitory computer-readable medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform steps comprising: receiving metadata for a plurality of frames included in a streaming session, wherein the metadata includes information related to a number of stages in a pipeline implemented by the streaming service, and wherein the pipeline includes at least one stage implemented by a server device and at least one additional stage implemented by a client device connected to the server device via a network; calculating, for each frame of the plurality of frames, a score and a weight for each component of a plurality of components derived from the metadata; as well as calculating a quality metric for the streaming session based on the scores and weights of the plurality of components; wherein the plurality of components include at least one of the following components: a first jitter component based on a delta between a first timestamp corresponding to a frame rendered by the application and a second timestamp corresponding to the frame captured by the first stage of the pipeline; - a second jitter component based on a delta between a first timestamp corresponding to presentation of the frame on the client device and a second timestamp corresponding to presentation of a previous frame on the client device; - a first delay component based on the round-trip delay of the frame; as well as - A second delay component based on the stack delay of the frame.

16. The computer-readable medium of claim 15, wherein the steps further comprise: generating a visual representation of the quality metric for one or more streaming sessions; or At least one component of the pipeline is adjusted based on the quality metric.

17. The computer-readable medium of claim 15, wherein: The quality metric is calculated as the sum of a picture quality score and a combined stall-latency score; The jank-latency score for the combination is calculated using the following steps: For each frame of the plurality of frames, calculating a rolling weighted score of each of at least one delay component and at least one jamming component based on a first sliding window; as well as For each of the at least one delay component and the at least one jam component, calculating a rolling mean of a combined weighted sum of the rolling weighted scores, wherein the rolling mean of the combined weighted sum is based on a second sliding window; as well as The picture quality score is calculated as a difference between a mean of a plurality of picture quality scores for the plurality of frames and a product of a coefficient and a standard deviation of the plurality of picture quality scores for the plurality of frames.

18. The computer-readable medium of claim 17, wherein the combined jam-latency score is calculated as a difference between a total weighted mean calculated based on a first set of parameters and a standard deviation based on a second set of parameters.

Images