Dynamic image smoothing based on network conditions

Abstract

Dynamic image smoothing based on network conditions is disclosed. Systems and methods for dynamically smoothing images based on network conditions to adjust the bit rate required to transmit the images are disclosed. Content in the images is smoothed to reduce the number of bits required to encode each image. Filtering an image modifies areas containing content with high frequency pixel changes, reducing the frequency, so the pixel colors in that area appear "smoothed" or uniform. In other words, an area of an image showing a lawn with grass can have high frequency changes from pixel to pixel due to the fine details of individual blades of grass that are similar in color but not uniform. Encoding that area as a single green shade (or a multi-pixel area of different green shades) allows a viewer to recognize it as a lawn with grass while greatly reducing the number of bits required to represent the area.

Description

BACKGROUND

[0001] Cloud gaming services enable clients to access games hosted on servers by performing many of the processing tasks in the server and streaming the results of the processing (e.g., graphics output) to the local client device. The quality of the stream depends on network conditions, such as the available channel bandwidth. When the bandwidth is low, users experience blurring and blocking artifacts in the frames, and sometimes even dropped frames resulting in stuttering. Conventionally, to improve the end user experience, dynamic resolution conversion (DRC) is used to reduce the frame resolution whenever the bandwidth is low, and transmit the lower resolution frames to the client, where frame upscaling is then performed. The problem with DRC is that every time the resolution is changed, the encoder and decoder must be reset. Also, after the reset, the encoder transmits an IDR (Intra Decoder Refresh) coded picture (a special type of I-frame) to clear the contents of the reference picture buffer in the decoder and prevent the use of any previous frames as reference frames. The first frame transmitted at the new resolution is coded as an IDR frame, resulting in a spike in bitrate even when reducing the resolution. The decoder reinitialization to use the new IDR frame can cause frame rate jitter. These problems and / or other problems associated with the prior art need to be addressed. SUMMARY

[0002] Embodiments of the present disclosure relate to dynamically filtering images based on network conditions. Systems and methods are disclosed that dynamically adjust the bitrate required to transmit an image by smoothing high spatial frequency content based on network conditions. Rather than changing the pixel resolution of the image to accommodate network conditions, as with conventional systems such as the one described above, regions and / or content in the image are smoothed to reduce the number of bits required to encode each image. Filtering the image modifies regions containing content with high frequency of pixel variation, reducing the frequency so that the pixel colors in that region appear “smoothed.” For example, an image region showing a lawn full of grass can have a high frequency of variation from pixel to pixel due to the fine details of individual blades of grass that are similar in color. Encoding the region as a single green shade (or a multi-pixel region of different green shades) allows a viewer to recognize it as a lawn full of grass while greatly reducing the number of bits required to represent the region and the bitrate required to transmit the encoded image. BRIEF DESCRIPTION OF DRAWINGS

[0003] The current systems and methods for dynamic image smoothing based on network conditions are described in detail below with reference to the attached drawing figures, wherein:

[0004] Figure 1A A block diagram of an example server / client system suitable for implementing some embodiments of the present disclosure is shown.

[0005] Figure 1B An image according to one embodiment and a version of the smoothed image are shown.

[0006] Figure 1C A flowchart of a method for performing dynamic image smoothing according to one embodiment is shown.

[0007] Figure 2A A DRC diagram for bit rate reduction is shown according to existing technology.

[0008] Figure 2B A diagram is shown for dynamic image smoothing for bitrate reduction according to some embodiments of the present disclosure.

[0009] Figure 2C A flowchart of another method for performing dynamic image smoothing according to one embodiment is shown.

[0010] Figure 3A Two versions are shown: an image according to one embodiment and a smoothed image.

[0011] Figure 3B Some embodiments suitable for implementing this disclosure are shown. Figure 1C The flowchart shown is a flowchart of the steps in the flowchart.

[0012] Figure 4 Example parallel processing units suitable for implementing some embodiments of this disclosure are shown.

[0013] Figure 5A This is applicable to the use of some embodiments of this disclosure. Figure 4 A conceptual diagram of the processing system implemented by the PPU.

[0014] Figure 5B Exemplary systems in which various architectures and / or functions of various prior embodiments can be implemented are shown.

[0015] Figure 6A This is applicable to implementing some embodiments of this disclosure. Figure 4 A conceptual diagram of the graphics processing pipeline implemented by the PPU.

[0016] Figure 6B An exemplary game streaming system suitable for implementing some embodiments of this disclosure is shown. Detailed Implementation

[0017] Systems and methods related to dynamically smoothing images based on network conditions are disclosed. Image smoothing can be applied to reduce the number of bits required to encode the image. Small-scale details (e.g., texture content in the image) can be smoothed by filtering high spatial frequency content within the image. In the context described below, smoothing reduces entropy within the image, which alters pixel values ​​and enables more efficient encoding of the smoothed image. As network conditions improve, filtering can be adjusted to reduce smoothing and increase the bit rate.

[0018] The amount of smoothing applied can be dynamically adjusted based on network conditions to provide clients accessing applications on the server with an interactive experience featuring visually meaningful images. In particular, important image details (e.g., object edges and structure) are preserved even as the smoothing amount increases. Dynamic smoothing responds quickly to changes in network conditions and does not require resetting the encoder and / or decoder. Perceptibly important features remain visible, thus improving the end-user experience compared to conventional techniques that use reduced image resolution and / or jittered frame rates.

[0019] Figure 1A A block diagram of an example server / client system 100 suitable for implementing some embodiments of this disclosure is shown. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware and / or software, or any combination thereof. For example, various functions can be performed by a processor executing instructions stored in memory. Moreover, those skilled in the art will understand that any system performing the operation of the server / client system 100 is within the scope and spirit of the embodiments of this disclosure.

[0020] like Figure 1A As shown, server 110 transmits encoded data to client 140 via a channel. Although server / client system 100 is described in the context of transmitting image data, in some embodiments, other types of data (encoded or unencoded), such as auxiliary information depth buffers, normals, etc., can be dynamically smoothed and transmitted.

[0021] Client 140 provides network condition data 142 to server 110. In another embodiment, network monitor 135 or an additional network monitor 135 is configured within server 110 to provide network condition data 142. Network condition data 142 may include data related to available bandwidth, latency, jitter, and channel packet loss. Based on network condition data 142, image smoothing control 125 generates parameters to dynamically control the filtering of high spatial frequency content via image smoothing filter 115 and / or game capture unit 105. In one embodiment, image smoothing control 125 adjusts parameters when a predetermined threshold of network condition data 142 is exceeded. In one embodiment, image smoothing control 125 increases the smoothing amount when network condition data indicates that channel capacity is exceeded, and decreases the smoothing amount when channel capacity is not exceeded.

[0022] To dynamically adjust the bitrate, image smoothing is performed at server 110 before the frames are encoded for transmission to client 140. Image smoothing is achieved by filtering high spatial frequency content within the image. Image smoothing can be applied by game capture unit 105 during image rendering, or it can be applied to the rendered image by image smoothing filter 115. Typically, image smoothing is performed before the image frames are encoded. Game capture unit 105 generates the rendered image and may include a game engine that renders at least a portion of the rendered image based on at least one parameter received from image smoothing control 125.

[0023] In one embodiment, parameters include a texture LOD (Level of Detail) bias, which controls the resolution of the texture mapping applied to a surface during rendering. For example, texture mappings can be applied to surfaces such as characters, scene backgrounds, terrain, buildings, or structures. Texture mappings are filtered to generate each lower-resolution LOD, resulting in a set of texture mappings (e.g., mip mappings) corresponding to different LODs. The biased LOD is calculated to reduce image quality using lower-resolution texture mappings, and typically improves rendering speed due to improved texture data caching. The image quality degradation caused by using lower-resolution texture mappings appears as blurry artifacts in the rendered image. Generally, texture mapping LODs can be biased to maintain a minimum frame rate for real-time rendering. The biased LOD is a first filtering mechanism that can be used to generate a smooth image. A second filtering mechanism can be used instead of the first filtering mechanism or in addition to the first filtering mechanism to apply filters to the rendered image.

[0024] The rendered image output by the game capturer 105 can be filtered by the image smoothing filter 115 based on one or more parameters received from the image smoothing control 125. Typically, the image smoothing filter 115 implements filters to process pixels to blur small-scale details while preserving structural edges of the geometry within the rendered image frame to generate a smooth image frame. In one embodiment, the image smoothing filter 115 implements a scroll-guided filter, which may include a bilateral filter and / or a domain transformation filter.

[0025] In one embodiment, the image smoothing control 125 also receives user-defined controls for determining parameters. The user-defined controls can specify the type of filtering performed by the image smoothing filter 115 and / or the game capture unit 105. For example, an edge-preserving smoothing filter function can be selected or defined for the image smoothing filter 115. In one embodiment, the user-defined controls can specify the levels of minimum and / or maximum smoothing (or LOD bias).

[0026] Encoder 120 encodes (compresses) the smoothed image frame to generate an encoded image frame for transmission to client 140. Decoder 130 decodes (decompresses) the encoded image frame to generate a smoothed image frame at client 140. Network monitor 135 detects changes in network conditions and provides updates to image smoothing control 125. In one embodiment, network monitor 135 continuously or periodically provides network condition data 142 to image smoothing control 125.

[0027] Image smoothing is compatible with existing encoders and decoders in conventional server / client systems. Unlike DRC, image smoothing does not require resetting encoder 120 or decoder 130 when the smoothing amount changes. The basic idea behind image smoothing is that an image consists of thousands of pixels, but not all pixels are equally important. According to the HVS (Human Visual System) model, only overall structural features are important to human perception, and individual details (e.g., high-frequency details such as texture and artifacts) are not important. Therefore, image smoothing can extract meaningful information from an image. Structural details in the image (e.g., edges and corners of geometric objects and text) are preserved, while complex details like textures and small detail patterns are removed. For example, while preserving tree branches (small primitives) and road edges, textures applied to the sky or ground can be smoothed to a constant color. Importantly, when image smoothing is applied to reduce the bit rate, text remains sharp. Conversely, when DRC is used to reduce the bit rate, text appears blurry.

[0028] Now, based on user needs, we will provide further illustrative information about the various optional architectures and features that can be used to implement the aforementioned framework. It should be particularly noted that the following information is presented for illustrative purposes only and should not be construed as limiting in any way. Any of the following features may be incorporated into or not exclude other features described, as needed.

[0029] Figure 1B Image 140 and a smoothed version, image 150, are shown according to one embodiment. Image smoothing filter 115 processes image 140 to produce image 150. Image 140 is rendered without any modification to the LOD bias value used for texture mapping. In other embodiments, image 140 may be rendered with a modified LOD bias value. Figure 1B As shown, certain high-frequency details of the content within region 142 are reduced in smoothing region 152. Specifically, pixels with similar colors that vary with pixel size are colored with a single color. The edges of bright areas of the content within region 142 are preserved within smoothing region 152, while darker areas appear uniform. Similarly, high-frequency details of the content within region 144 appear uniform in smoothing region 154. Conversely, compared to image 140, the outlines of characters and the shadows of trees remain sharp in smoothed image 150. The smoothed image 150 retains important details and structure, allowing gameplay to continue without reducing image resolution or jittering the frame rate. In some embodiments, bitrate savings of 20-35% are achieved using image smoothing filter 115. When the LOD bias value is modified, bitrate savings can be as high as 75% (e.g., a 4x reduction in bitrate).

[0030] Generally, images that are dynamically smoothed via filtering are sharper than those with DRC. When using DRC, small text becomes blurry due to the scaling down and up process, while smoothed images retain sharpness. For some smoothed images, object edges are sharper even compared to the original image, and smoothing also removes noise around object edges. Furthermore, for some image sequences, blocky artifacts are reduced in the smoothed version of the image sequence compared to the original image sequence.

[0031] Figure 1C A flowchart of a method 170 for performing dynamic image smoothing according to one embodiment is shown. Each block of the method 170 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. The method can be provided by a standalone application, service, or managed service (standalone or in combination with other managed services) or a plug-in to another product, to name a few. Additionally, by way of example, relative to Figure 1AThe system described herein describes method 170. However, this method may be additionally or alternatively performed by any system or any combination of systems, including but not limited to the systems described herein. Furthermore, those skilled in the art will understand that any system performing method 170 is within the scope and spirit of the embodiments of this disclosure.

[0032] In step 175, network status data 142 for the connection between server 110 and client 140 is received. In one embodiment, image smoothing control 125 receives network status data 142. The network status data may include data related to one or more of available bandwidth, latency, jitter, and packet loss for the connection, or data representing one or more of available bandwidth, latency, jitter, and packet loss for the connection.

[0033] In step 180, in response to changes in network condition data, parameters controlling the filtering of high spatial frequency content are adjusted. In one embodiment, the smoothing amount achieved by filtering is increased when network condition data 142 indicates that connection quality has deteriorated. In one embodiment, network condition data 142 is evaluated based on one or more predetermined thresholds associated with a Quality of Service (QoS) metric. In one embodiment, the smoothing amount achieved by filtering is decreased when network condition data 142 indicates that connection quality has improved.

[0034] In step 185, an image frame is generated based on the parameter. In one embodiment, the parameter corresponds to a LOD bias, and the LOD bias is used to render the image to reduce the entropy of the texture surface. In another embodiment, the parameter corresponds to a smoothing filter, and the rendered image is filtered based on the parameter to reduce high-frequency spatial variations in pixels. The image smoothing control 125 can generate one or more parameters to control the LOD bias and / or filtering for each rendered image.

[0035] Figure 2A A graph 200 is shown illustrating DRC for bitrate reduction according to prior art. Graph 200 shows the available bandwidth of the connection from server to client over time. Graph 200 also shows the change in bitrate consumed by dynamic smoothing techniques and DRC. At time 201, the available bandwidth of the connection from server to client decreases from level one to level two. In response, a conventional DRC system reduces the image resolution, thereby increasing the bitrate during the insertion of lower-resolution I-frames. The encoder and decoder are reset to change the resolution used for DRC.

[0036] The transmission of an IDR frame after time 201 causes a bit rate spike, after which the DRC bit rate drops below the available bandwidth. This bit rate spike can be particularly problematic because the available bandwidth has decreased. Furthermore, the client may be unable to maintain a constant frame rate, resulting in frame jitter visible to the end user. When the available bandwidth increases back to level 1 from level 2 at time 202, regular DRC increases the image resolution. Changing the resolution requires resetting the encoder and decoder, and transmitting another IDR frame after time 202. For DRC, the bit rate spikes again during the transmission of the second IDR frame and then adjusts to a stable bit rate. Even when available bandwidth increases, resetting the decoder and / or the bit rate spike can still cause frame jitter.

[0037] Compared to DRC, dynamic smoothing does not require resetting the encoder and / or decoder and does not cause peaks in bitrate or frame jitter. Furthermore, dynamic smoothing can continuously adapt the filtering to the available bandwidth. Due to the variations in each image, it is difficult to determine the amount of bitrate change that a specific adjustment to this parameter will result in. Therefore, in one embodiment, the adjusted parameter can reduce or increase the bitrate by a larger or smaller amount than required to not exceed the available bandwidth. In other words, the adjustment of the parameter can be conservative in response to changes in network condition data 142, and then fine-tuned or optimized. The parameter can be continuously adjusted when the bitrate measured by server 110 at encoder 120 is outside the threshold range of the available bandwidth.

[0038] Figure 2B A graph 250 illustrating dynamic image smoothing for bit rate reduction according to some embodiments of this disclosure is shown. As shown in graph 250, the dynamic smoothing bit rate can increase from a start time, decrease the available bandwidth at time 201, and then increase again at time 203 to approach the level of the available bandwidth. Adjusting parameters to rapidly increase the smoothing level reduces the bit rate used to transmit the dynamically smoothed image. Image smoothing control 125 can continue to adjust parameters after time 203 based on measured bit rate and network condition data 142, such that the bit rate increases while still remaining below the available bandwidth. In another embodiment, the parameter is constant until the available bandwidth changes beyond a predetermined value. Image smoothing control 125 can delay increasing the bit rate (decreasing the smoothing level) to ensure that the increase in available bandwidth is consistent. Similarly, when the available bandwidth decreases, image smoothing control 125 can delay decreasing the bit rate (increasing the smoothing level). Image smoothing control 125 can use different delays to adjust parameters to increase or decrease the bit rate. In one embodiment, a shorter delay is used to decrease the bit rate, while a longer delay is used to increase the bit rate. Delays can be scheduled, programmed, or calculated based on network condition data 142.

[0039] As shown in graph 250, the available bandwidth increases at time 202. In response, at time 204, the image smoothing control 125 adjusts its parameters to decrease the level of smoothing, and the bit rate used to transmit the encoded image frame increases. Since the increased bit rate at time 204 is higher than the available bandwidth, the image smoothing control 125 adjusts its parameters to increase smoothing, reducing the bit rate to below the available bandwidth at time 205. The image smoothing control 125 can continue to adjust its parameters to fine-tune the bit rate after time 205.

[0040] Unlike traditional DRC techniques, image smoothing does not require resetting the encoder and / or decoder to insert IDR frames. Instead, at least one of the encoded image frames transmitted before parameter adjustment can be used by the decoder as an I-frame to decode encoded image frames received after parameter adjustment. When bit rate resets and / or spikes are avoided, the frame rate of the decoded image frames is constant before and after parameter adjustment.

[0041] Figure 2C A flowchart of another method 220 for performing dynamic image smoothing according to one embodiment is shown. Method 220 includes steps 175, 180, and 185 of method 170. Steps 175, 180, and 185 are performed as previously described. At step 225, image smoothing control 125 determines whether a transmission from server 110 to client 140 is within a threshold range for network conditions (e.g., as determined by one or more QoS metrics). For example, if the transmission bit rate is greater than the available bandwidth indicated by network condition data 142, the transmission is not within the threshold range. If the transmission bit rate is equal to or less than the available bandwidth, the transmission is within the threshold range.

[0042] At step 225, if the image smoothing control 125 determines that the transmission is outside the threshold range, then at step 230, the image smoothing control 125 adjusts the parameters before returning to step 185. The image smoothing control 125 can adjust one or more parameters to control filtering for each rendered image via LOD bias and / or image smoothing.

[0043] At step 225, if the image smoothing control 125 determines that the transmission is within network conditions, then at step 235, the image smoothing control 125 maintains a constant parameter until the network conditions data 142 changes. In one embodiment, the image smoothing control 125 maintains a constant parameter until the network conditions data 142 and / or the transmission bit rate changes.

[0044] As previously described, the LOD bias used for rendering an image can be controlled via the image smoothing control 125 to adjust the bitrate of the encoded image frames. In some embodiments, the LOD bias is not exposed or inaccessible to the outside of the application running on the game capture unit 105. When control over the LOD bias is not exposed, the image smoothing control 125 may not be able to set the value of the LOD bias used during rendering.

[0045] Figure 3A Two versions, an image 300 and a smoothed image, are shown according to one embodiment. The game capture unit 105 renders image 300 using parameters provided by the image smoothing control 125 without modifying the texture LOD value. The game capture unit 105 may apply an LOD bias, which is not based on the parameters but is instead determined by the game program itself without knowledge of the network condition data 142. In one embodiment, the parameters provided by the image smoothing control 125 have higher priority than the LOD bias determined by the game program (if any), and take precedence over the LOD bias determined by the game program.

[0046] Image 320, rendered by game capture unit 105, has a first LOD bias value controlled by image smoothing control 125. Image 330, rendered by game capture unit 105, has a second LOD bias value controlled by image smoothing control 125. No smoothing filter is applied to image 300 to produce images 320 and 330. The first LOD bias value can reduce the highest texture mapping resolution from 256x256 to 64x64, and each lower resolution mip map is similarly biased downward by four in each dimension. The second LOD bias value can reduce the highest texture mapping resolution from 256x256 to 8x8, and each lower resolution mip map is similarly biased downward by 32 in each dimension. In one embodiment, image smoothing control 125 adjusts parameters according to the range of available bandwidth for controlling smoothing to control the LOD bias value to a predetermined value. In one embodiment, game capture unit 105 supports mip map LOD bias values ​​from 0 to 15 for 16 different levels of texture LOD.

[0047] In one or more embodiments, and as Figure 3AAs shown, high spatial frequency details within region 305 are reduced in smooth region 325, and even more so in the uniformly smooth region 335. Conversely, regions excluding texture-mapped surfaces are unaffected by LOD bias adjustments. For example, trees on the horizon are constructed as primitives (typically small triangles), so the structure of the trees appears preserved in images 320 and 330. Similarly, the structure and appearance of characters are preserved in images 320 and 330. Text within region 310 is preserved within smooth regions 328 and 338. Similarly, text, symbols, and icons within a head-up display (HUD) are preserved.

[0048] Smoothing images 320 and 330 preserve important details and structure well, allowing gameplay to continue without reducing image resolution or jittering frame rates. Bitrate savings of 58% and 62% are achieved using LOD bias adjustments for smoothed images 320 and 330, respectively. In one embodiment, image smoothing filter 115 can be configured to apply a filter to an image that has already been rendered using an LOD bias value provided by image smoothing control 125. In such an embodiment, image smoothing control 125 can provide a first parameter for controlling the LOD bias and a second parameter for controlling the smoothing filter.

[0049] Although the smoothing technique is described in the context of a rendered image, it can also be applied to other images, such as those captured "in the wild." However, for unrendered images, the LOD bias value may not be used to perform smoothing. Furthermore, when the game capture unit 105 does not support modification or control of the LOD bias used for texture mapping, modifying the LOD bias parameters has no effect on smoothing.

[0050] Figure 3B A flowchart of step 180 of the flowchart shown in Figure 1C, applicable to implementing some embodiments of this disclosure, is illustrated. At step 340, image smoothing control 125 determines whether the LOD bias used by game capture unit 105 to render an image can be controlled. If the LOD bias can be controlled, at step 345, image smoothing control 125 adjusts the LOD bias parameters provided to game capture unit 105 before proceeding to step 350. Adjusting the LOD bias parameters controls the filtering of high spatial frequency texture maps in the rendered image. If the LOD bias cannot be controlled, image smoothing control 125 proceeds directly from step 340 to step 350. At step 350, game capture unit 105 renders the image according to the LOD bias. At step 355, image smoothing control 125 adjusts the parameters controlling the filtering applied by image smoothing filter 115 to reduce high spatial frequency content. Reducing high spatial frequency content lowers the bitrate used for encoded image frames transmitted from server 110 to client 140.

[0051] Compared to conventional techniques such as DRC, using dynamic image smoothing via LOD bias and / or image filtering to adjust the bitrate of image transmission from server 110 to client 140 provides an improved user experience. The bitrate can be quickly increased or decreased in response to changes in available bandwidth to accommodate low latency without causing frame jitter. Dynamic image smoothing does not require resetting the encoder and / or decoder. In one embodiment, dynamic image smoothing can be implemented as a plugin for streaming tools, allowing applications to benefit from it without modification. Dynamic image smoothing technology provides an engaging experience for end users even at reduced bitrates by preserving structural details of text, head-up display (HUD), and rendered images.

[0052] Parallel processing architecture

[0053] Figure 4 A parallel processing unit (PPU) 400 according to one embodiment is shown. The PPU 400 can be used to implement server 110 and / or client 140 of server / client system 100. The PPU 400 can be used to implement one or more of a game capture unit 105, an image smoothing filter 115, an image smoothing control 125, an encoder 120, a decoder 130, and a network monitor within server / client system 100.

[0054] In one embodiment, PPU 400 is a multi-threaded processor implemented on one or more integrated circuit devices. PPU 400 is a latency-hidden architecture designed to process many threads in parallel. A thread (e.g., an execution thread) is an instance of a set of instructions configured to be executed by PPU 400. In one embodiment, PPU 400 is a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to generate two-dimensional (2D) image data for display on a display device. In other embodiments, PPU 400 may be used to perform general-purpose computing. While an exemplary parallel processor is provided herein for illustrative purposes, it should be strongly noted that such a processor is presented for illustrative purposes only and any processor may be employed to complement and / or replace it.

[0055] One or more PPU 400s can be configured to accelerate thousands of high-performance computing (HPC), data center, cloud computing, and machine learning applications. PPU 400s can be configured to accelerate numerous deep learning systems and applications for autonomous vehicles, simulations, computational graphics such as ray or path tracking, deep learning, high-precision speech, image, and text recognition systems, intelligent video analytics, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data analytics, astronomy, molecular dynamics simulations, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations.

[0056] like Figure 4 As shown, PPU 400 includes an input / output (I / O) unit 405, a front-end unit 415, a scheduler unit 420, a job allocation unit 425, a hub 430, a crossbar (Xbar) 470, one or more general purpose processing clusters (GPCs) 450, and one or more memory partitioning units 480. PPU 400 can be connected to a host processor or other PPU 400 via one or more high-speed NVLink 410 interconnects. PPU 400 can be connected to a host processor or other peripheral devices via interconnect 402. PPU 400 can also be connected to local memory 404, which includes multiple storage devices. In one embodiment, local memory may include multiple dynamic random access memory (DRAM) devices. The DRAM devices may be configured as a high-bandwidth memory (HBM) subsystem, with multiple DRAM dies stacked within each device.

[0057] The NVLink 410 interconnect enables systems to scale and include one or more PPUs 400s combined with one or more CPUs, supporting cache coherency between the PPUs 400 and the CPU, as well as the CPU controller. The NVLink 410 can transfer data and / or commands to or from other units of the PPU 400 (e.g., one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown)) via hub 430. Figure 5A The NVLink 410 is described in more detail.

[0058] I / O unit 405 is configured to transmit and receive communications (e.g., commands, data, etc.) from a host processor (not shown) via interconnect 402. I / O unit 405 may communicate directly with the host processor via interconnect 402 or via one or more intermediate devices (e.g., memory bridges). In one embodiment, I / O unit 405 may communicate with one or more other processors (e.g., one or more PPUs 400) via interconnect 402. In one embodiment, I / O unit 405 implements a Peripheral Component Interconnect Fast (PCIe) interface for communication on a PCIe bus, and interconnect 402 is a PCIe bus. In alternative embodiments, I / O unit 405 may implement other types of well-known interfaces for communication with external devices.

[0059] I / O unit 405 decodes packets received via interconnect 402. In one embodiment, the packets represent commands configured to cause PPU 400 to perform various operations. I / O unit 405 transmits the decoded commands to various other units of PPU 400 as specified by the commands. For example, some commands may be transmitted to front-end unit 415. Other commands may be transmitted to hub 430 or other units of PPU 400, such as one or more copy engines, video encoders, video decoders, power management units, etc. (not explicitly shown). In other words, I / O unit 405 is configured to route communication among and between various logical units of PPU 400.

[0060] In one embodiment, a program executed by the host processor encodes a command stream in a buffer that provides a workload to the PPU 400 for processing. The workload may include multiple instructions and data to be processed by those instructions. A buffer is an area in memory that can be accessed (e.g., read / write) by both the host processor and the PPU 400. For example, I / O unit 405 may be configured to access a buffer in system memory connected to interconnect 402 via a memory request transmitted over interconnect 402. In one embodiment, the host processor writes a command stream to the buffer and then transmits a pointer to the beginning of the command stream to the PPU 400. Front-end unit 415 receives pointers to one or more command streams. Front-end unit 415 manages one or more streams, reads commands from these streams, and forwards the commands to the respective units of the PPU 400.

[0061] Front-end unit 415 is coupled to scheduler unit 420, which configures various GPCs 450 to process tasks defined by one or more flows. Scheduler unit 420 is configured to track status information related to the various tasks managed by scheduler unit 420. This status can indicate which GPC 450 a task is assigned to, whether the task is active or inactive, the priority associated with the task, etc. Scheduler unit 420 manages the execution of multiple tasks on one or more GPCs 450.

[0062] Scheduler unit 420 is coupled to job allocation unit 425, which is configured to allocate tasks to be executed on GPC 450. Job allocation unit 425 can track multiple scheduled tasks received from scheduler unit 420. Job allocation unit 425 manages each GPC 450's pending task pool and active task pool. When GPC 450 completes task execution, the task is evicted from GPC 450's active task pool, and one of the other tasks from the pending task pool is selected and scheduled for execution on GPC 450. If an active task is idle on GPC 450 (e.g., waiting to resolve data dependencies), the active task is evicted from GPC 450 and returned to the pending task pool, while another task from the pending task pool is selected and scheduled for execution on GPC 450.

[0063] In one embodiment, the host processor executes a driver kernel that implements an application programming interface (API) that enables one or more applications executing on the host processor to schedule operations for execution on the PPU 400. In one embodiment, the PPU 400 executes multiple computing applications concurrently, and the PPU 400 provides isolation, Quality of Service (QoS), and independent address spaces for the multiple computing applications. Applications can generate instructions (e.g., API calls) that cause the driver kernel to generate one or more tasks for execution by the PPU 400. The driver kernel outputs tasks to one or more streams being processed by the PPU 400. Each task may include one or more associated thread groups, referred to herein as warps. In one embodiment, a warp includes 32 associated threads that can execute in parallel. Cooperating threads can refer to multiple threads that include instructions for executing tasks and can exchange data via shared memory. Tasks can be assigned to one or more processing units within the GPC 450, and instructions can be scheduled for execution by at least one warp.

[0064] The work allocation unit 425 communicates with one or more GPCs 450 via XBar 470. XBar 470 is an interconnect network that couples multiple units of PPU 400 to other units of PPU 400. For example, XBar 470 can be configured to couple the work allocation unit 425 to a specific GPC 450. Although not explicitly shown, one or more other units of PPU 400 can also be connected to XBar 470 via hub 430.

[0065] Tasks are managed by scheduler unit 420 and assigned to GPC 450 by job allocation unit 425. GPC 450 is configured to process tasks and generate results. Results may be consumed by other tasks within GPC 450, routed to different GPC 450 via XBar 470, or stored in memory 404. Results may be written to memory 404 via memory partitioning unit 480, which implements a memory interface for reading and writing data to / from memory 404. Results may be transferred to another PPU 400 or CPU via NVLink 410. In one embodiment, PPU 400 includes U number of memory partitioning units 480, which is equal to the number of separate and distinct memory devices of memory 404 coupled to PPU 400. Each GPC 450 may include a memory management unit to provide virtual address to physical address translation, memory protection, and arbitration of memory requests. In one embodiment, the memory management unit provides one or more Translation Lookahead Buffers (TLBs) for translating virtual addresses to physical addresses in memory 404.

[0066] In one embodiment, memory partitioning unit 480 includes a raster operation (ROP) unit, a secondary (L2) cache, and a memory interface coupled to memory 404. The memory interface can implement 32, 64, 128, or 1024-bit data buses for high-speed data transfer. PPU 400 can connect to up to Y storage devices, such as high-bandwidth memory stacks or Graphics Dual Data Rate version 5, synchronous dynamic random access memory, or other types of persistent storage. In one embodiment, the memory interface implements an HBM2 memory interface, and Y is equal to half of U. In one embodiment, the HBM2 memory stack and PPU 400 reside in the same physical package, resulting in significant power and area savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory dies, and Y equals 4, with each HBM2 stack including two 128-bit channels per die for a total of eight channels and a 1024-bit data bus width.

[0067] In one embodiment, memory 404 supports Single Error Corrected Double Error Detection (SECDED) error correction code (ECC) to protect data. ECC provides enhanced reliability for computational applications sensitive to data corruption. Reliability is especially critical in large-scale cluster computing environments where the PPU 400 processes very large datasets and / or long-running applications.

[0068] In one embodiment, PPU 400 implements a multi-level memory hierarchy. In one embodiment, memory partitioning unit 480 supports unified memory to provide a single unified virtual address space for the CPU and PPU 400 memory, thereby enabling data sharing between virtual memory systems. In one embodiment, the frequency of PPU 400 accesses to memory located on other processors is tracked to ensure that memory pages are moved to the physical memory of the PPU 400 that accesses the page more frequently. In one embodiment, NVLink 410 supports address translation services, allowing PPU 400 to directly access the CPU's page tables and providing full access to the CPU's memory.

[0069] In one embodiment, the replication engine transfers data between multiple PPUs 400 or between a PPU 400 and a CPU. The replication engine can generate page faults for addresses not mapped to page tables. Memory partitioning unit 480 can then service the page faults, mapping the addresses to page tables, after which the replication engine can perform the transfer. In conventional systems, multiple replication engines operating on fixed (e.g., non-pageable) memory across multiple processors significantly reduce available memory. Due to hardware page faults, addresses can be passed to the replication engine without concern for whether memory pages reside, and the replication process is transparent.

[0070] Data from memory 404 or other system memory can be fetched by memory partitioning unit 480 and stored in L2 cache 460, which is located on-chip and shared among the various GPCs 450. As shown, each memory partitioning unit 480 includes a portion of the L2 cache associated with the corresponding memory 404. Lower-level caches can then be implemented in individual cells within the GPC 450. For example, each processing unit within the GPC 450 can implement a Level 1 (L1) cache. The L1 cache is a dedicated memory for a specific processing unit. L2 cache 460 is coupled to memory interface 470 and XBar 470, and data from the L2 cache can be fetched and stored in each L1 cache for processing.

[0071] In one embodiment, the processing unit within each GPC 450 implements a SIMD (Single Instruction, Multiple Data) architecture, where each thread in a group of threads (e.g., a thread bundle) is configured to process different datasets based on the same instruction set. All threads in the thread group execute the same instructions. In another embodiment, the processing unit implements a SIMT (Single Instruction, Multiple Threads) architecture, where each thread in a group of threads is configured to process different datasets based on the same instruction set, but where individual threads in the thread group are allowed to diverge during execution. In one embodiment, a program counter, call stack, and execution state are maintained for each thread bundle, thereby enabling concurrency between the thread bundle and the serial execution within the thread bundle when threads diverge. In another embodiment, a program counter, call stack, and execution state are maintained for each individual thread, thereby ensuring equal concurrency among all threads within and between thread bundles. When an execution state is maintained for each individual thread, threads executing the same instructions can be converged and executed in parallel for maximum efficiency.

[0072] Collaboration groups are a programming model for organizing groups of communicating threads. This model allows developers to express the granularity at which threads are communicating, enabling richer and more efficient parallel decompositions. The Collaboration Startup API supports synchronization between blocks of threads to execute parallel algorithms. Conventional programming models provide a single, simple structure for synchronizing collaborating threads: a barrier across all threads in a block (e.g., the `syncthreads()` function). However, programmers often want to define thread groups at a smaller granularity than thread blocks and synchronize within the defined groups to achieve higher performance, design flexibility, and software reuse in the form of a collective group-wide functional interface.

[0073] Collaboration groups enable programmers to explicitly define thread groups at both the sub-block (e.g., as small as a single thread) and multi-block granularity, and to perform collective operations, such as synchronization, on threads within the collaboration group. This programming model supports clear composition across software boundaries, so library and utility functions can be safely synchronized in their local contexts without having to make assumptions about convergence. The collaboration group primitive enables new cooperative parallelism patterns, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across the entire thread block mesh.

[0074] Each processing unit comprises a large number (e.g., 128, etc.) of different processing cores (e.g., functional units), which may be fully pipelined, single-precision, double-precision, and / or mixed-precision, and includes floating-point arithmetic logic units and integer arithmetic logic units. In one embodiment, the floating-point arithmetic logic unit implements the IEEE 754-2008 standard for floating-point arithmetic. In one embodiment, the core comprises 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

[0075] Tensor cores are configured to perform matrix operations. Specifically, tensor cores are configured to perform deep learning matrix arithmetic, such as convolution operations used for neural network training and inference. In one embodiment, each tensor core operates on a 4×4 matrix and performs matrix multiplication and accumulation operations D = A×B + C, where A, B, C, and D are 4×4 matrices.

[0076] In one embodiment, matrix multiplication inputs A and B are 16-bit floating-point matrices, while accumulation matrices C and D can be either 16-bit or 32-bit floating-point matrices. The Tensor Core performs 32-bit floating-point accumulation on the 16-bit floating-point input data. The 16-bit floating-point multiplication requires 64 operations and produces a full-precision product, which is then accumulated with other intermediate products using 32-bit floating-point addition for a 4x4x4 matrix multiplication. In practice, the Tensor Core consists of larger two-dimensional or higher-dimensional matrix operations composed of these smaller elements. APIs such as the CUDA 9 C++ API expose specialized matrix loading, matrix multiplication and accumulation, and matrix storage operations to efficiently utilize the Tensor Core from CUDA-C++ programs. At the CUDA level, the warp-level interface assumes a 16x16 matrix spanning all 32 threads of the warp.

[0077] Each processing unit may further include M Special Function Units (SFUs) that perform special functions (e.g., attribute evaluation, inverse square root, etc.). In one embodiment, an SFU may include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, an SFU may include a texture unit configured to perform texture map filtering operations. In one embodiment, a texture unit is configured to load a texture map (e.g., a 2D array of texture pixels) from memory 404 and sample the texture map to produce sampled texture values ​​for use in a shader program executed by the processing unit. In one embodiment, the texture map is stored in shared memory, which may include or include an L1 cache. The texture unit implements texture operations, such as filtering operations using mip-maps (e.g., texture maps with different levels of detail). In one embodiment, each processing unit includes two texture units.

[0078] Each processing unit also includes N Load Memory Units (LSUs) that implement load and store operations between shared memory and the register file. Each processing unit includes an interconnect network that connects each core to the register file and the LSUs to the register file and shared memory. In one embodiment, the interconnect network is a crossbar switch that can be configured to connect any core to any register in the register file and the LSUs to storage locations in the register file and shared memory.

[0079] Shared memory is an array of on-chip memory that allows data storage and communication between processing units and between threads within a processing unit. In one embodiment, the shared memory includes a storage capacity of 128KB and is located in the path from each processing unit to memory partition unit 480. The shared memory can be used for cache reads and writes. One or more of the shared memory, L1 cache, L2 cache, and memory 404 are backup storage.

[0080] Combining data caching and shared memory functionality into a single memory block provides optimal overall performance for both types of memory access. This capacity is used as a cache by programs that do not use shared memory. For example, if shared memory is configured to use half its capacity, texture and load / store operations can use the remaining capacity. This integration within shared memory allows it to be used as a high-throughput pipeline for streaming data, while providing high-bandwidth and low-latency access to frequently reused data.

[0081] When configured for general-purpose parallel computing, a simpler configuration can be used compared to graphics processing. Specifically, a more streamlined programming model is created by bypassing fixed-function graphics processing units. In the general-purpose parallel computing configuration, the work allocation unit 425 directly allocates and distributes blocks of threads to processing units within the GPC 450. Threads execute the same program, using unique thread IDs in computation to ensure each thread produces unique results. One or more processing units are used to execute programs and perform computations. Shared memory is used for communication between threads, and LSUs are used to read from and write to global memory via shared memory and memory partitioning unit 480. When configured for general-purpose parallel computing, processing units can also write commands, which the scheduler unit 420 can use to start new work on the processing unit.

[0082] Each PPU 430 may include and / or be configured to perform the functions of one or more processing cores and / or components thereof, such as a tensor core (TC), tensor processing unit (TPU), pixel visualization core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element and / or similar content.

[0083] The PPU 400 can be included on desktop computers, laptop computers, tablet computers, servers, supercomputers, smartphones (e.g., wireless, handheld devices), personal digital assistants (PDAs), digital cameras, vehicles, head-mounted displays, handheld electronic devices, etc. In one embodiment, the PPU 400 is embodied on a single semiconductor substrate. In another embodiment, the PPU 400 is included in a system-on-a-chip (SoC) along with one or more other devices (e.g., additional PPU 400, memory 404, reduced instruction set computer (RISC) CPU, memory management unit (MMU), digital-to-analog converter (DAC), etc.).

[0084] In one embodiment, the PPU 400 may be included on a graphics card that includes one or more storage devices. The graphics card may be configured to interface with a PCIe slot on a desktop computer motherboard. In yet another embodiment, the PPU 400 may be an integrated graphics processing unit (iGPU) or a parallel processor included in the motherboard's chipset.

[0085] Example computing system

[0086] As developers expose and leverage more parallelism in applications such as artificial intelligence computing, systems with multiple GPUs and CPUs are being used across various industries. High-performance GPU-accelerated systems with tens of thousands of compute nodes have been deployed in data centers, research facilities, and supercomputers to solve increasingly larger problems. With the increasing number of processing devices in high-performance systems, communication and data transmission mechanisms need to be scaled to support the increased bandwidth.

[0087] Figure 5A This is based on the use of one embodiment. Figure 4 A conceptual diagram of a processing system 500 implemented by a PPU 400. An exemplary system 565 can be configured to implement... Figure 1C , Figure 2B and Figure 3BThe methods 170, 220 and / or step 185 are shown. The processing system 500 includes a CPU 530, a switch 510 and multiple PPUs 400 and corresponding memory 404.

[0088] The NVLink 410 provides a high-speed communication link between each PPU 400. Although in Figure 5A A specific number of NVLink 410 and interconnect 402 connections are shown, but the number of connections to each PPU 400 and CPU 530 can vary. Switch 510 interfaces between interconnect 402 and CPU 530. PPU 400, memory 404, and NVLink 410 can reside on a single semiconductor platform to form a parallel processing module 525. In one embodiment, switch 510 supports two or more protocols to interface between various different connections and / or links.

[0089] In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between each PPU 400 and CPU 530, and provides a switch 510 interface between interconnect 402 and each PPU 400. PPU 400, memory 404, and interconnect 402 may reside on a single semiconductor platform to form a parallel processing module 525. In yet another embodiment (not shown), interconnect 402 provides one or more communication links between each PPU 400 and CPU 530, and uses NVLink 410 to provide a switch 510 interface between each PPU 400 to provide one or more high-speed communication links between PPUs 400. In another embodiment (not shown), NVLink 410 provides one or more high-speed communication links between PPUs 400 and CPU 530 via switch 510. In yet another embodiment (not shown), interconnect 402 directly provides one or more communication links between each PPU 400. One or more of the NVLink 410 high-speed communication links can be implemented as physical NVLink interconnects or on-chip or die-on interconnects using the same protocol as the NVLink 410.

[0090] In the context of this specification, a single semiconductor platform can refer to a single integrated circuit fabricated on a bare die or on a chip, based on a single semiconductor. It should be noted that the term "single semiconductor platform" can also refer to a multi-chip module with increased connectivity, simulating on-chip operation and representing a substantial improvement over conventional bus implementations. Of course, various circuits or devices can be placed separately or in various combinations of semiconductor platforms, depending on user requirements. Alternatively, the parallel processing module 525 can be implemented as a circuit board substrate, and each PPU 400 and / or memory 404 can be a packaged device. In one embodiment, the CPU 530, switch 510, and parallel processing module 525 reside on a single semiconductor platform.

[0091] In one embodiment, the signaling rate of each NVLink 410 is 20 to 25 gigabits per second, and each PPU 400 includes six NVLink 410 interfaces (e.g., Figure 5A As shown, each PPU 400 includes five NVLink 410 interfaces. Each NVLink 410 provides a data transfer rate of 25 gigabits per second in each direction, with six links providing 400 gigabits per second. When the CPU 530 also includes one or more NVLink 410 interfaces, the NVLink 410 can be dedicated to PPU-to-PPU communication (e.g., ...). Figure 5A (as shown), or some combination of PPU-to-PPU and PPU-to-CPU.

[0092] In one embodiment, NVLink 410 allows direct load / store / atomic access from CPU 530 to memory 404 of each PPU 400. In one embodiment, NVLink 410 supports coherent operations, allowing data read from memory 404 to be stored in the cache hierarchy of CPU 530, thereby reducing cache access latency to CPU 530. In one embodiment, NVLink 410 includes support for Address Translation Service (ATS), allowing PPU 400 to directly access page tables in CPU 530. One or more of NVLink 410s can also be configured to operate in a low-power mode.

[0093] Figure 5B An exemplary system 565 is shown in which various architectures and / or functions of various prior embodiments can be implemented. The exemplary system 565 can be configured to implement, respectively... Figure 1C , Figure 2B and Figure 3B Methods 170, 220 and / or step 185 are shown.

[0094] As shown in the figure, a system 565 is provided, which includes at least one central processing unit 530 connected to a communication bus 575. The communication bus 575 may directly or indirectly couple one or more of the following devices: main memory 540, network interface 535, one or more CPUs 530, one or more display devices 545, one or more input devices 560, switch 510, and parallel processing system 525. The communication bus 575 may be implemented using any suitable protocol and may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. The communication bus 575 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Fast (PCIe) bus, HyperTransport, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, one or more CPUs 530 may be directly connected to main memory 540. Furthermore, one or more CPUs 530 can be directly connected to the parallel processing system 525. In cases where the connections between components are direct or point-to-point, the communication bus 575 may include PCIe links to perform the connections. In these examples, the PCI bus does not need to be included in system 565.

[0095] although Figure 5B The various blocks are shown as being connected to the line via a communication bus 575, but this is not intended to be limiting, but merely for clarity. For example, in some embodiments, presentation components, such as one or more display devices 545, can be considered I / O components, such as one or more input devices 560 (e.g., if the display is a touchscreen). As another example, one or more CPUs 530 and / or parallel processing systems 525 may include memory (e.g., in addition to the parallel processing systems 525, CPUs 530, and / or main memory 540 and / or other components, main memory 540 may represent a storage device). In other words, Figure 5B The computing devices described are for illustrative purposes only. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," and / or other device or system types; all are referred to as such. Figure 5B Within the scope of computing devices.

[0096] System 565 also includes main memory 540. Control logic (software) and data are stored in main memory 540, which may take the form of various computer-readable media. Computer-readable media can be any available media that system 565 can access. Computer-readable media can include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0097] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, main memory 540 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements such as an operating system). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disk (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by system 565. As used herein, computer storage media itself does not include signals.

[0098] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and include any information transmission medium. The term "modulated data signal" can refer to a signal having one or more characteristics set or altered in such a way as to encode information in the signal. By way of example and not limitation, computer storage media can include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0099] When executed, the computer program enables system 565 to perform various functions. One or more CPUs 530 may be configured to execute at least some of computer-readable instructions to control one or more components of system 565 to perform one or more of the methods and / or processes described herein. Each CPU 530 may include one or more cores capable of processing multiple software threads simultaneously (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.). One or more CPUs 530 may include any type of processor and may include different types of processors depending on the type of system 565 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of system 565, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or auxiliary coprocessors (e.g., math coprocessors), system 565 may also include one or more CPUs 530.

[0100] In addition to one or more CPUs 530, or starting with one or more CPUs 530, the parallel processing module 525 may be configured to execute at least some computer-readable instructions to control one or more components of the system 565 to perform one or more methods and / or processes described herein. The system 565 may use the parallel processing module 525 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, the parallel processing module 525 may be used for general-purpose computing on a GPU (GPGPU). In embodiments, one or more CPUs 530 and / or the parallel processing module 525 may execute methods, processes, and / or portions thereof discretely or jointly.

[0101] System 565 also includes one or more input devices 560, a parallel processing system 525, and one or more display devices 545. The one or more display devices 545 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The one or more display devices 545 may receive data from other components (e.g., the parallel processing system 525, one or more CPUs 530, etc.) and output data (e.g., as images, video, sound, etc.).

[0102] Network interface 535 enables system 565 to be logically coupled to other devices, including input device 560, one or more display devices 545, and / or other components, some of which may be built into (e.g., integrated into) system 565. Illustrative input device 560 includes microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. Input device 560 provides a natural user interface (NUI) to handle user-generated air gestures, voice, or other physiological input. In some cases, input may be transmitted to appropriate network elements for further processing. NUI can implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent gesture recognition, air gestures, head and eye tracking, and touch recognition (described in more detail below) associated with the display of system 565. System 565 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, system 565 may include an accelerometer or gyroscope capable of detecting motion (e.g., as part of an inertial measurement unit (IMU)). In some examples, system 565 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0103] Furthermore, for communication purposes, system 565 can be coupled to a network (e.g., a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network, etc.) via network interface 535. System 565 can be included within a distributed network and / or cloud computing environment.

[0104] Network interface 535 may include one or more receivers, transmitters, and / or transceivers that enable system 565 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communications. Network interface 535 may include components and functions to enable communication over any of many different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication via Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0105] System 565 may also include auxiliary storage (not shown). Auxiliary storage 610 includes, for example, hard disk drives and / or removable storage drives, representing floppy disk drives, magnetic tape drives, optical disc drives, digital versatile disc (DVD) drives, recording devices, Universal Serial Bus (USB) flash memory. The removable storage drive reads from and / or writes to the removable storage unit in a well-known manner. System 565 may also include a hardwired power supply, a battery power supply, or a combination thereof (not shown). The power supply can provide power to system 565 to enable the components of system 565 to operate.

[0106] Each of the aforementioned modules and / or devices may even reside on a single semiconductor platform to form system 565. Alternatively, depending on user requirements, the individual modules may also be placed individually or in various combinations of semiconductor platforms. Although various embodiments have been described above, it should be understood that they are merely exemplary and not limiting. Therefore, the breadth and scope of the preferred embodiments should not be limited by any of the exemplary embodiments described above, but should be defined only by the appended claims and their equivalents.

[0107] Machine Learning

[0108] Deep neural networks (DNNs), developed on processors such as the PPU 400, have been used in a variety of use cases, from self-driving cars to faster drug development, from automatic image captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technology that mimics the neural learning process of the human brain, continuously learning, becoming smarter, and providing more accurate results faster over time. Initially, adults teach children to correctly identify and classify various shapes, eventually enabling them to recognize shapes without any guidance. Similarly, deep learning, or neural learning systems, need to be trained on object recognition and classification as they become smarter and more efficient at recognizing basic objects, occluded objects, and so on, while also assigning context to objects.

[0109] At its simplest level, neurons in the human brain examine various inputs they receive, assign a level of importance to each of these inputs, and pass the output to other neurons for further processing. Artificial neurons, or perceptrons, are the most basic model of neural networks. In one example, a perceptron might receive one or more inputs representing various features of an object that the perceptron is being trained to recognize and classify, assigning a weight to each of these features based on their importance in defining the object's shape.

[0110] Deep neural network (DNN) models consist of multiple layers with many connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolutional layers, etc.) and can be trained with large amounts of input data to quickly solve complex and high-precision problems. In one example, the first layer of a DNN model decomposes an input image of a car into its parts and looks for basic patterns such as lines and angles. The second layer assembles the lines to find higher-level patterns, such as wheels, windshields, and rearview mirrors. The next layer identifies the vehicle type, and the final few layers generate labels for the input image to identify models of specific car brands.

[0111] Once a DNN is trained, it can be deployed and used to recognize and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include recognizing handwritten digits on a check deposited into an ATM, recognizing an image of a friend in a photograph, providing movie recommendations to over 50 million users, identifying and classifying different types of cars, pedestrians, and road hazards in self-driving cars, or translating human speech in real time.

[0112] During training, data flows through the DNN in the forward propagation phase until a prediction is produced indicating the label corresponding to the input. If the neural network does not correctly label the input, the error between the correct label and the predicted label is analyzed, and the weights of each feature are adjusted in the backpropagation phase until the DNN correctly labels the input and other inputs in the training dataset. Training complex neural networks requires significant parallel computing power, including floating-point multiplication and addition supported by a PPU 400. Inference is less computationally intensive than training and is a latency-sensitive process where the trained neural network is applied to new inputs that have never been seen before, for tasks such as classifying images, detecting sentiment, recognizing recommendations, recognizing and translating speech, and generally inferring new information.

[0113] Neural networks rely heavily on matrix mathematical operations, and complex multi-layered networks require significant floating-point performance and bandwidth to improve efficiency and speed. The PPU 400, with thousands of processing cores optimized for matrix mathematical operations and delivering tens to hundreds of TFLOPS of performance, is a computing platform capable of providing the performance required for deep neural network-based artificial intelligence and machine learning applications.

[0114] Furthermore, images generated using one or more of the techniques disclosed herein can be used to train, test, or certify DNNs for recognizing objects and environments in the real world. Such images can include scenes of roads, factories, buildings, urban environments, rural environments, humans, animals, and any other physical objects or real-world environments. These images can be used to train, test, or certify DNNs used in machines or robots to manipulate, process, or modify physical objects in the real world. Additionally, such images can be used to train, test, or certify DNNs used in autonomous vehicles to navigate and move vehicles in the real world. Furthermore, images generated using one or more of the techniques disclosed herein can be used to communicate information to users of these machines, robots, and vehicles.

[0115] Graphics processing pipeline

[0116] In one embodiment, PPU 400 includes a graphics processing unit (GPU). PPU 400 is configured to receive commands specifying a shader program for processing graphics data. Graphics data can be defined as a set of primitives, such as points, lines, triangles, quadrilaterals, triangle strips, etc. Typically, a primitive includes data specifying multiple vertices for that primitive (e.g., in model space coordinates) and attributes associated with each vertex of that primitive. PPU 400 can be configured to process the graphics primitives to generate framebuffers (e.g., pixel data for each pixel of a display).

[0117] The application writes model data of the scene (e.g., a set of vertices and attributes) to memory such as system memory or memory 404. The model data defines each object visible on the display. The application then makes API calls to the driver kernel to request rendering and display of the model data. The driver kernel reads the model data and writes commands to one or more streams to perform operations that process the model data. The commands may reference different shader programs to be implemented on processing units within the PPU 400, including one or more vertex shaders, shell shaders, domain shaders, geometry shaders, and pixel shaders. For example, one or more processing units may be configured to execute a vertex shader program that processes multiple vertices defined by the model data. In one embodiment, different processing units may be configured to execute different shader programs simultaneously. For example, a first subset of processing units may be configured to execute a vertex shader program, while a second subset of processing units may be configured to execute a pixel shader program. The first subset of processing units processes the vertex data to produce processed vertex data and writes the processed vertex data to L2 cache 460 and / or memory 404. After rasterizing the processed vertex data (e.g., converting 3D data into 2D data in screen space) to generate fragment data, a second subset of the processing unit executes pixel shaders to generate processed fragment data, which is then mixed with other processed fragment data and written to the frame buffer in memory 404. Vertex shader and pixel shader programs can execute concurrently, pipelinedly processing different data from the same scene until all model data for that scene has been rendered to the frame buffer. The contents of the frame buffer are then transferred to the display controller for display on the display device.

[0118] Figure 6A According to one embodiment, by Figure 4 A conceptual diagram of a graphics processing pipeline 600 implemented by a PPU 400. The graphics processing pipeline 600 is an abstract flowchart of the processing steps implemented to generate 2D computer-generated images from 3D geometric data. It is well known that pipeline architectures can more efficiently perform long-latency operations by dividing operations into multiple stages, where the output of each stage is coupled to the input of the next successive stage. Therefore, the graphics processing pipeline 600 receives input data 601 passed from one stage of the graphics processing pipeline 600 to the next stage to generate output data 602. In one embodiment, the graphics processing pipeline 600 may represent a system... The graphics processing pipeline is defined by the API. Alternatively, the graphics processing pipeline 600 can be implemented within the context of the functionality and architecture of the previous and / or any subsequent figures.

[0119] like Figure 6AAs shown, the graphics processing pipeline 600 includes a pipeline architecture comprising multiple stages. These stages include, but are not limited to, a data assembly stage 610, a vertex shading stage 620, a primitive assembly stage 630, a geometry shading stage 640, a viewport scaling, culling, and trimming (VSCC) stage 650, a rasterization stage 660, a fragment shading stage 670, and a raster operation stage 680. In one embodiment, input data 601 includes commands configuring the processing unit to implement the graphics processing pipeline 600 and the geometric primitives (e.g., points, lines, triangles, quadrilaterals, triangle strips, or sectors, etc.) to be processed by the stage. Output data 602 may include pixel data (e.g., color data) copied into a frame buffer or other type of surface data structure in memory.

[0120] The data assembly stage 610 receives input data 601, which specifies vertex data for higher-order surfaces, primitives, etc. The data assembly stage 610 collects vertex data in temporary storage or a queue, for example, by receiving commands from the host processor including pointers to buffers in memory and reading vertex data from the buffers. The vertex data is then transferred to the vertex shading stage 620 for processing.

[0121] The vertex shading stage 620 processes vertex data by performing a set of operations (e.g., a vertex shader or procedure) on each vertex at a time. A vertex can be, for example, specified as a 4-coordinate vector (e.g., color, texture coordinates, surface normals, etc.) associated with one or more vertex attributes (e.g., color, texture coordinates, surface normals, etc.).<x,y,z,w> The vertex shading stage 620 can manipulate individual vertex attributes, such as position, color, texture coordinates, etc. In other words, the vertex shading stage 620 performs operations on vertex coordinates or other vertex attributes associated with a vertex. Such operations typically include lighting operations (e.g., modifying the vertex's color attribute) and transformation operations (e.g., modifying the vertex's coordinate space). For example, a vertex can be specified using coordinates in object coordinate space, which is transformed by multiplying the coordinates by a matrix that converts the coordinates from object coordinate space to world space or normalized device coordinate (NCD) space. The vertex shading stage 620 generates transformed vertex data that is then transferred to the primitive assembly stage 630.

[0122] Primitive assembly stage 630 collects the vertices output by vertex shading stage 620 and groups these vertices into geometric primitives for processing by geometry shading stage 640. For example, primitive assembly stage 630 can be configured to group every three consecutive vertices as geometric primitives (e.g., triangles) to be transferred to geometry shading stage 640. In some embodiments, specific vertices can be reused for consecutive geometric primitives (e.g., two consecutive triangles in a triangle strip can share two vertices). Primitive assembly stage 630 transfers geometric primitives (e.g., sets of associated vertices) to geometry shading stage 640.

[0123] The geometry shading stage 640 processes geometry primitives by performing a set of operations (e.g., geometry shaders or procedures) on them. Tessellation operations can generate one or more geometry primitives from each primitive. In other words, the geometry shading stage 640 can subdivide each geometry primitive into a finer mesh of two or more geometry primitives for the rest of the graphics processing pipeline 600 to process. The geometry shading stage 640 then passes the geometry primitives to the viewport SCC stage 650.

[0124] In one embodiment, the graphics processing pipeline 600 can operate in a streaming multiprocessor, and the vertex shading stage 620, primitive assembly stage 630, geometry shading stage 640, fragment shading stage 670, and / or associated hardware / software can sequentially perform processing operations. Once the sequential processing operations are complete, in one embodiment, the viewport SCC stage 650 can utilize the data. In one embodiment, the graph data processed by one or more stages of the graphics processing pipeline 600 can be written to a cache (e.g., L1 cache, vertex cache, etc.). In this case, in one embodiment, the viewport SCC stage 650 can access the data in the cache. In one embodiment, the viewport SCC stage 650 and the rasterization stage 660 are implemented as fixed-function circuitry.

[0125] In the Viewport SCC stage 650, viewport scaling, culling, and trimming are performed on the geometry. Each surface rendered is associated with an abstract camera position. The camera position represents the viewer's position in viewing the scene and defines the viewing frustum that surrounds the scene objects. The viewing frustum may include the view plane, the back plane, and four trimming planes. Any geometry that is completely outside the viewing frustum can be culled (e.g., discarded) because it does not contribute to the final rendered scene. Any geometry that is partially inside and partially outside the viewing frustum can be trimmed (e.g., converted into new geometry that surrounds the viewing frustum). Furthermore, each geometry can be scaled based on the depth of the viewing frustum. All potentially visible geometry is then transferred to the rasterization stage 660.

[0126] Rasterization stage 660 converts 3D geometric primitives into 2D fragments (e.g., fragments usable for display, etc.). Rasterization stage 660 can be configured to use the vertices of the geometric primitives to set a set of plane equations from which various properties can be interpolated. Rasterization stage 660 can also compute an overlay mask for multiple pixels, indicating whether one or more sample locations of that pixel occlude a geometric primitive. In one embodiment, a z-test can also be performed to determine whether the geometric primitive is occluded by other geometric primitives that have already been rasterized. Rasterization stage 660 generates fragment data (e.g., interpolated vertex properties associated with specific sample locations of each covered pixel), which is then passed to fragment shading stage 670.

[0127] Fragment shading stage 670 processes fragment data by performing a set of operations (e.g., fragment shader or procedure) on each fragment. Fragment shading stage 670 may generate pixel data (e.g., color values) for the fragment, for example, by performing lighting operations or sampling a texture map using interpolated texture coordinates for the fragment. The pixel data generated by fragment shading stage 670 is then transferred to raster operation stage 680.

[0128] The rasterization stage 680 can perform various operations on the pixel data, such as performing alpha testing, stencil testing, and mixing the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the rasterization stage 680 has completed processing the pixel data (e.g., output data 602), the pixel data can be written to a rendering target such as a frame buffer, color buffer, etc.

[0129] It will be understood that, in addition to or replacing one or more of the stages described above, the graphics processing pipeline 600 may include one or more additional stages. Various implementations of the abstract graphics processing pipeline may implement different stages. Furthermore, in some embodiments, one or more of the stages described above may be excluded from the graphics processing pipeline (e.g., geometry shading stage 640). Other types of graphics processing pipelines are conceivable within the scope of this disclosure. Moreover, any stage of the graphics processing pipeline 600 may be implemented by one or more dedicated hardware units within a graphics processor such as the PPU 400. Other stages of the graphics processing pipeline 600 may be implemented by programmable hardware units such as processing units in the PPU 400.

[0130] The graphics processing pipeline 600 can be implemented via an application executed by a main processor such as a CPU. In one embodiment, the device driver can implement an application programming interface (API) that defines various functions that can be utilized by the application to generate graphics data for display. The device driver is a software program that includes multiple instructions that control the operation of the PPU 400. The API provides an abstraction for programmers, allowing them to utilize dedicated graphics hardware (e.g., the PPU 400) to generate graphics data without requiring programmers to utilize the specific instruction set of the PPU 400. The application can include API calls routed to the device driver of the PPU 400. The device driver interprets the API calls and performs various operations in response to the API calls. In some cases, the device driver can perform operations by executing instructions on the CPU. In other instances, the device driver can perform operations at least in part by initiating operations on the PPU 400 using the input / output interface between the CPU and the PPU 400. In one embodiment, the device driver is configured to utilize the hardware of the PPU 400 to implement the graphics processing pipeline 600.

[0131] Various programs can be executed within the PPU 400 to implement different stages of the graphics processing pipeline 600. For example, a device driver can launch a kernel on the PPU 400 to execute the vertex shading stage 620 on one or more processing units. The device driver (or the initial kernel executed by the PPU 400) can also launch other kernels on the PPU 400 to execute other stages of the graphics processing pipeline 600, such as the geometry shading stage 640 and the fragment shading stage 670. Furthermore, some stages of the graphics processing pipeline 600 can be implemented on fixed-function hardware, such as a rasterizer or data assembler implemented within the PPU 400. It should be understood that the results from one kernel can be processed by one or more intermediate fixed-function hardware units before subsequent kernel processing on the processed unit.

[0132] Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other display device. In some embodiments, the display device may be directly coupled to the system or processor that generates or renders the image. In other embodiments, the display device may be indirectly coupled to the system or processor, for example, via a network. Examples of such networks include the Internet, mobile telecommunications networks, Wi-Fi networks, and any other wired and / or wireless networking systems. When the display device is indirectly coupled, the image generated by the system or processor can be streamed to the display device over the network. This streaming allows, for example, video games or other applications that render images to be executed on a server or in a data center, and the rendered images will be transmitted and displayed on one or more user devices (e.g., computers, video game consoles, smartphones, other mobile devices, etc.) that are physically separate from the server or data center. Therefore, the techniques disclosed herein can be applied to enhance streamed images and services that enhance streamed images, such as NVIDIA GeForce Now (GFN), Google Stadia, etc.

[0133] Example game streaming system

[0134] Figure 6B This is an example system diagram of a game streaming system 605 according to some embodiments of the present disclosure. The game streaming system 605 can be used with... Figure 1A The server / client system 100 can be used in conjunction with or in place of it. Figure 6B Includes one or more game servers 603 (which may include...) Figure 5A Example processing system 500 and / or Figure 5B (Similar components, features, and / or functions to the exemplary system 565), one or more client devices 604 (which may include components, features, and / or functions similar to those of the exemplary system 565), and one or more client devices 604. Figure 5A Example processing system 500 and / or Figure 5B The exemplary system 565 may include similar components, features, and / or functions, as well as one or more networks 606 (which may be similar to one or more networks described herein). In some embodiments of this disclosure, system 605 may be implemented.

[0135] In system 605, for a game session, one or more client devices 604 can receive input data in response to input to one or more input devices 626, transfer the input data to one or more game servers 603, receive encoded display data from one or more game servers 603, and display the display data on the monitor 624. In this way, computationally intensive computation and processing are offloaded to one or more game servers 603 (e.g., rendering of the game session's graphics output—especially ray or path tracing—is performed by one or more GPUs 615 of one or more game servers 603). In other words, the game session is streamed from one or more game servers 603 to one or more client devices 604, thereby reducing the demand on graphics processing and rendering on one or more client devices 604.

[0136] For example, regarding an instance of a game session, client device 604 can display frames of the game session on display 624 based on display data received from one or more game servers 603. Client device 604 can receive input to one or more input devices 626 and generate input data in response. Client device 604 can transmit input data to one or more game servers 603 via communication interface 621 and through one or more networks 606 (e.g., the Internet), and one or more game servers 603 can receive input data via communication interface 618. One or more CPUs 608 can receive input data, process input data, and transmit data to one or more GPUs 615, thereby causing one or more GPUs 615 to generate renderings of the game session. For example, the input data can represent the movement of a user's character in the game, shooting weapons, reloading, passing, turning vehicles, etc. Rendering component 612 can render the game session (e.g., representing the result of the input data), and rendering capture component 614 can capture the rendering of the game session as display data (e.g., as image data capturing the rendered frames of the game session). The rendering of a game session may include lighting and / or shadow effects computed using ray or path tracing with one or more parallel processing units (e.g., GPU 615), which may further use one or more dedicated hardware accelerators or processing cores to perform ray or path tracing techniques on one or more game servers 603.

[0137] In one embodiment, rendering component 612 may be configured to receive LOD bias parameters controlled based on the conditions of one or more networks 606. One or more game servers 603 may include image smoothing controls 125 and / or network monitors 135 to adjust the LOD bias parameters and perform dynamic image smoothing to control the bitrate. In one embodiment, image smoothing filter 115 and image smoothing controls 125 are included between rendering capture component 614 and encoder 616.

[0138] The encoder 616 can then encode the display data to generate encoded display data, and can transmit the encoded display data to the client device 604 via communication interface 618 on one or more networks 606. The client device 604 can receive the encoded display data via communication interface 621, and the decoder 622 can decode the encoded display data to generate display data. The client device 604 can then display the display data via display 624.

[0139] Example network environment

[0140] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may... Figure 5A Processing system 500 and / or Figure 5B Implemented on one or more instances of the exemplary system 565 (e.g., each device may include similar components, features and / or functions of the processing system 500 and / or the exemplary system 565).

[0141] Components of a network environment can communicate with each other via one or more networks, which may be wired, wireless, or both. The network may include multiple networks or networks of networks. For example, the network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (e.g., the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. In cases where the network includes wireless telecommunications networks, components such as base stations, communication towers, and even access points (as well as other components) can provide wireless connectivity.

[0142] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for one or more servers can be implemented on any number of client devices.

[0143] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, and combinations thereof. The cloud-based network environment may include a framework layer, job scheduler, resource manager, and distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting software and / or application layers of one or more applications. The software or one or more applications may respectively include web-based service software or applications. In embodiments, one or more client devices may use the web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software web application framework, for example, one that can use a distributed file system for large-scale data processing (e.g., "big data").

[0144] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions can be distributed across multiple locations from a central or core server (e.g., one or more data centers may be distributed across states, regions, countries, globally, etc.). If the connection to a user (e.g., a client device) is relatively close to one or more edge servers, then one or more core servers can specify at least a portion of the functionality to one or more edge servers. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0145] One or more client devices may include Figure 5A Example processing system 500 and / or Figure 5B The exemplary system 565 includes at least some components, features, and functions. As an example and not a limitation, the client device may be embodied as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, device, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0146] Note that the techniques described herein can be embodied in executable instructions stored in a computer-readable medium for use by or in conjunction with a processor-based instruction execution machine, system, apparatus, or device. Those skilled in the art will understand that, for some embodiments, various types of computer-readable media may be included to store data. As used herein, “computer-readable medium” includes one or more of any suitable media for storing executable instructions of a computer program, such that an instruction execution machine, system, apparatus, or device can read (or retrieve) the instructions from the computer-readable medium and execute the instructions for performing the described embodiments. Suitable storage formats include one or more of electronic, magnetic, optical, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer floppy disks; random access memory (RAM); read-only memory (ROM); erasable programmable read-only memory (EPROM); flash memory devices; and optical storage devices, including portable optical discs (CDs), portable digital video discs (DVDs), etc.

[0147] It should be understood that the arrangement of components shown in the accompanying drawings is for illustrative purposes, and other arrangements are possible. For example, one or more elements described herein can be implemented wholly or partially as electronic hardware components. Other elements can be implemented in software, hardware, or a combination of software and hardware. Furthermore, some or all of these other elements can be combined, some other elements can be omitted entirely, and additional components can be added while still achieving the functionality described herein. Therefore, the subject matter described herein can be embodied in many different variations, and all such variations are considered to be within the scope of the claims.

[0148] To facilitate understanding of the subject matter described herein, numerous aspects are described in accordance with sequences of actions. Those skilled in the art will recognize that various actions can be performed by dedicated circuitry or circuits, by program instructions executed by one or more processors, or a combination of both. The description of any sequence of actions herein is not intended to imply that a specific order for performing that sequence must be followed. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order.

[0149] In the context of describing the subject matter (particularly in the context of the appended claims), the use of the terms “an,” “a,” and “the,” and similar references, should be interpreted to cover both singular and plural forms. The plural form is used unless otherwise stated herein or clearly contradicts the context. The term “at least one” followed by a list of one or more items (e.g., “at least one of A and B”) should be understood to refer to one item selected from the listed items (A or B) or any combination of two or more of the listed items (A and B), unless otherwise stated herein or clearly contradicts the context. Furthermore, the foregoing description is for illustrative purposes only and not for limiting purposes, as the scope of protection sought is defined by the claims and their equivalents described below. Unless otherwise required, the use of any and all example or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate the subject matter and does not constitute a limitation on the scope of the subject matter. The use of the term “based on” and other similar phrases in the claims and written description to indicate the conditions that produce the result is not intended to exclude any other conditions that produce that result. Nothing in the specification should be construed as indicating that any unclaimed element is essential for carrying out the claimed invention.

Claims

1. A computer-implemented method, comprising: Receive network status data used for connections from the server to the client; In response to changes in the network condition data, the first and second parameters are automatically adjusted; Render one or more images of the scene according to the first parameter, wherein the first parameter corresponds to the level of detail of the texture mapping applied to one or more surfaces of the scene; as well as One or more image frames are generated according to the second parameter, which controls a smoothing filter applied to one or more regions of the rendered one or more images, the one or more regions corresponding to regions with high spatial frequency content, and the pixel resolution of the one or more image frames is consistent with the pixel resolution of the rendered one or more images.

2. The computer-implemented method according to claim 1, wherein generating the image frame comprises: The smoothing filter is applied to each frame of a series of rendered images.

3. The computer-implemented method according to claim 2, wherein the smoothing filter comprises at least one of the following: Scrolling guide filter; bilateral filter; or Domain conversion filter.

4. The computer-implemented method according to claim 1 further includes: The image frame is encoded for transmission to the client.

5. The computer-implemented method according to claim 4, further comprising: The encoded image frame is decoded at the client to produce a decoded image frame.

6. The computer-implemented method of claim 5, wherein, prior to the change in the network condition data, one of the decoded image frames is used as an intra-coded I-frame, wherein at least one of the coded image frames is decoded based on the I-frame.

7. The computer-implemented method of claim 5, wherein the frame rate of the decoded image frame is substantially constant before and after the change in the network condition data.

8. The computer-implemented method according to claim 1, wherein generating the image frame comprises: The visible geometry is preserved within one or more scenes represented by the image frames.

9. The computer-implemented method of claim 1, wherein the network status data is monitored by the client.

10. The computer-implemented method of claim 1, wherein the network status data includes data relating to at least one of the following: The available bandwidth corresponding to the channel; Corresponding to the delay of the channel; Corresponding to the jitter of the channel; or Data packet loss corresponding to the channel.

11. The computer-implemented method according to claim 1, further comprising: Measure the bit rate corresponding to the image frame; as well as The first parameter and the second parameter are adjusted based on a comparison between the bit rate and the network condition data.

12. The computer-implemented method of claim 11, wherein the second parameter is adjusted to increase the level of smoothing in order to reduce the bit rate.

13. The computer-implemented method of claim 11, wherein the second parameter is adjusted to reduce the level of smoothing in order to increase the bit rate.

14. The computer-implemented method of claim 1, wherein the steps of receiving, adjusting, rendering, and generating are performed within a cloud computing environment.

15. The computer-implemented method of claim 1, wherein the steps of receiving, adjusting, rendering, and generating are performed on the server or in a data center to generate the image frame, and the client includes a user device.

16. The computer-implemented method of claim 1, wherein the image frame is used for at least one of the following: Game applications; or Applications used to train, test, or certify neural networks used in machines, robots, or autonomous vehicles.

17. The computer-implemented method of claim 1, wherein generating the image frame preserves the text and structure within the image frame.

18. A system comprising: The server, which includes a processor and memory, is configured as follows: Receive network status data for the connection from the server to the client; In response to changes in the network condition data, the first and second parameters are automatically adjusted; Render one or more images of the scene according to the first parameter, wherein the first parameter corresponds to the level of detail of the texture mapping applied to one or more surfaces of the scene; as well as One or more image frames are generated according to the second parameter, which controls a smoothing filter applied to one or more regions of the rendered one or more images, the one or more regions corresponding to regions with high spatial frequency content, and the pixel resolution of the one or more image frames is consistent with the pixel resolution of the rendered one or more images.

19. A non-transitory computer-readable medium storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: Receive network status data used for connections from the server to the client; In response to changes in the network condition data, the first and second parameters are automatically adjusted; Render one or more images of the scene according to the first parameter, wherein the first parameter corresponds to the level of detail of the texture mapping applied to one or more surfaces of the scene; as well as One or more image frames are generated according to the second parameter, which controls a smoothing filter applied to one or more regions of the rendered one or more images, the one or more regions corresponding to regions with high spatial frequency content, and the pixel resolution of the one or more image frames is consistent with the pixel resolution of the rendered one or more images.

Images