Method and system for dynamic user equipment upgrade of media streams

Abstract

The methods disclosed herein provide for receiving, at a user device, a media stream comprising frames generated by a graphics rendering application at a first resolution; and locally upgrading, at the user device, each received frame of the media stream to a second resolution greater than the first resolution using one or more weight matrices pre-trained in association with the graphics rendering application. The local upgrading of the media stream can be performed "on-the-fly" such as with respect to a single content stream (e.g., a game) or a content stream segment (e.g., a scene in a match).

Description

BACKGROUND

[0001] Increasingly, graphical media applications, such as games, are becoming cloud-based; however, streaming high resolution video is extremely bandwidth intensive. Some home internet plans are not sufficient to support streaming large amounts of high resolution video at nominal frame rates. Additionally, high resolution video streaming is memory and power intensive for cloud-based content providers. A lower memory, lower bandwidth, and lower cost solution is needed. SUMMARY

[0002] According to one implementation, a method for increasing resolution of streamed graphics without increasing bandwidth consumption includes receiving, at a user device, a media stream comprising frames of a first resolution generated by a graphics rendering application; and locally upgrading each frame of the media stream received at the user device to a second resolution greater than the first resolution using one or more weight matrices pre-trained in association with the graphics rendering application.

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Other implementations are described and recited herein. BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1 An example system is illustrated that utilizes artificial intelligence executing on a user device to dynamically upgrade resolution of a received video stream of media content.

[0005] Figure 2 A system is illustrated that performs example training operations for generating weight matrices that can be used to dynamically upgrade resolution of graphics frame data streamed to a user device.

[0006] Figure 3 A system is illustrated that performs operations for dynamically upgrading resolution and fidelity of graphics frame data received at a user device.

[0007] Figure 4 Another example system is illustrated that performs operations for dynamically upgrading resolution and fidelity of graphics frame data received at a user device.

[0008] Figure 5 Example operations are illustrated for upgrading frames of a media stream received at a user device.

[0009] Figure 6 An example schematic diagram of a processing device suitable for implementing aspects of the disclosed technology is illustrated. DETAILED DESCRIPTION

[0010] The technology disclosed herein provides a bandwidth-efficient solution for rendering, streaming, and upgrading media content to enable high-resolution (e.g., 4k) content presentation on user device displays. According to one implementation, a media stream is broadcast to a user device at a first resolution that is much lower than the final resolution at which the graphics are ultimately rendered on the user device. The user device locally performs actions to improve both resolution and image fidelity (e.g., detail).

[0011] While some prior art aims to locally enhance the resolution of streamed content on a user device, these conventional solutions tend to rely on uniform scaling methods such as bicubic interpolation, bilinear interpolation, or a “nearest neighbor” method that predicts pixel values from the values of nearest neighbor pixels. All of these methods generally provide a uniform scaling of the brightness / color of newly added frame pixels according to a static, predefined value set relative to the values of nearby pixels in the original image. In these solutions, the final image resolution is improved, but the results are not ideal. As the resolution is increased, the image becomes pixelated, resulting in a “blocky” appearance that looks less resolved than the actual resolution.

[0012] In contrast to these conventional solutions, the presently disclosed technology provides an image upgrade tool that dynamically improves the resolution of incoming frames (e.g., at 60 or 120 frames per second) by using artificial intelligence that is trained to improve image fidelity by “filling in the gaps” so that newly added pixels provide details that are not present in the original frame. According to one implementation, the proposed solution allows frames to be rendered (e.g., by a game engine) at a low resolution such as 720p or 1080p. The low resolution frames are transmitted across a network to a user device and locally upgraded to a higher quality (such as 4k) using an Al-trained image upgrade tool.

[0013] Notably, server-level streaming of 4k resolution requires the transfer of a data stream that is four times the size of a 1080p resolution stream. As such, high resolution media streaming places a cost burden on both the data center (e.g., high power and bandwidth costs) and the end user (e.g., ensuring that home internet streaming setups support both high bitrates and high bandwidth streaming (e.g., 20 to 40 Mbps)). In contrast, media streaming at 1080p resolution consumes only a fraction of the bandwidth consumed by 4k (e.g., 5 Mbps). This bandwidth savings is significant in terms of both bandwidth and power consumption savings for the data center. Moreover, the end user benefits from the bandwidth reduction being implemented, while also experiencing an increase in video quality and fidelity. For the user, the savings in bandwidth translates to an improvement in latency (as more bandwidth can be directed to controller responses) and monetary cost savings due to a reduction in the need for a high data cap on more expensive internet plans. The resolution enhancement solution described herein allows media streams to be transferred at a lower resolution and viewed at a relatively higher resolution without a corresponding loss in frame rate or content and with significant bandwidth savings. The foregoing can also be achieved while consuming less bandwidth and less power and reduced memory during actual frame rendering (e.g., by an application or game server) as such rendering occurs at a lower resolution. In various implementations, depending on how the technology is leveraged by users, game developers, and cloud application streaming providers, the technology can be applied to improve quality (image fidelity) and / or frame rate.

[0014] Figure 1 An example system 100 is illustrated that utilizes artificial intelligence (Al) executing on a user device to dynamically upgrade the resolution of a received video stream of media content. The system 100 includes a processing device 102, which can take various forms in different implementations, including but not limited to a laptop or tablet, desktop computer, game console, smartphone, etc. The processing device includes hardware, as shown by integrated circuit (IC) 104, which includes a matrix multiple engine 106, sometimes also referred to as a matrix math accelerator. For example, the IC 104 can include one or more standard system on a chip (SoC), application specific integrated circuit (ASICS), or FPGA. According to one implementation, the matrix multiple engine 106 is implemented as a systolic multiplier array that is easily programmed and effective for application to any neural network.

[0015] In addition to IC 104 and matrix multiplication engine 106, processing device 102 further includes memory 108 and processor 110. Memory 108 and processor 110 are shown as being external to IC 104; however, it should be understood that IC 104 can include additional memory and / or one or more processors in addition to processor 110. Super-resolution neural network (SRNN) inference engine 112 is stored within memory 108 and is executable by processor 110. In other implementations, some or all aspects of SRNN inference engine 112 are stored within memory of IC 104 and / or executed by processor(s) of IC 104.

[0016] SRNN inference engine 112 includes software executable by processor 110 to dynamically upgrade (or upsample) graphical frames of an incoming low-resolution media content stream 114 as they arrive at processing device 102 from media server 116. Although “low-resolution” is a relative term when used independently, throughout the specification, the term is used with respect to media content streams received at a user device and having lower resolution and / or less detail than a final media content stream (e.g., high-resolution media stream 122) rendered on display 118 of processing device 102. As used herein, “upgrade” refers to an increase in image resolution and fidelity (detail).

[0017] In different implementations, media server 116 can take various forms, including the form of one or more different web-based servers and / or edge devices that host and / or generate media content. For example, media server 116 can be a cloud-based gaming service (xCloud) or application server that stores and executes a graphics generation application to generate low-resolution media stream 114, which is in turn streamed to a user device (e.g., processing device 102).

[0018] In one implementation, SRNN inference engine 112 is pre-loaded with trained weights stored in weight matrix 120, which is used by SRNN inference engine 112 to convert each input frame of low-resolution media stream 114 to a corresponding output frame of high-resolution media stream 122. In various implementations, the trained weights can be communicated prior to or during streaming of low-resolution media stream 122 to processing device 102. For a single graphics rendering application, there can be multiple weight matrices that are trained to upgrade content of different segments of the relevant media stream. In one example where low-resolution media stream 122 is game content, a new game scene can use different video content and style than a previous game scene. In this case, the new game scene can load matrix multiplication engine 106 with a different weight matrix than the weight matrix used for the previous scene to upgrade.

[0019] In various implementations, the different weight matrices associated with the same media content stream can be delivered to the processing device 102 in different ways. In one implementation, the weight matrices are sent to the processing device 102 as a batched package and cached locally for a period of time. The locally stored weight matrices can be selectively loaded into the matrix multiplication engine 106 when their associated frame segments are streamed and / or processed for local rendering. In another implementation, the different weight matrices for the low resolution media stream 122 are independently sent to the processing device 102 prior to or during a scene transition associated with a new matrix. In this scenario, the processing device 102 does not have to cache a large amount of information. In yet another implementation, the different weight matrices are streamed concurrently with the low resolution media stream 122 at the upcoming point in time at which each matrix is expected to be loaded and used.

[0020] In one implementation in which the weight matrices are batched and / or locally cached on the processing device 122, the graphics rendering application at the media server 116 can inform the processing device through an established API to use the correct weight matrix for each individual scene being displayed. For example, this can be accomplished by tagging each weight matrix with metadata that indicates which frame segment(s) will be upscaled, where each weight matrix is stored locally in local memory.

[0021] When the low resolution media stream 122 is stopped (e.g., the game ends or the user terminates the stream), the locally stored data can be versioned, allowing the developer to change the quality of the trained weights at game updates.

[0022] In one implementation, the SRNN inference engine 202 is a convolutional neural network (CNN) that includes several convolutional layers that collectively provide for converting each feature in a raw, low resolution input frame to a corresponding higher resolution version of that feature. The matrix multiplication for upsampling each input image to a corresponding super resolution output image can be performed by the matrix multiplication engine 106. In other implementations, these mathematical computations are performed partially or entirely by software elements rather than hardware elements.

[0023] In one implementation, the weight matrix 120 is derived during a training process that utilizes stochastic descent or similar methods to identify a set of weights that achieve a desired effect in the conversion between input (low resolution) and output (high resolution) images. In contrast to graphics upgrade solutions that provide a uniform selection of pixel values based on static interpolation, the weight matrix 120 is trained to provide scaling weights for specific segments or groups of frames of a predefined granularity, such as groups of frames associated with the same media title (e.g., a game, application, or video). In one implementation, the weight matrix 120 is trained on a per-title basis, such that a different weight matrix is loaded into the memory of the SRNN inference engine 112 each time a user initiates a different cloud-based application hosted by the media server 116. For example, each time a user initiates a game hosted by the media server 116, the SRNN inference engine 112 identifies, retrieves, and loads a matrix of weights that were specifically trained for that game. In another implementation, the weight matrix 120 is trained on a per-scene basis, such that each individual scene within a video or game is upgraded using a different weight matrix that was specifically trained for that scene.

[0024] In Figure 1 the weight matrix 120 is shown as being retrieved from a cloud-based SRNN data store 124. In the example shown, the SRNN data store 124 stores weight matrices trained for different media titles (graphical rendering applications), with each weight matrix stored in association with a media title 126 and a weight matrix ID 128. For example, the media title “Minecraft” is shown as being stored in association with multiple different weight matrix IDs, with each title / ID pair (e.g., [media title, matrix ID]) uniquely identifying a weight matrix that was trained to upgrade a discrete, predefined segment of frames associated with that media title. As used herein, a “segment” of frames refers to a collection of consecutive or non-consecutive frames associated with the same media title. For example, each weight matrix is used to upgrade a different segment of frames, including, for example, a segment of frames that depict a particular animation, a segment of frames that depict a particular scene, or a segment of frames that are pre-identified as having common characteristics, such as text or menu options.

[0025] As an example and not a limitation, the weight matrix identified by the media title "Minecraft" and the weight matrix ID "Level 1" can be trained using a subset of frames rendered during "Level 1" in Minecraft and used by the SRNN inference engine 112 to perform upgrade actions for all frames rendered during Level 1. Similarly, the matrix identified by the media title "Minecraft" and the weight matrix ID "menu matrix" can be trained using frames generated by Minecraft that include menu icons or text and used by the SRNN inference engine 112 to upgrade all frames depicting menu content. Furthermore, an exemplary matrix identified by the media title "Minecraft" and the weight matrix ID "Animation matrix" can be trained using frames rendering some aspects of the same animation. This matrix can be used to upgrade all frames depicting various aspects of that animation.

[0026] Because each weight in the weight matrix is ​​trained on a finite number of predefined frames generated during the execution of the same media executable (e.g., the same game), the resulting fidelity enhancement for each output frame fills in details that may be unique to the frames used to train the corresponding weight matrix (e.g., color gradients, shadow effects), even though these details may vary significantly between titles or even between different segments of the same title. As a result, the rendering of frame content is more realistic and visually appealing compared to what can be achieved using weight matrices trained on frames from many different media titles or traditional upgrade methods such as pixel interpolation.

[0027] Figure 2 A system 200 is illustrated that performs an exemplary training operation to generate a weight matrix 208 for dynamically upgrading the resolution of graphics frame data streamed to a user device. System 200 includes an SRNN inference engine 202, which may include components related to those described above. Figure 1 The characteristics discussed are the same or similar. During the operation shown, the weight matrix 208 is trained to provide image upgrades associated with the graphics of a single media title. As used herein, "media title" refers to a specific graphics rendering application (e.g., a game) that includes an executable program for generating video content streams of media content. Figure 2 In this process, developer 204 initiates an operation to train the weights of weight matrix 208, which is associated with graphics rendering application 210.

[0028] In different implementations, the developer 204 can choose to generate a single weight matrix 208 for the graphics rendering application, or multiple different weight matrices for the graphics rendering application 210. For example, a single weight matrix can be generated and used to provide image upgrading for all frames generated by the graphics rendering application 210; alternatively, different weight matrices can be used to provide image upgrading for different frame segments (e.g., different scenes, different animations, and / or groupings based on other visual characteristics shared between frames).

[0029] In Figure 2 In the example, the developer 204 trains the weight matrix 208 with a subset of frames selected from a single, individual scene (“Scene 1”), which can be an animation, a game level, or a particular scene within a game level. In this example step, the developer 204 selects training frames 214 from Scene 1 that are representative of the types of graphics present in Scene 1. Although not shown, the developer 204 can also select one or more additional frames that will be used to train the weight matrix for Scene 1. Each selected representative frame is provided as a training input to the SRNN inference engine 202.

[0030] Upon receiving the training frames 214, the SRNN inference engine 202 generates a suggested upgraded version of the training frames 214 using the predefined (e.g., initial default) weights in the weight matrix 208. In one implementation, the SRNN inference engine 202 includes several convolutional layers in a traditional neural network design. Each different convolutional layer can create a subsequent feature map that is fed to the next layer. For example, each convolutional layer can be used to map a single image feature or set of features to a higher resolution feature or set of features. In Figure 2 The suggested upgraded version of the training frames 214 is represented as “upgraded frames 218” in Figure 2 The generation of the upgraded frames 218 entails (1) increasing the image resolution of the training frames 214 by a predetermined factor (e.g., from 720p to 4k), and (2) defining a suggested value for each pixel newly added to the frame based on the current weights included in the weight matrix 208. These weights can be adjusted throughout the training process using traditional techniques such as training, backpropagation, and gradient descent mechanisms. This training can be achieved via supervised or unsupervised training techniques.

[0031] In one implementation utilizing unsupervised learning, the SRNN training controller 212 compares each upscaled image (e.g., upscaled frame 218) to a traditionally and fully rendered true high resolution image (e.g., a 4k or higher resolution image). For example, the upscaled frame 218 can be compared to a high resolution image or set of images selected by the SRNN training controller 212 or by the developer 204. Adjustments can be implemented automatically by the SRNN training controller 212 to reduce the differences between the upscaled frame 218 and the true high resolution image. In this implementation, the training process can remain reiterative on the training set until the upscaled frame 218 outputs best approximate the true high resolution image.

[0032] Still other implementations implement a hybrid approach in which the adjuster 216 uses both supervised and unsupervised inputs to adjust the weights.

[0033] By way of example and not limitation, the system 200 is shown utilizing a supervised learning approach rather than an unsupervised approach. Here, the SRNN training controller 212 utilizes input from the developer 204 to best approximate the quality intended for each frame segment. For example, the developer 204 previews the upscaled frame 218 and provides a Yes / No input to the SRNN training controller 212, where the Yes / No input indicates whether the upscaled frame 218 meets certain aesthetically acceptable standards. For example, the developer 204 can notice that the upscaled frame 218 does not realistically enhance certain areas of the frame (e.g., certain objects appear overly pixelated or shadows appear to spill into areas that they should not).

[0034] In the illustrated supervised learning approach, the determination of whether the upscaled frame 218 meets certain aesthetically acceptable standards is subjective to the individual developer’s preference. In effect, this utilizes Al to identify weights of the weight matrix 208 that are effective to implement an artistically subjective resolution enhancement that improves image fidelity (adds image detail) rather than merely interpolating existing pixel values to determine new pixel values.

[0035] If the developer 204 is not satisfied with the upscaled frame 218, the developer 204 provides a "no" as the supervised input 220, and the weight adjuster 216 of the SRNN training controller 212 computes adjusted weight values for the weight matrix 208. Using the adjusted weight values, the SRNN inference engine 202 regenerates the upscaled frame 218, and the developer 204 can again provide the supervised input 220. The above-described weight adjustment and regeneration of the upscaled frame 218 can be performed multiple times in a loop until the developer 204 provides a "yes" as the supervised input 220 to the SRNN training controller 212, indicating that the upscaled frame 218 meets the aesthetically acceptable criteria.

[0036] In addition to the supervised input 220, the weight adjuster 216 can utilize one or more additional CNN traditional training techniques to determine the weight adjustment at each iteration.

[0037] If the developer has selected any other training frames for the same weight matrix (e.g., other training frames representing Scene 1 in addition to the training frame 214), the above-described operations can be repeated with the current values of the weight matrix 208 used as the starting values for the correction operation for the next received training input frame. In some implementations, the developer 204 can choose to train the weight matrix 208 on a single frame. In other implementations, the weight matrix 208 can be trained on multiple frames.

[0038] For each training frame (e.g., the training frame 214) selected for a given frame segment (e.g., Scene 1), one of the above-described training operations is performed, and the SRNN inference engine 202 outputs the weight matrix 208 to the SRNN data store 224, which can have the same or similar features as the SRNN data store 124 described with respect to Figure 1

[0039] In this manner, the weight matrix 208 is trained on a subset of frames selected from a given frame segment ("Scene 1"), and a final version of the weight matrix 208 is used during live media stream operations to dynamically adjust the resolution and fidelity of each frame of the given frame segment as it is received at the user device. Different weight matrices can be similarly generated and used for various different frame segments of the media content stream generated by the graphics rendering application 210.

[0040] Figure 3 A system 300 is illustrated that performs exemplary operations for dynamically upsampling the resolution and fidelity of graphics frame data received at a user device. In one implementation, the user device 302 is a mobile device having reduced memory and / or processing capabilities as compared to most PCs and game consoles. For example, the user device 302 can be a tablet or a cellular phone. The user device 302 includes a graphics rendering application 304 that receives graphics frame data from a remote server 306. The graphics rendering application 304 is configured to perform the operations described above with respect to the graphics rendering application 210.​Figure 3 Many of the similarly named elements described are the same or similar elements, including, for example, processor 304, memory 306, which can include a combination of volatile and non-volatile memory, IC chip 308, including matrix multiplication engine 310, and display 312.

[0041] Memory 306 of user device 302 stores media viewing application 314, which performs actions for presenting a received stream (e.g., media stream 316) of video content generated by graphics rendering application 328 hosted by media server 318 on display 312.

[0042] Graphics rendering application 328 retrieves or generates the content of media stream 316 and initiates transmission of such content to the user device. For example, the graphics rendering application can be a cloud-based game engine or a video streaming tool. In contrast, media viewing application 314 is a locally installed application (e.g., a web plug-in) that receives frame data within media stream 316 and generates rendering instructions that instruct low-level graphics hardware (not shown) to render the frame data to display 312. In one implementation, media viewing application 314 includes logic for communicating with a locally executed SRNN inference engine 322 that upscales (increases resolution and fidelity) each frame received within media stream 316 before the frame is rendered on display 312. In the illustrated implementation, media stream 316 includes video data that may, for example, include streamed video (e.g., movies, television), game graphics, etc. In the illustrated implementation, media server 318 also provides metadata 320 associated with media stream 316 to user device 302. Metadata 320 helps SRNN inference engine 322 identify one or more particular weight matrices needed to upscale frames of media stream 316 and load each identified weight matrix into memory 306 at the appropriate time.

[0043] Metadata 320 can be included with media stream 316 (e.g., attached to individual video frame packets) or transmitted separately, such as at a time prior to transmitting media stream 316. In one implementation, media viewing application 314 requests and receives metadata 320 from media server 318 when the user first initiates a particular cloud-based application, such as when the user initiates a game (or a new level of a game) hosted by media server 316.

[0044] For example, upon loading each new scene of a graphics rendering application, the media server 318 can transmit a packet(s) of metadata 320 that includes a weight matrix for upgrading frames of the scene, or that includes information that can be used by the user device 302 to independently fetch and store such weight matrices. In the illustrated implementation, the metadata 320 includes a list of weight matrix identifiers (e.g., matrix IDs) that each identify a corresponding weight matrix that has been pre-trained on a subset of frames from the media stream 316 to facilitate upgrading of discrete, predefined segments of frames within the media stream 316. Each matrix identifier in the metadata 320 is transmitted to the user device 302 in association with a checkpoint identifier that identifies a particular location within the media stream 316 at which the associated weight matrix is to be loaded into memory utilized by the SRNN inference engine 322. For example, the checkpoint identifier can include one or more timestamps or frame numbers. Each checkpoint identifier can be used to identify a segment of frames within the media stream 316 that are to be upgraded based on the weights of the corresponding weight matrix. For example, the metadata 320 can include a matrix identifier “Level 1 Matrix” that is associated with a timestamp or frame number that identifies the beginning of a segment of frames within the media stream 316 that are to be upgraded using the matrix with ID “Level 1 Matrix.” The media viewing application 314 provides the metadata 320 to the SRNN inference engine 322, and the SRNN inference engine 322 uses the metadata 320 to determine when to dynamically retrieve and load each different weight matrix that has been trained for the graphics rendering application 328. The SRNN inference engine 322 includes a matrix retrieval API 330 that dynamically requests selected weight matrices from the SRNN data store 334 during presentation of the media stream 316 on the display 312. For example, the matrix retrieval API 330 can track the location of a current play pointer within the media stream 316 and an offset between the read pointer and each checkpoint identifier identified in the metadata 320. When the read pointer is within a predefined offset of a location identified by a given one of the checkpoint identifiers, the matrix retrieval API 330 queries the SRNN data store 334 with the associated matrix ID (which may, in some cases, include both an identifier of the graphics rendering application 328 and an identifier of the weight matrix). In response, the SRNN data store 334 transmits the corresponding weight matrix back to the SRNN inference engine 322, and the weight matrix is loaded into temporary memory 336 on the user device 302. The weight matrix is dynamically moved from the temporary memory 336 into the SRNN inference engine 322 at a time selected based on the associated checkpoint identifier. Once loaded into the SRNN inference engine 322, the weight matrix is used to upgrade the segment of frames corresponding to the checkpoint identifier. In one implementation, the temporary memory 336 is a buffer that stores a small number of weight matrices of the graphics rendering application 328 at a time.Once used to upgrade an associated frame segment, each weight matrix in the buffer 336 can be overwritten with a newly retrieved weight matrix. In this sense, the user device 302 need not store all weight matrices that have been pre-defined with respect to the graphics rendering application at the same time, thereby reducing storage requirements.

[0045] Figure 4 Another example system 400 is illustrated that performs operations for dynamically upgrading the resolution and fidelity of graphics frame data received at a user device 402. The user device 402 includes many of the same hardware and hardware components described with respect to the user device 302, including, for example, a media server 418 that hosts a graphics rendering application 428. The graphics rendering application 428 renders graphics and transmits media streams 416 to the user device 402. At the user device 402, a SRNN inference engine 422 utilizes a matrix multiplication engine 410 (integrated within the IC 408) to perform actions to upgrade each frame of the media stream 416 before the media viewing application 414 renders the frame to a display 412 of the user device 402. Figure 3 The hardware and software components described are the same as those described with respect to the user device 302, including, for example, a media server 418 that hosts a graphics rendering application 428. The graphics rendering application 428 renders graphics and transmits media streams 416 to the user device 402. At the user device 402, a SRNN inference engine 422 utilizes a matrix multiplication engine 410 (integrated within the IC 408) to perform actions to upgrade each frame of the media stream 416 before the media viewing application 414 renders the frame to a display 412 of the user device 402.

[0046] The user device 402 differs from the user device 302 of Figure 3 The user device 402 differs from the user device 302 of Figure 3 The user device 402 differs from the user device 302 of

[0047] In one implementation, the SRNN inference engine 422 receives some or all of the weight matrices for the graphics rendering application prior to streaming and local rendering and stores them in the DRAM 432. For example, all of the weight matrices for the graphics rendering application 428 can be downloaded and stored when the user first begins streaming from the graphics rendering application 428 or when the media viewing application 414 is downloaded onto the user device 402.

[0048] Rather than requesting matrices one at a time as needed, the matrix retrieval API 430 of the SRNN inference engine 422 queries the SRNN data store 434 with an application ID (e.g., a media title or other identifier associated with the graphics rendering application 428) and in turn sends groups of weight matrices trained for different segments of the graphics rendering application 428.

[0049] Figure 5Example operations 500 for upgrading frames received at a user device within a media stream without increasing bandwidth, cost, memory, power, or bitrate are illustrated. A streaming operation 502 streams a media stream to the user device. Each frame of the media stream has a first resolution and has been generated by a cloud-based graphics rendering application.

[0050] A weight matrix identification operation 504 identifies an application-specific weight matrix that has been pre-trained on frames of the application to upgrade segments of the graphics frame from the first resolution to a higher second resolution. According to one implementation, the weight matrix identification operation 504 is performed by an SRNN inference engine executing locally on the user device. In one implementation, the identified weight matrix is one that has been trained based on a training set that specifically includes frames generated by the same graphics rendering application.

[0051] Once the appropriate weight matrix has been identified, a retrieval and loading operation 506 retrieves the application-specific weight matrix and loads the application-specific weight matrix into the SRNN inference engine. In one implementation, the application-specific weight matrix is loaded into an SRAM structure. Retrieving the weight matrix can require (1) locating the matrix from a repository stored on the user device, or (2) querying the matrix in a remote data store.

[0052] A graphics upgrade operation 508 uses the application-specific weight matrix to upgrade the graphics frame segment from the first resolution to the higher second resolution, and the upgraded frame is rendered locally to a display on the user device, and the graphics rendering application renders the upgraded frame locally to a display of the client device.

[0053] The above-described techniques allow a developer (e.g., a game studio) to produce one or more different weight matrices that are optimal for each game title. In effect, this uses Al to add pixels and details to each image based on a matrix that is specifically trained for the media title to better depict the overall look of each frame.

[0054] The matrix is passed to the user device and executed locally. According to one implementation, the weight matrix can be changed and versioned to allow for quality adjustments associated with any changes to the game, such as enabling the release of new levels, scenes, maps, etc.

[0055] Figure 6 An example schematic of a processing device 600 suitable for implementing aspects of the disclosed technology is illustrated. The processing device 600 includes one or more processing units 602, memory 604, a display 606, and other interfaces 608 (e.g., buttons). The memory 604 generally includes both volatile memory (e.g., RAM) and non-volatile memory (e.g., flash memory). An operating system 610 (such as Microsoft Windows®, Apple iOS®, Google Android®, etc.) is stored in the memory 604 and is executable by the processing unit(s) 602. Microsoft Windows® operating systems, Apple Macintosh® operating systems, UNIX® operating systems, LINUX® operating systems, Google Android® operating systems, Microsoft Phone operating systems, or a specific operating system designed for a gaming device) resides in memory 604 and is executed by processor unit(s) 602, although other operating systems can be employed.

[0056] One or more applications 612, such as a SRNN inference engine, are loaded in memory 604 and executed by processor unit(s) 602 on the operating system 610. Applications 612 can receive input from various input devices, such as microphone 634 or input accessory 636 (e.g., a keypad, mouse, stylus, touchpad, gamepad, racing wheel, joystick). Processing device 600 includes a power supply 616, which is powered by one or more batteries or other power sources and provides power to the other components of processing device 600. Power supply 616 can also be connected to an external power source that supercharges or recharges the internal battery or other power source.

[0057] Processing device 600 includes one or more communication transceivers 630 and antennas 632, which can provide network connectivity (e.g., mobile telephone networks, wireless data networks, BlueTooth® ) networks, etc.). Processing device 600 can also include various other components, such as a positioning system (e.g., a global positioning satellite transceiver), one or more accelerometers, one or more cameras, an audio interface (e.g., microphone 634, audio amplifiers and speakers, and / or audio jacks), and storage device 628. Other configurations can also be employed.

[0058] In example implementations, mobile operating systems, various applications (e.g., a SRNN inference engine), and other modules and services can have hardware and / or software instantiated by instructions stored in memory 604 and / or storage device 628 and processed by processor unit(s) 602. Memory 604 can be a memory of a host device or an accessory coupled to a host.

[0059] ​The processing device 600 can include various tangible computer-readable storage media and intangible computer-readable communication signals. Tangible computer- readable storage can be embodied by any available media that can be accessed by the processing device 600 and includes both volatile and nonvolatile storage media, removable and non-removable media. Tangible computer-readable storage media excludes intangible and transitory communications signals, but includes volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Tangible computer-readable media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other tangible medium which can be used to store the desired information and which can be accessed by the processing device 600. In contrast to tangible computer-readable storage media, intangible computer-readable communication signals can embody computer readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other signal transport mechanism. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, intangible communication signals include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency, infrared and other wireless media.

[0060] Some embodiments can include an article of manufacture. An article of manufacture can include a tangible storage medium to store logic. Examples of a storage medium can include one or more types of memory on a computer-readable storage medium, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writeable or re-writeable memory, and the like. Examples of logic can include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. For example, in one embodiment, an article of manufacture can store executable computer program instructions, which, when executed by a computer, cause the computer to perform methods and / or operations in accordance with various embodiments described. The executable computer program instructions can include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, and the like. The executable computer program instructions can be implemented according to a predefined computer language, manner or syntax, for instructing a computer to perform a certain function. The instructions can be implemented using any suitable high-level, low-level, object-oriented, visual, compiled and / or interpreted programming language.

[0061] The example methods disclosed herein include receiving, at a user device (e.g., processing device 102), a media stream comprising frames of a first resolution generated by a graphics rendering application (e.g., media stream 122), and locally upgrading each received frame of the media stream to a second resolution greater than the first resolution on the user device with one or more weight matrices (e.g., weight matrices 120) pre-trained in association with the graphics rendering application. The methods further provide rendering the upgraded frames of the media stream to a display of the user device. From an end user’s perspective, this allows for a reduction in bandwidth without negatively impacting video fidelity or quality. Figure 1 Figure 1 In another example method according to any of the foregoing methods, each of the frames is upgraded to the second resolution using a select weight matrix of the one or more weight matrices trained in association with the graphics rendering application, and the upgrading of each of the frames provides supplementing the frame with additional detail derived based on the associated select weight matrix. Using a scene-specific or application-specific weight matrix utilizes Al to add pixels and detail to each image based on matrices specifically trained for the media title to better depict the overall appearance of each frame. Figure 1

[0062] In another example method according to any of the foregoing methods, each of the frames is upgraded to the second resolution using a select weight matrix of the one or more weight matrices trained in association with the graphics rendering application, and the upgrading of each of the frames provides supplementing the frame with additional detail derived based on the associated select weight matrix. Using a scene-specific or application-specific weight matrix utilizes Al to add pixels and detail to each image based on matrices specifically trained for the media title to better depict the overall appearance of each frame.

[0063] ​​Yet another example method of any preceding method further provides transmitting a request from the user device for a selection weight matrix. The request specifies a matrix identifier stored in a remote database in association with a media title of the graphics rendering application. The weight matrix is received and stored locally on the user device in response to transmitting the request. The method further provides dynamically loading a super-resolution neural network (SRNN) inference engine (e.g., SRNN engine 112) using the selection weight matrix when a rendering operation reaches a predefined checkpoint within the media stream. Figure 1 This dynamic retrieval and loading of the weight matrix(es) reduces the amount of memory resources on the user device sufficient to implement the above-described solutions.

[0064] In still another example method of any preceding method, the plurality of weight matrices are pre-trained in association with the graphics rendering application, and the method further provides loading the SRNN inference engine using a different one of the weight matrices when each of a plurality of predefined checkpoints is reached during presentation of the media stream.

[0065] In still another example method of any preceding method, the plurality of weight matrices are pre-trained in association with the graphics rendering application, and each of the weight matrices is trained using frames sampled from an associated one of a plurality of different discrete frame segments of the media stream. For example, scene-specific weight matrices can be created to allow resolution and fidelity enhancements that are specifically tailored to features and tonal profiles that are unique to different scenes. In another example method of any preceding method, each of the weight matrices is used to dynamically upgrade frames of the associated one of the plurality of different discrete frame segments during media stream operations.

[0066] In still another example method of any preceding method, the user device is a game console, and the one or more weight matrices are stored on the console for dynamic loading into a super-resolution neural network (SRNN) inference engine at a selection time associated with different checkpoints within the media stream.

[0067] In still another example method of any preceding method, the user device is a mobile device, and the method further includes receiving and storing metadata of the graphics rendering application (e.g., Figure 3The metadata specifies matrix identification information that uniquely identifies a plurality of different weight matrices pre-trained for the graphics rendering application, and further specifies checkpoint identifiers that each identify a checkpoint location within the media stream at which an associated one of the plurality of different weight matrices is loaded into a super-resolution neural network (SRNN) inference engine. The metadata can provide a user device with sufficient information to identify weight matrices trained for different scenes in a media stream, and to dynamically load those weight matrices at the appropriate time.

[0068] An example device disclosed herein includes a super-resolution neural network (SRNN) inference engine (e.g., an application programming interface (API) 110) stored in a memory of a user device (e.g., a smartphone 102). The SRNN inference engine is configured to receive a media stream (e.g., a low resolution media stream 114) that includes frames of a first resolution generated by a graphics rendering application (e.g., a game application 106). The SRNN inference engine is further configured to locally upgrade each of a plurality of frames received within the media stream from the first resolution to a second resolution that is greater than the first resolution using one or more weight matrices (e.g., weight matrices 120) pre-trained in association with the graphics rendering application. The system further includes a media viewing application (e.g., a media viewing application 108) executing locally on the device that presents upgraded frames of the media stream to a display (e.g., a display 104) of the user device. Figure 1 Figure 1 The SRNN inference engine is configured to receive a media stream (e.g., a low resolution media stream 114) that includes frames of a first resolution generated by a graphics rendering application (e.g., a game application 106). The SRNN inference engine is further configured to locally upgrade each of a plurality of frames received within the media stream from the first resolution to a second resolution that is greater than the first resolution using one or more weight matrices (e.g., weight matrices 120) pre-trained in association with the graphics rendering application. The system further includes a media viewing application (e.g., a media viewing application 108) executing locally on the device that presents upgraded frames of the media stream to a display (e.g., a display 104) of the user device. Figure 1 Figure 1 The SRNN inference engine is configured to receive a media stream (e.g., a low resolution media stream 114) that includes frames of a first resolution generated by a graphics rendering application (e.g., a game application 106). The SRNN inference engine is further configured to locally upgrade each of a plurality of frames received within the media stream from the first resolution to a second resolution that is greater than the first resolution using one or more weight matrices (e.g., weight matrices 120) pre-trained in association with the graphics rendering application. The system further includes a media viewing application (e.g., a media viewing application 108) executing locally on the device that presents upgraded frames of the media stream to a display (e.g., a display 104) of the user device. Figure 1

[0069] In another example system of any of the foregoing systems, upgrading each of the plurality of frames received within the media stream includes supplementing each of the frames with additional detail derived based on an associated one of the one or more weight matrices.

[0070] In yet another example system of any of the foregoing systems, the device further includes an application programming interface (API) configured to transmit a request to select a weight matrix during presentation of the media stream. The request specifies a matrix identifier stored in a remote database in association with a media title of the graphics rendering application. The requested weight matrix is received on the user device and locally stored in response to transmitting the request, and the user device is adapted to use the selected weight matrix to dynamically load the SRNN inference engine in response to reaching a predetermined checkpoint during presentation of the media stream.

[0071] ​​​In yet another example system of any of the preceding systems, the plurality of weight matrices are pre-trained in association with the graphics rendering application, and a different one of the weight matrices is used to dynamically load the SRNN inference engine when each of a plurality of predefined checkpoints is reached during presentation of the media stream.

[0072] In another example system of any of the preceding systems, the plurality of weight matrices are pre-trained in association with the graphics rendering application, and each of the weight matrices is trained using frames sampled from an associated one of a plurality of different discrete frame segments of the media stream.

[0073] In yet another example system of any of the preceding systems, each of the weight matrices is used to dynamically upgrade frames of the associated one of the plurality of different discrete frame segments during media stream operation.

[0074] In another example system of any of the preceding systems, the user device is a game console, and the one or more weight matrices are jointly stored on the console for dynamic loading into the SRNN inference engine in association with different checkpoints within the media stream.

[0075] An example computer-readable storage medium disclosed herein encodes computer-executable instructions for performing a computer process comprising: receiving, at a user device (e.g., processing device 102), a media stream comprising frames of a first resolution generated by a graphics rendering application (e.g., graphics rendering application 210); locally upgrading each received frame of the media stream to a second resolution greater than the first resolution with one or more weight matrices (e.g., weight matrices 120) pre-trained in association with the graphics rendering application; and rendering the upgraded frames of media stream to a display of the user device (e.g., high resolution media stream 122). Figure 1 Figure 2 Figure 1 Figure 1 Figure 1

[0076] On an example computer-readable storage medium of any of the preceding computer-readable storage media, the computer process provides upgrading each of the frames to the second resolution using a selected weight matrix of the one or more weight matrices trained in association with the graphics rendering application. The upgrading of each frame includes supplementing each frame with additional detail derived based on the associated selected weight matrix.

[0077] ​​​​​In another example computer-readable storage medium of any previous computer-readable storage medium, the computer process further includes transmitting a request for a selection weight matrix from the user device. The request specifies a matrix identifier stored in a remote database in association with a media title of the graphics rendering application. The selection weight matrix is received on the user device and stored locally in response to transmitting the request. The computer process further provides dynamically loading a super-resolution neural network (SRNN) inference engine (e.g., SRNN engine 112) using the selection weight matrix when a rendering operation reaches a predefined checkpoint within the media stream. Figure 1

[0078] In yet another example computer-readable storage medium of any previous computer-readable storage medium, the plurality of weight matrices are pre-trained in association with the graphics rendering application, and the computer process further includes loading the SRNN inference engine using a different one of the weight matrices when each of a plurality of predefined checkpoints is reached during presentation of the media stream.

[0079] An example system disclosed herein includes means for receiving, at a user device, a media stream comprising frames generated by a graphics rendering application at a first resolution; and means for locally upgrading, at the user device, each received frame of the media stream to a second resolution greater than the first resolution with one or more weight matrices pre-trained in association with the graphics rendering application. The system further provides means for rendering the upgraded frames of the media stream to a display of the user device.

[0080] The above specification, examples, and data provide a complete description of the structure and use of example implementations. Since many implementations can be made without departing from the spirit and scope of the claimed application, the following claims are intended to cover all implementations falling within the scope of the claims. Furthermore, structural features of different examples can be combined with one another without departing from the claimed claims.​

Claims

1. A method for upscaling frames of a media stream, comprising: receiving, at a user device, a media stream comprising frames of a first resolution generated by a graphics rendering application; on the user device, utilizing one or more weight matrices to locally upscale each received frame of the media stream to a second resolution greater than the first resolution, wherein each weight matrix is pre-trained in association with the graphics rendering application to provide scaling weights for a predefined granularity of segments or frame groups; and rendering the upscaled frames of the media stream to a display of the user device.

2. The method of claim 1, wherein, each of the frames is upscaled to the second resolution using a select weight matrix of the one or more weight matrices trained in association with the graphics rendering application, the upscaling of each of the frames providing the frame with additional detail derived based on the associated select weight matrix.

3. The method of claim 1, further comprising: transmitting, from the user device, a request for a select weight matrix, the request specifying a matrix identifier stored in a remote database in association with a media title of the graphics rendering application; receiving the select weight matrix on the user device and storing the select weight matrix locally in response to transmitting the request; using the select weight matrix to dynamically load a super-resolution neural network (SRNN) inference engine when a rendering operation reaches a predefined checkpoint within the media stream.

4. The method of claim 3, wherein, the plurality of weight matrices are pre-trained in association with the graphics rendering application, and wherein the method further comprises: using a different weight matrix of the weight matrices to load the SRNN inference engine when each of a plurality of predefined checkpoints is reached during presentation of the media stream.

5. The method of claim 1, wherein, the plurality of weight matrices are pre-trained in association with the graphics rendering application, and each of the weight matrices is trained using frames sampled from an associated one of a plurality of different discrete frame segments of the media stream.

6. The method of claim 5, wherein, each of the weight matrices is used to dynamically upscale frames of the associated one of the plurality of different discrete frame segments during media stream operation.

7. The method of claim 1, wherein, the user device is a game console and the one or more weight matrices are stored on the console for dynamic loading into a super-resolution neural network (SRNN) inference engine at select times associated with different checkpoints within the media stream.

8. The method of claim 1, wherein, the user device is a mobile device and the method further comprises: receiving and storing metadata of the graphics rendering application, the metadata specifying matrix identification information uniquely identifying a plurality of different weight matrices pre-trained for the graphics rendering application, and further specifying checkpoint identifiers each identifying a checkpoint location within the media stream at which an associated one of the plurality of different weight matrices is loaded into a super-resolution neural network (SRNN) inference engine.

9. A system for upscaling frames of a media stream, comprising: a super-resolution neural network, SRNN, inference engine stored in memory of a user device and configured to: receive a media stream comprising frames of a first resolution generated by a graphics rendering application and utilize one or more weight matrices to locally upscale each of a plurality of frames received within the media stream from the first resolution to a second resolution greater than the first resolution, wherein each weight matrix is pre-trained in association with the graphics rendering application to provide scaling weights for a predefined granularity of segments or groups of frames; and a media viewing application executing locally on the device that presents the upscaled frames of the media stream to a display of the user device.

10. The system of claim 9, wherein, Upscaling each of the plurality of frames received within the media stream includes supplementing each of the frames with additional detail derived based on an associated one of the one or more weight matrices.

11. The system of claim 9, wherein, The device further includes: an application programming interface, API, configured to: communicate a request to select a weight matrix during presentation of the media stream, the request specifying a matrix identifier stored in association with a media title of the graphics rendering application in a remote database; and receive the requested weight matrix on the user device in response to communicating the request and locally store the requested weight matrix, the user device adapted to use the selected weight matrix to dynamically load the SRNN inference engine in response to reaching a predetermined checkpoint during presentation of the media stream.

12. The system of claim 9, wherein, The plurality of weight matrices are pre-trained in association with the graphics rendering application, and wherein a different one of the weight matrices is used to dynamically load the SRNN inference engine when each of a plurality of predefined checkpoints is reached during presentation of the media stream.

13. The system of claim 9, wherein, The plurality of weight matrices are pre-trained in association with the graphics rendering application, and each of the weight matrices is trained using frames sampled from an associated one of a plurality of different discrete frame segments of the media stream.

14. The system of claim 13, wherein, Each of the weight matrices is used to dynamically upscale frames of the associated one of the plurality of different discrete frame segments during media stream operation.

15. The system of claim 13, wherein, The user device is a game console, and the one or more weight matrices are jointly stored on the console for dynamic loading into the SRNN inference engine in association with different checkpoints within the media stream.

16. One or more computer-readable storage media encoding computer-executable instructions for executing a computer process comprising: receiving, at a user device, a media stream comprising frames of a first resolution generated by a graphics rendering application; at the user device, utilizing one or more weight matrices to locally upscale each received frame of the media stream to a second resolution greater than the first resolution, wherein each weight matrix is pre-trained in association with the graphics rendering application to provide scaling weights for a predefined granularity of segments or groups of frames; and rendering an upscaled frame of the media stream to a display of the user device.

17. The one or more computer-readable storage media of claim 16, wherein, each of the frames is upscaled to the second resolution using a select weight matrix of the one or more weight matrices trained in association with the graphics rendering application, the upscaled of each of the frames supplementing each of the frames with additional detail derived based on the associated select weight matrix.

18. The one or more computer-readable storage media of claim 16, wherein, The computer process further includes: transmitting, from the user device, a request for a select weight matrix, the request specifying a matrix identifier stored in a remote database in association with a media title of the graphics rendering application; receiving the select weight matrix on the user device and locally storing the select weight matrix in response to transmitting the request; and dynamically loading a super-resolution neural network (SRNN) inference engine using the select weight matrix when a rendering operation reaches a predefined checkpoint within the media stream.

19. The one or more computer-readable storage media of claim 18, wherein a plurality of weight matrices are pre-trained in association with the graphics rendering application, and wherein the computer process further includes: loading the SRNN inference engine using a different weight matrix of the weight matrices when each of a plurality of predefined checkpoints is reached during presentation of the media stream.

Images