Dynamic user device upscaling of media streams - Patent Application 20070122997

Abstract

The disclosed method provides for receiving, at a user device, a media stream including frames at a first resolution generated by a graphics rendering application, and locally upscaling each received frame of the media stream to a second resolution higher than the first resolution at the user device using one or more weighting matrices pre-trained in association with the graphics rendering application. The local upscaling of the media stream may be performed “on the fly,” for example, for an individual content stream (e.g., a game) or a segment of a content stream (e.g., a scene within a game).

Description

[Background technology]

[0001] Increasingly, more graphics media applications (e.g., games) are becoming cloud-based, but streaming high-definition video is bandwidth-intensive. Some home internet plans are insufficient to support streaming large amounts of high-definition video at advertised frame rates. Furthermore, streaming high-definition video requires cloud-based content providers. Take It is memory and power intensive. Lower memory, lower bandwidth and cheaper solutions are desired. Summary of the Invention

[0002] According to one implementation, a method for increasing the resolution of streamed graphics without increasing bandwidth consumption includes receiving, at a user device, a media stream including frames at a first resolution generated by a graphics rendering application, and locally upscaling each frame of the media stream received at the user device to a second resolution higher than the first resolution using one or more weighting matrices pre-trained in association with the graphics rendering application.

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Other implementations are also described and recited herein. [Brief explanation of the drawings]

[0004] [Figure 1] 1 illustrates an exemplary system for dynamically upscaling the resolution of a received video stream of media content using artificial intelligence executed on a user device. [Figure 2]1 illustrates a system that performs an exemplary training operation to generate a weighting matrix that can be used to dynamically upscale the resolution of graphics frame data streamed to a user device. [Figure 3] 1 illustrates a system that performs operations to dynamically upscale the resolution and fidelity of graphics frame data received at a user device. [Figure 4] 1 illustrates another exemplary system that performs operations to dynamically upscale the resolution and fidelity of graphics frame data received at a user device. [Figure 5] 10 illustrates an exemplary operation for upscaling frames of a media stream received at a user device. [Figure 6] 1 depicts an exemplary schematic diagram of a processing device suitable for implementing aspects of the disclosed technology. DETAILED DESCRIPTION OF THE INVENTION

[0005] Provided herein is a bandwidth-efficient solution for rendering, streaming, and upscaling media content to enable the presentation of high-resolution (e.g., 4k) content on a user device display. According to one implementation, a media stream is broadcast to a user device at a first resolution that is much lower than the final resolution at which graphics will ultimately be presented on the user device. The user device locally performs operations that enhance both the resolution and image fidelity (e.g., detail).

[0006] While some existing techniques are designed to locally increase the resolution of streamed content at the user device, these conventional techniques tend to rely on uniform scaling methods such as bicubic interpolation, bilinear interpolation, or "nearest-neighbor" methods that predict pixel values ​​from the values ​​of nearest neighbor pixels. All of these methods generally provide uniform scaling of the luminance / color of newly added frame pixels according to static, predefined values ​​that are set relative to the values ​​of nearby pixels in the original image. While these solutions improve the final image resolution, they are ineffective. As the resolution increases, the image becomes pixelated, resulting in a "blocky" appearance that makes it appear to have a lower resolution than it actually does.

[0007] In contrast to these conventional solutions, the presently disclosed technology provides an image upscaling tool that increases the resolution of incoming frames on the fly (e.g., at 60 or 120 frames per second) by using artificial intelligence trained to improve image fidelity by "filling in the gaps," so that newly added pixels provide detail not present in the original frame. According to one implementation, the proposed solution allows frames to be rendered (e.g., by a game engine) at a lower resolution, such as 720p or 1080p. The lower-resolution frames are transmitted over a network to a user device and locally upscaled to a higher quality, such as 4k, using an AI-trained upscaling tool.

[0008] In particular, server-level streaming at 4k resolution involves transmitting a data stream that is four times the size of a 1080p resolution stream. High-resolution media streaming therefore imposes cost burdens on both data centers (high power and bandwidth costs) and end users (e.g., ensuring home Internet streaming setups that can support both high-bitrate and high-bandwidth streaming (e.g., 20 to 40 Mbps)). In contrast, a media stream at 1080p resolution consumes only a small fraction of the bandwidth consumed by 4k (e.g., 5 Mbps). This bandwidth savings is significant for data centers in terms of both bandwidth and power savings. Furthermore, this reduction in bandwidth benefits end users, as it can be achieved while also achieving improved video quality and fidelity. For users, the bandwidth savings translate into improved latency (because more bandwidth can be directed toward controller response) and monetary cost savings due to a reduced need for high data caps offered by more expensive Internet plans.

[0009] The resolution enhancement solution described herein allows a media stream to be transmitted at a lower resolution and viewed at a relatively higher resolution, with significant bandwidth savings and without a corresponding loss in frame rate or content. This is achieved while also reducing bandwidth, power, and memory consumption during actual frame rendering (e.g., by an application or game server), since this rendering is done at a lower resolution. In various implementations, this technique can be applied to enhance quality (image fidelity) and / or frame rate, depending on how users, game developers, and cloud application streaming providers utilize the technology.

[0010] FIG. 1 illustrates an exemplary system 100 for dynamically upscaling the resolution of a received video stream of media content using artificial intelligence (AI) running on a user device. The system 100 includes a processing device 102, which may take a variety of forms, including, but not limited to, a laptop or tablet, a desktop computer, a gaming console, a smartphone, etc. The processing device includes hardware implemented as an integrated circuit (IC) 104 that includes a matrix multiplication engine 106, sometimes referred to as a matrix math accelerator. For example, the IC 104 may include one or more standard silicon systems-on-chips (SoCs), application-specific integrated circuits (ASICs), or FPGAs. According to one implementation, the matrix multiplication engine 106 is easily programmable and efficiently adaptable to any neural network.

[0011] In addition to IC 104 and matrix multiplication engine 106, processing device 102 further includes memory 108 and processor 110. Although memory 108 and processor 110 are shown as being external to IC 104, IC 104 may include additional memory and / or one or more processors in addition to processor 110. A super-resolution neural net (SRNN) inference engine 112 is stored in memory 108 and executed by processor 110. In other implementations, some or all aspects of SRNN inference engine 112 are stored within memory of IC 104 and / or executed by a processor of IC 104.

[0012] The SRNN inference engine 112 is software executed by the processor 110 to dynamically upscale (or upsample) graphics frames of an incoming low-resolution media content stream 114 as the frames arrive at the processing device 102 from the media server 116. While "low resolution" is a relative term when used independently, the term is used throughout this specification in reference to a media content stream received at a user device at a lower resolution and / or less detail than the resolution and / or detail of the final media content stream (e.g., high-resolution media stream 122) that will be rendered on the display 118 of the processing device 102. As used herein, "upscaling" refers to an increase in image resolution and fidelity (detail).

[0013] In different implementations, the media server 116 may take various forms, including those of one or more different web-based servers and / or edge devices that host and / or generate media content. For example, the media server 116 may be a cloud-based gaming service (xCloud) or an application server that stores and executes graphics generation applications to generate low-resolution media streams 114 that are then streamed to user devices (e.g., processing device 102).

[0014] In one implementation, the SRNN inference engine 112 is pre-loaded with trained weights stored in a weight matrix 120 that are used by the SRNN inference engine 112 to transform each input frame of the low-resolution media stream 114 into a corresponding output frame of the high-resolution media stream 122. In various implementations, the trained weights may be sent either before or during the streaming of the low-resolution media stream 114 to the processing device 102. For a single graphics rendering application, there may be multiple weight matrices trained to upscale content for different segments of the associated media stream. In one example where the low-resolution media stream 114 is game content, a new game scene may use different video content and style than a previous game scene. In this case, the new game scene may be upscaled by loading a different weight matrix into the matrix multiplication engine 106 than the weight matrix used for the previous scene.

[0015] In various implementations, different weighting matrices associated with the same media content stream may be transmitted to processing device 102 in various ways. In one implementation, the weighting matrices are transmitted to processing device 102 as a bulk package and cached locally for a period of time. The locally stored weighting matrices may be selectively loaded into matrix multiplication engine 106 when their associated frame segments are streamed and / or processed for local rendering. In another implementation, different weighting matrices for low-resolution media stream 114 are transmitted to processing device 102 individually before or during a scene transition associated with the new matrix. In this scenario, processing device 102 does not need to cache large amounts of information. In yet another implementation, the different weighting matrices are streamed simultaneously with low-resolution media stream 114, with anticipation of the next time each matrix will be loaded and used.

[0016] In one implementation where weighting matrices are transmitted in bulk and / or cached locally at processing device 102, the graphics rendering application at media server 116 may inform processing device 102, through an established API, to use the correct weighting matrix for each individual scene to be displayed. This may be achieved, for example, by tagging each of the weighting matrices with metadata that indicates which frame segments should be upscaled with each weighting matrix stored locally in local memory.

[0017] If the low-resolution media stream 114 stops (e.g., the game ends or the user stops the stream), the locally stored data may be versioned, allowing developers to change the quality of the trained weights when updating the game.

[0018] In one implementation, the SRNN inference engine 112 is a convolutional neural network (CNN) that includes several convolution layers that collectively provide a transformation of each feature of the original low-resolution input frame into a corresponding higher-resolution version of that feature. The matrix multiplications used to upscale each input image to a corresponding super-resolution output image may be performed by the matrix multiplication engine 106. In other implementations, these mathematical operations are performed partially or exclusively by software elements rather than hardware elements.

[0019] In one implementation, weight matrix 120 is derived during a training process that utilizes stochastic descent or similar methodologies to identify a set of weights that achieves a desired effect in the transformation between an input (low-resolution) image and an output (high-resolution) image. In contrast to graphics upscaling solutions that provide a uniform selection of pixel values ​​based on static interpolation, weight matrix 120 is trained to provide scaling weights for a specific segment or group of frames of a predefined granularity, such as a group of frames related to the same media title (e.g., a game, application, or video). In one implementation, weight matrix 120 is trained on a per-title basis, such that each time a user launches a different cloud-based application hosted by media server 116, a different weight matrix is ​​loaded into the memory of SRNN inference engine 112. For example, each time a user launches a game hosted by media server 116, SRNN inference engine 112 identifies, retrieves, and loads a matrix with weights that have been trained specifically for the game. In another implementation, weight matrix 120 is trained on a scene-by-scene basis, such that each individual scene in a video or game is upscaled using a different weight matrix that has been trained specifically for that scene.

[0020] In FIG. 1 , weight matrix 120 is shown retrieved from a cloud-based SRNN data store 124. In the depicted example, SRNN data store 124 stores trained weight matrices for different media titles (graphics rendering applications), with each weight matrix stored in association with a media title 126 and a weight matrix ID 128. For example, the media title “Minecraft” is shown stored in association with multiple different weight matrix IDs, with each title / ID pair (e.g., [media title, matrix ID]) uniquely identifying a weight matrix trained to upscale a distinct predefined segment of frames associated with the media title. As used herein, a “segment” of frames refers to a collection of either consecutive or non-consecutive frames associated with the same media title. For example, each weight matrix is ​​used to upscale a different segment of frames, including, for example, a segment of frames displaying a particular animation, a segment of frames displaying a particular scene, or a segment of frames pre-identified as having a common characteristic such as text or menu options.

[0021] By way of example, and not limitation, a weighting matrix identified by the media title "Minecraft" and weighting matrix ID "Level1_Matrix" may be trained using a subset of frames rendered during "Level 1" of Minecraft and used by the SRNN inference engine 112 to perform an upscaling operation on all frames rendered during Level 1. Similarly, a matrix identified by the media title "Minecraft" and weighting matrix ID "Menu_Matrix" may be trained using frames generated by Minecraft that include menu icons or text and used by the SRNN inference engine 112 to upscale all frames that display menu content. Still further, an exemplary matrix identified by the media title "Minecraft" and weighting matrix ID "Animation1_Matrix" may be trained using frames that render several aspects of the same animation. This matrix may be used to upscale all frames that display that aspect of the animation.

[0022] Because the weighting matrices are each trained based on a limited, predefined segment of frames produced during execution of the same media executable (e.g., the same game), the resulting increased fidelity of each output frame fills in those details (e.g., colors, gradients, shading effects) that may be unique to the segment of frames used to train the corresponding weighting matrix, despite the fact that the details may vary significantly between titles, and even between different segments of the same title. The result is a more realistic and visually appealing rendering of the frame content than is achievable through traditional upscaling approaches such as pixel interpolation, or by using weighting matrices trained on frames from many different media titles.

[0023] FIG. 2 illustrates a system 200 that performs an example training operation to generate a weighting matrix 208 used to dynamically upscale the resolution of graphics frame data streamed to a user device. The system 200 includes an SRNN inference engine 202, which may include the same or similar characteristics as those described above with respect to FIG. 1. During the illustrated operation, the weighting matrix 208 is trained to effect image upscaling in association with the graphics of a single media title. As used herein, a "media title" refers to a particular graphics-rendering application (e.g., a game) that includes an executable file for generating video content for a media content stream. In FIG. 2, a developer 204 initiates an operation to train weights for the weighting matrix 208 in association with a graphics-rendering application 210.

[0024] In different implementations, developer 204 may choose to generate a single weight matrix 208 for graphics rendering application 210 or multiple different weight matrices for the graphics rendering application. For example, a single weight matrix may be generated and used to perform image upscaling for all frames generated by graphics rendering application 210, or alternatively, different weight matrices may be used to perform image upscaling for different segments of frames (e.g., groupings based on different scenes, different animations, and / or other visual characteristics shared among each group of frames).

[0025] 2, developer 204 trains weight matrix 208 on a selected subset of frames from a single individual scene ("Scene 1"), which may be an animation, a game level, or a particular scene within a game level. In this exemplary step, developer 204 selects training frames 214 from Scene 1 that are representative of the type of graphics present in Scene 1. Although not shown, developer 204 may also select one or more additional frames to be used to train the weight matrix for Scene 1. Each selected representative frame is provided as a training input to SRNN inference engine 202.

[0026] Upon receiving a training frame 214, the SRNN inference engine 202 generates a proposed upscaled version of the training frame 214 using predefined (e.g., initially default) weights in the weight matrix 208. In one implementation, the SRNN inference engine 202 includes several convolutional layers in a traditional neural network design. Each different convolutional layer can generate a subsequent feature map that is fed to the next layer. For example, each convolutional layer can serve to map an individual image feature or set of features to a higher-resolution feature or set of features.

[0027] In Figure 2, the proposed upscaled version of training frame 214 is represented in Figure 2 as "upscaled frame 218." Generating upscaled frame 218 involves (1) increasing the image resolution of training frame 214 by a predefined factor (e.g., from 720p to 4k), and (2) defining a proposed value for each pixel newly added to the frame based on the current weights contained in weight matrix 208. These weights may be adjusted throughout the training process using conventional techniques such as training, backpropagation, and gradient descent mechanisms. This training may be accomplished by either supervised or unsupervised training techniques.

[0028] In one implementation utilizing unsupervised learning, the SRNN training controller 212 compares each upscaled image (e.g., upscaled image frame 218) with a traditionally and fully rendered authentic high-resolution image (e.g., a 4k or higher resolution image). For example, the upscaled frame 218 may be compared to a high-resolution image or set of images selected by the SRNN training controller 212 or by the developer 204. Adjustments may be performed automatically by the SRNN training controller 212 to reduce discrepancies between the upscaled frame 218 and the authentic high-resolution image. In this implementation, the training process may continue to iterate through the training set until the output of the upscaled frame 218 best approximates the authentic high-resolution image.

[0029] Yet another implementation implements a hybrid approach in which tuner 216 uses both supervised and unsupervised inputs to tune the weights.

[0030] By way of example, and not limitation, the system 200 is shown using a supervised learning approach rather than an unsupervised approach, where the SRNN training controller 212 uses input from the developer 204 to best approximate the intended quality for each frame segment. For example, the developer 204 previews the upscaled frame 218 and provides a Yes / No input to the SRNN training controller 212. The Y / N input indicates whether the upscaled frame 218 meets certain aesthetic acceptance criteria. For example, the developer 204 may find that the upscaled frame 218 does not realistically enhance certain areas of the frame (e.g., a particular object appears too pixelated, or a shadow appears to extend into areas where it should not be).

[0031] The determination of whether an upscaled frame 218 meets particular aesthetic acceptance criteria is subject to the preferences of the individual developer in the supervised learning approach depicted, which effectively involves simply interpolating existing pixel values ​​to determine new pixel values. In contrast, AI is utilized to identify weights in the weighting matrix 208 that are effective in implementing artistically subjective resolution enhancements that improve image fidelity.

[0032] If the developer 204 is not satisfied with the upscaled frame 218, the developer 204 provides "No" as the supervised input 220, and the weight adjuster 216 of the SRNN training controller 212 calculates adjusted weight values ​​for the weight matrix 208. Using the adjusted weight values, the SRNN inference engine 202 may regenerate the upscaled frame 218, and the developer 204 may again provide the supervised input 220. The above-described weight adjustment and regeneration of the upscaled frame 218 may be performed iteratively a number of different times until the developer 204 provides "Yes" as the supervised input 220 to the SRNN training controller 212, indicating that the upscaled frame 218 meets aesthetic acceptance criteria. In addition to the supervised input 220, the weight adjuster 216 may use one or more additional conventional CNN training techniques to determine the weight adjustments at each iteration.

[0033] If developer 204 selects any other training frames for the same weighting matrix (e.g., other training frames representative of scene 1 in addition to training frame 214), the above operations may be repeated, with the current values ​​of weighting matrix 208 being used as starting values ​​for correction operations on the next received training frame. In some implementations, developer 204 may select to train weighting matrix 208 on a single frame. In other implementations, weighting matrix 208 may be trained on multiple frames.

[0034] Once the above training operations have been performed for each training frame (e.g., training frame 214) selected for a given frame segment (e.g., scene 1), the SRNN inference engine 202 outputs the weight matrix 208 to the SRNN data store 224. The SRNN data store 224 may have the same or similar features as those described with respect to the SRNN data store 124 of FIG.

[0035] In this manner, weighting matrix 208 is trained on a selected subset of frames from a given frame segment (“Scene 1”), and the final version of weighting matrix 208 is used during live media streaming operations to dynamically adjust the resolution and fidelity of each frame as the given frame segment is received at the user device. Different weighting matrices may similarly be generated and used for various different frame segments of the media content stream generated by graphics rendering application 210.

[0036] 3 illustrates a system 300 that performs exemplary operations for dynamically upscaling the resolution and fidelity of graphics frame data received at a user device. In one implementation, the user device 302 is a mobile device with reduced memory and / or processing power compared to most PCs and game consoles. For example, the user device 302 may be a tablet or a mobile phone. The user device 302 includes many elements that are the same as or similar to the like-named elements described with respect to FIG. 1 , including, for example, a processor 304, memory 306, which may include a combination of volatile and non-volatile memory, an IC chip 308 that includes a matrix multiplication engine 310, and a display 312.

[0037] The memory 306 of the user device 302 stores a media viewing application 314 that performs operations to present on the display 312 a received stream of video content (e.g., media stream 316) generated by a graphics rendering application 328 hosted by a media server 318.

[0038] The graphics rendering application 328 retrieves or generates content for the media stream 316 and initiates transmission of such content to the user device. For example, the graphics rendering application may be a cloud-based game engine or a video streaming tool. In contrast, the media viewing application 314 is a locally installed application (e.g., a web plug-in) that receives frame data in the media stream 316 and generates rendering instructions that instruct low-level graphics hardware (not shown) to render the frame data on the display 312. In one implementation, the media viewing application 314 includes logic that communicates with a locally running SRNN inference engine 322 to upscale (increase the resolution and fidelity of) each frame received in the media stream 316 before the frame is rendered on the display 312.

[0039] In the illustrated implementation, the media stream 316 includes video data, which may be, for example, streamed video (e.g., movies, TV), game graphics, etc. In the illustrated implementation, the media server 318 also provides metadata 320 to the user device 302 in association with the media stream 316. The metadata 320 helps the SRNN inference engine 322 identify one or more specific weighting matrices needed to upscale frames of the media stream 316, as well as the appropriate time to load each identified weighting matrix into the memory 306.

[0040] The metadata 320 may be included with the media stream 316 (e.g., attached to individual video frame packets) or may be transmitted separately, e.g., prior to the transmission of the media stream 316. In one implementation, the media viewing application 314 requests and receives the metadata 320 from the media server 318 when a user first launches a particular cloud-based application, such as when the user launches a game (or a new level of a game) hosted on the media server 318.

[0041] For example, upon loading each new scene of a graphics rendering application, the media server 318 may send a packet of metadata 320 containing a weighting matrix for upscaling the frames of the scene or containing information used by the user device 302 to independently obtain and store such a weighting matrix.

[0042] In the illustrated implementation, the metadata 320 includes a list of weight matrix identifiers (e.g., matrix IDs), each identifying a corresponding weight matrix that has been pre-trained on a subset of frames from the media stream 316 to facilitate upscaling of individual pre-defined segments of frames within the media stream 316. Each matrix identifier in the metadata 320 is transmitted to the user device 302 in association with a checkpoint identifier that identifies a particular position within the media stream 316 where the associated weight matrix should be loaded into memory utilized by the SRNN inference engine 322. For example, the checkpoint identifier may include one or more time stamps or frame numbers. Each checkpoint identifier can be used to identify a frame segment within the media stream 316 that should be upscaled based on the weights of the corresponding weight matrix. For example, the metadata 320 may include a matrix identifier "Level1_Matrix" in association with a time stamp or frame number that identifies the start of a segment of frames within the media stream 316 that should be upscaled using a matrix with ID "Level1_Matrix." The media viewing application 314 provides metadata 320 to the SRNN inference engine 322, which uses this metadata 320 to determine when to dynamically retrieve and load each different weight matrix being trained for the graphics rendering application 328.

[0043] The SRNN inference engine 322 includes a matrix lookup API 330 that dynamically requests a selected weight matrix from the SRNN data store 334 during presentation of the media stream 316 on the display 312. For example, the matrix lookup API 330 may track the current playback pointer position within the media stream 316 and the offset between that retrieved pointer and each of the checkpoint identifiers identified in the metadata 320. If the retrieved pointer is within a predefined offset of the position identified by a given one of the checkpoint identifiers, the matrix lookup API 330 queries the SRNN data store 334 with the associated matrix ID (which, in some cases, may include both an identifier for the graphics rendering application 328 and an identifier for the weight matrix). In response, the SRNN data store 334 returns the corresponding weight matrix to the SRNN inference engine 322, which loads the weight matrix into temporary storage of the user device 302. The weight matrix is ​​dynamically loaded from temporary storage 336 into the SRNN inference engine 322 at a time selected based on the associated checkpoint identifier. Once loaded into the SRNN inference engine 322, the weighting matrix is ​​used to upscale the frame segment corresponding to the checkpoint identifier.

[0044] In one implementation, temporary storage 336 is a buffer that stores a small number of weight matrices for graphics rendering application 328 at a time. Once used to upscale the associated frame segment, each weight matrix in buffer 336 may be overwritten with the newly read weight matrix. In this sense, user device 302 does not need to simultaneously store all predefined weight matrices for a graphics rendering application, reducing storage requirements.

[0045] 4 illustrates another exemplary system 400 that performs operations to dynamically upscale the resolution and fidelity of graphics frame data received at a user device 402. The user device 402 includes many of the same hardware and software components as described with respect to FIG. 3, including, for example, a media server 418 that hosts a graphics rendering application 428. The graphics rendering application 428 renders graphics and sends a media stream 416 to the user device 402. At the user device 402, an SRNN inference engine 422 performs operations that utilize a matrix multiplication engine 410 (embedded in the IC 408) to upscale each frame of the media stream 416 before the frame is rendered by a media viewing application 414 to the display 412 of the user device 402.

[0046] 3 in that the user device 402 has a larger amount of local memory (e.g., DRAM 432) allocated for use by the SRNN inference engine 422 than the user device 302 of FIG 3. As a result, the SRNN inference engine 422 can simultaneously store multiple weight matrices in DRAM 432 rather than dynamically retrieving such matrices as needed.

[0047] In one implementation, the SRNN inference engine 422 receives and stores some or all weight matrices for the graphics rendering application in DRAM 432 prior to streaming and local rendering. For example, all of the weight matrices for the graphics rendering application 428 may be downloaded and stored when the media viewing application 414 is downloaded to the user device 402 or when the user first starts streaming from the graphics rendering application 428.

[0048] Rather than requesting matrices one by one as needed, the matrix lookup API 430 of the SRNN inference engine 422 queries the SRNN data store 434 using an application ID (e.g., a media title or other identifier associated with the graphics rendering application 428) and is returned a group of weight matrices trained for different segments of the graphics rendering application 428.

[0049] 5 illustrates example operations 500 for upscaling frames received at a user device in a media stream without increasing bandwidth, cost, memory, power, or bitrate. A streaming operation 502 streams the media stream to the user device. Each frame of the media stream has a first resolution and was generated by a cloud-based graphics rendering application.

[0050] A weight matrix identification operation 504 identifies an application-specific weight matrix that has been pre-trained on frames of the application for upscaling a segment of the graphics frame from a first resolution to a second resolution. According to one implementation, the weight matrix identification operation 504 is performed by an SRNN inference engine running locally on the user device. In one implementation, the identified weight matrix is ​​a weight matrix trained based on a training set that includes only frames produced by the same graphics rendering application.

[0051] Once an appropriate weight matrix is ​​identified, a lookup and load operation 506 retrieves and loads the application-specific weight matrix into the SRNN inference engine. In one implementation, the application-specific weight matrix is ​​loaded into an SRAM structure. Retrieving a weight matrix may involve either (1) locating the matrix from a repository stored on the user device or (2) querying a remote database of matrices.

[0052] An image upscaling operation 508 upscales a segment of the graphics frame from a first resolution to a second resolution using an application-specific weighting matrix, renders the upscaled frame locally to a display on the user device, and an image rendering application renders the upscaled frame locally to a display on the client device.

[0053] The above technology allows developers (e.g., game studios) to generate one or more different weighting matrices that are optimal for each game title, which effectively uses AI to add pixels and detail to each image based on a matrix trained specifically for the media title to better represent the overall look and feel of each frame.

[0054] The matrix is ​​sent to the user device and executed locally. In one implementation, the weight matrix can be modified and versioned to allow quality adaptation in connection with any changes to the game to allow for the release of new levels, scenes, maps, etc.

[0055] 6 depicts an exemplary schematic diagram of a processing device 600 suitable for implementing aspects of the disclosed technology. The processing device 600 includes one or more processor units 602, memory 604, a display 606, and other interfaces 608 (e.g., buttons). The memory 604 typically includes both volatile memory (e.g., RAM) and non-volatile memory (e.g., flash memory). An operating system 610, such as the Microsoft Windows® operating system, the Microsoft Windows Phone operating system, or specific operating systems designed for gaming devices, resides in the memory 604 and is executed by the processor unit 602, although it should be understood that other operating systems may be used.

[0056] One or more applications 612, such as an SRNN inference engine, are loaded into memory 604 by the processor unit 602 and executed by the operating system 610. The applications 612 may receive input from various input devices, such as a microphone 634 or input accessories 635 (e.g., a keypad, a mouse, a stylus, a touchpad, a gamepad, a racing wheel, a joystick). The processing device 600 includes a power supply 616, which is powered by one or more batteries or other power sources and provides power to the other components of the processing device 600. The power supply 616 may also be connected to an external power source that overrides or recharges the internal battery or other power source.

[0057] Processing device 600 includes one or more communications transceivers 630 and antennas 632 that can provide network connectivity (e.g., cellular network, Wi-Fi, Bluetooth). Processing device 600 may also include various other components, such as a positioning system (e.g., a global positioning satellite transceiver), one or more accelerometers, one or more cameras, an audio interface (e.g., a microphone 634, an audio amplifier, and a speaker and / or audio jack), and a storage device 628. Other configurations may also be used.

[0058] In an example implementation, the mobile operating system, various applications (e.g., the SRNN inference engine), and other modules and services may comprise hardware and / or software embodied in instructions stored in memory 604 and / or storage device 628 and processed by processor unit 602. Memory 604 may be memory of the host device or of an accessory coupled to the host.

[0059] Processing device 600 may include a variety of tangible computer-readable storage media and intangible computer-readable communication signals. Tangible computer-readable storage may be embodied by any available medium that can be accessed by processing device 600, including both volatile and nonvolatile storage media, removable and non-removable storage media. Tangible computer-readable storage media does not include intangible, transitory communication signals, but includes volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Tangible computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other tangible medium that can be used to store the desired information and that can be accessed by processing device 600. In contrast to tangible computer-readable storage media, intangible computer-readable communication signals may embody computer-readable instructions, data structures, program modules or other data residing in a modulated data signal such as a carrier wave or other signal transport mechanism. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, intangible communication signals include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.

[0060] Some embodiments may include an article of manufacture. The article of manufacture may include a tangible storage medium for storing logic. Examples of storage media may include one or more types of volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of logic may include various software elements such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (APIs), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. In one embodiment, for example, the article of manufacture may store executable computer program instructions that, when executed by a computer, cause the computer to perform methods and / or operations according to described embodiments. The executable computer program instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. The executable computer program instructions may be implemented according to a predefined computer language, methodology, or syntax for instructing a computer to perform certain functions. The instructions may be implemented using any suitable high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming language.

[0061] An example method disclosed herein provides for receiving, at a user device (e.g., processing device 102 of FIG. 1 ), a media stream (e.g., media stream 122 of FIG. 1 ) including frames at a first resolution generated by a graphics rendering application, and locally upscaling, at the user device, each received frame of the media stream to a second resolution higher than the first resolution using one or more weighting matrices (e.g., weighting matrix 120 of FIG. 1 ) pre-trained in association with the graphics rendering application. The method further provides for rendering the upscaled frames of the media stream to a display of the user device. This allows for bandwidth reduction without adversely affecting video fidelity or quality from the end user's perspective.

[0062] In another exemplary method following any of the above methods, each of the frames is subjected to one or more weight matrices trained in association with a graphics rendering application. Of The frames are upscaled to the second resolution using a selected weighting matrix, with each upscaling of the frames resulting in the frame being supplemented with additional detail derived based on the associated selected weighting matrix. The use of scene-specific or application-specific weighting matrices utilizes AI to add pixels and detail to each image based on a matrix trained specifically for the media title to better represent the overall look and feel of each frame.

[0063] Yet another exemplary method of any of the above methods further provides for transmitting a request for a selected weight matrix from a user device. The request identifies a matrix identifier stored in a remote database in association with a media title of a graphics rendering application. In response to transmitting the request, the weight matrix is ​​received and stored locally on the user device. The method further provides for dynamically loading the selected weight matrix into a super-resolution neural network (SRNN) inference engine (e.g., SRNN inference engine 112 of FIG. 1 ) when the rendering operation reaches a predefined checkpoint in the media stream. This dynamic retrieval and loading of the weight matrix reduces the amount of memory resources on the user device that are sufficient to implement the above solution.

[0064] In yet another exemplary method of any of the above methods, a plurality of weight matrices are pre-trained in association with a graphics rendering application, and the method includes: Of It is further provided to load a different one of the above into the SRNN inference engine.

[0065] In yet another exemplary method of any of the above methods, multiple weighting matrices are pre-trained in association with a graphics rendering application, each of the multiple weighting matrices being trained using frames sampled from an associated one of multiple different distinct frame segments of the media stream. For example, scene-specific weighting matrices may be generated to enable resolution and fidelity enhancements specifically tailored to the characteristics and tonal profiles unique to different scenes.

[0066] In another example of any of the above methods, each of the weighting matrices is used to dynamically upscale an associated one of a plurality of different distinct frame segments during a media streaming operation.

[0067] In yet another exemplary method of any of the above methods, the user device is a gaming console, and the one or more weight matrices are stored on the gaming console for dynamically loading into a super-resolution neural network (SRNN) inference engine at selected times associated with different checkpoints in the media stream.

[0068] In yet another exemplary method of any of the above methods, the user device is a mobile device, and the method further includes receiving and storing metadata for a graphics-rendering application (e.g., metadata 320 of FIG. 3 ). The metadata specifies matrix identification information that uniquely identifies a plurality of different weight matrices pre-trained for the graphics-rendering application, and further specifies checkpoint identifiers that each identify a checkpoint location within the media stream at which an associated one of the plurality of different weight matrices should be loaded into a super-resolution neural network (SRNN) inference engine. This metadata can provide the user device with sufficient information to identify the trained weight matrices associated with different scenes within the media stream and dynamically load those weight matrices at the appropriate time.

[0069] An example device disclosed herein includes a super-resolution neural network (SRNN) inference engine (e.g., element 112 of FIG. 1 ) stored in a memory of a user device (e.g., 102 of FIG. 1 ). The SRNN inference engine is configured to receive a media stream (e.g., low-resolution media stream 114 of FIG. 1 ) including frames at a first resolution generated by a graphics rendering application and to locally upscale each of a plurality of frames received in the media stream from the first resolution to a second resolution higher than the first resolution using one or more weighting matrices (e.g., weighting matrix 120 of FIG. 1 ) pre-trained in association with the graphics rendering application. The system further includes a media viewing application executing locally on the user device to present the upscaled frames of the media stream on a display of the user device (e.g., in FIG. 1 , user device 102 presents high-resolution media stream 122).

[0070] In another exemplary system of any of the above systems, upscaling each frame of the plurality of frames received in the media stream includes supplementing each of the frames with additional detail derived based on an associated one of the one or more weighting matrices.

[0071] In yet another exemplary system of any of the above systems, the user device further comprises an application programming interface (API) configured to send a request for a selected weight matrix during presentation of the media stream, the request identifying a matrix identifier stored in the remote database in association with a media title in a graphics rendering application, and in response to sending the request, the requested weight matrix is ​​received and stored locally on the user device, and the user device is configured to dynamically load the selected weight matrix into the SRNN inference engine upon reaching a predefined checkpoint during presentation of the media stream.

[0072] In yet another exemplary system of any of the above systems, multiple weight matrices are pre-trained in association with a graphics rendering application, and the SRNN inference engine is dynamically loaded with a different one of the multiple weight matrices upon reaching each of multiple predefined checkpoints during presentation of the media stream.

[0073] In yet another exemplary system of any of the above systems, a plurality of weight matrices are pre-trained in association with a graphics rendering application, and each of the plurality of weight matrices is trained using frames sampled from an associated one of a plurality of different distinct frame segments of the media stream.

[0074] In yet another exemplary system of any of the above systems, each of a plurality of weighting matrices is used to dynamically upscale an associated one of a plurality of different distinct frame segments during a media streaming operation.

[0075] In another exemplary system of any of the above systems, the user device is a gaming console, and one or more weight matrices are co-stored on the gaming console for dynamic loading into the SRNN inference engine in association with different checkpoints in the media stream.

[0076] An example computer-readable storage medium disclosed herein encodes computer-executable instructions for performing a computer process that includes receiving, at a user device (e.g., processing device 102 of FIG. 1 ), a media stream (e.g., low-resolution media stream 114 of FIG. 1 ) including frames at a first resolution generated by a graphics rendering application (e.g., graphics rendering application 210 of FIG. 2 ); locally upscaling, at the user device, each received frame of the media stream to a second resolution higher than the first resolution using one or more weighting matrices (e.g., weighting matrix 120 of FIG. 1 ) pre-trained in association with the graphics rendering application; and rendering the upscaled frames of the media stream (e.g., high-resolution media stream 122 of FIG. 1 ) on a display of the user device.

[0077] An exemplary computer-readable storage medium of any of the above computer-readable storage media provides for upscaling each of the frames to a second resolution using a selected weighting matrix from one or more weighting matrices trained in association with a graphics rendering application, wherein upscaling each of the frames includes supplementing the frame with additional detail derived based on the associated selected weighting matrix.

[0078] In another exemplary computer-readable storage medium of any of the above computer-readable storage media, the computer process further includes sending a request for the selected weight matrix from the user device. The request identifies a matrix identifier stored in the remote database in association with a media title of the graphics rendering application. In response to sending the request, the selected weight matrix is ​​received and stored locally on the user device. The computer process is further provided for dynamically loading the selected weight matrix into a super-resolution neural network (SRNN) inference engine (e.g., SRNN inference engine 112 of FIG. 1 ) when the rendering operation reaches a predefined checkpoint in the media stream.

[0079] In yet another exemplary computer-readable storage medium of any of the above computer-readable storage media, the plurality of weight matrices are pre-trained in association with a graphics rendering application, and the computer process further includes loading a different one of the plurality of weight matrices into the SRNN inference engine upon reaching each of a plurality of pre-defined checkpoints during presentation of the media stream.

[0080] An example system disclosed herein includes means for receiving, at a user device, a media stream including frames at a first resolution generated by a graphics rendering application, and means for locally upscaling, at the user device, each received frame of the media stream to a second resolution higher than the first resolution using one or more weighting matrices pre-trained in association with the graphics rendering application. The system further provides means for rendering the upscaled frames of the media stream to a display of the user device.

[0081] The above specification, examples, and data provide a complete description of the structure and use of exemplary implementations. Since many implementations can be made without departing from the spirit and scope of the claimed invention, the following appended claims define the invention. Furthermore, structural features of different examples may be combined in yet other implementations without departing from the recited claims.

Claims

1. receiving, at a user device, a media stream including frames at a first resolution generated by a graphics rendering application; locally upscaling, at the user device, each received frame of the media stream to a second resolution higher than the first resolution using one or more weighting matrices pre-trained in association with the graphics rendering application, each of the frames being upscaled using a selected weighting matrix from the one or more weighting matrices associated with a frame segment to which the frame belongs; rendering the upscaled frames of the media stream on a display of the user device; and A method having the following.

2. The upscaling of each of the frames results in supplementing that frame with additional detail derived based on the associated selected weight matrix. The method of claim 1.

3. transmitting, from the user device, a request for the selected weight matrix, the request specifying a matrix identifier stored in a remote database in association with a media title of the graphics rendering application, the media title being a particular frame segment including a plurality of frames; receiving and locally storing the selection weight matrix at the user device in response to transmitting the request; dynamically loading the selected weight matrix into a Super Resolution Neural Network (SRNN) inference engine when the rendering operation reaches a predefined checkpoint in the media stream corresponding to the media title; The method of claim 1 further comprising:

4. A plurality of weight matrices are pre-trained in association with the graphics rendering application, the method comprising: and loading a different one of the plurality of weight matrices into a super-resolution neural network (SRNN) inference engine as the selected weight matrix upon reaching each of a plurality of predefined checkpoints during presentation of the media stream, each of the plurality of predefined checkpoints corresponding to a different one of a plurality of frame segments. The method of claim 1.

5. a plurality of weight matrices are pre-trained in association with the graphics rendering application, each of the plurality of weight matrices being trained with frames sampled from an associated one of a plurality of different distinct frame segments of the media stream; The method of claim 1.

6. the user device is a gaming console, and the one or more weight matrices are stored on the gaming console for dynamic loading into a super-resolution neural network (SRNN) inference engine at selected times in association with different checkpoints within the media stream; The method of claim 1.

7. the user device is a mobile device, the method further comprising receiving and storing metadata of the graphics rendering application; The metadata includes: identifying matrix identification information that uniquely identifies a plurality of different weight matrices pre-trained for the graphics rendering application; and further specifying checkpoint identifiers, each of which identifies a checkpoint location within the media stream at which an associated one of the plurality of different weighting matrices should be loaded as the selected weighting matrix into a super-resolution neural network (SRNN) inference engine. The method of claim 1.

8. It has a super-resolution neural network (SRNN) inference engine and a media viewing application, The SRNN inference engine comprises: stored in a memory of the user device; a media stream including frames at a first resolution generated by a graphics rendering application, and configured to locally upscale each of a plurality of frames received in the media stream to a second resolution higher than the first resolution using one or more weighting matrices pre-trained in association with the graphics rendering application, each of the frames being upscaled using a selected weighting matrix from the one or more weighting matrices associated with a frame segment to which the frame belongs; The media viewing application a processor, executed locally on the user device, for presenting the upscaled frames of the media stream on a display of the user device; system.

9. upscaling each frame of the plurality of frames received in the media stream includes supplementing each of the frames with additional detail derived based on the selected weight matrix, the selected weight matrix being an associated one of the one or more weight matrices. The system of claim 8.

10. The user device further comprises an application programming interface (API), the API comprising: transmitting a request for the selected weight matrix during presentation of the media stream, the request identifying a matrix identifier stored in a remote database associated with a media title of the graphics rendering application, the media title being a particular frame segment including a plurality of frames; receiving and storing the requested selection weight matrix locally on the user device in response to transmitting the request; It is configured as the user device is configured to dynamically load the selection weight matrix into the SRNN inference engine upon reaching a predefined checkpoint corresponding to the media title during presentation of the media stream. The system of claim 8.

11. a plurality of weight matrices are pre-trained in association with the graphics rendering application, and the SRNN inference engine is dynamically loaded with a different one of the plurality of weight matrices as the selected weight matrix upon reaching each of a plurality of pre-defined checkpoints during presentation of the media stream; The system of claim 8.

12. a plurality of weight matrices are pre-trained in association with the graphics rendering application, each of the plurality of weight matrices being trained with frames sampled from an associated one of a plurality of different distinct frame segments of the media stream; The system of claim 8.

13. the user device is a gaming console, and the one or more weight matrices are stored on the gaming console for dynamic loading into the SRNN inference engine in relation to different checkpoints within the media stream. The system of claim 8.

Images