Frame stacking for coding artifacts

By generating stacks for frame difference analysis in the game streaming system, identifying and highlighting coding artifacts, the problem of difficult artifacts in frame encoding in the prior art is solved, and system performance and user experience are improved.

CN113597765BActive Publication Date: 2024-11-01GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080021766.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-12-13
Filing Date
2020-03-17
Publication Date
2024-11-01
Estimated Expiration
2040-03-17

AI Technical Summary

Technical Problem

Existing game streaming systems are difficult to effectively identify and reduce encoding artifacts during frame encoding, especially because the streaming frame has a large number of features, which makes encoding artifacts visually difficult to recognize.

Method used

By analyzing the differences between the original frame and the encoded frame, a stack is generated to indicate the encoded artifacts, and the encoded artifacts are identified and highlighted using the stack generator, allowing developers or users to adjust the encoding parameters to reduce artifacts. .

Benefits of technology

The encoding artifacts are quickly and easily identified and adjusted, which improves the performance and user experience of the game streaming system and improves the efficiency of regulating encoding parameters.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113597765B_ABST
    Figure CN113597765B_ABST
Patent Text Reader

Abstract

A game streaming system encodes an original frame (120) to generate an encoded frame (122) of a game stream (115), the game stream (115) comprising a plurality of frames for display. The system identifies a first encoding artifact based on a first difference between the encoded frame and the original frame, and applies an overlay (125) to the encoded frame. The overlay includes a first visual indicator indicating the first encoding artifact.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Video streaming systems transmit video frames from a server to a client device via a network, utilizing the server's large storage space and computing power to provide video content. For example, a game streaming system streams video frames representing a game environment, while allowing a user to interact with the game environment via a game controller or other input devices. In order to ensure the fast and effective transmission of video frames, the game streaming system can transmit frames in a coded format by encoding frames at the server before transmission. However, encoding frames can introduce coding artifacts, such as color changes, feature blur, etc. Some game streaming systems solve these problems by adjusting the parameters of the coding process, such as by encoding some parts of the frame with less compression compared to other parts of the frame. However, the efficacy of these improved techniques often depends on identifying the specific features that are most affected by coding in the frames of the streaming, and it is difficult to identify these features using conventional techniques. Summary of the invention

[0002] The proposed solution particularly relates to a method comprising encoding an original frame to generate a coded frame of a game stream, the game stream comprising a plurality of frames for display; identifying a first coding artifact based on a first difference between the coded frame and the original frame; and applying an overlay to the coded frame, the overlay comprising a first visual indicator indicating the first coding artifact.

[0003] In an exemplary embodiment, the first difference may include a difference in at least one characteristic of one or more pixels between the encoded frame and the original frame, such as a color difference, a texture difference, and / or a blur difference between the encoded frame and the original frame. In this regard, identifying the first difference may include identifying a difference in pixel changes for at least a portion of the encoded frame and a corresponding portion of the original frame.

[0004] In an exemplary embodiment, including the first visual indicator with the overlay may be responsive to the first difference exceeding a threshold. For example, the corresponding threshold may involve a pixel difference value, i.e., when the pixel value difference between the pixels of the encoded frame and the original frame exceeds a predetermined threshold, the first visual indicator is included with the overlay.

[0005] For example, the first visual indicator may be based on the magnitude and / or type of the first difference. In an exemplary embodiment, the method may include determining, for a first coding artifact, a type of the first difference from at least two different types of differences. Depending on the determined type, the visual indicator may therefore differ so as to allow a visual indication of the type of the coding artifact by the stack. For example, the method may include identifying a second coding artifact based on a second difference between the encoded frame and the original frame, wherein the stack includes a second visual indicator indicating the second coding artifact, the second visual indicator being different from the first visual indicator. The first and second visual indicators may differ, for example, in at least one of size, shape, and color. The difference between the first and second visual indicators may thereby indicate a difference between the first and second coding artifacts in type, magnitude, or a combination thereof.

[0006] In an exemplary embodiment, the first visual indicator may be positioned at or above at least a portion of the first encoding artifact, or positioned at or above the entire first encoding artifact. Accordingly, when the stack and the encoded frame are displayed together, the first visual indicator visibly highlights the corresponding encoding artifact.

[0007] In general, the first visual indicator may include, for example, color and / or texture.For example, all pixels within a specified pixel radius of the pixel corresponding to the artifact may be inverted.

[0008] In an exemplary embodiment, an overlay generator of a server providing a game stream may be used, wherein the overlay generator is configured to generate an overlay to provide a first visual indicator.

[0009] Based on the first visual indicator, at least one encoding parameter used to encode the original frame may be adjusted. This may implicitly mean that at least two different adjustable encoding parameters were used to encode the original frame, and one or more of the encoding parameters were adjusted automatically or by a user based on the first visual indicator.

[0010] The proposed solution also relates to a method comprising identifying a coding artifact based on a difference between a pixel of a coded frame and a corresponding pixel in an original frame, the coded frame being provided via a game stream; applying an overlay to the coded frame, the overlay indicating the coding artifact; and displaying the coded frame with the overlay.

[0011] In addition, the proposed solution also relates to a non-transitory computer-readable medium implementing an executable instruction set. The executable instruction set is provided to manipulate at least one processor to encode an original frame to generate an encoded frame of a game stream, the game stream comprising a plurality of frames for display; identify a first encoding artifact based on a first difference between the encoded frame and the original frame; and apply an overlay to the encoded frame, the overlay comprising a first visual indicator indicating the first encoding artifact. In particular, the proposed medium can implement the executable instruction set to manipulate at least one processor to implement the proposed method.

[0012] Another aspect of the proposed solution relates to a real-time video streaming system, comprising a server for generating coded frames for transmission to at least one client device based on a game state of a video game program executed at the server. The server can be configured to: encode an original frame to generate an coded frame of a game stream of the video game program, the game stream comprising a plurality of frames for display; identify a first coding artifact based on a first difference between the coded frame and the original frame; and apply an overlay to the coded frame, the overlay comprising a first visual indicator indicating the first coding artifact. In particular, the proposed real-time video streaming system may include a server configured to implement the proposed method.

[0013] The present disclosure relates in particular to a technique for generating an overlay that identifies coding artifacts in coded frames to be streamed by a real-time video streaming system, such as a game streaming system. A server of the game streaming system generates each frame, for example, based on a game state of a video game program executed at the server. The server generates frames in a "raw" or uncoded format, encoding the frames for transmission to a client device via a network. For at least one coded frame, the server identifies coding artifacts by identifying differences between the coded frame and the corresponding original frame, such as differences in color, texture, blur or other characteristics of one or more pixels of the coded frame and the original frame, for example. The server generates an overlay with a visual indicator to highlight the identified coding artifacts and displays the overlay with the coded frame. A user (e.g., a developer or a user of a client device) is therefore able to quickly and easily identify coding artifacts, and is therefore able to quickly and easily adjust aspects of the game streaming system that resolve such artifacts, thereby improving the performance of the game streaming system.

[0014] To illustrate, in some embodiments, the encoder uses one or more parameters to manage the encoding process, such as quantization parameters, compression parameters, etc. Adjustment of one or more of these parameters can reduce encoding artifacts. By identifying specific encoding artifacts of one or more encoded frames to be streamed, the effectiveness of these adjustments in reducing encoding artifacts can be enhanced. For example, the encoder can implement variance-based adaptive quantization (VBAQ), and the developer can adjust the VBAQ parameters based on specific encoding artifacts identified in one or more encoded frames to be streamed. However, since the frames of the streams often have a large number of features and the encoding artifacts of the frames may be small relative to the overall size of the frames of the streams, it may be difficult to visually identify the encoding artifacts. By using an overlay to visually highlight the encoding artifacts, the game streaming system allows developers or other users of the system to quickly identify the artifacts and make corresponding adjustments to the encoding parameters, thereby improving the development and performance of the game streaming system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By referring to the accompanying drawings, the present disclosure may be better understood and its features and advantages may be more apparent to those skilled in the art. The same reference numerals are used in different drawings to indicate similar or identical items.

[0016] Figure 1 is a block diagram of a game streaming system that generates a frame stack that identifies coding artifacts in encoded frames of a game stream, according to some embodiments.

[0017] Figure 2 is a diagram illustrating the Figure 1 3. An illustration of an example of encoding artifacts generated when encoding a raw frame at a game streaming system.

[0018] Figure 3 is a diagram illustrating a method according to some embodiments Figure 1 Schematic diagram of an example of a stackup.

[0019] Figure 4 is a diagram illustrating a method according to some embodiments Figure 1 A diagram of an example of a stack of layers for identifying coding artifacts in a frame.

[0020] Figure 5 is a diagram illustrating a method according to some embodiments Figure 1 A diagram of an example of a stack of layers for identifying multiple coding artifacts in a coded frame.

[0021] Figure 6 is a diagram illustrating a method according to some embodiments Figure 1 Block diagram of a stackup generator that generates a stackup.

[0022] Figure 7 is a flow chart illustrating a method of generating an overlay that identifies coding artifacts in encoded frames of a game stream in accordance with some embodiments. DETAILED DESCRIPTION

[0023] Figure 1 A game streaming system 100 is illustrated that is generally configured to stream rendered game content from a server 102 to a client device 104. In the depicted example, the server 102 and the client device 104 communicate via a network 110, where the network 110 is a packet-switched network or other network generally configured to transmit data (including video data) via one or more routers, servers, communication towers (e.g., cellular communication towers), etc., or any combination thereof. Thus, in various embodiments, the network 110 is a wide area network (e.g., the Internet), a local area network, etc., or any combination thereof.

[0024] The server 102 and the client 104 are generally configured to jointly implement a streamed gaming session, wherein interactive gaming content is streamed from the server 102 to the client device 104. Thus, the server 102 may be any type of computer device that implements the functionality further described herein, such as a rack-mounted server, a clustered server (i.e., a server device implemented in a cluster of server devices), a mobile server device, etc., or a combination thereof. The client device 104 may be any computer device capable of displaying video frames to a user and capable of receiving user input, and thus may be a desktop or laptop computer, a digital media player, a game console, a smart phone, a tablet computer, etc.

[0025] To support streaming of game content to client devices during a streamed game session, server 102 includes multiple processing units, such as a central processing unit (CPU) 106 and a graphics processing unit (GPU) 108. CPU 106 is a processing unit that is generally configured to execute a general-purpose set of instructions organized in the form of a computer program to perform tasks on behalf of server 102. Examples of such computer programs include operating systems, virtual machines, data security programs (e.g., data encryption / decryption programs), web pages, database programs, and the like.

[0026] The GPU 108 is a processing unit that is generally configured to perform operations associated with graphics and vector processing based on commands received from the CPU 106. For example, in the course of executing one or more general-purpose programs, the processor 106 generates commands for generating and manipulating a graphics model for display and provides these commands to the GPU 108. In response, the GPU 108 executes these commands by performing one or more corresponding graphics operations, thereby manipulating the graphics model to generate one or more frames for display. In some embodiments, the frames generated by the GPU 108 are rendered frames that are ready for display at a display device.

[0027] To support the reception and display of streaming game content, the client device 104 includes a processing unit such as a CPU and a GPU ( Figure 1 106 and 108 of the server 102). Each of these modules is similar to a corresponding module in the server 102. Thus, for example, the CPU is generally configured to execute general instructions on behalf of the client device 104, such as executing one or more of an operating system, a web browser, etc. It is to be understood that in some embodiments, the CPU and GPU of the client device generally have relatively small computing power relative to the CPU 106 and GPU 108 of the server 102. The game streaming system 100 thus utilizes the computing power of the server 102 to stream game content of higher quality to the client device 104 than the client device 104 alone can generate.

[0028] In operation, the client 104 initiates a game streaming session in response to a user requesting a game session via a computer program (not shown) executed at the client 104 (such as via a web browser, a dedicated game streaming program, a video streaming service program, etc.). In response to the user request, the CPU 112 sends the game session request to the server 102 via the network 110. In response, the server 102 initiates execution of the game program 109. In some embodiments, the server 102 is configured to execute any one of a library of game programs, and the request provided by the client 104 indicates a specific one of the game programs to be executed.

[0029] During execution, the game program 109 maintains a collection of data called a game state, which represents the condition or state of the game. A user of the client 104 uses a game controller or other input device to provide input data to the CPU 112, which communicates the input data to the game program 109 via the network 110. In response to the input data, the game program 109 modifies the game state according to the rules of the particular game implemented by the game program 109.

[0030] Based on the game state and in accordance with the rules of the implemented game, the game program 109 uses the CPU 106 and GPU 108 to generate rendered frames representing visual game information, such as a virtual environment, a game board, a selection screen, a heads-up display (HUD), etc., or any combination thereof. The server 102 provides these rendered frames, designated as streamed frames 115, to the network 110 for communication to the client 104. The client device 104 receives at least a subset of the streamed frames 115 and displays the received frames at the display 117 using the GPU 108. The client device 104 thus displays the visual game information to the user, in response to the user providing further input data using a game controller or other input device, resulting in further changes to the game state saved by the game program 109 and corresponding changes to the frames in the frame stream 115. In this way, the user of the client device 104 interacts with the game program 109 to play the game executed at the server 102.

[0031] In order to save network bandwidth and ensure relatively fast transmission of frames to the client device 104, the server 102 may encode each streaming frame 115 prior to transmission. Figure 1 In the depicted example, the GPU 108 includes an encoder 121, which is generally configured to encode the original frame 120 based on a specified video encoding format, such as H.262 (MPEG-2 Part 2), MPEG-4 Part 2, H.264 (MPEG-4 Part 10), or other video encoding formats to generate an encoded frame 122. The GPU 108 provides the encoded frame 122 to a network interface (not shown), which transmits the encoded frame 122 as part of the streaming frame 115. The client device 104 may include a decoder (not shown) to decode the encoded frame in the streaming frame before displaying the decoded frame to the user.

[0032] In some cases, the encoder process results in an encoded frame 122 that has visible differences from the original frame 120 (or has visible differences when the encoded frame 122 is decoded and displayed). These differences are generally referred to as encoding artifacts, and may include artifacts such as color changes, pixelation, image blur, etc. Coding artifacts may be visually distracting or otherwise adversely affect the user experience of the game streaming system 100. In some cases, encoding artifacts can be reduced or eliminated by adjusting the encoding parameters of the encoder 121, such as parameters associated with variable-based adaptive quantization (VBAQ). These parameter adjustments can be improved by identifying the characteristics of specific encoding artifacts presented in the encoded frame. Therefore, during the development and execution of the game program 109, it is often useful to visually identify encoding artifacts in the encoded frame so that the developer or user can implement corresponding adjustments to the encoding parameters of the encoder 121. However, the encoding artifacts in a given encoded frame may be small relative to the overall size of the encoded frame, so it is difficult for the user or developer to visually identify them. Therefore, to help a user or developer identify encoding artifacts, GPU 108 includes an overlay generator 123 configured to generate an overlay 125 that provides visible indicators of encoding artifacts for encoded frames 122 .

[0033] For example, in some embodiments, the overlay generator 123 is configured to identify differences between features of the raw frame 120 and the encoded frame 122. As used herein, a feature of an image is one or more pixels of the image, and may be a single pixel, a set of pixels corresponding to a particular visual feature such as an edge, a corner, an object, etc. A given feature, such as an individual pixel or a set of pixels, may have different characteristics between the raw frame 120 and the encoded frame 122, such as differences in color, color pattern, pixelation level, color variation of a set of pixels, etc., or any combination thereof. The overlay generator 125 identifies the characteristic differences for each feature, and generates the overlay 125 to have a visual indicator for each identified feature and to indicate any encoding artifacts in the encoded frame 122.

[0034] In some embodiments, the stack generator 123 uses color subtraction to identify coding artifacts. For example, the stack generator 125 can subtract the color value of each pixel of the coded frame 122 from the color value of the corresponding pixel of the original frame 120 (or vice versa) to identify the color difference for each pixel, and generate the stack 125 based on the identified color difference. In other embodiments, the stack generator 123 identifies coding artifacts by identifying the difference in the amount of blur between the set of pixels of the coded frame 122 and the corresponding set of pixels of the original frame 120. In some embodiments, the stack generator 123 determines the amount of blur associated with a given set of pixels based on the change in the values ​​of the pixels in the given set of pixels. The difference in the amount of blur between the corresponding set of pixels of the coded frame 122 and the original frame 120 is therefore based on the difference in the change of the corresponding set of pixels.

[0035] In some embodiments, the size of the overlay 125 corresponds to the size of the coded frame 122, and the GPU 108 positions the visual indicator for the feature so that the indicator is located at or near the corresponding coding artifact in one of the coded frames 122, so that when the overlay and the coded frame 122 are displayed together, the visual indicator visibly highlights the corresponding coding artifact. In some embodiments, the GPU 108 selects a visual indicator for the feature based on one or more identified coding artifacts of the feature. For example, in some embodiments, the size, shape, or color of the visual indicator, or a combination thereof, indicates the type, magnitude, or a combination thereof of the corresponding coding artifact. Thus, for example, in some cases, a particular color of the visual indicator indicates one type of coding artifact, such as a pixel color difference, while a different color of the visual indicator indicates a different type of coding artifact, such as a color or pixel variation difference.

[0036] In some embodiments, GPU 108 composites frame 122 and overlay 125 into a single frame for display on a display device ( Figure 1 Thus, when the composite frame is displayed, the visual indicator of the overlay 125 visibly indicates the encoding artifacts associated with the encoded frame 122, even in those cases where the discontinuity itself is difficult or impossible to see with the naked eye. The developer of the game program 109 can adjust the encoding parameters of the encoder 121 to better address the identified encoding artifacts and improve the overall gaming experience at the client device 104.

[0037] Figure 2An example of encoding artifacts associated with feature 230 of frames 120 and 122 is illustrated in accordance with some embodiments. In the depicted example, feature 230 is illustrated as having no fill in original frame 120, but having a cross-hatched fill at encoded frame 122, indicating that feature 230 has encoding artifacts in encoded frame 122. Feature 230 may thus be any visual feature that may have encoding artifacts, such as an individual pixel, a set of pixels, a visual feature such as a corner of an object or an edge of an object, and so forth.

[0038] Figure 3 An example of an overlay 125 providing a visual indicator for a feature 230 in accordance with some embodiments is illustrated. In the depicted example, the overlay 125 is illustrated with a dashed outline and is depicted as being slightly offset from the frame 122 in order to distinguish the overlay from the frame. However, it is to be understood that the GPU 116 may composite the overlay 125 with the frame 122 for display, such that the frame 122 and the overlay 125 may be effectively displayed together at the display 117. In particular, the indicator 335 is displayed on or near the feature 230 to provide a visual indication of the associated discontinuity.

[0039] In various implementations, the indicator 335 is a color, shape, texture, informational text, icon, or other visual indicator that indicates one or more of the type of discontinuity associated with the feature 230, the magnitude of the discontinuity, a specified characteristic of the discontinuity (e.g., a spatial direction associated with a positional discontinuity), or any combination thereof. For example, in the case of a positional discontinuity, the indicator 335 may be a line or arrow indicating the size and direction of the positional discontinuity.

[0040] In some cases, multiple features of the encoded frame 122 are associated with encoding artifacts. For example, in some cases, some features of the encoded frame 122 have a change in color relative to the original frame 120, while other features have a change in the amount of blur relative to the original frame 120. In some embodiments, the GPU 116 generates the overlay 125 to include a visual indicator for each encoding artifact of the encoded frame 122. Figure 4 An example according to some embodiments is illustrated in . In the depicted example, the original frame 120 includes a plurality of features, each of which is illustrated by a corresponding unfilled circle. The encoded frame 122 includes encoding artifacts associated with each feature, which are illustrated by circles with a gray fill. The overlay 125 includes a visual indicator for each encoding artifact, which is illustrated by a circle with a black fill.

[0041] The type of each visual indicator. Thus, each visual indicator can have different characteristics, such as different size, color, shape, orientation, texture information, icon, etc., or a combination thereof, which visually indicates the type of encoding artifact associated with the corresponding feature. For example, in some embodiments, one visual indicator is red to indicate that the feature is associated with a color change artifact, while another visual indicator has a blue color to indicate the corresponding feature 440.

[0042] In some embodiments, the type of visual indicator or its characteristics (such as the size, color, or shape of the visual indicator) indicates the magnitude of the associated encoding artifact. For example, in some embodiments, a visual indicator of one color indicates one level of change between the encoded frame 122 and the original frame 120, and a visual indicator of a different color indicates a different level of change between the encoded frame 122 and the original frame 120. Figure 4 In the example of FIG. 1 , the stack 125 includes a plurality of visual indicators of encoding artifacts, where each visual indicator may indicate a type, magnitude, and other characteristics of an associated encoding artifact.

[0043] In some embodiments, the overlay generator 123 generates a visual indicator for a given coding artifact only when the frame difference associated with the given coding artifact exceeds a threshold. For example, the overlay generator 123 can identify a coding artifact based on the pixel value difference between the pixels of the coded frame 122 and the original frame 120. In response to the pixel value difference exceeding the threshold, the overlay generator 123 generates a visual indicator for the coding artifact to be displayed via the overlay 125. In response to the pixel value difference being below the threshold, the overlay generator 123 omits the visual indicator for the coding artifact from the overlay 125. The overlay generator 123 thus omits visual indicators for coding artifacts that are unlikely to be seen, which allows developers to focus on more visible coding artifacts. In some embodiments, the overlay generator 123 can change the magnitude of the threshold in response to a specified event, such as a frame associated with a screen change of the game program 109.

[0044] In some embodiments, the color, brightness, or intensity of the visual indicator is based on the magnitude of the difference associated with the corresponding encoding artifact. For example, in some embodiments, the overlay generator 123 generates the overlay 125 as a heat map, where the intensity of each pixel of the visual indicator corresponds to the magnitude of the difference associated with the corresponding encoding artifact.

[0045] In some embodiments, the overlay generator 123 generates the visual indicator to have a color or other characteristic based on the color or other characteristic of the corresponding portion of the coded frame 122. For example, the overlay generator 123 may generate the visual indicator by inverting the pixel values ​​of one or more pixels in the coded frame corresponding to the encoding artifact. In some embodiments, the overlay generator 123 inverts all pixels within a specified pixel radius of the pixel corresponding to the encoding artifact. In still other embodiments, the visual indicator may include texture information that identifies a characteristic of the associated encoding artifact.

[0046] Figure 5 1 is a block diagram of an overlay generator 123 according to some embodiments. In various embodiments, the overlay generator 123 may be a software module executed at one or more of the CPU 106 and the GPU 108 of the server 102. In other embodiments, one or more aspects of the overlay generator 123 are implemented by dedicated hardware of the server 102.

[0047] In the depicted example, the overlay generator 123 includes a color subtraction module 652 and a blur detection module 654. The color subtraction module 652 is configured to perform a color subtraction process based on the original frame 120 and the encoded frame 122. For example, in some embodiments, the color subtraction module 652 subtracts the pixel color value of each pixel of the original frame 120 from the pixel color value of the corresponding pixel of the encoded frame. Based on the color difference of a given pixel, the color subtraction module 652 assigns a color value to the corresponding pixel of the overlay 125. For example, in some embodiments, if the difference associated with the pixel is within a first range, the color subtraction module 652 assigns a first color to the corresponding pixel of the overlay 125, and if the difference associated with the pixel is within a different second range, the color subtraction module 652 assigns a different second color to the corresponding pixel of the overlay 125. The color of the overlay 125 thus represents the color difference between the original frame 120 and the encoded frame 122 for each pixel.

[0048] The blur detection module 654 is configured to determine the difference in blur level between the original frame 120 and the encoded frame 122 for a specified segment of pixels. To illustrate, in some embodiments, the blur detection module 654 applies a window of a specified size (in terms of number of pixels) and shape to the original frame 120 and the encoded frame 122, determines the difference in blur level for the applied window, and then translates the window to another set of pixels. For example, in some embodiments, the blur detection module 654 uses an N×N window, where N is an integer number of pixels. In some embodiments, the blur detection module 654 determines the blur level of the window by determining the pixel variation of the window, such as the difference between the brightest pixel and the darkest pixel in the window. To determine the difference in blur level, the blur detection module 654 determines the blur level of the window by determining a power spectrum of the pixel values ​​of the window, where the blur level is indicated by the spatial frequency of the power spectrum.

[0049] Based on the blur difference for a given window, the color subtraction module 652 assigns a color value to the corresponding pixel of the overlay 125. For example, in some embodiments, if the blur difference associated with the window is within a first range, the color subtraction module 652 assigns a first color to the corresponding pixel of the overlay 125, and if the blur difference associated with the pixel is within a second, different range, the color subtraction module 652 assigns a different second color to the corresponding pixel of the overlay 125. As with the color subtraction process, the color of the overlay 125 thus represents the blur difference between the original frame 120 and the encoded frame 122 for each window.

[0050] Figure 6 A flow diagram of a method 700 of generating an overlay having visual indicators showing encoding artifacts for frames of a game stream according to some embodiments is illustrated. Figure 1 The method 700 is described with reference to an example implementation at the game streaming system 100 of FIG. 702. At block 702, the encoder 121 encodes the original frame 120 to generate the encoded frame 122. At block 704, the overlay generator 123 identifies differences between the encoded frame 122 and the original frame 120, such as differences in color, differences in blur levels, and the like. At block 706, the overlay generator 123 identifies encoding artifacts based on the differences identified at block 704. At block 706, the overlay generator 123 generates an overlay 125 to include a visual indicator of the encoding artifact identified at block 704. At block 708, the GPU 108 composites the overlay 125 with the encoded frame 122 and sends the composite frame to a display device for display, thereby allowing a user to quickly and visually identify encoding artifacts in the encoded frame and make adjustments to the encoding parameters of the encoder 121.

[0051] In some embodiments, some aspects of the technology described above can be implemented by one or more processors of a processing system that executes software. The software includes one or more executable instruction sets stored on a non-transient computer-readable storage medium or otherwise tangibly implemented on a non-transient computer-readable storage medium. The software may include instructions and certain data, which manipulate the one or more processors to perform one or more aspects of the technology described above when executed by the one or more processors. For example, the non-transient computer-readable storage medium may include a disk or optical disk storage device, a solid-state storage device such as a flash memory, a cache, a random access memory (RAM), or one or more other non-volatile memory devices, etc. The executable instructions stored on the non-transient computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats interpreted by one or more processors or otherwise executable by the one or more processors.

[0052] Computer-readable storage media may include any storage medium or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, tapes, or magnetic hard disks), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical systems (MEMES)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard disk), removably attached to a computing system (e.g., an optical disk or a flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired or wireless network (e.g., network accessible storage (NAS)).

[0053] It should be noted that not all activities or elements described above in the general description are required, a portion of a specific activity or device may not be required, and one or more additional activities may be performed or one or more additional elements may be included in addition to those described. Further, the order in which the activities are listed is not necessarily the order in which they are performed. Moreover, the concepts have been described with reference to specific embodiments. However, those skilled in the art will appreciate that various modifications and variations may be made without departing from the scope of the present disclosure as given in the appended claims. Therefore, the description and drawings are to be understood in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0054] The above has described benefits, other advantages and solutions to problems with respect to specific embodiments. However, these benefits, advantages, solutions to problems and any (multiple) features that may cause any benefit, advantage or solution to the problem to occur or become more prominent are not understood to be key, required or necessary features in any or all claims. In addition, the specific embodiments disclosed above are only illustrative, because the disclosed subject matter can be modified and practiced in different but equivalent ways that are clearly understood by those skilled in the art who have benefited from the teachings of this article. Except as described in the attached claims, it is not expected that the details of the construction or design shown in this article are limited. Therefore, the specific embodiments disclosed above can obviously be changed or modified, and all such changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection given in the attached claims is sought herein.

Claims

1. A method for indicating coding artifacts, comprising: encoding the original frame [120] to generate an encoded frame [122] of a game stream, the game stream comprising a plurality of frames for display [115]; identifying a first encoding artifact based on a first difference between the encoded frame and the original frame exceeding a threshold; synthesizing the coded frame with an overlay [125] to generate a synthesized frame, the overlay comprising a first visual indicator [335] indicating the first coding artifact and having characteristics that visually indicate a type of the first coding artifact; as well as The synthesized frame is sent to a display device for display, wherein the first visual indicator is positioned at or over at least a portion of the first encoding artifact, or at or over the entire first encoding artifact.

2. The method according to claim 1, wherein: The first difference comprises a difference in at least one characteristic of one or more pixels between the encoded frame and the original frame.

3. The method according to claim 2, wherein: The first difference comprises a color difference between the encoded frame and the original frame.

4. The method according to claim 2, wherein: The first difference comprises a texture difference between the encoded frame and the original frame.

5. The method according to claim 2, wherein: The first difference comprises a blurred difference between the encoded frame and the original frame.

6. The method according to claim 5, wherein: Identifying the first difference includes identifying a difference in pixel changes between at least a portion of the encoded frame and a corresponding portion of the original frame.

7. The method according to claim 1, wherein: The first visual indicator is based on a magnitude and / or type of the first difference.

8. The method according to claim 7, further comprising: A type of the first difference among at least two different types of differences of the first encoding artifacts is determined.

9. The method according to claim 1, wherein: The first visual indicator comprises a color.

10. The method according to claim 1, wherein: The first visual indicator comprises a texture.

11. The method according to claim 1, further comprising: identifying a second encoding artifact based on a second difference between the encoded frame and the original frame; and Wherein the stack comprises a second visual indicator indicative of the second encoding artifact, the second visual indicator being different from the first visual indicator. 12 . The method of claim 11 , wherein the first visual indicator and the second visual indicator differ in at least one of size, shape, and color.

13. The method according to claim 1, further comprising: Using an overlay generator [123] of a server [102] providing the game stream, wherein the overlay generator is configured to generate the overlay to provide the first visual indicator.

14. The method according to claim 1, further comprising: At least one encoding parameter used to encode the original frame is adjusted based on the first visual indicator.

15. The method according to claim 14, wherein: At least two different encoding parameters are used to encode the original frame and are adjustable based on the first visual indicator.

16. The method according to claim 1, further comprising: The encoded frame is displayed for provision with the overlay via a game stream.

17. A method for indicating coding artifacts, comprising: identifying the coding artifact based on a difference between a pixel of a coded frame [122] and a corresponding pixel of an original frame [120], the coded frame being provided via a game stream [115]; compositing the coded frame with an overlay [125] to generate a composite frame, the overlay indicating the coding artifact and having characteristics visually indicating a type of the coding artifact; as well as The synthesized frame is displayed with the overlay positioned at or over at least a portion of the coding artifact, or positioned at or over the entire coding artifact.

18. The method according to claim 17, wherein: The coding artifacts include at least one of color differences, texture differences, and blur differences.

19. A non-transitory computer readable medium implementing an executable instruction set for manipulating at least one processor to: encoding the original frame [120] to generate an encoded frame [122] of a game stream, the game stream comprising a plurality of frames for display [115]; identifying a first encoding artifact based on a first difference between the encoded frame and the original frame exceeding a threshold; as well as synthesizing the coded frame with an overlay [125] to generate a synthesized frame, the overlay comprising a first visual indicator [335] indicating the first coding artifact and having characteristics that visually indicate a type of the first coding artifact; as well as The synthesized frame is sent to a display device for display, wherein the first visual indicator is positioned at or over at least a portion of the first encoding artifact, or at or over the entire first encoding artifact.

20. The computer-readable medium of claim 19, wherein: The first difference comprises a difference in at least one characteristic of one or more pixels between the encoded frame and the original frame.

21. The computer-readable medium of claim 19, wherein: The first difference comprises a color difference between the encoded frame and the original frame.

22. The computer-readable medium of claim 19, wherein: The first difference comprises a texture difference between the encoded frame and the original frame.

23. The computer-readable medium of claim 19, wherein: The first difference comprises a blurred difference between the encoded frame and the original frame.

24. The computer-readable medium of claim 23, wherein: Identifying the first difference includes identifying a difference in pixel changes between at least a portion of the encoded frame and a corresponding portion of the original frame.

25. The computer-readable medium of claim 19, wherein the first visual indicator is based on a magnitude of the first difference.

26. The computer-readable medium of claim 19, wherein the first visual indicator comprises a color.

27. The computer-readable medium of claim 26, wherein the first visual indicator comprises, for example, a texture.

28. The computer-readable medium of claim 19, wherein the executable instructions direct the processor to: identifying a second encoding artifact based on a second difference between the encoded frame and the original frame; and in, The overlay includes a second visual indicator indicative of the second encoding artifact, the second visual indicator being different from the first visual indicator.

29. A real-time video streaming system [100] comprising a server [102] for generating encoded frames for transmission to at least one client device [104] based on a game state of a video game program [109] executed at the server, the server being configured to: encoding raw frames [120] to generate encoded frames [122] of a game stream [115] of the video game program, the game stream comprising a plurality of frames for display; identifying a first encoding artifact based on a first difference between the encoded frame and the original frame exceeding a threshold; synthesizing the coded frame with an overlay [125] to generate a synthesized frame, the overlay comprising a first visual indicator [335] indicating the first coding artifact and having characteristics that visually indicate a type of the first coding artifact; as well as The synthesized frame is sent to a display device for display, wherein the first visual indicator is positioned at or over at least a portion of the first encoding artifact, or at or over the entire first encoding artifact.