Frame overlay for inter-frame disparity in game streams

Abstract

To provide a method and program capable of generating an overlay which identifies discontinuities and other changes in features between streamed frames in a game streaming system.SOLUTION: A method 700 in the game streaming system receives 702 a stream of frames representing a game content for display, from a server via a network, analyzes the received frames to identify 704, 706 discontinuities and other changes in features of the plurality of frames, such as a particular feature changing one or more of position, size, texture, or other visual feature between successively received frames, generates the overlay to indicate the identified features, and displays 708 the overlay with at least one of the received frames.SELECTED DRAWING: Figure 7

Description

[Background technology]

[0001] background An interactive video streaming system streams video frames from a server to a client device over a network while allowing a user to interact with the virtual environment represented by the streamed video. For example, a game streaming system streams video frames representing a game environment while allowing a user to interact with the game environment via a game controller or other input device. The user manipulates the input device to interact with the game environment in a desired manner, and the client device sends signaling representing the interaction to the server over the network. In response, the server modifies the game state corresponding to the game environment and transmits video frames representing the game environment over the network to the client device based on the new game state.

[0002] Because user input and the resulting video frames are transmitted repeatedly, network performance can have a significant impact on the user's experience. For example, network transmission errors can cause transmission delays and dropped or lost video frames in the transmitted stream. These errors can result in video stuttering, a perceived unresponsiveness of the game, and other problems that negatively impact the user experience. Some game streaming systems address these issues by implementing improvement techniques such as frame interpolation and motion smoothing. However, the effectiveness of these improvement techniques often depends on identifying the specific characteristics of the streamed frames that are most affected by transmission errors, and identifying these characteristics is difficult using conventional techniques. Summary of the Invention

[0003] overview The proposed solution particularly relates to a method, the method comprising: receiving a game stream including a plurality of frames for display, the plurality of frames including a first frame including a first graphical feature and a second frame including the first graphical feature; identifying a first difference between a first display characteristic of the first graphical feature associated with the first frame and a second display characteristic of the first graphical feature associated with the second frame; and applying an overlay to the first frame, the overlay including a first visual indicator indicating the first difference.

[0004] In an exemplary embodiment, the first display feature is: - may include a first position of the first graphical feature, and the second display feature includes a second position of the first graphical feature, the second position being different from the first position; and / or - may include a first texture of the first graphical feature, and the second display feature includes a second texture of the first graphical feature, the second texture being different from the first texture; and / or The display characteristic may include a first size of the first graphical feature, and the second display characteristic includes a second size of the first graphical feature, the second size being different from the first size.

[0005] Additionally or alternatively, the method may further comprise including a first overlay with a first visual indicator in response to the first difference exceeding a threshold value. For example, in some embodiments, a first graphical feature may have a similar pattern in displayed pixel characteristics, such as, for example, a similar pattern of pixel color, intensity, pixel activity, or other characteristics, or any combination thereof, in the first display feature. In some embodiments, pairs of first and second display features between the first and second frames may be assigned a correspondence score based on the similarity of the patterns of corresponding pixel characteristics, and features having a correspondence score equal to or greater than a (correspondence) threshold may be identified as representing the same graphical feature. At the same time, features that do not correspond to each other (i.e., features having a correspondence score below the (correspondence) threshold) may be identified as differing by a first difference and exceeding the difference threshold.

[0006] In an example embodiment, the first visual indicator may be based on the magnitude of the first difference, such that the color and / or texture of the first visual indicator may vary depending on the magnitude of the first difference, for example.

[0007] In one example embodiment, the characteristics of the visual indicator correspond to the type of discontinuity associated with the first graphical feature.

[0008] Generally, the method may further comprise applying an enhancement technique to ameliorate the effect of discontinuities between the streamed frames based on the visual indicator. For example, the enhancement technique may include motion smoothing and / or frame interpolation for a gaming stream that blurs the first difference. In some embodiments, the degree and / or type of blurring may be adjusted by: based on a visual indicator, for example based on the type of visual indicator, which may depend on whether the first difference relates to a difference in position, texture or size, and / or may be determined based on the size of the visual indicator.

[0009] For example, motion smoothing can be improved by identifying how one or more graphical features have changed between frames, such as the direction the feature has moved, the distance the feature has moved, and differences in the texture of the feature, and such (first) differences are visualized by a visual indicator.

[0010] In an example embodiment, the method may further comprise identifying a second difference between a third display characteristic of the second graphical feature associated with the first frame and a fourth display characteristic of the second graphical feature associated with the second frame, wherein the overlay includes a second visual indicator indicating the second difference, the second visual indicator being different from the first visual indicator.

[0011] The proposed solution further relates to a computer readable medium tangibly embodying a set of instructions which, when executed by a processor, cause the processor to: receiving a game stream including a plurality of frames for display, the plurality of frames including a first frame including a first graphical feature and a second frame including the first graphical feature; - identifying a first difference between a first display characteristic of a first graphical feature associated with a first frame and a second display characteristic of a first graphical feature associated with a second frame; and - operating to apply an overlay to the first frame, the overlay including a first visual indicator indicating the first difference;

[0012] Furthermore, the proposed solution relates to a game streaming system comprising a server for generating a stream of frames for a game stream and a client device for receiving the stream of frames, the client device comprising: receiving a game stream including a plurality of frames for display, the plurality of frames including a first frame including a first graphical feature and a second frame including the first graphical feature; - identifying a first difference between a first display characteristic of a first graphical feature associated with a first frame and a second display characteristic of a first graphical feature associated with a second frame; and configured to apply an overlay to the first frame, the overlay including a first visual indicator that indicates the first difference.

[0013] In particular, the present disclosure relates to techniques for generating overlays that identify discontinuities and other changes in characteristics between streamed frames in a game streaming system. A client device of the game streaming system receives a stream of frames representing game content for display from a server over a network. The client device analyzes the received frames to identify discontinuities and other changes in characteristics of the frames, such as particular features that change one or more of their position, size, texture, or other visual characteristics between successively received frames. The client device generates an overlay to indicate the identified features and displays the overlay with at least one of the received frames. A user of the client device can quickly and easily identify features that are discontinuities between frames and can therefore quickly and easily adjust aspects of the game streaming system that correspond to such discontinuities to improve performance of the game streaming system.

[0014] To illustrate, in some embodiments, a game streaming system can implement frame interpolation, motion smoothing, and similar techniques that hide or otherwise ameliorate the effects of discontinuities between streamed frames, which may result from, for example, network transmission errors. The effectiveness of these techniques can be increased by identifying particular features of the stream that are discontinuous between frames. For example, motion smoothing can be made more effective by identifying particular objects, textures, or other features that experience discontinuities at the client device. However, because streamed frames often have numerous features, and the discontinuities in the features may be small relative to the overall size of the streamed frames, visually identifying the particular features that are discontinuous between frames can be difficult. By employing overlays to visually highlight the discontinuous features, the game streaming system can enable system developers or other users to quickly identify the discontinuous features and make corresponding adjustments to motion smoothing or other improvement techniques to improve the development and performance of the game streaming system.

[0015] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by reference to the accompanying drawings, in which: The use of the same reference symbols in the various drawings indicates similar or identical items. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a block diagram of a game streaming system that generates frame overlays that identify discontinuities and other changes in characteristics between frames in a game stream, according to some embodiments. [Figure 2] 2A-2C illustrate examples of discontinuities in features between frames of a game stream in the game streaming system of FIG. 1, according to some embodiments. [Figure 3]2 illustrates an example overlay of FIG. 1 identifying feature discontinuities in a frame, according to some embodiments. [Figure 4] 2 illustrates an example overlay of FIG. 1 identifying discontinuities in multiple features in a frame, according to some embodiments. [Figure 5] 2 illustrates another example of the overlay of FIG. 1 that identifies discontinuities in features, according to some embodiments. [Figure 6] FIG. 2 is a block diagram illustrating an overlay generator that generates the overlay of FIG. 1 in accordance with some embodiments. [Figure 7] FIG. 1 is a flow diagram illustrating a method for generating an overlay that identifies discontinuities in features between frames of a game stream, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0017] Detailed Description 1 illustrates a game streaming system 100 generally configured to stream rendered game content from a server 102 to a client device 104. In the illustrated example, the server 102 and the client device 104 communicate over a network 110, which is generally a packet-switched or other network configured to transfer data, including video data, through one or more routers, servers, and communication towers (e.g., cellular communication towers), or the like, or any combination thereof. Thus, in various embodiments, the network 110 is a wide area network (e.g., the Internet), a local area network, or the like, or any combination thereof.

[0018] The server 102 and clients 104 are typically configured to collectively implement a streamed game session, with interactive game content being streamed from the server 102 to the client devices 104. As such, the server 102 may be any type of computing device that implements functionality described further herein, such as a rack-mounted server, a cluster server (i.e., a server device implemented in a cluster of server devices), a mobile server device, and the like, or a combination thereof. The client devices 104 may be any computing device capable of displaying video frames to a user and receiving user input, and thus may be a desktop or laptop computer, a digital media player, a game console, a smartphone, a tablet, and the like.

[0019] To support streaming of game content to client devices during streamed game sessions, server 102 includes multiple processing units, such as a central processing unit (CPU) 106 and a graphics processing unit (GPU) 108. CPU 106 is a processing unit configured to execute a general-purpose set of instructions, generally organized in the form of a computer program, and to perform tasks on behalf of server 102. Such computer programs include, for example, operating systems, virtual machines, data security programs (e.g., data encryption / decryption programs), web pages, and database programs.

[0020] GPU 108 is a processing unit configured to perform operations generally associated with graphics and vector processing based on instructions received from CPU 106. For example, while executing one or more general-purpose programs, CPU 106 generates instructions to generate and manipulate graphical models for display and provides these instructions to GPU 108. In response, GPU 108 executes the instructions by performing one or more corresponding graphical operations to manipulate the graphical models to generate one or more frames for display. In some embodiments, the frames generated by GPU 108 are rendered frames ready for display on a display device.

[0021] To support receiving and displaying streamed game content, The client device 104 includes a CPU 112 and a GPU 116. Each of these modules is similar to a corresponding module of the server 102. Thus, for example, the CPU 112 is generally configured to execute general-purpose instructions on behalf of the client device 104, such as running one or more of an operating system and a web browser, etc. It should be noted that in some embodiments, the CPU 112 and the GPU 116 generally have relatively little computing power compared to the CPU 106 and GPU 108 of the server 120. The game streaming system 100 therefore leverages the computing power of the server 102 to stream higher quality game content to the client 104 than could be generated by the client device 104 alone.

[0022] In operation, client 104 initiates a game streaming session in response to a user requesting a game session via a computer program (not shown) executing on client 104, such as via a web browser, a dedicated game streaming program, a video streaming service program, etc. In response to the user's request, CPU 112 transmits a game session request to server 102 over network 110. In response, server 102 begins executing game program 109. In some embodiments, server 102 is configured to execute any one of a library of game programs, and the request provided by client 104 indicates a particular one of the game programs to be executed.

[0023] During execution, game program 109 maintains a collection of data called the game state, which represents the conditions or state of the game. A user of client 104 uses a game controller or other input device to provide input data to CPU 112, which communicates the input data to game program 109 over network 110. In response to the input data, game program 109 modifies the game state according to the rules of the particular game implemented by game program 109.

[0024] Based on the game state and in accordance with the rules of the implemented game, the game program 109 uses the CPU 106 and GPU 108 to generate rendered frames representing visual game information, such as a virtual environment, a game board, a selection screen, a heads-up display (HUD), and the like, or any combination thereof. The server 102 provides these rendered frames, designated streamed frames 115, over the network 110 for communication to the client 104. The client device 104 receives at least a subset of the streamed frames 115 and uses the GPU 116 to display the received frames on the display 117. Thus, the client device 104 displays the visual game information to the user, who in response provides further input data using a game controller or other input device, resulting in further changes to the game state maintained by the game program 109 and corresponding changes to the frames of the frame stream 115. In this manner, the user of the client device 104 interacts with the game program 109 to play the game executed by the server 102.

[0025] In some cases, transmission or other errors in the network 110 may prevent one or more of the streamed frames 115 from being received, or may be received in a corrupted manner at the client 104 and unable to be displayed. Such frames are referred to herein as "missing" frames. Missing frames may cause jumps, stuttering, and other visual discontinuities that may adversely affect a user's experience with the game program 109. In some cases, the game program 109 may not be able to add graphics to the streamed frames 115 as the frames are being generated. The effect of dropped frames can be addressed by applying graphical effects, such as to make discontinuities between frames less perceptible to the user. For example, the game program 109 can apply motion smoothing, which blurs the differences between successive frames of the streamed frames 115, making the discontinuities less visible to the user. In some embodiments, these improving graphical effects are applied in whole or in part by a program running on the client 104.

[0026] In some cases, the improved graphical effect can be improved by identifying certain discontinuity characteristics between frames displayed by the client 104. For example, motion smoothing can be improved by identifying the manner in which one or more graphical features changed between frames, such as the direction the feature moved, the distance the feature moved, and differences in the feature's texture. Therefore, during development or execution of the game program 109, it may be useful to visually identify discontinuities between frames displayed by the client 104 so that a developer or user can make corresponding adjustments to the improved graphical effect achieved by the game program 109. However, feature discontinuities between displayed frames may be small relative to the overall size of the displayed frames and therefore difficult for a user or developer to visually identify. Therefore, to assist a user or developer in identifying discontinuities between displayed frames, the GPU 116 is configured to generate an overlay 125 that provides a visual indicator of the discontinuities between displayed frames.

[0027] For example, in some embodiments, GPU 116 is configured to analyze each of two consecutively received frames (e.g., frames 120 and 122). In some cases, frames 120 and 122 correspond to discontinuous frames in streamed frames 115 because one or more of streamed frames 115 were dropped during transmission over network 110. GPU 116 identifies visual features in each of frames 120 and 122 and further identifies correspondences between the visual features. For example, in some embodiments, GPU 116 analyzes pixel features of each frame and identifies features based on patterns of pixel features, such as color, intensity, pixel activity, or other features of one or more subsets of pixels, or any combination of these features. For example, in some cases, GPU 116 identifies features of frame 120 based on the color of a subset of pixels of the frame having a particular pattern, such as a color pattern representing an object edge, a corner, or other visual feature. In some embodiments, the server 102 provides metadata with the streamed frames 115 that identifies expected characteristics of each streamed frame, and the GPU 116 employs the metadata instead of or in addition to analyzing the received frames to identify characteristics in each frame.

[0028] After or in parallel with identifying one or more features in each of frames 120 and 122, GPU 116 identifies correspondences between the features. For example, in some embodiments, GPU 116 identifies features in frames 120 and 122 that have similar patterns of corresponding pixel features, such as similar patterns of pixel color, intensity, pixel activity, or other features, or any combination thereof. In some embodiments, each pair of features between frames 120 and 122 is assigned a correspondence score based on the similarity of the patterns of corresponding pixel features, and features with a correspondence score above a threshold are identified as the same feature.

[0029] In response to identifying the same feature in both frames 120 and 122, GPU 116 identifies discontinuities between the features in frames 120 and 122. As used herein, discontinuities refer to differences in the location of the feature in each frame, the location of the feature in each frame, and the position of the feature in each frame. It may be a difference, or a combination of differences, in a feature identified between at least one pair of frames, such as a difference in the color of one or more pixels, and a difference in the graphical texture of a feature in each frame.

[0030] GPU 116 generates overlay 125 to have a visual indicator for each identified feature and to indicate feature discontinuities between frames 120 and 122. In some embodiments, the size of overlay 125 corresponds to the size of frames 120 and 122, and GPU 116 positions the visual indicators for the features such that the indicators are located at or near the location of the corresponding feature in one of frames 122, such that when overlay and frame 122 are displayed together, the visual indicators visually highlight the corresponding feature. In some embodiments, GPU 116 selects the visual indicators for the features based on one or more of the discontinuities identified for the features. For example, in some embodiments, the size, shape, or color, or a combination thereof, of the visual indicator indicates the magnitude of the corresponding discontinuity. Thus, for example, a larger visual indicator may indicate a larger location of the discontinuity associated with the feature.

[0031] Further, in some embodiments, different characteristics of the visual indicator correspond to different types of discontinuities associated with the feature. Thus, for example, in some embodiments, the size of the visual indicator indicates the magnitude of the position discontinuity associated with the feature (i.e., the amount of difference in the feature's position between frames 120 and 122), while the color of the visual indicator indicates whether there is a texture discontinuity between the feature (e.g., white indicates no difference in the feature's texture between frames 120 and 122, while red indicates a difference in texture).

[0032] In some embodiments, GPU 116 composites frame 122 and overlay 125 into a single frame for display on display 117. As such, when the composite frame is displayed, visual indicators in overlay 125 visually indicate discontinuities in features between frames 120 and 122, even if the discontinuities themselves are difficult or impossible to see with the naked eye. Based on the identified discontinuities, a user of client device 104 can make adjustments, such as adjustments to display settings of display 117, settings of GPU 116 or a game client program (not shown) running on client 104, settings of game program 109, or a combination thereof. For example, a user can adjust one or more motion smoothing settings of game program 109 to adjust aspects of the behavior of the motion smoothing portion of the program to better accommodate the identified discontinuities and improve the overall gaming experience at client device 104.

[0033] 2 shows an example of a discontinuity of a feature 230 between frames 120 and 122, according to some embodiments. In the illustrated example, feature 230 is in one position in frame 120 but in a different position in frame 122, specifically, shifted to the right. Thus, FIG. 2 shows a position discontinuity for feature 230. Feature 230 may therefore be any visual feature that can change position between frames, such as a corner of an object, an edge of an object, a particular texture feature of an object, a portion of an object having particular characteristics such as a particular shape, color, texture, or a combination thereof.

[0034] It should be noted that although Figure 2 shows position discontinuities associated with feature 230, in other embodiments, feature 230 may have other discontinuities. These other discontinuities may be in place of, or in addition to, the position discontinuities shown in Figure 2, and may be combined in any manner. For example, feature 230 may be in different translational positions, different rotational positions, or may have different textures. The color may be brown and a different color, etc., or any combination thereof.

[0035] 3 shows an example of an overlay 125 that provides a visual indicator 335 for a feature 230, according to some embodiments. In the illustrated example, the overlay 125 is shown with a dashed outline and depicted as being slightly offset from the frame 122 to distinguish the overlay from the frame. However, the GPU 116 can composite the overlay 125 with the frame 122 for display such that the frame 122 and the overlay 125 are effectively displayed together on the display 117. In particular, the indicator 335 is displayed on or near the feature 230 to provide a visual indicator of the associated discontinuity.

[0036] In different embodiments, indicator 335 is a color, shape, texture, informational text, icon, or other visual indicator, or combination thereof, that indicates one or more of the type of discontinuity associated with feature 230, the magnitude of the discontinuity, and a particular characteristic of the discontinuity (e.g., a spatial direction associated with a position discontinuity), or any combination thereof. For example, in the case of a position discontinuity, indicator 335 may be a line or arrow that indicates the size and direction of the position discontinuity.

[0037] In some cases, features in frames 120 and 122 have discontinuities between the frames. For example, features in frames 120 and 122 may shift position between the frames. In some embodiments, GPU 116 generates overlay 125 to include a visual indicator for each discontinuity between frames 120 and 122. An example according to some embodiments is shown in FIG. 4. In the illustrated example, overlay 125 includes visual indicators 335, 442, and 446, each of which visually indicates a discontinuity associated with a corresponding feature, designated feature 230, 440, and 444. In some embodiments, each of features 230, 440, and 444 is a different type of feature identified by GPU 116 in frames 120 and 122. For example, in some embodiments, feature 230 is an edge of an object, feature 440 is a corner of the same object, and feature 444 is a portion of a different object having an identified texture. In other embodiments, one or more of the features 230, 440, and 440 are the same type of feature, such as an edge or corner of an object.

[0038] The type of each visual indicator 335, 442, and 446 depends on the type of discontinuity associated with the corresponding feature. Therefore, each visual indicator 335, 442, and 446 may have different characteristics, such as different sizes, colors, shapes, orientations, texture information, icons, and the like, or combinations thereof, that visually indicate the type of discontinuity associated with the corresponding feature. For example, in some embodiments, visual indicator 335 is red to indicate that feature 230 has a positional discontinuity, while visual indicator 442 is red to indicate that feature 440 has a texture discontinuity. In other embodiments, visual indicator 442 has an "X" shape to indicate a color discontinuity for feature 440, while visual indicator 444 has a "plus" shape to indicate a rotational discontinuity for feature 444.

[0039] In some embodiments, the type of visual indicator, or a characteristic thereof, such as the size, color, or shape of the visual indicator, indicates the magnitude of the associated discontinuity. For example, in some embodiments, a larger visual indicator indicates a greater discontinuity than a smaller visual indicator. In other embodiments, a visual indicator of one particular color indicates a greater discontinuity than a visual indicator of a different particular color. Thus, in the example of FIG. 4, overlay 125 includes visual indicators for multiple characteristic discontinuities, each visual indicator indicating a magnitude of the associated discontinuity. The sensory indicator may indicate the type, magnitude, and other characteristics of the associated discontinuity.

[0040] FIG. 5 shows another example of an overlay 125 composited with a frame 122, according to some embodiments. In the example of FIG. 5, the overlay 125 includes rectangular visual indicators, each indicating a discontinuity in a feature of frame 122 at or near the corresponding rectangle. The size of the rectangle indicates the magnitude of the corresponding discontinuity. For example, visual indicator 561 is larger than visual indicator 562, indicating that the discontinuity associated with visual indicator 561 is greater than the discontinuity associated with visual indicator 562. A user of the overlay 125 can thus quickly identify the features with the greatest discontinuities and, accordingly, adjust one or more aspects of the game program 109 to accommodate the greatest discontinuities, for example, by modifying motion blur characteristics and frame interpolation, or a combination thereof.

[0041] 6 is a block diagram of a system for generating an overlay 125, including a feature identifier 652 and an overlay generator 655, according to some embodiments. In some embodiments, the feature identifier 652 and the overlay generator 655 are software modules that execute on one or more of the CPU 112 and the GPU 116 of the client device 104. In other embodiments, one or more aspects of the feature identifier 652 and the overlay generator 655 are implemented by dedicated hardware of the client device 104.

[0042] Feature identifier 652 is configured to analyze frames 120 and 122 to identify features included in each frame, identify correspondences between the features, and store the identified features and correspondences in feature specified data structure 654. To identify features, feature identifier 652 performs one or more feature identification processes, such as edge detection, corner detection, blob detection, ridge detection, texture detection, and the like, or any combination thereof. Additionally, in the example of FIG. 6, feature detector receives feature metadata 650 from server 102 (FIG. 1) that identifies or provides hints for identifying one or more features of frames 120 and 122. In some embodiments, game program 109 generates feature metadata during frame generation for streamed frames 115 based on object data associated with the game state. For example, in some embodiments, game program 109 determines the position, texture, and other specific characteristics of objects in the frames based on the game state to generate a visual representation of the game state, such as one or more of the virtual environment, player character, and game information overlay. The game program 106 may store position information, texture information, and other feature information for one or more of the objects in feature metadata 650. Feature identifier 652 can use the stored information to assist in feature detection, such as by assuming features exist for each object represented by feature metadata 650 and using object feature information to identify edges, corners, textures, and the like.

[0043] After identifying one or more features in each of frames 120 and 122, feature identifier 650 determines correspondences between the features. That is, feature identifier 650 determines which features in frame 122 correspond to which features in frame 120. Feature identifier 650 determines correspondences between the features based on similar patterns of characteristics of corresponding pixels in each feature pair, such as similar patterns of pixel color, intensity, pixel activity, or other characteristics, or any combination thereof. In some embodiments, feature identifier 650 identifies correspondences between features only if the features are within the same region of each frame, simplifying correspondence identification.

[0044] To identify whether a given feature in frame 120 corresponds to a given feature in frame 122, feature identifier 650 assigns each pair of features a correspondence score that indicates the similarity of the patterns of corresponding pixel features. The correspondence score may be based on any of a variety of factors, such as similarity of pixel color, pixel activity, pixel intensity, and patterns of pixel color or intensity. Feature identifier 650 identifies features that have a correspondence score above a threshold to be corresponding features.

[0045] Overlay generator 655 receives features 654 from feature identifier 650. For each set of corresponding features indicated by features 654, overlay generator 655 determines discontinuities between the corresponding features, such as differences in translational position, rotational position, color, texture, etc. Overlay generator 655 generates overlay 125 to have a visual indicator for each identified feature of features 654 that indicates the discontinuity associated with the feature, as described herein.

[0046] FIG. 7 is a flow diagram illustrating a method 700 for generating an overlay with visual indicators between received frames of a game stream, according to some embodiments. For illustrative purposes, method 700 is described with reference to an implementation in the game streaming system 100 of FIG. 1. At block 702, client device 104 receives at least a portion of streamed frames 115, including frames 120 and 122. At block 704, GPU 116 identifies features in each of frames 120 and 122 and identifies correspondences between the features. At block 706, GPU 116 identifies discontinuities between corresponding features. At block 708, GPU 116 generates overlay 125 including visual indicators for the discontinuities identified at block 706. GPU 116 composites overlay 125 with frame 122 and sends the composite frame to display device 117 for display, allowing a user to quickly visually identify discontinuities in the received frames.

[0047] In some embodiments, certain aspects of the above techniques can be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the above techniques. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage, solid-state storage such as flash memory, cache, random access memory (RAM), or one or more other non-volatile memory devices. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that can be interpreted or otherwise executed by one or more processors.

[0048] A computer-readable storage medium may include any storage medium, or combination of storage media, that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-Ray® discs), magnetic media (e.g., floppy disks, magnetic tape, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical systems (MEMS)-based storage media. Computer-Readable Storage Medium The body may be embedded in a computing system (e.g., system RAM or ROM), permanently attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disk or Universal Serial Bus (USB)-based flash memory), or coupled to a computer system via a wired or wireless network (e.g., a Network Accessible Storage (NAS)).

[0049] It should be noted that not all of the above activities or elements in the overall description are required, that some of certain activities or devices may be unnecessary, and that one or more other activities or elements may be performed in addition to those described above. Furthermore, the order in which the activities are described does not necessarily dictate the order in which the activities are performed. Also, concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the following claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0050] Benefits, other advantages, and solutions to problems have been described with regard to specific embodiments. However, these benefits, advantages, solutions to problems, and any feature(s) that give rise to or make more prominent any benefit, advantage, or solution are not intended to be construed as critical, necessary, or essential features of any or all of the claims. Moreover, the specific embodiments disclosed above are exemplary only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as set forth in the following claims. It will therefore be apparent that the particular embodiments disclosed are capable of alterations or modifications and that all such variations are within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the following claims.

Claims

1. receiving a video stream including a plurality of frames for display, the plurality of frames including a first frame including one or more graphical features and a second frame including the one or more graphical features; identifying a difference between a first display characteristic of each of the one or more graphical features associated with the first frame and a second display characteristic of each of the one or more graphical features associated with the second frame, the difference indicating a discontinuity between the first frame and the second frame; applying an overlay to the first frame, the overlay including one or more visual indicators indicating each of the one or more differences between the first frame and the second frame; and identifying, from among the one or more visual indicators, the visual indicator that exhibits the greatest difference.

2. 2. The method of claim 1 , wherein each of the first display features includes a first position of each of the one or more graphical features, and each of the second display features includes a second position of each of the one or more graphical features, the second positions being different from the first positions.

3. 3. The method of claim 1, wherein each of the first display features includes a first texture for each of the one or more graphical features, and each of the second display features includes a second texture for each of the one or more graphical features, the second texture being different from the first texture.

4. 4. The method of claim 1, wherein each of the first display characteristics includes a first size of each of the one or more graphical features, and each of the second display characteristics includes a second size of each of the one or more graphical features, the second sizes being different from the first sizes.

5. 5. The method of claim 1, wherein applying the overlay to the first frame comprises applying the overlay to the first frame in response to at least one of the one or more differences exceeding a threshold.

6. The method of any one of claims 1 to 5, further comprising: varying a size of each of the one or more visual indicators based on a magnitude of each of the one or more differences.

7. 7. The method of claim 1, further comprising applying an improvement technique to improve the effect of discontinuities between streamed frames based on the visual indicator of the greatest difference.

8. The method of claim 7 , wherein the enhancement techniques include motion smoothing and / or frame interpolation for the video stream that blurs the one or more differences.

9. A program for causing a computer to execute the method according to any one of claims 1 to 8.

10. a storage unit that stores the program according to claim 9; and one or more processors configured to execute the program.

Images