Method and apparatus for cloud gaming

Abstract

To provide a method and a device for a cloud game in various aspects of the present disclosure.SOLUTION: In some examples, a device for a cloud game comprises a processing circuit. For example, the processing circuit receives a video sequence, and meta data associated with the video sequence. The video sequence contains a sequence image frame generated in response to game control information. The meta game is determined on the basis of the game control information. The processing circuit can determine a coding construction for coding the video sequence on the basis of the meta game. Then, the processing circuit codes the video sequence to a video bitstream to be coded on the basis of the coding construction.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims benefit of priority to U.S. Patent Application No. 17 / 184,082, entitled "Method and Apparatus for Cloud Gaming," filed February 24, 2021, which claims benefit of priority to U.S. Provisional Patent Application No. 62 / 990,204, entitled "Method of Video Coding with Metadata in a Cloud Gaming System," filed March 16, 2020. The entire disclosures of the prior applications are hereby incorporated by reference in their entirety.

[0002] [Technical field] This disclosure describes embodiments generally relating to multimedia technology in the field of cloud gaming. [Background technology]

[0003] The discussion of the background art provided herein is intended to provide an overall context for the present disclosure. The extent of the work of the currently signed inventors described in this background section and in each aspect of this specification has not been presented as prior art at the time of the filing of this disclosure, and is not expressly or impliedly admitted as prior art to the present disclosure.

[0004] Cloud gaming services are trending online services available from multiple service providers. Cloud gaming may also be called games on demand. In cloud gaming systems, games are executed on remote servers and are called cloud-based games. Users can play cloud-based games through user devices. Game content may be generated on remote servers and streamed and displayed on user devices. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure provide methods and apparatus for cloud gaming. In some examples, the apparatus for cloud gaming includes a processing circuit. For example, the processing circuit receives a video sequence and metadata associated with the video sequence. The video sequence includes a series of image frames generated in response to game control information, and the metadata is determined based on the game control information. The processing circuit can determine an encoding configuration for encoding the video sequence based on the metadata. Then, the processing circuit encodes the video sequence into a coded video bitstream based on the encoding configuration.

[0006] In one embodiment, the metadata includes a motion vector, and the processing circuitry determines a starting point for a motion search for encoding a block within an image frame of the video sequence based on the motion vector.

[0007] In another embodiment, the metadata indicates at least one of image characteristics of a scene change characteristic, a lighting change characteristic, a blinking characteristic, a panning characteristic, a zooming characteristic, a fading characteristic, and an overlay characteristic of an image frame of the video sequence, and the processing circuitry can determine an encoding configuration for encoding the image frame based on the image characteristics.

[0008] In another embodiment, the metadata indicates a bounding box in the image frame, and the processing circuitry can determine, based on the bounding box, a reference area in a previous image frame for an area not covered by the bounding box in the current image frame for an image frame previous to the current image frame, and encode, based on the reference area in the previous image frame, the area not covered by the bounding box in the current image frame.

[0009] In another embodiment, the metadata indicates a model transformation from the first image frame to the second image frame, and the processing circuitry can determine an encoding configuration based on the model transformation and can encode blocks in the second image frame based on reference blocks in the first image frame based on the encoding configuration.

[0010] In another embodiment, the metadata indicates a presence of an object of the current image frame in a subsequent image frame. The processing circuitry determines the buffer configuration based on the metadata.

[0011] In another embodiment, the metadata indicates a noise level of the image frame, and the processing circuitry can pre-process / post-process the image frame based on the noise level.

[0012] In another embodiment, the metadata indicates a region of interest in the image frame, and the processing circuitry encodes the region of interest using a first encoding configuration having a higher quality than a second encoding configuration used to encode blocks outside the region of interest.

[0013] In another embodiment, the metadata indicates a graphic overlay in a region of the image frame, and the processing circuitry determines an encoding scheme for encoding the region based on the graphic overlay.

[0014] In another embodiment, the metadata indicates a message box in the image frame for the overlay message. The processing circuitry includes information of the overlay message in a particular Network Abstraction Layer Unit (NALU) of the coded video bitstream.

[0015] Aspects of the present disclosure also provide a non-volatile computer-readable storage medium having instructions stored thereon that, when executed by a computer for video processing, cause the computer to perform a method for cloud gaming.

[0016] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings, in which: [Brief description of the drawings]

[0017] [Figure 1] A block diagram of a cloud gaming system according to one embodiment of the present disclosure is shown. [Diagram 2] FIG. 1 illustrates a cloud gaming scenario according to some embodiments of the present disclosure. [Diagram 3] FIG. 1 illustrates a block diagram of a cloud gaming system according to some embodiments of the present disclosure. [Figure 4] 1 shows a block diagram of a video encoder according to one embodiment of the present disclosure. [Diagram 5] 1 shows a diagram of a video decoder according to some embodiments of the present disclosure. [Figure 6] 1 shows a flowchart outlining an example process according to one embodiment of the present disclosure. [Figure 7] 1 shows a flowchart outlining another example process according to one embodiment of the present disclosure. [Figure 8] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Aspects of the present disclosure provide systems, devices, and methods for metadata-assisted video coding (encoding / decoding) design in the field of cloud gaming.

[0019] FIG. 1 illustrates a block diagram of a cloud gaming system (100) according to an embodiment of the present disclosure. The cloud gaming system (100) includes a server system (101) and a client system (102) connected by a network (150). The server system (101) includes one or more servers (110) coupled to each other. For example, the server system (101) may include data centers (multiple), server farms (multiple), etc. The server system (101) may render game content and compress the game content into a bitstream. The bitstream may be distributed to the client systems (102) by the network (150). The client systems (102) include one or more client devices (160), each of which may interact with a game user (also referred to as a player). For example, the client devices (160) may receive user control information from the game user and may provide output to the game user, such as video output, audio output, etc.

[0020] The servers (110) may each be implemented using any suitable technology. In FIG. 1, the servers (110) include a processing circuit (120), a memory (115), and an interface circuit (111) coupled together. The memory (115) is configured to store software instructions and various data (e.g., media data, configuration information, control information, etc.) before, during, and after processing.

[0021] The processing circuitry (120) may include any suitable processing circuitry, such as, for example, one or more central processing units (CPUs), one or more graphic processing units (GPUs), an application specific integrated circuit, etc. In the example of FIG. 1, the processing circuitry (120) may be configured to include a game engine (125). For example, the one or more CPUs and / or GPUs may function as the game engine (125) by executing game software stored in the memory (115). Furthermore, the processing circuitry (120) may be configured to include an encoder, such as, for example, a video encoder, an audio encoder, etc. In the example of FIG. 1, the processing circuitry (120) is configured to include a video encoder (130). For example, the one or more CPUs and / or GPUs may be configured to function as the video encoder (130). In another example, the video encoder (130) may be implemented using an application specific integrated circuit.

[0022] The interface circuit (111) can interface the server (110) with the network (150). The interface circuit (111) can include a receiver for receiving signals from the network (150) and a transmitter for transmitting signals to the network (150). For example, the interface circuit (111) can transmit signals carrying encoded media data, such as encoded video bitstreams, encoded audio bitstreams, etc., to other devices, such as client devices (160), via the network (150). The interface circuit (111) can receive signals carrying user information, such as user input, from the client devices (160) via the network (150).

[0023] The network (150) is suitably coupled to the server system (101) and the client system (102) via wired and / or wireless connections, such as, for example, Ethernet connections, fiber optic connections, WiFi connections, cellular network connections, etc. The network (150) may include network server devices, storage devices, network devices, etc. The components of the network (150) are suitably coupled via wired and / or wireless connections.

[0024] The client device (160) is configured to interact with a user. In the example of Figure 1, the client device (160) includes various components, such as an interface circuit (161), a processing circuit (170), a display (165), and a user input device (169), which are coupled together.

[0025] The interface circuit (161) can interface the client device (160) with the network (150). The interface circuit (161) can include a receiver for receiving signals from the network (150) and a transmitter for transmitting signals to the network (150). For example, the interface circuit (161) can receive signals from the network (150) carrying data, such as signals carrying coded video bitstreams, signals carrying audio bitstreams, etc., transmitted by the server (110), and can transmit signals, such as signals carrying user input, via the network (150) to other devices, such as the server (110).

[0026] The user input device (169) can receive user input, which can be appropriately transmitted to the server system (101). In one example, the user input can be provided to the processing circuit (170), which can convert the user input into an appropriate format, such as a message, packet, etc., and transmit the user input via the interface circuit (161). The user input can be transmitted to the server system (101) via the network (150). The user input device (169) can be any suitable user input device, such as, for example, a joystick, a motion sensor, a microphone, a keyboard, a mouse, a touch screen, a touch pad, etc.

[0027] The processing circuit (170) may include any suitable processing circuit, such as, for example, a CPU, a GPU, an application specific integrated circuit, etc. The processing circuit (170) may be configured to include media decoder(s), such as, for example, a video decoder, an audio decoder, etc. For example, the processing circuit (170) is configured to include a video decoder (180). In one example, a GPU is configured as the video decoder (180). In another example, a CPU can function as the video decoder (180) by executing software instructions. The video decoder (180) can decode information in a coded video bitstream to reconstruct a video sequence (e.g., a series of image frames).

[0028] Further, the processing circuitry (170) may be configured to include a game client (175). For example, the CPU may function as the game client (175) by executing cloud gaming client software. The game client (175) may receive user input (171) and transmit data representing the user input (171) to the game engine (125) at the server (110).

[0029] The display (165) may be any suitable display device, such as, for example, a television, a smartphone, a wearable display, a head-mounted device, etc. In one example, the display (165) may receive a video sequence and suitably display the video sequence.

[0030] In some examples, to play a game, the game engine (125) can generate game content for the game, such as video sequences, audio, and the like, which illustrates various game environments of the game. In one example, the game engine (125) can generate video content (122) (e.g., video sequences) and provide the video content (122) to a video encoder (130). The video encoder (130) can compress the video content (122) into a coded video bitstream (123), which can be transmitted by an interface circuit (111) and streamed over a network (150) to a client device (160). In the client device (160), the interface circuit (161) can receive a signal carrying the coded video bitstream and provide the coded video bitstream (172) to a video decoder (180). The video decoder (180) can reconstruct a video sequence from the coded video bitstream (172), and this reconstructed video sequence (173) can be provided to a display (165) to present the game environment to a game user.

[0031] A game user may interact with the game client (175). For example, the game client (175) may receive user control information (also referred to as interaction data in some examples), such as the user input (171) shown in FIG. 1. The user control information (which is displayed by (121) on the server (110), such as commands, position information, depth information, motion information, etc., may be transmitted from the game client (175) to the game engine (125) via the interface circuit (161), the network (150), and the interface circuit (111). The game engine (130) may then perform operations and generate updated game content in response to the received user control information (121). The updated game content may be compressed and provided from the server (110) to the client device (160). As the game continues, the game process is repeated.

[0032] In some embodiments, the game engine (125) can generate game video content in the form of raw video content (122), such as a series of image frames in YUV format. The raw video content (122) can be provided to a video encoder (130), which compresses the raw video content and generates a coded video bitstream (123) according to a particular video coding format or standard, such as H.264, H.265, AV1, AVS2, etc. This coded video bitstream (123) can be sent to a client device (160) and decoded by a client-side video decoder for display using a compatible format or standard, such as the video encoder (130).

[0033] The cloud gaming system (100) provides a means of playing video games by running game software on a remote server, rather than on a game console or other local device in the relevant example. For example, a game company may own a server (110) that is a dedicated server for game streaming, and the server (110) may be significantly more powerful than the consumer-side hardware (e.g., client device). Furthermore, in some examples, the streaming service may transmit similar data to multiple client devices (160) where the same game is played by multiple players (users). Thus, multiple players may play the same game at a lower total operating cost (to the game company) than would be expected in the relevant example where the cost is proportional to the number of players.

[0034] FIG. 2 shows a diagram (200) illustrating a cloud gaming scenario according to some embodiments of the present disclosure. In the example of FIG. 2, a server (210) is a cloud gaming server that provides game services for a game, and client devices 260A-260C are client devices that interact with players A-C of the game, respectively. The server (210) may be configured similarly to the server (110) in FIG. 1. The client devices 260A-260C may each be configured similarly to the client device (160) in FIG. 1.

[0035] In the example of FIG. 2, a game engine (not shown) in the server (210) renders game content for a game environment and provides the game content to the client devices (260A)-(260C) over a network (not shown). At one time, as shown as (201) in FIG. 2, the client device (260B) receives user control information, such as user input from player B. As shown as (202) in FIG. 2, the client device 260B sends data representing the user input to the server (210) over the network. In response to the data representing the user input, as shown as (203) in FIG. 2, the game engine in the server (210) performs operations to update the game environment. The server (210) then sends the updated game content to the client devices (260A)-(260C) over a network (204) in FIG. 2. On the client side, as shown as (205) in FIG. 2, each client device (260A)-(260C) can refresh the game environment, for example, refresh the display, in response to the updated game content.

[0036] In a related example, the client device may be a game console or a personal computer, which is required to play most games and may constitute a large upfront cost for the game. The price of these game consoles or personal computers may reach a cost of hundreds to thousands of dollars. The local machines (game consoles and personal computers) also wear out over time and have degraded performance. As technology develops, cloud game servers are continually upgraded at no additional cost to the consumer.

[0037] According to some aspects of the present disclosure, cloud gaming may require high and reliable bandwidth requirements and low latency required for two-way communication to achieve a good user experience. In some examples, cloud gaming may require a fast and stable Internet connection and proximity to game servers to obtain high and reliable bandwidth. For example, streaming services for cloud gaming may require bandwidth between 10 Mbps and 35 Mbps. The present disclosure provides techniques for improving coding speed and coding efficiency to reduce bandwidth requirements and improve visual quality.

[0038] In some instances, high latency can be detrimental to the gaming experience. Latency can be contributed by many factors, video encoding for streaming being one of these factors. The present disclosure also provides techniques to reduce encoding time while maintaining coding efficiency, thus improving the cloud gaming experience.

[0039] Aspects of the present disclosure provide techniques for metadata-assisted video encoding design in a cloud gaming system. In some examples, metadata for video content (e.g., raw video) refers to data that provides information about the video content, such as, for example, camera information, color information, lighting information, motion information, viewing information, user control, and the like. The metadata along with the raw video may be rendered by a game engine (e.g., game (125)) and provided as input by the game engine to a video encoder (e.g., video encoder (130)) to aid in the encoding process. In some embodiments, the video encoder (e.g., video encoder (130)) can encode the raw video into a coded video bitstream with the aid of the metadata. Furthermore, in some embodiments, the metadata may be provided along with the coded video bitstream to a video decoder (e.g., video decoder (180)). The video decoder (e.g., video decoder (180)) can decode the coded video bitstream with the aid of the metadata.

[0040] 3 illustrates a block diagram of a cloud gaming system (300) according to some embodiments of the present disclosure. The cloud gaming system (300) operates similarly to the cloud gaming system (100) described above and utilizes certain components that are the same or equivalent to those used in the cloud gaming system (100). For example, the server system (301) is configured similarly to the server system (101), the client system (302) is configured similarly to the client system (102), the network (350) is configured similarly to the network (150), the server (310) is configured similarly to the server (110), the processing circuitry (320) is configured similarly to the processing circuitry (120), the memory (315) is configured similarly to the memory (115), the interface circuitry (311) is configured similarly to the interface circuitry (111), the game engine (325) is configured similarly to the game engine (125), and the like. The video encoder (330) is configured similarly to the video encoder (130), the interface circuit (361) is configured similarly to the interface circuit (161), the processing circuit (370) is configured similarly to the processing circuit (170), the display (365) is configured similarly to the display (165), the user input device (369) is configured similarly to the user input device (169), the game client (375) is configured similarly to the game client (175), and the video decoder (380) is configured similarly to the video decoder (180). Descriptions of these components are provided above and are omitted here for clarity.

[0041] Further, in the example of FIG. 3, the video encoder (330) is configured to encode the video content into a video bitstream with the aid of metadata. For example, the video encoder (330) includes a controller (331), which is a metadata-based controller. The game engine (325) provides the raw video content (322) along with metadata (326) to the video encoder (330). The metadata (326) includes information associated with the raw video content (322), such as camera information, color information, lighting information, motion information, viewing information, user control, and the like. The controller (331) receives the metadata (326) and controls the configuration of the video encoder (330) based on the metadata (326) to encode the raw video content (322) and generate a coded video bitstream (323). With the aid of the metadata, the speed and efficiency of encoding can be improved, and the bandwidth requirements and delays can be reduced.

[0042] In some embodiments, the coded video bitstream (323) may be output along with metadata, for example as indicated by the metadata output (327).

[0043] Further, in the example of FIG. 3, the video decoder (380) is configured to decode the video bitstream with the aid of metadata. For example, the video decoder (380) includes a controller (381) that is a metadata-based controller. The controller (381) receives the metadata (377) along with the coded video bitstream (372). In one example, the coded video bitstream (372) may be the coded video bitstream (323) generated by the video encoder (330), and the metadata (377) may be the metadata output (327). The controller (381) may control the configuration of the video decoder (380) based on the metadata (377) to decode the coded video bitstream (372) and generate a reconstructed video sequence (373).

[0044] In some embodiments, the metadata (326) from the game engine (325) to the video encoder (330) may include any suitable game control information, such as game scene information (e.g., camera information, color information, lighting information, motion information, viewing information), user control information, etc., that can help improve coding efficiency and / or reduce the complexity of the encoding process.

[0045] Specifically, in some examples, user input (371) (also referred to as user control information) from a user input device (369), such as a joystick, keypad, mouse, and / or keyboard, may be transmitted from the client device (360) to the server (310). In one example, the game engine (325) receives user control information (321) corresponding to the user input (371) at the client device (360). The game engine (325) may generate game scene content (e.g., video content, audio content, etc.) based on the user control information (321). In some embodiments, the metadata (326) may be determined based on the game scene information and the user control information (321). For example, a global movement of the game scene and / or a local movement of an object may be identified based on the game scene information and the user control information (321). In one example, the user control information (321) includes a change in gaze direction, whereby a global movement of the game scene may be identified based on the change in gaze direction. In another example, the user control information (321) includes object motion, whereby local motion of the object may be identified. In one example, the metadata (326) may include global motion of the game scene, local motion of the object, and bounding boxes of the object. The metadata (326) may aid the video encoding process as additional input data to the video encoder (330). The metadata (326) may be provided to the video encoder (330) along with the raw video content (322) rendered by the game engine (325). The video encoder (330) may utilize the information in the metadata (326) to efficiently perform motion estimation / compensation processing using the raw video content (322).

[0046] In some embodiments, the metadata (326) may be provided to the video encoder (330) as configurable encoder settings, such as parameters of the video encoder (330). In one example, the metadata (326) may be used to generate a global motion model (e.g., an affine model, a perspective model, a rotation model, a zooming model, etc.) for an image or region of an image. In another example, the metadata (326) may be used to select encoding tools in the video encoder (330). However, in one example, if any of such parameters are unavailable, a set of default values ​​may be assumed by the video encoder (330).

[0047] In some embodiments, the metadata (326) provided to the video encoder (330) may be processed and indicated by metadata (327) and transmitted along with the coded video data stream (323). The metadata (327) may assist a video decoder in decoding the coded video bitstream (323). For example, metadata (327) that may assist in decoding and displaying the coded video bitstream (323), such as timing information, overlay messages, overlay graphics, bounding boxes, etc., may be transmitted as supplemental enhancement information (SEI) messages. The SEI messages may be conveyed within the coded video bitstream or separately.

[0048] In some embodiments, the client device (360) receives the coded video bitstream (372) along with metadata (377). The metadata (377) may be used by the client device (360) to aid in decoding or post-processing of the video sequence carried in the coded video bitstream (372). For example, the metadata (377) may include timing information, overlay messages, overlay graphics, bounding box information that can aid in decoding and displaying the coded video bitstream (372).

[0049] In some embodiments, the metadata may be coded (encoded / decoded) using an additional codec. In some examples, the game engine (325) may output separated content, video, audio, overlay messages, overlay graphics, etc. In one example, the overlay messages (or overlay graphics) are considered as metadata for the video content. The overlay messages (or overlay graphics) may be compressed into a separate bitstream (second bitstream) using a separate encoder from the video encoder 330. And in one example, the coded video bitstream for the video content and the second bitstream for the overlay messages (or overlay graphics) may be multiplexed into a single bitstream.

[0050] According to some aspects of the present disclosure, various information that can assist in the video encoding process or the video decoding process may be transmitted as metadata along with the game video input (e.g., a video sequence as game video input for a video encoding process, a coded video bitstream as game video input for a video decoding process).

[0051] In some examples, the metadata may include location information for regional game content, such as the location of a game character (eg, the main character of the game), a region of interest (ROI) in a game scene, and the like.

[0052] In some examples, the metadata may include camera information such as main camera position information, camera field of view angle, and various camera parameters (e.g., focal length, distance to main character, field of view angle, distortion parameters, vignette parameters, etc.).

[0053] In some examples, the metadata may include systematic information of the game scene, such as, for example, the main color of the scene, the brightness of the scene, the noise level of the current image, scene change indicators, fading indicators, and / or fading parameters (e.g., weightings for weighted prediction), etc.

[0054] In some examples, the metadata may include regional and global motion information. In one example, the metadata includes translational motion vectors for one or more of the previous images at the pixel level or at a block level having a predefined block size.

[0055] In some examples, the metadata may include a global motion model or a regional motion model. In one example, the metadata may include a global motion model, such as a panning model, a zooming model, a rotation model, a perspective motion model, an affine motion model, etc. for the image. In another example, the metadata may include a regional model, such as a panning model, a zooming model, a rotation model, a perspective motion model, an affine motion model, etc.

[0056] In some examples, the metadata may include user input information such as keystrokes, mouse movements, mouse button clicks, motion sensor input, and the like.

[0057] In various embodiments, metadata is provided, for example, from the game engine (325) to a video encoder (330) on the server side, and includes various information that can assist the encoding process in order to reduce the complexity of the encoding process or to improve coding efficiency, such as reducing bitrate and / or reducing distortion.

[0058] In one embodiment, the game engine (325) can provide metadata including recommended motion vectors at a pixel or block level. The recommended motion vectors can be used to aid motion estimation by the video encoder (330), thereby reducing motion vector search operations and improving motion estimation accuracy. In one example, the recommended motion vectors from the game engine (325) can be used as a starting point in a motion search algorithm by the video encoder (330). In some examples, the game engine (325) generates video content in response to game scene motion information and user input, and the motion vectors recommended by the game engine (325) can be very accurate. This allows the video encoder (330) to reduce the time for motion vector search and achieve accurate motion estimation.

[0059] In another embodiment, the game engine (325) can provide metadata including parameters that can identify image features in the video content, such as scene changes, lighting changes, blinking, panning, zooming, fading, message overlays, etc. In view of the parameters, the video encoder (330) can select appropriate configuration settings, parameter settings, code tools, etc. to drive the encoding process, which can improve compression efficiency.

[0060] In another embodiment, the game engine (325) can provide metadata including bounding box information, and the video encoder (330) can use the bounding box information to determine the relationship between uncovered areas in an image frame and valuable image frames in a video sequence, and can use the relationship to avoid unnecessary calculations and improve processing speed. In one example, a character in a bounding box may have a different motion from the rest of the image. The video encoder (330) can use the bounding box information to apply separate motion models to the character and the rest of the image. In another example, the video encoder (330) can use the bounding box information to identify, for example, from a previous image, certain areas (also called uncovered areas) that are not covered by a bounding box in a current image (e.g., the certain area is covered by a bounding box in the previous image), and can also identify previous images that exposed certain regions (e.g., before a bounding box covered the certain region). Such information allows the video encoder (330) to avoid unnecessary calculations in coding modes that are unlikely to be selected for those regions. For example, based on such information, the video encoder (330) may directly select a temporal prediction to predict a particular region in the current image based on the identified previous image(s) without attempting other prediction tools.

[0061] In another embodiment, the game engine (325) may provide metadata indicating model transformations in an image region (e.g., the same object with different imaging characteristics) relative to a previous image(s), such as changes in motion blur effects, changes in lens blur effects, changes in lens distortion, changes in lens vignetting, etc. The video encoder (330) may determine the coding tool, coding method, coding parameters, etc. to be used for a particular region based on the model transformation information indicated by the metadata. For example, if the metadata indicates lens blur in a region of the current image frame relative to a reference region in the previous image frame, the video encoder (330) may encode the region of the current image frame based on the reference region in the previous image frame and a blur filter.

[0062] In another embodiment, the game engine (325) provides metadata indicating the presence of an image (current image) or an object in the current image in the subsequent image(s). Based on the metadata, the video encoder (330) can determine a buffer configuration, such as buffering a particular content / area of ​​the current image to be used as a reference for the subsequent image(s). In one example, the metadata indicates a movement of an object causing a movement of the object in the subsequent image. Based on the metadata, the video encoder (330) can buffer the content of an area (area having the object) in the current image having the object, and the buffered content can be used as a reference for the subsequent image(s) in motion compensation. In another example, the metadata indicates a map to be used for the current image and the subsequent image. Based on the metadata, the video encoder (330) can buffer an area of ​​the map in the current image as a reference for coding the subsequent image. In another example, the metadata indicates that the current image (or a region of the current image) is a background for the subsequent image. Based on the metadata, the video encoder (330) can buffer the current picture (or a region of the current picture) to be the long-term reference and can code subsequent pictures accordingly. For example, a skip mode can be used to code blocks in subsequent pictures that can be referenced to the long-term reference.

[0063] In another embodiment, the game engine (325) provides metadata indicative of noise characteristics, such as noise level information of an image. Based on the metadata, pre-processing and / or post-processing of the image can be performed. In one example, the game engine (325) provides a noisy image (in the video content) and metadata indicative of the noise level to the video encoder (330). Based on the metadata indicative of the noise level, the video encoder (330) can remove noise (e.g., apply a filter to remove noise) from the noisy image in a pre-processing stage, generate a low-noise image, and encode the low-noise image. The strength / threshold of the noise removal can be determined by the noise level information in the metadata. In one example, the encoded image (low-noise image) and the metadata indicative of the noise level are transmitted to the client device (360). The video decoder (380) can then reconstruct the low-noise image and then add noise to the low-noise image in a post-processing stage based on the noise level information in the metadata.

[0064] In another example, the game engine (325) provides a low-noise (or noise-free) image (in the video content) and metadata indicative of the noise level to the video encoder (330). The video encoder (330) then encodes the low-noise image. In one example, the encoded image (low-noise image) and metadata indicative of the noise level are transmitted to the client device (360). The video decoder (380) then reconstructs the low-noise image and can then add noise to the low-noise image in a post-processing stage based on the noise level information in the metadata. In some examples, the video encoder (330) includes a local decoder that can add noise in a post-processing stage based on the noise level information in the metadata.

[0065] It should be noted that a low noise image may be coded with a higher coding efficiency as compared to a high noise image.

[0066] In another embodiment, the game engine (325) provides metadata indicating regions of interest in an image. Based on the metadata, the video encoder (330) can achieve better quality or rate control. In one example, a higher bitrate can be allocated for encoding the regions of interest than other regions in the image. In another example, if the bandwidth is not sufficient to code the entire image at a higher bitrate, the bitrate for coding areas outside the regions of interest can be reduced first to maintain a higher bitrate for the regions of interest.

[0067] In another embodiment, the game engine (325) provides metadata indicating regions of graphic overlays in an image, such as logos, banners, radar images, maps, message boxes, etc. Based on the metadata, the video encoder (330) can apply appropriate coding tools for the regions of the graphic overlays, which may differ from coding tools for other regions in the image. In one example, the logos may be coded using skip mode to improve coding efficiency. Using appropriate coding tools for the regions of the graphic overlays can avoid erroneous motion information, artifacts / distortions, etc.

[0068] In some embodiments, the game engine (325) provides metadata indicating a message box for the overlay message. The overlay message may be compressed and included in the video bitstream as a separate network abstraction layer unit (NALU) used specifically for the overlay message. In some examples, the NALU is actually a packet. The video bitstream may be organized and transmitted via NALUs (packets), one of which may be a special NALU for the overlay message. The NALU for the overlay message may include the area of ​​the bounding box of the overlay message, the position of the overlay message, the font, size, color, style of text, overlay text content, etc. In one example, the overlay text content may be coded in Unicode. In one example, the game engine (325) provides the overlay message to the video encoder (330) separately from the video content. The video encoder (330) may encode the video content and the overlay message separately. The overlay message may be compressed as a separate NALU in the video bitstream. A video bitstream can be transmitted from the server (310) to the client device (360). The processing circuitry (370) can decode the video content and the overlay message separately from the video bitstream and then add the overlay message to the video content, such as adding closed captions.

[0069] 4 shows a block diagram of a video encoder (430) according to one embodiment of the present disclosure. The video encoder (430) is included in an electronic device (410). In one example, the electronic device (410) is a server device such as the server (310) in the example of FIG. 3. The electronic device (410) also includes a transmitter (411) (e.g., transmission circuitry, etc.). The video encoder (430) may be used in place of the video encoder (330) in the example of FIG. 3.

[0070] The video encoder (430) can receive raw video content from a video source, such as the game engine (325) in the example of Figure 3. Additionally, the video encoder (430) can receive metadata, such as camera information, color information, lighting information, motion information, viewing information, overlay information (e.g., overlay information, overlay graphics, etc.), user controls, etc., to provide information associated with the raw video content.

[0071] In some examples, the raw video content is in the form of a digital video sample stream that may be in any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). The raw video content may be provided as multiple separate images that convey motion when viewed sequentially. The images themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc., in use.

[0072] According to one embodiment, the video encoder (430) can code and compress images of raw video content (also called video sequence) into a coded video bitstream (443). In the example of FIG. 4, the video encoder (430) includes a controller (431), which is configured to control the encoding configuration of the video encoder (430) based on metadata to improve encoding speed and efficiency. In some embodiments, the controller (431) can determine the encoding configuration based on the metadata. Furthermore, the controller (431) is functionally coupled to other functional units and controls the other functional units for the encoding process. This coupling is not shown for clarity. It should be noted that the controller (431) can set various configurations and parameters, such as picture skip, quantizer, lambda value of rate-distortion optimization technique, picture size, group of pictures (GOP) layout, maximum motion vector search range, motion model, etc.

[0073] In one example, the controller (431) can be configured to determine a global motion model for the image, such as an affine model, a perspective model, a rotation model, a scaling model, etc., based on the metadata and configure the encoding parameters accordingly. In another example, the controller (431) can be configured to determine a region in a bounding box within the image and a local motion model for the region, based on the metadata. In some embodiments, the controller (431) includes a separate encoder (not shown) that is distinct from other parts of the video encoder (430). The separate encoder can compress the metadata into compressed metadata. For example, the metadata includes overlay content, such as overlay messages, overlay graphics, etc. The overlay content can be compressed into an overlay content bitstream (444). In one example, the overlay content bitstream (444) can be combined with the coded video bitstream (443) into a single bitstream, for example by the multiplexer (412). In another example, the controller 431 can include the metadata in an SEI message associated with the coded video bitstream (443). The controller (431) may be configured to have other suitable functionality associated with the video encoder (430) that is optimized for a particular system design.

[0074] In some embodiments, the video encoder (430) is configured to operate in an encoding loop. As an oversimplified explanation, in one example, the encoding loop can include a source coder (432) (e.g., responsible for creating symbols, such as a symbol stream, based on an input image to be coded and a reference image) and a (local) decoder (434) embedded in the video encoder (430). The decoder (434) reconstructs the symbols to create sample data in a similar manner to how the (remote) decoder creates the sample data (because in the video compression techniques contemplated in the disclosed subject matter, any compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (435). The decoding of the symbol stream provides bit-exact results regardless of the location of the decoder (local or remote), so that the contents of the reference image memory (435) correspond bit-exactly between the local and remote decoders. In other words, the reference image samples "seen" by the prediction part of the encoder are exactly the same sample values ​​"seen" by the decoder when using prediction during decoding.

[0075] The operation of the "local" decoder (434) may be the same as that of the "remote" decoder already described in detail above in relation to FIG. 5. During operation, in some embodiments, the source coder (432) may perform motion-compensated predictive coding, which predictively codes an input image with reference to one or more previously coded images designated as "reference images" from the video sequence. In this manner, the coding engine (433) codes differences between pixel blocks of the input image and pixel blocks of reference images that can be selected as prediction references for the input image. In some examples, the source coder (432) may receive control signals from the controller (431). The control signals may be generated based on metadata of the video sequence and may control the source coder (432) to perform motion-compensated predictive coding at high speed.

[0076] The local video decoder (434) can decode the coded video data of the image that can be designated as a reference image based on the symbols generated by the source coder (432). The operation of the coding engine (433) can advantageously be a lossy process. When the coded video data is decoded by a video decoder (such as the video decoder described with reference to FIG. 5), the reconstructed video sequence can be a replica of the source video sequence, usually with some errors. The local video decoder (434) can copy the decoding process that can be performed by the video decoder on the reference image and store the reconstructed reference image in the reference image cache (435). In this way, the video encoder (430) can locally store a copy of the reconstructed reference image that has a common content with the reconstructed reference image obtained by the far-end video decoder (in the absence of transmission errors).

[0077] The predictor (436) can perform a prediction search for the coding engine (433). That is, for a new image to be coded, the predictor (436) can search the reference image memory (435) for sample data (as candidate reference pixel blocks). In some examples, the predictor (436) receives a control signal from the controller (431). The control signal can be generated based on metadata, camera information, color information, lighting information, motion information, view information, user control, block shape, etc. The control signal can control the predictor (436) to reduce the time for the predictive search of a suitable prediction reference for the new image. In one example, the control signal can indicate a reference (e.g., a reference image, a reference block, etc.). In another example, the control signal can indicate a starting point for the search. The predictor (436) can operate on a pixel block basis based on the sample block to find a suitable prediction reference. In some cases, the input image can have a prediction reference drawn from multiple reference images stored in the reference image memory (435), as determined by the search results obtained by the predictor (436).

[0078] The controller (431) may also manage the encoding operations of the source coder (432), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0079] The output of all the above mentioned functional units can be entropy coded in an entropy coder (437), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0080] The transmitter (411) can buffer the coded video sequence created by the entropy coder (437) in preparation for transmission over the network. The transmitter (411) can merge the coded video data from the video coder (4303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown). In some examples, the coded video bitstream (443) and the overlay content bitstream (444) are multiplexed into a single bitstream. In another example, metadata may be included in an SEI message associated with the coded video bitstream (443).

[0081] The controller (431) can perform various management operations for the video encoder (430). During encoding, the controller (431) can assign a particular coded picture type to each coded picture, which can affect the coding technique that can be applied to each picture. For example, pictures are often assigned as one of the picture types, such as an I-picture, a P-picture, a B-picture, etc.

[0082] An intraframe picture (I-picture) may be one that can be coded and decoded without using any other picture in the sequence as a source of prediction. Some video codecs allow different types of intraframe pictures, such as Independent Decoder Refresh ("IDR") pictures.

[0083] A predicted image (P-image) may be one that can be encoded and decoded using intra-frame or inter-frame prediction, which predicts the sample values ​​of each block using at most one motion vector and reference index.

[0084] Bidirectionally predicted images (B-pictures) may be those that can be encoded and decoded using intra-frame or inter-frame prediction, which predicts the sample values ​​of each block using at most two motion vectors and reference indices. Similarly, multiple predicted images can use more than one reference image and associated metadata to reconstruct a single block.

[0085] A source image is generally spatially subdivided into a number of sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and can be coded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective image of the block. For example, blocks of an I image may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same image (spatial prediction or intraframe prediction). Pixel blocks of a P image may be predictively coded via spatial prediction or via temporal prediction with reference to the previous coded reference image. Blocks of a B image may be predictively coded via spatial prediction or via temporal prediction with reference to the previous or two coded reference images.

[0086] The video encoder (430) may perform encoding operations according to a given video coding technique or standard, in which the video encoder (430) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence, such that the coded video data may conform to a syntax specified by the video coding technique or standard used.

[0087] In one embodiment, the transmitter (411) can transmit additional data along with the coded video. The source coder (432) can include such data as part of the encoded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures or slices, SEI messages, VUI parameter set fragments, etc.

[0088] A video (or video sequence, raw video content) can be captured as multiple source images (video pictures) in a time sequence. Intraframe image prediction (often shortened to intraframe prediction) exploits spatial correlation in a given image, while interframe image prediction exploits correlation (temporal or other) between images. In one example, a particular image being encoded / decoded, called the current image, is divided into blocks. If a block of the current image is similar to a reference block in a previously coded and still buffered reference image in the video, the block of the current image can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference image, and can have a third dimension that identifies the reference image if multiple reference images are used.

[0089] In some embodiments, bidirectional prediction techniques can be used for interframe image prediction. According to bidirectional prediction techniques, two reference images are used, such as a first reference image and a second reference image, both of which are before the current image in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current image can be coded by a first motion vector that points to a first reference block in the first reference image and a second motion vector that points to a second reference block in the second reference image. A block can be predicted by a combination of the first reference block and the second reference block.

[0090] Furthermore, to improve coding efficiency, merge mode techniques can be used in inter-frame image prediction.

[0091] According to some embodiments of the present disclosure, prediction, such as inter-frame image prediction or intra-frame image prediction, is performed on a block-by-block basis. For example, according to the high efficiency video coding (HEVC) standard, images in a sequence of video images are divided into coding tree units (CTUs) for compression, and the CTUs in an image have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively divided into one or more coding units (CUs) in a quadtree. For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU, four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter-frame prediction type or an intra-frame prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Usually, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in encoding / decoding is performed in units of a prediction block. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0092] 5 shows a diagram of a decoder (580) according to some embodiments of the present disclosure. The video decoder (580) is configured to receive coded images that are part of a coded video sequence and decode the coded images to generate reconstructed images. In one example, the video decoder (580) is used in place of the video decoder (380) of FIG. 3.

[0093] In the example of FIG. 5, the video decoder (580) includes an entropy decoder (582), an interframe decoder (585), a residual decoder (583), a reconstruction module (586), and an intraframe decoder (584), coupled together as shown in FIG. 5.

[0094] The entropy decoder (582) may be configured to reconstruct from the coded image certain symbols that represent syntax elements that make up the coded image. Such symbols may include, for example, a mode for coding the block (e.g., intra mode, inter mode, bi-predictive mode, merged submode of the latter two or another submode), prediction information (e.g., intra prediction information or inter prediction information, etc.) that may identify certain samples or metadata used for prediction by the intra decoder (584) or the inter decoder (585), respectively, and residual information, for example in the form of quantized transform coefficients. In one example, if the prediction mode is an inter prediction mode or a bi-predictive mode, the inter prediction information is provided to the inter decoder (585). And if the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (584). The residual information may undergo inverse quantization and be provided to the residual decoder (583). The metadata may be provided to the controller (581).

[0095] The inter-frame decoder (585) is configured to receive the inter-frame prediction information and to generate an inter-frame prediction result based on the inter-frame prediction information.

[0096] The intraframe decoder (584) is configured to receive the intraframe prediction information and to generate a prediction result based on the intraframe prediction information.

[0097] The residual decoder (583) is configured to perform inverse quantization to extract inverse quantized transform coefficients, and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (583) may also require certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (582) (data path not shown as this is only low volume control information).

[0098] The reconstruction module (586) is configured to combine, in the spatial domain, the residual as output by the residual decoder (583) and the prediction result (possibly as output by an inter-frame prediction module or an intra-frame prediction module) to form a reconstructed block, which may be part of a reconstructed image, which may then be part of a reconstructed video. It should be noted that it may perform other suitable operations, such as a deblocking operation, to improve visual quality. The reconstructed image may be buffered, for example, in a buffer (587), which may provide a reference image to the inter-frame decoder (585).

[0099] In some embodiments, the controller (581) can receive the metadata and configure decoding parameters of the decoder (580) based on the metadata. In some examples, the controller (581) can provide control signals to the intraframe decoder (584) and / or the interframe decoder (585) to control prediction based on the metadata. In another example, the controller (581) can decode overlay content from the metadata and provide the overlay content to the reconstruction module (586). Thus, the reconstruction module (586) can reconstruct the image using the overlay content.

[0100] FIG. 6 shows a flow chart outlining a process (600) according to one embodiment of the present disclosure. The process (600) may be used to encode a video sequence in a cloud gaming system, such as the cloud gaming system (100), the cloud gaming system (300), etc. In various embodiments, the process (600) is performed by a processing circuit, such as the processing circuit (120) in the server (110), the processing circuit (320) in the server (310), the video encoder (130), the video encoder (330), the video encoder (430), etc. In some embodiments, the process (600) is implemented in software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit performs the process (600). The process starts at (S601) and proceeds to (S602).

[0101] At (S610), a video sequence and metadata associated with the video sequence are received. The video sequence includes a series of image frames generated in response to game control information. The metadata is determined based on the game control information and is indicative of the game control information. In one example, a game engine, such as game engine (325), generates game content, such as video content, audio content, etc., based on game control information, such as game scene control information, user information, etc. The video content may be generated as a video sequence including a series of image frames. Further, metadata associated with the video sequence may be identified based on the game control information. The video sequence and metadata may be provided to a video encoder, such as video encoder (330). The video encoder (330) receives the video sequence and metadata associated with the video sequence.

[0102] At (S620), an encoding configuration is determined based on the metadata. In one embodiment, the metadata includes pixel-level or block-level motion vectors. Based on the motion vectors, a starting point for a motion search for encoding a block within an image frame of the video sequence can be determined.

[0103] In another embodiment, the metadata indicates at least one of image features of a scene change feature, a lighting change feature, a blinking feature, a panning feature, a zooming feature, a fading feature, and an overlay feature of an image frame of the video sequence. Based on the image features, an encoding configuration (e.g., a coding tool, a parameter set, a configuration setting, etc.) for encoding the image frame may be determined.

[0104] In another embodiment, the metadata indicates a bounding box in the image frame, and based on the bounding box a reference area in a previous image frame for an area not covered by the bounding box in the current image frame for an image frame previous to the current image frame can be determined, and based on the reference area in the previous image frame the area not covered by the bounding box in the current image frame can be encoded.

[0105] In another embodiment, the metadata indicates a model transformation (e.g., a change in motion blur effect, a change in lens blur effect, a change in lens distortion, a change in lens vignetting, etc.) from the first image frame to the second image frame. An encoding configuration (e.g., a filter tool, a parameter set, a configuration setting, etc.) may be determined based on the model transformation. Based on the encoding configuration, blocks in the second image frame may be encoded based on reference blocks in the first image frame.

[0106] In another embodiment, the metadata indicates the presence of an object of the current image frame in a subsequent image frame, and a buffer configuration (e.g., buffer the reconstructed area containing the object for encoding / decoding the subsequent image frame) may be determined based on the metadata.

[0107] In another embodiment, the metadata indicates a noise level of the image frame. A pre-processing or post-processing configuration for the image frame may then be determined based on the noise level. In one example, the image frame may be denoised in pre-processing based on the pre-processing configuration. In another example, noise may be added to the reconstructed frame in post-processing based on the post-processing configuration.

[0108] In another embodiment, the metadata indicates a region of interest in the image frame. In some examples, a first encoding configuration having a higher quality (e.g., higher resolution, higher bit rate) may be determined for encoding the region of interest, and a second encoding configuration having a lower quality than the first encoding configuration may be determined for encoding an area outside the region of interest.

[0109] In another embodiment, the metadata indicates a graphic overlay in a region of the image frame, and an encoding configuration for encoding the region based on the graphic overlay may be determined. In one example, a skip mode may be used to code certain graphic overlays, such as logos, to improve coding efficiency.

[0110] At (S630), the video sequence is encoded into a coded video bitstream based on the coding configuration. The coded video bitstream may be transmitted over a network to a destination device, such as the client device (360). In some embodiments, metadata is also transmitted to the destination device to aid in decoding the coded video bitstream at the destination device. In one embodiment, the metadata is transmitted in an SEI message along with the coded video bitstream. In another embodiment, the metadata includes overlay content. The overlay content may be encoded into an overlay content bitstream. The overlay content bitstream and the coded video bitstream may be multiplexed into a single output bitstream. The single output bitstream may be transmitted over a network to the destination device.

[0111] In another embodiment, the metadata indicates a message box in the image frame for the overlay message. The overlay message and information of the overlay message may be compressed and included in a specific NALU of the coded video bitstream. The coded video bitstream may include a plurality of NALUs (e.g., packets) for carrying the coded video data and may include a specific NALU for carrying the overlay message. The coded video bitstream may be transmitted to a destination device over a network. The process then proceeds to (S699) and ends.

[0112] FIG. 7 shows a flow chart outlining a process (700) according to one embodiment of the disclosure. The process (700) may be used to decode a video sequence in a cloud gaming system, such as the cloud gaming system (100), the cloud gaming system (300), etc. In various embodiments, the process (700) is performed by a processing circuit, such as the processing circuit (170), the processing circuit (370), the video decoder (180), the video decoder (380), the video decoder (480), etc. In some embodiments, the process (700) is implemented with software instructions, and thus, when the processing circuit executes the software instructions, the processing circuit performs the process (700). The process begins at (S701) and proceeds to (S710).

[0113] At (S710), a coded video bitstream and metadata associated with the coded video bitstream are received, the metadata being determined based on game control information that is also used to generate video content for a game server, such as server (310).

[0114] At (S720), a decoding configuration and a reconstruction configuration are determined based on the metadata.

[0115] In one embodiment, the metadata indicates a noise level of the image frame, and a post-processing configuration (configuration in the reconstruction configuration) for the image frame may be determined based on the noise level. In one example, the post-processing configuration may add noise of the noise level to the reconstructed image frame.

[0116] At (S730), the coded video bitstream is decoded based on the decoding configuration, and the video sequence is reconstructed based on the reconstruction configuration. The process then proceeds to (S799) and ends.

[0117] In some embodiments, the coded video bitstream includes NALUs for carrying video content and specific NALUs for carrying overlay content. The specific NALUs can be identified and decoded separately (e.g., by a different decoder) to generate the overlay content. The video sequence can then be reconstructed with the overlay content.

[0118] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 8 illustrates a computer system (800) suitable for implementing certain embodiments of the disclosed subject matter.

[0119] Computer software may be encoded using any suitable machine code or computer language and may undergo assembly, compilation, linking, or similar mechanisms to create code containing instructions, which may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or may be executed by code interpretation, microcode execution, etc.

[0120] The instructions may be executed in various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, internet of things devices, and the like.

[0121] The components of the computer system (800) illustrated in Figure 8 are exemplary in nature and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of the computer system (800).

[0122] The computer system (800) may include a number of human interface input devices. Such human interface input devices may respond to input by one or more users through tactile input (e.g., keystrokes, swipes, data glove movements, etc.), audio input (e.g., voice, clapping, etc.), visual input (e.g., gestures, etc.), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds, etc.), images (e.g., scanned images, photographic images obtained from still image cameras, etc.), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic vision, etc.).

[0123] The human interface input devices may include one or more of a keyboard (801), a mouse (802), a trackpad (803), a touch screen (810), a data glove (not shown), a joystick (805), a microphone (806), a scanner (807), and a camera (808) (only one of each is shown).

[0124] The computer system (800) may also include several human interface output devices. Such human interface output devices may stimulate one or more of the user's senses, for example, by haptic output, sound, light, and smell / taste. Such human interface output devices may be haptic output devices (e.g., touch screens (810), haptic feedback via data gloves (not shown) or joysticks (805), but may also be haptic feedback devices that do not act as input devices), audio output devices (e.g., speakers (809), headphones (not shown)), visual output devices (e.g., screens (810), including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have touch screen input capabilities, each of which may or may not have haptic feedback capabilities, some of which may output two-dimensional visual output or three or more dimensional visual output, for example, by stereographic output, virtual reality glasses (not shown), holographic displays and smoke tanks (not shown), and printers (not shown).

[0125] The computer system (800) may include human accessible storage devices and their associated media, such as optical media or similar media (821), including CD / DVD ROM / RW (820) with CDs / DVDs, thumb drives (822), removable hard drives or solid state drives (823), legacy magnetic media such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0126] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitory signals.

[0127] The computer system (800) may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local networks, wide area networks, metropolitan area networks, vehicular and industrial networks, real-time networks, delay tolerant networks, and the like. Examples of networks include LANs such as Ethernet, wireless LANs, cellular networks (GSM, 3G, 4G, 5G, LTE, and the like), television cable or wireless wide area digital networks (including cable television, satellite television, terrestrial broadcast television), vehicular and industrial networks (including CANBus), and the like. Some networks generally require an external network interface adapter connected to some general-purpose data port or peripheral bus (849) (e.g., a USB port on the computer system (800)), while other systems are typically integrated into the core of the computer system (800) by connecting to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (800) may communicate with other entities. Such communication may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., from the Canbus to a particular Canbus device), or two-way, e.g., to other computer systems using local or wide area digital networks. As mentioned above, specific protocols and protocol stacks may be used for each of these networks and network interfaces.

[0128] The above-mentioned human interface devices, human accessible storage devices, and network interfaces can be connected to the core (840) of the computer system (800).

[0129] The cores (840) may include one or more central processing units (CPUs) (841), graphics processing units (GPUs) (842), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (843), hardware accelerators (844) for specific tasks, etc. These devices may be connected via a system bus (848), along with read-only memory (ROM) (845), random access memory (846), internal mass storage (847), e.g., an internal non-user-accessible hard disk drive, SSD, etc. In some computer systems, the system bus (848) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (1348) or may be connected via a peripheral bus (849). Peripheral bus architectures include Peripheral to External Controller Interface (PCI), Universal Serial Bus (USB), etc.

[0130] The CPU (841), GPU (842), FPGA (843), and accelerator (844) can execute some instructions, which can be combined to constitute the above-mentioned computer code. The computer code can be stored in ROM (845) or RAM (846). Also, temporary data can be stored in RAM (846), while permanent data can be stored in, for example, internal mass storage (847). The use of cache memory, which can be closely associated with one or more of the CPU (841), GPU (842), mass storage (847), ROM (845), RAM (846), etc., allows for fast storage and retrieval from any memory device.

[0131] The computer-readable medium can have computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be any medium and code known and available to those skilled in the computer software arts.

[0132] By way of example and not limitation, a computer system having the architecture (800), and in particular the core (840), may provide functionality as a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible, computer-readable media. Such computer-readable media may be media related to the user-accessible mass storage described above, and may be specific storage with the core (840), such as the core internal mass storage (847) or ROM (845), which is non-volatile. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (840). The computer-readable media may include one or more memory devices or chips, depending on the particular needs. This software may cause the core (840), and in particular the processor therein (including CPU, GPU, FPGA, etc.) to perform certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM (846) and modifying such data structures according to the software-defined processes. Additionally or alternatively, the computer system may provide functionality because of logic hardwired or otherwise incorporated into circuitry (e.g., accelerator (844)) that may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. Where appropriate, references to software may include logic, and vice versa. Where appropriate, references to computer-readable media may include circuitry (such as integrated circuits (ICs)) that store software to be executed, circuitry embodying logic to be executed, or both. The present disclosure encompasses any appropriate combination of hardware and software.

[0133] While this disclosure has described several exemplary embodiments, there are modifications, arrangements, and various equivalent substitutions which are within the scope of the disclosure. It should thus be understood that those skilled in the art will be able to design various systems and methods which, although not explicitly shown or described herein, embody the principles of the disclosure and are therefore within the spirit and scope of the disclosure. [Explanation of symbols]

[0134] 100 Cloud Gaming System 101 Server System 102 Client Systems 110 Server 111 Interface circuit 115 Memory 120 Processing circuit 125 Game Engine 130 Video Encoder 150 Network 160 client devices 161 Interface Circuit 165 Display 169 User Input Devices 170 Processing circuit 175 Game Clients 180 Video Decoder

Claims

1. A method for cloud gaming executed by an electronic device, comprising: receiving a video sequence and metadata associated with the video sequence from a game engine, the metadata being created based on game control information relating to a game environment, the game control information including user control information and game scene information, the metadata being updated in the game engine based on the user control information being changed in response to user input, and the game scene information being updated in the game engine based on the game scene information being changed in response to game scene movement, the video sequence including a series of image frames generated in response to the game control information; encoding the video sequence into a coded video bitstream based on the metadata; The method according to claim 1, further comprising:

2. The step of encoding the video sequence into a coded video bitstream based on the metadata comprises: determining, based on the metadata, an encoding configuration for encoding the video sequence; encoding the video sequence into a coded video bitstream based on the encoding configuration; 2. The method of claim 1, comprising:

3. The metadata includes a motion vector, and the method further comprises: determining, based on the motion vector, a starting point for a motion search for encoding a block within an image frame of the video sequence; 2. The method of claim 1 .

4. The metadata further indicates at least one image feature of a scene change feature, a lighting change feature, a blinking characteristic, a panning feature, a zooming feature, a fading feature, and an overlay feature of an image frame of the video sequence, and the method further comprises: encoding the video sequence into a coded video bitstream based on the image features.

2. The method of claim 1 .

5. The method according to claim 1, wherein the metadata indicates a bounding box for an image frame, the method further comprising: determining a reference area in a previous image frame for an area not covered by a bounding box in the current image frame for a previous image frame of the current image frame based on the bounding box; and encoding an area not covered by a bounding box in the current image frame based on a reference area in the previous image frame.

2. The method of claim 1 .

6. The metadata indicates a model transformation from a first image frame to a second image frame, the method further comprising: determining an encoding structure for encoding the video sequence based on the model transformation; and encoding the block in the second image frame based on the encoding configuration and based on a reference frame in the first image frame.

2. The method of claim 1 .

7. The method according to claim 1, wherein the metadata indicates a presence of an object of a current image frame in a subsequent image frame, the method further comprising: determining a buffer configuration based on the metadata; 2. The method of claim 1 .

8. The method according to claim 7, wherein the metadata indicates a noise level of an image frame, the method further comprising: pre-processing / post-processing the image frames based on the noise level.

2. The method of claim 1 .

9. The method according to claim 8, wherein the metadata indicates an area of ​​interest in an image frame, the method further comprising: encoding the region of interest using a first encoding scheme having a higher quality than a second encoding scheme used to encode blocks outside the region of interest; 2. The method of claim 1 .

10. The metadata indicates a graphic overlay in a region of an image frame; The step of encoding the video sequence into a coded video bitstream based on the metadata further comprises: determining an encoding scheme for encoding said region based on said graphic overlay; and encoding the video sequence into a coded video bitstream based on the encoding configuration.

2. The method of claim 1 .

11. The method according to claim 10, wherein the metadata indicates a message box in an image frame for an overlay message, the method further comprising: including information of the overlay message in a specific network abstraction layer unit (NALU) of the coded video bitstream; 2. The method of claim 1 .

12. A device for cloud gaming, comprising: A processing circuit is provided, the processing circuit comprising: receiving a video sequence and metadata associated with the video sequence from a game engine, the metadata being created based on game control information relating to a game environment, the game control information including user control information and game scene information, the metadata being updated in the game engine based on the user control information being changed in response to user input, and the game scene information being updated in the game engine based on the game scene information being changed in response to game scene movement, the video sequence including a series of image frames generated in response to the game control information; encoding the video sequence into a coded video bitstream based on the metadata. An apparatus comprising:

13. The method of claim 12, further comprising: encoding the video sequence into a coded video bitstream based on the metadata, the method comprising: determining an encoding configuration for encoding the video sequence based on the metadata; and encoding the video sequence into a coded video bitstream based on the encoding structure; 13. The apparatus of claim 12, comprising:

14. The metadata includes a motion vector, and the processing circuitry further comprises: determining, based on the motion vector, a starting point for a motion search for encoding a block in an image frame of the video sequence.

13. The apparatus of claim 12.

15. The method of claim 1, wherein the metadata further indicates at least one of image features of a scene change feature, a lighting change feature, a blinking characteristic, a panning feature, a zooming feature, a fading feature, and an overlay feature of an image frame of the video sequence, and the processing circuitry further comprises: configured to perform encoding the video sequence into a coded video bitstream based on the image features.

13. The apparatus of claim 12.

16. The metadata is indicative of a bounding box for an image frame, the processing circuitry further comprising: determining a reference area in a previous image frame for an area not covered by a bounding box in the current image frame for a previous image frame of the current image frame based on the bounding box; and encoding an area not covered by a bounding box in the current image frame based on a reference area in the previous image frame.

13. The apparatus of claim 12.

17. The metadata indicates a model transformation from a first image frame to a second image frame, the processing circuitry further comprising: determining an encoding structure for encoding the video sequence based on the model transformation; and and encoding the block in the second image frame based on a reference frame in the first image frame based on the encoding configuration.

13. The apparatus of claim 12.

18. The method of claim 17, wherein the metadata indicates a presence of an object of a current image frame in a subsequent image frame, the processing circuitry further comprising: determining a buffer configuration based on the metadata.

13. The apparatus of claim 12.

19. The metadata is indicative of a noise level of an image frame, the processing circuitry further comprising: and performing pre-processing / post-processing of the image frames based on the noise level.

13. The apparatus of claim 12.

20. The metadata indicates a region of interest in an image frame, the processing circuitry further comprising: encoding the region of interest using a first encoding scheme having a higher quality than a second encoding scheme used to encode blocks outside the region of interest.

13. The apparatus of claim 12.

21. A computer program comprising: The computer program, when loaded and executed by an electronic device, causes the electronic device to carry out the method according to any one of claims 1 to 11. A computer program comprising:

Images