Method, device, electronic device and computer-readable storage medium for cloud gaming

Abstract

Aspects of the present disclosure provide methods, devices, electronic devices, and computer-readable storage media for cloud gaming. In some examples, the device for cloud gaming includes a processing circuit. For example, the processing circuit receives a video sequence and metadata associated with the video sequence. The video sequence includes a sequence of picture frames generated in response to game control information, and the metadata indicates the game control information. The processing circuit can configure encoding parameters based on the metadata indicating the game control information. Then, the processing circuit can encode the video sequence into an encoded video stream based on the encoding parameters.

Description

[0001] Incorporation by Reference

[0002] This application claims priority to U.S. Patent Application No. 17 / 176,502, filed on February 16, 2021, entitled "Method and Apparatus for Cloud Gaming," which claims priority to U.S. Provisional Application No. 62 / 990,193, filed on March 16, 2020, entitled "Video Encoding and Decoding Method in Cloud Gaming System." The entire disclosure of the prior application is hereby incorporated by reference in its entirety. Technical Field

[0003] This disclosure describes embodiments generally relating to multimedia techniques in the field of cloud gaming. Background Art

[0004] The background description provided herein is intended to generally present the context of the present disclosure. The work of the inventors currently named, to the extent that the work is described in this background section, and aspects of the description that may not constitute prior art at the time of filing, are neither explicitly nor implicitly admitted to be prior art to the present disclosure.

[0005] Cloud gaming services are trending online services available from multiple service providers. Cloud gaming can also be referred to as on-demand gaming. In a cloud gaming system, games are run on a remote server and are referred to as cloud-based games. Users can play cloud-based games via user devices. Game content can be generated at a remote server and streamed and displayed on a user device. Summary of the invention

[0006] Aspects of the present disclosure provide methods and devices for cloud gaming. In some examples, the device for cloud gaming includes a processing circuit. For example, the processing circuit receives a video sequence and metadata associated with the video sequence. The video sequence includes a sequence of picture frames generated in response to game control information, and the metadata indicates the game control information. The processing circuit can configure encoding parameters based on the metadata indicating the game control information. Then, the processing circuit can encode the video sequence into an encoded video stream based on the encoding parameters.

[0007] In some embodiments, the metadata includes at least one of camera information, color information, lighting information, motion information, view information, overlay information, and user control information.

[0008] In an embodiment, the processing circuit receives a signal carrying user control information via a network and generates a video sequence in response to the user control information.

[0009] In some examples, the processing circuit determines motion information of at least one of a global motion of the game scene and a local motion of an object in a bounding box within the game scene based on the game scene control information and the user control information.

[0010] In some embodiments, the processing circuit determines a motion model for encoding the video sequence based on metadata indicating the game control information. The motion model may be one of an affine model, a perspective model, a rotation model, and a scaling model.

[0011] In some examples, the metadata may be transmitted to a destination device of the encoded video stream via an interface circuit and a network. In an example, the metadata is transmitted with the encoded video stream in a Supplemental Enhancement Information (SEI) message.

[0012] In another example, the metadata includes the overlay content. The processing circuit encodes the overlay content into an overlay content bitstream, and multiplexes the encoded video bitstream and the overlay content bitstream into an output bitstream. The output bitstream can then be transmitted.

[0013] In some other examples, a device for cloud gaming includes a processing circuit that receives an encoded video stream and metadata associated with the encoded video stream. The metadata indicates game control information. The processing circuit configures decoding parameters and reconstruction parameters based on the metadata. The processing circuit then decodes the encoded video stream based on the decoding parameters and reconstructs the video sequence based on the reconstruction parameters. In the example, the metadata includes overlay content. The processing circuit demultiplexes the input stream into an encoded video stream and an overlay content stream. The overlay content stream is decoded to generate overlay content. The overlay content can then be used to reconstruct the video sequence.

[0014] Aspects of the present disclosure also provide a non-volatile computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for cloud gaming. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0016] Figure 1 A block diagram of a cloud gaming system according to an embodiment of the present disclosure is shown.

[0017] Figure 2 A schematic diagram of a cloud gaming scenario according to some embodiments of the present disclosure is shown.

[0018] Figure 3 A block diagram of a cloud gaming system according to some embodiments of the present disclosure is shown.

[0019] Figure 4 A block diagram of a video encoder according to an embodiment of the present disclosure is shown.

[0020] Figure 5 A diagram of a video decoder according to some embodiments of the present disclosure is shown.

[0021] Figure 6 A flowchart outlining an example of a process according to an embodiment of the present disclosure is shown.

[0022] Figure 7 A flow chart outlining another process example according to an embodiment of the present disclosure is shown.

[0023] Figure 8 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION

[0024] Various aspects of the present disclosure provide systems, devices, and methods for metadata-assisted video codec (encoding / decoding) design in the field of cloud gaming.

[0025] Figure 1 A block diagram of a cloud gaming system (100) according to an embodiment of the present disclosure is shown. The cloud gaming system (100) includes a server system (101) and a client system (102) connected via a network (150). The server system (101) includes one or more servers (110) coupled together. For example, the server system (101) may include one or more data centers, one or more server farms, etc. The server system (101) can present game content and compress the game content into a code stream. The code stream can be delivered to the client system (102) via the network (150). The client system (102) includes one or more client devices (160) that can interact with game users (also referred to as players) respectively. For example, the client device (160) can receive user control information from a game user, and can provide output to the game user, such as video output, audio output.

[0026] The servers (110) may be implemented using any suitable technology. Figure 1 In the example of , the server (110) includes a processing circuit (120), a memory (115), and an interface circuit (111) coupled together. The memory (115) is configured to store software instructions and various data (such as media data, configuration information, control information, etc.) before, during, and after processing.

[0027] The processing circuit (120) may include any suitable processing circuit, such as one or more central processing units (CPUs), one or more graphics processing units (GPUs), application specific integrated circuits, etc. Figure 1 In the example of , the processing circuit (120) can be configured to include a game engine (125). For example, one or more CPUs and / or GPUs can execute game software stored in the memory (115) to serve as the game engine (125). Further, the processing circuit (120) can be configured to include an encoder, such as a video encoder, an audio encoder, etc. Figure 1 In an example, the processing circuit (120) is configured to include a video encoder (130). For example, one or more CPUs and / or GPUs may be configured to function as the video encoder (130). In another example, the video encoder (130) may be implemented using an application specific integrated circuit.

[0028] The interface circuit (111) can connect the server (110) to the network (150). The interface circuit (111) can include a receiving part for receiving signals from the network (150) and a transmitting part for transmitting signals to the network (150). For example, the interface circuit (111) can transmit a signal carrying encoded media data (such as an encoded video stream, an encoded audio stream, etc.) to other devices (such as a client device (160)) via the network (150). The interface circuit (111) can receive a signal carrying user information such as user input from the client device (160) via the network (150).

[0029] The network (150) is suitably coupled to the server system (101) and the client system (102) via wired and / or wireless connections, such as Ethernet connections, fiber optic connections, WiFi connections, cellular network connections, etc. The network (150) may include network server devices, storage devices, network devices, etc. The components of the network (150) are suitably coupled together via wired and / or wireless connections.

[0030] The client device (160) is configured to interact with a user. Figure 1 In the example of , the client device (160) includes various components coupled together, such as an interface circuit (161), a processing circuit (170), a display (165), and a user input device (169).

[0031] The interface circuit (161) can interface the client device (160) with the network (150). The interface circuit (161) can include a receiving part that receives signals from the network (150) and a transmitting part that transmits signals to the network (150). For example, the interface circuit (161) can receive a signal carrying data (such as a signal carrying an encoded video stream, a signal carrying an audio stream, etc.) sent by the server (110) from the network (150), and can transmit the signal (such as a signal carrying user input, etc.) to other devices (such as the server (110)) via the network (150).

[0032] The user input device (169) may receive user input, and the user input may be appropriately sent to the server system (101). In an example, the user input may be provided to the processing circuit (170), and the processing circuit (170) may convert the user input into an appropriate form, such as a message, a packet, etc., and send the user input via the interface circuit (161). The user input may be sent to the server system (101) via the network (150). The user input device (169) may be any suitable user input device, such as a joystick, a motion sensor, a microphone, a keyboard, a mouse, a touch screen, a touch pad, etc.

[0033] The processing circuit (170) may include appropriate processing circuits, such as a CPU, a GPU, an application specific integrated circuit, etc. The processing circuit (170) may be configured to include one or more media decoders, such as a video decoder, an audio decoder, etc. For example, the processing circuit (170) is configured to include a video decoder (180). In an example, the GPU is configured as a video decoder (180). In another example, the CPU may execute software instructions to function as a video decoder (180). The video decoder (180) may decode information in an encoded video stream and reconstruct a video sequence (e.g., a sequence of picture frames).

[0034] Further, the processing circuit (170) can be configured to include a game client (175). For example, the CPU can execute cloud gaming client software to serve as the game client (175). The game client (175) can receive user input (171) and send data representing the user input (171) to the game engine (125) in the server (110).

[0035] The display ( 165 ) may be any suitable display device, such as a television, a smart phone, a wearable display, a head mounted device, etc. In an example, the display ( 165 ) may receive a video sequence and display the video sequence appropriately.

[0036] In some examples, in order to play the game, the game engine (125) can generate game content for the game, such as video sequences, audio, etc. depicting various game environments of the game. In an example, the game engine (125) can generate video content (122) (e.g., a video sequence) and provide the video content (122) to a video encoder (130). The video encoder (130) can compress the video content (122) into an encoded video stream (123), and the encoded video stream (123) can be transmitted by the interface circuit (111) and streamed to the client device (160) via the network (150). At the client device (160), the interface circuit (161) can receive a signal carrying the encoded video stream and provide the encoded video stream (172) to a video decoder (180). The video decoder (180) can reconstruct the video sequence from the encoded video stream (172), and the reconstructed video sequence (173) can be provided to a display (165) to provide a game environment to a game user.

[0037] The game user can interact with the game client (175). For example, the game client (175) can receive user control information (also referred to as interaction data in some examples), such as Figure 1 , and the like. The game engine (130) may receive user input (171) as shown in FIG. 1 . User control information (such as commands, position information, depth information, motion information, etc.) may be transmitted from the game client (175) to the game engine (125) via the interface circuit (161), the network (150), and the interface circuit (111) (the user control information is shown by (121) at the server (110)). Then, in response to the received user control information (121), the game engine (130) may perform operations and generate updated game content. The updated game content may be compressed and provided from the server (110) to the client device (160). When the game continues, the game process repeats.

[0038] In some embodiments, the game engine (125) can generate game video content in the form of raw video content (122), such as a sequence of picture frames in a YUV format. The raw video content (122) can be provided to a video encoder (130). The video encoder (130) compresses the raw video content and generates an encoded video stream (123) that complies with a specific video encoding format or standard such as H.264, H.265, AV1, AVS2, etc. The encoded video stream (123) is sent to a client device (160) and can be decoded by a video decoder of the client using a format or standard consistent with the video encoder (130) for display.

[0039] The cloud gaming system (100) provides a way to play video games by running the game software on a remote server, as opposed to a game console or other local device in the related examples. For example, a gaming company owns a server (110) that is a dedicated server for game streaming, and the server (110) can be much more powerful than the hardware on the consumer side (e.g., client device). Further, in some examples, the streaming service can send similar data to multiple client devices (160) where multiple players (users) are playing the same game. Therefore, the total amount of operating costs (for the gaming company) that multiple players can play the same game is less than the total amount of operating costs expected in the related examples, where the cost is proportional to the number of players.

[0040] Figure 2 A diagram (200) illustrating a cloud gaming scenario according to some embodiments of the present disclosure is shown. Figure 2 In the example of FIG. 2 , the server ( 210 ) is a cloud gaming server that provides gaming services for the game, and the client devices 260A-260C are client devices that interact with players AC of the game, respectively. The server ( 210 ) can communicate with Figure 1 The server (110) in FIG. 1 is configured similarly. The client devices 260A-260C can respectively communicate with Figure 1 The client device (160) in is similarly configured.

[0041] exist Figure 2 In the example of FIG. 1 , a game engine (not shown) in the server ( 210 ) renders game content for a game environment and provides the game content to client devices ( 260A )-( 260C ) via a network (not shown). At some point, as Figure 2 As shown in (201) of FIG. 1 , the client device (260B) receives user control information, such as user input from player B. Figure 2 As shown in (202) in FIG. 1 , the client device 260B sends data representing the user input to the server (210) via the network. In response to the data representing the user input, as shown in FIG. Figure 2 As shown in (203) in FIG. 1 , the game engine in the server (210) operates the game environment and updates it. Then, as shown in FIG. Figure 2 As shown in (204) in FIG. 1 , the server (210) sends updated game content to the client devices (260A)-(260C). Figure 2 As shown in (205), each of the client devices (260A)-(260C) can refresh the game environment in response to the updated game content, such as refreshing the display.

[0042] In a related example, the client device may be a game console or PC, which is required to play most games and constitutes a high cost of the game. The cost of a game console or PC can be from a few thousand to tens of thousands of dollars. The local machine (game console or PC) also wears out over time and has reduced performance. As technology develops, cloud gaming servers are constantly upgraded at no additional cost to the consumer.

[0043] According to some aspects of the present disclosure, cloud gaming may require high and reliable bandwidth requirements and low latency requirements for two-way communication in order to achieve a good user experience. In some examples, cloud gaming may require a fast and stable Internet connection and close proximity to the game server in order to have high and reliable bandwidth. For example, a streaming service for cloud gaming may require a bandwidth between 10Mbps and 35Mbps. The present disclosure provides techniques for improving encoding speed and encoding efficiency in order to reduce bandwidth requirements and improve visual quality.

[0044] In some examples, high latency can be detrimental to the gaming experience. Latency can be caused by many factors, and video encoding for streaming media is one of the factors. The present disclosure also provides techniques for reducing encoding time while maintaining encoding efficiency and thus improving the cloud gaming experience.

[0045] Various aspects of the present disclosure provide techniques for metadata-assisted video encoding design in a cloud gaming system. In some examples, metadata of video content (e.g., raw video) refers to data that provides information about the video content (such as camera information, color information, lighting information, motion information, view information, user control, etc.). The metadata together with the raw video can be presented by a game engine (e.g., game (125)) and can also be provided by the game engine as input to a video encoder (e.g., video encoder (130)) to assist in the encoding process. In some embodiments, the video encoder (e.g., video encoder (130)) can encode the raw video into an encoded video stream with the help of metadata. Further, in some embodiments, the metadata can be provided to a video decoder (e.g., video decoder (180)) together with the encoded video stream. The video decoder (e.g., video decoder (180)) can decode the encoded video stream with the help of metadata.

[0046] Figure 3A block diagram of a cloud gaming system (300) according to some embodiments of the present disclosure is shown. The cloud gaming system (300) operates similarly to the cloud gaming system (100) described above, and utilizes certain components that are the same or equivalent to those used in the cloud gaming system (100). For example, the server system (301) is configured similarly to the server system (101); the client system (302) is configured similarly to the client system (102); the network (350) is configured similarly to the network (150); the server (310) is configured similarly to the server (110); the processing circuit (320) is configured similarly to the processing circuit (120); the memory (315) is configured similarly to the memory (115); the interface circuit (311) is configured similarly to the interface circuit (111); the game engine (325) is configured similarly to the game engine ( 125); video encoder (330) is configured similarly to video encoder (130); interface circuit (361) is configured similarly to interface circuit (161); processing circuit (370) is configured similarly to processing circuit (170); display 365 is configured similarly to display (165); user input device 369 is configured similarly to user input device (169); game client (375) is configured similarly to game client (175); video decoder (380) is configured similarly to video decoder (180). Descriptions of these components have been provided above and will be omitted here for clarity.

[0047] Further, in Figure 3 In an example, a video encoder (330) is configured to encode video content into a video stream with the help of metadata. For example, the video encoder (330) includes a controller (331) that acts as a metadata-based controller. The game engine (325) provides the original video content (322) together with the metadata (326) to the video encoder (330). The metadata (326) includes information associated with the original video content (322), such as camera information, color information, lighting information, motion information, view information, and user control associated with the original video content (322). The controller (331) receives the metadata (326) and controls the configuration of the video encoder (330) based on the metadata (326) to encode the original video content (322) and generate an encoded video stream (323). The help of metadata improves encoding speed and efficiency, and can reduce bandwidth requirements and reduce latency.

[0048] In some embodiments, the encoded video stream (323) may be output along with metadata, as indicated by metadata output (327).

[0049] Further, in Figure 3In an example, a video decoder (380) is configured to decode a video stream with the help of metadata. For example, the video decoder (380) includes a controller (381) that acts as a metadata-based controller. The controller (381) receives metadata (377) and an encoded video stream (372). In the example, the encoded video stream (372) can be an encoded video stream (323) generated by a video encoder (330), and the metadata (377) can be metadata output (327). The controller (381) can control the configuration of the video decoder (380) based on the metadata (377) to decode the encoded video stream (372) and generate a reconstructed video sequence (373).

[0050] In some embodiments, the metadata (326) from the game engine (325) to the video encoder (330) may include any suitable game control information that can help improve encoding efficiency and / or reduce the complexity of the encoding process, such as game scene information (e.g., camera information, color information, lighting information, motion information, view information), user control information, etc.

[0051] Specifically, in some examples, user input (371) (also referred to as user control information) from a user input device (369) (such as a joystick, keypad, mouse, and / or keyboard) can be transmitted from a client device (360) to a server (310). In an example, a game engine (325) receives user control information (321) corresponding to the user input (371) at the client device (360). The game engine (325) can generate game scene content (such as video content, audio content, etc.) based on the user control information (321). In some embodiments, metadata (326) can be determined based on the game scene information and the user control information (321). For example, the global motion of the game scene and / or the local motion of the object (321) can be identified based on the game scene information and the user control information. In an example, the user control information (321) includes a change in viewing direction, so the global motion of the game scene can be identified based on the change in viewing direction. In another example, the user control information (321) includes the movement of the object, so the local motion of the object can be identified. In an example, the metadata (326) may include the global motion of the game scene, the local motion of objects, and the bounding boxes of objects. The metadata (326) may assist the video encoding process as additional input data to the video encoder (330). The metadata (326) may be fed to the video encoder (330) along with the original video content (322) rendered by the game engine (325). The video encoder (330) may utilize the information in the metadata (326) to efficiently perform a motion estimation / compensation process on the original video content (322).

[0052] In some embodiments, the metadata (326) can be fed to the video encoder (330) as configurable encoder settings (e.g., parameters of the video encoder (330), etc. In an example, the metadata (326) can be used to generate a global motion model (e.g., an affine model, a perspective model, a rotation model, a scaling model, etc.) for a picture or a picture region. In another example, the metadata (326) can be used to select encoding tools in the video encoder (330). However, in an example, when any of these parameters are not available, the video encoder (330) can assume a set of default values.

[0053] In some embodiments, metadata (326) fed into the video encoder (330) may be processed and sent along with the coded video stream (323) by metadata (327). The metadata (327) may assist the video decoder in decoding the coded video stream (323). For example, metadata (327) (such as timing information, overlay messages, overlay graphics, bounding boxes, etc.) that may assist in decoding and displaying the coded video stream (323) may be sent as a supplemental enhancement information (SEI) message. The SEI message may be delivered within the coded video stream or separately.

[0054] In some embodiments, the client device (360) receives the encoded video stream (372) and metadata (377). The metadata (377) can be used by the client device (360) to help decode or post-process the video sequence carried in the encoded video stream (372). For example, the metadata (377) can include timing information, overlay messages, overlay graphics, bounding box information, which can help decode and display the encoded video stream (372).

[0055] In some embodiments, metadata may be encoded and decoded (encoded / decoded) using an additional codec. In some examples, the game engine (325) may output separate content, video, audio, overlay messages, overlay graphics, etc. In an example, the overlay message (or overlay graphic) is considered metadata for the video content. The overlay message (or overlay graphic) may be compressed into a separate bitstream (second bitstream) using an encoder separate from the video encoder 330. Then, in an example, the encoded video bitstream for the video content and the second bitstream for the overlay message (or overlay graphic) may be multiplexed into a single bitstream.

[0056] Figure 4 A block diagram of a video encoder (430) according to an embodiment of the present disclosure is shown. The video encoder (430) is included in an electronic device (410). In an example, the electronic device (410) is a server device, such as Figure 3The electronic device (410) further includes a transmitter (411) (eg, a transmitting circuit, etc.). The video encoder (430) may be used to replace Figure 3 A video encoder (330) in an example of FIG.

[0057] The video encoder (430) can be obtained from, for example, Figure 3 In the example of FIG. 4 , the video source of the game engine (325) receives the raw video content. In addition, the video encoder (430) can receive metadata that provides information associated with the raw video content, such as camera information, color information, lighting information, motion information, view information, overlay information (e.g., overlay messages, overlay graphics, etc.), user controls, etc.

[0058] In some examples, the raw video content is in the form of a stream of digital video samples that may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any appropriate sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). The raw video content may be provided as a plurality of individual pictures that impart motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, where each pixel may include one or more samples, depending on the sampling structure, color space, etc. used.

[0059] According to an embodiment, the video encoder (430) may encode and compress pictures of the original video content (also referred to as a video sequence) into an encoded video stream (443). Figure 4 In an example of , the video encoder (430) includes a controller (431) configured to control the encoding configuration of the video encoder (430) based on metadata to improve encoding speed and efficiency. In some embodiments, the controller (431) controls other functional units as described below and is functionally coupled to the other functional units. For clarity, the coupling is not shown. It should be noted that the controller (431) can set various parameters, such as picture skipping, quantizer, lambda value of rate-distortion optimization technology, picture size, picture group (GOP) layout, maximum motion vector search range, motion model, etc.

[0060] In an example, the controller (431) is configured to determine a global motion model (such as an affine model, a perspective model, a rotation model, a scaling model, etc.) of the picture based on the metadata, and can configure encoding parameters accordingly. In another example, the controller (431) is configured to determine a region in a bounding box within the picture and a local motion model of the region based on the metadata. In some embodiments, the controller (431) includes an encoder (not shown) separated from other parts of the video encoder (430). The separate encoder can compress the metadata into compressed metadata. For example, the metadata includes overlay content, such as overlay messages and overlay graphics. The overlay content can be compressed into an overlay content stream (444). In an example, the overlay content stream (444) can be (for example) combined into a single stream with the encoded video stream (443) by a multiplexer (412). In another example, the controller 431 can include the metadata in an SEI message (443) associated with the encoded video stream. The controller (431) can be configured to have other appropriate functions related to the video encoder (430) optimized for a specific system design.

[0061] In some embodiments, the video encoder (430) is configured to operate in an encoding loop. As an oversimplified description, in an example, the encoding loop may include a source encoder (432) (e.g., responsible for creating symbols such as a symbol stream based on an input picture to be encoded and one or more reference pictures), and a (local) decoder (434) embedded in the video encoder (430). The decoder (434) reconstructs the symbols to create sample data in a manner similar to that which the (remote) decoder would also create (because any compression between the symbols and the encoded video code stream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (435). Since the decoding of the symbol stream results in a bit-accurate result that is independent of the decoder location (local or remote), the contents of the reference picture memory (435) are also bit-accurate between the local decoder and the remote decoder. In other words, when prediction is used during decoding, the predicted part of the encoder "sees" the same sample values ​​as the sample values ​​"seen" by the decoder as reference picture samples.

[0062] The operation of the "local" decoder (434) can be combined with Figure 5 The operation of the "remote" decoder is identical as described in detail.

[0063] During operation, in some examples, the source encoder (432) can perform motion compensated predictive coding, which predictively encodes an input picture with reference to one or more previously encoded pictures from a video sequence designated as "reference pictures". In this way, the encoding engine (433) encodes the difference between a pixel block of the input picture and a pixel block of a reference picture that can be selected as a prediction reference for the input picture. In some examples, the source encoder (432) can receive control signals from the controller (431). The control signals are generated based on metadata of the video sequence, and these control signals control the source encoder (432) to quickly perform motion compensated predictive coding and decoding.

[0064] The local video decoder (434) can decode the encoded video data of the picture that can be designated as the reference picture based on the symbols created by the source encoder (432). The operation of the encoding engine (433) can advantageously be a lossy process. When encoding at a video decoder (such as a reference picture) Figure 5 When the encoded video data is decoded at a video decoder (described in the foregoing), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (434) replicates the decoding process that may be performed by the video decoder on the reference pictures, and may cause the reconstructed reference pictures to be stored in a reference picture cache (435). In this way, the video encoder (430) may locally store copies of the reconstructed reference pictures that have the same content (without transmission errors) as the reconstructed reference pictures that will be obtained by the remote video decoder.

[0065] The predictor (436) may perform a prediction search on the encoding engine (433). That is, for a new picture to be encoded, the predictor (436) may search for sample data (as candidate reference pixel blocks) in the reference picture memory (435). In some examples, the predictor (436) receives a control signal from the controller (431). The control signal may be generated based on metadata, camera information, color information, lighting information, motion information, view information, user control, block shape, etc. The control signal may control the predictor (436) to reduce the time of a prediction search for an appropriate prediction reference for the new picture. In an example, the control signal may indicate a reference (e.g., a reference picture, a reference block, etc.). In another example, the control signal may indicate a starting point for the search. The predictor (436) may operate on a pixel-by-pixel sample block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (436), the input picture may have a prediction reference derived from multiple reference pictures stored in the reference picture memory (435).

[0066] The controller (431) may also manage encoding operations of the source encoder (432), including, for example, setting parameters and sub-group parameters for encoding video data.

[0067] The outputs of all the above functional units may be entropy encoded in an entropy encoder (437). The entropy encoder (437) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0068] The transmitter (411) can buffer one or more encoded video sequences created by the entropy encoder (437) in preparation for transmission via the network. The transmitter (411) can merge the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown). In some examples, the encoded video stream (443) and the overlay content stream (444) are multiplexed into a single stream. In another example, the metadata can be included in an SEI message (443) associated with the encoded video stream.

[0069] The controller (431) may perform various management operations of the video encoder (430). During encoding, the controller (431) may assign a certain coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the picture types such as an I picture, a P picture, a B picture, etc.

[0070] An intra picture (I picture) may be a picture that is encoded and decoded without using any other picture in the sequence as a prediction source.Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures.

[0071] A predicted picture (P picture) may be a picture that may be encoded and decoded using intra prediction or inter prediction that predicts sample values ​​of each block using at most one motion vector and a reference index.

[0072] Bidirectional prediction pictures (B pictures) can be pictures that can be encoded and decoded using intra prediction or inter prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-prediction pictures can use more than two reference pictures and associated metadata to reconstruct a single block.

[0073] A source picture may typically be spatially subdivided into blocks of samples (e.g., 4×4, 8×8, 4×8, or 16×16 blocks per sample), and encoded on a block-by-block basis. Blocks may be predictively encoded with reference to other (already encoded) blocks determined by the coding assignment applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively encoded, or they may be predictively encoded with reference to already encoded blocks of the same picture (either spatially or intra-predicted). Blocks of pixels of a P picture may be predictively encoded by spatial prediction with reference to one previously encoded reference picture or by temporal prediction. Blocks of a B picture may be predictively encoded by spatial prediction with reference to one or two previously encoded reference pictures or by temporal prediction.

[0074] The video encoder (430) may perform encoding operations according to a predetermined video encoding technique or standard. In its operation, the video encoder (430) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0075] In an embodiment, the transmitter (411) may transmit additional data along with the encoded video. The source encoder (432) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0076] A video (or video sequence, raw video content) can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated as intra-frame prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits (temporal or other) correlations between pictures. In an example, a specific picture under encoding / decoding, referred to as the current picture, is partitioned into blocks. When a block in the current picture is similar to a reference block in a reference picture that has been previously encoded and still buffered in the video, the block in the current picture can be encoded by a vector called a motion vector. In the case of using multiple reference pictures, the motion vector points to a reference block in a reference picture and may have a third dimension that identifies the reference picture.

[0077] In some embodiments, bidirectional prediction techniques may be used in inter-picture prediction. According to the bidirectional prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture may be encoded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block may be predicted by a combination of the first reference block and the second reference block.

[0078] Furthermore, merge mode techniques may be used in inter-picture prediction to improve coding efficiency.

[0079] According to some embodiments of the present disclosure, prediction is performed in units of blocks, such as inter-picture prediction and intra-picture prediction. For example, according to the High Efficiency Video Coding (HEVC) standard, a picture in a video picture sequence is partitioned into coding tree units (CTUs) for compression, and the CTUs in the picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), which are a luminance CTB and two chrominance CTBs. Each CTU can be recursively quadtree-divided into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be divided into a CU of 64×64 pixels, or 4 CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In the example, each CU is analyzed to determine the prediction type of the CU, such as an inter-frame prediction type or an intra-frame prediction type. The CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In an embodiment, the prediction operation in the codec (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values) such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0080] Figure 5 A diagram of a video decoder (580) according to some embodiments of the present disclosure is shown. The video decoder (580) is configured to receive an encoded picture as part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In an example, the video decoder (580) is used instead of Figure 3 The video decoder (380) in the example.

[0081] exist Figure 5 In the example of FIG. 5 , the video decoder ( 580 ) includes Figure 5An entropy decoder (582), an inter-frame decoder (585), a residual decoder (583), a reconstruction module (586) and an intra-frame decoder (584) coupled together are shown in FIG.

[0082] The entropy decoder (582) may be configured to reconstruct certain symbols from the encoded picture, which represent syntax elements constituting the encoded picture. Such symbols may include, for example, a mode for encoding a block (such as, for example, an intra-frame mode, an inter-frame mode, a bidirectional prediction mode, the inter-frame mode and the bidirectional prediction mode being in a merge sub-mode or another sub-mode), prediction information (such as, for example, intra-frame prediction information or inter-frame prediction information) that can identify certain samples or metadata used for prediction by an intra-frame decoder (584) or an inter-frame decoder (585), respectively, residual information in the form of, for example, quantized transform coefficients, etc. In an example, when the prediction mode is an inter-frame prediction mode or a bidirectional prediction mode, the inter-frame prediction information is provided to the inter-frame decoder (585); and when the prediction type is an intra-frame prediction type, the intra-frame prediction information is provided to the intra-frame decoder (584). The residual information may be dequantized and provided to the residual decoder (583). The metadata may be provided to the controller (581).

[0083] The inter-frame decoder (585) is configured to receive the inter-frame prediction information and generate an inter-frame prediction result based on the inter-frame prediction information.

[0084] The intra decoder (584) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0085] The residual decoder (583) is configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (583) may also require certain control information (to include quantizer parameters (QP)), and this information may be provided by the entropy decoder (582) (the data path is not shown because this may only be a small amount of control information).

[0086] The reconstruction module (586) is configured to combine the residual output by the residual decoder (583) and the prediction result (output by the inter-frame prediction module or the intra-frame prediction module as appropriate) in the spatial domain to form a reconstructed block, which can be part of a reconstructed picture, which in turn can be part of a reconstructed video. It should be noted that other appropriate operations (such as deblocking operations, etc.) can be performed to improve visual quality. For example, the reconstructed picture can be buffered in the buffer (587), and the buffer (587) can provide a reference picture to the inter-frame decoder (585).

[0087] In some embodiments, the controller (581) receives metadata and can configure decoding parameters of the decoder (580) based on the metadata. In some examples, the controller (581) can provide control signals to the intra decoder (584) and / or the inter decoder (585) to control the prediction based on the metadata. In another example, the controller (581) can decode the overlay content from the metadata and provide the overlay content to the reconstruction module (586). Thus, the reconstruction module (586) can reconstruct the picture using the overlay content.

[0088] Figure 6 A flow chart outlining a process (600) according to an embodiment of the present disclosure is shown. The process (600) can be used to encode a video sequence in a cloud gaming system (such as a cloud gaming system (100), a cloud gaming system (300), etc.). In various embodiments, the process (600) is performed by a processing circuit, such as a processing circuit (120) in a server (110), a processing circuit (320) in a server (310), a video encoder (130), a video encoder (330), a video encoder (430), etc. In some embodiments, the process (600) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (600). The process starts at (S601) and proceeds to (S610).

[0089] At (S610), a video sequence and metadata associated with the video sequence are received. The video sequence includes a sequence of picture frames generated in response to game control information. The metadata is determined based on the game control information, and the metadata indicates the game control information. In an example, a game engine (such as game engine (325)) generates game content (such as video content, audio content) based on game control information, such as game scene control information, user information, etc. The video content can be generated as a video sequence including a sequence of picture frames. Further, metadata associated with the video sequence can be identified based on the game control information. The video sequence and metadata can be provided to a video encoder, such as video encoder (330). The video encoder (330) receives the video sequence and metadata associated with the video sequence.

[0090] According to an aspect of the present disclosure, metadata may be any suitable information that may assist in encoding, such as camera information, color information, lighting information, motion information, view information, overlay information, user control information, and the like.

[0091] In some embodiments, a signal is received over a network and carries user control information captured by a remote user input device such as a user input device (369). A video sequence may be generated in response to the user control information. In some examples, the metadata may include motion information identified based on the user control information and the game scene information. In an example, the motion information includes global motion of the game scene. In another example, the motion information includes local motion of objects in a bounding box within the game scene.

[0092] At (S620), encoding parameters are configured based on the metadata. In an example, a motion model for encoding the video sequence is determined based on the metadata indicating the game control information. In some examples, the motion model may be one of an affine model, a perspective model, a rotation model, and a scaling model.

[0093] At (S630), the video sequence is encoded into an encoded video stream based on the encoding parameters. The encoded video stream can be transmitted to a destination device, such as a client device (360) via a network. In some embodiments, metadata is also transmitted to the destination device to assist in decoding the encoded video stream at the destination device. In an embodiment, the metadata is transmitted in an SEI message along with the encoded video stream. In another embodiment, the metadata includes overlay content. The overlay content can be encoded into an overlay content stream. The overlay content stream and the encoded video stream can be multiplexed into a single output stream. The single output stream can be transmitted to the destination device via a network. Then, the process proceeds to (S699) and ends.

[0094] Figure 7 A flow chart outlining a process (700) according to an embodiment of the present disclosure is shown. The process (700) can be used to decode a video sequence in a cloud gaming system (such as a cloud gaming system (100), a cloud gaming system (300), etc.). In various embodiments, the process (700) is performed by a processing circuit, such as a processing circuit (170), a processing circuit (370), a video decoder (180), a video decoder (380), a video decoder (480), etc. In some embodiments, the process (700) is implemented in software instructions, so when the processing circuit executes the software instructions, the processing circuit performs the process (700). The process starts at (S701) and proceeds to (S710).

[0095] At (S710), an encoded video stream and metadata associated with the encoded video stream are received. The metadata indicates game control information.

[0096] At (S720), decoding parameters and reconstruction parameters are determined based on the metadata.

[0097] At (S730), the encoded video stream is decoded based on the decoding parameters, and the video sequence is reconstructed based on the reconstruction parameters. Then, the process proceeds to (S799) and ends.

[0098] In some embodiments, the input code stream can be demultiplexed into an encoded video code stream and an overlay content code stream. The overlay content code stream can be decoded separately (e.g., by different decoders) to generate overlay content. The overlay content can then be used to reconstruct the video sequence.

[0099] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Figure 8 A computer system (800) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0100] Computer software may be encoded using any suitable machine code or computer language that may be assembled, compiled, linked or similarly structured to create code comprising instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through interpretation, microcode execution, etc.

[0101] These instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0102] Figure 8 The components for the computer system (800) shown in the example are exemplary in nature and are not intended to impose any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of components should also not be interpreted as having any dependency or requirement on any one component or combination of components illustrated in the exemplary embodiment of the computer system (800).

[0103] The computer system (800) may include certain human interface input devices. Such human interface input devices may be responsive to input from one or more human users through, for example, tactile input (such as keystrokes, swipes, data glove motions), audio input (such as voice, tapping), visual input (such as gestures), olfactory input (not depicted). The human interface devices may also be used to capture certain media that are not necessarily directly related to a person's conscious input, such as audio (such as voice, music, ambient sounds), pictures (such as scanned images, photographic pictures obtained from a still picture camera), and videos (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0104] The input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (801), mouse (802), touchpad (803), touch screen (810), data gloves (not shown), joystick (805), microphone (806), scanner (807), camera (808).

[0105] The computer system (800) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (810), a data glove (not shown), or a joystick (805), although there may also be tactile feedback devices that are not used as input devices), audio output devices (such as: speakers (809), headphones (not shown)), visual output devices (such as for screens (810) including CRT screens, LCD screens, plasma screens, OLED screens, each screen with or without touch screen input capabilities, each screen with or without tactile feedback capabilities, some of which are capable of outputting two-dimensional visual output or more than three-dimensional output through means such as stereo output; virtual reality glasses (not depicted), holographic displays and smoke canisters (not shown), and printers (not shown).

[0106] The computer system (800) may also include human-accessible storage devices and their associated media, such as optical media including media (821) such as CD / DVD ROM / RW (820) with CD / DVD, thumb drives (822), removable hard drives or solid-state drives (823), traditional magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), and the like.

[0107] Those skilled in the art should also understand that the term "computer-readable media" used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other volatile signals.

[0108] The computer system (800) may also include an interface to one or more communication networks. The network may be, for example, wireless, wired, optical. The network may further be local, wide-area, metropolitan, vehicle-mounted, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks (such as Ethernet), wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide-area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicle and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter attached to some common data port or peripheral bus (849) (such as a USB port of the computer system (800)); other networks are typically integrated into the kernel of the computer system (800) by attaching to a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smart phone computer system). Using any of these networks, the computer system (800) can communicate with other entities. Such communication may be one-way, receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way (e.g., to other computer systems using a local area digital network or a wide area digital network). Certain protocols and protocol stacks may be used on each of the networks and network interfaces such as those described above.

[0109] The above-mentioned human interface device, human-accessible storage device, and network interface may be attached to the kernel ( 840 ) of the computer system ( 800 ).

[0110] The core (840) may include one or more central processing units (CPUs) (841), graphics processing units (GPUs) (842), dedicated programmable processing units (FPGAs) (843) in the form of field programmable gate arrays (FPGAs), hardware accelerators (844) for certain tasks, etc. These devices, along with read-only memory (ROM) (845), random access memory (846), internal mass storage devices (847) such as internal non-user accessible hard drives, SSDs, etc., may be connected via a system bus (848). In some computer systems, the system bus (848) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (848) or to the core's system bus (848) via a peripheral bus (849). Peripheral bus architectures include PCI, USB, etc.

[0111] The CPU (841), GPU (842), FPGA (843), and accelerator (844) can execute certain instructions, and the combination of these instructions can constitute the aforementioned computer code. The computer code can be stored in ROM (845) or RAM (846). Transition data can also be stored in RAM (846), while permanent data can be stored in, for example, an internal mass storage device (847). Fast storage and retrieval of any memory device can be enabled by using a cache memory, which can be closely associated with one or more CPUs (841), GPUs (842), mass storage devices (847), ROM (845), RAM (846), etc.

[0112] The computer readable medium may have computer codes for performing various computer-implemented operations. The media and computer codes may be those specially designed and constructed for the purposes of the present disclosure, or they may be of a type well known and available to those skilled in the art of computer software.

[0113] As an example and not by way of limitation, a computer system having the architecture (800) and in particular the kernel (840) may provide functionality as a result of one or more processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software contained in one or more tangible computer-readable media. Such computer-readable media may be media associated with a user-accessible mass storage device as described above, as well as certain storage devices of the kernel (840) having non-volatile properties (such as a kernel internal mass storage device (847) or ROM (845)). Software implementing various embodiments of the present disclosure may be stored in such a device and executed by the kernel (840). Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software may cause the kernel (840) and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (846) and modifying such data structures according to software-defined processes. In addition or as an alternative, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (844)) that may operate in place of or in conjunction with software to perform a particular process or particular portion of a particular process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (such as an integrated circuit (IC)) storing software for execution, circuitry including logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0114] Although the present disclosure has described several exemplary embodiments, there are changes, permutations, and various alternative equivalents that fall within the scope of the present disclosure. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. A method for cloud gaming, characterized in that: include: receiving a video sequence and metadata associated with the video sequence, the video sequence comprising a sequence of picture frames generated in response to game control information, and the metadata determined based on the game control information and user control information; Determining motion information of at least one of a global motion of a game scene and a local motion of an object in a bounding box within the game scene based on the metadata; configuring encoding parameters based on motion information of at least one of a global motion of the determined game scene and a local motion of an object in a bounding box within the game scene, the encoding parameters comprising a motion model for encoding a video sequence, wherein the motion model comprises at least one of an affine model, a perspective model, a rotation model, and a scaling model; and Based on the motion model, the video sequence is encoded into an encoded video stream.

2. The method according to claim 1, characterized in that The metadata includes at least one of camera information, color information, lighting information, motion information, view information, overlay information, and user control information.

3. The method according to claim 1, characterized in that Further including: receiving a signal carrying user control information via a network; as well as The video sequence is generated in response to the user control information.

4. The method according to claim 1, characterized in that Further including: The metadata is transmitted to a destination device of the encoded video stream via a network.

5. The method according to claim 4, characterized in that Further including: The metadata is transmitted along with the coded video stream in a Supplemental Enhancement Information (SEI) message.

6. The method according to claim 4, characterized in that The metadata includes overlay content, and the method further comprises: encoding the overlay content into an overlay content code stream; and multiplexing the encoded video code stream and the superimposed content code stream into an output code stream; and The output code stream is transmitted.

7. A device for cloud gaming, characterized in that: include: A processing circuit, the processing circuit being configured to: receiving a video sequence and metadata associated with the video sequence, the video sequence comprising a sequence of picture frames generated in response to game control information, and the metadata determined based on the game control information and user control information; Determining motion information of at least one of a global motion of a game scene and a local motion of an object in a bounding box within the game scene based on the metadata; configuring encoding parameters based on motion information of at least one of a global motion of the determined game scene and a local motion of an object in a bounding box within the game scene, the encoding parameters comprising a motion model for encoding a video sequence, wherein the motion model comprises at least one of an affine model, a perspective model, a rotation model, and a scaling model; and Based on the motion model, the video sequence is encoded into an encoded video stream.

8. The device according to claim 7, characterized in that The metadata includes at least one of camera information, color information, lighting information, motion information, view information, overlay information, and user control information.

9. The device according to claim 7, characterized in that The processing circuit is configured to: receiving a signal carrying user control information via a network; and The video sequence is generated in response to the user control information.

10. The device according to claim 7, characterized in that The processing circuit is configured to: The metadata is transmitted to a destination device of the encoded video stream via an interface circuit and a network.

11. The device according to claim 10, characterized in that The processing circuit is configured to: The metadata is transmitted along with the coded video stream in a Supplemental Enhancement Information (SEI) message.

12. The device according to claim 10, characterized in that The metadata includes overlay content, and the processing circuit is configured to: encoding the overlay content into an overlay content code stream; and multiplexing the encoded video code stream and the superimposed content code stream into an output code stream; and The output code stream is transmitted.

13. A method for cloud gaming, characterized in that: include: receiving an encoded video stream and metadata associated with the encoded video stream, the metadata indicating game control information and user control information; Determining motion information of at least one of a global motion of a game scene and a local motion of an object in a bounding box within the game scene based on the metadata; configuring decoding parameters and reconstruction parameters based on the determined motion information of at least one of a global motion of the game scene and a local motion of an object in a bounding box within the game scene, the decoding parameters comprising a motion model for decoding a video sequence, wherein the motion model comprises at least one of an affine model, a perspective model, a rotation model, and a scaling model; and The encoded video stream is decoded based on the motion model and the encoded video stream is reconstructed into a video sequence based on the reconstruction parameters.

14. The method according to claim 13, characterized in that The metadata includes overlay content, and the method further comprises: Demultiplexing the input code stream into the encoded video code stream and the superimposed content code stream; Decoding the overlay content code stream to generate the overlay content; and The video sequence is reconstructed using the superimposed content.

15. An electronic device, characterized in that: Including: processor, memory; The memory stores computer executable instructions, which, when executed by the processor, implement the method according to any one of claims 1-6 and 13-14.

16. A non-volatile computer-readable medium storing instructions, characterized in that: When the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1-6 and 13-14.

Images