Systems, methods, apparatuses, and computer program products for providing metadata in bitstreams

Abstract

Information messages can be used to signal changes to graphics rendering information (GRI) with coded rendered graphics / video data in a bitstream. Syntax elements can be added to an information message to represent updated parameters in GRI categories such as rendering engine parameters, projection matrix, world to cam matrix, and / or depth parameters. Category presence indicators can be set to signal when information messages included syntax elements representing parameters in a particular GRI category. If a particular category presence indicator is set, a decoder can parse from the information message the syntax elements representing parameters in the associated GRI category. Alternatively, if a particular category presence indicator is not set, the information message will not include syntax elements representing parameters in the associated GRI category and the decoder can use previously-received and / or default parameter values when generating target graphics from the rendered graphics data.

Description

SYSTEMS, METHODS, APPARATUSES, AND COMPUTER PROGRAM PRODUCTS FOR PROVIDING METADATA IN BITSTREAMSCross-Reference to Related Applications

[0001] This application claims the benefitof priority to U.S. Provisional Patent Application Serial No.63 / 710,822 filed October 23, 2024 and entitled “Systems, Methods, Apparatuses, and Computer Program Products for Providing Metadata in Bitstreams,” the entire disclosure of which is hereby incorporated herein by reference in its entirety for all purposes.Technological Field

[0002] This disclosure relates generally to systems, methods, apparatuses, and computer program products for video coding and decoding, and more specifically for providing graphics rendering information in supplemental enhancement information messages in bitstreams.Background of the Invention

[0003] This section is intended to provide a background or context to the invention that is recited in the claims. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.

[0004] Scalable video coding refers to coding structure where one bitstream can contain multiple representations of the content at different bitrates, resolutions or frame rates. In these cases the receiver can extract the desired representation depending on its characteristics. Alternatively, a server or a network element can extract the portions of the multi-layer bitstream to be transmitted to the receiver depending on e.g., the network characteristics or processing capabilities of the receiver. A scalable bitstream typically consists of a base layer providing the lowest quality video available and one or more enhancement layers that enhance the video quality when received and decoded together with the lower layers. In order to improve coding efficiency for the enhancement layers, the coded representation of that layer typically depends on the lower layers.

[0005] A multi-layer bitstream is a bitstream comprising multiple layers, which may be, but are not limited to, base and enhancement layers, e.g., for scalable video coding. A multi-layer bitstream may additionally or alternatively comprise independent layers that do not have an inter-layer prediction relationship between each other and may even represent different types of content. A multi-layerbitstream may be regarded as a scalable video bitstream. Each layer in a multi-layer bitstream is identified by a layer identifier in a network abstraction layer (NAL) unit header. Additionally, a multi-layer bitstream may include one or more picture units (PUs) that comprise a set of NAL units that contain all VCL NAL units of a coded picture and their associated non-VCL NAL units. The multi-layer bitstream may also include one or more access units (AUs) that comprise a set of PUs that belong to different layers of the multi-layer bitstream and contain coded pictures associated with a same output time.

[0006] The Versatile Supplemental Enhancement Information (VSEI) standard for coded video bitstreams is a standard maintained by the Moving Picture Experts Group (MPEG). The VSEI standard requires, among other things, that layer identifiers of PUs in an AU have increasing layer identifier values. Supplemental enhancement information (SEI) messages may be used to provide metadata to facilitate or enable formation of a target display picture formed by overlaying multiple ordered display overlays in a specified order. A display overlay contains texture and, optionally, an alpha channel, each contained within a cropped decoded picture, a subpicture, and / or a constituent rectangle.Brief Summary

[0007] Some embodiments of the present invention comprise systems, methods, apparatuses, and computer program products for preparing, providing, receiving, and interpreting graphics rendering information in supplemental enhancement information messages in bitstreams. For example, described herein are various enhancements to information messages, such as supplemental enhancement information (SEI) messages, for providing graphics rendering information between encoder-side devices and decoder-side devices in bitstreams carrying images, video, audio, and / or the like. For example, enhanced SEI messages are described for providing graphics rendering information (GRI) in bitstreams. GRI SEI messagescan comprise GRI organized into GRI categories, e.g., rendering engine parameters, projection matrix, world to cam matrix (also known as world to camera matrix), depth parameters, and / or the like. If a GRI SEI message includes updated GRI in a particular GRI category, a category-specific updated GRI indicator in the GRI SEI message can be set. A decoder can, if a category-specific updated GRI indicator is set, review all GRI values in that specific GRI category for which the category-specific updated GRI indicator is set to extract updated GRI. Alternatively, any category-specific updated GRI indicator that is not set indicates to the decoder that updated GRI is not provided in this GRI category and the decoder can use previously provided or default GRI values for the GRI in that GRI category.

[0008] According to a first aspect, there is provided the subject matter of: receiving, from a graphics rendering application, source video data comprising rendered graphics data; receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data; encoding pictures of thesource video data into coded pictures; generating an information message comprising one or more GRI indicators configured to indicate the one or more parameters; and including the coded pictures and the information message in a bitstream. The first aspect may be implemented in algorithm(s), encoded in a distribution medium or computer program product(s), and as method(s) performed by apparatus(es). An apparatus may comprise means for causing the apparatus at least to perform the method(s). The means may comprise at least one processor; and at least one memory storing algorithm as instructions that, when executed by the at least one processor, cause the apparatus at least to perform the method(s).

[0009] The first aspect may include any single feature or any combination of features from:• providing the coding pictures and the information message to a decoding device in the bitstream,• wherein the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category, • in an instance in which the GRI received from the graphics rendering application includes at least one first parameter of the first category, setting a first GRI indicator to a first value to indicate that at least one first parameter of the first category is provided in the information message,• in an instance in which the GRI received from the graphics rendering application includes at least one second parameter of the second category, setting a second GRI indicator to the first value to indicate that at least one second parameter of the second category is provided in the information message, and / or• in an instance in which the GRI received from the graphics rendering application includes no updated parameters, setting the plurality of GRI indicators to a second value to indicate that the information message does not comprise any parameters of any categories.

[0010] According to a second aspect, there is provided the subject matter of: receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures, wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, and wherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data. The second aspect may be implemented in algorithm(s), encoded in a distribution medium or computer program product(s), and as method(s) performed by apparatus(es). An apparatus may comprise means for causing the apparatus at least toperform the method(s). The means may comprise at least one processor; and at least one memory storing algorithm as instructions that, when executed by the at least one processor, cause the apparatus at least to perform the method (s).

[0011] The second aspect may include any single feature or any combination of features from:• decoding the bitstream to form decoded video data; and / or performing image analysis on the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters,• decoding the bitstream to form decoded video data; and / or generating, from the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters, adjusted video data through any of transcoding, editing, or reprojection,• wherein the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category, • in an instance in which a first GRI indicator in the information message is set to a first value, determining that at least one first parameter of the first category is provided in the information message,• in an instance in which a second GRI indicator in the information message is set to the first value, determining that at least one second parameter of the second category is provided in the information message,• in an instance in the first GRI indicator and the second GRI indicator in the information message are both set to a second value, determining that the information message does not comprise any first parameters of the first category or any second parameters of the second category,• wherein the source video data comprises a depth map,• wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters,• wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements, • wherein the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphics data,• wherein the characteristics comprise one or more of: a vertical axis identification, a coordinate system handedness, or a projection format,• wherein the one or more GRI indicators comprise a world to camera matrix or a projection matrix,• wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients,• wherein the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix,• wherein the one or more parameters are grouped into two or more categories,• wherein a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI,• including, in the information message, respective category presence indicators for respective categories of the two or more categories,• wherein the respective category presence indicators indicate whether at least one parameter for a respective category is included in the information message,• in an instance in which the respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, adding, to the information message, a syntax element representing the at least one parameter of the respective category,• determining, based at least upon a syntax element value of respective category presence indicators in the information message for respective categories of the two or more categories, whether at least one parameter for a respective category is included in the information message,• in an instance in which the syntax element value of a respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, parsing, from the information message, a syntax element representing the at least one parameter of the respective category, • wherein at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1 ,• wherein the information message is provided or received in a first coded picture in the bitstream,• wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are not present in thefirst information message, one or more syntax elements are not included in the first information message for parameters of the first category, and the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the first coded picture, • wherein the first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message,• wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category,• wherein a second information message is contained within a second coded picture in the bitstream,• wherein a second category presence indicator for the first category in the second information message indicates that at least one parameter of the first category is not present in the second information message,• wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message,• wherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture,• wherein the bitstream comprises a depth map,• wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and / or• wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements.

[0012] According to some aspects, an apparatus can be provided that comprises at least one processor and at least one memory storing instructions stored thereon that, when executed by the at least one processor, cause the apparatus to perform at least: receiving, from a graphics rendering application, source video data comprising rendered graphics data; receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data; encoding pictures of the source video data into coded pictures; generating an information message comprising one or more GRIindicators configured to indicate the one or more parameters; and including the coded pictures and the information message in the bitstream.

[0013] In some embodiments, the instructions stored on the at least one memory, when executed by the at least one processor, further cause the apparatus to perform at least: providing, to a decoding device, the bitstream. In some embodiments, the source video data comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters. In some embodiments, the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format.

[0014] In some embodiments, the one or more GRI indicators comprise a world to camera matrix or a projection matrix. In some embodiments, at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0015] In some embodiments, the one or more parameters are grouped into two or more categories. In some embodiments, a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI. In some embodiments, the instructions stored on the at least one memory, when executed by the at least one processor, further cause the apparatus to perform: including, in the information message, respective category presence indicators for respective categories of the two or more categories, wherein the respective category presence indicators indicate whether at least one parameter for a respective category is included in the information message; and, in an instance in which the respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, adding, to the information message, a syntax element representing the at least one parameter of the respective category.

[0016] In some embodiments, at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1. In some embodiments, the information message is provided in a coded picture or frame in the bitstream. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least oneparameter of the first category are not present in the first information message, wherein one or more syntax elements are not included in the first information message for parameters of the first category, and wherein the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the first coded picture. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message, and wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category. In some embodiments, a second information message is contained within a second coded picture in the bitstream, wherein a second category presence indicator for the first category in the second information message indicates that at least one parameter of the first category is not present in the second information message, wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message, and wherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture.

[0017] In some embodiments, the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category. In some embodiments, the instructions stored on the at least one memory, when executed by the at least one processor, further cause the apparatus to perform: in an instance in which the GRI received from the graphics rendering application includes at least one first parameter of the first category, setting a first GRI indicator to a first value to indicate that at least one first parameter of the first category is provided in the information message; in an instance in which the GRI received from the graphics rendering application includes at least one second parameter of the second category, setting a second GRI indicator to the first value to indicate that at least one second parameter of the second category is provided in the information message; and, in an instance in which the GRI received from the graphics rendering application includes no updated parameters, setting the plurality of GRI indicators to a second value to indicate that the information message does not comprise any parameters of any categories.

[0018] According to some aspects, an apparatus can be provided that comprises at least one processor and at least one memory storing instructions stored thereon that, when executed by the at least one processor, cause the apparatus to perform at least: receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures,wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, wherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data.

[0019] In some embodiments, the instructions stored on the at least one memory, when executed by the at least one processor, further cause the apparatus to perform at least: decoding the bitstream to form decoded video data; and performing image analysis on the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters. In some embodiments, the instructions stored on the at least one memory, when executed by the at least one processor, further cause the apparatus to perform at least: decoding the bitstream to form decoded video data; and generating, from the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters, adjusted video data through any of transcoding, editing, or reprojection.

[0020] In some embodiments, the bitstream comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format. In some embodiments, the one or more GRI indicators comprise a world to camera matrix or a projection matrix, and wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0021] In some embodiments, the one or more parameters are grouped into two or more categories, wherein a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI. In some embodiments, the instructions stored on the at least one memory, when executed by the at least one processor, further cause the apparatus to perform: determining, based at least upon a syntax element value of respective category presence indicators in the information message for respective categories of the two or more categories, whether at least one parameter for a respective category is included in the information message; and, in an instance in which the syntax element value of a respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, parsing,from the information message, a syntax element representing the at least one parameter of the respective category.

[0022] In some embodiments, at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1. In some embodiments, the information message is encoded in a coded picture or frame received in the bitstream. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are not present in the first information message, wherein one or more syntax elements are not included in the first information message for parameters of the first category, and wherein the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the first coded picture. In some embodiments, a first information message is contained within a first coded picture of the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message, and wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category. In some embodiments, a second information message is contained within a second coded picture of the bitstream, wherein a second category presence indicator for the first category in the second information message indicates that at least one parameter of the first category is not present in the second information message, wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message, and wherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture.

[0023] In some embodiments, the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category. In some embodiments, the instructions stored on the at least one memory, when executed by the at least one processor, further cause the apparatus to perform: in an instance in which a first GRI indicator in the information message is set to a first value, determining that at least one first parameter of the first category is provided in the information message; in an instance in which a second GRI indicator in the information message is set to the first value, determining that at least one second parameter of the second category is provided in the information message; and, in an instance in the first GRI indicator and the second GRI indicator in theinformation message are both set to a second value, determining that the information message does not comprise any first parameters of the first category or any second parameters of the second category.

[0024] According to some aspects, an apparatus can be provided that comprises at least one processor and at least one memory storing instructions stored thereon that, when executed by the at least one processor, cause the apparatus to perform at least: receiving, from a graphics rendering application, source video data comprising rendered graphics data; receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data; generating an information message comprising a plurality of GRI indicators configured to indicate whether the information message comprises at least one parameter of respective category are provided in the information message, the plurality of GRI indicators comprising a first GRI indicator associated with a first category and a second GRI indicator associated with a second category; in an instance in which the GRI received from the graphics rendering application includes one or more first parameters of the first category, setting the first GRI indicator to a first value that indicates that at least one parameter of the first category is provided in the information message; in an instance in which the GRI received from the graphics rendering application includes one or more second parameters of the second category, setting the second GRI indicator in the information message to a second value that indicates that at least one parameter of the second category is provided in the information message; and providing, in a bitstream, coded pictures and the information message.

[0025] In some embodiments, the instructions stored on the at least one memory, when executed by the at least one processor, further cause the apparatus to perform at least: providing, to a decoding device, the bitstream comprising the coded pictures and the information message. In some embodiments, the bitstream comprises a depth map, wherein the plurality of GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the plurality of GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format. In some embodiments, the plurality of GRI indicators comprise a world to camera matrix or a projection matrix, and wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the plurality of GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0026] In some embodiments, the one or more first parameters and the one or more second parameters are grouped into two or more categories, wherein a respective category of the two or more categories comprises at least one parameter of the one or more first parameters or at least one parameter of the one or more second parameters.

[0027] According to some aspects, a method can be carried out by, e.g., an apparatus comprising one or more processors and one or more memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform the method. In some embodiments, the method can comprise: receiving, from a graphics rendering application, source video data comprising rendered graphics data; receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data; encoding pictures of the source video data into coded pictures; generating an information message comprising one or more GRI indicators configured to indicate the one or more parameters; and including the coded pictures and the information message in the bitstream.

[0028] In some embodiments, the method can further comprise: providing, to a decoding device, the bitstream. In some embodiments, the source video data comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters. In some embodiments, the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format.

[0029] In some embodiments, the one or more GRI indicators comprise a world to camera matrix or a projection matrix. In some embodiments, at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0030] In some embodiments, the one or more parameters are grouped into two or more categories. In some embodiments, a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI. In some embodiments, In some embodiments, the method can further comprise: including, in the information message, respective category presence indicators for respective categories of the two or more categories, wherein the respective category presence indicators indicate whether at least one parameter for a respective category is included in the information message; and, in an instance in which the respective category presenceindicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, adding, to the information message, a syntax element representing the at least one parameter of the respective category.

[0031] In some embodiments, at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1. In some embodiments, the information message is provided in a coded picture or frame in the bitstream. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are not present in the first information message, wherein one or more syntax elements are not included in the first information message for parameters of the first category, and wherein the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the first coded picture. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message, and wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category. In some embodiments, a second information message is contained within a second coded picture in the bitstream, wherein a second category presence indicator for the first category in the second information message indicates that at least one parameter of the first category is not present in the second information message, wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message, and wherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture.

[0032] In some embodiments, the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category. In some embodiments, In some embodiments, the method can further comprise: in an instance in which the GRI received from the graphics rendering application includes at least one first parameter of the first category, setting a first GRI indicator to a first value to indicate that at least one first parameter of the first category is provided in the information message; in an instance in which the GRI received from the graphics rendering application includes at least one second parameter of the second category, setting a second GRI indicator to the firstvalue to indicate that at least one second parameter of the second category is provided in the information message; and, in an instance in which the GRI received from the graphics rendering application includes no updated parameters, setting the plurality of GRI indicators to a second value to indicate that the information message does not comprise any parameters of any categories.

[0033] According to some aspects, a method can be carried out by, e.g., an apparatus comprising one or more processors and one or more memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform the method. In some embodiments, the method can comprise: receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures, wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, wherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data.

[0034] In some embodiments, the method can further comprise: decoding the bitstream to form decoded video data; and performing image analysis on the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters. In some embodiments, the method can further comprise: decoding the bitstream to form decoded video data; and generating, from the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters, adjusted video data through any of transcoding, editing, or reprojection.

[0035] In some embodiments, the bitstream comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format. In some embodiments, the one or more GRI indicators comprise a world to camera matrix or a projection matrix, and wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0036] In some embodiments, the one or more parameters are grouped into two or more categories, wherein a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI. In some embodiments, the method can further comprise:determining, based at least upon a syntax element value of respective category presence indicators in the information message for respective categories of the two or more categories, whether at least one parameter for a respective category is included in the information message; and, in an instance in which the syntax element value of a respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, parsing, from the information message, a syntax element representing the at least one parameter of the respective category.

[0037] In some embodiments, at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1. In some embodiments, the information message is encoded in a coded picture or frame received in the bitstream. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are not present in the first information message, wherein one or more syntax elements are not included in the first information message for parameters of the first category, and wherein the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the first coded picture. In some embodiments, a first information message is contained within a first coded picture of the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message, and wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category. In some embodiments, a second information message is contained within a second coded picture of the bitstream, wherein a second category presence indicator for the first category in the second information message indicates that at least one parameter of the first category is not present in the second information message, wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message, and wherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture.

[0038] In some embodiments, the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category. In some embodiments, the method can further comprise: in an instance in which a first GRI indicator in the information message is set to afirst value, determining that at least one first parameter of the first category is provided in the information message; in an instance in which a second GRI indicator in the information message is set to the first value, determining that at least one second parameter of the second category is provided in the information message; and, in an instance in the first GRI indicator and the second GRI indicator in the information message are both set to a second value, determining that the information message does not comprise any first parameters of the first category or any second parameters of the second category.

[0039] According to some aspects, a method can be carried out by, e.g., an apparatus comprising one or more processors and one or more memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform the method. In some embodiments, the method can comprise: receiving, from a graphics rendering application, source video data comprising rendered graphics data; receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data; generating an information message comprising a plurality of GRI indicators configured to indicate whether the information message comprises at least one parameter of respective category are provided in the information message, the plurality of GRI indicators comprising a first GRI indicator associated with a first category and a second GRI indicator associated with a second category; in an instance in which the GRI received from the graphics rendering application includes one or more first parameters of the first category, setting the first GRI indicator to a first value that indicates that at least one parameter of the first category is provided in the information message; in an instance in which the GRI received from the graphics rendering application includes one or more second parameters of the second category, setting the second GRI indicator in the information message to a second value that indicates that at least one parameter of the second category is provided in the information message; and providing, in a bitstream, coded pictures and the information message.

[0040] In some embodiments, the method can further comprise: providing, to a decoding device, the bitstream comprising the coded pictures and the information message. In some embodiments, the bitstream comprises a depth map, wherein the plurality of GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the plurality of GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format. In some embodiments, the plurality of GRI indicators comprise a world to camera matrix or a projection matrix, and wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, theplurality of GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0041] In some embodiments, the one or more first parameters and the one or more second parameters are grouped into two or more categories, wherein a respective category of the two or more categories comprises at least one parameter of the one or more first parameters or at least one parameter of the one or more second parameters.

[0042] According to some aspects, a computer program product can be provided that comprises, e.g., a non-transitory computer-readable storage medium comprising instructions (e.g., program codes) stored thereon that, when executed by at least one processor of an apparatus, cause the apparatus to perform at least: receiving, from a graphics rendering application, source video data comprising rendered graphics data; receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data; encoding pictures of the source video data into coded pictures; generating an information message comprising one or more GRI indicators configured to indicate the one or more parameters; and including the coded pictures and the information message in the bitstream.

[0043] In some embodiments, the instructions stored thereon, when executed by the at least one processor, further cause the apparatus to perform at least: providing, to a decoding device, the bitstream. In some embodiments, the source video data comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters. In some embodiments, the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format.

[0044] In some embodiments, the one or more GRI indicators comprise a world to camera matrix or a projection matrix. In some embodiments, at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0045] In some embodiments, the one or more parameters are grouped into two or more categories. In some embodiments, a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI. In some embodiments, the instructionsstored thereon, when executed by the at least one processor, further cause the apparatus to perform: including, in the information message, respective category presence indicators for respective categories of the two or more categories, wherein the respective category presence indicators indicate whether at least one parameter for a respective category is included in the information message; and, in an instance in which the respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, adding, to the information message, a syntax element representing the at least one parameter of the respective category.

[0046] In some embodiments, at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1. In some embodiments, the information message is provided in a coded picture or frame in the bitstream. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are not present in the first information message, wherein one or more syntax elements are not included in the first information message for parameters of the first category, and wherein the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the first coded picture. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message, and wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category. In some embodiments, a second information message is contained within a second coded picture in the bitstream, wherein a second category presence indicator for the first category in the second information message indicates that at least one parameter of the first category is not present in the second information message, wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message, and wherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture.

[0047] In some embodiments, the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category. In some embodiments, the instructions stored thereon, when executed by the at least one processor, further cause the apparatus to perform: inan instance in which the GRI received from the graphics rendering application includes at least one first parameter of the first category, setting a first GRI indicator to a first value to indicate that at least one first parameter of the first category is provided in the information message; in an instance in which the GRI received from the graphics rendering application includes at least one second parameter of the second category, setting a second GRI indicator to the first value to indicate that at least one second parameter of the second category is provided in the information message; and, in an instance in which the GRI received from the graphics rendering application includes no updated parameters, setting the plurality of GRI indicators to a second value to indicate that the information message does not comprise any parameters of any categories.

[0048] According to some aspects, a computer program product can be provided that comprises, e.g., a non-transitory computer-readable storage medium comprising instructions (e.g., program codes) stored thereon that, when executed by at least one processor of an apparatus, cause the apparatus to perform at least: receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures, wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, wherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data.

[0049] In some embodiments, the instructions stored thereon, when executed by the at least one processor, further cause the apparatus to perform at least: decoding the bitstream to form decoded video data; and performing image analysis on the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters. In some embodiments, the instructions stored thereon, when executed by the at least one processor, further cause the apparatus to perform at least: decoding the bitstream to form decoded video data; and generating, from the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters, adjusted video data through any of transcoding, editing, or reprojection.

[0050] In some embodiments, the bitstream comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format. In some embodiments, the one or more GRI indicators comprise a world to camera matrix or a projectionmatrix, and wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0051] In some embodiments, the one or more parameters are grouped into two or more categories, wherein a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI. In some embodiments, the instructions stored thereon, when executed by the at least one processor, further cause the apparatus to perform: determining, based at least upon a syntax element value of respective category presence indicators in the information message for respective categories of the two or more categories, whether at least one parameter for a respective category is included in the information message; and, in an instance in which the syntax element value of a respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, parsing, from the information message, a syntax element representing the at least one parameter of the respective category.

[0052] In some embodiments, at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1. In some embodiments, the information message is encoded in a coded picture or frame received in the bitstream. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are not present in the first information message, wherein one or more syntax elements are not included in the first information message for parameters of the first category, and wherein the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the first coded picture. In some embodiments, a first information message is contained within a first coded picture of the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message, and wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category. In some embodiments, a second information message is contained within a second coded picture of the bitstream, wherein a second category presence indicator for the first category in the second information message indicates that at least one parameter of the first category is not present in the second information message, wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message, andwherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture.

[0053] In some embodiments, the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category. In some embodiments, the instructions stored thereon, when executed by the at least one processor, further cause the apparatus to perform: in an instance in which a first GRI indicator in the information message is set to a first value, determining that at least one first parameter of the first category is provided in the information message; in an instance in which a second GRI indicator in the information message is set to the first value, determining that at least one second parameter of the second category is provided in the information message; and, in an instance in the first GRI indicator and the second GRI indicator in the information message are both set to a second value, determining that the information message does not comprise any first parameters of the first category or any second parameters of the second category.

[0054] According to some aspects, a computer program product can be provided that comprises, e.g., a non-transitory computer-readable storage medium comprising instructions (e.g., program codes) stored thereon that, when executed by at least one processor of an apparatus, cause the apparatus to perform at least: receiving, from a graphics rendering application, source video data comprising rendered graphics data; receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data; generating an information message comprising a plurality of GRI indicators configured to indicate whether the information message comprises at least one parameter of respective category are provided in the information message, the plurality of GRI indicators comprising a first GRI indicator associated with a first category and a second GRI indicator associated with a second category; in an instance in which the GRI received from the graphics rendering application includes one or more first parameters of the first category, setting the first GRI indicator to a first value that indicates that at least one parameter of the first category is provided in the information message; in an instance in which the GRI received from the graphics rendering application includes one or more second parameters of the second category, setting the second GRI indicator in the information message to a second value that indicates that at least one parameter of the second category is provided in the information message; and providing, in a bitstream, coded pictures and the information message.

[0055] In some embodiments, the instructions stored thereon, when executed by the at least one processor, further cause the apparatus to perform at least: providing, to a decoding device, the bitstreamcomprising the coded pictures and the information message. In some embodiments, the bitstream comprises a depth map, wherein the plurality of GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the plurality of GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format. In some embodiments, the plurality of GRI indicators comprise a world to camera matrix or a projection matrix, and wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the plurality of GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0056] In some embodiments, the one or more first parameters and the one or more second parameters are grouped into two or more categories, wherein a respective category of the two or more categories comprises at least one parameter of the one or more first parameters or at least one parameter of the one or more second parameters.

[0057] According to some aspects, an apparatus can be provided that comprises means, such as at least one processor and at least one memory storing instructions stored thereon that, when executed by the at least one processor, cause the apparatus to perform one or more elements of a method. For example, the apparatus can comprise: means for receiving, from a graphics rendering application, source video data comprising rendered graphics data; receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data; encoding pictures of the source video data into coded pictures; generating an information message comprising one or more GRI indicators configured to indicate the one or more parameters; and including the coded pictures and the information message in the bitstream.

[0058] In some embodiments, the apparatus can further comprise: means for providing, to a decoding device, the bitstream. In some embodiments, the source video data comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters. In some embodiments, the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphicsdata, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format.

[0059] In some embodiments, the one or more GRI indicators comprise a world to camera matrix or a projection matrix. In some embodiments, at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0060] In some embodiments, the one or more parameters are grouped into two or more categories. In some embodiments, a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI. In some embodiments, the apparatus can further comprise: means for including, in the information message, respective category presence indicators for respective categories of the two or more categories, wherein the respective category presence indicators indicate whether at least one parameter for a respective category is included in the information message; and means for, in an instance in which the respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, adding, to the information message, a syntax element representing the at least one parameter of the respective category.

[0061] In some embodiments, at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1. In some embodiments, the information message is provided in a coded picture or frame in the bitstream. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are not present in the first information message, wherein one or more syntax elements are not included in the first information message for parameters of the first category, and wherein the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the first coded picture. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message, and wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category. In some embodiments, a second information message is contained within a second coded picture in the bitstream, wherein a second category presence indicator for the firstcategory in the second information message indicates that at least one parameter of the first category is not present in the second information message, wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message, and wherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture.

[0062] In some embodiments, the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category. In some embodiments, the apparatus can further comprise: means for, in an instance in which the GRI received from the graphics rendering application includes at least one first parameter of the first category, setting a first GRI indicator to a first value to indicate that at least one first parameter of the first category is provided in the information message; means for, in an instance in which the GRI received from the graphics rendering application includes at least one second parameter of the second category, setting a second GRI indicator to the first value to indicate that at least one second parameter of the second category is provided in the information message; and means for in an instance in which the GRI received from the graphics rendering application includes no updated parameters, setting the plurality of GRI indicators to a second value to indicate that the information message does not comprise any parameters of any categories.

[0063] According to some aspects, an apparatus can be provided that comprises means, such as at least one processor and at least one memory storing instructions stored thereon that, when executed by the at least one processor, cause the apparatus to perform one or more elements of a method. For example, the apparatus can comprise: means for receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures, wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, wherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data.

[0064] In some embodiments, the apparatus can further comprise: means for decoding the bitstream to form decoded video data; and performing image analysis on the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters. In some embodiments, the apparatus can further comprise: means for decoding the bitstream to form decoded video data; and generating, from the decoded video data, based at leastupon the at least one syntax element parsed from the information message that represents the one or more parameters, adjusted video data through any of transcoding, editing, or reprojection.

[0065] In some embodiments, the bitstream comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format. In some embodiments, the one or more GRI indicators comprise a world to camera matrix or a projection matrix, and wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0066] In some embodiments, the one or more parameters are grouped into two or more categories, wherein a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI. In some embodiments, the apparatus can further comprise: means for determining, based at least upon a syntax element value of respective category presence indicators in the information message for respective categories of the two or more categories, whether at least one parameter for a respective category is included in the information message; and means for, in an instance in which the syntax element value of a respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, parsing, from the information message, a syntax element representing the at least one parameter of the respective category.

[0067] In some embodiments, at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1. In some embodiments, the information message is encoded in a coded picture or frame received in the bitstream. In some embodiments, a first information message is contained within a first coded picture in the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are not present in the first information message, wherein one or more syntax elements are not included in the first information message for parameters of the first category, and wherein the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the firstcoded picture. In some embodiments, a first information message is contained within a first coded picture of the bitstream, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message, and wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category. In some embodiments, a second information message is contained within a second coded picture of the bitstream, wherein a second category presence indicator for the first category in the second information message indicates that at least one parameter of the first category is not present in the second information message, wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message, and wherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture.

[0068] In some embodiments, the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category. In some embodiments, the apparatus can further comprise: means for, in an instance in which a first GRI indicator in the information message is set to a first value, determining that at least one first parameter of the first category is provided in the information message; means for, in an instance in which a second GRI indicator in the information message is set to the first value, determining that at least one second parameter of the second category is provided in the information message; and means for, in an instance in the first GRI indicator and the second GRI indicator in the information message are both set to a second value, determining that the information message does not comprise any first parameters of the first category or any second parameters of the second category.

[0069] According to some aspects, an apparatus can be provided that comprises means, such as at least one processor and at least one memory storing instructions stored thereon that, when executed by the at least one processor, cause the apparatus to perform one or more elements of a method. For example, the apparatus can comprise: means for receiving, from a graphics rendering application, source video data comprising rendered graphics data; means for receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data; means for generating an information message comprising a plurality of GRI indicators configured to indicate whether the information message comprises at least one parameter of respective category are provided in the information message, the plurality of GRI indicators comprising a first GRI indicator associated with a firstcategory and a second GRI indicator associated with a second category; means for, in an instance in which the GRI received from the graphics rendering application includes one or more first parameters of the first category, setting the first GRI indicator to a first value that indicates that at least one parameter of the first category is provided in the information message; means for, in an instance in which the GRI received from the graphics rendering application includes one or more second parameters of the second category, setting the second GRI indicator in the information message to a second value that indicates that at least one parameter of the second category is provided in the information message; and means for providing, in a bitstream, coded pictures and the information message.

[0070] In some embodiments, the apparatus can further comprise: means for providing, to a decoding device, the bitstream comprising the coded pictures and the information message. In some embodiments, the bitstream comprises a depth map, wherein the plurality of GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements. In some embodiments, the plurality of GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format. In some embodiments, the plurality of GRI indicators comprise a world to camera matrix or a projection matrix, and wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients. In some embodiments, the plurality of GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

[0071] In some embodiments, the one or more first parameters and the one or more second parameters are grouped into two or more categories, wherein a respective category of the two or more categories comprises at least one parameter of the one or more first parameters or at least one parameter of the one or more second parameters.

[0072] The above-noted aspects, elements, clauses, and features may be implemented in systems, apparatuses, methods, articles and non-transitory computer-readable media depending on the desired configuration. The subject disclosure may be implemented in and used with a number of different types of devices, such as one or more computing devices, one or more codecs, one or more encoders, one or more user equipment, one or more rendering engines, one or more servers, one or more network access nodes, one or more relay stations, one or more display devices, and / or the like. An example device can comprise at least one processor and at least one memory that stores thereon instructions which, when executed by the at least one processor, cause the device to perform some or all of the elements of theabove-described method, according to various embodiments. In other examples, a computer program product, such as a non-transitory computer-readable storage medium can be provided that comprises instructions stored thereon that, when executed by at least one processor of an apparatus, cause the apparatus to perform some or all elements of a method such as that descried above, according to some embodiments. In other examples, an apparatus can be provided that comprises means for carrying out a method - such means can include, e.g., a processor and a memory storing computer-executable instructions or computer codes thereon that, when executed by the processor, cause the apparatus to perform some or all of a method such as one of the methods described herein.

[0073] The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention taken in conjunction with the accompanying drawings.

[0074] This summary is intended to provide a brief overview of some of the aspects and features according to the subject disclosure. Accordingly, it will be appreciated that the above-described features are merely examples and should not be construed to narrow the scope of the subject disclosure in any way. Other features, aspects, and advantages of the subject disclosure will become apparent from the following detailed description, drawings and claims.Brief Description of the Drawings

[0075] Having thus described the invention in general terms, reference will now be made to the accompanying drawings. The skilled artisan will understand that the drawings primarily are for illustrative purposes and are not intended to limit the scope of the inventive subject matter described herein. The drawings are not necessarily to scale; in some instances, various aspects of the inventive subject matter disclosed herein may be shown exaggerated or enlarged in the drawings to facilitate an understanding of different features. In the drawings, like reference characters generally refer to like features (e.g., functionally similar and / or structurally similar elements).

[0076] In the accompanying drawings:

[0077] FIG. 1 is a block flow diagram illustrating a process for video encoding and decoding, in accordance with various embodiments of the present disclosure;

[0078] FIG. 2 shows schematically an example of a system for video encoding and decoding, in accordance with various embodiments of the present disclosure;

[0079] FIG. 3A illustrates example configurations for left-handed coordinate systems depending, e.g., upon whether an up vector is assumed to be along the x-axis or along the y-axis, in accordance with various embodiments of the present disclosure;

[0080] FIG. 3B illustrates example configurations for right-handed coordinate systems depending, e.g., upon whether an up vector is assumed to be along the x-axis or along the y-axis, in accordance with various embodiments of the present disclosure;

[0081] FIG. 4 shows schematically an example of a decoder-side device configured for carrying out video decoding, in accordance with various embodiments of the present disclosure;

[0082] FIG. 5 shows schematically an example of an encoder-side device configured for carrying out video encoding, in accordance with various embodiments of the present disclosure;

[0083] FIG. 6 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure;

[0084] FIG. 7 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure;

[0085] FIG. 8 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure;

[0086] FIG. 9 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure;

[0087] FIG. 10 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure;

[0088] FIG. 11 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure;

[0089] FIG. 12 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure;

[0090] FIG. 13 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure;

[0091] FIG. 14 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure;

[0092] FIG. 15 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure; and

[0093] FIG. 16 is a block flow diagram illustrating a method for enhancing metadata signaling in bitstreams between encoder-side devices and decoder-side devices to include graphics rendering information, in accordance with various embodiments of the present disclosure.Detailed Description

[0094] The present disclosure more fully describes various embodiments with reference to the accompanying drawings. It should be understood that some, but not all embodiments are shown and described herein. Indeed, the embodiments may take many different forms, and accordingly this disclosure should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will satisfy applicable legal requirements. Like numbers refer to like elements throughout.

[0095] FIG. 1 shows a process 100 in which video compression is applied.

[0096] The process 100 can comprising generating, recording, rendering, receiving, retrieving, or otherwise providing original video data 101. The original video data 101 can be encoded by a video encoder 102, using, e.g., one or more algorithms. Algorithms, such as Discrete Cosine Transform-based video compression algorithms, e.g., MPEG-2, MPEG-4, H.263, and H.264, can be used by the video encoder 102 to encode the original video data 101. The output from the video encoder 102 is compressed video data 103. Compressed video data 103 is sent to a network 104 that provides the compressed video data 105 to a video decoder 106. The video decoder 106 decodes the compressed video data 105 to generate decoded video data 107, which is approximately equivalent to the original video data 101.

[0097] The video encoder 102 compresses the original video data 101 in such a way that the compressed video data 103 does not exceed an available bandwidth of the network 104 in order for the video decoder 106 to be able to receive and decode the compressed video data 105. However, communication bandwidth may vary depending on the type of the network 104. For example, the available communication bandwidth of an Ethernet is different from that of a wireless local area network (WLAN). The network 104, which may be e.g., a cellular communication network, may have a very narrow bandwidth. Thus, it can be important to generate compressed video data 103 at various bitrates from the same original video data 101, such as by using scalable video coding. Scalable videocoding is a video compression technique that allows video data to provide scalability. Scalability is the ability to generate video sequences at different resolutions, frame rates, and qualities from the same compressed bitstream. In some embodiments, the video encoder 102 can achieve temporal scalability can be provided using, e.g., Motion Compensation Temporal filtering (MCTF), Unconstrained MCTF, Successive Temporal Approximation and Referencing, and / or the like. In some embodiments, the video encoder 102 can achieve Signal-to-Noise Ratio (SNR) scalability or Signal-to-Noise-plus-lnterference Ratio (SNIR) scalability using, e.g., Embedded ZeroTrees Wavelet (EZW), Set Partitioning in Hierarchical Trees (SPIHT), Embedded ZeroBlock Coding (EZBC), Embedded Block Coding with Optimized Truncation (EBCOT), etc. In some embodiments, the video encoder 102 can transmit only a portion of a scene, image, or picture need be transmitted as compressed video data 105 to the video decoder 106, which may improve bit-rate efficiency of video compression / coding.

[0098] In some embodiments, the video encoder 102 can achieve spatial scalability by using, e.g., a wavelet transform algorithm or multi-layer coding. For example, in some embodiments the video encoder 102 can use a multi-layer bitstream to transmit different portions of a scene, image, picture, inlay, overlay, background imagery, and / or the like, as separate layers, overlays, textures, alphas, or the like, to the video decoder 106 via the network 104.

[0099] When the video encoder 102 uses a multi-layer bitstream to transmit different portions of a scene, image, picture, inlay, overlay, background imagery, and / or the like, as separate layers, overlays, textures, alphas, or the like, to the video decoder 106 via the network 104, the video encoder 102 must typically provide metadata before, with, or after transmitting a portion or component of the multi-layer bitstream. For example, metadata may include bitstream information, such as attributes of a frame, overlay, layer, texture, alpha, or the like. In some embodiments, the metadata can be provided in a network abstraction layer (NAL) unit, e.g., in accordance with the H.264 / AVC and HEVC video coding standards, the entire disclosures of which are hereby incorporated herein by reference in their entireties for all purposes.

[0100] Among other elements of the metadata, supplemental enhancement information (SEI) can be provided as additional data inserted into the multi-layer bitstream to convey information that may be helpful for the network 104 to properly transmit the compressed video data 105 to the video decoder 106. Additionally or alternatively, SEI can be provided as additional data inserted into the multi-layer bitstream to convey information that may be helpful for the video decoder 106 to properly synchronize related audio and video content from the multi-layer bitstream, properly orient the video content from among the layers, determine a proper texture overlay order, determine characteristics about each layer such as whether a texture overlay / layer includes displayable content in every portion / region of the texture overlay / layer, etc.

[0101] SEI can be inserted by the video encoder 102 during the encoding and transmission of the original video data 101. Various types of information can be added into SEI, such as camera parameters, encoder parameters, time codes, closed captions, lyrics, etc. Syntax and semantics can be established for SEI messages and / or how SEI messages are to be inserted into a bitstream such that the video decoder 106 can predicably decode the SEI messages from the multi-layer bitstream.

[0102] SEI can also be used for application-specific purposes, such as to convey rendered graphics data alongside captured images / pictures and information about how to overlay or inlay the rendered graphics data with / over the captured images / pictures. In other embodiments, SEI can be used to indicate characteristics about display overlays in different layers of the multi-layer bitstream.

[0103] A multi-layer bitstream (such as for scalable coded video) may contain different layers comprising different image sequences, overlay sequences, texture sequences, and / or the like. For example, a scalable coded video bitstream may comprise different layers each containing different representations of an original image sequence. In one specific example, a first layer in the multi-layer bitstream can contain a relatively lower quality version of an original image sequence, and a second layer in the multi-layer bitstream can contain a relatively higher quality version of the original image sequence. In a second specific example, a first layer in the multi-layer bitstream can contain a first image sequence of background images, and a second layer in the multi-layer bitstream can contain a second image sequence of foreground images to be overlayed over respective background images from the first image sequence of background images. Other examples will be readily apparent to those skilled in the art, such as more sophisticated examples that include a plurality of different image layers within the multi-layer bitstream, one or more alpha representations, initialization values for a target picture to be rendered (e.g., at the video decoder 106), a combination of these, and / or the like. In some embodiments, SEI can be provided in a SEI Raw Byte Sequency Payload as a detached NAL unit.

[0104] Various SEI messages can be used to indicate / signal various information between the video encoder 102 and the video decoder 106. For example, information about camera-captured content, such as a shutter interval used, can be conveyed to the video decoder 106 using a Shutter Interval Information SEI message. Information such as parameters of annotated regions using bounding boxes can be communicated to the video decoder 106 using an Annotated Regions SEI message. Other information or parameters associated with video content being transmitted by the video decoder 106, such as content light level information, equirectangular projection information, fisheye information, color content volume information, color remapping information, motion-constrained tile set (MCTS) information, cubemap projection information, sphere rotation information, region-wise packing information, omnidirectionalviewport information, SEI manifest information, SEI prefix information, and / or the like, can be communicated by one or more other SEI messages.

[0105] Some metadata and / or other enhancement information can be provided via in-band and / or SEI messages. For example, information about a recovery point in a group of pictures (GOP) or sequence of NAL units, can be indicated using a presentation time stamp or decoding time stamp identifier, NAL unit identifier, frame number, etc. In some embodiments, a Recovery Point SEI message can be used to provide such information. In other embodiments, an l-frame / slice (intra-coded picture) can be provided before / between P-frames / slices (predicted pictures) and / or B-frames / slices (bidirectional predicted pictures) to indicate a latest recovery point as the latest decoded l-frame / slice. In other embodiments, such as under the H.264 video coding standard, in addition to l-frames / slices, P-frames / slices and / or B-frames / slices, switching l-frames / slices, switching P-frames / slices, and multi-frame motion estimation frames / slices can be provided. In some embodiments, a recovery point can be indicated in, e.g., the Recovery Point SEI message, that indicates a recovery point objective, system restore point, orphaned recovery point, recovery point chain, etc.

[0106] FIG. 2 illustrates a system 200, according to an embodiment, within which embodiments of the present invention can be utilized is shown. The system 200 comprises multiple communication devices which can communicate through one or more networks. The system 200 may comprise any combination of wired or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM, UMTS, CDMA network etc.), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet.

[0107] The system 200 may include both wired and wireless communication devices and / or electronic devices suitable for implementing select, various, or all of the embodiments described herein.

[0108] For example, the system 200, as shown in FIG. 2, is illustrated as comprising a mobile network 210 and a representation of the internet 220. The mobile network 210 can be or comprise, e.g., a fourth generation (4G) network, a Long Term Evolution (LTE), a fifth generation (5G) network, a sixth generation (6G) network, and / or the like. Connectivity to the internet 220 may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.

[0109] The example communication devices shown in the system 200 may include, but are not limited to, an electronic device or apparatus, such as mobile device 211, user equipment 212, etc. User equipment 212 can be connected to the internet 220 by way of at least an access point 213, which maybe or comprise a gNodeB (gNB), an eNodeB (eNB), base station, access network node, radio access network (RAN) node, and / or the like. The mobile device 211 can be connected to the internet 220 by way of at least a cell tower 214, e.g., via radio signaling 215 with the cell tower 214, short messaging service (SMS) with the cell tower 214, and / or the like.

[0110] Additionally or alternatively, the mobile device 211 and / or user device 212 can be connected to the internet 220 by way of a WiFi access point 216 or the like. Access point 213, cell tower 214, and / or WiFi access point 216 can be configured to communicate directly with the internet 220 or with the internet 220 by way of a network server 219, which can comprise, be comprised in, hosted on, or otherwise functionalized via any suitable network-side device. Such network-side devices can include, but are not limited to, a server, a computing device, a centralized processing unit (CPU), a graphics processing unit (GPU), a processor, processing circuitry, a controller, a network element, a virtualized network function, a mobility management entity (MME), a serving gateway (SGW), a packet data network (PDN) gateway (PGW), a home subscriber server (HSS), a public data network (PDN), an access and mobility management function (AMF), a user plane function (UPF), a data network (DN), an authentication server function (AUSF), a session management function (SMF), a network slice selection function (NSSF), a network exposure function (NEF), a network function repository function (NRF), a policy control function (PCF), a unified data management (UDM) function, an application function (AF), or any other suitable network-side device, element, function, hardware, device, etc.

[0111] In the system 200, electronic devices such as, e.g., 211, 212, 217, etc. may be stationary or mobile when carried by an individual who is moving. For example, the user equipment 212 can be or comprise a head-mounted display, a body-worn display, an immersive gaming system, a smartphone, a laptop (e.g., 217), or the like. In other embodiments, electronic devices in the system 200, such as 211, 212, 217, can be mobile by virtue of being located in, mounted on, coupled to, or otherwise supported by a device configured for transportation, including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle, or any similar suitable mode of transport. In other embodiments, the computing device 218 can also be stationary or can be configured to be used while stationary and while mobile, or to be operationally or configurationally switched from a mobile use mode to a stationary use mode. For example, in embodiments in which the user device 212 is or comprises a head mounted display, the user device 212 may be effectively used by a user wearing the user device 212 while the user is stationary or while the user is moving.

[0112] Additionally or alternatively, electronic devices in the system 200 can be stationary. For example, the system 200 can comprise a computing device 218, which can be or comprise a gaming console, a desktop computer, a three-dimensional gaming system, a virtual reality display system, anaugmented reality display system, an interactive-display system, an image projection system, and / or the like. In other embodiments, the computing device 218 can also be mobile or can be configured to be used while stationary and while mobile, or to be operationally or configurationally switched from a stationary use mode to a mobile use mode.

[0113] In some embodiments, one or more of the electronic devices in the system 200, e.g., one of 211, 212, 217, 218 may be or comprise a set-top box, a digital TV receiver, a device configured to transmit / receive streaming content or audio / video via a bitstream, etc., but which may / may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware or software or combination of the encoder / decoder implementations, in various operating systems, and in chipsets, processors, DSPs and / or embedded systems offering hardware / software based coding.

[0114] In some embodiments, certain electronic devices (e.g., 211, 212, 217, 218) in the system 200 may be configured to send and receive calls and messages and communicate with service providers through a wireless connection, such as 215, to the cell tower 214 or the access point 213. The cell tower 214 and / or the access point 213 may be connected to the network server 219 that allows communication between the mobile network 210 and the internet 220. The system 200 may include additional communication devices and communication devices of various types, such as electronic devices 221, 222, and 223, which may be outside of the mobile network 210 but nevertheless connected to the internet 220, e.g., by way of a wired or wireless connection 224.

[0115] The various communication devices (e.g., 211, 212, 217, 218, 221, 222, 223) illustrated in the system 200 of FIG. 2 may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11 and any similar wireless communication technology. A communications device involved in implementing various embodiments of the present disclosure may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.

[0116] Among other transmissions between two or more of the communication devices (e.g., 211, 212, 217, 218, 221, 222, 223) illustrated in the system 200 of FIG. 2, video and / or audio transmissions can be carried out. In order to improve the efficiency and / or effectiveness of resource use, reduce bit-rate, reduce and / or improve signaling, and reduce transmission-side (TX-side) and / or receiverside (RX-side) computational complexity, data to be transmitted can be compressed (i.e., encoded) atthe TX-side and decoded at the RX-side. To carry out such video / audio data compression, a coder / decoder device (i.e., codec), or one or more codecs, can be used. In some embodiments, the video encoder 102 and / or video decoder 106 described above with reference to FIG. 1 can comprise at least one codec.

[0117] Real-time Transport Protocol (RTP) is widely used for real-time transport of timed media such as audio and video. RTP may operate on top of the User Datagram Protocol (UDP), which in turn may operate on top of the Internet Protocol (IP). RTP is specified in Internet Engineering Task Force (IETF) Request for Comments (RFC) 3550, available from www.ietf.org / rfc / rfc3550.txt. In RTP transport, media data is encapsulated into RTP packets. Each media type or media coding format may have a dedicated RTP payload format.

[0118] An RTP session is an association among a group of participants communicating with RTP. It is a group communications channel which can potentially carry a number of RTP streams. An RTP stream is a stream of RTP packets comprising media data.

[0119] Communication systems may include any number of media-aware network elements (MANEs). For example, many multipoint audio-visual conferences operate utilizing a centralized unit called Multipoint Control Unit (MCU). An MCU may implement the functionality of an RTP translator or an RTP mixer. An RTP translator may be a media translator that may modify the media inside the RTP stream. A media translator may for example decode and re-encode the media content (i.e., transcode the media content). An RTP mixer is a middlebox that aggregates multiple RTP streams that are part of a session by generating one or more new RTP streams. An RTP mixer may manipulate the media data. One common application for a mixer is to allow a participant to receive a session with a reduced amount of resources compared to receiving individual RTP streams from all endpoints. A mixer can be viewed as a device terminating the RTP streams received from other endpoints in the same RTP session. Using the media data carried in the received RTP streams, a mixer generates derived RTP streams that are sent to the receiving endpoints. In another example, a MANE is a selective forward unit (SFU) that selectively forwards incoming RTP packets from one or more senders to one or more receivers.

[0120] According to some embodiments, a video codec can consist of an encoder that transforms the input video into a compressed representation suited for storage / transmission and / or a decoder that can uncompress the compressed video representation back into a viewable form. A video encoder (e.g., 102) and / or a video decoder (e.g., 106) may be combined within a singular device or can be separate from each other, i.e., need not form a codec within a singular device. Typically, a video encoder(e.g., 102) discards some information in the original video data 101 in order to represent the original video data 101 in a more compact form (that is, at a lower bit-rate).

[0121] Hybrid video encoders, for example many encoder implementations of ITU-T H.263 and H.264, often may encode the original video data 101 in two or more phases. According to some embodiments, pixel values in a certain picture area (or “block”) are initially predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner), and thereafter a prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This can be done by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT), a DCT algorithm, or a variant of the same), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the video encoder (e.g., 102) can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate), as reflected for example in the compressed video data 103 and / or the compressed video data 105.

[0122] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0123] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction.

[0124] Referring now to FIG. 3, a video coding system or device is illustrated, according to an example embodiment, as a schematic block diagram of an exemplary apparatus, referred to herein as decoding device 300, which is configured to carry out at least a portion of the video compression / encoding / decoding processes and tasks described herein, e.g., 100.

[0125] The decoding device 300 may for example be configured to function as the video decoder 106. In other embodiments, the decoding device 300 can be, comprise, or be comprised within,e.g., mobile device 211, user equipment 212, computing device 217, or computing device 218 in the wireless network 210. In other embodiments, the decoding device 300 can be, comprise, or be comprised within a heads-up display, a head-mounted display, a gaming console, a user’s computer, and / or the like. However, it will be appreciated that embodiments of the invention may be implemented within any electronic device or apparatus which may require encoding and decoding, or encoding or decoding, of video images, multi-layer bitstreams, scalable coded video, bitstreams comprising audio and video, bidirectional bitstreams, conversational content bitstreams, non-conversational content bitstreams, and / or the like.

[0126] The decoding device 300 may comprise a controller 301 in operable communication with a memory 302 and a radio interface 303. The decoding device 300 may further comprise a display 308, e.g., in the form of a liquid crystal display (LCD), light emitting diode (LED) display, organic LED (OLED) display, plasma display, Active-Matrix OLED (AMOLED) display, Quantum dot LED (QLED) display, micro-LED display, augmented reality display, virtual reality display, projected image display, any combination thereof, and / or the like. In other embodiments, the display 308 may be any other display technology suitable to display an image and / or video. The decoding device 300 may, optionally, further comprise a keypad 309. In other embodiments, any suitable data or user interface mechanism may be employed. For example a user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

[0127] The decoding device 300 may comprise a microphone (not shown) or any suitable audio input which may be a digital or analogue signal input. The decoding device 300 may further comprise an audio output device which, in some embodiments, may be any one of: an earpiece, a speaker, or an analogue audio or digital audio output connection. The decoding device 300 may also comprise a battery (not shown). In other embodiments, the decoding device 300 may be powered by any suitable mobile energy device such as a solar cell, a fuel cell, a clockwork generator, etc. The decoding device 300 may, optionally, further comprise a camera 310 capable of recording or capturing images and / or video. The decoding device 300 may further comprise an infrared port (not shown) for short range line of sight communication to other devices. In other embodiments the decoding device 300 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB / firewire wired connection.

[0128] According to some embodiments, the controller 301 can comprise, e.g., a processor or the like configured for controlling at least some aspects, functionalities, equipment, subcomponents, or subsystems of the decoding device 300. The controller 301 may be connected either directly or indirectly to the memory 302 which, in some embodiments, may store both data in the form of image and audiodata and / or may also store instructions for implementation on the controller 301. The controller 301 may further be connected to codec circuitry 305 and the codec circuitry 305 can be arranged and dimensioned for, operably programmed for, programmatically capable of, or otherwise suitably configured for carrying out coding and / or decoding of audio and / or video data or assisting in coding and decoding carried out by the controller 301.

[0129] The decoding device 300 may further comprise a card reader (not shown) and / or a smart card (not shown), for example a III CC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network (e.g., 210).

[0130] The decoding device 300 may comprise radio interface circuitry 303 connected to, or otherwise in operable communication with, the controller 301. The radio interface circuitry 303 can be arranged and dimensioned for, operably programmed for, programmatically capable of, or otherwise suitably configured for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system, or a wireless local area network. The decoding device 300 may further comprise an antenna array 304 connected to the radio interface circuitry 303 for transmitting radio frequency signals generated at the radio interface circuitry 303 to other apparatus(es) and for receiving radio frequency signals from other apparatus(es).

[0131] The antenna array 304 can be configured to receive radio signals comprising or representing the compressed / coded audio / video content from, e.g., an encoder-side device. The antenna array 304 can relay the radio signals to the radio interface 303, which can be configured to convert the radio signals to decodable information, which it then passes along to the codec circuitry 305. The codec circuitry 305 then decodes, e.g., with the aid of, and / or upon receiving instructions from, the controller 301. The codec circuitry 305 can then, once the decodable information is decoded, provide decoded information to the controller 301. The controller 301 can interpret the decoded information to synchronize the audio / video content, and otherwise determine how to reconstitute, build, reconstruct, render, overlay, display, emit, broadcast, and / or present the decoded audio, images, and / or video frames to one or more users, either directly on the decoding device 300 (e.g., via the display 308) or by transmitting / providing the decoded audio, images, and / or video frames to another device (e.g., 211, 212, 217, 218) for display thereon.

[0132] The decoding device 300 may, optionally, further comprise a camera 310 capable of recording or detecting individual frames which are then passed to the codec 305 or the controller 301 for processing. The decoding device 300 may receive the video image data for processing from another device prior to transmission and / or storage. The decoding device 300 may also receive either wirelessly or by a wired connection the image for coding / decoding. The incorporation of the camera 310 into / withthe decoding device 300 can be helpful in certain circumstances when, e.g., the image and / or video content is or comprises virtual reality content or augmented reality content in which real world objects and imagery located about the decoding device 300 or other device (e.g., 211, 212, 217, 218) displaying thereon the content may need to record and return images and / or video to assist with rendering subsequent images and / or video frames of the content for the user(s).

[0133] Referring now to FIG. 4, a video coding system or device is illustrated, according to an example embodiment, as a schematic block diagram of an exemplary apparatus, referred to herein as an encoding device 400, which is configured to carry out at least a portion of the video compression / encoding / decoding processes and tasks described herein, e.g., 100.

[0134] The encoding device 400 may for example be configured to function as the video encoder 102. In other embodiments, the encoding device 400 can be, comprise, or be comprised within a network-side or encoder-side device, e.g., 219. However, it will be appreciated that the encoding device 400 can be, comprise, or be comprised within another device within the mobile network (e.g., 210) in which the decoding device 300 is located. In other embodiments, the encoding device 400 can be, comprise, or be comprised within a device operationally and / or physically located outside the system or network (e.g., 210) in which the decoding device 300 is located. For example, the encoding device 400 can be, comprise, or be comprised within, e.g., 221, 222, 223, or the like. However, it will be appreciated that embodiments of the invention may be implemented within any electronic device or apparatus which may require encoding and decoding, or encoding or decoding, of video images, multi-layer bitstreams, scalable coded video, bitstreams comprising audio and video, bidirectional bitstreams, conversational content bitstreams, non-conversational content bitstreams, and / or the like.

[0135] The encoding device 400 may comprise a controller 401 in operable communication with a memory 402 and a radio interface 403. The encoding device 400 can be further may comprise codec circuitry 405 in operable communication with one or both of the controller 401 and / or the radio interface 403. The encoding device 400 can further comprise an antenna array 404 in operable communication with the radio interface 403.

[0136] In some embodiments, the encoding device 400 can be configured to capture audio and / or video content to be encoded and transmitted to a decoder-side device (e.g., 300). In such embodiments, the encoding device 400 can, optionally, comprise a camera 407, a microphone (not shown), and / or the like.

[0137] In other embodiments, the encoding device 400 can be configured to generate audio and / or video content to be encoded and transmitted to a decoder-side device (e.g., 300). In such embodiments, the encoding device 400 can, optionally, comprise a graphics generator 409.

[0138] In other embodiments, encoding device 400 can be configured to request, retrieve, or otherwise receive audio and / or video from one or more external devices, subcomponents, systems, etc. For example, the encoding device 400 can be configured to receive audio from an external microphone (not shown) or an external audio generation device (not shown). In some embodiments, the encoding device 400 can be configured to receive video from an external camera 408. Whether video content is captured by the camera 407 or by the external camera 408, these cameras are capable of recording or capturing images and / or video.

[0139] In some embodiments, the encoding device 400 can be configured to receive generated graphics or other rendered content from an external rendering device or graphics generating device, such as the graphics generator 409.

[0140] According to some embodiments, the controller 401 can be or comprise, e.g., a processor, processing circuitry, or the like, that is configured for controlling at least some aspects, functionalities, equipment, subcomponents, or subsystems of the encoding device 400. The controller 401 may be connected either directly or indirectly to the memory 402 which, in some embodiments, may store both data in the form of image and / or audio data and / or may also store instructions for implementation of the same using the controller 401. The controller 401 may further be connected to the codec circuitry 405 and the codec circuitry 405 can be arranged and dimensioned for, operably programmed for, programmatically capable of, or otherwise suitably configured for carrying out coding and / or decoding of audio and / or video data or assisting in coding and decoding carried out by the controller 401.

[0141] The radio interface circuitry 403 of the encoding device 400 can be configured to be connected to, or otherwise in operable communication with, the controller 401. The radio interface circuitry 403 can be arranged and dimensioned for, operably programmed for, programmatically capable of, or otherwise suitably configured for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system, or a wireless local area network. The encoding device 400 may further comprise the antenna 404 connected to the radio interface circuitry 403 for transmitting radio frequency signals generated at the radio interface circuitry 403 to other apparatus(es) and for receiving radio frequency signals from other apparatus(es).

[0142] The codec circuitry 405 of the encoding device 400 may be configured to receive images and / or video (e.g., as bitstream data) from the controller 401. The codec circuitry 405 can be further configured to encode / compress this image and / or video data, and optionally metadata for decoder-side use in decoding / interpreting the encoded / compressed image and / or video data. The codec circuitry 405 can then provide the encoded / compressed image and / or video data to the radio interface 403, which canconvert the encoded / compressed image and / or video data to a form that can be provided / transmitted to a decoder-side device (e.g., 300) via radio signaling using the antenna array 404.

[0143] In order to ensure that the encoded / compressed image and / or video data being provided from, e.g., the encoder device 400 to the decoder device 300, is properly and synchronously encoded and decoded, standard means can be defined for how the encoded / compressed image and / or video data is encoded. Likewise, metadata that is associated with the encoded / compressed image and / or video data can likewise be provided between the encoder-side and the decoder-side (e.g., from the encoder device 400 to the decoder device 300). Standard means can also be defined for what the metadata associated with the encoded / compressed image and / or video data comprises, the form in which the metadata associated with the encoded / compressed image and / or video data is provided, how syntax used in the metadata associated with the encoded / compressed image and / or video data is selected and used, and / or how the metadata associated with the encoded / compressed image and / or video data is encoded / decoded.

[0144] As described briefly above, among other elements of the metadata, supplemental enhancement information (SEI) can be provided from the encoder device 400 as additional data inserted into the multi-layer bitstream to convey information that may be helpful for the decoder device 300 to properly decode and interpret the encoded / compressed image and / or video data received therewith from the encoder device 400. SEI can likewise be helpful for the decoder device 300, once the encoded / compressed image and / or video data is decoded, if the decoder device 300 generates displayable or renderable image(s) or video frame(s) from the encoded / compressed image and / or video data. Additionally and / or alternatively, SEI can be helpful for the decoder device 300 if the decoder device 300 generates data or information about the encoded / compressed image and / or video data that, when transmitted to a displaying device (e.g., 112, 113, 117, 118, etc.), enables the displaying device to generate and display displayable image(s) or video frame(s). Alternatively, SEI can be helpful for the decoder device 300 if the decoder device 300 generates data or information about the encoded / compressed image and / or video data that, when transmitted to a rendering device (e.g., 112, 113, 117, 118, etc.), enables the rendering device to render and display / present rendered imagery, rendered graphics, and / or rendered video frame(s).

[0145] The High Efficiency Video Coding (H.265 / HEVC a.k.a. HEVC) standard was originally developed by the Joint Collaborative Team - Video Coding (JCT-VC) of VCEG and MPEG. The standard was published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Version 2 of the H.265 / HEVC standard included scalable,multiview, fidelity range, three-dimensional, and screen content coding extensions which may be abbreviated SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0146] Versatile Video Coding (VVC) (MPEG-I Part 3), a.k.a. ITU-T H.266, is a video compression standard developed by the Joint Video Experts Team (JVET) of the Moving Picture Experts Group (MPEG), (formally ISO / IEC JTC1 SC29 WG11) and Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU) to be the successor to HEVC / H.265.

[0147] A specification of the AV1 bitstream format and decoding process were developed by the Alliance for Open Media (AOM). The AV1 specification was published in 2018. AOM is reportedly working on the AV2 specification.

[0148] Some key definitions, bitstream and coding structures, and concepts of some video coding standards and specifications are described in this section for providing background for a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein the embodiments may be implemented. It is to be understood that embodiments are not limited to the referenced video coding standards or specifications.

[0149] An elementary unit for the input to an encoder and the output of a decoder, respectively, in many cases is a picture. A picture given as an input to an encoder may also be referred to as a source picture, and a picture decoded by a decoded may be referred to as a decoded picture or a reconstructed picture.

[0150] The source and decoded pictures are each comprised of one or more sample arrays. The sample arrays of a picture may be referred to as luma (or L or Y) and chroma, where the two chroma arrays may be referred to as Cb and Cr; regardless of the actual color representation method in use. The actual color representation method in use can be indicated e.g., in a coded bitstream e.g., using the Video Usability Information (VUI) syntax of HEVC or alike. A component may be defined as an array or single sample from one of the three sample arrays (luma and two chroma) or the array or a single sample of the array that compose a picture in monochrome format.

[0151] Chroma sample arrays may be absent (and hence monochrome sampling may be in use) or chroma sample arrays may be subsampled when compared to luma sample arrays. Chroma formats comprise monochrome format and non-monochrome formats, and these may be summarized as follows:

[0152] In monochrome sampling there is only one sample array, which may be nominally considered the luma array.

[0153] In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array.

[0154] In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luma array.

[0155] In 4:4:4 sampling when no separate color planes are in use, each of the two chroma arrays has the same height and width as the luma array.

[0156] Samples of a sample array have a certain bit depth, such as 8 bits per sample or 10 bits per sample. A bit depth implicitly specifies a value range, which may be referred to as the full range. For example, the full range is from 0 to 255, inclusive, for 8 bits per sample, or from 0 to 1023, inclusive, for 10 bits per sample. The source video may use allocate a narrower sample value range than the full range. A specific value range, sometimes referred to as the studio range, has been specified in the ITU-T H.273 standard specifying coding-independent code points for video. A source value range may interchangeably be referred to as a source sample value range, and may be defined as the sample value range of the video that is given as input to a video encoder to be encoded.

[0157] A picture may be defined to be either a frame or a field. A frame comprises a matrix of luma samples and possibly the corresponding chroma samples. A field is a set of alternate sample rows of a frame and may be used as encoder input, when the source signal is interlaced.

[0158] A bitstream may be defined as a sequence of bits or a sequence of syntax structures. A bitstream format may constrain the order of syntax structures in the bitstream.

[0159] A syntax element may be defined as an element of data represented in a bitstream. A syntax structure may be defined as zero or more syntax elements present together in a bitstream in a specified order.

[0160] Syntax structures may be specified, for example, using arithmetic, logical, relational, bit-wise, and assignment operators similar to those available in many programming languages. For example, & may indicate a bit-wise ‘AND’ operation. Furthermore, syntax structures may be specified with reference to mathematical functions.

[0161] Syntax structures and semantics may use the values of variables derived from the values of syntax elements. Naming conventions may be defined for variables. For example, variables may be named by a mixture of lower case and upper case letter and without any underscore characters. Variables starting with an upper case letter may be derived for the decoding of the current syntax structure and all depending syntax structures. Variables starting with an upper case letter may, in some cases, be used in the decoding process for later syntax structures without mentioning the originating syntax structure of the variable. Variables starting with a lower case letter may only be used in relation to the syntax structure or function for which they have been defined.

[0162] Video coding specifications may define an elementary unit that for the output an of an encoder and / or for the input to a decoder. For example, such an elementary unit may be an open bitstream unit (OBU), as specified e.g., in AV1, or a Network Abstraction Layer (NAL) unit, as specified e.g., in HEVC orWC.

[0163] In some video codecs, an elementary unit for the output of an encoder and the input of a decoder, respectively, may be a Network Abstraction Layer (NAL) unit. For transport over packet-oriented networks or storage into structured files, NAL units may be encapsulated into packets or similar structures. A bytestream format has been specified in some video coding standards for transmission or storage environments that do not provide framing structures. The bytestream format separates NAL units from each other by attaching a start code in front of each NAL unit. To avoid false detection of NAL unit boundaries, encoders run a byte-oriented start code emulation prevention algorithm, which adds an emulation prevention byte to the NAL unit payload if a start code would have occurred otherwise. In order to enable straightforward gateway operation between packet- and stream-oriented systems, start code emulation prevention may always be performed regardless of whether the bytestream format is in use or not. A NAL unit may be defined as a syntax structure containing an indication of the type of data to follow and bytes containing that data in the form of an RBSP interspersed as necessary with emulation prevention bytes. A raw byte sequence payload (RBSP) may be defined as a syntax structure containing an integer number of bytes that is encapsulated in a NAL unit. An RBSP is either empty or has the form of a string of data bits containing syntax elements followed by an RBSP stop bit and followed by zero (0) or more subsequent bits equal to zero (0).

[0164] A bitstream may be defined to logically include a syntax structure, such as a NAL unit, when the syntax structure is transmitted along the bitstream but may be included in the bitstream according to the bitstream format. A bitstream may be defined to natively comprise a syntax structure, when the bitstream includes the syntax structure.

[0165] In some coding formats or standards, a bitstream may be in the form of a network abstraction layer (NAL) unit stream or a byte stream, that forms the representation of coded pictures and associated data forming one or more coded video sequences.

[0166] In some coding formats, such as AV1, a bitstream may comprise a sequence of open bitstream units (OBUs). An OBU comprises a header and a payload, wherein the header identifies a type of the OBU. Furthermore, the header may comprise a size of the payload in bytes.

[0167] In some coding standards, NAL units include a header and payload. In some coding standards, the NAL unit header indicates the type of the NAL unit. In some coding standards, the NAL unit header indicates a scalability layer identifier (e.g., called nuhjayerjd), which may be used, e.g., forindicating spatial or quality layers, views of a multiview video, or auxiliary layers (such as depth maps or alpha planes). In some coding standards, the NAL unit header includes a temporal sublayer identifier, which may be used for indicating temporal subsets of the bitstream, such as a 30-frames-per-second subset of a 60-frames-per-second bitstream.

[0168] Scalable video coding refers to coding structure where one bitstream can contain multiple representations of the content, for example at different bitrates, resolutions or frame rates. In these cases, the receiver can extract the desired representation depending on its characteristics (e.g., resolution that matches best the display device). Alternatively, a server or a network element can extract the portions of the bitstream to be transmitted to the receiver depending on e.g., the network characteristics or processing capabilities of the receiver. A scalable bitstream may consist of a “base layer” providing the lowest quality video available and one or more enhancement layers that enhance the video quality when received and decoded together with the lower layers. To improve coding efficiency for the enhancement layers, the coded representation of that layer typically depends on the lower layers. For example, the motion and mode information of the enhancement layer can be predicted from lower layers. Similarly, the pixel data of the lower layers can be used to create prediction for the enhancement layer.

[0169] A scalable video codec for quality scalability (also known as Signal-to-Noise or SNR) and / or spatial scalability may be implemented as follows. For a base layer, a conventional non-scalable video encoder and decoder is used. The reconstructed / decoded pictures of the base layer are included in the reference picture buffer for an enhancement layer. In H.265 / HEVC and similar codecs using reference picture list(s) for inter prediction, the base layer decoded pictures may be inserted into a reference picture list(s) for coding / decoding of an enhancement layer picture similarly to the decoded reference pictures of the enhancement layer. Consequently, the encoder may choose a base-layer reference picture as inter prediction reference and indicate its use typically with a reference picture index in the coded bitstream. The decoder decodes from the bitstream, for example from a reference picture index, that a base-layer picture is used as inter prediction reference for the enhancement layer. When a decoded base-layer picture is used as prediction reference for an enhancement layer, it is referred to as an inter-layer reference picture.

[0170] While the previous paragraph described a scalable video codec with two scalability layers with an enhancement layer and a base layer, it needs to be understood that the description can be generalized to any two layers in a scalability hierarchy with more than two layers. In this case, a second enhancement layer may depend on a first enhancement layer in encoding and / or decoding processes, and the first enhancement layer may therefore be regarded as the base layer for the encoding and / or decoding of the second enhancement layer. Furthermore, it needs to be understood that there may beinter-layer reference pictures from more than one layer in a reference picture buffer or reference picture lists of an enhancement layer, and each of these inter-layer reference pictures may be considered to reside in a base layer or a reference layer for the enhancement layer being encoded and / or decoded. Furthermore, it needs to be understood that other types of inter-layer processing than reference-layer picture upsampling may take place instead or additionally. For example, the bit-depth of the samples of the reference-layer picture may be converted to the bit-depth of the enhancement layer and / or the sample values may undergo a mapping from the color space of the reference layer to the color space of the enhancement layer.

[0171] In addition to quality scalability, there are also other scalability modes, such as spatial scalability and bit-depth scalability. In spatial scalability, enhancement layer pictures are coded at a higher resolution than the base layer pictures. In bit-depth scalability, enhancement layer pictures are coded at higher bit-depth (e.g., 10 bits or 12 bits) than base layer pictures (e.g., 8 bits).

[0172] A multi-layer bitstream is a bitstream comprising multiple layers, which may be, but are not limited to, base and enhancement layers as discussed above for scalable video coding. A multi-layer bitstream may additionally or alternatively comprise independent layers that do not have inter-layer prediction relationship between each other and may even represent different types of content.

[0173] Layers of a multi-layer bitstream may be identified by a layer identifier or layer ID. In some video coding specifications, such as HEVC and WC, the layer ID is represented by the nuhjayerjd syntax element. In some video coding specifications, such as AV1, the spatial Jd syntax element of an extension of an OBU header (obu_extension_header) may be regarded as a layer ID.

[0174] NAL units can be categorized into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are typically coded slice NAL units.

[0175] A non-VCL NAL unit may be for example one of the following types: a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), an adaptation parameter set (APS), a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end of sequence (EOS) NAL unit, an end of bitstream (EOB) NAL unit, or a filler data NAL unit. Parameter sets may be needed for the reconstruction of decoded pictures, whereas many of the other non-VCL NAL units may not be necessary for the reconstruction of decoded sample values.

[0176] In some coding formats, picture unit (PU) may be defined as a set of data units, such as NAL units, that are associated with each other, are consecutive in decoding order, and contain exactly one coded picture. For example, certain non-video-coding data units, such as non-VCL NAL units, may be next to coded video data units in decoding order and the respective picture unit may comprise both these non-video-coding data units and the video coding data units of a coded picture.

[0177] In some coding formats, an access unit (AU) may be defined as a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and include at most one coded picture at any scalability layer (e.g., with any specific value of nuh JayerJd in some coding formats, such as HEVC or VVC). In some coding formats, an access unit comprises one or more complete picture units. In some coding formats, in addition to including the VCL NAL units of a coded picture, an access unit may also include non-VCL NAL units associated with the coded picture. Said specified classification rule may, for example, associate pictures with the same output time or picture order count value into the same access unit.

[0178] In some coding formats, a coded video sequence (CVS) may be defined as a sequence of coded pictures in decoding order that is independently decodable and is followed by another coded video sequence or the end of the bitstream.

[0179] In some coding formats, such as AV1, a coded video sequence comprises one or more temporal units. A temporal unit consists of a series of OBUs starting from a temporal delimiter, optional sequence headers, optional metadata OBUs, a sequence of one or more frame headers, each followed by zero or more tile group OBUs as well as optional padding OBUs. A temporal unit may be defined to comprise all the OBUs that are associated with a specific, distinct time instant. A temporal unit may comprise a temporal delimiter OBU, and all the OBUs that follow, up to but not including the next temporal delimiter. A temporal delimiter OBU may be defined as an indication that the following OBUs will have a different presentation / decoding time stamp from the one of the last frame prior to the temporal delimiter.

[0180] A coded layer video sequence (CLVS) may be defined as a sequence of coded pictures and associated other data within the same scalable layer (e.g., with the same value of nuhjayerjd) that is decodable independently of other pictures in the same layer.

[0181] In some video coding formats, such as WC, a subpicture may be defined as a rectangular region of one or more slices within a picture, wherein the one or more slices are complete and a slice is a unit (e.g., a NAL unit) that can be decoded independently of other slices of the same coded picture. Thus, a subpicture consists of one or more slices that collectively cover a rectangular region of a picture. Consequently, each subpicture boundary is also always a slice boundary. The slices of a subpicture may be required to be rectangular slices.

[0182] An independent subpicture (a.k.a. an extractable subpicture) may be defined as a subpicture with subpicture boundaries that are treated as picture boundaries. Additionally, it may be required that an independent subpicture has no loop filtering across the subpicture boundaries.

[0183] Output order may be defined as the order in which the decoded pictures are output by a decoder.

[0184] Some coding formats use a concept of picture order count (POC). A value of POC is derived for each picture and is non-decreasing with increasing picture position in output order. In some coding formats, an increasing value of POC indicates the output order of pictures within a single scalability layer and a single CVS. POC may be used in the decoding process for example for implicit scaling of motion vectors and for reference picture list initialization. Furthermore, POC may be used in the verification of output order conformance.

[0185] Some video coding specifications enable metadata OBUs. A metadata OBU comprises a type field, which specifies the type of metadata. A metadata OBU may be understood to be similar to an SEI NAL unit or an SEI message.

[0186] Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or alike. Some video coding specifications include SEI NAL units, and some video coding specifications contain both prefix SEI NAL units and suffix SEI NAL units, where the former type can start a picture unit or alike and the latter type can end a picture unit or alike. An SEI NAL unit contains one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, post-processing of decoded pictures, rendering, error detection, error concealment, and resource reservation.

[0187] ITU-T Recommendation H.274, which is equivalent to ISO / IEC 23002-7, may be called "versatile supplemental enhancement information messages for coded video bitstreams" and be referred to as "versatile supplemental enhancement information" or VSEI. The VSEI standard specifies the syntax and semantics of video usability information (VUI) parameters and supplemental enhancement information (SEI) messages. The VUI parameters and SEI messages defined in the VSEI standard are designed to be conveyed within coded video bitstreams in a manner specified in a video coding specification or to be conveyed by other means determined by the specifications for systems that make use of such coded video bitstreams. The VSEI standard is intended for use with WC coded video bitstreams, although it is drafted in a manner intended to be sufficiently generic that it may also be used with other types of coded video bitstreams.

[0188] A source picture for encoding may be formed from a composition of multiple constituent rectangles. The rectangles may themselves be constituent pictures of different types, such as texture, depth, alpha, or object mask, or may contain multiple constituent pictures of the same type, such as multiple views or multiple Al feature channels.

[0189] A constituent rectangles SEI message may be encoded in or along a video bitstream to describe the constituent rectangles present in decoded pictures. Likewise, a constituent rectangles SEImessage may be decoded from or along a video bitstream to determine constituent rectangles that are present in decoded pictures and properties of the constituent rectangles, such as their types.

[0190] In some embodiments, SEI can be provided as additional data inserted by the encoder device 400 into the multi-layer bitstream to convey information that may be helpful for the decoder device 300 to properly synchronize related audio and video content from the multi-layer bitstream, properly orient the video content from among the layers, determine a proper texture overlay order, determine characteristics about each layer such as whether a texture overlay / layer includes displayable content in every portion / region of the texture overlay / layer, etc.

[0191] SEI can be inserted by the encoder device 400 during the encoding and transmission of the image and / or video data to the decoder device 300. Various types of information can be added into SEI, such as camera parameters, encoder parameters, time codes, closed captions, lyrics, etc. Syntax and semantics can be established for SEI messages and / or how SEI messages are to be inserted into a bitstream such that the decoder device 300 can predicably decode the SEI messages from the multilayer bitstream.

[0192] SEI can also be used for application-specific purposes, such as to convey rendered graphics data alongside captured images / pictures and information about how to overlay or inlay the rendered graphics data with / over the captured images / pictures. In other embodiments, SEI can be used to indicate characteristics about display overlays in different layers of the multi-layer bitstream.

[0193] A multi-layer bitstream (such as for scalable coded video) may contain different layers comprising different image sequences, overlay sequences, texture sequences, and / or the like. For example, a scalable coded video bitstream may comprise different layers each containing different representations of an original image sequence. In one specific example, a first layer in the multi-layer bitstream can contain a relatively lower quality version of an original image sequence, and a second layer in the multi-layer bitstream can contain a relatively higher quality version of the original image sequence. In a second specific example, a first layer in the multi-layer bitstream can contain a first image sequence of background images, and a second layer in the multi-layer bitstream can contain a second image sequence of foreground images to be overlayed over respective background images from the first image sequence of background images. Other examples will be readily apparent to those skilled in the art, such as more sophisticated examples that include a plurality of different image layers within the multi-layer bitstream, one or more alpha representations, initialization values for a target picture to be rendered (e.g., at the decoder device 300), a combination of these, and / or the like. In some embodiments, SEI can be provided in a SEI Raw Byte Sequence Payload as a detached NAL unit.

[0194] Various SEI messages can be used to indicate / signal various information between the encoder device 400 and the decoder device 300. For example, information about camera-captured content, such as a shutter interval used, can be conveyed to the decoder device 300 using a Shutter Interval Information SEI message. Information such as parameters of annotated regions using bounding boxes can be communicated to the decoder device 300 using an Annotated Regions SEI message. Other information or parameters associated with video content being transmitted by the decoder device 300, such as content light level information, equirectangular projection information, fisheye information, color content volume information, color remapping information, motion-constrained tile set (MCTS) information, cubemap projection information, sphere rotation information, region-wise packing information, omnidirectional viewport information, SEI manifest information, SEI prefix information, and / or the like, can be communicated by one or more other SEI messages.

[0195] Some metadata and / or other enhancement information can be provided via in-band and / or SEI messages. For example, information about a recovery point in a group of pictures (GOP) or sequence of NAL units, can be indicated using a presentation time stamp or decoding time stamp identifier, NAL unit identifier, frame number, etc. In some embodiments, a Recovery Point SEI message can be used to provide such information. In other embodiments, an l-frame / slice (intra-coded picture) can be provided before / between P-frames / slices (predicted pictures) and / or B-frames / slices (bidirectional predicted pictures) to indicate a latest recovery point as the latest decoded l-frame / slice. In other embodiments, such as under the H.264 video coding standard, in addition to l-frames / slices, P-frames / slices and / or B-frames / slices, switching l-frames / slices, switching P-frames / slices, and multi-frame motion estimation frames / slices can be provided. In some embodiments, a recovery point can be indicated in, e.g., the Recovery Point SEI message, that indicates a recovery point objective, system restore point, orphaned recovery point, recovery point chain, etc.

[0196] In addition to captured images and video, a bitstream can include / carry rendered graphics, rendered images, rendered video, and / or the like. Video content generated by graphics rendering using, e.g., GPUs, rather than video content captured by a camera, may be coded with video coding standards like HEVC or WC. In some embodiments, generated content, rendered graphics, and / or the like, can include or be provided with metadata files containing information that provides parameters used in the rendering of the content. In some embodiments, such metadata can be provided in, e.g., JSON format. Examples of some information that can be provided with rendered graphics, GPU-rendered video, and / or the like, can include, e.g., depth maps and dense motion vectors. A wide variety of graphics rendering applications, such as game engines and authoring tools are available and can be used with GPUs to render 2D image or video content from 3D models, as well as associated depth maps. Differentapplication implementations use different conventions for representing 3D models with X, Y, and Z axes, including whether the axes use a right handed or left handed coordinate system and whether the Y or Z axis is vertical.

[0197] For example, as illustrated in FIGs. 3A and 3B, various 3D coordinate systems are used by different rendering graphics applications, examples of which include Unreal Engine, Maya, Unity, Lightwave, DirectX, and Blender, among others. Some graphics rendering applications use Normalized Device Coordinates (NDC) in the range [- 1, 1] while others use the range [0, 1].

[0198] In some embodiments, a multiview acquisition information (MAI) SEI message and / or a depth representation information (DRI) SEI message can be used to provide certain parameters. For example, the MAI SEI message can be used to specify various parameters of the acquisition environment for the layers that may be present in the current CVS, i.e. , the CVS containing the MAI SEI message. In some embodiments, intrinsic and extrinsic camera parameters can be specified. These parameters could be used for processing the decoded views prior to rendering on a 3D display.

[0199] In some embodiments, syntax elements in the DRI SEI message may be used to specify various parameters for auxiliary pictures of type AUX_DEPTH for the purpose of processing decoded primary and auxiliary pictures prior to rendering on a 3D display, such as view synthesis. For example, depth or disparity ranges for depth pictures may be specified.

[0200] In other embodiments, a JSON file may include metadata for associated video content, e.g., such as per frame auxiliary information for gaming content provided by the gaming engine used to generate the gaming content. An example JSON file may include information about projectionMatrix, worldToCameraMatrix, farClipPlane, nearClipPlane, and focalLength.

[0201] However, none of the MAI SEI message, DRI SEI message, or JSON file comprising metadata for associated gaming content provide information necessary for various use cases and activities involving video content generated by GPUs, such as transcoding, image / video analysis, reprojection, and creation of derivative content may benefit from having access to the parameters used by the GPU in creation of the video content, among others.

[0202] As such, described herein is a graphics rendering information (GRI) SEI message that is configured for carrying auxiliary information describing content that was created by graphics rendering, such as gaming content or Al generated content.

[0203] In some embodiments, a GRI SEI message can comprise syntax elements to represent parameters in one or more of the following categories, each with a presence flag:game engine parameters,projection matrix,- world to cam matrix, and / or- depth parameters

[0204] In some embodiments, the game engine parameters category includes syntax elements for the vertical axis identification, the coordinate system handedness, and indication of perspective vs. orthographic projection.

[0205] In some embodiments, if the first graphics rendering SEI message applicable to a CLVS has the presence flag for a particular category set to 0, default values are inferred for the syntax elements gated by that presence flag. In some embodiments, the values of syntax elements persist in decoding order until the graphics rendering SEI message is cancelled or until the syntax element is present in an applicable graphics rendering SEI message, so in subsequent SEI messages, if a presence flag for a particular category is set to 0, the inference values for the syntax elements in that category are inferred to be equal to the values in the previous SEI graphics rendering SEI message.

[0206] An SEI message is defined to carry parameters used by a GPU to render source video content and optionally associated depth video. The parameters can be grouped into categories, such as one or more of the following categories:- game engine parameters,- projection matrix,- world to cam matrix, and / or- depth parameters.

[0207] In some embodiments, a presence flag syntax element can be included in the SEI message, e.g., for each category of parameters. In some embodiments, some categories may contain parameters that are likely to be updated infrequently while other categories contain parameters that are likely to be update more frequently. For example, parameters of the graphics rendering application, such as a gaming engine or authoring tool, used to generate the source video content are included in the game engine parameters category. The parameters in this category are often likely to be unchanged over the video sequence and / or throughout the bitstream.

[0208] In some embodiments, some GRI, parameters, values, etc. may be more or less likely to be updated together or separately based upon one or more correlative or causative connections between the GRI, parameters, values, etc. For example, rendering / game engine parameters are associated with the particular rendering engine or game engine used to render the associated graphics, meaning that if a change in rendering / game engine is performed mid-bitstream, it may be more likely that the grouped GRI, parameters, values, etc. in the “rendering / game engine parameters” category will change together than for many GRI, parameters, values, etc. in other categories.

[0209] In some embodiments, the coordinate system used by the gaming engine is described using a vertical axis identification flag, which indicates if the Y or Z axis is vertical, and a coordinate system handedness flag which indicates whether a right handed or left handed coordinate system is used.

[0210] In some embodiments, a normalization range flag indicates whether the Normalized Device Coordinates (NDC) used are in the range [- 1, 1] or in the range [0, 1].

[0211] In some embodiments, a perspective projection flag indicates whether perspective or orthographic projection was used in the generation of the source video content.

[0212] In some embodiments, a data precision format indicator syntax element indicates the format and length of the syntax elements that signal the coefficients of the projection matrix and world to camera matrix, and the depth parameters.

[0213] In some embodiments, another category of parameters is the projection matrix parameters. The semantics can describe a 4x4 projection matrix variable, GriProjectionCoeff[ i ][ j ], which represents both rotation and translation. In some embodiments, the first three rows of that matrix are directly signaled by syntax elements, e.g., in the GRI SEI message. In some embodiments, the 4throw can be derived based on the perspective projection flag, e.g., to be {0, 0, 1, 0} for perspective projection or { 0, 0, 0, 1} for orthographic projection.

[0214] In some embodiments, another category of parameters is the world to camera matrix parameters. In some embodiments, semantics can describe a 4x4 projection matrix variable, GriWorldToCamCoeff[ i ][ j ], which can represent both rotation and translation. In some embodiments, the first three rows of that matrix are directly signaled by syntax elements in the SEI message. In some embodiments, the 4throw is set to {0, 0, 0, 1}. In some embodiments, the parameters in this category may change for each picture for gaming content.

[0215] In some embodiments, another category of parameters is depth parameters, which describe the samples in the picture containing the depth map. In some embodiments, the presence of the depth map is optional, so these parameters may not need to be signaled.

[0216] In some embodiments, syntax elements for a near plane and a far plane of the depth map can be included in this category. In some embodiments, a depth inverse flag indicates whether the samples in the depth map indicate the Z depth directly or an inverse of the Z depth. In some embodiments, a focal plane syntax element indicates the focal length.

[0217] In some embodiments, a cancel flag is used to cancel the persistence of the GRI SEI message (and the parameters provided therein or in previous GRI SEI messages in other coded pictures in the same coded picture / video sequence), which otherwise persists. In some embodiments, for the first occurrence of a GRI SEI message in a CLVS, if a presence flag for a category indicates that the syntaxelements for a category are not present, default values are determined, retrieved, or inferred for each syntax element. In some embodiments, in any subsequent occurrence of a GRI SEI message in a CVLS, if a presence flag for a category indicates that the syntax elements for a category are not present, the values of the syntax elements are inferred to be equal to the value of the corresponding syntax elements in the previous GRI SEI message in the CLVS.

[0218] Example syntax for the GRI SEI message, according to some embodiments, is provided in Table 1, and example semantics for the GRI SEI message are shown below.Table 1. Example syntax for a GRI SEI message.graphics_rendering_info( payloadSize ) { Descriptor gri_cancel_flag u(1) if ( !gri_cancel_flag ) {gri_game_engine_params_present_flag u(1) gri_projection_matrix_present_flag u(1) gri_world_to_cam_matrix_present_flag u(1) gri_depth_params_present_flag u(1) if( gri_game_engine_parameters_present_flag ) {gri_y_axis_up_flag u(1) gri_right_handed_flag u(1) gri normalization range flag u(1) gri_perspective_projection_flag u(1) gri_data_precision_format_idc u(2) }if( gri_projection_matrix_present_flag )for( i = 0; i < 3; i++ )for(j = 0; j < 4; j++)gri_projection_coeff[ i ][ j ] u(v) if( gri_world_to_cam_matrix_present_flag )for( i = 0; i < 3; i++ )for(j = 0; j < 4; j++)gri_world_to_cam_coeff[ i ][ j ] u(v) if( gri_depth_params_present_flag ) {gri_near_plane u(v) gri_far_plane u(v) gri_depth_inverse_flag u(1) gri_focal_length u(v) }}}

[0219] The graphics rendering information (GRI) SEI message provides parameters used to create the source content by a graphics renderer. A GRI SEI message may optionally indicate various parameters for auxiliary pictures of type AUX_DEPTH used to create of the source content by a graphics renderer.

[0220] In some embodiments, when the GRI SEI message is present in the current CVS, it may be present in a primary layer. In some embodiments, the GRI SEI message additionally applies to auxiliary layers of type AUXJDEPTH associated with the primary layer containing the GRI SEI message.

[0221] In some embodiments, gri_cancel_flag equal to ‘1’ indicates that the SEI message cancels the persistence of any GRI SEI message in output order that applies to the current layer. gri_cancel_flag equal to ‘0’ indicates that graphics rendering information follows.

[0222] In some embodiments, the graphics rendering information applies to the current picture and all subsequent pictures in the same layer in output order until one or more of the following conditions are true:- A new CLVS of the current layer begins.- The bitstream ends.- A picture in the current layer in a PU containing a GRI SEI message with gri_cancel_flag equal to ‘1’ is output that follows the current picture in output order.

[0223] In some embodiments, the variable PrevGriSei PresentFlag is derived as follows:- If gri_cancel_flag equal to ‘1 ’, the variable PrevGriSei P resentFI ag is set equal to ‘O’.- Otherwise, the following applies:o If within the current CLVS there is a GRI SEI message present in a PU preceding the current PU in decoding order, PrevGriSeiPresentFlag is set equal to ‘1’ - Otherwise, PrevGriSeiPresentFlag is set equal to ‘O’.

[0224] In some embodiments, gri_game_engine_params_present_flag equal to ‘1’ indicates that the gri_y_axis_up_flag, gri_left_hand_flag, gri_normalization_range_flag, and gri_data_precision_format_idc syntax elements are present. gri_game_engine_params_present_flag equal to ‘0’ indicates that the gri_y_axis_up_flag, gri_right_hand_flag, gri_normalization_range_flag, and gri_data_precision_format Jdc syntax elements are not present.

[0225] In some embodiments, gri_world_to_cam_matrix_present_flag equal to ‘1’ indicates that the gri_projection_coeff[ i ][ j ] syntax elements are present. gri_world_to_cam_matrix_present_flag equal to ‘0’ indicates that the gri_projection_coeff[ i ][ j ] syntax elements are not present.

[0226] In some embodiments, gri_world_to_cam_matrix_present_flag equal to ‘1’ indicates that the gri_world_to_cam_coeff[ i ][ j ] syntax elements are present. gri_world_to_cam_matrix_present_flag equal to ‘0’ indicates that the gri_world_to_cam_coeff[ i ][ j ] syntax elements are not present.

[0227] In some embodiments, gri_depth_params_present_flag equal to ‘1’ indicates that the gri_disparity_flag, gri_near_plane, gri_far_plane, and gri_focal Jength syntax elements are present. gri_depth_params_present_flag equal to ‘0’ indicates that the gri_depth_inverse_flag, gri_near_plane, gri_far_plane, and gri_focal Jength syntax elements are not present.

[0228] In some embodiments, gri_y_axis_up_flag equal to ‘1’ indicates that the graphics engine used to create of the source content has a vertical Y axis in its coordinate system. gri_y_axis_up_flag equal to ‘0’ indicates that the graphics engine used to create of the source content has a vertical Z axis in its coordinate system.

[0229] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_y_axis_up_flag is inferred to be equal to ‘1’.

[0230] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_y_axis_up_flag is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0231] In some embodiments, gri_right_handed_flag equal to ‘1’ indicates that the the graphics engine used to create of the source content indicates that the graphics engine used to create of the source content uses a right handed coordinate system. gri_right_handed_flag equal to ‘1’ indicates that the the graphics engine used to create of the source content indicates that the graphics engine used to create of the source content uses a left handed coordinate system.

[0232] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_right_handed_flag is inferred to be equal to ‘1’.

[0233] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_ right_handed_flag is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0234] In some embodiments, gri_normalization_range_flag equal to ‘1’ indicates that the graphics engine used to create of the source content uses normalized coordinates in range of -1 to 1 inclusive. gri_normalization_range_flag equal to ‘0’ indicates that the graphics engine used to create of the source content uses normalized coordinates in range of 0 to 1, inclusive.

[0235] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_normalization_range_flag is inferred to be equal to ‘1’.

[0236] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_normalization_range_flag is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0237] In some embodiments, gri_perspective_projection_flag equal to ‘1’ indicates that the graphics engine used to create the source content uses projection projection. In some embodiments, gri_perspective_projection_flag equal to ‘0’ indicates that the graphics engine used to create the source content uses orthographic projection.

[0238] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_perspective_projection_flag is inferred to be equal to ‘1’.

[0239] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_perspective_projection_flag is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0240] In some embodiments, gri_data_format_idc indicates the data format used to signal the gri_projection_coeff[ i ][j ], gri_world_to_cam_coeff[ i ][j ], gri_near_plane, gri_far_plane, and gri_focal Jength syntax elements, as specified in Table 2.

[0241] In some embodiments, the value of gri_data_format_idc shall be in the range of 0 to 1, inclusive of 0, 1, and any values therebetween. In some embodiments, values 2 and 3 may be reserved for future use by ITU-T | ISO / IECTable 2. Examp e mapping of gri_data_format_idc gri_data_format_idc DataFormatType GriParamLen0 binary32 32in ISO / IEC60559:20201 binaryl 6 16in ISO / IEC60559:20202..3 Reserved

[0242] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_data_format_idc is inferred to be equal to ‘O’.

[0243] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_data_format_idc is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0244] In some embodiments, gri_projection_coeff[ i ][ j ] indicates the (i, j)-th coefficient of a 4x4 projection matrix used by a game engine to render the source picture, in a floating point format of type DataFormatType, as specified in Table 3. The length of the syntax element is GriParamLen bits, as determined by Table 2.

[0245] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_projection_coeff[ i ][ j ] is inferred to be equal to (i = = j) ? 1 : 0.

[0246] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_projection_coeff[ i ][ j ] is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0247] In some embodiments, the variable GriProjectionCoeff[ i ][ j ] is a 2-dimensional array of size 4x4 derived as follows:for (i = 0; i < 3; i++)for 0 = 0; j < 4; j++)GriProjectionCoeff [ i ][ j ] = gri_projection_coeff[ i ][ j ]if ( grLperspecti ve_projection_flag )GriProjectionCoeff

[0003] = { 0, 0, -1, 0}elseGriProjectionCoeff

[0003] = { 0, 0, 0, 1}

[0248] In some embodiments, gri_world_to_cam_coeff[ i ][ j ] indicates the (i, j)-th coefficient of a 4x3 array that represents the first three rows of a 4x4 world to camera matrix used by a game engine to render the source picture, in a floating point format of type DataFormatType, as specified in Table 3. In some embodiments, the length of the syntax element is GriParamLen bits, as determined by Table 2.

[0249] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_world_to_cam_coeff[ i ][ j ] is inferred to be equal to (i = = j) ? 1 : 0.

[0250] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_world_to_cam_coeff[ i ][ j ] is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0251] In some embodiments, the variable GriWorldToCamCoeff[ i ][ j ] is a 2-dimensional array of size 4x4 derived as follows:for (i = 0; i < 3; i++)for 0 = 0; j < 4; j++)GriWorldToCamCoeff[ i ][ j ] = gri_world_to_cam_coeff [ i ][ j ]GriWorldToCamCoeff

[0003] = { 0, 0, 0, 1}

[0252] In some embodiments, gri_near_plane specifies the the closest distance to the camera used for rendering the source content, as used in the depth mapping process. In some embodiments, the length of the syntax element is GriParamLen bits.

[0253] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_near_plane is inferred to be equal to 0.01.

[0254] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_near_plane is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0255] In some embodiments, gri_far_plane specifies the the farthest distance to the camera used for rendering the source content, as used in the depth mapping process. In some embodiments, the length of the syntax element is GriParamLen bits.

[0256] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_far_plane is undefined.

[0257] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_far_plane is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0258] In some embodiments, gri_depth_inverse_flag equal to ‘1’ specifies that each decoded luma sample value of an associated auxiliary picture of type AUX_DEPTH represents an inverse of Z value in the range of gri_far_plane to gri_near_plane, . In some embodiments, gri_depth Jnverse_flag equal to ‘0’ specifies that each decoded luma sample value of an associated auxiliary picture of type AUX_DEPTH represents a Z value in the range of gri_near_plane to gri_far_plane.

[0259] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_depth_inverse_flag is inferred to be equal to ‘1’.

[0260] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_depth Jnverse_flag is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0261] In some embodiments, gri_focal_plane specifies the focal length of the camera used for rendering the source content. In some embodiments, the length of the syntax element is GriParamLen bits.

[0262] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_focal_plane is undefined.

[0263] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_focal_plane is inferred to be equal to the value of the corresponding syntax element in the previous GRI SEI message.

[0264] In some embodiments, this process can be used to interpret the sample values of auxiliary pictures of type AUX_DEPTH associated with the GRI SEI message.

[0265] In some embodiments, the value of depth Z associated with a luma sample value of an auxiliary picture of type AUX_DEPTH is represented in a normalized device coordinates with a range indicated by gri_normalization_range_flag and gri_depth_inverse_flag. In some embodiments, the normalized device coordinate depth ZNDCis evaluated as:aZNDC=+C

[0266] In some embodiments, in instances in which gri_normalization_range_flag is equal to ‘O’:a = 2 ■ gri_near_plane, c = — 1.

[0267] In some embodiments, when gri_normalization_range_flag is equal to ‘1’:2 ■ gri_near_plane ■ gri_far_plane gri_far_plane + gri_near_plane a = - , c = - ; gri_far_plane — gri_near_plane gri_far_plane — gri_near_plane

[0268] In some embodiments, in instances in which gri_depth Jnverse_flag is equal to ‘O’:„ >ZNDC + 1 >zbuf Engine >

[0269] In some embodiments, when gri_depth_inverse_flag is equal to ‘1’:„ > 1— ZNDCzbuf Engine

[0270] In some embodiments, this process can be used to to convert the projection matrix and the world to camera matrix depending on the values of gri_y_axis_up_flag and gri_right_handed_flag.

[0271] In some embodiments, if a further process utilizing the decoded bitstream is using a coordinate system that is different from a coordinate system indicated by gri_y_axis_up_flag and gri_right_handed_flag syntax elements in the VSEI message a coordinate system conversion process can be invoked.

[0272] In some embodiments, to convert projection matrix gri_projection_coeff[ i ][j ] or world to camera matrix gri_world_to_cam_coeff[ i ][ j ] M from vertical Y axis (gri_y_axis_up_flag is equal to 1) to projection matrix gri_projection_coeff[ i ][ j ] or world to camera matrix gri_world_to_cam_coeff[ i ][ j ] M' to verical Z axis, the following process can be applied.1 0 0 0- M' = M ■0001 01 00 ■-0 0 0 1-

[0273] In some embodiments, to convert projection matrix gri_projection_coeff[ i ][j ] or world to camera matrix gri_world_to_cam_coeff[ i ][ j ] M from vertical Z axis (gri_y_axis_up_flag is equal to 0) toprojection matrix gri_projection_coeff[ i ][ j ] or world to camera matrix gri_world_to_cam_coeff[ i ][ j ] M' to verical Y axis, the following process can be applied.1 0 0 0- M' = M ■0 0 1 00 1 0 0 ■-0 0 0 1-

[0274] In some embodiments, further processing of the decoded picture may be performed with a graphics engine with a left handed coordinate system when gri_right_handed_flag is equal to ‘1’, or the graphics engine with a right handed coordingate system when gri_right_handed_flag is equal to ‘O’.

[0275] In some embodiments, to convert projection matrix gri_projection_coeff[ i ][j ] or world to camera matrix gri_world_to_cam_coeff[ i ][j ] M from the right hand coordinate system (gri_right_handed_flag is equal to 1) to projection matrix gri_projection_coeff[ i ][j ] or world to camera matrix gri_world_to_cam_coeff[ i ][ j ] M' to the left hand coordinate system, the following process can be applied.-1 0 0 0- M' = M ■01 0 00 0 1 0 ■- 0 0 0 1-

[0276] In some embodiments, to convert projection matrix gri_projection_coeff[ i ][j ] or world to camera matrix gri_world_to_cam_coeff[ i ][j ] M from the left hand coordinate system (gri_right_handed_flag is equal to 0) to projection matrix gri_projection_coeff[ i ][ j ] or world to camera matrix gri_world_to_cam_coeff[ i ][ j ] M' to the right hand coordinate system, the following process can be applied.-1 0 0 0- M' = M ■01 0 00 0 1 0 ■- 0 0 0 1-

[0277] In some instances, the focal plane may be explicitly indicated for vertical and horizontal components. In some embodiments, a symmetric focus flag is added, and when it is equal to ‘O’, two separate syntax elements are signaled for the focal plane in the X and Y direction.Table 3. Example syntax for a GRI SEI message.graphics_rendering_info( payloadSize ) { Descriptor gri_cancel_flag u(1) if ( !gri_cancel_flag ) {gri_game_engine_params_present_flag u(1) gri_projection_matrix_present_flag u(1) gri_world_to_cam_matrix_present_flag u(1) gri_depth_params_present_flag u(1) if (gri_game_engine_parameters_present_flag ) {gri_y_axis_up_flag u(1) gri_right_handed_flag u(1) gri normalization range flag u(1) gri_perspective_projection_flag u(1) gri_symmetric_focus_flag u(1) gri_data_precision_format_idc u(2) }if ( gri_projection_matrix_present_flag )for( i = 0; i < 3; i++ )for(j = 0; i < 4; j++)gri_projection_coeff[ i ][ j ] u(v) if ( gri_world_to_cam_matrix_present_flag )for( i = 0; i < 3; i++ )for(j = 0; i < 4; j++)gri_world_to_cam_coeff[ i ][ j ] u(v) if (gri_depth_params_present_flag) {gri_near_plane u(v) gri_far_plane u(v) gri_depth_inverse_flag u(1) if ( gri_symmetric_focus_flag )gri_focal_length u(v) else {gri_focal_length_x u(v) gri_focal_length_y u(v) }}}}

[0278] In some embodiments, gri_symmetric_focus_flag equal to ‘1’ indicates that the graphics engine used to create the source content uses symmetric focal plane along X and Y axis.

[0279] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘O’, the value of gri_symmetric_focus_flag is inferred to be equal to ‘1’.

[0280] In some embodiments, when not present and PrevGriSei PresentFlag equal to ‘1’, the value of gri_symmetric_focus_flag is inferred to be equal to the value of the corresponding sytnax element in the previous GRI SEI message.

[0281] In some embodiments, gri_focal_length_x and gri_focal_length_y specify the horizontal and vertical focal lengths, respectively, of the camera used for rendering the source content. In some embodiments, the length of the syntax element is GriParamLen bits.

[0282] In some embodiments, when not present and PrevGriSeiPresentFlag equal to ‘O’, the value of gri_focal_plane_x and gri_focal_plane_y are undefined.

[0283] In some embodiments, when not present and PrevGriSeiPresentFlag equal to ‘1’, the value of gri_focal_plane_x and gri_focal_plane_y are inferred to be equal to the value of the corresponding sytnax element in the previous GRI SEI message.

[0284] In some embodiments, the encoder device 400 may be configured to receive source video, e.g., from a graphics rendering application such as a game rendering application or a rendering engine. The source video can comprise, e.g., captured images / video, rendered graphics, rendered images / video, captured image / video data, rendered graphics data, rendered images / video data, and / or the like. In some embodiments, the source video can further comprise or information about one or more parameters that were used, e.g., by the graphics rendering application and / or image / video capturing device(s), to capture or generate the various elements / layers of the source video. In other embodiments, the source video can be received at the encoder device 400 and separately the encoder device 400 can receive information about one or more parameters that were used, e.g., by the graphics rendering application and / or image / video capturing device(s), to capture or generate the various elements / layers of the source video. In some embodiments, the encoder device 400 can be further configured to form an SEI message containing presence flags for one or more categories of syntax elements and syntax elements representing the parameters, including any one or more of: a vertical axis identification category, a coordinate system handedness category, a projection format category, a length of the coefficients of the projection matrix category, a world to camera matrix category, and / or the like.

[0285] The encoder device 400 can be further configured to encode the source video and form a bitstream with the coded source video and SEI message. The encoder device 400 can be further configured to receive graphics data (e.g., a depth map video, a graphics sequence, a video graphicssequence, rendered video frames, rendered images, etc.) generated by a graphics rendering application. The graphics data (e.g., depth map video) corresponds to the source video received from the graphics rendering application. The encoder device 400 can be further configured to generate an SEI message for at least one picture or frame in the video sequence, depth map video, etc., and include a presence flag for depth parameters, and include the near plane, far plane, and inverse flag syntax elements in the SEI message.

[0286] The encoder device 400 can be further configured to include a first SEI message in a first coded picture to be transmitted in the bitstream and set each respective presence flag in the first SEI message to a value of ‘1’ if updated / new GRI is being provided / to be provided in the first SEI message for respective parameters in a respective GRI category represented by the respective presence flag in the first SEI message, and / or set each respective presence flag in the first SEI message to a value of ‘0’ if updated / new GRI is not being provided / to be provided in the first SEI message for respective parameters in a respective GRI category represented by the respective presence flag in the first SEI message. The encoder device 400 can be further configured to encode a second coded picture. The encoder device 400 can be further configured to generate a second SEI message for the second coded picture. If updated / new GRI is being provided / to be provided in the second SEI message relative to the GRI provided in the first SEI message, at least one respective presence flag for at least one parameter in at least one respective GRI category is to be set to a value of ‘1’ in the second SEI message. Alternatively, if no updated / new GRI is being provided / to be provided in the second SEI message relative to the GRI provided in the first SEI message, all presence flags are to be set to a value of ‘0’ in the second SEI message.

[0287] In some embodiments, SEI messages, e.g., the first SEI message and / or the second SEI message, can be encoded in / with a coded picture to be transmitted / encoded in-band (i.e., together in the bitstream).

[0288] In some embodiments, the encoder device 400 can be configured to include a first SEI message in a first coded picture, and set a first presence flag to a value of ‘0’ to indicate to the decoder device 300 that that no updated / new GRI is provided in the first SEI message for any of the parameters in a first GRI category associated with the first presence flag, and the encoder device 400 can also set a second presence flag in the first SEI message to a value of ‘1’ to indicate to the decoder device 300 that at least one updated / new GRI is provided in the first SEI message for at least one parameter in a second GRI category associated with the second presence flag. Then, in a second coded picture for which the at least one updated / new GRI from the first SEI message still applies, the encoder device 400 can include a second SEI message in which at least the second presence flag (and possibly also the first presenceflag) is set to a value of ‘0’ to indicate to the decoder device 300 that, in the second SEI message in the second coded picture, no additional updated / new GRI are being provided for parameters in the second GRI category. Based on the second presence flag in the second SEI message being set to a value of ‘O’, the decoder device 300 may determine that the updated / new GRI provided in the first SEI message for the parameters in the second GRI category still apply when the decoder device 300 is decoding, rendering, creating, displaying, or otherwise processing the second coded picture.

[0289] In some embodiments, the decoder device 300 can be configured to receive a bitstream containing coded pictures and SEI messages in / with those coded pictures. The SEI messages can be or comprise GRI SEI message(s). In some embodiments, the decoder device 300 can be configured to decode the coded pictures received in the bitstream. In some embodiments, the decoder device 300 can be configured to parse syntax elements in the SEI messages. In some embodiments, the syntax elements can comprise one or more presence flags for one or more categories of syntax elements. In some embodiments, the syntax elements in the SEI messages can comprise syntax elements representing one or more parameters, such as one or more of: a vertical access identification flag, a coordinate system handedness flag, a format of the coefficients of the projection matrix, a world to camera matrix, etc.

[0290] In some embodiments, the decoder device 300 can be configured to use the parameters of the SEI message in one or more of: transcoding, analysis, reprojection, generation of derivative content of the decoded video pictures, etc.

[0291] In some embodiments, the decoder device 300 can be configured to decode pictures representing, e.g., a depth map video or the like. In some embodiments, the decoder device 300 can be configured to receive and parse depth map parameters in SEI messages. In some embodiments, the decoder device 300 can be configured to perform mapping of decoded samples of, e.g., a depth map video, based on one or more parameters, e.g., depth map parameters, in SEI messages.

[0292] In some embodiments, the decoder device 300 can be configured to receive a bitstream containing at least one coded picture containing a GRI SEI message. In some embodiments, the GRI SEI message can comprise one or more presence flags. In an instance in which at least one presence flag in the GRI SEI message is set to a value of ‘O’, the decoder device 300 can be configured to interpret this presence flag value of ‘0’ as indicating that no updated / new GRI is provided in the GRI SEI message for any parameter in at least one GRI category associated with the at least one presence flag.

[0293] In some embodiments, the decoder device 300 can be configured to decode the at least one coded picture and parse the GRI SEI message contained within the at least one coded picture. In some embodiments, the decoder device 300 can be configured to, in an instance in which a particular presence flag is set to a value of ‘O’, infer that all syntax element values of parameters in a particular GRI categoryassociated with the particular presence flag should be equal to one or more previously received syntax element values of the parameters in the particular GRI category associated with the particular presence flag and / or one or more default syntax element values of the parameters in the particular GRI category associated with the particular presence flag.

[0294] In some embodiments, the decoder device 300 can be configured to receive a bitstream comprising / containing at least a first coded picture and a second picture. Each of the first and second coded pictures comprises metadata coded therein / therewith, such as a SEI message (e.g., a GRI SEI message). In some embodiments, the first coded picture can comprise a first SEI message having at least one presence flag set to a value of ‘1’, and the second coded picture can comprise a second SEI message having the same at least one presence flag set to a value of ‘O’. In such a circumstance, the decoder device 300 according to some embodiments can be configured to decode the first coded picture and the second coded picture to generate a first decoded picture and a second decoded picture, parse from the first SEI message one or more syntax element values for one or more parameters in at least one GRI category for which the at least one presence flags were set to a value of ‘1’ in the first SEI message, apply the one or more syntax element values when rendering or generating a first target picture from the first decoded picture, infer that the one or more syntax element values from the first SEI message remain the same in the second SEI message due to the same at least one presence flag being set to a value of ‘0’ in the second SEI message, and apply, when rendering or generating a second target picture from the second decoded picture, the same one or more syntax element values for the same one or more parameters in the same at least one GRI category for which the at least one presence flags were set to a value of ‘1’ in the first SEI message.

[0295] In some embodiments, the decoder device 300 can be configured to receive a bitstream containing at least a first coded picture containing or comprising a first GRI SEI message and a second coded picture containing a second GRI SEI message. Each of the first and second GRI SEI messages can comprise a first presence flag associated with a first GRI category of first parameters used when capturing / rendering the first coded picture and a second presence flag associated with a second GRI category of second parameters used when capturing / rendering the first coded picture. In some embodiments, the first coded picture can be a first coded picture in a coded picture sequence or a coded video sequence and the second coded picture can be a subsequent coded picture in the same coded picture sequence or the same coded video sequence. The first presence flag is configured to binarily indicate whether the GRI SEI message includes updated / new GRI / syntax element values for at least one parameter in the first GRI category of first parameters. Likewise, the second presence flag is configured to binarily indicate whether the GRI SEI message includes updated / new GRI / syntax element values forat least one parameter in the second GRI category of second parameters. This binary indication using a presence flag can be performed or achieved using a bit value of ‘0’ to indicate that the presence flag is ‘not set’ and a bit value of ‘1’ to indicate that the presence flag is ‘set’.

[0296] In some embodiments, in the first GRI SEI message, the first presence flag can be set to a value of ‘O’, indicating that the first GRI SEI message does not contain / comprise any updated / new GRI / syntax element values for any parameter in the first GRI category of first parameters and the second presence flag in the first GRI SEI message can be set to a value of ‘1’, indicating that the first GRI SEI message contains / comprises at least one updated / new GRI / syntax element value for at least one parameter in the second GRI category of second parameters. In some embodiments, the decoder device 300 can be configured to, based at least on these bit values of the first and second presence flags in the first GRI SEI message, determine that the first GRI SEI message does not contain any updated / new GRI / syntax element values for any parameter in the first GRI category of first parameters and that the first GRI SEI message does contain at least one updated / new GRI / syntax element value for at least one parameter in the second GRI category of second parameters. Based on this, the decoder device 300 can determine that it does not need to parse out from the first GRI SEI message any of the GRI / syntax element values from the first GRI category of first parameters, and the decoder device 300 can further determine that it does need to parse out from the first GRI SEI message the GRI / syntax element values from the second GRI category of second parameters.

[0297] In some embodiments, the decoder device 300 can be further configured to determine whether the decoder device 300 has stored therein first previously provided GRI / syntax element values for the first GRI category of first parameters and / or first default GRI / syntax element values for the first GRI category of first parameters, as well as whether the decoder device 300 has stored therein second previously provided GRI / syntax element values for the second GRI category of second parameters and / or second default GRI / syntax element values for the second GRI category of second parameters.

[0298] In some embodiments, such as in an instance in which the first coded picture is a first coded picture in a coded picture sequence or a coded video sequence, and further in an instance in which the first GRI SEI message includes the first presence flag set to a value of ‘0’ and the second presence flag set to a value of ‘1 ’, the decoder device 300 can be configured to parse out from the first GRI SEI message the GRI / syntax element values for the second parameters in the second GRI category, and the decoder device 300 can be further configured to, instead of parsing out from the first GRI SEI message the GRI / syntax element values for the first parameters in the first GRI category, retrieve / provide one or more default GRI / syntax element values associated with the first parameters in the first GRI category. Thedefault GRI / syntax element values can be stored at the decoder device 300 or can be provided by / retrieved from another system or device, such as the encoder device 400.

[0299] In some embodiments, such as in an instance in which the first coded picture is an nthcoded picture in a coded picture sequence or a coded video sequence that follows a first coded picture of the coded picture sequence of the coded video sequence, and further in an instance in which the first GRI SEI message includes the first presence flag set to a value of ‘0’ and the second presence flag set to a value of ‘1’, the decoder device 300 can be configured to parse out from the first GRI SEI message the GRI / syntax element values for the second parameters in the second GRI category, and the decoder device 300 can be further configured to, instead of parsing out from the first GRI SEI message the GRI / syntax element values for the first parameters in the first GRI category, retrieve / provide one or more previously provided GRI / syntax element values associated with the first parameters in the first GRI category. The previously provided GRI / syntax element values are provided in a prior coded picture that precedes the first coded picture in the coded picture sequence or the coded video sequence, assuming other, newer previously provided GRI / syntax element values for those same parameters were not been provided to the decoder device 300 in other coded pictures between the prior coded picture that precedes the first coded picture and the first coded picture in the coded picture sequence or the coded video sequence.

[0300] In some embodiments, the decoder device 300 can be configured to, after decoding the first coded picture, retrieving / providing the previously provided or default GRI / syntax element values for the first parameters in the first GRI category, and parsing out from the first GRI SEI message the updated / new GRI / syntax element values for the second parameters in the second GRI category, apply the previously provided or default GRI / syntax element values for the first parameters in the first GRI category and apply the updated / new GRI / syntax element values for the second parameters in the second GRI category when rendering / generating a target picture from the decoded first picture.

[0301] Once the second coded picture is decoded, the decoder device 300 can look to the first and second presence flags in the second GRI SEI message in the second decoded picture to determine whether updated / new GRI / syntax element values are provided in the second GRI SEI message. If, for example, the first presence flag is set to a value of ‘1’ in the second GRI SEI message and the second presence flag is set to a value of ‘0’ in the second GRI SEI message, the decoder device 300 can determine from these flag values that the second GRI SEI message contains updated / new GRI / syntax element values for one or more parameters in the second GRI category, and that the second GRI SEI message does not contain any updated / new GRI / syntax element values for any parameter in the first GRI category. Other permutations and examples with regard to the first presence flag value, the secondpresence flag value, and the value(s) of other presence flag(s) in the first or second GRI SEI message will be readily apparent. Several non-limiting examples are provided below.

[0302] If the first presence flag in the first GRI SEI message is set to a value of ‘0’ and the second presence flag in the first GRI SEI message is set to a value of ‘1’, and thereafter in the picture / video sequence both the first and second presence flags in the second SEI message are set to a value of ‘O’, the decoder device 300 can determine that the GRI / syntax element values used for the first and second parameters in the first and second GRI categories when rendering / generating the first target picture from the first decoded picture should also be used for the first and second parameters in the first and second GRI categories when rendering / generating the second target picture from the second decoded picture.

[0303] If the first presence flag in the first GRI SEI message is set to a value of ‘0’ and the second presence flag in the first GRI SEI message is set to a value of ‘1’, and thereafter in the picture / video sequence the first presence flag is set to a value of ‘1’ while the second presence flag is set to a value of ‘O’, the decoder device 300 can parse out from the second GRI SEI message the updated / new GRI / syntax element values for the first parameters in the first GRI category to be used for the first parameters in the first GRI category when rendering / generating the second target picture from the second decoded picture, and determine that the GRI / syntax element values used for the second parameters in the second GRI categories when rendering / generating the first target picture from the first decoded picture should also be used for the second parameters in the second GRI category when rendering / generating the second target picture from the second decoded picture.

[0304] Once the decoder device 300 determines whether updated / new GRI / syntax element values are provided for parameters in certain GRI categories in a GRI SEI message, it can then parse out all GRI / syntax element values for all parameters in the certain GRI categories and apply those GRI / syntax element values during target picture generation. Additionally or alternative, in certain embodiments, the decoder device 300 can be configured to compare the GRI / syntax element values in the certain GRI categories to previously provided and default GRI / syntax element values stored at the decoder device 300 to determine which of the GRI / syntax element values in the certain GRI categories in the GRI SEI message contain updated / new GRI / syntax element values based on the comparison.

[0305] In other embodiments, a GRI SEI message can comprise additional flags and / or syntax for some or all parameters in some or all GRI categories to indicate specifically which of the parameters in a particular GRI category is updated in the GRI SEI message.

[0306] Some or all of the elements, steps, or components of the approaches described herein can be carried out by a computing device or an apparatus comprising a processor and memory. Examples of such computing devices and apparatuses are described in more detail below. Referring now to bothFIG. 4 and FIG. 5, various aspects related to the functionality of the decoder device 300 and / or the encoder device 400 and components / configurations thereof are described. Embodiments of the present invention can be implemented as an apparatus or device, such as described above with regard to one or more embodiments of the decoder device 300 and / or one or more embodiments of the encoder device 400. In other embodiments, the present invention can be implemented as a computer program product that is executable on a computing device - such as by execution of program codes or computer-readable instructions stored on at least one memory device.

[0307] Some embodiments of the present invention may be implemented in various other ways, such as an article of manufacture. One example of an article of manufacture in the context of the invention disclosed herein is a computer program product that includes one or more software components including, for example, software objects, methods, data structures, program codes, computer-readable instructions, application-specific software, and / or the like. A software component may be coded in any of a variety of programming languages. An illustrative programming language may be a lower-level programming language such as an assembly language associated with a particular hardware architecture and / or operating system platform. A software component comprising assembly language instructions may require conversion into executable machine code by an assembler prior to execution by the hardware architecture and / or platform. Another example programming language may be a higher-level programming language that may be portable across multiple architectures. A software component comprising higher-level programming language instructions may require conversion to an intermediate representation by an interpreter or a compiler prior to execution.

[0308] Other examples of programming languages include, but are not limited to, a macro language, a shell or command language, a job control language, a script language, a database query or search language, and / or a report writing language. In one or more example embodiments, a software component comprising instructions in one of the foregoing examples of programming languages may be executed directly by an operating system or other software component without having to be first transformed into another form. A software component may be stored as a file or other data storage construct. Software components of a similar type or functionally related may be stored together such as, for example, in a particular directory, folder, or library. Software components may be static (e.g., pre-established or fixed) or dynamic (e.g., created or modified at the time of execution).

[0309] A computer program product may include a non-transitory computer-readable storage medium storing applications, programs, program modules, scripts, source code, program code, object code, byte code, compiled code, interpreted code, machine code, executable instructions, and / or the like (also referred to herein as executable instructions, instructions for execution, computer program products,program code, and / or similar terms used herein interchangeably). Such non-transitory computer-readable storage media include all computer-readable media (including volatile and non-volatile media).

[0310] In one embodiment, a non-volatile computer-readable storage medium may include a floppy disk, flexible disk, hard disk, solid-state storage (SSS) (e.g., a solid-state drive (SSD), solid state card (SSC), solid state module (SSM), enterprise flash drive, magnetic tape, or any other non-transitory magnetic medium, and / or the like. A non-volatile computer-readable storage medium may also include a punch card, paper tape, optical mark sheet (or any other physical medium with patterns of holes or other optically recognizable indicia), compact disc read only memory (CD-ROM), compact disc-rewritable (CD-RW), digital versatile disc (DVD), Blu-ray disc (BD), any other non-transitory optical medium, and / or the like. Such a non-volatile computer-readable storage medium may also include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory (e.g., Serial, NAND, NOR, and / or the like), multimedia memory cards (MMC), secure digital (SD) memory cards, SmartMedia cards, CompactFlash (CF) cards, Memory Sticks, and / or the like. Further, a non-volatile computer-readable storage medium may also include conductive-bridging random access memory (CBRAM), phase-change random access memory (PRAM), ferroelectric random-access memory (FeRAM), nonvolatile random-access memory (NVRAM), magnetoresistive random-access memory (MRAM), resistive random-access memory (RRAM), Silicon-Oxide-Nitride-Oxide-Silicon memory (SONOS), floating junction gate random access memory (FJG RAM), Millipede memory, racetrack memory, and / or the like.

[0311] In one embodiment, a volatile computer-readable storage medium may include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), fast page mode dynamic random access memory (FPM DRAM), extended data-out dynamic random access memory (EDO DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), double data rate type two synchronous dynamic random access memory (DDR2 SDRAM), double data rate type three synchronous dynamic random access memory (DDR3 SDRAM), Rambus dynamic random access memory (RDRAM), Twin Transistor RAM (TTRAM), Thyristor RAM (T-RAM), Zerocapacitor (Z-RAM), Rambus in-line memory module (RIMM), dual in-line memory module (DIMM), single in-line memory module (SIMM), video random access memory (VRAM), cache memory (including various levels), flash memory, register memory, and / or the like. It will be appreciated that where embodiments are described to use a computer-readable storage medium, other types of computer-readable storage media may be substituted for or used in addition to the computer-readable storage media described above.

[0312] As should be appreciated, various embodiments of the present invention may also be implemented as methods, apparatus, systems, computing devices, computing entities, and / or the like. As such, embodiments of the present invention may take the form of an apparatus, system, computing device, computing entity, and / or the like executing instructions stored on a computer-readable storage medium to perform certain steps or operations. Thus, embodiments of the present invention may also take the form of an entirely hardware embodiment, an entirely computer program product embodiment, and / or an embodiment that comprises combination of computer program products and hardware performing certain steps or operations.

[0313] In some embodiments, the decoding device 300 and / or the encoding device 400 according to one embodiment of the present invention. In general, the terms computing device, computing entity, computer, entity, device, system, and / or similar words used herein interchangeably may refer to, for example, one or more computers, computing entities, desktops, mobile phones, tablets, phablets, notebooks, laptops, distributed systems, kiosks, input terminals, servers or server networks, blades, gateways, switches, processing devices, processing entities, set-top boxes, relays, routers, network access points, base stations, the like, and / or any combination of devices or entities adapted to perform the functions, operations, and / or processes described herein. Such functions, operations, and / or processes may include, for example, transmitting, receiving, operating on, processing, displaying, storing, determining, creating / generating, monitoring, evaluating, comparing, and / or similar terms used herein interchangeably. In one embodiment, these functions, operations, and / or processes can be performed on data, content, information, and / or similar terms used herein interchangeably.

[0314] In some embodiments, the decoding device 300 and / or the encoding device 400 may include or be in communication with one or more processing elements (also referred to as processors, processing circuitry, and / or similar terms used herein interchangeably) that communicate with other elements within the decoding device 300 and / or the encoding device 400 via a bus, for example. As will be understood, the processing element of the decoding device 300 and / or the encoding device 400 may be embodied as one or more complex programmable logic devices (CPLDs), microprocessors, multi-core processors, coprocessing entities, application-specific instruction-set processors (ASIPs), microcontrollers, and / or controllers. Further, the processing element may be embodied as one or more other processing devices or circuitry. The term circuitry may refer to an entirely hardware embodiment or a combination of hardware and computer program products. Thus, the processing element may be embodied as integrated circuits, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), hardware accelerators, other circuitry, and / or the like. As will therefore be understood, the processing element may be configured for a particular use or configured to executeinstructions stored in volatile or non-volatile media or otherwise accessible to the processing element 402. As such, whether configured by hardware or computer program products, or by a combination thereof, the processing element may be capable of performing steps or operations according to embodiments of the present invention when configured accordingly.

[0315] In one embodiment, the decoding device 300 and / or the encoding device 400 may further include or be in communication with non-volatile media (also referred to as non-volatile storage, memory, memory storage, memory circuitry and / or similar terms used herein interchangeably). In one embodiment, the non-volatile storage or memory may include one or more non-volatile storage or memory media, including but not limited to hard disks, ROM, PROM, EPROM, EEPROM, flash memory, MMCs, SD memory cards, Memory Sticks, CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM, Millipede memory, racetrack memory, and / or the like. As will be recognized, the non-volatile storage or memory media may store databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, byte code, compiled code, interpreted code, machine code, executable instructions, and / or the like. The term database, database instance, database management system, and / or similar terms used herein interchangeably may refer to a collection of records or data that is stored in a computer-readable storage medium using one or more database models, such as a hierarchical database model, network model, relational model, entity-relationship model, object model, document model, semantic model, graph model, and / or the like.

[0316] In one embodiment, the decoding device 300 and / or the encoding device 400 may further include or be in communication with volatile media (also referred to as volatile storage, memory, memory storage, memory circuitry and / or similar terms used herein interchangeably). In one embodiment, the volatile storage or memory may also include one or more volatile storage or memory media 404, including but not limited to RAM, DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM, cache memory, register memory, and / or the like. As will be recognized, the volatile storage or memory media may be used to store at least portions of the databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, byte code, compiled code, interpreted code, machine code, executable instructions, and / or the like being executed by, for example, the processing element. Thus, the databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, byte code, compiled code, interpreted code, machine code, executable instructions, and / or the like may be used to control certain aspects of the operation of the decoding device 300 and / or the encoding device 400 with the assistance of the processing element and operating system.

[0317] In some embodiments, the decoding device 300 and / or the encoding device 400 may also include one or more network interfaces, such as a transceiver for communicating with various computing entities, such as by communicating data, content, information, and / or similar terms used herein interchangeably that can be transmitted, received, operated on, processed, displayed, stored, and / or the like. Such communication may be executed using a wired data transmission protocol, such as fiber distributed data interface (FDDI), digital subscriber line (DSL), Ethernet, asynchronous transfer mode (ATM), frame relay, data over cable service interface specification (DOCSIS), or any other wired transmission protocol. Similarly, the decoding device 300 and / or the encoding device 400 may be configured to communicate via wireless external communication networks using any of a variety of protocols, such as general packet radio service (GPRS), Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access 2000 (CDMA2000), CDMA2000 1X (1xRTT), Wideband Code Division Multiple Access (WCDMA), Global System for Mobile Communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), Time Division-Synchronous Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Evolved Universal Terrestrial Radio Access Network (E-UTRAN), Evolution-Data Optimized (EVDO), High Speed Packet Access (HSPA), High-Speed Downlink Packet Access (HSDPA), IEEE 802.11 (Wi-Fi), Wi-Fi Direct, 802.16 (WiMAX), ultra-wideband (UWB), infrared (IR) protocols, near field communication (NFC) protocols, Wibree, Bluetooth protocols, wireless universal serial bus (USB) protocols, and / or any other wireless protocol.

[0318] Although not shown, the decoding device 300 and / or the encoding device 400 may include or be in communication with one or more input elements, such as a keyboard input, a mouse input, a touch screen / display input, motion input, movement input, audio input, pointing device input, joystick input, keypad input, and / or the like. The decoding device 300 and / or the encoding device 400 may also include or be in communication with one or more output elements (not shown), such as audio output, video output, screen / display output, motion output, movement output, and / or the like.

[0319] The signals provided to and received from the decoding device 300 and / or the encoding device 400 may include signaling information / data in accordance with air interface standards of applicable wireless systems. In this regard, the decoding device 300 and / or the encoding device 400 may be capable of operating with one or more air interface standards, communication protocols, modulation types, and access types. More particularly, the decoding device 300 and / or the encoding device 400 may operate in accordance with any of a number of wireless communication standards and protocols, such as those described above. In a particular embodiment, the decoding device 300 and / or the encoding device 400 may operate in accordance with multiple wireless communication standards and protocols, such as UMTS, CDMA2000, 1xRTT, WCDMA, GSM, EDGE, TD-SCDMA, LTE, E-UTRAN, EVDO, HSPA,HSDPA, Wi-Fi, Wi-Fi Direct, WiMAX, UWB, IR, NFC, Bluetooth, USB, and / or the like. Similarly, the decoding device 300 and / or the encoding device 400 may operate in accordance with multiple wired communication standards and protocols, such as those described above, via a network interface.

[0320] Via these communication standards and protocols, the decoding device 300 and / or the encoding device 400 can communicate with various other entities using concepts such as Unstructured Supplementary Service Data (USSD), Short Message Service (SMS), Multimedia Messaging Service (MMS), Dual-Tone Multi-Frequency Signaling (DTMF), and / or Subscriber Identity Module Dialer (SIM dialer). The decoding device 300 and / or the encoding device 400 can also download changes, add-ons, and updates, for instance, to its firmware, software (e.g., including executable instructions, applications, program modules), and operating system.

[0321] According to one embodiment, the decoding device 300 and / or the encoding device 400 may include location determining aspects, devices, modules, functionalities, and / or similar words used herein interchangeably. For example, the decoding device 300 and / or the encoding device 400 may include outdoor positioning aspects, such as a location module adapted to acquire, for example, latitude, longitude, altitude, geocode, course, direction, heading, speed, universal time (UTC), date, and / or various other information / data. In one embodiment, the location module can acquire data, sometimes known as ephemeris data, by identifying the number of satellites in view and the relative positions of those satellites (e.g., using global positioning systems (GPS)). The satellites may be a variety of different satellites, including Low Earth Orbit (LEO) satellite systems, Department of Defense (DOD) satellite systems, the European Union Galileo positioning systems, the Chinese Compass navigation systems, Indian Regional Navigational satellite systems, and / or the like. This data can be collected using a variety of coordinate systems, such as the Decimal Degrees (DD); Degrees, Minutes, Seconds (DMS); Universal Transverse Mercator (UTM); Universal Polar Stereographic (UPS) coordinate systems; and / or the like.

[0322] Alternatively, the location information / data can be determined by triangulating a position of the decoding device 300 and / or the encoding device 400 in connection with a variety of other systems, including cellular towers, Wi-Fi access points, and / or the like. Similarly, the decoding device 300 and / or the encoding device 400 may include indoor positioning aspects, such as a location module adapted to acquire, for example, latitude, longitude, altitude, geocode, course, direction, heading, speed, time, date, and / or various other information / data. Some of the indoor systems may use various position or location technologies including RFID tags, indoor beacons or transmitters, Wi-Fi access points, cellular towers, nearby computing devices (e.g., smartphones, laptops) and / or the like. For instance, such technologies may include the iBeacons, Gimbal proximity beacons, Bluetooth Low Energy (BLE) transmitters, NFCtransmitters, and / or the like. These indoor positioning aspects can be used in a variety of settings to determine the location of someone or something to within inches or centimeters.

[0323] The decoding device 300 and / or the encoding device 400 may also comprise a user interface (that can include a display coupled to the processing element / controller) and / or a user input interface (coupled to the processing element / controller). For example, the user interface may be a user application, browser, user interface, and / or similar words used herein interchangeably executing on and / or accessible via the decoding device 300 and / or the encoding device 400 to interact with and / or cause display of information / datafrom the decoding device 300 and / or the encoding device 400, as described herein. The user input interface can comprise any of a number of devices or interfaces allowing the decoding device 300 and / or the encoding device 400 to receive data, such as a keypad (hard or soft), a touch display, voice / speech or motion interfaces, or other input device. In embodiments in which the decoding device 300 and / or the encoding device 400 comprises a keypad, the keypad can include (or cause display of) the conventional numeric (0-9) and related keys (#, *), and other keys used for operating the decoding device 300 and / or the encoding device 400 and may include a full set of alphabetic keys or set of keys that may be activated to provide a full set of alphanumeric keys. In addition to providing input, the user input interface can be used, for example, to activate or deactivate certain functions, such as screen savers and / or sleep modes.

[0324] The decoding device 300 and / or the encoding device 400 can also include volatile storage or memory and / or non-volatile storage or memory, which can be embedded and / or may be removable. For example, the non-volatile memory may be ROM, PROM, EPROM, EEPROM, flash memory, MMCs, SD memory cards, Memory Sticks, CBRAM, PRAM, FeRAM, NVRAM, MRAM, RRAM, SONOS, FJG RAM, Millipede memory, racetrack memory, and / or the like. The volatile memory may be RAM, DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM, DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, RDRAM, TTRAM, T-RAM, Z-RAM, RIMM, DIMM, SIMM, VRAM, cache memory, register memory, and / or the like. The volatile and non-volatile storage or memory can store databases, database instances, database management systems, data, applications, programs, program modules, scripts, source code, object code, byte code, compiled code, interpreted code, machine code, executable instructions, and / or the like to implement the functions of the decoding device 300 and / or the encoding device 400. As indicated, this may include a user application that is resident on the entity or accessible through a browser or other user interface for the decoding device 300 to communicate with the encoding device 400 and / or for the encoding device 400 to communication with the decoder device 300.

[0325] In another embodiment, the decoding device 300 and / or the encoding device 400 may include other components or functionalities. As will be recognized, these architectures and descriptions are provided for exemplary purposes only and are not limiting to the various embodiments.

[0326] Referring now to FIG. 6, a method 500 according to a particular embodiment of the present disclosure is illustrated. The method 500 can comprise: receiving, from a graphics rendering application, source video data comprising rendered graphics data, as shown at block 502. In some embodiments, the method 500 can further comprise: receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data, as shown at block 504. In some embodiments, the method 500 can further comprise: encoding pictures of the source video data into coded pictures, as shown at block 506. In some embodiments, the method 500 can further comprise: generating an information message comprising one or more GRI indicators configured to indicate the one or more parameters, as shown at block 508. In some embodiments, the method 500 can further comprise: including the coded pictures and the information message in the bitstream, as shown at block 510.

[0327] Some or all of the method 500 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 500. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 500.

[0328] Referring now to FIG. 7, a method 600 according to a particular embodiment of the present disclosure is illustrated. The method 600 can comprise: receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures, wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, and wherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data, as shown at block 602. In some embodiments, the method 600 can further comprise: decoding the bitstream to form decoded video data, as shown at block 604. In some embodiments, the method 600 can further comprise: performing image analysis on the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters, as shown at block 606.

[0329] Some or all of the method 600 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 600. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 600.

[0330] Referring now to FIG. 8, a method 700 according to a particular embodiment of the present disclosure is illustrated. The method 700 can comprise: receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures, wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, and wherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data, as shown at block 702. In some embodiments, the method 700 can further comprise: decoding the bitstream to form decoded video data, as shown at block 704. In some embodiments, the method 700 can further comprise: generating, from the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters, adjusted video data through any of transcoding, editing, or reprojection, as shown at block 706.

[0331] Some or all of the method 700 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 700. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 700.

[0332] Referring now to FIG. 9, a method 800 according to a particular embodiment of the present disclosure is illustrated. The method 800 can comprise: receiving, from a graphics rendering application, source video data comprising rendered graphics data, as shown at block 802. In some embodiments, the method 800 can further comprise: receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data, as shown at block 804. In some embodiments, the method 800 can further comprise: generating a supplemental enhancement information (SEI) message comprising one or more updated GRI indicators configured to indicate whether the SEI message comprises updated GRIassociated with the rendered graphics data in the source video data, as shown at block 806. In some embodiments, the method 800 can further comprise: in an instance in which the GRI received from the graphics rendering application includes updated GRI that is different from both existing GRI previously provided to a decoder-side device and default GRI, adding the updated GRI to the SEI message and setting at least one updated GRI indicator from among the one or more updated GRI indicators in the SEI message to a value that indicates that updated GRI associated with the rendered graphics data is provided in the SEI message, as shown at block 808. In some embodiments, the method 800 can further comprise: providing, to the decoder-side device, a bitstream comprising coded source video data that comprises coded graphics data, the bitstream further comprising the SEI message comprising the at least one GRI indicator set to the value that indicates that updated GRI associated with the rendered graphics data is provided in the SEI message, as shown at block 810.

[0333] Some or all of the method 800 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 800. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 800.

[0334] Referring now to FIG. 10, a method 900 according to a particular embodiment of the present disclosure is illustrated. The method 900 can comprise: receiving, from a graphics rendering application, source video data comprising rendered graphics data, as shown at block 902. In some embodiments, the method 900 can further comprise: receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data, as shown at block 904. In some embodiments, the method 900 can further comprise: generating a supplemental enhancement information (SEI) message comprising a plurality of updated GRI indicators configured to indicate whether the SEI message comprises updated GRI associated with the rendered graphics data in the source video data, the plurality of updated GRI indicators comprising a first updated GRI indicator associated with a first GRI category and a second updated GRI indicator associated with a second GRI category, as shown at block 906. In some embodiments, the method 900 can further comprise: if GRI received from the graphics rendering application includes first updated GRI associated with the first GRI category that is different from both first existing GRI associated with the first GRI category that was previously provided to a decoder-side device and first default GRI associated with the first GRI category, adding the first updated GRI to theSEI message and setting the first updated GRI indicator to a first value that indicates that first updated GRI from the first GRI category is provided in the SEI message associated with the rendered graphics data, as shown at block 908. In some embodiments, the method 900 can further comprise: if GRI received from the graphics rendering application includes second updated GRI that is different from both second existing GRI from the second GRI category that was previously provided to the decoder-side device and second default GRI associated with the second GRI category, adding the second updated GRI to the SEI message and setting the second updated GRI indicator in the SEI message to a second value that indicates that second updated GRI from the second GRI category is provided in the SEI message associated with the rendered graphics data, as shown at block 910. In some embodiments, the method 900 can further comprise: providing a bitstream comprising coded source video data comprising coded graphics data and the SEI message that comprises the first updated GRI and first GRI indicator set to the first value, the SEI message further comprising the second updated GRI and second GRI indicator set to the second value, as shown at block 912.

[0335] Some or all of the method 900 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 900. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 900.

[0336] Referring nowto FIG. 11, a method 1000 according to a particular embodiment of the present disclosure is illustrated. The method 1000 can comprise: receiving, from a graphics rendering application, source video data comprising rendered graphics data, as shown at block 1002. In some embodiments, the method 1000 can further comprise: receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data, as shown at block 1004. In some embodiments, the method 1000 can further comprise: generating a supplemental enhancement information (SEI) message comprising a plurality of updated GRI indicators configured to indicate whether the SEI message comprises updated GRI associated with the rendered graphics data in the source video data, the plurality of updated GRI indicators comprising a first updated GRI indicator associated with a first GRI category and a second updated GRI indicator associated with a second GRI category, as shown at block 1006. In some embodiments, the method 1000 can further comprise: if GRI received from the graphics rendering application includes first updated GRI associated with the first GRI category that is different from bothfirst existing GRI associated with the first GRI category that was previously provided to a decoder-side device and first default GRI associated with the first GRI category, adding the first updated GRI to the SEI message and setting the first updated GRI indicator to a first value that indicates that first updated GRI from the first GRI category is provided in the SEI message associated with the rendered graphics data, as shown at block 1008. In some embodiments, the method 1000 can further comprise: if GRI received from the graphics rendering application includes only GRI that are the same as either second existing GRI from the second GRI category that was previously provided to the decoder-side device or second default GRI associated with the second GRI category, setting the second updated GRI indicator in the SEI message to a second value that indicates that the SEI message does not comprise any updated GRI from the second GRI category, as shown at block 1010. In some embodiments, the method 1000 can further comprise: providing, to the decoder-side device, a bitstream comprising coded source video data that comprises coded graphics data, the bitstream further comprising the SEI message comprising the first updated GRI and the first GRI indicator set to the first value that indicates that the first updated GRI from the first GRI category is being provided in the SEI message associated with the rendered graphics data in the bitstream, the SEI message further comprising the second GRI indicator set to the second value that indicates that the SEI message does not include any updated GRI from the second GRI category, as shown at block 1012.

[0337] Some or all of the method 1000 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 1000. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 1000.

[0338] Referring now to FIG. 12, a method 1100 according to a particular embodiment of the present disclosure is illustrated. The method 1100 can comprise: receiving, from a graphics rendering application, first source video data comprising first rendered graphics data and first graphics rendering information (GRI) associated with the first rendered graphics data, as shown at block 1102. In some embodiments, the method 1100 can further comprise: generating a first supplemental enhancement information (SEI) message comprising one or more updated GRI indicators configured to indicate whether the first SEI message comprises updated GRI associated with the first rendered graphics data in the first source video data, as shown at block 1104. In some embodiments, the method 1100 can further comprise: in an instance in which the first GRI received from the graphics rendering application includesfirst updated GRI that is different from default GRI, adding the first updated GRI to the first SEI message and setting at least one updated GRI indicator from among the one or more updated GRI indicators in the first SEI message to a first value that indicates that the first updated GRI is being provided in the first SEI message, as shown at block 1106. In some embodiments, the method 1100 can further comprise: providing, to a decoder-side device, in a bitstream, first coded source video data that comprises first coded graphics data and the first SEI message, as shown at block 1108. In some embodiments, the method 1100 can further comprise: receiving, from the graphics rendering application, second source video data comprising second rendered graphics data and second GRI associated with the second rendered graphics data, as shown at block 1110. In some embodiments, the method 1100 can further comprise: generating a second SEI message comprising the one or more updated GRI indicators configured to indicate whether the second SEI message comprises updated GRI associated with the second rendered graphics data in the second source video data, as shown at block 1112. In some embodiments, the method 1100 can further comprise: if second GRI includes updated GRI that is different from the first updated GRI previously provided to the decoder-side device in the first SEI message and the default GRI, adding the second updated GRI to the second SEI message and setting at least one updated GRI indicator from among the one or more updated GRI indicators in the second SEI message to the first value that indicates that the second updated GRI is being provided in the second SEI message, as shown at block 1114. In some embodiments, the method 1100 can further comprise: if second GRI does not include any GRI that is different from the first updated GRI previously provided to the decoderside device in the first SEI message or the default GRI, setting the one or more updated GRI indicators in the second SEI message to a second value that indicates that the second SEI message does not contain further updated GRI, as shown at block 1116. In some embodiments, the method 1100 can further comprise: providing, to the decoder-side device, in the bitstream, second coded source video data that comprises second coded graphics data and the second SEI message, as shown at block 1118.

[0339] Some or all of the method 1100 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 1100. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 1100.

[0340] Referring now to FIG. 13, a method 1200 according to a particular embodiment of the present disclosure is illustrated. The method 1200 can comprise: receiving a bitstream comprising coded sourcevideo data that comprises rendered graphics data generated by a graphics rendering application, the bitstream further comprising a graphics rendering information (GRI) supplemental enhancement information (SEI) message associated with the rendered graphics data in the coded source video data, wherein the GRI SEI message comprises an updated GRI indicator configured to indicate whether updated GRI is provided in the GRI SEI message, as shown at block 1202. In some embodiments, the method 1200 can further comprise: in an instance in which the updated GRI indicator is set to a value that indicates that updated GRI is provided in the GRI SEI message, extracting the updated GRI from the GRI SEI message, as shown at block 1204. In some embodiments, the method 1200 can further comprise: decoding the coded source video data comprising the rendered graphics data from the bitstream, as shown at block 1206. In some embodiments, the method 1200 can further comprise: preparing target graphics from the rendered graphics data in the source video data decoded from the bitstream and based at least on the updated GRI extracted from the GRI SEI message, as shown at block 1208. In some embodiments, the method 1200 can further comprise: preparing a target video based on the coded source video data, the target video comprising the target graphics prepared from the rendered graphics decoded from the bitstream based at last upon the updated GRI extracted from the GRI SEI message, as shown at block 1210.

[0341] Some or all of the method 1200 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 1200. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 1200.

[0342] Referring now to FIG. 14, a method 1300 according to a particular embodiment of the present disclosure is illustrated. The method 1300 can comprise: receiving a bitstream comprising coded source video data that comprises rendered graphics data generated by a graphics rendering application, the bitstream further comprising a graphics rendering information (GRI) supplemental enhancement information (SEI) message associated with the rendered graphics data in the coded source video data, wherein the GRI SEI message comprises a first updated GRI indicator configured to indicate whether first updated GRI in a first GRI category is provided in the GRI SEI message and a second updated GRI indicator configured to indicate whether second updated GRI in a second GRI category is provided in the GRI SEI message, as shown at block 1302. In some embodiments, the method 1300 can further comprise: in an instance in which the first updated GRI indicator is set to a value that indicates that firstupdated GRI in the first GRI category is provided in the GRI SEI message, extracting the first updated GRI from the GRI SEI message, as shown at block 1304. In some embodiments, the method 1300 can further comprise: in an instance in which the second updated GRI indicator is set to a value that indicates that second updated GRI in the second GRI category is provided in the GRI SEI message, extracting the second updated GRI from the GRI SEI message, as shown at block 1306. In some embodiments, the method 1300 can further comprise: decoding the coded source video data comprising the rendered graphics data from the bitstream, as shown at block 1308. In some embodiments, the method 1300 can further comprise: preparing target graphics from the rendered graphics data in the source video data decoded from the bitstream and based at least on the first updated GRI and the second updated GRI extracted from the GRI SEI message, as shown at block 1310. In some embodiments, the method 1300 can further comprise: preparing a target video based on the coded source video data, the target video comprising the target graphics prepared from the rendered graphics decoded from the bitstream based at last upon the first updated GRI and the second updated GRI extracted from the GRI SEI message, as shown at block 1312.

[0343] Some or all of the method 1300 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 1300. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 1300.

[0344] Referring now to FIG. 15, a method 1400 according to a particular embodiment of the present disclosure is illustrated. The method 1400 can comprise: receiving a bitstream comprising coded source video data that comprises rendered graphics data generated by a graphics rendering application, the bitstream further comprising a graphics rendering information (GRI) supplemental enhancement information (SEI) message associated with the rendered graphics data in the coded source video data, wherein the GRI SEI message comprises a first updated GRI indicator configured to indicate whether first updated GRI in a first GRI category is provided in the GRI SEI message and a second updated GRI indicator configured to indicate whether second updated GRI in a second GRI category is provided in the GRI SEI message, as shown at block 1402. In some embodiments, the method 1400 can further comprise: in an instance in which the first updated GRI indicator is set to a first value that indicates that first updated GRI in the first GRI category is provided in the GRI SEI message, extracting the first updated GRI from the GRI SEI message, as shown at block 1404. In some embodiments, the method 1400 canfurther comprise: in an instance in which the second updated GRI indicator is set to a second value that indicates that second updated GRI in the second GRI category is not provided in the GRI SEI message, determining that the GRI SEI message does not comprise any updated GRI from the second GRI category, as shown at block 1406. In some embodiments, the method 1400 can further comprise: decoding the coded source video data comprising the rendered graphics data from the bitstream, as shown at block 1408. In some embodiments, the method 1400 can further comprise: preparing target graphics from the rendered graphics data in the source video data decoded from the bitstream and based at least on the first updated GRI extracted from the GRI SEI message, as shown at block 1410. In some embodiments, the method 1400 can further comprise: preparing a target video based on the coded source video data, the target video comprising the target graphics prepared from the rendered graphics decoded from the bitstream based at last upon the first updated GRI extracted from the GRI SEI message, as shown at block 1412.

[0345] Some or all of the method 1400 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 1400. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 1400.

[0346] Referring now to FIG. 16, a method 1500 according to a particular embodiment of the present disclosure is illustrated. The method 1500 can comprise: receiving, from an encoder-side device, in a bitstream, first coded source video data that comprises first coded graphics data and a first supplemental enhancement information (SEI) message, the first SEI message comprising one or more updated GRI indicators configured to indicate whether the first SEI message comprises first updated GRI associated with the first rendered graphics data, as shown at block 1502. In some embodiments, the method 1500 can further comprise: if at least one updated GRI indicator from among the one or more updated GRI indicators in the first SEI message is set to a first value, extracting the first updated GRI associated with the first rendered graphics data from the first SEI message, as shown at block 1504. In some embodiments, the method 1500 can further comprise: decoding the first coded source video data comprising the first rendered graphics data received in the bitstream, as shown at block 1506. In some embodiments, the method 1500 can further comprise: preparing, from the first rendered graphics data, based at least on the first updated GRI extracted from the first GRI SEI message, first target graphics from the first rendered graphics data, as shown at block 1508. In some embodiments, the method 1500can further comprise: preparing a first target video based on the first coded source video data and the first target graphics, as shown at block 1510. In some embodiments, the method 1500 can further comprise: receiving, from the encoder-side device, in the bitstream, second coded source video data that comprises second coded graphics data and a second SEI message, the second SEI message comprising one or more updated GRI indicators configured to indicate whether the second SEI message comprises second updated GRI associated with the second rendered graphics data, as shown at block 1512. In some embodiments, the method 1500 can further comprise: if the one or more updated GRI indicators in the second SEI message are set to a second value, determining that the second SEI message does not contain second updated GRI associated with the second rendered graphics data, as shown at block 1514. In some embodiments, the method 1500 can further comprise: decoding the second coded source video data comprising the second rendered graphics data received in the bitstream, as shown at block 1516. In some embodiments, the method 1500 can further comprise: preparing, from the second rendered graphics data, based at least on the first updated GRI extracted from the first GRI SEI message, second target graphics from the second rendered graphics data, as shown at block 1518. In some embodiments, the method 1500 can further comprise: preparing a second target video based on the second coded source video data and the second target graphics, as shown at block 1520.

[0347] Some or all of the method 1500 can be carried out using a system (e.g., 100), by an apparatus, or by a computing device, such as 102, 104, 106, 219, 211, 212, 217, 218, 300, 400, etc. For example, an apparatus can comprise at least one processor and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the apparatus to perform some or all of the method 1500. Additionally, a computer program product can be provided that comprises a non-transitory computer readable storage medium storing instructions thereon that, when executed by a processor, cause a machine or apparatus to perform some or all of the method 1500.

[0348] Some example embodiments have been described with reference to an information message, such as a SEI message, and in certain embodiments as a GRI SEI message. It needs to be understood, however, that embodiments may similarly be realized with any similar structures or data units, such as a metadata open bitstream unit (OBU), as specified in AV1 or AV2, for example. Specific SEI message syntax structures have been presented in some example embodiments, but it needs to be understood that embodiments generally apply to any SEI messages or any syntax structures that may have at least partly a similar intent as the intent of those specific SEI messages.

[0349] Some example embodiments have been described with the help of syntax of the bitstream. It needs to be understood, however, that the corresponding structure and / or computer program may reside at the encoder for generating the bitstream and / or at the decoder for decoding the bitstream.

[0350] Where example embodiments have been described with reference to an encoder, it needs to be understood that the resulting bitstream and the decoder have corresponding elements in them. Likewise, where example embodiments have been described with reference to a decoder, it needs to be understood that the encoder has structure and / or computer program for generating the bitstream to be decoded by the decoder.

[0351] The above-noted aspects and features may be implemented in systems, apparatuses, methods, articles and non-transitory computer-readable media depending on the desired configuration. The subject disclosure may be implemented in and used with a number of different types of devices, such as one or more computing devices, one or more codecs, one or more encoders, one or more user equipment, one or more rendering engines, one or more servers, one or more network access nodes, one or more relay stations, one or more display devices, and / or the like.

[0352] An example device can comprise at least one processor and at least one memory that stores thereon instructions which, when executed by the at least one processor, cause the device to perform some or all of the elements of the above-described method, according to various embodiments. In other examples, a computer program product, such as a non-transitory computer-readable storage medium can be provided that comprises instructions stored thereon that, when executed by at least one processor of an apparatus, cause the apparatus to perform some or all elements of a method such as that descried above, according to some embodiments. In other examples, an apparatus can be provided that comprises means for carrying out a method - such means can include, e.g., a processor and a memory storing computer-executable instructions or computer codes thereon that, when executed by the processor, cause the apparatus to perform some or all of a method such as one of the methods described herein.

[0353] As used herein, the terms “instructions,” “file,” “designs,” “data,” “content,” “information,” and similar terms may be used interchangeably, according to some example embodiments of the present invention, to refer to data capable of being transmitted, received, operated on, displayed, and / or stored. Thus, use of any such terms should not be taken to limit the spirit and scope of the disclosure. Further, where a computing device is described herein to receive data from another computing device, it will be appreciated that the data may be received directly from the other computing device or may be received indirectly via one or more computing devices, such as, for example, one or more servers, relays, routers, network access points, base stations, and / or the like.

[0354] As used herein, the term “computer-readable medium” refers to any medium configured to participate in providing information to a processor, including instructions for execution. Such a medium may take many forms, including, but not limited to a non-transitory computer-readable storage medium (for example, non-volatile media, volatile media), and transmission media. Transmission media include,for example, coaxial cables, copper wire, fiber optic cables, and carrier waves that travel through space without wires or cables, such as acoustic waves and electromagnetic waves, including radio, optical, and infrared waves. Signals include man-made transient variations in amplitude, frequency, phase, polarization, or other physical properties transmitted through the transmission media. Examples of non-transitory computer-readable media include a floppy disk, a flexible disk, hard disk, magnetic tape, any other non-transitory magnetic medium, a compact disc read only memory (CD-ROM), compact disc compact disc-rewritable (CD-RW), digital versatile disc (DVD), Blu-Ray, any other non-transitory optical medium, punch cards, paper tape, optical mark sheets, any other physical medium with patterns of holes or other optically recognizable indicia, a random access memory (RAM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), a FLASH-EPROM, any other memory chip or cartridge, a carrier wave, or any other non-transitory medium from which a computer can read. The term computer-readable storage medium is used herein to refer to any computer-readable medium except transmission media. However, it will be appreciated that where embodiments are described to use a computer-readable storage medium, other types of computer-readable mediums may be substituted for or used in addition to the computer-readable storage medium in alternative embodiments. By way of example only, a design file for a printed article may be stored on a computer-readable medium and may be read by a computing device, such as described hereinbelow, for controlling part or all of a three-dimensional (3D) printing process and associated apparatuses and components, according to various embodiments described herein.

[0355] As used herein, the term “circuitry” refers to all of the following: (a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry); (b) to combinations of circuits and computer program product(s) comprising software (and / or firmware instructions stored on one or more computer readable memories), such as (as applicable): (i) to a combination of processor(s) or (ii) to portions of processor(s) / software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions described herein); and (c) to circuits, such as, for example, a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present. This definition of “circuitry” applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term “circuitry” would also cover an implementation of merely a processor (or multiple processors) or portion of a processor and its (or their) accompanying software and / or firmware. The term “circuitry” would also cover, for example and if applicable to the particular claim element, a baseband integrated circuit or applicationsprocessor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, other network device, and / or other computing device.

[0356] As used herein, the term “computing device” refers to a specialized, centralized device, network, or system, comprising at least a processor and a memory device including computer program code, and configured to provide guidance or direction related to the charge transactions carried out in one or more charging networks.

[0357] As used herein, the terms “about,” “substantially,” and “approximately” generally mean plus or minus 10% of the value stated, e.g., about 250 m would include 225 pm to 275 pm, about 1,000 pm would include 900 pm to 1,100 pm. Any provided value, whether or not it is modified by terms such as “about,” “substantially,” or “approximately,” all refer to and hereby disclose associated values or ranges of values thereabout, as described above.

[0358] The described and illustrated embodiments are examples. Although the specification may refer to “an”, “one”, or “some” embodiment(s) in several locations of the text, this does not necessarily mean that each reference is made to the same embodiment(s), or that a particular feature only applies to a single embodiment. Single features of different embodiments may also be combined to provide other embodiments. Further, when a particular feature, structure, or characteristic is described in connection of an embodiment, it is within the knowledge of one skilled in the art to apply such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. It shall be understood that although the terms “first,” “second” and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0359] For the purposes of the present disclosure, the phrases “at least one of A or B”, “at least one of A and B”, and “A and / or B” means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase “A, B, and / or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0360] As used herein, "plurality" means two or more. As used herein, a "set" of items may include one or more of such items. As used herein, whether in the subject disclosure or the claims, the terms "comprising", "including", "carrying", "having", "containing", "involving", and the like are to be understood to be open-ended, i.e. , to mean including but not limited to. Only the transitional phrases "consisting of' and "consisting essentially of", respectively, are closed or semi-closed transitional phrases with respect to claims. Use ofordinal terms such as "first", "second", "third", etc., in the claims or the subject disclosure to modify an element does not by itself connote any priority, precedence, or order of one element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one element having a certain name from another element having a same name (but for useof the ordinal term) to distinguish the elements. As used herein, "and / or" and "at least one of' means that the listed items are alternatives, but the alternatives also include any combination of the listed items.

[0361] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, the combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. It should be appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning consistent with the particular concepts disclosed herein.

[0362] In some embodiments, one or more of the operations, steps, elements, or processes described herein may be modified or further amplified as described below. Moreover, in some embodiments, additional optional operations may also be included. It should be appreciated that each of the modifications, optional additions, and / or amplifications described herein may be included with the operations previously described herein, either alone or in combination, with any others from among the features described herein.

[0363] The provided method description, illustrations, and process flow diagrams are provided merely as illustrative examples and are not intended to require or imply that the steps of the various embodiments must each or all be performed and / or should be performed in the order presented or described. As will be appreciated by one of skill in the art, the order of steps in some or all of the embodiments described may be performed in any order. Words such as “thereafter,” “then,” “next,” etc. are not intended to limit the order of the steps; these words are simply used to guide the reader through the description of the methods. Further, any reference to claim elements in the singular, for example, using the articles “a,” “an,” or “the” is not to be construed as limiting the element to the singular.

[0364] Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these inventions pertain having the benefit of teachings presented in the foregoing descriptions and the associated drawings. Although the figures only show certain components of the apparatus and systems described herein, it is understood that various other components may be used in conjunction with the system. Therefore, it is to be understood that the inventions are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, the steps in the method described above may not necessarily occur in the order depicted in the accompanying diagrams, and in some cases one or more of the steps depicted may occur substantially simultaneously, or additional steps may be involved. Although specific terms are employed herein, they are used in ageneric and descriptive sense only and not for purposes of limitation. Specific equipment, apparatuses, systems, computing devices, communications equipment, and / or components described in the examples are for illustration only and not for purposes of limitation. For instance, any and all electronic devices, user equipment, mobile devices, terminal devices, network nodes, access nodes, radio access network nodes, cell towers, base stations, network functions, network elements, servers, structures, and / or the like, having any form factor, scale, dimensions, aesthetic attributes, internal structures or circuitry, external ports or antennas, and / or functional or mechanical properties, which are formed according to any of the disclosed methods, approaches, processes, or variations thereof, using any devices, equipment, apparatuses, systems, or variations thereof, or variations thereof, are all contemplated and covered by the present disclosure. None of the examples provided are intended to, nor should they, limit in any way the scope of the present disclosure.

[0365] Every document cited or referenced herein, including any cross referenced or related patent or application is hereby incorporated herein by reference in its entirety unless expressly excluded or otherwise limited. The citation of any document and / or the mention of methods or apparatuses as being conventional, typical, usual, or the like is not, and should not be taken as an acknowledgement or any form of suggestion that the reference or mentioned method / apparatus is prior art with respect to any invention disclosed or claimed herein or that it alone, or in any combination with any other reference or references, teaches, suggests or discloses any such invention or forms part of the common general knowledge in any country in the world. Further, to the extent that any meaning or definition of a term in this document conflicts with any meaning or definition of the same term in a document incorporated by reference, the meaning or definition assigned to that term in this document shall govern.

Claims

Claims1. An apparatus comprising:at least one processor; andat least one memory comprising instructions stored therein that, when executed by the at least one processor, cause the apparatus to perform at least:receiving, from a graphics rendering application, source video data comprising rendered graphics data;receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data;encoding pictures of the source video data into coded pictures;generating an information message comprising one or more GRI indicators configured to indicate the one or more parameters; andincluding the coded pictures and the information message in a bitstream.

2. The apparatus of claim 1 , wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:providing the coding pictures and the information message to a decoding device in the bitstream.

3. The apparatus of claim 1 or claim 2, wherein the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category.

4. The apparatus of claim 3, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:in an instance in which the GRI received from the graphics rendering application includes at least one first parameter of the first category, setting a first GRI indicator to a first value to indicate that at least one first parameter of the first category is provided in the information message.

5. The apparatus of claim 3 or claim 4, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:in an instance in which the GRI received from the graphics rendering application includes at least one second parameter of the second category, setting a second GRI indicator to the first value to indicate that at least one second parameter of the second category is provided in the information message.

6. The apparatus of any of claims 3 to 5, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:in an instance in which the GRI received from the graphics rendering application includes no updated parameters, setting the plurality of GRI indicators to a second value to indicate that the information message does not comprise any parameters of any categories.

7. An apparatus comprising:at least one processor; andat least one memory comprising instructions stored therein that, when executed by the at least one processor, cause the apparatus to perform at least:receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures,wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, andwherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data.

8. The apparatus of claim 7, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:decoding the bitstream to form decoded video data; andperforming image analysis on the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters.

9. The apparatus of claim 7, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:decoding the bitstream to form decoded video data; andgenerating, from the decoded video data, based at least upon the at least one syntax element parsed from the information message that represents the one or more parameters, adjusted video data through any of transcoding, editing, or reprojection.

10. The apparatus of claim 7, wherein the information message comprises a plurality of GRI indicators configured to indicate whether the information message comprises at least one first parameter of a first category and at least one second parameter of a second category.

11. The apparatus of claim 10, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:in an instance in which a first GRI indicator in the information message is set to a first value, determining that at least one first parameter of the first category is provided in the information message.

12. The apparatus of claim 10 or claim 11 , wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:in an instance in which a second GRI indicator in the information message is set to the first value, determining that at least one second parameter of the second category is provided in the information message.

13. The apparatus of any of claims 10 to 12, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:in an instance in the first GRI indicator and the second GRI indicator in the information message are both set to a second value, determining that the information message does not comprise any first parameters of the first category or any second parameters of the second category.

14. The apparatus of any of claims 7 to 13, wherein the source video data comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters, and wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements.

15. The apparatus of any of claims 7 to 14, wherein the one or more GRI indicators comprise characteristics of a graphics engine used to generate the graphics data, the characteristics comprising one or more of: a vertical axis identification, a coordinate system handedness, or a projection format.

16. The apparatus of any of claims 7 to 15, wherein the one or more GRI indicators comprise a world to camera matrix or a projection matrix, and wherein at least one of the world to camera matrix or the projection matrix comprises multiple coefficients.

17. The apparatus of claim 16, wherein the one or more GRI indicators comprise a length of matrix coefficient, which specifies a length of syntax elements signaled in the information message for respective coefficients of the world to camera matrix or the projection matrix.

18. The apparatus of any of claims 7 to 17, wherein the one or more parameters are grouped into two or more categories, wherein a respective category of the two or more categories comprises at least one parameter of the one or more parameters comprised in the GRI.

19. The apparatus of claim 18, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:including, in the information message, respective category presence indicators for respective categories of the two or more categories, wherein the respective category presence indicators indicate whether at least one parameter for a respective category is included in the information message.

20. The apparatus of claim 19, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:in an instance in which the respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for a respective category, adding, to the information message, a syntax element representing the at least one parameter of the respective category.

21. The apparatus of claim 18, wherein the instructions stored in the at least one memory, when executed by the at least one processor, further cause the apparatus to perform:determining, based at least upon a syntax element value of respective category presence indicators in the information message for respective categories of the two or more categories, whether at least one parameter for a respective category is included in the information message; andin an instance in which the syntax element value of a respective category presence indicator for a respective category indicates the presence in the information message of at least one parameter for arespective category, parsing, from the information message, a syntax element representing the at least one parameter of the respective category.

22. The apparatus of any of claims 7 to 21 , wherein at least one category presence indicator of the respective category indicators in the information message is a syntax flag that indicates the presence of at least one parameter of the respective category when the syntax flag has a value of 1.

23. The apparatus of any of claims 7 to 22, wherein the information message is provided or received in a first coded picture in the bitstream.

24. The apparatus of claim 23, wherein a first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are not present in the first information message, one or more syntax elements are not included in the first information message for parameters of the first category, and the first category presence indicator for the first category in the first information message is configured to cause the use of default values of the first category when processing the first coded picture.

25. The apparatus of claim 23, wherein the first category presence indicator for a first category in the first information message indicates that at least one parameter of the first category are present in the first information message, and wherein one or more syntax elements are included in the first information message for the at least one parameter of the first category.

26. The apparatus of any of claims 23 to 25, wherein a second information message is contained within a second coded picture in the bitstream, wherein a second category presence indicator for the first category in the second information message indicates that at least one parameter of the first category is not present in the second information message, wherein one or more syntax elements are not included for the at least one parameter of the first category in the second information message, and wherein the second category presence indicator for the first category in the second information message is configured to cause the use of the at least one parameters of the first category received in the first information message when processing the second coded picture.

27. The apparatus of any of claims 7 to 26, wherein the bitstream comprises a depth map, wherein the one or more GRI indicators comprise a depth parameters GRI indicator for one or more depth parameters.

28. The apparatus of claim 27, wherein the one or more depth parameters comprise one or more of: near plane elements, far plane elements, or depth inverse flag syntax elements.

29. A method comprising:receiving, from a graphics rendering application, source video data comprising rendered graphics data;receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data;encoding pictures of the source video data into coded pictures;generating an information message comprising one or more GRI indicators configured to indicate the one or more parameters; andincluding the coded pictures and the information message in a bitstream.

30. A method comprising:receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures,wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, andwherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data.

31. A computer program product comprising at least one non-transitory computer-readable storage medium, the at least one non-transitory computer-readable storage medium comprising instructions stored therein that, when executed by at least one processor of an apparatus, cause the apparatus to perform at least:receiving, from a graphics rendering application, source video data comprising rendered graphics data;receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data;encoding pictures of the source video data into coded pictures;generating an information message comprising one or more GRI indicators configured to indicate the one or more parameters; andincluding the coded pictures and the information message in a bitstream.

32. A computer program product comprising at least one non-transitory computer-readable storage medium, the at least one non-transitory computer-readable storage medium comprising instructions stored therein that, when executed by at least one processor of an apparatus, cause the apparatus to perform at least:receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures,wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, andwherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data.

33. An apparatus comprising:means for receiving, from a graphics rendering application, source video data comprising rendered graphics data;means for receiving, from the graphics rendering application, graphics rendering information (GRI) comprising one or more parameters used by the graphics rendering application to generate the rendered graphics data;means for encoding pictures of the source video data into coded pictures;means for generating an information message comprising one or more GRI indicators configured to indicate the one or more parameters; andmeans for including the coded pictures and the information message in a bitstream.

34. An apparatus comprising:means for receiving, in a bitstream, an information message comprising one or more graphics rendering information (GRI) indicators and coded pictures,wherein the one or more GRI indicators are configured to indicate one or more parameters used by a graphics rendering application to generate rendered graphics data associated with the coded pictures, andwherein the coded pictures comprise encoded pictures of source video data comprising the rendered graphics data.

Images