ICC profile metadata for video streams

Abstract

An exemplary method of video decoding comprises receiving a video bitstream containing a picture set, said video bitstream corresponding to a source device. The method also comprises identifying color profile metadata for said source device based on supplementary enhancement information (SEI) messages for said video bitstream, and reconstructing said picture set using information from said video bitstream. The method further comprises using color characteristics from said color profile metadata to present said picture set at an output device.

Description

Technology Field

[0001] preference

[0002] This application is a continuation of U.S. Patent Application No. 19 / 061,900 filed on February 24, 2025, claiming priority to U.S. Provisional Patent Application No. 63 / 558,070, titled “ICC Profile Metadata for Video Streams,” filed on February 26, 2024, the entirety of which is incorporated by reference herein.

[0003] Technology field

[0004] The disclosed embodiments relate to video coding, including but not limited to systems and methods that generally incorporate color profile metadata into a video bitstream and use color profile metadata for video reconstruction. Background Technology

[0005] Digital video is supported by various electronic devices such as digital televisions, laptops or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video teleconferencing devices, and video streaming devices. Electronic devices transmit and receive digital video data over communication networks or communicate in other ways, and / or store digital video data on storage devices. Due to the limited bandwidth capacity of communication networks and the limited memory resources of storage devices, video coding may be used to compress video data according to one or more video coding standards before the video data is transmitted or stored. Video coding may be performed by hardware and / or software on electronic / client devices or servers that provide cloud services.

[0006] Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that take advantage of the inherent redundancy in video data. Video coding aims to compress video data into a form that uses lower bit rates while avoiding or minimizing degradation in video quality. Numerous video codec standards have been developed. For example, High-Efficiency Video Coding (HEVC / H.265) is a video compression standard designed as part of the MPEG-H project. The ITU-T and ISO / IEC released the HEVC / H.265 standard in 2013 (Version 1), 2014 (Version 2), 2015 (Version 3), and 2016 (Version 4). Versatile Video Coding (VVC / H.266) is a video compression standard intended as a successor to HEVC. The ITU-T and ISO / IEC released the VVC / H.266 standards in 2020 (Version 1) and 2022 (Version 2). AOMedia Video 1 (AV1) is an open video coding format designed as an alternative to HEVC. On January 8, 2019, Version 1.0.0 was released, verified by Errata 1 of the specification. The Enhanced Compression Model (ECM) is a video coding standard currently under development. ECM aims to significantly improve compression efficiency beyond existing standards such as HEVC / H.265 and VVC, thereby enabling higher quality video at inherently lower bitrates.

[0007] Another technique used in video coding standards is Supplemental Enhancement Information (SEI) messages, which enable the carrying of information within the coded bitstream that is supplementary to the coded video. This SEI information may or may not be directly related to the video coding process, that is, as specified by the video standard. In some cases, the information within the SEI messages is related to application processes that run in parallel with, or closely following, the video decoding process. These applications may include rendering processes that use specific SEI messages to adjust the brightness or color space of decoded video frames prior to presentation by a display device. Another such application process arranges portions of the decoded video into specific patterns defined by SEI messages, for example, for 360-degree video displayed on a head-mounted device. Generally, multiple applications may be supported by the information provided in the SEI messages.

[0008] For the H.264, AVC, H.265, and HEVC standards, SEI messages are specified in the main coding specifications, namely "Advanced Video Coding" for H.266 and "High Efficiency Video Coding" for VVC. For example, SEI messages strictly intended for application use only may be specified in a separate specification titled "Versatile supplemental enhancement information messages for coded video bitstreams (VSEI)," whereas SEI messages that can affect the decoding process may be specified in the main coding specification, "Versatile video coding." Additionally, the International Color Consortium develops profiles to assist in color management between different input and output devices. means of solving the problem

[0009] The present disclosure describes, in particular, providing color profile metadata (e.g., International Color Association (ICC) profile metadata) in SEI messages. ICC profile specifications may be those published by ISO, for example, as ISO 15076. For example, an input device (e.g., a digital camera) may use specific settings that describe the color characteristics used to capture raw red, green, and blue image samples in images captured by the input device. These settings may be described in the ICC color profile used by the input device. The input device may store its ICC profile as metadata in the image captured by the camera. Similarly, an output device (e.g., a color printer) may have its own color characteristics, which are also described in the ICC profile settings. Through the availability of the input device's ICC color profile stored in the metadata of the captured image, the output device may harmonize the color values ​​used between the two devices by using the input device's ICC profile and the output device's ICC profile (e.g., so that color fidelity in the image is maintained when the captured image is printed on a color printer). In a similar manner, a display device (another output device) can have its own ICC profile describing the color characteristics of the display. If an ICC color profile exists for the camera used to capture the image, the display can manage RGB colors more effectively when the image is displayed to ensure that the fidelity of the image's appearance is maintained.

[0010] According to some embodiments, a video decoding method comprises: (i) receiving a video bitstream (e.g., a coded video sequence) containing a picture set—the video bitstream corresponds to a source device—; (ii) identifying color profile metadata for the source device based on supplementary enhancement information (SEI) messages for the video bitstream; (iii) reconstructing a picture set using information from the video bitstream; and (iv) presenting the picture set at an output device using color characteristics from the color profile metadata.

[0011] According to some embodiments, a video encoding method comprises: (i) receiving video data (e.g., a source video sequence) including a set of pictures corresponding to a source device; (ii) identifying color profile metadata for the source device; (iii) encoding and signaling the set of pictures in a video bitstream; and (iv) signaling the color profile metadata in a supplementary enhancement information (SEI) message for the video bitstream.

[0012] According to some embodiments, a method for processing visual media data comprises (i) acquiring a source video sequence including a picture set corresponding to a source device; (ii) identifying color profile metadata for the source device; and (iii) encoding the picture set in a video bitstream, wherein the video bitstream includes supplementary enhancement information (SEI) messages representing the encoded picture set and color profile metadata.

[0013] According to some embodiments, a video processing method comprises: (i) setting an image format metadata type identifier to indicate that ICC profile metadata is included in an SEI message associated with the current image; (ii) signaling the image format metadata type identifier in a bitstream; and (iii) encoding the current image in a bitstream based on the image format metadata type identifier.

[0014] According to some embodiments, a computing system such as a streaming system, a server system, a personal computer system, or other electronic device is provided. The computing system includes a control circuit and a memory storing one or more instruction sets. One or more instruction sets include instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder). According to some embodiments, a non-transient computer-readable storage medium is provided. The non-transient computer-readable storage medium stores one or more instruction sets for execution by the computing system. One or more instruction sets include instructions for performing any of the methods described herein.

[0015] Accordingly, devices and systems having methods for encoding and decoding video are disclosed. These methods, devices, and systems may complement or replace conventional methods, devices, and systems for video encoding / decoding. The features and benefits described herein are not necessarily all-encompassing, and in particular, a person skilled in the art will be aware of some additional features and benefits by considering the drawings, specification, and claims provided in this disclosure. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and educational purposes and is not necessarily chosen to explain or limit the subject matter described herein. Brief explanation of the drawing

[0016] To enable a more detailed understanding of the present disclosure, a more specific description may be made by referring to the features of various embodiments, some of which are illustrated in the accompanying drawings. However, the accompanying drawings merely illustrate relevant features of the present disclosure and, therefore, should not be considered limiting, as such description may allow for other effective features that a person skilled in the art would understand when reading the present disclosure. FIG. 1 is a block diagram illustrating an exemplary communication system according to some embodiments. FIG. 2a is a block diagram illustrating exemplary elements of an encoder component according to some embodiments. FIG. 2b is a block diagram illustrating exemplary elements of a decoder component according to some embodiments. FIG. 3 is a block diagram illustrating an exemplary server system according to some embodiments. FIG. 4a illustrates exemplary NAL units and SEI headers according to some embodiments. FIG. 4b illustrates an exemplary generative AI post-filtering process according to some embodiments. FIG. 4c illustrates an exemplary capture system that embeds ICC profile metadata within an image according to some embodiments. FIG. 4d illustrates an exemplary carrying of ICC profile metadata within the payload of an SEI message according to some embodiments. FIG. 4e illustrates an exemplary reference of ICC profile metadata via a Uniform Resource Identifier (URI) within the payload of an SEI message according to some embodiments. FIG. 4f illustrates exemplary syntax of an ICC profile metadata SEI message according to some embodiments. FIG. 5a illustrates an exemplary video decoding process according to some embodiments. FIG. 5b illustrates an exemplary video encoding process according to some embodiments. In accordance with general practice, the various features illustrated in the drawings are not necessarily drawn to scale, and similar reference numbers may be used throughout the specification and drawings to denote similar features. Specific details for implementing the invention

[0017] The present disclosure describes video / image compression techniques comprising signaling color profile metadata (e.g., ICC profile metadata) associated with a source device in SEI messages for a video bitstream. Including color profile metadata for a source device allows the color characteristics of the source device to be taken into account at the output device, which can improve the fidelity / appearance of the video data.

[0018] Exemplary systems and devices

[0019] FIG. 1 is a block diagram illustrating a communication system (100) according to some embodiments. The communication system (100) includes a source device (102) and a plurality of electronic devices (120) (e.g., electronic device (120-1) to electronic device (120-m)) that are coupled to communicate with each other through one or more networks. In some embodiments, the communication system (100) is a streaming system for use with video-enabled applications, such as, for example, video conferencing applications, digital TV applications, and media storage and / or distribution applications.

[0020] The source device (102) includes a video source (104) (e.g., a camera component or media storage) and an encoder component (106). In some embodiments, the video source (104) is a digital camera (e.g., configured to generate an uncompressed video sample stream). The encoder component (106) generates one or more encoded video bitstreams from the video stream. The video stream from the video source (104) may have a higher data volume compared to the encoded video bitstream (108) generated by the encoder component (106). Because the encoded video bitstream (108) has a lower data volume (less data) compared to the video stream from the video source, the encoded video bitstream (108) requires less bandwidth for transmission and less storage space for storage compared to the video stream from the video source (104). In some embodiments, the source device (102) does not include an encoder component (106) (e.g., configured to transmit uncompressed video to network(s) (110)).

[0021] One or more networks (110) represent any number of networks that transmit information between a source device (102), a server system (112), and / or electronic devices (120), including, for example, wired (wired) and / or wireless communication networks. One or more networks (110) may exchange data on circuit-switched and / or packet-switched channels. Representative networks include telecommunication networks, local area networks, wide area networks, and / or the Internet.

[0022] One or more networks (110) include a server system (112) (e.g., a distributed / cloud computing system). In some embodiments, the server system (112) is or includes a streaming server (e.g., configured to store and / or distribute video content, such as an encoded video stream from a source device (102)). The server system (112) includes a coder component (114) (e.g., configured to encode and / or decode video data). In some embodiments, the coder component (114) includes an encoder component and / or a decoder component. In various embodiments, the coder component (114) is instantiated as hardware, software, or a combination thereof. In some embodiments, the coder component (114) is configured to decode an encoded video bitstream (108) and re-encode the video data using a different encoding standard and / or methodology to produce encoded video data (116). In some embodiments, the server system (112) is configured to generate multiple video formats and / or encodings from the encoded video bitstream (108). In some embodiments, the server system (112) functions as a Media-Aware Network Element (MANE). For example, the server system (112) may be configured to prune the encoded video bitstream (108) to potentially customize different bitstreams for one or more of the electronic devices (120). In some embodiments, the MANE is provided separately from the server system (112).

[0023] The electronic device (120-1) includes a decoder component (122) and a display (124). In some embodiments, the decoder component (122) is configured to decode encoded video data (116) to generate an outgoing video stream that can be rendered on a display or other type of rendering device. In some embodiments, one or more of the electronic devices (120) do not include a display component (e.g., are communicably coupled to an external display device and / or include media storage). In some embodiments, the electronic devices (120) are streaming clients. In some embodiments, the electronic devices (120) are configured to access a server system (112) to obtain the encoded video data (116).

[0024] The source device and / or a plurality of electronic devices (120) are sometimes referred to as "terminal devices" or "user devices." In some embodiments, one or more of the source device (102) and / or electronic devices (120) are instances of a server system, a personal computer, a portable device (e.g., a smartphone, tablet, or laptop), a wearable device, a video conferencing device, and / or other types of electronic devices.

[0025] In an exemplary operation of the communication system (100), the source device (102) transmits an encoded video bitstream (108) to the server system (112). For example, the source device (102) may encode a stream of pictures captured by the source device. The server system (112) receives the encoded video bitstream (108) and may decode and / or encode the encoded video bitstream (108) using a coder component (114). For example, the server system (112) may apply an encoding for video data that is more optimal for network transmission and / or storage. The server system (112) may transmit the encoded video data (116) (e.g., one or more encoded video bitstreams) to one or more of the electronic devices (120). Each electronic device (120) may decode the encoded video data (116) and optionally display the video pictures.

[0026] FIG. 2a is a block diagram illustrating exemplary elements of an encoder component (106) according to some embodiments. The encoder component (106) receives video data (e.g., source video sequence) from a video source (104). In some embodiments, the encoder component includes a receiver (e.g., transceiver) component configured to receive the source video sequence. In some embodiments, the encoder component (106) receives a video sequence from a remote video source (e.g., a video source that is a component of a device different from the encoder component (106)). The video source (104) may provide the source video sequence in the form of a digital video sample stream which may have any suitable bit depth (e.g., 8-bit, 10-bit, or 12-bit), any color space (e.g., BT.601 Y CrCB, or RGB), and any suitable sampling structure (e.g., Y CrCb 4:2:0 or Y CrCb 4:4:4). In some embodiments, the video source (104) is a storage device that stores previously captured / prepared video. In some embodiments, the video source (104) is a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that convey motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, wherein each pixel may contain one or more samples depending on the sampling structure, color space, etc., being used. A person skilled in the art can easily understand the relationship between the pixels and the samples.

[0027] The encoder component (106) is configured to code and / or compress pictures of a source video sequence into a coded video sequence (216) in real time or under other time constraints as required by the application. In some embodiments, the encoder component (106) is configured to perform a conversion between the source video sequence and a bitstream of visual media data (e.g., a video bitstream). Enforcing an appropriate coding rate is one function of the controller (204). In some embodiments, the controller (204) controls other function units as described below and is functionally coupled to those other function units. Parameters set by the controller (204) may include rate-control-related parameters (e.g., picture skips, quantizers, and / or lambda values ​​of rate-distortion optimization techniques), picture size, picture group (GOP) layout, maximum motion vector search range, etc. A person skilled in the art in the relevant technical field can easily identify other functions of the controller (204) because they may be related to encoder components (106) that are optimized for a specific system design.

[0028] In some embodiments, the encoder component (106) is configured to operate in a coding loop. In a simplified example, the coding loop includes a source coder (202) (responsible for generating symbols, such as a symbol stream, based on the input picture to be coded and reference picture(s), e.g.,) and a (local) decoder (210). The decoder (210) reconstructs the symbols to generate sample data in a manner similar to the (remote) decoder (when the compression between the symbols and the coded video bitstream is lossless). The reconstructed sample stream (sample data) is input into the reference picture memory (208). Because the decoding of the symbol stream yields bit-exact results independently of the decoder location (local or remote), the content within the reference picture memory (208) is also bit-exact between the local encoder and the remote encoder. In this way, the prediction part of the encoder interprets the same sample values ​​that the decoder will interpret as reference picture samples when using the prediction during decoding.

[0029] The operation of the decoder (210) may be the same as the operation of a remote decoder such as the decoder component (122) described in detail below in relation to FIG. 2b. However, with brief reference to FIG. 2b, since symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (214) and the parser (254) may be lossless, the entropy decoding portions of the decoder component (122), including the buffer memory (252) and the parser (254), may not be fully implemented in the local decoder (210).

[0030] Except for parsing / entropy decoding, the decoder techniques described herein may exist in substantially the same functional form in the corresponding encoder. For this reason, the subject matter disclosed focuses on decoder operations. Additionally, descriptions of encoder techniques may be abbreviated because they may be the inverses of decoder techniques.

[0031] As part of the operation, the source coder (202) may perform motion-compensated predictive coding, which predictively codes the input frame by referencing one or more previously coded frames from a video sequence designated as reference frames. In this way, the coding engine (212) codes the differences between pixel blocks of the input frame and pixel blocks of the reference frame(s) that can be selected as predictive reference(s) for the input frame. The controller (204) may manage the coding operations of the source coder (202), including, for example, the settings of parameters and subgroup parameters used to encode video data.

[0032] The decoder (210) decodes the coded video data of frames that can be designated as reference frames based on symbols generated by the source coder (202). The operations of the coding engine (212) may advantageously be lossy processes. When the coded video data is decoded in a video decoder (not shown in FIG. 2a), the reconstructed video sequence may be a copy of the source video sequence having some errors. The decoder (210) may duplicate the decoding processes that can be performed by a remote video decoder on the reference frames and allow the reconstructed reference frames to be stored in the reference picture memory (208). In this way, the encoder component (106) locally stores copies of the reconstructed reference frames having common content as reconstructed reference frames to be acquired by the remote video decoder (without transmission errors).

[0033] The predictor (206) can perform predictive searches for the coding engine (212). That is, for a new frame to be coded, the predictor (206) can search the reference picture memory (208) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which can serve as appropriate predictive references for the new pictures. The predictor (206) can operate on a sample block-by-pixel block basis to find appropriate predictive references. As determined by the search results obtained by the predictor (206), the input picture may have predictive references derived from a number of reference pictures stored in the reference picture memory (208).

[0034] The outputs of all the aforementioned functional units may undergo entropy coding in an entropy coder (214). The entropy coder (214) converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to a person skilled in the art (e.g., Huffman coding, variable length coding, and / or arithmetic coding).

[0035] In some embodiments, the output of the entropy coder (214) is coupled to a transmitter. The transmitter may be configured to buffer the coded video sequence(s) generated by the entropy coder (214) to prepare them for transmission over a communication channel (218), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter may be configured to merge the coded video data from the source coder (202) with other data to be transmitted, e.g., coded audio data and / or auxiliary data streams (sources not shown). In some embodiments, the transmitter may transmit additional data along with the encoded video. The source coder (202) may include this data as part of the coded video sequence. The additional data may include other forms of redundant data such as time / space / SNR enhancement layers, redundant pictures and slices, Supplementary Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0036] The controller (204) can manage the operation of the encoder component (106). During coding, the controller (204) can assign a specific coded picture type to each coded picture, which can affect the coding techniques applied to each picture. For example, pictures may be assigned as intra-pictures (I-pictures), prediction pictures (P-pictures), or bidirectional prediction pictures (B-pictures). Intra-pictures can be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh (IDR) pictures. A person skilled in the art is familiar with these variations of I-pictures and their respective applications and characteristics, and therefore, they are not described repeatedly here. Prediction pictures can be coded and decoded using inter-prediction or intra-prediction, which use at most one motion vector and reference index to predict the sample values ​​of each block. Bidirectional prediction pictures can be coded and decoded using inter-prediction or intra-prediction, which use up to two motion vectors and reference indices to predict sample values ​​of each block. Similarly, multiple prediction pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0037] Source pictures are generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and can be coded on a block-by-block basis. Blocks can be predictively coded by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the blocks. For example, blocks of I pictures can be non-predictively coded, or they can be predictively coded by referencing already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of P pictures can be non-predictively coded via spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of B pictures can be non-predictively coded via spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0038] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra prediction) uses spatial correlation within a given picture, while inter-picture prediction uses (temporal or other) correlation between pictures. In the example, a specific picture being encoded / decoded, referred to as the current picture, is partitioned into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector referred to as a motion vector. The motion vector points to a reference block within the reference picture and, if multiple reference pictures are in use, may have a third dimension that identifies the reference picture.

[0039] The encoder component (106) may perform coding operations according to a predetermined video coding technique or standard, such as any one described herein. In such operations, the encoder component (106) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancies in the input video sequence. Accordingly, the coded video data may comply with the syntax specified by the video coding technique or standard used.

[0040] FIG. 2b is a block diagram illustrating exemplary elements of a decoder component (122) according to some embodiments. The decoder component (122) of FIG. 2b is coupled to a channel (218) and a display (124). In some embodiments, the decoder component (122) includes a transmitter coupled to a loop filter (256) and configured to transmit data to the display (124) (e.g., via a wired or wireless connection).

[0041] In some embodiments, the decoder component (122) includes a receiver coupled to the channel (218) and configured to receive data from the channel (218) (e.g., via a wired or wireless connection). The receiver may be configured to receive one or more coded video sequences to be decoded by the decoder component (122). In some embodiments, the decoding of each coded video sequence is independent of other coded video sequences. Each coded video sequence may be received from the channel (218), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver may receive the encoded video data along with other data, e.g., coded audio data and / or auxiliary data streams, which may be forwarded to their respective user entities (not described). The receiver may separate the coded video sequence from the other data. In some embodiments, the receiver receives additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence(s). Additional data may be used by a decoder component (122) to decode the data and / or reconstruct the original video data more accurately. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0042] According to some embodiments, the decoder component (122) includes a buffer memory (252), a parser (254) (sometimes referred to as an entropy decoder), a scaler / inverse conversion unit (258), an intra-picture prediction unit (262), a motion compensation prediction unit (260), an aggregator (268), a loop filter unit (256), a reference picture memory (266), and a current picture memory (264). In some embodiments, the decoder component (122) is implemented as an integrated circuit, a series of integrated circuits, and / or other electronic circuits. The decoder component (122) may be implemented at least partially in software.

[0043] A buffer memory (252) is coupled between the channel (218) and the parser (254) (e.g., to prevent network jitter). In some embodiments, the buffer memory (252) is separate from the decoder component (122). In some embodiments, a separate buffer memory is provided between the output of the channel (218) and the decoder component (122). In some embodiments, in addition to the buffer memory (252) inside the decoder component (122) (e.g. configured to handle playback timing), a separate buffer memory is provided outside the decoder component (122) (e.g. to prevent network jitter). When receiving data from a storage / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (252) may not be required or may be small. For use on best effort packet networks such as the Internet, a buffer memory (252) may be required, may be relatively large and / or adaptive in size, and may be implemented at least partially in an operating system or similar elements outside the decoder component (122).

[0044] The parser (254) is configured to reconstruct symbols (270) from a coded video sequence. The symbols may include, for example, information used to manage the operation of a decoder component (122) and / or information for controlling a rendering device such as a display (124). The control information for the rendering device(s) may be, for example, in the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not described). The parser (254) parses (entropy-decodes) the coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow principles widely known to a person skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (254) can extract a set of subgroup parameters for at least one of the subgroups of pixels in a video decoder from a coded video sequence based on at least one parameter corresponding to a group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (254) can also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0045] The reconstruction of the symbols (270) may involve a number of different units depending on the type of the coded video picture or its parts (such as inter- and intra-pictures, inter- and intra-blocks) and other factors. Which units are involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser (254). The flow of this subgroup control information between the parser (254) and the number of units below is not described for clarity.

[0046] The decoder component (122) can be conceptually subdivided into a number of functional units, and in some implementations, these units may interact closely with each other and be integrated at least partially. However, for clarity, the conceptual subdivision of functional units is retained in this specification.

[0047] The scaler / inverse transform unit (258) receives quantized transform coefficients as well as control information (such as which transform to use, block size, quantization factor, and / or quantization scaling matrices) as symbol(s) (270) from the parser (254). The scaler / inverse transform unit (258) may output blocks containing sample values ​​that can be input to the aggregator (268). In some cases, the output samples of the scaler / inverse transform unit (258) relate to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed pictures, but can use predictive information from previously reconstructed parts of the current picture. Such predictive information may be provided by the intra-picture prediction unit (262). The intra-picture prediction unit (262) may generate a block of the same size and shape as the block being reconstructed by using surrounding already reconstructed information fetched from the current (partially reconstructed) picture from the current picture memory (264). The aggregator (268) can add, on a sample-by-sample basis, the prediction information generated by the intra-picture prediction unit (262) to the output sample information provided by the scaler / inverse transformation unit (258).

[0048] In other cases, the output samples of the scaler / inverse unit (258) are intercoded and potentially related to a motion-compensated block. In these cases, the motion compensation prediction unit (260) may access the reference picture memory (266) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (270) for the block, these samples may be added by the aggregator (268) to the output of the scaler / inverse unit (258) (referred to here as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (266) where the motion compensation prediction unit (260) fetches the prediction samples may be controlled by motion vectors. Motion vectors may be available to the motion compensation prediction unit (260) in the form of symbols (270) that may have, for example, X, Y, and reference picture components. Motion compensation may also include motion vector prediction mechanisms, for example, interpolation of sample values ​​fetched from reference picture memory (266) when subsample accurate motion vectors are in use.

[0049] Various loop filtering techniques may be applied to the output samples of the aggregator (268) in the loop filter unit (256). Video compression techniques may include in-loop filter techniques that are controlled by parameters included in the coded video bitstream and made available to the loop filter unit (256) as symbols (270) from the parser (254), but may respond not only to meta-information acquired during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, but also to previously reconstructed and loop-filtered sample values. The output of the loop filter unit (256) may be a sample stream that may be output to a render device such as a display (124), and may also be stored in a reference picture memory (266) for use in future inter-picture predictions.

[0050] Certain coded pictures, once reconstructed, can be used as reference pictures for future predictions. When a coded picture is reconstructed and the coded picture is identified as a reference picture (e.g., by a parser (254)), the current reference picture may become part of the reference picture memory (266), and a new current picture memory may be reallocated before starting the reconstruction of the next coded picture.

[0051] The decoder component (122) may perform decoding operations according to a predetermined video compression technology that may be documented in a standard such as any of the standards described in this specification. The coded video sequence may comply with the syntax specified by the video compression technology or standard used in that it adheres to the syntax of the video compression technology or standard as specified in the video compression technology document or standard and specifically in the profile document therein. Additionally, to comply with some video compression technologies or standards, the complexity of the coded video sequence may be within boundaries defined by the levels of the video compression technology or standard. In some cases, the levels limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the levels may, in some cases, be further limited by HRD (Hypothetical Reference Decoder) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0052] FIG. 3 is a block diagram illustrating a server system (112) according to some embodiments. The server system (112) includes a control circuit (302), one or more network interfaces (304), memory (314), a user interface (306), and one or more communication buses (312) for interconnecting these components. In some embodiments, the control circuit (302) includes one or more processors (e.g., CPU, GPU, and / or DPU). In some embodiments, the control circuit includes field programmable gate array(s), hardware accelerators, and / or integrated circuit(s) (e.g., application integrated circuit).

[0053] Network interface(s) (304) may be configured to interface with one or more communication networks (e.g., wireless, wired, and / or optical networks). The communication networks may be local, wide-area, metropolitan, automotive and industrial, real-time, latency-tolerant, etc. Examples of communication networks include local area networks such as Ethernet and wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, automotive and industrial including CANBus, etc. Such communication may be unidirectional, receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus to specific CANbus devices), or bidirectional (e.g., to other computer systems using local or wide-area digital networks). Such communication may include communication to one or more cloud computing networks.

[0054] The user interface (306) includes one or more output devices (308) and / or one or more input devices (310). The input device(s) (310) may include one or more of a keyboard, mouse, trackpad, touch screen, data glove, joystick, microphone, scanner, camera, etc. The output device(s) (308) may include one or more of an audio output device (e.g., speaker), a visual output device (e.g., display or monitor), etc.

[0055] The memory (314) may include high-speed random access memory (e.g., DRAM, SRAM, DDR RAM, and / or other random access solid-state memory devices) and / or non-volatile memory (e.g., one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, and / or other non-volatile solid-state storage devices). The memory (314) optionally includes one or more storage devices located remotely from the control circuit (302). The memory (314), or alternatively, the non-volatile solid-state memory device(s) within the memory (314), includes a non-transient computer-readable storage medium. In some embodiments, the memory (314), or the non-transient computer-readable storage medium of the memory (314), stores the following programs, modules, instructions, and data structures, or a subset or superset thereof:

[0056] ● An operating system (316) including procedures for handling various basic system services and performing hardware-dependent tasks;

[0057] ● A network communication module (318) used to connect the server system (112) to other computing devices through one or more network interfaces (304) (e.g., through wired and / or wireless connections);

[0058] ● A coding module (320) for performing various functions related to encoding and / or decoding data, such as video data. In some embodiments, the coding module (320) is an instance of the coder component (114). The coding module (320) includes, but is not limited to, one or more of the following:

[0059] o A decoding module (322) for performing various functions related to decoding encoded data, such as those previously described in relation to the decoder component (122); and

[0060] o Encoding module (340) for performing various functions related to encoding data, such as those previously described in relation to the encoder component (106); and

[0061] ● A picture memory (352) for use with, for example, a coding module (320), storing pictures and picture data. In some embodiments, the picture memory (352) includes one or more of a reference picture memory (208), a buffer memory (252), a current picture memory (264), and a reference picture memory (266).

[0062] In some embodiments, the decoding module (322) includes a parsing module (324) configured to perform various functions previously described in relation to, for example, a parser (254), a conversion module (326) configured to perform various functions previously described in relation to, for example, a scalar / inverse conversion unit (258), a prediction module (328) configured to perform various functions previously described in relation to, for example, a motion compensation prediction unit (260) and / or an intra-picture prediction unit (262), and a filter module (330) configured to perform various functions previously described in relation to, for example, a loop filter (256).

[0063] In some embodiments, the encoding module (340) includes a code module (342) (e.g., configured to perform various functions previously described in relation to the source coder (202) and / or coding engine (212)) and a prediction module (344) (e.g., configured to perform various functions previously described in relation to the predictor (206). In some embodiments, the decoding module (322) and / or the encoding module (340) include a subset of the modules illustrated in FIG. 3. For example, a shared prediction module is used by both the decoding module (322) and the encoding module (340).

[0064] Each of the modules identified above stored in memory (314) corresponds to a set of instructions for performing the functions described herein. The modules identified above (e.g., sets of instructions) do not need to be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or rearranged in other ways in various embodiments. For example, the coding module (320) may not optionally include separate decoding and encoding modules, but rather use the same set of modules to perform the functions of both. In some embodiments, memory (314) stores a subset of the modules and data structures identified above. In some embodiments, memory (314) stores additional modules and data structures not mentioned above.

[0065] FIG. 3 illustrates a server system (112) according to some embodiments, but FIG. 3 is intended more as a functional description of various features that may exist in one or more server systems rather than as a structural schematic diagram of the embodiments described herein. In practice, items depicted individually may be combined and some items may be separated. For example, some items depicted separately in FIG. 3 may be implemented on single servers, and single items may be implemented by one or more servers. The actual number of servers used to implement the server system (112), and how features are allocated among them, will vary from implementation to implementation and, optionally, depend in part on the amount of data traffic processed by the server system during average usage periods as well as during peak usage periods.

[0066] Exemplary coding techniques

[0067] The coding processes and techniques described below may be performed on the aforementioned devices and systems (e.g., source device (102), server system (112), and / or electronic device (120)). As previously discussed, video codecs generally include various modalities including partitioning, intra / inter prediction, transform coding, quantization, entropy coding, and in-loop filtering. Additionally, a video coder may include, signal, and / or parse additional data for the video bitstream. The additional data may include temporal, spatial, and / or SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplementary enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc. The discussion below concerns this additional data information provided with the video bitstream.

[0068] As previously described, SEI messages can be used to signal color profile metadata in video streams. Depending on the capabilities of video coding standards, the metadata may be stored directly within the payload of the SEI message, or, as an alternative, the SEI message may be generated with information (e.g., a Uniform Resource Identifier (URI)) identifying an ICC profile metadata resource to be obtained from an external source of the video bitstream.

[0069] Many applications employ SEI messages along with associated coded video streams to access metadata critical or essential to their operation. One example is for stereoscopic video applications, where frame-packing SEI messages describe the configuration of left-eye versus right-eye views for individual video frames, packing both views into a single frame. Dolby Vision® is another example application that relies on carrying metadata within SEI messages; it uses the metadata carried in SEI messages to describe the transfer function applied to the input video signal by display systems capable of emitting high dynamic range outputs.

[0070] Emerging applications for Generative Artificial Intelligence (AI) are also an area where SEI messages can facilitate the deployment of new services. In this application space, imagery can be provided to one or more neural network models that generate or "induce" new or similar image-based results, along with corresponding supplementary information. An example of a popular non-image-based Generative AI tool is ChatGPT. Adobe Firefly is an example of an image-based Generative AI tool. However, for Generative AI applications that rely on imagery within coded video streams, it is not practical to define separate SEI messages to handle each of the nuanced inputs that the underlying models may require. An example of metadata that may be useful for Generative AI (and other applications) is information stored in ICC profile specifications, which may also be published by ISO as ISO 15076.

[0071] As previously discussed, compressed video may be augmented by supplementary enhancement information, for example, in the corresponding video bitstream, in the form of SEI messages or VUI. Video coding standards may include specification parts for SEI and VUI. SEI and VUI information may also be specified in standalone specifications that may be referenced by video coding specifications.

[0072] FIG. 4a shows an exemplary layout of a coded video sequence (CVS). The coded video sequence of FIG. 4a is subdivided into network abstraction layer units (NAL units). An example NAL unit (501) includes a NAL unit header (502) that may include one or more reserved bits. In some embodiments, the NAL unit header (502) includes 16 bits, such as forbidden_zero_bit (503) and nuh_reserved_zero_bit (504), which may not be used in the NAL unit and may be 0. The NAL unit header (502) may further include a nuh_layer_id (505). Several bits of the nuh_layer_id (505) (e.g., 3 bits) may indicate the layer to which the NAL unit belongs (e.g., spatial, SNR, or multiview enhancement). The NAL unit header (502) may further include a nuh_unit_type (505). Several bits of the nuh_nal_unit_type (505) (e.g., 5 bits) indicate the type of the NAL unit (501). In some embodiments, 22 NAL unit type values ​​are defined, for example, 6 NAL unit types are reserved, and 4 NAL unit type values ​​are unspecified and may be used by other specifications (e.g., outside of a video codec). Several bits of the NAL unit header (e.g., 3) may indicate the temporal hierarchy to which the NAL unit belongs, for example, nuh_temporal_id_plus1 (506).

[0073] A coded picture may contain one or more VCL NAL units and / or one or more non-VCL NAL units. VCL NAL units may contain coded data that conceptually belongs to the video coding layer, as previously discussed. Non-VCL NAL units may contain data that conceptually does not belong to the video coding layer. Using H.266 as an example, NAL units may be classified into parameter sets, picture headers, NAL markers, prefix / suffix SE NAL units, filler data, and reserved / unspecified NAL unit types.

[0074] Parameter sets contain information that may be required for the decoding process and may be applied to two or more coded pictures. Parameter sets and conceptually similar NAL units may consist of NAL unit types such as DCI_NUT (e.g., corresponding decoding capability information (DCI)), VPS_NUT (e.g., corresponding to the video parameter set (VPS), which, among other things, establishes hierarchical relationships), SPS_NUT (e.g., corresponding to the sequence parameter set (SPS), which, among other things, establishes parameters used and maintained constant across the CVS), PPS_NUT (e.g., corresponding to the picture parameter set (PPS), which, among other things, establishes parameters used and maintained constant within the coded picture), and prefix and suffix adaptation parameter sets (e.g., PREFIX_APS_NUT and SUFFIX_APS_NUT). Parameter sets may contain information required for the decoder to decode VCL NAL units and may therefore be referred to herein as "normative" NAL units. The picture header (PH_NUT) can also be considered a "normative" NAL unit.

[0075] NAL units marking specific locations in a NAL unit stream may include NAL units having NAL unit types AUD_NUT (e.g., corresponding to an access unit delimiter), EOS_NUT (e.g., corresponding to the end of a sequence), and EOB_NUT (e.g., corresponding to the end of a bitstream). These NAL units may be considered non-normative and are also known as informative in that a conforming decoder must be able to receive them in the NAL unit stream but does not require them for its decoding process.

[0076] Prefix and suffix SEI NAL unit types (e.g., PREFIX_SEI_NUT and SUFFIX_SEI_NUT) represent NAL units containing prefix and suffix supplemental enhancement information. For example, in H.266, these NAL units are considered for informational purposes as they are not required for the decoding process.

[0077] The filler data NAL unit type (e.g., FD_NUT) represents filler data, which can be random and can be used to "waste" bits in a NAL unit stream or bitstream. Such filler data may be required for transmission in a specific isochronous transmission environment.

[0078] Referring still to FIG. 4a, a layout of a NAL unit stream of a decoding sequence (510) containing a coded picture (511) containing some of the previously discussed types of NAL units is illustrated. For example, at the beginning of the NAL unit stream, DCI (512), VPS (513), and SPS (514) can establish parameters that a decoder can use to decode the coded pictures of the CVS containing the coded picture (511) of the NAL unit stream.

[0079] The coded picture (511) may contain a prefix-APS (516), a picture header (517), a prefix-SEI (518), one or more VCL NAL units (519), and a suffix-SEI (520) (e.g., in the described order or any other order compliant with the video coding technology or standard in use).

[0080] Prefix and suffix SEI NAL units are used so that for some SEI messages, the content of the message is known before the coding of a given picture begins, while for others, the content is known only after the picture is coded. Allowing specific SEI messages to appear early or late in the NAL unit stream of the coded picture through prefix and suffix SEIs enables the reduction or avoidance of buffering. For example, the sampling time of the picture to be coded in the encoder is known before the picture is coded, and thus the picture timing SEI message can be a prefix SEI message (e.g., prefix-APS (516)). On the other hand, a decoded picture hash SEI message containing the hash of the sample values ​​of the decoded picture, which may be useful for debugging, for example, encoder implementations, is a suffix SEI message (e.g., suffix-SEI (518)) because the encoder cannot calculate the hash for the samples reconstructed before the picture was coded. The locations of the prefix and suffix SEI NAL units may not be limited to their locations in the NAL unit stream. The terms "prefix" and "suffix" imply the coded pictures or NAL units to which the prefix / suffix SEI messages may be related, and details of this applicability may be specified, for example, in the semantic description of a given SEI message.

[0081] FIG. 4a also illustrates a simplified syntax diagram of a NAL unit containing a prefix or suffix SEI message (520). This syntax is a container format for multiple SEI messages that can be carried in a single NAL unit. Like other NAL units, the SEI NAL unit (520) begins with a NAL unit header (521). One or more SEI messages follow the header (521). Two SEI messages, 530-1 and 530-2, are depicted in FIG. 4a. Each SEI message (530) within the SEI NAL unit (520) includes a payload_type_byte (522) (e.g., 8 bits) (e.g., one of 256 different SEI types) that can specify which of the plurality of SEI types applies to the SEI message, a payload_size_byte (523) (e.g., 8 bits) that specifies the number of bytes of the SEI payload, and corresponding payloads (524). This structure may be repeated until a payload_type_byte identical to 0xff, which indicates the end of the NAL unit, is observed. The syntax of the payload (524) may vary depending on the SEI message (e.g., may have a length between 0 and 255 bytes).

[0082] FIG. 4b is a functional block diagram illustrating an exemplary encoding (106) and decoding (122) system employing an exemplary generative AI post-filtering process (609). The system may include a video source (104), e.g., a digital camera, which generates a source video sequence, e.g., input to an encoder (106). FIG. 4b also illustrates a separate source (601) of complementary metadata (602) for the source (104). In some embodiments, the complementary metadata (602) may originate from the video source (104) itself (e.g., as many digital cameras generate complementary metadata in parallel with capturing source images). FIG. 4b illustrates the encoder (106) receiving both the complementary metadata and the source video sequence. The complementary metadata (602) may be obtained from the separate source (601) by the encoder (106) or directly as an output from the video source (104).

[0083] In FIG. 4b, the output from the encoder (106) is a coded video stream (603) comprising sequences of one or more coded picture data (604) and SEI messages (605)—which may refer to or transmit complementary metadata (602) within the payload of the SEI messages (605). The coded video stream (603) is input to a decoder (122), and the decoder outputs a decoded video stream (606) comprising sequences of reconstructed picture data (607) and payloads (608) of complementary metadata SEI messages. The decoded video stream (606) may be input to a generative AI post-filtering process (609). The output (610) may originate from the generative AI post-filtering process (609).

[0084] FIG. 4c illustrates a capture system that embeds ICC profile metadata within a JPEG image. FIG. 4c illustrates an example with a JPEG image, but similar techniques may be used for other types of images. In the drawing, a digital camera (701) captures a scene and generates a corresponding JPEG image (705). A first part of the JPEG image (705) is represented by a sequence of hexadecimal numbers (and corresponding ASCII interpretation) specified in the drawing, where '0xFFD8' represents the "image start" JPEG marker (703) defined in the JPEG image coding standard. A second part of the JPEG image (705) is represented as a continuation (704) of the JPEG image (705). In this continuation (704), the APP2 marker (706) (defined by the sequence '0xFFE2' in the JPEG standard) indicates the start of ICC profile metadata (708) (or other types of metadata in other embodiments). The end of the ICC profile metadata (707) exemplifies the end of the profile metadata, and its address can be calculated by adding 0228 hexadecimal bytes to the address of the APP2 marker.

[0085] FIG. 4d illustrates ICC profile metadata (708) starting with an APP2 marker (706), packaged within an SEI message (802) that may be specified by a video standard for the purpose of conveying an ICC profile metadata payload (and / or other types of payloads) in a coded video stream generated by an encoder. In some embodiments, the presence of the SEI message (802) is signaled by an SEI NAL unit (801).

[0086] FIG. 4e illustrates an SEI message for carrying ICC profile metadata, wherein the ICC profile metadata (708) is referenced by a URI (901) within the payload of the SEI message (901). In FIG. 4e, the ICC profile metadata exists at or within a location (902) separate from the SEI message.

[0087] FIG. 4f illustrates a system that uses ICC profile metadata embedded within a video sequence (1001) generated by a source (104) in a generative AI post-filtering process (609). The sequence (1001) from the source (104) is input to an encoder (106). The output from the encoder (106) is a coded video stream (603) containing, for example, coded video data (604) and ICC profile metadata SEI messages (1002). The coded video stream (603) is reconstructed by a decoder (122) that outputs a decoded video stream (606). The decoded stream (606) consists of reconstructed picture data (605) and ICC profile metadata payloads (1003). The decoded stream (606) is input to a generative AI process (609) that generates a generative AI process output (610).

[0088] Table 1 below shows exemplary syntax for ICC profile SEI messages.

[0089]

[0090] Table 1 - ICC Profile Metadata SEI Message Syntax Example

[0091] As described above, the cancel flag (icc_profile_cancel_flag) can be used to disable the persistence of a previously processed ICC profile SEI message. For example, if icc_profile_cancel_flag is set to a value indicating 'false', then the ICC profile mode ID (icc_profile_mode_id) may indicate whether the payload of the SEI message is the ICC profile metadata itself or a URI for the location of the ICC profile metadata. For example, if the mode ID is 0, then the ICC profile data payload byte (icc_profile_data_payload_byte) receives bytes of data from the SEI payload. If the mode ID is 1, then the ICC profile data URI (icc_profile_data_URI) receives a string of data from the SEI payload.

[0092] In some embodiments, various types of image format metadata may be signaled via SEI messages. For example, image format types may include interchangeable image file format (EXIF) type, JPEG file exchange format (JFIF) type, extensible metadata platform (XMP) type, ICC type, and / or tagged image file format (TIFF) type. Examples of syntax for SEI messages containing two or more image format types are presented in Table 2.

[0093]

[0094] Table 2 - Exemplary Image Profile Metadata SEI Message Syntax

[0095] As previously discussed, a cancel_flag such as 1 can indicate that the SEI message cancels the persistence of any previous image format metadata SEI message in the output order, and a cancel_flag such as 0 can indicate that image format information follows. The persistence_flag specifies the persistence of the image format metadata SEI message for the current layer. For example, a persistence_flag such as 0 can indicate that the image format metadata SEI message applies only to the current picture, and a persistence_flag such as 1 can indicate that the image format metadata SEI message applies to the current picture and persists to subsequent pictures in the current layer in the output order (e.g., until a new CLVS starts or the bitstream ends). The metadata payloads variable (num_metadata_payloads) indicates the number of metadata payloads following the SEI message. The zero bit flag (bit_equal_to_zero) must always be 0 and functions as an acknowledgment for the SEI message. The URI present flag (uri_present_flag) indicates whether metadata is contained within the payload or obtained via a URI specified in the payload. For example, if the URI present flag is equal to 0, image format information can be obtained directly from the payload of the SEI message, and if the URI present flag is equal to 1, image format information is obtained using a URI. The payload length variable (payload_len_minus1) indicates the length of the image format information payload. In some embodiments, the payload length (e.g., payload size) is limited to be less than (or optionally equal to) a threshold. For example, the payload length may be limited to less than 65,535 bytes. The data payload byte (data_payload_byte) contains the syntax and semantics for the image format type.For example, when an ICC profile type appears, the data payload bytes may be used to indicate the ICC version (e.g., major version and / or minor version). The data URI variable (data_uri) contains a URI (with syntax and semantics as specified in, for example, IETF Internet Standard 66).

[0096] The image format type (type_id) indicates the type of metadata payload for the SEI message. For example, a value of 0 can indicate an EXIF ​​type, values ​​1-3 can indicate a JFIF type, a value of 4 can indicate an XMP type, and a value of 5 can indicate an ICC profile type.

[0097] In some embodiments, the image format (e.g., ICC profile) SEI message is a prefix SEI message. For example, the SEI payload can be specified as shown in Table 3 below.

[0098]

[0099] Table 3 - Exemplary Syntax for SEI Payloads

[0100] FIG. 5a is a flowchart illustrating a method (500) for decoding video according to some embodiments. The method (500) may be performed in a computing system (e.g., a server system (112), a source device (102), or an electronic device (120)) having a control circuit and a memory storing instructions for execution by the control circuit. In some embodiments, the method (500) is performed by executing instructions stored in the memory of the computing system (e.g., memory (314)).

[0101] The system receives a video bitstream (e.g., a coded video sequence) containing a picture set (502), wherein the video bitstream corresponds to a source device (e.g., source device (102)). The system identifies color profile metadata (e.g., image format metadata) for the source device based on an SEI message for the video bitstream (504). The system reconstructs the picture set using information from the video bitstream (506). The system uses color characteristics from the color profile metadata to present the picture set on an output device (508). In some embodiments, the color profile metadata includes ICC metadata (e.g., an ICC profile). In this way, the ICC metadata can be extracted from or referenced by the SEI message.

[0102] FIG. 5b is a flowchart illustrating a method (550) for encoding video according to some embodiments. The method (550) may be performed in a computing system (e.g., a server system (112), a source device (102), or an electronic device (120)) having a control circuit and a memory storing instructions for execution by the control circuit. In some embodiments, the method (550) is performed by executing instructions stored in the memory of the computing system (e.g., memory (314)).

[0103] The system receives video data (e.g., source video sequence) containing a set of pictures corresponding to a source device (e.g., source device (102)) (552). The system identifies color profile metadata for the source device (554). The system encodes and signals the set of pictures into a video bitstream (556). The system signals the color profile metadata in an SEI message for the video bitstream (558). As previously described, the encoding process may mirror the decoding processes described herein (e.g., signaling and parsing additional data for the video bitstream). For brevity, these details are not repeated here.

[0104] FIGS. 5a and 5b illustrate a number of logical stages in a specific order, but stages that are not order-dependent may be rearranged, and other stages may be combined or separated. Since some rearrangements or other groupings not specifically mentioned will be obvious to those skilled in the art, the orderings and groupings presented herein are not comprehensive. Furthermore, it should be recognized that the stages may be implemented in hardware, firmware, software, or any combination thereof.

[0105] Now, refer to the following exemplary embodiments:

[0106] (A1) In one embodiment, some embodiments include a method of video decoding (e.g., method (500)). In some embodiments, the method is performed in a computing system having memory and one or more processors (e.g., a server system (112)). In some embodiments, the method is performed in a coding module (e.g., a coding module (320)). The method includes (i) receiving a video bitstream containing a picture set (e.g., a coded video sequence)—the video bitstream corresponds to a source device—; (ii) identifying color profile metadata (and / or other image format metadata) for the source device based on an SEI message for the video bitstream; (iii) reconstructing a picture set using information from the video bitstream; and (iv) using color characteristics from the color profile metadata to present the picture set at an output device. For example, the SEI message may contain ICC profile metadata. As an example, the SEI message may be included as part of a video bitstream obtained from an image source.

[0107] The advantage of using color metadata (e.g., ICC profiles) is that it provides a mechanism to clearly map the video stream to the output-reference color space, whereas the color information in the video stream is insufficient to perform such mapping. In situations where video is displayed on a computer display, mapping to the output color space can be particularly important, as fidelity may otherwise be degraded.

[0108] (A2) In some embodiments of A1, the method further comprises the step of parsing an image format metadata (IFM) type identifier indicating the type of metadata contained in the SEI message, wherein the color profile metadata is identified when the IFM type identifier indicates that the SEI message contains color profile metadata. For example, IFM types may include metadata of Exif, JFIF, XMP, ICC, TIFF, and / or other types.

[0109] (A3) In some embodiments of A1 or A2, the color profile metadata includes an International Color Association (ICC) profile. For example, the information stored in the International Color Association (ICC) profile specifications may likewise be published by ISO as ISO 15076.

[0110] (A4) In some embodiments of A3, color profile metadata represents an ICC major version number and an ICC minor version number. For example, the first variable (ICCmajorVer) may be set to be equal to data_payload_byte[ i ]

[0008] >> 4 and the second variable (ICCminorVer) may be set to be equal to data_payload_byte[ i ]

[0009] >> 4, and are interpreted as the ICC profile major and minor versions.

[0111] (A5) In some embodiments of any one of A1 to A4, color profile metadata is obtained from the payload of the SEI message. For example, the metadata may be stored directly in the payload of the SEI message according to the capabilities of video coding standards.

[0112] (A6) In some embodiments of any one of A1 to A4, the SEI message includes a Uniform Resource Identifier (URI) string for color profile metadata. For example, the SEI message may be generated with a URI that identifies the exact ICC profile metadata resource to be obtained from a source outside the video bitstream. For example, the ICC profile metadata may exist in or at a location separate from the SEI message.

[0113] (A7) In some embodiments of any one of A1 to A6, the method further comprises applying a filtering process to a picture set using color profile metadata. In some embodiments, the filtering process is a generative artificial intelligence (AI) filtering process, as exemplified in FIG. 4f.

[0114] (A8) In some embodiments of any one of A1 to A7, the video bitstream corresponds to a source video sequence containing image data and color profile metadata. For example, a first part of the image may be represented by a sequence of hexadecimal numbers (e.g., as specified in the JPEG image coding standard), and a second part of the same image may contain profile metadata for the image.

[0115] (A9) In some embodiments of any one of A1 through A8, the method further comprises determining that an SEI message exists based on a syntax element of the Network Abstraction Layer (NAL). For example, the existence of an SEI message may be signaled by an SEI NAL unit. In some embodiments, the SEI message is parsed only when the NAL indicates that the SEI message exists. In some embodiments, a picture set is presented without identifying color profile metadata based on the determination that the NAL indicates that the SEI message does not exist.

[0116] (A9) In some embodiments of any one of A1 to A9, the SEI message includes a first indicator indicating whether a previously processed SEI message of the same type will not persist. For example, a cancellation flag may be used to disable persistence of a previously processed ICC profile SEI message. The first indicator may be referred to as a cancellation flag (e.g., icc_profile_cancel_flag or cancel_flag).

[0117] (A10) In some embodiments of A9, the method further comprises determining whether to obtain color profile metadata from the payload or URI of the SEI message according to the second indicator when the first indicator indicates that the previously processed SEI message will persist. For example, an ICC profile mode ID may indicate whether the payload of the SEI message is the ICC profile metadata itself or a URI for the location of the ICC profile metadata. The second indicator may be referred to as a URI flag (e.g., uri_present_flag or icc_profile_mode_id).

[0118] (B1) In other embodiments, some embodiments include a method of video encoding (e.g., method (550)). In some embodiments, the method is performed in a computing system having memory and one or more processors (e.g., a server system (112)). In some embodiments, the method is performed in a coding module (e.g., a coding module (320)). The method includes (i) receiving video data (e.g., a source video sequence) containing a set of pictures corresponding to a source device; (ii) identifying color profile metadata for the source device; (iii) encoding and signaling the set of pictures in a video bitstream; and (iv) signaling the color profile metadata in an SEI message for the video bitstream.

[0119] (B2) In some embodiments of B1, the color profile metadata includes an International Color Consortium (ICC) profile.

[0120] (B3) In some embodiments of B1 or B2, color profile metadata is signaled in the payload of the SEI message.

[0121] (B4) In some embodiments of any one of B1 to B3, the SEI message includes a uniform resource identifier (URI) string for color profile metadata. For example, the SEI message may include a payload containing some metadata and a URI containing additional metadata.

[0122] (B5) In some embodiments of any one of B1 to B4, the video data includes image data and color profile metadata.

[0123] (B6) In some embodiments of any one of B1 to B5, the method further comprises signaling whether an SEI message exists in the network abstraction layer (NAL) of the video bitstream.

[0124] (B7) In some embodiments of any one of B1 to B6, the method further comprises signaling whether a previously processed SEI message of the same type should not be continued through a first indicator in the SEI message.

[0125] (C1) In other embodiments, some embodiments include a method for processing visual media data. In some embodiments, the method is performed in a computing system having memory and one or more processors (e.g., a server system (112)). In some embodiments, the method is performed in a coding module (e.g., a coding module (320)). The method comprises (i) acquiring a source video sequence comprising a plurality of frames; and (ii) performing a conversion between the source video sequence and a video bitstream of visual media data according to a format rule, wherein the video bitstream comprises a plurality of encoded frames, and the format rule specifies that (a) color profile metadata must be acquired for a picture set using an SEI message for the video bitstream; and (b) the plurality of encoded frames must be reconstructed using color characteristics from the color profile metadata.

[0126] (D1) In other aspects, some embodiments include a video processing method. In some embodiments, the method is performed in a computing system having memory and one or more processors (e.g., a server system (112)). In some embodiments, the method is performed in a coding module (e.g., a coding module (320)). The method includes (i) setting an image format metadata type identifier to indicate that ICC profile metadata is included in an SEI message associated with the current image; (ii) signaling the image format metadata type identifier in a bitstream; and (iii) encoding the current image in a bitstream based on the image format metadata type identifier.

[0127] In another embodiment, some embodiments include a computing system (e.g., a server system (112)) comprising a control circuit (e.g., a control circuit (302)) and a memory coupled to the control circuit (e.g., a memory (314)), wherein the memory stores one or more instruction sets configured to be executed by the control circuit, and the one or more instruction sets include instructions for performing any of the methods described herein (e.g., A1 to A10, B1 to B7, C1, and D1 above). In yet another embodiment, some embodiments include a non-transient computer-readable storage medium storing one or more instruction sets for execution by the control circuit of the computing system, wherein the one or more instruction sets include instructions for performing any one of the methods described herein (e.g., A1 to A10, B1 to B7, C1, and D1 above).

[0128] Unless otherwise specified, any of the syntax elements described herein may be high-level syntax (HLS). As used herein, HLS is signaled at a level higher than the block level. For example, HLS may correspond to the sequence level, frame level, slice level, or tile level. As another example, HLS elements may be signaled in the video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), adaptation parameter set (APS), slice header, picture header, tile header, and / or CTU header.

[0129] It will be understood that while terms such as “first,” “second,” etc., may be used in this specification to describe various elements, these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, singular forms (“a,” “an,” and “the”) are intended to include plural forms unless the context otherwise clearly indicates. It will also be understood that the term “and / or,” as used in this specification, refers to and encompasses any and all possible combinations of one or more of the associated enumerated items. It will be further understood that the terms “comprising” and / or “comprising,” when used in this specification, specify the presence of the mentioned features, integers, steps, actions, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof. When used in this specification,N refers to a variable number. Unless explicitly stated, N Different instances of may refer to the same number (e.g., the same integer value such as the number 2) or different numbers.

[0130] As used in this specification, the term “if” may be interpreted, depending on the context, to mean “when” a mentioned antecedent condition is true, or at the time it is true, or in response to a decision that it is true, or according to a decision that it is true, or in response to a detection that it is true. Similarly, the phrase “when it is determined that the mentioned antecedent condition is true,” or “when the mentioned antecedent condition is true,” or “when the mentioned antecedent condition is true,” may be interpreted, depending on the context, to mean “when the mentioned antecedent condition is determined to be true,” or in response to a decision that it is true, or according to a decision that it is true, or at the time of detection that it is true, or in response to a detection that it is true.

[0131] The foregoing description has been described with reference to specific embodiments for the purposes of explanation. However, the exemplary discussions above are not intended to be comprehensive or to limit the claims to the exact forms disclosed. Many modifications and variations are possible in consideration of the teachings above. The embodiments have been selected and described to best illustrate the principles of operation and actual applications so that other persons skilled in the art can practice them.

Claims

Claim 1 A video decoding method performed on a computing system having memory and one or more processors, the method comprising: receiving a video bitstream including a picture set—the video bitstream corresponds to a source device—; identifying color profile metadata for the source device based on supplementary enhancement information (SEI) messages for the video bitstream; reconstructing the picture set using information from the video bitstream; and presenting the picture set at an output device using color characteristics from the color profile metadata. Claim 2 The method of claim 1 further comprises the step of parsing an image format metadata (IFM) type identifier indicating the type of metadata included in the SEI message, wherein the color profile metadata is identified when the IFM type identifier indicates that the SEI message contains the color profile metadata. Claim 3 A method according to claim 1, wherein the color profile metadata includes an International Color Association (ICC) profile. Claim 4 In paragraph 3, the color profile metadata represents an ICC major version number and an ICC minor version number, a method. Claim 5 A method according to claim 1, wherein the color profile metadata is obtained from the payload of the SEI message. Claim 6 A method according to claim 1, wherein the SEI message includes a Uniform Resource Identifier (URI) string for the color profile metadata. Claim 7 A method according to claim 1, further comprising the step of applying a filtering process to the picture set using the color profile metadata. Claim 8 A method according to claim 1, wherein the video bitstream corresponds to a source video sequence comprising image data and color profile metadata. Claim 9 A method according to claim 1, further comprising the step of determining that the SEI message exists based on the syntax elements of the network abstraction layer (NAL). Claim 10 A method according to claim 1, wherein the SEI message includes a first indicator indicating whether a previously processed SEI message of the same type will not continue. Claim 11 A method according to claim 10, further comprising the step of determining whether to obtain the color profile metadata from the payload or URI of the SEI message according to the second indicator when the first indicator indicates that the previously processed SEI message will continue. Claim 12 A video encoding method performed on a computing system having memory and one or more processors, the method comprising: receiving video data including a picture set corresponding to a source device; identifying color profile metadata for the source device; encoding and signaling the picture set in a video bitstream; and signaling the color profile metadata in a supplementary enhancement information (SEI) message for the video bitstream. Claim 13 In paragraph 12, the method wherein the color profile metadata includes an International Color Association (ICC) profile. Claim 14 In paragraph 12, the color profile metadata is signaled in the payload of the SEI message. Claim 15 A method according to claim 12, wherein the SEI message comprises a Uniform Resource Identifier (URI) string for the color profile metadata. Claim 16 In claim 12, the method comprises the video data including image data and the color profile metadata. Claim 17 A method according to claim 12, further comprising the step of signaling whether the SEI message exists in the network abstraction layer (NAL) of the video bitstream. Claim 18 A method according to claim 12, further comprising the step of signaling whether a previously processed SEI message of the same type will not continue through a first indicator within the SEI message. Claim 19 A non-transient computer-readable storage medium storing a video bitstream generated by a video encoding method, wherein the video encoding method comprises receiving video data including a picture set corresponding to a source device; identifying color profile metadata for the source device; and encoding the picture set; and wherein the video bitstream includes a supplementary enhancement information (SEI) message representing the encoded picture set and the color profile metadata. Claim 20 In paragraph 19, the SEI message is a non-transient computer-readable storage medium containing the color profile metadata or indicating the storage location of the color profile metadata.

Images