Extensible supplemental enhancements for binary metadata of video streams
By introducing a purpose identifier into the SEI message, the problem of SEI message payload limitation is solved, enabling flexible carrying of multiple binary metadata formats in the video encoded stream, supporting artificial intelligence and machine learning applications, and improving the flexibility and scalability of video stream processing.
Patent Information
- Application Number
- CN202580018086.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-04-17
- Filing Date
- 2025-04-18
- Publication Date
- 2026-11-17
AI Technical Summary
In existing video coding standards, the payload size limitation of SEI messages makes it impossible to effectively carry or reference neural network models, thus limiting video streaming capabilities in artificial intelligence and machine learning applications.
Employing an extensible SEI message, which defines a purpose identifier in the SEI message payload, allows the carrying or referencing of various binary metadata formats, including image metadata such as Exif, JFIF, XMP, and ICC profiles, thus enabling flexible carrying and referencing of metadata.
It enables flexible carrying of multiple binary metadata formats in video encoding streams, supports artificial intelligence and machine learning applications, and improves the flexibility and scalability of video stream processing.
Smart Images

Figure CN123125870A_ABST
Abstract
Description
Cross-reference to related applications
[0001] This application claims priority to U.S. Provisional Application No. 63 / 636,491, filed April 19, 2024, and U.S. Application No. 19 / 182,164, filed April 17, 2025, the entire disclosure of which is incorporated herein by reference. Technical Field
[0002] The disclosed topics relate to video encoding and decoding, and in particular to carrying and / or referencing common image metadata formats within encoded video streams for video-based applications. Background Technology
[0003] The technique of using inter-frame prediction with motion compensation for video encoding and decoding has been known for decades. Uncompressed digital video can consist of a series of images, each with a spatial size of, for example, 1920×1080 luminance samples and associated chrominance samples. This series of images can have a fixed or variable image rate (commonly known as the frame rate), for example, 60 images per second, or 60 Hz. Uncompressed video has extremely high bitrate requirements. For example, 1080p60 4:2:0 video with 8 bits per sample (frame rate of 60 Hz, luminance sample resolution of 1920×1080) requires nearly 1.5 Gbit / s of bandwidth. One hour of such video would require more than 600 gigabytes of storage space.
[0004] One purpose of video encoding and decoding is to reduce redundancy in the input video signal through compression. Compression helps reduce bandwidth or storage space requirements, in some cases by two orders of magnitude or more. Lossless compression, lossy compression, and combinations of both can be used. Lossless compression refers to the technique of reconstructing an accurate copy of the original signal from the compressed original signal. When using lossy compression, the reconstructed signal may not be exactly the same as the original signal, but the distortion between the original and reconstructed signals is small enough that the reconstructed signal is suitable for the intended application. Lossy compression is widely used for video. The amount of distortion that is tolerated depends on the specific application; for example, users of some consumer streaming applications may tolerate a higher level of distortion compared to users of television program delivery applications. The achievable compression ratio reflects the higher the allowable / tolerable distortion, the higher the achievable compression ratio.
[0005] Video encoders and decoders can employ a variety of broad categories of techniques, including motion compensation, transform, quantization, entropy coding, and the carrying of supplementary information (e.g., metadata describing the images in the encoded bitstream), some of which will be described below.
[0006] Another technique used in video coding standards is Supplemental Enhancement Information (SEI) messages, which carry supplementary information to the encoded video within the encoded bitstream. This SEI information may or may not be directly related to the video coding process, as specified by video standards (e.g., H.264 | AVC, H.265 | HEVC, and H.266 | VVC). In most cases, the information in SEI messages relates to application processes that execute synchronously with or immediately after the video decoding process. Such applications may include rendering processes that utilize specific SEI messages to adjust the brightness or color space of decoded video frames before rendering them on a display device. Another such application process arranges portions of the decoded video into a specific pattern defined by SEI messages for 360-degree video, for example, a specific pattern displayed on a head-mounted display device. Generally, the information provided in SEI messages can support a wide range of applications.
[0007] Within current standards that utilize SEI messages, such as H.264 | AVC, H.265 | HEVC, and H.266 | VVC, the size of information that can be carried in the payload of an SEI message is limited to no more than 255 bytes. For H.266 | VVC, SEI messages specifically for application use are specified in a separate specification (VSEI) entitled "Versatile supplemental enhancement information messages for coded video bitstreams," while SEI messages that can affect the decoding process are specified in the main coding specification "Versatile Video Coding."
[0008] A recent standardization direction from the ITU-T / ISO / IEC Joint Video Experts Team (JVET) anticipates the use of encoded video streams in applications leveraging artificial intelligence (AI) and machine learning technologies. For this standardization work, version 3.0 of the VSEI specification defines a series of SEI messages for use in such applications. Currently, these SEI messages can carry (or reference via Uniform Resource Identifiers (URIs)) neural networks that will be applied to at least one decoded image in the video stream. However, not all applications will choose to utilize these newly defined SEI messages, as they are specified for referencing or carrying neural network models. Conversely, in some AI applications, it is not necessary for the encoded video stream to carry (or reference) neural networks. Summary of the Invention
[0009] According to one aspect of the invention, a method executed by at least one processor in a decoder includes: receiving a bitstream, the bitstream including visual media data, an SEI message, and a first identifier (ID) included in the payload of the SEI message; extracting metadata or extracting information referencing the metadata from the SEI message based on the first identifier; and decoding the visual media data based on the metadata, wherein the metadata includes binary data, wherein the SEI message references the metadata by using a URI in the payload of the SEI message, and wherein the interpretation of the first identifier is defined outside the payload of the SEI message.
[0010] According to one aspect of the invention, a method executed by at least one processor in an encoder includes: receiving visual media data; generating an SEI message, wherein the payload of the SEI message includes a first identifier and metadata or a reference to metadata; encoding the visual media data according to the metadata; generating a bitstream including the visual media data and the SEI message, wherein the metadata includes binary data, wherein the first identifier indicates whether the payload of the SEI message includes metadata or a reference to metadata, wherein the reference to metadata in the SEI message is performed by using a URI in the payload of the SEI message, and wherein the interpretation of the first identifier is defined outside the payload of the SEI message.
[0011] According to one aspect of the present invention, a method for processing visual media data includes: processing a bitstream of visual media data according to a format rule, wherein the bitstream includes an SEI message, the payload of the SEI message including a first identifier and metadata or a reference to metadata, wherein the metadata includes binary data, wherein the first identifier indicates whether the payload of the SEI message includes metadata or a reference to metadata, wherein the reference to metadata in the SEI message is performed by using a URI in the payload of the SEI message, and wherein the interpretation of the first identifier is defined outside the payload of the SEI message. Attached Figure Description
[0012] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:
[0013] Figure 1 This is a simplified block diagram of a communication system according to one embodiment.
[0014] Figure 2 This is a simplified block diagram of a communication system according to one embodiment.
[0015] Figure 3 This is a simplified block diagram of a decoder according to one embodiment.
[0016] Figure 4 This is a simplified block diagram of an encoder according to one embodiment.
[0017] Figure 5 This is a schematic diagram of a Network Abstraction Layer (NAL) unit and an SEI header according to one embodiment.
[0018] Figure 6 This is a schematic diagram of a simplified encoding / decoding system that employs a simplified Neural Network Post Filtering (NNPF) process, where the neural network model is carried by or referenced from SEI messages.
[0019] Figure 7 This is a schematic diagram of the generative AI post-filtering process when the neural network model is not carried or referenced in the SEI message.
[0020] Figure 8 This is a schematic diagram of an acquisition system that embeds image metadata within a JPEG image.
[0021] Figure 9 This is a schematic diagram illustrating how image metadata is carried within the payload of an SEI message.
[0022] Figure 10 This is a schematic diagram of JFIF metadata, including JPEG File Interchange Format (JFIF) extended tag segments, carried within a JPEG image.
[0023] Figure 11 This is a diagram illustrating how the metadata of an Exchangeable Image File (Exif) is referenced via a URI within the payload of an SEI message.
[0024] Figure 12 This is a schematic diagram of the simple generative AI post-filtering process for capturing image metadata SEI messages.
[0025] Figure 13 This is a schematic diagram of the Exif metadata SEI message syntax according to one embodiment.
[0026] Figure 14 This is a schematic diagram of the JFIF metadata SEI message syntax according to one embodiment.
[0027] Figure 15 This is a schematic diagram of the metadata SEI message syntax of the Extensible Metadata Platform (XMP) according to one embodiment.
[0028] Figure 16 This is a schematic diagram of the SEI message syntax of the International Color Consortium (ICC) profile metadata according to one embodiment.
[0029] Figure 17 It is a schematic diagram of the syntax of a single SEI message that carries binary metadata for EXIF, JFIF, XMP, or ICC profile formats.
[0030] Figure 18 It is a schematic diagram of the syntax of a single SEI message that carries binary metadata for EXIF, JFIF, XMP, or IFI profile formats.
[0031] Figure 19 This is a diagram of a computer system suitable for implementing embodiments of the present invention. Detailed Implementation
[0032] The following detailed description of exemplary embodiments is with reference to the accompanying drawings. The same reference numerals appearing in different drawings may identify the same or similar elements.
[0033] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive, nor is it intended to limit the implementation to the exact forms disclosed. Modifications and variations can be made based on the foregoing disclosure, and these modifications and variations can also be obtained from the practice of the implementation. Furthermore, at least one feature or component of one embodiment may be incorporated into or combined with another embodiment (or with at least one feature of another embodiment). Additionally, in the operation flowcharts and descriptions provided below, it should be understood that at least one operation may be omitted, at least one operation may be added, at least one operation may be performed simultaneously (at least partially), or the order of at least one operation may be reversed.
[0034] Obviously, the systems and / or methods described in this application can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, this application describes the operation and behavior of the systems and / or methods without mentioning specific software code—it should be understood that software and hardware can be designed to implement these systems and / or methods based on the description in this application.
[0035] Although specific combinations of features are listed in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible embodiments. In fact, many of these features can be combined in ways not expressly listed in the claims and / or not expressly disclosed in the specification. While each dependent claim listed below may depend directly on only one claim, the disclosure of possible embodiments includes combinations of each dependent claim with all other claims in the claim set.
[0036] Any element, action, or instruction used in this application shall not be construed as critical or necessary unless expressly stated otherwise. Furthermore, the articles “a” and “an” used in this application are intended to include at least one item and are interchangeable with “at least one.” If only one item is referred to, the term “a” or similar language is used. Similarly, the terms “have,” “possess,” “include,” “comprise,” “include,” etc., used in this application are open-ended terms. Furthermore, unless expressly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.” Additionally, expressions such as “at least one of [A] and [B]” or “at least one of [A] or [B]” should be understood to include only A, only B, or both A and B.
[0037] In this specification, references to "an embodiment," "an embodiment," or similar language mean that a particular feature, structure, or characteristic associated with the referred embodiment is included in at least one embodiment of the solution. Therefore, the phrases "in an embodiment," "in one embodiment," and similar language appearing in this specification may, but not necessarily all, refer to the same embodiment.
[0038] Furthermore, the features, advantages, and characteristics described in this invention can be combined in any suitable manner in at least one embodiment. Those skilled in the art will recognize from the description of this application that the invention can be practiced without employing at least one specific feature or advantage of a particular embodiment. In other instances, additional features and advantages may be found in certain embodiments, and these features and advantages may not necessarily be present in all embodiments of the invention.
[0039] According to embodiments of the present invention, a single scalable SEI message is disclosed for carrying multiple binary metadata formats within an encoded video stream. That is, this application does not specify a unique SEI message for each binary metadata format, but rather discloses a single scalable SEI message for binary metadata, wherein a "purpose identifier" indicates the type or purpose of the binary metadata in the SEI payload. Examples of binary metadata formats include commonly used image metadata formats, such as Exif metadata, JFIF, XMP, and ICC profiles. The metadata can be carried in the payload of the SEI message itself, or alternatively, the SEI message can be created using a URI that identifies the exact metadata resource to be obtained from an external source of the video stream. In video coding specifications, in addition to the syntax of the SEI message, a table is defined that allows for precise definition of each unique value of the purpose identifier.
[0040] In at least one example, a single SEI message can be a generic SEI message providing text information for various purposes, thus eliminating the need to define multiple SEI messages providing specific types of text information, while also providing future scalability. The same technique can be employed for image metadata formats. Therefore, a single SEI message is provided to carry binary metadata for image metadata formats SEIs, achieving similar benefits by eliminating the need to define multiple SEI messages providing specific image metadata information.
[0041] Each embodiment includes at least three separate SEI messages for carrying binary information associated with metadata for EXIF, JFIF, and XMP. These SEI messages are collectively referred to as “Image Format Metadata SEI Messages”. The main syntactic element in the payload of these SEI messages is a payload byte with a descriptor b(8), which is read from the video stream to collect the binary payload of each SEI. A single SEI message for carrying the binary metadata associated with the Image Format Metadata SEI message can benefit from the same principles used to create a single SEI employing a text string-based syntax. Such SEI messages will utilize a common syntax for binary payload bytes and may optionally carry the payload via a URI.
[0042] In at least one example, it is proposed to use a single SEI message to carry individual image format metadata SEI messages. For example, the "Type" syntax element can be defined to indicate, via signaling, which metadata format is carried in the SEI message payload. Furthermore, the option to reference the image metadata at a location determined by the URI can be reserved for this single SEI, making the image metadata payload accessible from the location provided by the URI.
[0043] Figure 1 A simplified block diagram of a communication system (100) according to an embodiment of the present invention is shown. The system (100) may include at least two terminals (110-120) interconnected via a network (150). For unidirectional data transmission, the first terminal (110) may encode video data locally for transmission to the other terminal (120) via the network (150). The second terminal (120) may receive the encoded video data from the other terminal from the network (150), decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in scenarios such as media service applications.
[0044] Figure 1 A second pair of terminals (130, 140) is shown, which are used to support, for example, bidirectional transmission of encoded video that may occur during video conferencing. For bidirectional data transmission, each terminal (130, 140) can encode locally acquired video data for transmission to the other terminal via the network (150). Each terminal (130, 140) can also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.
[0045] exist Figure 1In this invention, terminals (110-140) may be represented as servers, personal computers, and smartphones, but the principles of the invention are not limited thereto. Embodiments of the invention are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. A network (150) represents any number of networks that can transmit encoded video data between terminals (110-140), including, for example, wired and / or wireless communication networks. Communication networks (150) can exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, unless otherwise stated below, the architecture and topology of the network (150) may be irrelevant to the implementation of the invention. The network (150) may include Media Aware Network Elements (MANEs, 160), which may be included in, for example, the transmission path between terminals (130) and (140). MANEs can be used to selectively forward portions of media data to address network congestion, media switching, media mixing, archiving, and similar tasks typically performed by service providers rather than end users. Such MANEs are capable of parsing and reacting to limited portions of media transmitted over a network, such as syntax elements related to video coding technologies or standard NAL syntax.
[0046] Figure 2 As an application example of the disclosed subject matter, the arrangement of a video encoder and decoder in a streaming media environment is shown. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, and storing compressed video on digital media including CDs, DVDs, memory sticks, etc.
[0047] The streaming media system may include an acquisition subsystem (213) that may include a video source (201), such as a digital camera, for creating, for example, an uncompressed sample video stream (202). This sample stream (202) is depicted as a thick line to emphasize its high data volume compared to an encoded video stream, and may be processed by an encoder (203) coupled to the camera (201). The encoder (203) may include hardware, software, or a combination of hardware and software to enable or implement aspects of the disclosed subject matter, which are described in more detail below. The encoded video stream (204) is depicted as a thin line to emphasize its lower data volume compared to the sample stream (202), and may be stored on a streaming media server (205) for future use. At least one streaming media client (206, 208) may access the streaming media server (205) to retrieve a copy (207, 209) of the encoded video stream (204). The client (206) may include a video decoder (210) that decodes an input copy (207) of the encoded video stream and creates an output video sample stream (211) that can be displayed on a monitor (212) or other display device (not depicted). In some streaming media systems, the video stream (204, 207, 209) may be encoded according to certain video encoding / compression standards. For example, those standards include ITU-T Recommendation H.265 and ITU-T Recommendation H.266. The disclosed topics can be used in the context of VVC.
[0048] Figure 3 This can be a functional block diagram of a video decoder (210) according to an embodiment of the present invention.
[0049] The receiver (310) can receive at least one encoded video sequence to be decoded by the decoder (210); in the same embodiment or another embodiment, one encoded video sequence (CVS) is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequences can be received from a channel (312), which can be a hardware / software link connected to a storage device storing the encoded video data. The receiver (310) can receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams that can be forwarded to their respective user entities (not depicted). The receiver (310) can separate the encoded video sequences from other data. To combat / mitigate network jitter, a buffer memory (315) can be coupled between the receiver (310) and the entropy decoder / parser (320) (hereinafter referred to as the "parser"). When the receiver (310) receives data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network, a buffer (315) may not be necessary or may be small. When used on a best-effort packet network such as the Internet, a buffer (315) may be required, which may be relatively large and advantageously have an adaptive size.
[0050] The video decoder (210) may include a parser (320) for reconstructing symbols (321) from the entropy-coded video sequence. The categories of those symbols include information for managing the operation of the decoder (210) and may include information for controlling a display device, such as a display (212), which is not part of the decoder but may be coupled to it. Figure 2As shown. Control information for one or more display devices may be in the form of SEI messages or fragments of Video Usability Information (VUI) parameter sets (not depicted). The parser (320) may parse / entropy decode the received encoded video sequence. Encoding of the encoded video sequence may be performed according to video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (320) may extract a subgroup parameter set from the encoded video sequence for at least one pixel subgroup in the video decoder based on at least one parameter corresponding to a group. Subgroups may include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.
[0051] The parser (320) can perform entropy decoding / parsing operations on the video sequence received from the buffer (315) in order to create symbols (321).
[0052] Depending on the type of the encoded video image or its portions (e.g., inter-frame and intra-frame images, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (321) may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by the parser (320). For clarity, the flow of such subgroup control information between the parser (320) and the multiple units described below is not depicted.
[0053] In addition to the functional blocks already mentioned, decoder 210 can be conceptually subdivided into several functional units as described below. In practical implementations under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the functional units described below.
[0054] The first unit is the scaler / inverse transform unit (351). The scaler / inverse transform unit (351) receives from the parser (320) quantization transform coefficients as one or more symbols (321) and control information, including the transform to be used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit can output blocks containing sample values, which can be input into the aggregator (355).
[0055] In some cases, the output samples of the scaler / inverse transform (351) may involve intra-coded blocks that do not use predictive information from previously reconstructed images, but instead use predictive information from previously reconstructed portions of the current image. This predictive information can be provided by the intra-prediction unit (352). In some cases, the intra-prediction unit (352) uses surrounding reconstructed information obtained from the current (partially reconstructed) image (356) to generate blocks of the same size and shape as the blocks in the reconstruction. In some cases, the aggregator (355) adds the predictive information already generated by the intra-prediction unit (352) to the output sample information provided by the scaler / inverse transform unit (351) on a per-sample basis.
[0056] In other cases, the output samples of the scaler / inverse transform unit (351) may relate to inter-frame coded and possibly motion-compensated blocks. In this case, the motion-compensated prediction unit (353) can access the reference image memory (357) to obtain samples for prediction. After motion compensation of the obtained samples according to the block-related symbols (321), these samples can be added to the output of the scaler / inverse transform unit (referred to in this case as residual samples or residual signals) by the aggregator (355) to generate output sample information. The address at which the motion compensation unit obtains the predicted samples in the reference image memory can be controlled by motion vectors, which are provided to the motion compensation unit in the form of symbols (321), which may have, for example, X components, Y components, and reference image components. Motion compensation may also include interpolation of sample values obtained from the reference image memory when using subsampled precise motion vectors, motion vector prediction mechanisms, etc.
[0057] The output samples of the aggregator (355) can be processed in the loop filter unit (356) using various loop filtering techniques. The video compression techniques may include loop filtering techniques controlled by parameters included in the encoded video bitstream, which are provided to the loop filter unit (356) in the form of symbols (321) by the parser (320). The loop filtering techniques may also respond to metadata obtained during decoding of a previous portion of the encoded image or encoded video sequence (in the decoding order), and to previously reconstructed and loop-filtered sample values.
[0058] The output of the loop filter unit (358) can be a sample stream, which can be output to the display device (212) and stored in the reference image memory (358) for future inter-frame prediction.
[0059] Some encoded images can be used as reference images for future predictions after full reconstruction. Once an encoded image has been fully reconstructed and identified as a reference image (e.g., by the parser (320)), the current reference image (358) can become part of the reference image buffer (357), and new current image memory can be reallocated before the reconstruction of subsequent encoded images begins.
[0060] The video decoder 320 can perform decoding operations according to a predetermined video compression technology, which may be described in standards such as ITU-T Recommendation H.266. The encoded video sequence may conform to the syntax specified by the video compression technology or standard used; that is, the encoded video sequence follows the syntax of the video compression technology or standard, such as the syntax specified in the video compression technology document or standard, particularly in its configuration file document. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the hierarchy of the video compression technology or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further restricted by the Hypothetical Reference Decoder (HRD) specification and the HRD buffer management metadata indicated by signaling in the encoded video sequence.
[0061] In one embodiment, the receiver (310) may receive supplemental (redundant) data along with the encoded video. This supplemental data may be part of one or more encoded video sequences. The supplemental data may be used by the video decoder (320) to properly decode the data and / or more accurately reconstruct the original video data. The supplemental data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant images, forward error correction codes, etc.
[0062] Figure 4 This can be a functional block diagram of a video encoder (203) according to an embodiment of the present invention.
[0063] The encoder (203) can receive video samples from a video source (201) (a part of the non-encoder), which can capture one or more video images to be encoded by the encoder (203).
[0064] The video source (201) can provide a sequence of source video samples in the form of a digital video sample stream to be encoded by the encoder (203). This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (201) can be a storage device storing previously prepared video. In a video conferencing system, the video source (201) can be a camera that captures local image information as a video sequence. The video data can be provided as multiple individual images that, when viewed sequentially, produce an animated effect. The images themselves can be organized as spatial pixel arrays, where each pixel can include at least one sample, depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. Samples are described in detail below.
[0065] According to one embodiment, the encoder (203) can encode and compress images of a source video sequence into an encoded video sequence (443) in real time or as needed by the application, subject to any other time constraints. Enforcing an appropriate encoding rate is a function of the controller (450). The controller controls and is functionally coupled to other functional units described below. For clarity, the coupling is not depicted. Parameters set by the controller may include rate control-related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, GOP layout, maximum motion vector search range, etc. Other functions of the controller (450) will be readily identifiable to those skilled in the art, as these functions may involve a video encoder (203) optimized for a particular system design.
[0066] Some video encoders operate in a mode readily recognizable to those skilled in the art as an "encoding loop." As a highly simplified description, an encoding loop can consist of the encoding portion of an encoder (430) (hereinafter referred to as the "source encoder") (responsible for creating symbols based on the input image to be encoded and the reference image) and a (local) decoder (433) embedded in the encoder (203), which reconstructs these symbols to create sample data that a (remote) decoder would also create (since any compression between the symbols and the encoded video stream is lossless in the video compression techniques considered in the disclosed subject matter). This reconstructed sample stream is input to a reference image memory (434). Since decoding the symbol stream produces a bit-by-bit accurate result unaffected by the decoder's location (local or remote), the contents of the reference image buffer are also bit-by-bit accurate between the local and remote encoders. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same as the sample values "seen" by the decoder during decoding when using prediction. This fundamental principle of reference image synchronicity (and the drift that occurs, for example, due to channel errors, when synchronicity cannot be maintained) is well known to those skilled in the art.
[0067] The operation of the "local" decoder (433) can be the same as that of the "remote" decoder (210), which has been described above. Figure 3 It has been described in detail. However, a brief reference is provided. Figure 3 When symbols are available and the entropy encoder (445) and parser (320) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the decoder (210), which includes the channel (312), receiver (310), buffer (315) and parser (320), may not be fully implemented in the local decoder (433).
[0068] It can be noted that, apart from parsing / entropy decoding, any decoder technique present in the decoder must also exist in the corresponding encoder in essentially the same functional form. Therefore, the subject matter presented focuses on decoder operations. The description of encoder techniques can be simplified, as encoder techniques are the reverse operation of the comprehensively described decoder techniques. More detailed descriptions are only required in certain specific areas, and these descriptions are provided below.
[0069] As part of the operation of the source encoder (430), the source encoder may perform motion-compensated predictive coding, which references at least one previously encoded frame designated as a "reference frame" in the video sequence, to predictively encode the input frame. In this way, the encoding engine (432) encodes the differences between pixel blocks of an input frame and pixel blocks of one or more reference frames, wherein the one or more reference frames may be selected as one or more prediction references for the input frame.
[0070] The local video decoder (433) can decode encoded video data of frames that can be designated as reference frames based on symbols created by the source encoder (430). Advantageously, the operation of the encoding engine (432) can be a lossy process. When the encoded video data can be decoded by the video decoder (430), Figure 4 During decoding at (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (433) replicates the decoding process, which can be performed by the video decoder on the reference frame, and can store the reconstructed reference frame in the reference image buffer (434). In this way, the encoder (203) can store copies of the reconstructed reference frames locally, which have the same content as the reconstructed reference frames that will be obtained by the remote video decoder (no transmission errors).
[0071] The predictor (435) can perform a prediction search against the encoding engine (432). That is, for a new frame to be encoded, the predictor (435) can search the reference image memory (434) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., which can be used as suitable prediction references for the new image. The predictor (435) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (435), the input image may have prediction references obtained from multiple reference images stored in the reference image memory (434).
[0072] The controller (450) can manage the encoding operations of the video encoder (430), including, for example, setting parameters and subgroup parameters for encoding video data.
[0073] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder (445). By losslessly compressing the symbols generated by the various functional units according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, arithmetic coding, etc.), the entropy encoder converts these symbols into an encoded video sequence.
[0074] The transmitter (440) may buffer one or more encoded video sequences created by the entropy encoder (445) in preparation for transmission via a communication channel (460), which may be a hardware / software link connected to a storage device that will store the encoded video data. The transmitter (440) may combine the encoded video data from the video encoder (430) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (source not shown).
[0075] The controller (450) can manage the operation of the encoder (203). During encoding, the controller (450) can assign a specific encoded image type to each encoded image, which may affect the encoding techniques applicable to the corresponding image. For example, one of the following frame types can typically be assigned to an image.
[0076] An intra-frame image (I-image) can be an image that is encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs support different types of intra-frame images, including, for example, images refreshed by a standalone decoder. Those skilled in the art will recognize these variations of I-images and their corresponding applications and characteristics.
[0077] A predictive image (P-image) can be an image that is encoded and decoded by predicting sample values for each block via intra-frame prediction or via inter-frame prediction using at most one motion vector and a reference index.
[0078] A bidirectional predictive image (B-image) can be an image encoded and decoded by predicting sample values for each block via intra-frame prediction or via inter-frame prediction using up to two motion vectors and reference indices. Similarly, a multi-predictive image can reconstruct a single block using more than two reference images and associated metadata.
[0079] The source image can typically be spatially subdivided into multiple sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, which are determined by the coding assignments of the corresponding images applied to these blocks. For example, blocks of an I-image can be unpredictably coded, or these blocks can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be unpredictably coded, coded via spatial prediction, or coded via temporal prediction with reference to a previously coded reference image. Blocks of a B-image can be unpredictably coded, coded via spatial prediction, or coded via temporal prediction with reference to one or two previously coded reference images.
[0080] The video encoder (203) can perform encoding operations according to predetermined video coding techniques or standards, such as ITU-T H.266 Recommendation. In operation, the video encoder (203) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.
[0081] In one embodiment, the transmitter (440) may transmit additional data along with the encoded video. The video encoder (430) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant images and slices, SEI messages, VUI parameter set fragments, etc.
[0082] In a video stream, compressed video can be enhanced using SEI (Search Engine Information), which can take the form of SEI messages or VUI (Video User Interface). Video coding standards can include specification sections for SEI and VUI. SEI and VUI information can also be specified in separate specifications that can be referenced by other video coding standards.
[0083] refer to Figure 5 This illustrates an exemplary layout of an encoded video sequence according to H.266. The encoded video sequence is subdivided into NAL units. An exemplary NAL unit (501) may include a NAL unit header (502), which in turn includes 16 bits: the forbidden_zero_bit (503) and nuh_reserved_zero_bit (504) may not be used in H.266 and can be zero in H.266 compliant NAL units. The three bits of Nuh_layer_id (505) can indicate the layer (spatial, SNR, or multi-view enhancement) to which the NAL unit belongs. The five bits of nuh_nal_unit_type define the type of the NAL unit. In H.266 (04 / 2022), 22 NAL unit type values are defined for the NAL unit types defined in H.266, six NAL unit types are reserved, and four NAL unit type values are unspecified and available for use by specifications outside of H.266. Finally, the three bits of the NAL cell header indicate the time-domain layer nuh_temporal_id_plus1 (506) to which the NAL cell belongs.
[0084] An encoded image may contain at least one Video Coding Layer (VCL) NAL unit and zero or more non-VCL NAL units. A VCL NAL unit may contain encoded data that conceptually belongs to the VCL, as previously described. A non-VCL NAL unit may contain data that conceptually belongs to but is not conceptually part of the VCL. Taking H.266 as an example, NAL units can be classified as follows:
[0085] (1) A parameter set, which includes information that may be necessary for the decoding process and can be applied to more than one encoded image. Parameter sets and conceptually similar NAL units can belong to, for example, the following NAL unit types: DCI_NUT (Decoding Capability Information, DCI), VPS_NUT (Video Parameter Set, VPS, one function of which is to establish inter-layer relationships), SPS_NUT (Sequence Parameter Set, SPS, one function of which is to establish parameters used and unchanged throughout the encoded video sequence), PPS_NUT (Picture Parameter Set, PPS, one function of which is to establish parameters used and unchanged within the encoded image), and PREFIX_APS_NUT and SUFFIX_APS_NUT (prefix and suffix adaptive parameter sets). The parameter set may include information required by the decoder to decode the VCL NAL unit, and is therefore referred to as a "canonical" NAL unit in this application.
[0086] (2) Picture Header (PH) (PH_NUT), which is also a “standard” NAL unit.
[0087] (3) NAL units used to mark certain positions in the NAL unit stream. These NAL units include NAL units of type AUD_NUT (Access Unit (AU) delimiter), EOS_NUT (end of sequence), and EOB_NUT (end of stream). These NAL units are non-canonical, also known as informative, meaning that compliant decoders do not need these NAL units during their decoding process, but the decoder needs to be able to receive these NAL units in the NAL unit stream.
[0088] (4) Prefix and suffix SEI NAL unit types (PREFIX_SEI_NUT and SUFFIX_SEI_NUT), which indicate NAL units containing prefix and suffix SEI. In H.266 (04 / 2022), these NAL units are informational because they are not needed in the decoding process.
[0089] (5) Padding data NAL cell type FD_NUT indicates padding data; this data can be random and can be used to “waste” bits in the NAL cell stream or bit stream, which may be necessary for transmission in some isochronous transmission environments.
[0090] (6) Reserved and unspecified NAL cell types.
[0091] Continue to refer to Figure 5 The diagram illustrates the NAL unit stream layout in decoding order (510), which contains encoded images (511) comprising some of the previously described types of NAL units. At the beginning of the NAL unit stream, DCI (512), VPS (513), and SPS (514) can jointly establish parameters that the decoder can use to decode encoded images of the encoded video sequence, including the encoded images (511) of the NAL unit stream.
[0092] The encoded image (511) may contain, in the order depicted or in any other order conforming to the video coding technology or standard used (here: H.266): prefix APS (516), image header (517), prefix SEI (518), at least one VCL NAL unit (519), and suffix SEI (520).
[0093] Prefix SEI NAL units (518) and suffix SEI NAL units (520) were proposed during the standardization process because, for some SEI messages, the content of the message is known before encoding of a given image begins, while the content of others is only known after the image has been encoded. Making certain SEI messages appear earlier or later in the NAL unit stream of the encoded image as prefix and suffix SEIs avoids buffering. As an example, in an encoder, the sampling time of the image to be encoded is known before encoding, so the image timing SEI message can be a prefix SEI message (516). On the other hand, the decoded image hash SEI message, containing the hash values of the sample values of the decoded image, and used, for example, for debugging encoder implementations, is a suffix SEI message (518) because the encoder cannot compute hash values for the reconstructed samples until image encoding is complete. The positions of the prefix and suffix SEI NAL units are not limited by their positions in the NAL unit stream. The terms “prefix” and “suffix” may imply the encoded image or NAL unit to which the prefix / suffix SEI message applies, and the details of this applicability may be specified, for example, in the semantic description of a given SEI message.
[0094] Continue to refer to Figure 5 A simplified syntax diagram of a NAL unit 520 containing prefixed or suffixed SEI messages is shown. This syntax is a container format for carrying multiple SEI messages within a single NAL unit. For clarity, details of the anti-race syntax specified in H.266 are omitted here. Like other NAL units, the SEI NAL unit begins with a NAL unit header (521). This header is followed by at least one SEI message; two SEI messages (530, 531) are depicted and described below. Each SEI message within the SEI NAL unit includes: an 8-bit payload_type_byte (522) specifying one of 256 different SEI types; an 8-bit payload_size_byte (523) specifying the number of bytes of the SEI payload; and a payload of payload_size_byte bytes (524). This structure can be repeated until a payload_type_byte equals 0xff, indicating the end of the NAL unit. The syntax of the payload (524) depends on the SEI message, and the length of the payload can be any value between 0 and 255 bytes.
[0095] Figure 6This could be a functional block diagram of a simple encoding / decoding system employing the NNPF process (613), where the neural network model is either carried in the payload of an SEI message or referenced in the SEI message via a URI pointing to an external source of the encoded video stream (not depicted). Such an exemplary system could include a video source (201), such as a digital camera, which creates, for example, a source video sequence that is input to the encoder (203). In addition to the source video sequence, the encoder (203) could also receive input from, for example, a separate source (601) containing at least one neural network model that can be used in the post-filtering process (613). The output of the encoder (203) is an encoded video stream (604) comprising at least one sequence of two items: encoded image data (602) and SEI messages (603), which could be referenced in the payload of the SEI message (603) (not depicted) or carry neural network model information. An encoded video stream (604) is input to a decoder (210), which outputs a decoded video stream (607) comprising a sequence of two items: reconstructed image data (605) and a payload (606) of a neural network SEI message. The decoded video stream (604) can be input to an NNPF process (613), in which a neural network filter controller (608) performs any series of steps, including: 1) selecting image data (609) from the data in the decoded video stream (607) and constructing a sequence of at least one neural network filter (611) based on the SEI payload (606), which constitutes a “pipeline” (610) of the neural network filter. The output (612) of the neural network pipeline (610) can also be the output of the NNPF process (613).
[0096] Figure 7 This can be a functional block diagram of a simple encoder-decoder system employing a generative AI post-filtering process (703). Such an exemplary system may include a video source (201), such as a digital camera, for creating, for example, a source video sequence, which is input to the encoder (203). The figure depicts a separate source (701) for supplementary metadata (705). However, an alternative source for the supplementary metadata (705) could be the original video source (201) itself, such as the digital camera, since most digital cameras already create supplementary metadata when capturing the source image. It should be noted that the figure does not depict the supplementary metadata (705) being emitted directly from the video source (201) to the encoder (203). However, Figure 7 The encoder (203) is shown to receive both supplementary metadata and the source video sequence. The supplementary metadata can be obtained by the encoder (203) from an independent source (701), or it can be obtained directly as the output of a video source (201) not shown in the figure.
[0097] The output of the encoder (203) is an encoded video stream (604) comprising at least one sequence of the following two items: encoded image data (602) and SEI messages (702), which may be referenced (not depicted) in the payload of the SEI messages (702) or carry supplementary metadata. The encoded video stream (604) is input to a decoder (210), which outputs a decoded video stream (607) comprising a sequence of the following two items: reconstructed image data (605) and a payload (706) of supplementary metadata SEI messages. The decoded video stream (604) may be input to a generative AI post-filtering process (703). The output (704) may originate from the generative AI post-filtering process (703).
[0098] Figure 8 This can be an illustration of an acquisition system that embeds JFIF and Exif metadata within a JPEG image. In this illustration, a digital camera (801) acquires a scene (802) and can transmit a JPEG image (806). A portion of the JPEG image (805) can be represented by the hexadecimal number sequence shown in the illustration (and its corresponding ASCII interpretation), where '0xFFD8' represents the "start of image" JPEG marker (803) as specified in the JPEG image coding standard (formally titled "Digital compression and coding of continuous-tone still images - Requirements and guidelines").
[0099] exist Figure 8 In this context, the APP0 marker segment (804), defined by the sequence '0xFFE0' in the JPEG standard, marks the beginning of the JFIF metadata, which will be further explained below. JFIF metadata may be specified as per ITU-T recommendations.
[0100] Also in Figure 8In the JPEG standard, the APP1 marker segment (805), defined by the sequence '0xFFE1', marks the beginning of the Exif metadata. Exif metadata is defined by any existing Exif specification jointly developed by the Camera and Imaging Products Association (CIPA) and the Japan Electronics and Information Technology Industries Association (JEITA).
[0101] Figure 8 No exact example of XMP metadata is provided, but when stored in a JPEG image, XMP metadata can also be carried by the APP1 tag segment.
[0102] Figure 9 This is an illustration of carrying binary image metadata in an SEI message. In this illustration, a portion of the Exif metadata (903) starting with the APP1 marker (805) serves as an example of binary metadata that can be encapsulated in an SEI message (902), which, as specified in the video standard, carries the Exif metadata payload in an encoded video stream created by an encoder (not shown). According to the video standard specification, the SEI NAL unit (901) indicates the presence of the SEI message (902) using signaling.
[0103] Figure 10 This is another illustration of JFIF metadata, which includes JFIF extensions according to the ITU-T Recommendation on JFIF. In this illustration, the start portion (1001) of the JFIF metadata can be identified by the hexadecimal value of '0x4A46494600' stored in the APP0 tag segment '0xFFE0', where the ASCII representation (1002) of '0x4A46494600' is "JFIF". Following the start portion (1001) is the JFIF extension (1003), which can be identified by the hexadecimal value of '0x4A46585800' that can be stored in the subsequent APP0 tag segment '0xFFE0'. The ASCII representation (904) of '0x4A46585800' is "JFXX". In this illustration, the extension (1003) carries a "thumbnail" representation (1005) of the original image (805). In this illustration, the thumbnail is also compressed using the encoding scheme specified in ITU-T Recommendation T.81, and is therefore indicated by signaling via the '0xFFD8' image start marker segment.
[0104] Figure 11This is an illustration of an alternative embodiment of an SEI message (1101) for carrying Exif metadata (903), wherein this portion of the Exif metadata (903) can be referenced from within the payload of the SEI message (1101) via a URI (1103). In this alternative embodiment, this portion of the Exif metadata resides at or within a location separate from the SEI message (1101).
[0105] Figure 12 This is an illustration of a system that utilizes binary image metadata embedded in a video sequence (1201) created from a source (201) during a generative AI post-filtering process (703). The sequence (1201) can be input to an encoder (203). The output of the encoder (203) can be an encoded video stream (604), which includes, for example, encoded video data (602) and a binary image metadata SEI message (902). The encoded video stream (604) can be reconstructed by a decoder (210), which can output a decoded video stream (607). The stream (607) can include reconstructed image data (605) and an image metadata payload (1102). The stream (607) can be input to a generative AI process (703), which can then create a generative AI process output (704).
[0106] Figure 13 This can be one embodiment of the syntax of an Exif SEI message. The cancellation flag (1301) can be used to disable the persistence of a previously processed Exif SEI message. If flag (1301) is set to a value indicating 'true', processing of the current Exif SEI message can be completed. If flag (1301) is set to a value indicating 'false', the Exif persistence flag (1302) uses signaling to indicate the scope of persistence of the current Exif SEI message. In the specification of SEI messages, it is common practice for such messages to define both a "cancel" flag and a "persistence" flag, as shown by flags (1301) and (1302) respectively. In this case, the semantics of such flags can be applied consistently throughout the VSEI standard.
[0107] Continue to refer to Figure 13 The Exif pattern identifier (1303) indicates whether the payload of the SEI message is the Exif metadata itself or the URI of the location of the Exif metadata. If the pattern identifier (1303) is zero, the Exif data payload byte (1304) receives one data byte from the SEI payload. If the pattern identifier (1303) is one, the Exif data URI (1305) receives a data string from the SEI payload.
[0108] Figure 14 This can be one embodiment of the syntax of a JFIF SEI message. The cancellation flag (1401) can be used to disable the persistence of a previously processed JFIF SEI message, and the JFIF type identifier (1402) can be signaled to indicate the type of JFIF payload carried in the remainder of the SEI payload. If flag (1401) is set to a value indicating 'true', processing of the current JFIF SEI message can be completed. If flag (1401) is set to a value indicating 'false', the JFIF persistence flag (1403) signals to indicate the extent to which the current JFIF SEI message persists. As mentioned above, in the specification of SEI messages, it is common practice to define both a "cancel" flag and a "persistence" flag, as shown by flags (1401) and (1403) respectively, in which case the semantics of such flags can be applied consistently throughout the VSEI standard.
[0109] Continue to refer to Figure 14 If the type identifier (1402) is equal to zero, the type of the JFIF payload can include bytes in two types of JFIF tag segments, which include: 1) the start portion of the JFIF data indicated by signaling via the string "JFIF" (1002) (in... Figure 10 The following are examples of JFIF data (indicated by 1001) and 2) the subsequent portion of the JFIF data indicated by signaling via the string "JFXX" (1004). Figure 10 (shown as 1003); in this case, the JFIF data payload byte (1404) receives one data byte from the SEI payload. If the type identifier (1402) is equal to the value one, the type of the JFIF payload may include the JFIF tag segment of the JFIF data indicated by signaling via the string "JFXX" (1004). Figure 10 The byte in (1003) is used; in this case, the JFIF extended payload byte (1405) receives a data byte from the SEI payload. If the type identifier (1402) is equal to the value two, the type of the JFIF payload may include a JFIF tag segment of JFIF data indicated by signaling via the string "JFIF" (1002). Figure 10 The byte in (1001) is shown in the middle; in this case, the JFIF header payload byte (1406) receives a data byte from the SEI payload.
[0110] Figure 15This can be one embodiment of the syntax of an XMP SEI message. The cancellation flag (1501) can be used to disable the persistence of a previously processed XMP SEI message. If flag (1501) is set to a value indicating 'true', processing of the current XMP SEI message can be completed. If flag (1501) is set to a value indicating 'false', the XMP persistence flag (1502) uses signaling to indicate the scope of persistence of the current XMP SEI message. In the SEI message specification, it is common practice for such messages to define both a "cancel" flag and a "persistence" flag, as shown by flags (1501) and (1502) respectively. In this case, the semantics of such flags can be applied consistently throughout the VSEI standard.
[0111] Continue to refer to Figure 15 If the flag (1501) is set to a value indicating 'false', the XMP data payload byte (1503) receives a data byte from the SEI payload.
[0112] Figure 16 This can be one embodiment of the syntax for an ICC profile SEI message. The cancellation flag (1601) can be used to disable the continued effectiveness of previously processed ICC profile SEI messages. If flag (1601) is set to a value indicating 'false', the ICC profile pattern identifier (1602) can indicate whether the payload of the SEI message is the ICC profile metadata itself or the URI of the location of the ICC profile metadata. If the pattern identifier (1602) is equal to zero, the ICC profile data payload byte (1603) receives one data byte from the SEI payload. If the pattern identifier (1602) is equal to one, the ICC profile data URI (1604) receives a data string from the SEI payload.
[0113] Figure 17 This is an example of the syntax for a single scalable binary metadata SEI message, which can carry any type of binary payload based on a purpose identifier. In this example, a message from... Figure 13 , Figure 14 , Figure 15 and Figure 16 The payload is in the metadata format shown. The cancellation flag (1701) can be used to disable the continued effectiveness of previously processed messages with the same (1702) purpose identifier. The purpose identifier (1702) can be used with signaling to indicate the type or purpose of the metadata payload carried in the remainder of the SEI payload. The interpretation of the identifier (1702) can be defined in a separate table (1707). Figure 17In the sample table 1707 shown, a purpose identifier equal to zero indicates that the binary payload is EXIF metadata; a purpose identifier equal to one indicates that the binary payload is a JFIF header (1406); a purpose identifier equal to two indicates that the binary payload includes at least one (cascaded) JFIF extension (1405); a purpose identifier equal to three indicates that the binary payload consists of a JFIF header followed by at least one JFXX extension (1404); a purpose identifier equal to four indicates that the binary payload consists of XMP metadata (1503); a purpose identifier equal to five indicates that the binary payload consists of ICC profile metadata (1603); and purpose identifiers equal to six or greater up to 255 (inclusive) are reserved for future use. In this way, Figure 17 The described SEI syntax can be extended to be used for other binary payloads to be determined, such as identifier values of six to 255 (inclusive).
[0114] Continue to refer to Figure 17 If flag (1701) is set to a value indicating 'true', processing of the current binary metadata SEI message can be completed. If flag (1701) is set to a value indicating 'false', the binary persistent flag (1703) signals the scope of the current binary metadata SEI message's persistence. As mentioned above, in the SEI message specification, it is common practice to define both a "cancel" flag and a "persistent" flag, as shown by flags (1701) and (1703) respectively. In this case, the semantics of such flags can be consistently applied throughout the VSEI standard.
[0115] Continue to refer to Figure 17 If the flag (1701) is set to the value 'false', the pattern identifier (1704) indicates the pattern for carrying binary metadata in the SEI payload. If the pattern identifier (1704) is equal to the value zero, the binary metadata is carried directly in the payload of the SEI message itself, and the binary metadata payload byte (1705) receives one data byte from the SEI payload. If the pattern identifier (1704) is equal to one, the binary metadata is stored at a location outside the video stream; this location is determined by the URI carried in the SEI payload. When the pattern identifier (1704) is equal to one, the binary URI (1706) receives a data text string from the SEI message payload.
[0116] Figure 18 It is an embodiment of the syntax of a single scalable binary metadata SEI message, which can carry any type of binary payload depending on the purpose identifier.
[0117] The Image Format Information SEI message (1800) specifies an SEI message in which the payload of EXIF, JFIF, or XMP image format metadata can be carried in the video stream. `ifi_cancel_flag` equal to 1 indicates that the SEI message cancels the persistence of any previous Image Format Information SEI messages in the output order. `ifi_cancel_flag` equal to 0 indicates that the image format information immediately follows. `ifi_persistence_flag` specifies the persistence of the Image Format Information SEI message for the current layer. `ifi_persistence_flag` equal to 0 specifies that the Image Format Information SEI message applies only to the currently decoded image. `ifi_persistence_flag` equal to 1 specifies that the Image Format Information SEI message applies to the currently decoded image and persists for all subsequent images of the current layer in the output order until at least one of the following conditions is true: (i) a new CLVS for the current layer begins; (ii) the stream ends; (iii) an image of the current layer in the AU associated with the Exif metadata SEI message is output, which immediately follows the current image in the output order.
[0118] `ifi_num_metadata_payloads` indicates the number of metadata payloads immediately following it. `ifi_bit_equal_to_zero` should be equal to zero. `ifi_type_id[i]` indicates the type of metadata payload, such as... Figure 18 As defined in Table (1802) shown.
[0119] Human-machine interface input devices may include at least one of the following (only one of each is depicted): keyboard 1901, mouse 1902, touchpad 1903, touch screen 1910, data glove 1904, joystick 1905, microphone 1906, scanner 1907, and camera 1908.
[0120] Computer system 1900 may also include certain human-machine interface output devices. Such human-machine interface output devices can stimulate the senses of at least one human user through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback provided by touchscreen 1910, data gloves 1904, or joystick 1905, but there may also be tactile feedback devices that do not act as input devices), audio output devices (e.g., speakers 1909, headphones (not depicted)), visual output devices (e.g., screens 1910, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality, each with or without tactile feedback functionality—some of which are capable of outputting two-dimensional or more than three-dimensional visual output through stereoscopic output, etc.; virtual reality glasses (not depicted), holographic displays, and smoke boxes (not depicted)), and printers (not depicted).
[0121] The computer system 1900 may also include user-accessible storage devices and their associated media, such as optical media or similar media 1921 including CD / DVD ROM / RW 1920 equipped with CD / DVD, flash memory drives 1922, removable hard disk drives or solid-state drives 1923, conventional magnetic media such as magnetic tapes and floppy disks (not depicted), and dedicated ROM / ASIC / PLD devices such as security dongles (not depicted).
[0122] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.
[0123] Computer system 1900 may also include an interface connected to at least one communication network. The network may be, for example, a wireless network, a wired network, or an optical network. The network may also be a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), an in-vehicle and industrial network, a real-time network, a latency-tolerant network, etc. Examples of networks include: local area networks such as Ethernet and wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and in-vehicle and industrial networks including CAN buses, etc. Some networks typically require external network interface adapters (e.g., the USB port of computer system 1900) attached to certain general-purpose data ports or peripheral buses (1949); other networks are typically integrated into the core of computer system 1900 via a system bus described below (e.g., integrated into a PC computer system via an Ethernet interface, or integrated into a smartphone computer system via a cellular network interface). Using any of these networks, computer system 1900 can communicate with other entities. This communication can be one-way (e.g., broadcasting TV), one-way (e.g., CAN bus to certain CAN bus devices), or it can be bidirectional, such as connecting to other computer systems using a local area or wide area digital network. Certain protocols and protocol stacks can be used on each of the networks and network interfaces described above.
[0124] The aforementioned human-machine interface device, user-accessible storage device, and network interface can be attached to the core 1940 of the computer system 1900.
[0125] The core 1940 may include at least one Central Processing Unit (CPU) 1941, a Graphics Processing Unit (GPU) 1942, a dedicated programmable processing unit in the form of a Field Programmable Gate Array (FPGA) 1943, hardware accelerators 1944 for certain tasks, etc. These devices, along with read-only memory (ROM) 1945, random-access memory (RAM) 1946, and internal high-capacity storage devices 1947 such as internal non-user-accessible hard disk drives (HDDs) and SSDs, can be connected via the system bus 1948. In some computer systems, the system bus 1948 may be accessed via at least one physical connector to allow for expansion through additional CPUs, GPUs, etc. Peripheral devices can be directly attached to the core's system bus 1948 or connected via a peripheral bus 1949. Peripheral bus architectures include PCI, USB, etc.
[0126] CPUs 1941, GPUs 1942, FPGAs 1943, and accelerators 1944 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 1945 or RAM 1946. Temporary data can also be stored in RAM 1946, while permanent data can be stored, for example, in an internal mass storage device 1947. Fast storage and retrieval of any of these storage devices can be achieved using a cache memory, which can be closely associated with at least one CPU 1941, GPU 1942, mass storage device 1947, ROM 1945, RAM 1946, etc.
[0127] Computer-readable media may contain computer code for performing operations of various computer implementations. This medium and computer code may be designed and constructed specifically for the purposes of this invention, or may belong to a class well-known and available to those skilled in the art of computer software. For example, but not as a limitation, a computer system having architecture 1900, particularly core 1940, can provide functionality generated when a processor (including CPUs, GPUs, FPGAs, accelerators, etc.) executes software embodied in at least one tangible computer-readable medium. Such computer-readable media may be media associated with certain storage devices (e.g., internal core mass storage 1947 or ROM 1945) as described above, and with the non-volatile nature of core 1940. Software implementing various embodiments of the invention may be stored in such devices and executed by core 1940. Depending on specific needs, the computer-readable medium may include at least one storage device or chip. The software enables core 1940, particularly its processors (including CPU, GPU, FPGA, etc.), to execute a specific process or a specific portion of a specific process described in this application, including defining data structures stored in RAM 1946 and modifying such data structures according to the process defined by the software. Alternatively, the computer system may provide functionality generated by hard-wired logic or otherwise embodied in circuitry (e.g., accelerator 1944), which may operate in place of or in conjunction with the software to execute the specific process or a specific portion of a specific process described in this application. Where appropriate, references to software may cover logic, and vice versa. Where appropriate, references to computer-readable media may cover circuitry (e.g., integrated circuits) storing software for execution, circuitry embodying logic for execution, or both. This invention covers any suitable combination of hardware and software.
[0128] While several exemplary embodiments have been described in this invention, modifications, substitutions, and various alternative equivalents exist that fall within the scope of this invention. Therefore, it should be understood that those skilled in the art will be able to design numerous systems and methods that, while not explicitly stated or described herein, embody the principles of the invention and are therefore within its spirit and scope.
[0129] The above disclosure also covers the embodiments listed below:
[0130] (1) A method performed by at least one processor in a decoder includes: receiving a bitstream, the bitstream including visual media data, an SEI message, and a first identifier included in the payload of the SEI message; extracting metadata or extracting information referencing the metadata from the SEI message based on the first identifier; and decoding the visual media data based on the metadata, wherein the metadata includes binary data, wherein the reference to the metadata in the SEI message is performed by using a URI in the payload of the SEI message, wherein the interpretation of the first identifier is defined outside the payload of the SEI message.
[0131] (2) According to the method of feature (1), the method further includes: determining the value of a first identifier; extracting metadata from the payload of the SEI message in response to determining that the value of the first identifier is a first value; and referencing the metadata by using a URI included in the payload of the SEI message in response to determining that the value of the first identifier is a second value.
[0132] (3) The method according to feature (1) or (2) further includes: wherein the bitstream further includes a payload size, and wherein, in response to determining that the value of the first identifier is a first value, the amount of metadata extracted from the payload of the SEI message conforms to the payload size.
[0133] (4) The method according to any one of features (1) to (3) further includes: wherein the bitstream includes a second identifier that indicates the type of binary metadata.
[0134] (5) The method according to feature (4) further includes: wherein the second identifier has a first value indicating that the type of the binary metadata is Exif, a second value indicating that the type of the binary metadata is JFXX, and a third value indicating that the type of the binary metadata is XMP.
[0135] (6) According to the method of feature (5), wherein the second identifier has a fourth value indicating that the type of binary metadata is only JFIF header, a fifth value indicating that the type of binary metadata is only JFIF extension, and a sixth value indicating that the type of binary metadata is both JFIF header and JFIF extension.
[0136] (7) The method according to any one of features (1) to (6), wherein the bitstream further includes a cancellation flag, and wherein the first identifier is extracted according to the value of the cancellation flag.
[0137] (8) The method according to any one of features (1) to (7), wherein the SEI message is the ICC profile SEI message.
[0138] (9) A method executed by at least one processor in an encoder, the method comprising: receiving visual media data; generating an SEI message, wherein the payload of the SEI message includes a first identifier and metadata or a reference to metadata; encoding the visual media data according to the metadata; generating a bitstream including the visual media data and the SEI message, wherein the metadata includes binary data, wherein the first identifier indicates whether the payload of the SEI message includes metadata or a reference to metadata, wherein the reference to metadata in the SEI message is performed by using a URI in the payload of the SEI message, and wherein the interpretation of the first identifier is defined outside the payload of the SEI message.
[0139] (10) According to the method of feature (9), the method further includes: wherein the first identifier has a first value indicating that metadata is included in the payload of the SEI message, and wherein the first identifier has a second value indicating that the payload includes a URI referencing the metadata.
[0140] (11) According to the method of feature (9) or (10), wherein the bitstream further includes a payload size, and wherein, in response to the first identifier having a first value, the amount of metadata extracted from the payload of the SEI message conforms to the payload size.
[0141] (12) The method according to any one of features (9) to (11), wherein the bitstream includes a second identifier that indicates the type of binary metadata.
[0142] (13) According to the method of feature (12), wherein the second identifier has a first value indicating that the type of binary metadata is Exif, a second value indicating that the type of binary metadata is JFXX, and a third value indicating that the type of binary metadata is XMP.
[0143] (14) According to the method of feature (13), wherein the second identifier has a fourth value indicating that the type of binary metadata is only JFIF header, a fifth value indicating that the type of binary metadata is only JFIF extension, and a sixth value indicating that the type of binary metadata is both JFIF header and JFIF extension.
[0144] (15) The method according to any one of features (9) to (14), wherein the bitstream further includes a cancellation flag, and wherein the first identifier is extracted according to the value of the cancellation flag.
[0145] (16) The method according to any one of features (9) to (15), wherein the SEI message is an ICC profile SEI message.
[0146] (17) A method for processing visual media data, the method comprising: processing a stream of visual media data according to a format rule, wherein the stream includes an SEI message, the payload of the SEI message including a first identifier and metadata or a reference to metadata, wherein the metadata includes binary data, wherein the first identifier indicates whether the payload of the SEI message includes metadata or a reference to metadata, wherein the reference to metadata in the SEI message is performed by using a URI in the payload of the SEI message, and wherein the interpretation of the first identifier is defined outside the payload of the SEI message.
[0147] (18) According to the method of feature (17), the method further includes: wherein the first identifier has a first value indicating that metadata is included in the payload of the SEI message, and wherein the first identifier has a second value indicating that the payload includes a URI referencing the metadata.
[0148] (19) According to the method of feature (18), wherein the bitstream further includes a payload size, and wherein, in response to the first identifier having a first value, the amount of metadata extracted from the payload of the SEI message conforms to the payload size.
[0149] (20) According to the method of feature (19), the bitstream includes a second identifier that indicates the type of binary metadata.
Claims
1. A method performed by at least one processor in a decoder, the method comprising: The method includes: Receive a bitstream, the bitstream including visual media data, supplementary enhancement information (SEI) messages, and a first identifier included in the payload of the SEI messages; Based on the first identifier, extract metadata or information referencing the metadata from the SEI message; and The visual media data is decoded based on the metadata. The metadata includes binary data. The reference to the metadata in the SEI message is performed by using the Uniform Resource Identifier (URI) in the payload of the SEI message. The interpretation of the first identifier is defined outside the payload of the SEI message.
2. The method of claim 1, wherein, The method further includes: Determine the value of the first identifier; In response to determining that the value of the first identifier is a first value, the metadata is extracted from the payload of the SEI message; and In response to determining that the value of the first identifier is a second value, the metadata is referenced by using the URI included in the payload of the SEI message.
3. The method of claim 1, wherein, The bitstream also includes a payload size, and wherein, in response to determining that the value of the first identifier is the first value, the amount of the metadata extracted from the payload of the SEI message conforms to the payload size.
4. The method of claim 1, wherein, The bitstream includes a second identifier that indicates the type of the binary metadata.
5. The method of claim 4, wherein, The second identifier has a first value indicating that the type of the binary metadata is an exchangeable image file (Exif), a second value indicating that the type of the binary metadata is JFXX, and a third value indicating that the type of the binary metadata is an extreme memory profile (XMP).
6. The method of claim 5, wherein, The second identifier has a fourth value indicating that the type of the binary metadata is only a JPEG File Exchange Format (JFIF) header, a fifth value indicating that the type of the binary metadata is only a JFIF extension, and a sixth value indicating that the type of the binary metadata is both the JFIF header and the JFIF extension.
7. The method of claim 1, wherein, The bitstream also includes a cancellation flag, and wherein the first identifier is extracted based on the value of the cancellation flag.
8. The method of claim 1, wherein, The SEI message is the International Color Consortium (ICC) profile SEI message.
9. A method performed by at least one processor in an encoder, the method comprising: The method includes: Receive visual media data; Generate a Supplemental Enhancement Information (SEI) message, wherein the payload of the SEI message includes a first identifier and metadata or a reference to the metadata; The visual media data is encoded based on the metadata; Generate a bitstream that includes the visual media data and the SEI message. The metadata includes binary data. The first identifier indicates whether the payload of the SEI message includes the metadata or a reference to the metadata. The reference to the metadata in the SEI message is performed by using the Uniform Resource Identifier (URI) in the payload of the SEI message, and The interpretation of the first identifier is defined outside the payload of the SEI message.
10. The method of claim 9, wherein, The method also include: Wherein, the first identifier has a first value, the first value indicating that the metadata is included in the payload of the SEI message, and The first identifier has a second value, which indicates that the payload includes the URI that references the metadata.
11. The method of claim 9, wherein, The bitstream also includes a payload size, and wherein, in response to the first identifier having the first value, the amount of the metadata extracted from the payload of the SEI message conforms to the payload size.
12. The method of claim 9, wherein, The bitstream includes a second identifier that indicates the type of the binary metadata.
13. The method of claim 12, wherein, The second identifier has a first value indicating that the type of the binary metadata is an exchangeable image file (Exif), a second value indicating that the type of the binary metadata is JFXX, and a third value indicating that the type of the binary metadata is an extreme memory profile (XMP).
14. The method of claim 13, wherein, The second identifier has a fourth value indicating that the type of the binary metadata is only a JPEG File Exchange Format (JFIF) header, a fifth value indicating that the type of the binary metadata is only a JFIF extension, and a sixth value indicating that the type of the binary metadata is both the JFIF header and the JFIF extension.
15. The method of claim 9, wherein, The bitstream also includes a cancellation flag, and wherein the first identifier is extracted based on the value of the cancellation flag.
16. The method of claim 9, wherein, The SEI message is the International Color Consortium (ICC) profile SEI message.
17. A method of processing visual media data, characterized by The method includes: Process the bitstream of visual media data according to formatting rules. The bitstream includes Supplemental Enhancement Information (SEI) messages, the payload of which includes a first identifier and metadata or a reference to the metadata. The metadata includes binary data. The first identifier indicates whether the payload of the SEI message includes the metadata or a reference to the metadata. The reference to the metadata in the SEI message is performed by using the Uniform Resource Identifier (URI) in the payload of the SEI message, and The interpretation of the first identifier is defined outside the payload of the SEI message.
18. The method of claim 17, wherein, The method also include: Wherein, the first identifier has a first value, the first value indicating that the metadata is included in the payload of the SEI message, and The first identifier has a second value, which indicates that the payload includes the URI that references the metadata.
19. The method of claim 18, wherein, The bitstream also includes a payload size, and wherein, in response to the first identifier having the first value, the amount of the metadata extracted from the payload of the SEI message conforms to the payload size.
20. The method of claim 19, wherein, The bitstream includes a second identifier that indicates the type of the binary metadata.
