Bitstream merging

Abstract

To provide a video codec for efficiently merging video bitstreams.SOLUTION: A video encoder (2) for providing an encoded video stream (12) including one or more merge identifiers indicates encoded parameter information describing a plurality of encoding parameters (20, 22, 26, 30), encoded video content information, and whether and / or how the encoded video representation (12) can be merged with another encoded video representation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to video encoding / decoding. [Background technology]

[0002] Video compositing is used in numerous applications where a composite of multiple video sources is presented to the user. A common example is picture-in-picture (PiP) compositing and fusion with overlay video content, e.g., for advertising or user interfaces. Generating such a composite in the pixel domain requires parallel decoding of the input video bitstreams, which is computationally complex and may even be infeasible on a single hardware decoder or device with otherwise limited resources. For example, in current IPTV system designs, the corresponding set-top box performs the compositing, which is a major service cost driver due to its complexity, distribution, and limited lifespan. Reducing these cost drivers motivates ongoing efforts to virtualize set-top box functions, e.g., shifting user interface generation to cloud resources. In such approaches, only the video decoder—a so-called zero client—is the only hardware remaining at the customer premises. The state-of-the-art in such system designs is compositing in its simplest form, based on transcoding: decoding, pixel-domain compositing, and re-encoding before or during transport. To reduce the workload from a full decoding and encoding cycle, operations in the transform coefficient domain rather than the pixel domain were first proposed for PiP compositing. Since then, numerous techniques have been proposed to fuse or shorten the individual compositing steps and apply them to current video codecs. However, transcoding-based approaches for general compositing remain computationally complex, impairing the scalability of the system. Depending on the transcoding approach, such compositing can also affect rate distortion (RD) performance.

[0003] Additionally, a wide range of applications exist that build on tiles, where a tile is a spatial subset of a video plane that is coded independently of neighboring tiles. Tile-based streaming systems for 360° video work by splitting 360° video into tiles that are coded into sub-bitstreams of various resolutions and then merged into a single bitstream on the client side depending on the current user's viewing direction. Another application involving sub-bitstream merging is, for example, combining traditional video content with banner ads. Sub-bitstream recombination can also be an important part of video conferencing systems, where multiple users send individual video streams to a receiver where the streams are ultimately merged. Furthermore, tile-based cloud coding systems, in which video is split into tiles and tile coding is distributed to separate, independent instances, rely on fail-safe checks before the resulting tile bitstreams are merged back into a single bitstream. In all of these applications, streams are merged to enable decoding by a single video decoder using a known compatibility point. Merging here refers to lightweight bitstream rewriting that does not require full decoding and encoding of the entropy-coded data or reconstructing pixel values. However, the techniques used to ensure successful bitstream merging, i.e., conformance of the resulting merged bitstream, have stemmed from very different application scenarios.

[0004] For example, legacy codecs support a technique called Motion Constrained Tile Set (MCTS), in which the encoder constrains inter-prediction between pictures to be limited within the boundaries of a tile or picture, i.e., without using sample or syntax element values ​​that do not belong to the same tile or lie outside the picture boundary. This technique has its origins in Region of Interest (ROI) decoding, where the decoder can decode only a specific subsection of the bitstream and coded picture without encountering unresolvable dependencies and avoiding drift. Another state-of-the-art technology in this context is the Structure of Pictures (SOP) SEI message, which gives an indication of the applied bitstream structure, i.e., the picture ordering and inter-prediction reference structure. This information is provided by summarizing, for each picture between two Random Access Points (RAPs), the Picture Order Count (POC) value, the active Sequence Parameter Set (SPS) identifier, and the Reference Picture Set (RPS) index to the active SPS. Based on this information, a transcoder, middlebox, or media-aware network entity (MANE) or media player can identify bitstream structures that can help manipulate or modify the bitstream, such as adjusting the bitrate, skipping frames, or fast-forwarding.

[0005] While both of the above exemplary signaling techniques are essential in understanding whether lightweight merging of sub-bitstreams can be performed without nontrivial syntax modifications or even complete transcoding, they are by no means sufficient. More specifically, lightweight merging in this context is characterized by interleaving NAL units of source bitstreams with only minor rewriting operations, i.e., writing parameter sets used together with new picture sizes and tile structures, so that all bitstreams to be merged are located in separate tile regions. The next level of merging complexity is constituted by minor rewriting of slice header elements, ideally without changing the variable length codes in the slice headers. Further levels of merging complexity exist, such as re-performing entropy coding on slice data to modify certain syntax elements that can be changed without requiring reconstruction of pixel values, which, while entropy coded, can be beneficial compared to a full transcoding involving video decoding and encoding, and can rather be considered lightweight.

[0006] In the merged bitstream, all slices must reference the same parameter set. If the parameter sets of the original bitstreams use significantly different configurations, lightweight merging may not be possible, as many of the parameter set's syntax elements have further influence on the syntax of the slice header and slice payload, to different degrees. The further downstream in the decoding process a syntax element is involved, the more complex the merging / rewriting becomes. Several notable general categories of syntactic dependencies (of parameter sets and other structures) can be distinguished: A. Syntactic Presence Indicators B. Value calculation dependencies C. Slice Payload Encoding Tool Control a. Syntax used early in the decoding process (coefficient sign hiding, block division restrictions, etc.) b. Syntax used later in the decoding process (motion comp, loop filter, etc.) or general decoding process control (reference pictures, bitstream ordering) D. Source Format Parameters

[0007] In category A, the parameter set contains a number of presence flags for various tools (such as dependent_slice_segments_enabled_flag and output_flag_present_flag). If there are differences in such flags, the flags can be set to be valid in the combined parameter set, and default values ​​can be explicitly written into the merged slice headers of slices in the bitstream that did not contain the syntax element before the merge; i.e., in this case, the merge requires changes to the parameter set and slice header syntax.

[0008] In Category B, signaled values ​​of the parameter set syntax may be used together with parameters of other parameter sets or slice headers in the calculation, e.g., in HEVC, the slice quantization parameter (QP) of a slice is used to control the coarseness of quantization of the transform coefficients of the residual signal of the slice. The slice QP signaling in the bitstream (SliceQpY) depends on the QP signaling at the picture parameter set (PPS) level as follows: SliceQpY=26+init_qp_minus26(from PPS)+slice_qp_delta(from slice header)

[0009] Since all slices in the (merged) coded video picture must reference the same activated PPS, any differences in init_qp_minus26 in the PPSs of the individual streams to be merged require slice header adjustments to reflect the new common value of init_qp_minus26, i.e., in this case, merging requires changes to the parameter set and slice header syntax, just as in Category 1.

[0010] In category C, further parameter set syntax elements control coding tools that affect the bitstream structure of the slice payload. Subcategories Ca and Cb can be distinguished according to how far downstream in the decoding process the syntax elements are included and (relatedly) the complexity associated with modifications to these syntax elements, i.e., where the syntax elements are included between entropy coding and pixel-level reconstruction.

[0011] For example, one element in category Ca is the sign_data_hiding_enabled_flag, which controls the derivation of sign data for coded transform coefficients. Sign data hiding can be easily deactivated, and the corresponding inferred sign data is explicitly written to the slice payload. However, such changes to the slice payload in this category do not require going into the pixel domain with full decoding before re-encoding the pixel-by-pixel merged image. Another example is the determination of inferred block partitioning, or any other syntax that allows inferred values ​​to be easily written to the bitstream. That is, in this case, merging requires changes to the parameter set, slice header syntax, and slice payload that require entropy decoding / encoding.

[0012] However, subcategory Cb requires syntax elements related to processes far downstream in the decoding process, and therefore many of the complex decoding processes must be performed in a way that avoiding the remaining decoding steps is implementationally and computationally undesirable. For example, differences in syntax elements connected to various decoding process-related syntax, such as motion compensation constraints, "temporal motion constraint tile set SEI (Structure of Picture Information) messages," or "SEI messages," make pixel-level transcoding unavoidable. In contrast to subcategory Ca, there are many encoder decisions that affect the slice payload in ways that cannot be changed without complete pixel-level transcoding.

[0013] In category D, there are syntax elements in the parameter set (such as chroma subsampling indicated by chroma_format_idc) that necessitate pixel-level transcoding to merge sub-bitstreams with different values, i.e., in this case, merging requires a full decoding, pixel-level merging, and full encoding process.

[0014] While the above list is by no means exhaustive, it becomes clear that a wide range of parameters affect the benefits of merging sub-bitstreams into a common bitstream in different ways, and that tracking and analyzing these parameters can be tedious. Summary of the Invention [Problem to be solved by the invention]

[0015] SUMMARY OF THE INVENTION It is an object of the present invention to provide a video codec for efficiently merging video bitstreams. [Means for solving the problem]

[0016] This object is achieved by the subject matter of the claims of the present application.

[0017] The basic idea underlying the present invention is that improved merging of multiple video streams is achieved by including one or more merge identifiers, an approach that allows accelerating the merging process while reducing the burden on computational resources.

[0018] According to an embodiment of the present invention, a video encoder is configured to provide a video stream including encoded parameter information describing a plurality of encoding parameters, encoded video content information (i.e., information encoded using the encoding parameters defined by the parameter information), and one or more merging identifiers indicating whether and / or how, e.g., using what complexity, an encoded video representation can be merged with another encoded video representation. The complexity to be used may be determined based on the parameter values ​​defined by the parameter information. The merging identifier may be a concatenation of a plurality of encoding parameters that must be equal in the two different encoded video representations to enable them to be merged using a predetermined complexity. Additionally, the merging identifier may be a hash value of a concatenation of a plurality of encoding parameters that must be equal in the two different encoded video representations to enable them to be merged using a predetermined complexity.

[0019] According to embodiments of the present invention, a merging identifier may indicate a merging procedure by rewriting parameter sets, or by rewriting parameter sets and slice headers, or by rewriting parameter sets, slice headers, and slice payloads, or generally, a merging identifier type that indicates the complexity of a "suitable" merging method, where the merging identifier is associated with a merging identifier type, and the merging identifier associated with the merging identifier type includes coding parameters that must be equal in two different coded video representations such that the two different coded video representations are mergeable using the merging procedure complexity indicated by the merging identifier type. The value of the merging identifier type may indicate the merging process, and the video encoder is configured to switch between at least two of the following values ​​of the merging identifier type: a first value of the merging identifier type that indicates a merging process by rewriting parameter sets, a second value of the merging identifier type that indicates a merging process by rewriting parameter sets and slice headers, and a third value of the merging identifier type that indicates a merging process by rewriting parameter sets, slice headers, and slice payloads.

[0020] According to an embodiment of the present invention, multiple merging identifiers are associated with different complexities of the merging procedure, e.g., each identifier indicating a parameter set / hash and type of the merging procedure. The encoder may be configured to check whether the coding parameters evaluated for providing the merging identifier are identical for all units of the video sequence and to provide the merging identifier depending on the check.

[0021] According to an embodiment of the present invention, the plurality of encoding parameters may include merging-related parameters that must be identical in the different video streams, i.e., coded video representations, to enable merging with less complexity than merging by full pixel decoding, and the video encoder is configured to provide one or more merging identifiers based on the merging-related parameters, i.e., if there are no common parameters between the two coded video representations, full pixel decoding is performed, i.e., there is no possibility to reduce the complexity of the merging process. The merge-related parameters include one or more or all of the following parameters: parameters describing motion constraints at tile boundaries, e.g., parameters describing motion constrained tile set supplemental enhancement information (MCTS SEI); group of pictures (GOP) structure, i.e., mapping of coding order to display order; random access point indication, e.g., temporal layering using SOP: Structure of Pictures; information about SEI; parameters describing chroma coding formats and parameters describing luma coding formats, e.g., a set including at least chroma format and bit depth luma / chroma; parameters describing advanced motion vector prediction, parameters describing sample adaptive offset, parameters describing temporal motion vector prediction, and parameters describing loop filters and other coding parameters, i.e., parameter sets (including reference picture sets, reference quantization parameters, etc.), slice headers, slice payloads are rewritten to merge two coded video representations.

[0022] According to embodiments of the present invention, a merging identifier associated with a first complexity of the merging procedure may be determined based on a first encoding parameter set, while a merging parameter associated with a second complexity of the merging procedure higher than the first complexity may be determined based on a second encoding parameter set that is a proper subset of the first encoding parameter set. A merging identifier associated with a third complexity of the merging procedure higher than the second complexity may be determined based on a third encoding parameter set that is a proper subset of the second encoding parameter set. The video encoder is configured to determine a merging identifier associated with a first complexity of the merging procedure based on a first set that must be equal in two different video streams, e.g., coded video representations, to enable merging of video streams that only change the parameter sets applicable to multiple slices while leaving the coding parameter sets, e.g., slice headers and slice payloads, unchanged, i.e., a merging process with (only) parameter sets rewriting. The video encoder may be configured to determine a merge identifier associated with a second complexity of the merge procedure based on a second set that must be equal in two different video streams, e.g., coded video representations, to enable merging of video streams that change coding parameter sets, e.g., parameter sets applicable to multiple slices, while leaving slice payloads unchanged, and also modifying slice headers, i.e., a merging process that involves rewriting parameter sets and slice headers. The video encoder may be configured to determine a merge identifier associated with a third complexity of the merge procedure based on a third set that must be equal in two different video streams, e.g., coded video representations, to enable merging of video streams that change coding parameter sets, e.g., parameter sets applicable to multiple slices, and also modifying slice headers and slice payloads, but without performing full pixel decoding and pixel re-encoding, i.e., a merging process that involves rewriting parameter sets, slice headers, and slice payloads.

[0023] According to an embodiment of the present invention, a video merger for providing a merged video representation based on a plurality of provided coded video representations, e.g., video streams, is configured to receive a plurality of video streams including coded parameter information describing a plurality of coding parameters, coded video content information (i.e., information coded using the coding parameters defined by the parameter information), and one or more merging identifiers indicating whether and / or how, e.g., with what complexity, a coded video representation can be merged with another coded video representation, and the video merger is configured to decide on a merging method usage, e.g., a merge type, regarding the merging process, depending on the merging identifiers, i.e., depending on a comparison of the merging identifiers of the different video streams. The video merger is configured to select a merging method from a plurality of merging methods depending on the merging identifiers. The video merger may be configured to select from at least two of the following merging methods: a first merging method, which is a merging of video streams that only changes the parameter sets applicable to multiple slices while leaving the slice headers and slice payloads unchanged, and which are merging methods of different complexity depending on one or more merge identifiers; a second merging method, which is a merging of video streams that changes the parameter sets applicable to multiple slices while leaving the slice payloads unchanged and also changes the slice headers; and a third merging method, which is a merging of video streams that changes the parameter sets applicable to multiple slices and also changes the slice headers and slice payloads, but does not perform full pixel decoding and pixel re-encoding.

[0024] According to embodiments of the present invention, a video merger is configured to compare merge identifiers of two or more video streams associated with the same given merge method or associated with the same merge identifier type, and to perform a decision on whether to perform merging using the given merge method depending on the result of the comparison. The video merger may be configured to selectively perform merging using the given merge method if the comparison indicates that the merge identifiers of two or more video streams associated with the given merge method are equal. The video merger may be configured to use a merge method having a higher complexity than the given merge method with which the compared merge identifiers are associated if the comparison of the merge identifiers indicates that the merge identifiers of two or more video streams associated with the given merge method are different, i.e., without further comparison of the encoding parameters themselves. The video merger may be configured to selectively compare encoding parameters that must be equal in two or more video streams to enable merging of the video streams using a given merging method when a comparison of the merge identifiers indicates that the merge identifiers of two or more video streams associated with the given merging method are equal, and the video merger is configured to selectively perform merging using a given merging method when a comparison of the encoding parameters, i.e., the encoding parameters that must be equal in two or more video streams to enable merging of the video streams using the given merging method, indicates that the encoding parameters are equal, and the video merger is configured to perform merging using a merging method having a higher complexity than the given merging method when a comparison of the encoding parameters indicates that the encoding parameters include differences.

[0025] According to an embodiment of the present invention, a video merger may be configured to compare merge identifiers associated with merge methods having different complexities, i.e., compare hashes; the video merger is configured to identify a merge method of lowest complexity whose associated merge identifiers are equal in the two or more video streams to be merged; the video merger is configured to compare encoding parameter sets, i.e., individual encoding parameters, rather than hashed versions of the encoding parameters, which must be equal in the two or more video streams to be merged to enable merging using the identified merge method; different, usually overlapping, encoding parameter sets are associated with merge methods of different complexity; the video merger is configured to selectively merge two or more video streams using the identified merge method if the comparison indicates that the encoding parameters of the encoding parameter sets associated with the identified merge method are equal in the video streams to be merged; and the video merger is configured to merge two or more video streams using a merge method having a higher complexity than the identified merge method if the comparison indicates that the encoding parameters of the encoding parameter sets associated with the identified merge method include differences. The video merger is configured to determine which coding parameters should be modified in the merging process, i.e., merging of video streams, depending on, for example, one or more differences between merge identifiers associated with the same merge method or "merge identifier type" of different video streams to be merged.

[0026] According to an embodiment of the present invention, the video merger is configured to obtain joint coding parameters or joint coding parameter sets, e.g., sequence parameter sets SPS and picture parameter sets PPS, associated with slices of all video streams to be merged based on the coding parameters of the video streams to be merged, and include the joint coding parameters in the merged video stream, e.g., if all coding parameter values ​​of one coded video representation and the other coded video representation are the same; if there are differences between the coding parameters of the coded video representations, the coding parameters are updated by copying common parameters; the coding parameters are updated based on the main (i.e., one is main and the other is secondary) coded video representation; e.g., some coding parameters may be adapted according to the combination of the video streams, e.g., the total picture size; and to adapt coding parameters individually associated with each video slice, e.g., defined in a slice header, or when using a merging method with higher complexity than the minimum, to obtain modified slices to be included in the merged video stream. The adapted encoding parameters include parameters representing the image size of the merged encoded video representation, the image size being calculated based on the image sizes of the encoded video representations to be merged, i.e. in each dimension, in the context of their spatial arrangement.

[0027] According to an embodiment of the present invention, a video encoder for providing a provided encoded video representation, i.e., a video stream, the video encoder may provide a coarse granularity capability demand information (CGR) describing the compatibility of the video stream with video decoders having a functionality level among a plurality of predetermined functionality levels, e.g., the decodability of the video stream by the video decoder. The video encoder may be configured to provide fine-grained feature demand information, e.g., merge level limit information, describing what fraction of an allowable feature demand, i.e., decoder capability, associated with one of the predetermined feature levels is necessary to decode the encoded video representation, and / or what fraction of the allowable feature demand, i.e., “merged bitstream level limit,” a sub-bitstream contributes to the encoded video representation, i.e., a merged video stream into which video streams whose feature demands match one of the predetermined feature levels, i.e., whose feature demands are less than or equal to the allowable feature demand, correspond to the “merged bitstream level limit.” The video encoder is configured to provide fine-grained feature demand information, such that the fine-grained feature demand information includes a ratio or percentage value that refers to one of the predetermined feature levels. The video encoder is configured to provide the fine-grained feature demand information, such that the fine-grained feature demand information includes reference information and fraction information, the reference information describing which of the predetermined feature levels the fraction information refers to, such that the fine-grained feature demand information as a whole describes a fraction of one of the predetermined feature levels.

[0028] According to an embodiment of the present invention, a video merger for providing a merged video representation based on a plurality of provided encoded video representations, i.e., video streams, may be configured to receive a plurality of video streams, including: encoded parameter information describing a plurality of encoding parameters; encoded video content information (e.g., information encoded using the encoding parameters defined by the parameter information); coarse-grained feature demand information (e.g., level information, level 3, level 4, or level 5) describing compatibility of the video streams with a video decoder having a feature level among a plurality of predetermined feature levels, i.e., decodability of the video stream by the video decoder; and fine-grained feature demand information (e.g., merge level restriction information), wherein the video merger is configured to merge two or more video streams depending on the coarse-grained feature demand information and the fine-grained feature demand information. The video merger may be configured to determine which video streams can and will be included in the merged video stream without violating allowable feature demands, i.e., such that the feature demands of the merged video streams match one of the predetermined feature levels depending on the fine-grained feature demand information. The video merger may be configured to determine, for example, whether a valid merged video stream can be obtained by merging two or more video streams depending on the fine-resolution feature demand information, such that the valid merged video stream does not violate acceptable feature demands, i.e., the feature demands of the merged video streams match one of the predetermined feature levels. The video merger is configured to summarize the fine-grained feature demand information of the multiple video streams to be merged, for example, to determine which video streams can be included or to determine whether a valid merged video stream can be obtained.

[0029] Preferred embodiments of the present invention will now be described with reference to the drawings. [Brief explanation of the drawings]

[0030] [Figure 1]1 is a block diagram illustrating an apparatus for providing an encoded video representation as an example of a video encoder capable of implementing bitstream merging concepts according to embodiments of the present invention; [Figure 2] FIG. 2 is a schematic diagram illustrating an example of a bitstream structure according to an embodiment of the present invention. [Figure 3] FIG. 10 is a block diagram illustrating an apparatus for providing an encoded video representation as another example of a video encoder capable of implementing the bitstream merging concept according to embodiments of the present invention. [Figure 4-1] FIG. 4 is a schematic diagram illustrating an example of encoding parameters according to an embodiment of the present invention. [Figure 4-2] FIG. 4 is a schematic diagram illustrating an example of encoding parameters according to an embodiment of the present invention. [Figure 4-3] FIG. 4 is a schematic diagram illustrating an example of encoding parameters according to an embodiment of the present invention. [Figure 4-4] FIG. 4 is a schematic diagram illustrating an example of encoding parameters according to an embodiment of the present invention. [Figure 5a] FIG. 5 is a diagram showing an example of a detailed sequence parameter set (SPS) shown in FIG. 4. [Figure 5b] FIG. 5 is a diagram showing an example of a detailed sequence parameter set (SPS) shown in FIG. 4. [Figure 6a] FIG. 5 is a diagram showing an example of a detailed picture parameter set (PPS) shown in FIG. 4. [Figure 6b] FIG. 5 is a diagram showing an example of a detailed picture parameter set (PPS) shown in FIG. 4. [Figure 7a] FIG. 5 is a diagram illustrating an example of a detailed slice header shown in FIG. 4. [Figure 7b] FIG. 5 is a diagram illustrating an example of a detailed slice header shown in FIG. 4. [Figure 7c] FIG. 5 is a diagram illustrating an example of a detailed slice header shown in FIG. 4. [Figure 7d] FIG. 5 is a diagram illustrating an example of a detailed slice header shown in FIG. 4. [Figure 8]FIG. 5 is a diagram showing a detailed structure of the Supplementary Enhancement Information (SEI) message shown in FIG. 4. [Figure 9] FIG. 5 is a diagram showing detailed motion constraint tile sets in the SEI message shown in FIG. 4. [Figure 10] 1 is a block diagram illustrating an apparatus for providing a merged video representation as an example of a video merger capable of implementing the bitstream merging concept according to an embodiment of the present invention; [Figure 11] FIG. 2 is a schematic diagram illustrating a merge complexity determination process according to an embodiment of the present invention. [Figure 12] 1 is a schematic diagram illustrating the bitstream structure of multiple video representations to be merged and the bitstream structure of the merged video representation according to the bitstream merging concept of the present invention; FIG. [Figure 13] FIG. 10 is a block diagram illustrating an apparatus for providing a merged video representation as another example of a video merger that can implement the bitstream merging concept according to embodiments of the present invention. [Figure 14] 1 is a block diagram illustrating an apparatus for providing an encoded video representation as an example of a video encoder for providing functional demand information for a merged video representation according to an embodiment of the present invention; [Figure 15] 1 is a block diagram illustrating an apparatus for providing a merged video representation as an example of a video merger that provides functional demand information for a merged video representation according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0031] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular embodiments, procedures, techniques, etc. Those skilled in the art will understand that other embodiments may be used apart from these specific details. For example, the following description proceeds using a non-limiting example application, but the technology may be applied to any type of video codec. In some instances, detailed descriptions of well-known methods, interfaces, circuits, and devices are omitted so as not to obscure the description with unnecessary detail.

[0032] In the following description, equivalent or similar elements having equivalent or similar functions are designated with equivalent or similar reference signs.

[0033] The invention herein is to provide future video codecs, such as VVC (Versatile Video Coding), with a means to include in each sub-bitstream an indication that allows for the identification of sub-bitstreams that can be merged together into a legal bitstream or that cannot be merged together into a legal bitstream at a given level of complexity. This indication is hereinafter referred to as a "merge identifier," and further provides information about the appropriate merging method via an indication called a "merge identifier type." Assuming two bitstreams have a "merge identifier" and the same value, it is possible to merge the sub-bitstreams into a new combined bitstream at a given level of merge complexity (associated merge method).

[0034] FIG. 1 illustrates a video encoder 2 for providing an encoded video representation, i.e., encoded video stream 12, based on a provided (input) video stream 10, including an encoder core 4 including an encoding parameter determination member 14 and a merge identifier provider 6. Provided video stream 10 and encoded video stream 12 each have a bitstream structure, e.g., as shown in a simplified configuration in FIG. 2. This bitstream structure is composed of a number of Network Abstraction Layer (NAL) units, each of which includes various parameters and / or data, e.g., a sequence parameter set (SPS) 20, a picture parameter set (PPS) 22, an instantaneous decoder refresh (IDR) 24, a supplemental enhancement information (SEI) 26, and a number of slices 28. SEI 26 includes various messages, e.g., a structure of picture, a motion constraint tile set, etc. Slices 28 include a header 30 and a payload 32. The coding parameter determination member 14 determines coding parameters based on the SPS 20, PPS 22, SEI 26 and slice header 30. The IDR 24 is not necessarily a factor in determining coding parameters in accordance with the present invention, but the IDR 24 may optionally be included to determine coding parameters. The merge identifier provision 6 provides a merge identifier that indicates whether and / or how (and with what complexity) a coded video stream can be merged with another coded video stream.The merge identifier determines the complexity of the merge procedure, e.g., merging by rewriting parameter sets, or merging by rewriting parameter sets and slice headers, or merging by rewriting parameter sets, slice headers, slice payloads, or generally, the merge identifier indicates a merge identifier type that represents the associated "appropriate" merging method, the merge identifier being associated with a merge identifier type, and the merge identifier associated with the merge identifier type includes coding parameters that must be equal in two different coded video representations so that the two different coded video representations can be merged using the complexity of the merge procedure represented by the merge identifier type.

[0035] As a result, the encoded video stream 12 includes encoded parameter information describing multiple encoding parameters, encoded video content information, and one or more merge identifiers. In this embodiment, the merge identifiers are determined based on the encoding parameters. However, the merge identifier values ​​are set according to the desires of the encoder operator, and there may be little guarantee of sufficient collision avoidance in a closed system. Third-party entities, such as Digital Video Broadcasting (DVB) and the Advanced Television Systems Committee (ATSC), may also define merge identifier values ​​to be used within their systems.

[0036] The merge identifier types are described below in view of another embodiment according to the invention using FIGS.

[0037] FIG. 3 illustrates an encoder 2 (2a) including an encoder core 4 and a merge identifier 6a, and illustrates the data flow in the encoder 2a. The merge identifier 6a includes a first hash member 16a and a second hash member 16b that provide a hash value as a merge identifier to indicate a merge type. That is, the merge value may be formed from a concatenation of coded values ​​of a defined set of bitstream syntax elements (coding parameters), hereinafter referred to as a hash set. The merge identifier value may also be formed by subjecting the concatenation of coded values ​​to a well-known hash function, such as MD5, SHA-3, or any other suitable function. As shown in FIG. 3, an input video stream 10 includes input video information, and the input video stream 10 is processed by the encoder core 4. The encoder core 4 encodes the input video content, and the encoded video content information is stored in the payload 32. The encoding parameter determination member 14 receives parameter information, including the SPS, PPS, slice header, and SEI message. The parameter information is stored in each corresponding unit and received by the merge identifier provider 6a, i.e., by the first hash member 16a and the second hash member 16b, respectively. The first hash member 16a generates a hash value indicating a merge identifier type 2 based on the encoding parameters, and the second hash member 16b generates a hash value indicating a merge identifier type 1 based on the encoding parameters, for example.

[0038] The contents of the hash set, ie, the syntactic element (ie, merge parameter) values ​​that are concatenated to form the merge identifier value, determine the quality of the mergeability indication for the above syntactic categories.

[0039] For example, the merge identifier type indicates the appropriate merging method for the merge identifier, ie, different levels of merging possibilities corresponding to the syntax elements incorporated in the hash set, as follows: Type 0: Merge identifier for merging with parameter set rewriting Type 1: Merge identifier for merging by rewriting parameter sets and slice headers Type 2: Merge identifier for merging by rewriting parameter sets, slice headers, and slice payloads

[0040] For example, given two input sub-bitstreams to be merged in the context of an application, a device can compare the values ​​of the merge identifier and the merge identifier type and draw a conclusion about the feasibility of using the method associated with the merge identifier type for the two sub-bitstreams.

[0041] The following table shows the mapping between syntax element categories and associated merging methods: [Table 1]

[0042] As already mentioned above, the syntax categories are by no means exhaustive, but it becomes clear that a wide range of parameters affect the merits of merging sub-bitstreams into a common bitstream in different ways, and that tracking and analyzing these parameters can be tedious. In addition, there is not a perfect correspondence between syntax categories and merging methods (types), so for example, some parameters are required for category B, but the same parameters are not required for merging method 1.

[0043] To allow a device to easily identify applicable merge methods, merge identifier values ​​according to two or more of the above merge identifier type values ​​are generated and written to the bitstream: merge method (type) 0 indicates a first value of merge identifier type, merge method (type) 1 indicates a second value of merge identifier type, and merge method (type) 2 indicates a third value of merge identifier type.

[0044] The following example syntax needs to be incorporated into the hashset: <Temporal Motion Constraint Tile Set SEI Message (Merge Methods 0, 1, 2, Syntax Category C.b) Instructing Motion Constraints at Tiles and Image Borders> <Structure of Picture Information SEI Message (Merge Methods 0, 1, 2, Syntax Category C.b) Defining GOP Structure (i.e., Mapping from Encoding Order to Display Order, Random Access Point Indication, Temporal Hierarchy, Reference Structure)> <Syntax Element Values of Parameter Set> Reference Picture Set (Merge Method 0, Syntax Category B) Chroma Format (Merge Methods 0, 1, 2, Syntax Category D) Reference QP, Chroma QP Offset (Merge Method 0, Syntax Category B) Bit Depth Luma / Chroma (Merge Methods 0, 1, 2, Syntax Category D) <hrd Parameters> Initial Arrival Delay (Merge Method 0, Syntax Category B) Initial Removal Delay (Merge Method 0, Syntax Category B) <Encoding Tools> Encoding Block Structure (Maximum / Minimum Block Size, Predicted Partition) (Merge Methods 0, 1, Syntax Category C.a) Transform Size (Minimum / Maximum) (Merge Methods 0, 1, Syntax Category C.a) PCM Block Usage (Merge Methods 0, 1, Syntax Category C.a) High Motion Vector Prediction (Merge Methods 0, 1, 2, Syntax Category C.b) Sample Adaptive Offset (Merge Method 0, Syntax Category C.b) Temporal Motion Vector Prediction (Merge Methods 0, 1, 2, Syntax Category C.b) Intra Smoothing (Merge Methods 0, 1, Syntax Category C.a) Dependent Slices (Merge Method 0, Syntax Category A) Sign Concealment (Merge Methods 0, 1, Syntax Category C.a) Weighted Prediction (Merge Method 0, Syntax Category A) Transquant Bypass (Merge Methods 0, 1, Syntax Category C.a) Entropy-encoded synchronization (merge methods 0 and 1, syntax category Ca) (Skup4) Loop Filter (Merge Method 0, 1, 2, Syntax Category Cb) <slice header value> Parameter set ID (Merge method 0, Syntax category Ca) Reference image set (merging methods 0 and 1, syntax category B) <Usage of implicit CTU address signaling cp. (referenced by European patent application EP18153516) (Merge method 0, Syntax category A)>

[0045] That is, for the first value of the merge identifier type, i.e., type 0, syntax elements (parameters) describing motion constraints at tile and image boundaries, GOP structure, reference picture set, chroma format, reference and chroma quantization parameters, bit depth luma / chroma, virtual reference decoder parameters including parameters for initial arrival delay and parameter initial deletion delay, coded block structure, transform minimum and / or maximum size, usage of pulse code modulation blocks, advanced motion vector prediction, sample adaptive offset, temporal motion vector prediction, intra smoothing, dependent slices, code concealment, weighted prediction, transquant bypass, entropy coding synchronization, loop filter, slice header value with parameter set ID, slice header value with reference picture set, and usage of implicit coded transform unit address and notification must be incorporated into the hash set.

[0046] The second value of the merge identifier type, i.e., Type 1, includes syntax elements (parameters), namely, motion constraints at tile and picture boundaries, GOP structure, chroma format, bit depth luma / chroma, coding block structure, transform minimum and / or maximum size, pulse code modulation block usage, advanced motion vector prediction, sample adaptive offset, temporal motion vector prediction, intra smoothing, code concealment, transquant bypass, entropy coding synchronization, loop filter, and slice header value including reference picture set.

[0047] The third value of the merge identifier type, i.e., type 2, includes syntax elements (parameters), namely, motion constraints at tile and image boundaries, GOP structure, chroma format, bit depth luma / chroma, advanced motion vector prediction, sample adaptive offset, temporal motion vector prediction, and loop filter.

[0048] Figure 4 shows a schematic diagram illustrating an example of coding parameters according to an embodiment of the present invention. In Figure 4, reference numeral 40 denotes Type 0, and syntax elements belonging to Type 0 are indicated with dotted lines. Reference numeral 42 denotes Type 1, and syntax elements belonging to Type 1 are indicated with normal lines. Reference numeral 44 denotes Type 2, and syntax elements belonging to Type 2 are indicated with dashed lines.

[0049] 5a and 5b are an example of a sequence parameter set (SPS) 20, where the syntax elements required for Type 0 are designated by reference numeral 40. Similarly, the syntax elements required for Type 1 are designated by reference numeral 42, and the syntax elements required for Type 2 are designated by reference numeral 44.

[0050] 6a and 6b are an example of a picture parameter set (PPS) 22, where syntax elements required for Type 0 are indicated by reference numeral 40 and syntax elements required for Type 1 are indicated by reference numeral 42.

[0051] 7a to 7d are examples of slice headers 30, where only one syntax element of the slice header is required for type 0 as indicated by reference numeral 40 in FIG. 7c.

[0052] Figure 8 shows an example of a Structure of Picture (SOP) 26a, where all syntax elements are required to belong to the SOP for Type 2.

[0053] FIG. 9 is an example of a motion constrained tile set (MCTS) 26b, where Type 2 requires that all syntax elements belong to the MCTS.

[0054] 3 generates a merge identifier value using a hash function in hash member 16a and hash member 16b. When two or more merge identifier values ​​are generated using a hash function, the hashes are concatenated in the sense that an input to the hash function of a second merge identifier that encompasses additional elements in the hash set for the first merge identifier uses the first merge identifier (the hash result) instead of the respective syntax element values ​​in the concatenation of the inputs to the hash function.

[0055] Additionally, the presence of a merge identifier also provides a guarantee that the syntax elements incorporated in the hash set will have the same value in all access units (AUs) of a coded video sequence (CVS) and / or bitstream. This guarantee also has the form of a constraint flag in the profile / level syntax of a parameter set.

[0056] The merging process will now be described in view of another embodiment according to the invention using FIGS.

[0057] Figure 10 shows a video merger for providing a merged video stream based on multiple coded video representations. The video merger 50 includes a receiver 52 at which input video streams 12 (designated 12a and 12b in Figure 12) are received, a merging method identifier 54, and a merging processor 56. The merged video stream 60 is transmitted to a decoder. If the video merger 50 is included in a decoder, the merged video stream is transmitted to a user equipment or any other device for displaying the merged video stream.

[0058] The merging process is driven by the merge identifier and merge identifier type described above. The merging process can only generate parameter sets and interleave NAL units, which is the lightest form of merging associated with a merge identifier type value of 0, i.e., the first complexity. Figure 12 illustrates an example of a first complexity merging method. As indicated in Figure 12, parameter sets SPS1 of video stream 12a and SPS2 of video stream 12b are merged (a merged SPS is generated based on SPS1 and SPS2), and PPS1 of video stream 12a and PPS2 of video stream 12b are merged (a merged PPS is generated based on PPS1 and PPS2). The IDR is optional data and therefore will not be described. Additionally, slices 1,1 and 1,2 of video stream 12a and slices 2,1 and 2,2 of video stream 12b are interleaved, as shown as merged video stream 60. 10 and 12, two video streams 12a and 12b are input as an example, however, more video streams may be input and merged in the same way.

[0059] If necessary, the merging process may also include rewriting slice headers in the bitstream during NAL unit interleaving, which is associated with a merge identifier type value of 1, i.e., the second complexity. And finally, syntax elements in the slice payload that require entropy decoding and encoding during NAL unit interleaving, which is associated with a merge identifier type value of 2, i.e., the third complexity, may need to be adjusted. The merge identifier and merge identifier type drive the selection of one of the merging processes to be performed and the determination of its details.

[0060] The input to the merging process is a list of input sub-bitstreams, which also represent their spatial arrangement. The output of the process is the merged bitstream. In general, in every bitstream merging process, a parameter set for the new output bitstream needs to be generated, which can be based on the parameter sets of the input sub-bitstreams, e.g., the first input sub-bitstream. The required parameter set updates include image size. For example, the image size of the output bitstream is calculated as the sum, in each dimension, of the image sizes of the input sub-bitstreams in the context of their spatial arrangement.

[0061] It is a requirement of the bitstream merging process that all input sub-bitstreams have the same value of the merge identifier and at least one instance of the merge identifier type. In one embodiment, the merging process is performed associated with the lowest value of the merge identifier type for which all sub-bitstreams have the same value of the merge identifier.

[0062] For example, the difference in merge identifier values ​​having a particular merge identifier type value is used in the merging process to determine the details of the merging process according to the difference value of the merge identifier type with which the merge identifier value matches. For example, as shown in Figure 11, if a first merge identifier 70 with a merge identifier type value equal to 0 does not match between two input sub-bitstreams 70a, 70b, but a second merge identifier 80 with a merge identifier type value equal to 1 matches between two input sub-bitstreams 80a, 80b, the difference in bit positions of the first merge identifier 70 indicates a syntax element (associated with the slice header) that requires adjustment in all slices.

[0063] 13 illustrates a video merger 50 (50a) including a receiver (not shown), a merging method identifier including a merging identifier comparator 54a and an encoding parameter comparator 54b, and a merging processor 56. If the input video streams 12 include a hash value as a merging identifier value, the values ​​of each input video stream are compared in the merging identifier comparator 54a. For example, if both input video streams have the same merging identifier value, the individual encoding parameters of each input video stream are compared in the encoding parameter comparator 54b. Based on the encoding parameter comparison results, a merging method is determined, and the merging processor 56 merges the input video streams 12 using the determined merging method. If the merging identifier value (hash value) also indicates the merging method, comparison of the individual encoding parameters is not necessary.

[0064] Three merging methods are described above: the first, second, and third complexity merging methods. The fourth merging method is merging video streams using full pixel decoding and pixel re-encoding. The fourth merging method is applied when all three merging methods cannot be applied.

[0065] 14 and 15, a process for identifying the level of a merged result will be described below in consideration of another embodiment according to the present invention. Identifying the level of a merged result means placing information in a sub-bitstream, how large the contribution of the sub-bitstream is to the level limit of the merged bitstream that incorporates it.

[0066] FIG. 14 shows an encoder 2 (2b) including an encoder core including encoding parameter determination 14, merge identifier provision (not shown), and granularity function provision 8.

[0067] In general, when sub-bitstreams are to be merged into a combined bitstream, an indication of how each sub-bitstream contributes to the level-specific limitations of the codec system to which the potential merged bitstream must adhere is essential to ensure the creation of a legal combined bitstream. While traditionally, the granularity of codec levels is fairly coarse, e.g., identifying primary resolutions such as 720p, 1080p, or 4K, the indication of merge level limitations requires much finer granularity. The granularity of this traditional level indication is insufficient to represent the contribution of each individual sub-bitstream to the merged bitstream. Given that the number of tiles to be merged is unknown in advance, a reasonable tradeoff between flexibility and bitrate cost must be found, which generally far exceeds the granularity of traditional level limitations. One exemplary use case is a 360° video stream where a service provider needs the freedom to choose from different tile structures, such as 12 tiles, 24 tiles, or 96 tiles per 360° video, with each tile stream contributing 1 / 12, 1 / 24, or 1 / 96 of the overall level limit, such as 8K, assuming equal rate distribution. Furthermore, arbitrarily fine granularity may be required, assuming uneven rate distribution between tiles, for example, to achieve uniform quality across the video plane.

[0068] Such signaling may be, for example, a signaled ratio and / or an additional signaled level percentage. For example, in a four-participant conference scenario, each participant would transmit a legal Level 3 bitstream that also includes an indication, i.e., the level information included in the coarse-grained capability information, indicating that the transmitted bitstream adheres to 1 / 3 and / or 33% of the Level 5 limits, i.e., this information may be included in the fine-grained capability information. A receiver of multiple such streams, i.e., a video merger 50 (50b) as shown in FIG. 15, for example, may thus know that three such bitstreams may be mergeable into a single combined bitstream that adheres to Level 5.

[0069] The granularity capability information may have ratio and / or percentage indications as a vector of values, with each dimension relating to a different aspect of the codec level restriction, e.g., maximum allowed amount of luma samples per second, maximum picture size, bit rate, buffer occupancy, number of tiles, etc. Additionally, the ratios and / or percentages refer to the general codec level of the video bitstream.

[0070] While some aspects are described in the context of an apparatus or system, it will be apparent that these aspects also represent descriptions of corresponding methods, in which blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent descriptions of corresponding apparatus and / or system blocks or items or features. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0071] The data stream of the present invention can be stored on a digital storage medium or transmitted over a transmission medium, such as a wireless or wired transmission medium, such as the Internet.

[0072] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementations can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to perform the respective methods. Thus, the digital storage medium can be computer-readable.

[0073] Some embodiments according to the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0074] Generally, embodiments of the present invention may be implemented as a computer program product having program code that operates to perform one of the methods when the computer program product is run on a computer, and the program code may, for example, be stored on a machine-readable carrier.

[0075] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0076] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein when the computer program runs on a computer.

[0077] Another embodiment of the inventive method is therefore a data carrier (or digital storage medium, or computer-readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-transitory.

[0078] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or sequence of signals can for example be adapted to be transferred via a data communication connection, for example via the Internet.

[0079] Another embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0080] Another embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0081] Another embodiment according to the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0082] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0083] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0084] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part as hardware and / or as software.

[0085] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0086] The methods described herein, or any components of the methods described herein, may be implemented, at least in part, by hardware and / or by software.

[0087] The above-described embodiments are merely illustrative of the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore the intention to be limited only by the scope of the appended claims and not by the specific details presented herein by way of description and illustration of the embodiments.

Claims

1. 1. A method for providing an encoded video representation, comprising: providing a merge identifier in the bitstream that indicates a method for merging two or more video streams; providing level information in the bitstream indicating compatibility of the video stream with video decoders having one of a plurality of predetermined functionality levels; providing fraction level information in the bitstream that references one of the plurality of predetermined levels of functionality and indicates a fraction of the one of the plurality of predetermined levels of functionality, the fraction of the one of the plurality of predetermined levels of functionality being specified as a ratio value; The method, wherein the merge identifier, the level information, and the fractional level information are usable by a video decoder to merge the two or more video streams.

2. The fraction level information is A fraction of the maximum allowed amount of luma samples per second, Fraction of maximum image size, Fraction of maximum bitrate, a fraction of the buffer occupancy, and / or fraction of the maximum number of tiles, The method of claim 1 , wherein the method exhibits one or more of the following:

3. 1. A method for decoding a provided video representation, comprising: receiving a merge identifier in a bitstream that indicates a method for merging two or more video streams; receiving level information in the bitstream indicating compatibility of the video stream with a functionality level of a plurality of predetermined functionality levels; receiving fraction level information in the bitstream that references one of the plurality of predetermined levels of functionality and indicates a fraction of the one of the plurality of predetermined levels of functionality, the fraction of the one of the plurality of predetermined levels of functionality being specified as a ratio value; merging the two or more video streams according to the merge identifier, the level information, and the fractional level information.

4. The fraction level information is A fraction of the maximum allowed amount of luma samples per second, Fraction of maximum image size, Fraction of maximum bitrate, a fraction of the buffer occupancy, and / or fraction of the maximum number of tiles, The method of claim 3 , wherein the method exhibits one or more of the following:

5. 1. A video encoder for providing an encoded video representation, comprising:

3. A video encoder comprising at least one processor including processing circuitry configured to perform the method of claim 1 or 2.

6. 1. A video decoder for decoding a provided video representation, comprising:

5. A video decoder comprising at least one processor including processing circuitry configured to perform the method of claim 3 or 4.

7. A non-transitory storage medium storing a computer program for executing the method according to any one of claims 1 to 4.

Images