Bitstream Merge

Abstract

It provides a video codec for efficiently merging video bitstreams. [Solution] A video encoder 2 provides an encoded video stream 12, the video encoder 2 encodes a video stream 12 which includes encoded parameter information describing a plurality of encoding parameters (sequence parameter set, picture parameter set, auxiliary enhancement information and slice header), encoded video content information and one or more merge identifiers indicating whether and / or how the encoded video stream 12 can be merged with another encoded video stream.

Description

Technical Field

[0001] The present invention relates to video encoding / decoding.

Background Art

[0002] Video composition is used in a number of applications where the composition of multiple video sources is presented to the user. Common examples are, for example, picture-in-picture (PiP) composition and fusion with overlay video content for advertising or user interfaces. Generating such a composition in the pixel domain requires complex calculations and may even be infeasible on a device with a single hardware decoder or other limited resources, necessitating parallel decoding of the input video bitstream. For example, in current IPTV system designs, the corresponding set-top boxes perform the composition, which is a major service cost factor due to their complexity, distribution, and limited lifespan. Reduction of these cost factors has motivated the ongoing effort to virtualize the set-top box functionality, for example, the shift of user interface generation to cloud resources. In such an approach, only the video decoder, so-called zero client, is the only hardware remaining in the customer's home. The state-of-the-art technology for such system designs is the simplest form of composition based on transcoding, that is, decoding, composition in the pixel domain, and re-encoding before or during transport. To reduce the workload from the full decoding and encoding cycles, operations in the transform coefficient domain rather than the pixel domain were first proposed for PiP composition. Since then, numerous techniques have been proposed to fuse or shorten the individual composition steps and apply them to current video coders. However, the transcoding-based approach for general composition is still computationally complex and impairs the scalability of the system. Depending on the transcoding approach, such composition may also affect the rate distortion (RD) performance.

[0003] In addition, there are a wide range of applications built on tiles, where a tile is a spatial subset of a video plane encoded independently of adjacent tiles. A tile-based streaming system for 360° video works by dividing the 360° video into tiles, which are encoded into sub-bitstreams of varying resolutions and merged into a single bitstream on the client side depending on the current user's viewing orientation. Another application involving sub-bitstream merging is, for example, the combination of conventional video content and banner advertisements. Furthermore, sub-bitstream recombination can also be an important part of video conferencing systems where multiple users send individual video streams to a receiver where the streams are ultimately merged. Moreover, a tile-based cloud encoding system where video is divided into tiles and the tile encoding is distributed to separate, independent instances relies on fail-safe checks before the resulting tile bitstreams are merged back into a single bitstream. In all these applications, the streams are merged to enable decoding in a single video decoder using known compatibility points. Merging here means a lightweight bitstream rewrite that does not require a full decoding and encoding operation of the entropy-encoded data of the bitstream or pixel value reconstruction. However, the techniques used to ensure the success of bitstream merge formation, i.e., the conformity of the resulting merged bitstream, originated from very different application scenarios.

[0004] For example, legacy codecs support a technique called Motion Constrained Tile Set (MCTS), in which the encoder constrains inter-predictions between images to be limited to tile or image boundaries, i.e., without using sample values ​​or syntactic element values ​​that do not belong to the same tile or are located outside the image boundary. The origin of this technique is Region of Interest (ROI) decoding, in which the decoder can decode only specific subsections of the bitstream and encoded image without encountering unresolved dependencies and drift. Another state-of-the-art technique in this context is the Structure of Pictures (SOP) SEI message, which gives instructions for the applied bitstream structure, i.e., the order of images and the reference structure for inter-predictions. This information is provided by summarizing, for each image between two Random Access Points (RAPs), the Picture Order Count (POC) value, the Active Sequence Parameter Set (SPS) identifier, and the Reference Picture Set (RPS) index to the Active SPS. Based on this information, a transcoder, middlebox, or media-aware network entity (MANE) or media player can identify the bitstream structure that can be used to manipulate or modify the bitstream, such as adjusting the bitrate, dropping frames, or fast-forwarding.

[0005] While both of the exemplary notification techniques described above are essential in understanding whether lightweight merging of subbitstreams can be performed without complex syntax changes or even full transcoding, they are by no means sufficient. More specifically, lightweight merging in this context is characterized by interleaving the NAL units of the source bitstream with only minor rewrite operations, i.e., describing the parameter sets used together along with the new image size and tile structure, so that all bitstreams to be merged reside in separate tile regions. The next level of merging complexity consists, ideally, of minor rewrites of slice header elements that do not change the variable-length codes within the slice header. Further levels of merging complexity exist, and these levels are not considered lightweight, for example, by re-running entropy coding on the slice data to change certain syntactic elements that can be modified without requiring reconstruction of pixel values ​​that are entropy coded but can be considered lightweight compared to full transcoding involving decoding and coding the image.

[0006] In a merged bitstream, all slices must refer to the same parameter set. If the parameter sets of the original bitstreams use significantly different configurations, many of the syntactic elements of the parameter sets will further affect the syntax of the slice headers and slice payloads, and because the degree of this impact varies, a lightweight merge may not be possible. The further downstream in the decoding process the syntactic elements are, the more complex the merging / rewriting becomes. Several notable common categories of syntactic dependencies (of parameter sets and other structures) can be distinguished as follows: A. Existence indication of syntax B. Dependencies in value calculation C. Slice Payload Encoding Tool Control a. Syntax used in the initial stages of the decryption process (e.g., coefficient sign hiding, block partitioning restrictions) b. Syntax used later in the decoding process (e.g., motion compensation, loop filtering) or general decoding process controls (e.g., reference image, bitstream order) D. Source Format Parameters

[0007] In Category A, the parameter set includes numerous presence flags for various tools (such as dependent_slice_segments_enabled_flag and output_flag_present_flag). If there are differences in such flags, the flags can be set to be valid in the merged parameter set, and the default values ​​can be explicitly written to the merged slice header of slices in the bitstream that did not contain syntactic elements before merging; i.e., in this case, the merge requires changes to the syntax of the parameter set and slice header.

[0008] In Category B, the notified values ​​of the parameter set syntax may be used in the calculation together with parameters from other parameter sets or slice headers. For example, in HEVC, the slice quantization parameter (QP) of a slice is used to control the coarseness of the quantization of the conversion coefficients of the slice's residual signal. Notification of the slice QP in the bitstream (SliceQpY) depends on QP notification at the picture parameter set (PPS) level, as follows: SliceQpY=26+init_qp_minus26(from PPS)+slice_qp_delta(from slice header)

[0009] Since all slices in the (merged) encoded video image must refer to the same activated PPS, differences in the init_qp_minus26 of the PPS of the individual streams to be merged require adjustments to the slice headers so that they reflect a new common value for init_qp_minus26. In other words, in this case, the merge requires changes to the parameter set and the syntax of the slice headers, as in Category 1.

[0010] In Category C, syntactic elements of a further parameter set control the encoding tool that affects the bitstream structure of the slice payload. Subcategories Ca and Cb can be distinguished depending on how far downstream the syntactic elements are in the decoding process and (relatively) the complexity associated with the changes to these syntactic elements, i.e., where the syntactic elements lie between entropy coding and pixel-level reconstruction.

[0011] For example, one element of category Ca is the sign_data_hiding_enabled_flag, which controls the derivation of the sign data for the encoded transformation coefficients. Sign data hiding can be easily deactivated, and the corresponding inferred sign data is explicitly written to the slice payload. However, such a change to the slice payload in this category does not require proceeding to the pixel region by full decoding before re-encoding the pixel-by-pixel merged image. Another example is the inferred block partitioning decision, or any other syntax that allows the inferred values ​​to be easily written to the bitstream. That is, in this case, the merge requires changes to the parameter set, the syntax of the slice header, and the slice payload that requires entropy decoding / encoding.

[0012] However, subcategory Cb requires syntactic elements related to processes much further downstream in the decoding process, and therefore many complex decoding processes must be performed in a way that, in any case, avoiding the remaining decoding steps is undesirable from an implementation and computational standpoint. For example, differences in syntactic elements connected to various decoding process-related syntaxes, such as motion-compensated "temporal motion-constrained tileset SEI (Structure of Picture Information) messages" or "SEI messages," make it impossible to avoid pixel-level transcoding. In contrast to subcategory Ca, there are numerous encoder decisions that affect the slice payload in a way that cannot be altered without complete pixel-level transcoding.

[0013] Category D includes a syntactic element of a parameter set that makes pixel-level transcoding unavoidable in order to merge subbitstreams by different values ​​(such as chroma subsampling as indicated by chroma_format_idc). In other words, in this case, merging requires the processes of full decoding, pixel-level merging, and full encoding.

[0014] While the list above is by no means exhaustive, it becomes clear that a wide range of parameters influence the benefits of merging sub-bitstreams into a common bitstream in various ways, and that tracking and analyzing these parameters is time-consuming. [Overview of the project] [Problems that the invention aims to solve]

[0015] The object of the present invention is to provide a video codec for efficiently merging video bitstreams. [Means for solving the problem]

[0016] This objective is achieved by the subject matter of the claims of this application.

[0017] The fundamental idea underlying this invention is that improvements in merging multiple video streams are achieved by including one or more merge identifiers. This approach reduces the load on computing resources and accelerates the merging process.

[0018] According to embodiments of the present invention, the video encoder is configured to provide a video stream including encoded parameter information describing a plurality of encoding parameters, encoded video content information, i.e., information encoded using the encoding parameters defined by the parameter information, and one or more merge identifiers indicating whether and / or how an encoded video representation can be merged with another encoded video representation, for example, using what complexity. The complexity to use may be determined based on the parameter values ​​defined by the parameter information. The merge identifier may be a concatenation of a plurality of encoding parameters that must be equal in two different encoded video representations in order to enable the two different encoded video representations to be merged using a predetermined complexity. In addition, the merge identifier may be a hash value of the concatenation of a plurality of encoding parameters that must be equal in two different encoded video representations in order to enable the two different encoded video representations to be merged using a predetermined complexity.

[0019] According to embodiments of the present invention, a merge identifier can indicate a merge identifier type that represents the complexity of a merge procedure, or a “suitable” merge method, which involves merging by rewriting a parameter set, or by rewriting a parameter set and a slice header, or by rewriting a parameter set, a slice header, and a slice payload. The merge identifier is associated with a merge identifier type, and the merge identifier associated with a merge identifier type includes encoding parameters that must be equal in two different encoded video representations so that the two different encoded video representations can be merged using the complexity of the merge procedure represented by the merge identifier type. The value of the merge identifier type can indicate a merge process, and the video encoder is configured to switch between at least two of the following values ​​of the merge identifier type: a first value of the merge identifier type representing a merge process by rewriting a parameter set, a second value of the merge identifier type representing a merge process by rewriting a parameter set and a slice header, and a third value of the merge identifier type representing a merge process by rewriting a parameter set, a slice header, and a slice payload.

[0020] According to embodiments of the present invention, multiple merge identifiers are associated with different complications of the merge procedure, for example, each identifier indicates the parameter set / hash and type of the merge procedure. The encoder may be configured to check whether the encoding parameters evaluated for providing the merge identifier are identical in all units of the video sequence and to provide the merge identifier depending on this check.

[0021] According to embodiments of the present invention, the multiple encoding parameters may include merge-related parameters that must be identical in different video streams, i.e., encoded video representations, in order to enable merging with less complexity than merging by full pixel decoding, and the video encoder is configured to provide one or more merge identifiers based on the merge-related parameters, i.e., if there are no common parameters between the two encoded video representations, full pixel decoding is performed, i.e., there is no possibility of reducing the complexity of the merging process. Merging-related parameters include one, several, or all of the following parameters: motion constraints at tile boundaries, e.g., parameters describing motion constraint tail set supplemental enhancement information (MCTS SEI); group of picture (GOP) structure, i.e., mapping of encoding order to display order; random access point indication, e.g., temporal hierarchical structuring using SOP: Structure of Picture; information about SEI; parameters describing chroma encoding format and luma encoding format, e.g., a set containing at least chroma format and bit depth luma / chroma; parameters describing advanced motion vector prediction; parameters describing sample adaptive offset; parameters describing temporal motion vector prediction; and parameters describing loop filters and other encoding parameters. In other words, the parameter set (including reference image set, reference quantization parameters, etc.), slice header, and slice payload are rewritten to merge two encoded video representations.

[0022] According to embodiments of the present invention, a merge identifier associated with a first complexity of a merge procedure may be determined based on a first encoding parameter set, when a merge parameter associated with a second complexity of a merge procedure that is higher than the first complexity may be determined based on a second encoding parameter set that is a true subset of the first encoding parameter set. A merge identifier associated with a third complexity of a merge procedure that is higher than the second complexity may be determined based on a third encoding parameter set that is a true subset of the second encoding parameter set. The video encoder is configured to determine the merge identifier associated with the first complexity of a merge procedure based on a first set that must be equal in the encoded video representation to enable a merge of video streams by rewriting (only) the parameter set, i.e., a merge process that modifies only the parameter set applicable to multiple slices while keeping the slice header and slice payload immutable. The video encoder may be configured to determine a merge identifier associated with a second complexity of the merge procedure based on a second set of two different video streams, e.g., a second set that must be equal in the encoded video representation, in order to enable a merge of video streams in which the encoded parameter set is modified, for example, a parameter set applicable to multiple slices is modified while the slice payload remains immutable, and the slice header is also modified, i.e., a merge process involving rewriting the parameter set and slice header. The video encoder may be configured to determine a merge identifier associated with a third complexity of the merge procedure based on a third set of two different video streams, e.g., a third set that must be equal in the encoded video representation, in order to enable a merge of video streams in which the encoded parameter set is modified, for example, a parameter set applicable to multiple slices is modified, and the slice header and slice payload are also modified, but full pixel decoding and pixel re-encoding are not performed, i.e., a merge process involving rewriting the parameter set, slice header, and slice payload.

[0023] According to embodiments of the present invention, a video merger for providing a plurality of encoded video representations, for example, a video representation merged based on video streams, the video merger is configured to receive a plurality of video streams, each containing encoded parameter information describing a plurality of encoding parameters, encoded video content information, i.e., information encoded using the encoding parameters defined by the parameter information, and one or more merge identifiers indicating whether and / or how an encoded video representation can be merged with another encoded video representation, and / or how it can be merged, for example, using what complexity, and the video merger is configured to determine, with respect to the use of a merge method, for example, a merge type, a merge process, depending on the merge identifiers, i.e., depending on a comparison of the merge identifiers of different video streams. The video merger is configured to select a merge method from among a plurality of merge methods depending on the merge identifiers. The video merger may be configured to select from at least two of the following merging methods, namely merging methods of varying complexity depending on one or more merge identifiers: a first merging method which is a merging of video streams that only changes the parameter set applicable to multiple slices while keeping the slice header and slice payload immutable; a second merging method which is a merging of video streams that changes the parameter set applicable to multiple slices while keeping the slice payload immutable and also changes the slice header; and a third merging method which is a merging of video streams that changes the parameter set applicable to multiple slices and also changes the slice header and slice payload, but does not perform full pixel decoding and pixel recoding.

[0024] According to an embodiment of the present invention, a video merger compares merge identifiers of two or more video streams associated with the same given merge method or associated with the same merge identifier type, and determines whether to perform a merge using the given merge method depending on the result of the comparison. The video merger may be configured to selectively perform a merge using the given merge method when the comparison indicates that the merge identifiers of two or more video streams associated with the given merge method are equal. When the comparison of the merge identifiers indicates that the merge identifiers of two or more video streams associated with the given merge method are different, that is, without further comparing the encoding parameters themselves, the video merger may be configured to use a merge method having a higher complexity than the given merge method associated with the compared merge identifiers. When the comparison of the merge identifiers indicates that the merge identifiers of two or more video streams associated with the given merge method are equal, the video merger may be configured to selectively compare encoding parameters that must be equal in two or more video streams to enable the merge of the video streams using the given merge method. When the comparison of the encoding parameters, that is, the encoding parameters that must be equal in two or more video streams to enable the merge of the video streams using the given merge method, indicates that the encoding parameters are equal, the video merger is configured to selectively perform a merge using the given merge method. When the comparison of the encoding parameters indicates that the encoding parameters include a difference, the video merger is configured to perform a merge using a merge method having a higher complexity than the given merge method.

[0025] According to embodiments of the present invention, the video merger may be configured to compare merge identifiers associated with merge methods of different complications, i.e., compare hashes, the video merger is configured to identify the lowest complexity merge method in which the associated merge identifiers are equal in two or more video streams to be merged, the video merger is configured to compare sets of encoding parameters, i.e., individual encoding parameters, rather than hash versions of encoding parameters, which must be equal in two or more video streams to be merged in order to enable merging using the identified merge method, different, usually overlapping sets of encoding parameters are associated with merge methods of different complications, the video merger is configured to selectively merge two or more video streams using the identified merge method if the comparison indicates that the encoding parameters of the sets of encoding parameters associated with the identified merge method are equal in the video streams to be merged, the video merger is configured to merge two or more video streams using a merge method of higher complexity than the identified merge method if the comparison indicates that the encoding parameters of the sets of encoding parameters associated with the identified merge method contain differences. The video merger is configured, for example, to determine which encoding parameters should be modified during the merging process, i.e., during the merging of video streams, depending on one or more differences between merge identifiers associated with the same merging method or "merge identifier type" of different video streams to be merged.

[0026] According to an embodiment of the present invention, the video merger obtains a combined encoding parameter or a set of combined encoding parameters, such as a sequence parameter set SPS and a picture parameter set PPS, associated with slices of all video streams to be merged, based on the encoding parameters of the video streams to be merged. For example, when the values of all encoding parameters of one encoded video representation and another encoded video representation are the same, the combined encoding parameter is configured to be included in the merged video stream. When there are differences between the encoding parameters of the encoded video representations, the encoding parameters are updated by copying common parameters. The encoding parameters are updated based on the main (i.e., one is main and the other is sub) encoded video representation. For example, some encoding parameters can be adapted according to the combination of video streams, such as the total image size. The video merger is configured to adapt the encoding parameters associated individually with the individual video slices, for example, defined in the slice header or when using a merge method having a complexity higher than the minimum, to obtain the modified slices to be included in the merged video stream. The adapted encoding parameters include parameters representing the image size of the merged encoded video representation, and the image size is calculated based on the image sizes of the encoded video representations to be merged, i.e., in each dimension, in the context of their spatial arrangement.

[0027] According to embodiments of the present invention, a video encoder for providing a provided encoded video representation, i.e., a video stream, the video encoder provides coarse granularity capability demand information that describes the compatibility of the video stream with a video decoder having a functional level among a plurality of predetermined functional levels, for example, the decryptability of the video stream by the video decoder. The video encoder may be configured to provide fine-grained feature demand information, such as merge level limit information, which describes what fraction of an acceptable feature demand associated with one of a predetermined feature level, i.e., the decoder function is required to decode the encoded video representation, and / or what fraction of an acceptable feature demand, i.e., the "level limit of the merged bitstream", describes which video streams contribute to the merged video streams that are merged, i.e., the video streams whose feature demand matches one of a predetermined feature level, i.e., the "level limit of the merged bitstream", where the "acceptable feature demand of the merged bitstream matching one of a predetermined feature level" corresponds to the "level limit of the merged bitstream". The video encoder may be configured to provide fine-grained feature demand information such that the fine-grained feature demand information includes a ratio or percentage value that refers to one of a predetermined feature level. The video encoder is configured to provide fine-grained function demand information such that the fine-grained function demand information includes reference information and fraction information, the reference information describing which of the given function levels the fraction information refers to, such that the fine-grained function demand information as a whole describes a fraction of one of the given function levels.

[0028] According to embodiments of the present invention, a video merger for providing a plurality of encoded video representations, i.e., video representations merged based on video streams, the video merger may be configured to receive a plurality of video streams, each including encoded parameter information describing a plurality of encoding parameters, encoded video content information, e.g., information encoded using the encoding parameters defined by the parameter information, coarse-grained functional demand information, e.g., level information, level 3, level 4, or level 5, and fine-grained functional demand information, e.g., merge level limit information, describing the compatibility of the video streams with a video decoder having a functional level among a plurality of predetermined functional levels, i.e., the decryptability of the video streams by the video decoder, and the video merger is configured to merge two or more video streams depending on the coarse-grained functional demand information and the fine-grained functional demand information. The video merger may be configured to determine which video streams can and should be included in the merged video stream, i.e., without violating acceptable functional demands, i.e., such that the functional demands of the merged video streams match one of the predetermined functional levels depending on the fine-grained functional demand information. A video merger may be configured to determine, for example, whether a valid merged video stream can be obtained by merging two or more video streams, depending on fine-grained resolution feature requirement information, such that it does not violate acceptable feature requirements, i.e., the feature requirements of the merged video streams match one of a given feature level. The video merger may be configured to summarize the fine-grained feature requirement information of multiple video streams to be merged, for example, in order to determine which video streams can be included or whether a valid merged video stream can be obtained.

[0029] Preferred embodiments of the present invention will be described below with reference to the figures. [Brief explanation of the drawing]

[0030] [Figure 1]This block diagram shows an example of a video encoder capable of implementing the bitstream merging concept according to an embodiment of the present invention, and is a device for providing an encoded video representation. [Figure 2] This is a schematic diagram showing an example of a bitstream structure according to an embodiment of the present invention. [Figure 3] A block diagram shows an apparatus for providing an encoded video representation, as another example of a video encoder capable of implementing the bitstream merging concept according to embodiments of the present invention. [Figure 4-1] This is a schematic diagram showing an example of coding parameters according to an embodiment of the present invention. [Figure 4-2] This is a schematic diagram showing an example of coding parameters according to an embodiment of the present invention. [Figure 4-3] This is a schematic diagram showing an example of coding parameters according to an embodiment of the present invention. [Figure 4-4] This is a schematic diagram showing an example of coding parameters according to an embodiment of the present invention. [Figure 5a] Figure 4 shows an example of a detailed sequence parameter set (SPS). [Figure 5b] Figure 4 shows an example of a detailed sequence parameter set (SPS). [Figure 6a] Figure 4 shows an example of a detailed Picture Parameter Set (PPS). [Figure 6b] Figure 4 shows an example of a detailed Picture Parameter Set (PPS). [Figure 7a] Figure 4 shows an example of a detailed slice header. [Figure 7b] Figure 4 shows an example of a detailed slice header. [Figure 7c] Figure 4 shows an example of a detailed slice header. [Figure 7d] Figure 4 shows an example of a detailed slice header. [Figure 8]This figure shows a detailed structure of picture in the Supplemental Enhancement Information (SEI) message shown in Figure 4. [Figure 9] Figure 4 shows a detailed motion constraint tileset in the SEI message. [Figure 10] This is a block diagram showing an example of a video merger capable of implementing the bitstream merging concept according to an embodiment of the present invention, which is an apparatus for providing a merged video representation. [Figure 11] This is a schematic diagram illustrating the process for determining the complexity of a merge according to an embodiment of the present invention. [Figure 12] This is a schematic diagram showing the bitstream structures of multiple video representations to be merged and the bitstream structure of the merged video representation according to the bitstream merging concept of the present invention. [Figure 13] This block diagram shows another example of a video merger capable of implementing the bitstream merging concept according to embodiments of the present invention, which is an apparatus for providing merged video representations. [Figure 14] This block diagram shows an example of a video encoder that provides functional demand information for merged video representations according to an embodiment of the present invention, and is a device for providing encoded video representations. [Figure 15] This block diagram shows an example of a video merger that provides functional demand information for merged video representations according to an embodiment of the present invention, specifically an apparatus for providing merged video representations. [Modes for carrying out the invention]

[0031] The following description provides specific details, such as particular embodiments, procedures, and techniques, for illustrative purposes only, not limitation. Those skilled in the art will understand that, apart from these specific details, other embodiments may also be used. For example, the following description proceeds using non-limiting illustrative applications, but this technique can be applied to any type of video codec. In some cases, detailed descriptions of well-known methods, interfaces, circuits, and devices are omitted to avoid complicating the description with unnecessary details.

[0032] In the following description, equivalent or equivalent elements with equivalent or equivalent functions are represented by equivalent or equivalent reference symbols.

[0033] The invention of this specification provides a means for inserting instructions into each sub-bitstream that enable future video codecs, such as VVC (Versatile Video Coding), to identify sub-bitstreams that can be merged together into a legal bitstream or sub-bitstreams that cannot be merged together into a legal bitstream with a given level of complexity. These instructions are hereafter referred to as “merge identifiers” and further provide information regarding appropriate merging methods through instructions referred to as “merge identifier types.” Assuming that two bitstreams have “merge identifiers” and the same value, it is possible to merge the sub-bitstreams into a new combined bitstream with a given level of merging complexity (the associated merging method).

[0034] Figure 1 shows a video encoder 2 that provides an encoded video representation, i.e., an encoded video stream 12, based on a provided (input) video stream 10, including an encoder core 4 that includes an encoding parameter determination member 14 and a merge identifier provider 6. The provided video stream 10 and the encoded video stream 12 each have a bitstream structure, for example, as shown in a simplified configuration in Figure 2. This bitstream structure consists of multiple Network Abstraction Layer (NAL) units, each NAL unit containing various parameters and / or data, such as a sequence parameter set (SPS) 20, a picture parameter set (PPS) 22, an instantaneous decoder refresh (IDR) 24, auxiliary enhancement information (SEI) 26, and multiple slices 28. The SEI 26 contains various messages, i.e., structure of picture, motion constraint tile set, etc. The slices 28 contain a header 30 and a payload 32. The encoding parameter determination member 14 determines the encoding parameters based on the SPS20, PPS22, SEI26, and slice header 30. The IDR24 is not necessarily a factor in determining the encoding parameters according to the present invention, but it can be optionally included to determine the encoding parameters. The merge identifier provider 6 provides a merge identifier that indicates whether and / or how (to what complexity) an encoded video stream can be merged with another encoded video stream.A merge identifier determines a merge identifier that indicates the complexity of a merge procedure, such as merging by rewriting a parameter set, or merging by rewriting a parameter set and slice header, or merging by rewriting a parameter set, slice header, and slice payload, or generally indicates a merge identifier type that is associated with an "appropriate" merge method. A merge identifier is associated with a merge identifier type, and a merge identifier associated with a merge identifier type contains encoding parameters that must be equal in two different encoded image representations so that the two different encoded image representations can be merged using the complexity of the merge procedure represented by the merge identifier type.

[0035] As a result, the encoded video stream 12 includes encoded parameter information describing a plurality of encoding parameters, encoded video content information, and one or more merge identifiers. In this embodiment, the merge identifier is determined based on the encoding parameters. However, the merge identifier value is set according to the wishes of the encoder operator, and in a closed system, there may be little guarantee of sufficient collision avoidance. Third-party entities such as DVB (Digital Video Broadcasting) and ATSC (Advanced Television Systems Committee) may also define the values ​​of the merge identifier to be used within their systems.

[0036] The merge identifier type is described below with reference to another embodiment of the present invention using Figures 3 to 9.

[0037] Figure 3 shows encoder 2(2a), including encoder core 4 and merge discriminator 6a, and illustrates the data flow in encoder 2a. Merge discriminator 6a includes a first hash member 16a and a second hash member 16b that provide a hash value as a merge identifier to indicate the merge type. That is, the merge value can be formed from the concatenation of encoded values ​​of a defined set of syntactic elements (encoding parameters) of the bitstream, which is hereafter called a hash set. The merge identifier value can also be formed by passing the above concatenation of encoded values ​​to a well-known hash function such as MD5, SHA-3, or any other suitable function. As shown in Figure 3, the input video stream 10 contains input video information, and the input video stream 10 is processed by encoder core 4. Encoder core 4 encodes the input video content, and the encoded video content information is stored in payload 32. Encoding parameter determination member 14 receives parameter information, including SPS, PPS, slice header, and SEI message. Parameter information is stored in each corresponding unit and received by the merge identifier provider 6a, i.e., by the first hash member 16a and the second hash member 16b, respectively. The first hash member 16a generates a hash value indicating merge identifier type 2 based on the encoding parameters, for example, and the second hash member 16b generates a hash value indicating merge identifier type 1 based on the encoding parameters, for example.

[0038] The quality of the mergeability indication for the above syntax categories is determined by the contents of the hash set, i.e., the syntactic element (i.e., merge parameter) values ​​that are concatenated to form the merge identifier value.

[0039] For example, a merge identifier type indicates the appropriate merging method for a merge identifier, i.e., different levels of merging possibilities corresponding to syntactic elements embedded in a hash set, as follows: Type 0: Merge identifier for merging by rewriting parameter sets Type 1: Merge identifier for merging by rewriting parameter sets and slice headers Type 2: Merge identifier for merging by rewriting parameter sets, slice headers, and slice payloads.

[0040] For example, given two input sub-bitstreams to be merged in the context of an application, the device can compare the values ​​of the merge identifier and the merge identifier type and conclude which method to use for the two sub-bitstreams that is associated with the merge identifier type.

[0041] The table below shows the mapping between syntactic element categories and associated merge methods. [Table 1]

[0042] As already mentioned above, the syntax categories are by no means exhaustive, but it becomes clear that a wide range of parameters affect the benefits of merging sub-bitstreams into a common bitstream in various ways, and that tracking and analyzing these parameters is time-consuming. In addition, the syntax categories and merge methods (types) do not correspond perfectly; therefore, for example, category B requires several parameters, while merge method 1 does not require the same parameters.

[0043] To enable the device to easily identify applicable merge methods, a merge identifier value is generated using two or more of the merge identifier type values ​​described above and written to the bitstream. Merge method (type) 0 indicates the first value of the merge identifier type, merge method (type) 1 indicates the second value of the merge identifier type, and merge method (type) 2 indicates the third value of the merge identifier type.

[0044] The following example syntax needs to be incorporated into the hash set. <Temporal Motion Constraint Tile Set SEI Message (Merge Methods 0, 1, 2, Syntax Category C.b) Instructing Motion Constraints at Tiles and Image Borders> <Structure of Picture Information SEI Message (Merge Methods 0, 1, 2, Syntax Category C.b) Defining GOP Structure (i.e., Mapping from Encoding Order to Display Order, Random Access Point Indication, Temporal Hierarchy, Reference Structure)> <Syntax Element Values of Parameter Set> Reference Picture Set (Merge Method 0, Syntax Category B) Chroma Format (Merge Methods 0, 1, 2, Syntax Category D) Reference QP, Chroma QP Offset (Merge Method 0, Syntax Category B) Bit Depth Luma / Chroma (Merge Methods 0, 1, 2, Syntax Category D) <hrd Parameters> Initial Arrival Delay (Merge Method 0, Syntax Category B) Initial Removal Delay (Merge Method 0, Syntax Category B) <Coding Tools> Coding Block Structure (Maximum / Minimum Block Size, Speculated Partitioning) (Merge Methods 0, 1, Syntax Category C.a) Transform Size (Minimum / Maximum) (Merge Methods 0, 1, Syntax Category C.a) PCM Block Usage (Merge Methods 0, 1, Syntax Category C.a) High Motion Vector Prediction (Merge Methods 0, 1, 2, Syntax Category C.b) Sample Adaptive Offset (Merge Method 0, Syntax Category C.b) Temporal Motion Vector Prediction (Merge Methods 0, 1, 2, Syntax Category C.b) Intra Smoothing (Merge Methods 0, 1, Syntax Category C.a) Dependent Slices (Merge Method 0, Syntax Category A) Sign Concealment (Merge Methods 0, 1, Syntax Category C.a) Weighted Prediction (Merge Method 0, Syntax Category A) Transquant Bypass (Merge Methods 0, 1, Syntax Category C.a) Entropy-coded synchronization (merge method 0, 1, syntactic category Ca) (Skup4) Loop filter (Merge method 0, 1, 2, Syntax category Cb) <Slice header value> Parameter set ID (Merge method 0, Syntax category Ca) Reference image set (Merge method 0, 1, Syntax category B) <Implicit CTU address signaling cp. usage (referenced by European Patent Application EP18153516) (Merge Method 0, Syntax Category A)>

[0045] In other words, for the first value of the merge identifier type, i.e., type 0, the hashset must incorporate syntactic elements (parameters), namely motion constraints at tile and image boundaries, GOP structure, reference image set, chroma format, reference quantization parameters and chroma quantization parameters, bit depth luma / chroma, virtual reference decoder parameters including parameters for initial arrival delay and parameter initial deletion delay, coding block structure, transform minimum and / or maximum size, pulse code modulation block usage, advanced motion vector prediction, sample adaptive offset, temporal motion vector prediction, intra-smoothing, dependent slice, code hiding, weighted prediction, transquant bypass, entropy coding synchronization, loop filter, slice header value including parameter set ID, slice header value including reference image set, and implicit coding transform unit address notification usage.

[0046] The second value of the merge identifier type, namely, in type 1, is a slice header value that includes syntactic elements (parameters), namely motion constraints at tile and image boundaries, GOP structure, chroma format, bit depth chroma / chroma, coded block structure, transformation minimum and / or maximum size, pulse code modulation block usage, advanced motion vector prediction, sample adaptive offset, temporal motion vector prediction, intra-smoothing, code hiding, transquant bypass, entropy coded synchronization, loop filter, and reference image set.

[0047] The third value for the merge identifier type, namely type 2, is the syntactic elements (parameters), namely motion constraints at tile and image boundaries, GOP structure, chroma format, bit depth chroma / chroma, advanced motion vector prediction, sample adaptive offset, temporal motion vector prediction, and loop filter.

[0048] Figure 4 shows a schematic diagram illustrating an example of encoding parameters according to an embodiment of the present invention. In Figure 4, reference numeral 40 represents type 0, and syntactic elements belonging to type 0 are indicated by dotted lines. Reference numeral 42 represents type 1, and syntactic elements belonging to type 1 are indicated by regular lines. Reference numeral 44 represents type 2, and syntactic elements belonging to type 2 are indicated by dashed lines.

[0049] Figures 5a and 5b show an example of a sequence parameter set (SPS) 20, where the syntactic elements required for type 0 are indicated by reference numeral 40. Similarly, the syntactic elements required for type 1 are indicated by reference numeral 42, and the syntactic elements required for type 2 are indicated by reference numeral 44.

[0050] Figures 6a and 6b show examples of picture parameter sets (PPS) 22, where the syntactic elements required for type 0 are indicated by reference numeral 40, and the syntactic elements required for type 1 are indicated by reference numeral 42.

[0051] Figures 7a to 7d show examples of slice headers 30, and as indicated by reference numeral 40 in Figure 7c, type 0 requires only one syntactic element of the slice header.

[0052] Figure 8 shows an example of a Structure of Picture (SOP) 26a, and it is required for Type 2 that all syntactic elements belong to the SOP.

[0053] Figure 9 shows an example of a motion-constrained tileset (MCTS) 26b, where all syntactic elements must belong to the MCTS for type 2.

[0054] As described above, the merge identifier 6a in Figure 3 generates merge identifier values ​​using hash functions in hash members 16a and 16b. When two or more merge identifier values ​​are generated using hash functions, the hash is concatenated in the sense that the input to the hash function of the second merge identifier, which includes additional elements in the hash set with respect to the first merge identifier, uses the first merge identifier (hash result) instead of the respective syntactic element values ​​in concatenating the inputs to the hash function.

[0055] In addition, the presence of merge identifiers also provides a guarantee that syntactic elements incorporated into the hash set have the same value in all access units (AUs) of the encoded video sequence (CVS) and / or bitstream. Furthermore, this guarantee takes the form of constraint flags in the parameter set's profile / level syntax.

[0056] The merge process will be described below with reference to another embodiment of the present invention using Figures 10 to 13.

[0057] Figure 10 shows a video merger for providing a video stream merged based on multiple encoded video representations. The video merger 50 includes a receiver 52 that receives the input video streams 12 (12a and 12b as indicated in Figure 12), a merge method discriminator 54, and a merge processor 56. The merged video stream 60 is transmitted to a decoder. If the video merger 50 is included in the decoder, the merged video stream is transmitted to user equipment or any other device to display the merged video stream.

[0058] The merge process is driven by the merge identifier and merge identifier type described above. The merge process can only perform parameter set generation and NAL unit interleaving, which is the lightest form of merge associated with merge identifier type value 0, i.e., first complexity. Figure 12 shows an example of how to merge first complexity. As indicated in Figure 12, the parameter sets, i.e., SPS1 of video stream 12a and SPS2 of video stream 12b, are merged (a merged SPS is produced based on SPS1 and SPS2), and PPS1 of video stream 12a and PPS2 of video stream 12b are merged (a merged PPS is produced based on PPS1 and PPS2). IDR is optional data and is therefore omitted from the explanation. In addition, slices 1,1 and 1,2 of video stream 12a and slices 2,1 and 2,2 of video stream 12b are interleaved, as shown as merged video stream 60. In Figures 10 and 12, two video streams 12a and 12b are used as example inputs. However, more video streams may be input and merged in the same manner.

[0059] If necessary, the merge process may also include rewriting the slice header in the bitstream during NAL unit interleaving, associated with merge identifier type value 1, i.e., the second complexity. Finally, syntactic elements in the slice payload that require entropy decoding and encoding during NAL unit interleaving may need to be adjusted, associated with merge identifier type value 2, i.e., the third complexity. The merge identifier and merge identifier type drive the selection of one of the merge processes to be performed and the determination of its details.

[0060] The input to the merge process is a list of input sub-bitstreams, each representing a spatial arrangement. The output to the process is the merged bitstream. In general, every bitstream merge process requires the generation of a new output bitstream parameter set, which can be based on the parameter set of the input sub-bitstreams, e.g., the first input sub-bitstream. Necessary updates to the parameter set include the image size. For example, the image size of the output bitstream is calculated as the sum of the image sizes of the input sub-bitstreams in each dimension, in the context of their spatial arrangements.

[0061] A requirement of the bitstream merge process is that all input sub-bitstreams have the same value for the merge identifier and at least one instance of the merge identifier type. In one embodiment, a merge process is performed associated with the lowest value of the merge identifier type, where all sub-bitstreams have the same value for the merge identifier.

[0062] For example, the difference between merge identifier values ​​having a specific merge identifier type value is used in the merge process to determine the details of the merge process according to the difference value of the merge identifier type to which the merge identifier value matches. For example, as shown in Figure 11, if a first merge identifier 70 with a merge identifier type value equal to 0 does not match between the two input subbitstreams 70a and 70b, but a second merge identifier 80 with a merge identifier type value equal to 1 matches between the two input subbitstreams 80a and 80b, then the difference in the bit position of the first merge identifier 70 indicates the syntactic elements (related to the slice header) that require adjustment in all slices.

[0063] Figure 13 shows a video merger 50(50a) including a receiver (not shown), a merge method discriminator which includes a merge identifier comparator 54a and an encoding parameter comparator 54b, and a merge processor 56. If the input video stream 12 includes a hash value as the merge identifier value, the values ​​of each input video stream are compared in the merge identifier comparator 54a. For example, if both input video streams have the same merge identifier value, the individual encoding parameters of each input video stream are compared in the encoding parameter comparator 54b. Based on the encoding parameter comparison result, the merge method is determined, and the merge processor 56 merges the input video streams 12 using the determined merge method. If the merge identifier value (hash value) also indicates the merge method, comparison of individual encoding parameters is unnecessary.

[0064] The above describes three merging methods: the first complexity, the second complexity, and the third complexity. The fourth merging method is the merging of video streams using complete pixel decoding and pixel re-encoding. The fourth merging method is applied when all three merging methods are not applicable.

[0065] Using Figures 14 and 15 to consider another embodiment of the present invention, the process for identifying the level of the merged result is described below. Identifying the level of the merged result means placing information in a sub-bitstream and determining how significant the contribution of that sub-bitstream is to the level limit of the merged bitstream that incorporates it.

[0066] Figure 14 shows encoder 2(2b), which includes an encoder core with coding parameter determination 14, a merge identifier provider (not shown), and a granularity function provider 8.

[0067] Generally, when sub-bitstreams are to be merged into a combined bitstream, instructions on how each individual sub-bitstream contributes to the codec system's level-specific limitations that the potentially merged bitstream must adhere to are essential to ensure the creation of a legally combined bitstream. Traditionally, codec level granularity has been quite coarse, e.g., identifying primary resolutions such as 720p, 1080p, or 4K, but instructions for merge level limitations require much finer granularity. This conventional level instruction granularity is insufficient to represent the contribution of individual sub-bitstreams to the merged bitstream. Assuming the number of tiles to be merged is unknown beforehand, a reasonable trade-off between flexibility and bitrate cost must be found, which generally goes far beyond the granularity of conventional level limitations. One exemplary use case is a 360-degree video stream where the service provider needs the freedom to choose from different tile structures, such as 12, 24, or 96 tiles per 360-degree video, with each tile stream contributing 1 / 12, 1 / 24, or 1 / 96 of the overall level limit, such as 8K, assuming an equal rate distribution. Furthermore, arbitrary fineness of granularity may be required, for example, assuming a non-uniform rate distribution between tiles to achieve uniform quality across the video plane.

[0068] Such signaling may be, for example, the signaled ratio and / or additionally the signaled level percentage. For example, in a conference scenario with four participants, each participant would transmit a legal level 3 bitstream, which also includes level information contained in coarse-grained functional information, and the signal indicates that the transmitted bitstream will comply with 1 / 3 and / or 33% of the level 5 limit, i.e., this information may be contained in fine-grained functional information. A receiver of multiple such streams, i.e., a video merger 50(50b) as shown in Figure 15, can know, for example, that three such bitstreams may thus be merged into a single combined bitstream that complies with level 5.

[0069] The granularity function information may have ratio and / or percentage indications as a vector of values, where each dimension relates to another aspect of the codec level limit, such as the maximum allowable luma samples per second, maximum image size, bitrate, buffer occupancy, number of tiles, etc. In addition, the ratio and / or percentage refer to the general codec level of the video bitstream.

[0070] While some embodiments are described in the context of apparatus or systems, it is clear that these embodiments also represent descriptions of corresponding methods, where a block or device corresponds to a method step or a feature of a method step. Similarly, embodiments described in the context of method steps also represent descriptions of blocks, items, or features of corresponding apparatus and / or systems. Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0071] The data stream of the present invention can be stored in a digital storage medium or transmitted via a transmission medium such as a wireless transmission medium like the Internet or a wired transmission medium.

[0072] Depending on the specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. The embodiments can be implemented using a digital storage medium containing electronically readable control signals, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, which cooperates (or can cooperate) with a programmable computer system to perform each method. Thus, the digital storage medium may be computer-readable.

[0073] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system so that one of the methods described herein is performed.

[0074] Generally, embodiments of the present invention can be implemented as a computer program product having program code, which operates to perform one of the methods when the computer program product runs on a computer. The program code may be stored, for example, in a machine-readable carrier.

[0075] Other embodiments include a computer program stored in a machine-readable carrier for performing one of the methods described herein.

[0076] In other words, one embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is running on a computer.

[0077] Another embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) on which a computer program for performing one of the methods described herein is recorded. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-temporary.

[0078] Another embodiment of the method of the present invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0079] Another embodiment includes processing means configured or adapted to perform one of the methods described herein, such as a computer or a programmable logic device.

[0080] Another embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0081] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0082] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field-programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, these methods are preferably performed by any hardware device.

[0083] The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0084] The apparatus described herein, or any component of the apparatus described herein, may be implemented, at least in part, as hardware and / or software.

[0085] The methods described herein may be performed using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0086] The methods described herein, or any component thereof, may be implemented at least in part by hardware and / or software.

[0087] The embodiments described above are merely illustrative of the principles of the present invention. Those skilled in the art will understand that modifications and variations of the configurations and details described herein will become apparent. Therefore, it is intended that the invention be limited only by the appended claims and not by the specific details presented herein as descriptions and explanations of embodiments.

Claims

1. A method for providing an encoded video representation, To provide a merge identifier that indicates a method for merging two or more video streams in a bitstream, In the bitstream, level information is provided that indicates the compatibility of the video stream with a video decoder having one of a plurality of predetermined functional levels. The bitstream is provided with fraction level information that references one of the plurality of predetermined functional levels and indicates a fraction of one of the plurality of predetermined functional levels, wherein the fraction of one of the plurality of predetermined functional levels is specified as a ratio value. The merge identifier is the concatenation of multiple coding parameters, or the hash value of the concatenation of multiple coding parameters. The merge identifier, the level information, and the fractional level information are available to the video decoder for merging the two or more video streams. method.

2. The aforementioned fractional level information is, The fraction of the maximum allowable amount of luma sample per second, Fraction of maximum image size, Percentage of maximum bitrate, The fraction of buffer occupancy, and / or, Percentage of the maximum number of tiles, The method according to claim 1, wherein one or more of the following are represented.

3. A method for decoding a provided video representation, In a bitstream, receive a merge identifier that indicates a method for merging two or more video streams, In the bitstream, level information is received that indicates compatibility of the video stream with a functional level among a plurality of predetermined functional levels. The bitstream receives fraction level information that references one of the plurality of predetermined functional levels and indicates a fraction of one of the plurality of predetermined functional levels, wherein the fraction of one of the plurality of predetermined functional levels is specified as a ratio value. This includes merging the two or more video streams according to the merge identifier, the level information, and the fractional level information, A method wherein the merge identifier is encoded within the bitstream as a concatenation of multiple encoding parameters, or as a hash value of a concatenation of multiple encoding parameters.

4. The aforementioned fractional level information is, The fraction of the maximum allowable amount of luma sample per second, Fraction of maximum image size, Percentage of maximum bitrate, The fraction of buffer occupancy, and / or, Percentage of the maximum number of tiles, The method according to claim 3, wherein one or more of the following are shown.

5. A video encoder for providing an encoded video representation, A video encoder comprising at least one processor including a processing circuit configured to perform the method according to claim 1 or 2.

6. A video decoder for decoding a provided video representation, A video decoder comprising at least one processor including a processing circuit configured to perform the method described in claim 3 or 4.

7. A non-temporary storage medium storing a computer program for performing the method described in any one of claims 1 to 4.

Images