Use of access unit delimiters and adaptive parameter sets

Abstract

To provide a decoder and a method for deriving a necessary parameter from an access unit.SOLUTION: A video decoder 20 comprises an in-loop filter 90 which filters a reconfiguration version of a decoded picture, and a parameterizing unit. The parameterizing unit reads in-loop filter control information for parameterizing the in-loop filter from a parameter set located in an access unit of a decoded picture following a video encoding unit along the sequence of a data stream 14 and / or a part of a video encoding unit following data included in a video encoding unit carrying block-based prediction parameter data and predicted residue data along the sequence of the data stream, and also parameterizes the in-loop filter so as to filter the reconfiguration version of the encoded picture by a method corresponding to the in-loop filter control information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This application relates to the use of access delimiters and adaptive parameter sets for signaling encoding parameters.

Background Art

[0002] Modern video encoding standards utilize in-loop filters such as an Adaptive Loop Filter (ALF), Sample Adaptive Offset (SAO), and a Deblocking Filter.

[0003] In-loop filters are placed within the decoder loop of an encoder. During all video encoding stages, and particularly in the lossy compression performed at the quantization stage, the subjective quality of a video sequence may be reduced as a result of the appearance of blocking, ringing, or blurred artifacts. To remove these artifacts and improve the subjective and objective quality of the reconstructed sequence, a set of in-loop filters is used. The in-loop filter within the encoder estimates the optimal filter parameters that most improve the objective quality of a frame. These parameters are then sent to the decoder, and the in-loop filter of the decoder uses these parameters to optimally filter the reconstructed frame and achieve the same quality improvement as achieved for the reconstructed frame within the encoder.

[0004] The Deblocking Filter aims to remove blocking artifacts that appear at the edges of Coding Units (CUs), specifically Prediction Units (PUs) and Transform Units (TUs), as a result of using a block structure in the processing of all stages of the encoder.

[0005] SAO filters aim to reduce undesirable visible artifacts such as ringing. The key idea of ​​SAO is to first categorize the reconstructed samples into different categories, obtain an offset for each category, and then reduce sample distortion by adding the offset to each sample in that category.

[0006] The key idea behind ALF is to minimize the mean squared error between the original and decoded pixels using Wiener-based adaptive filter coefficients. ALF can be seen as a tool placed in the final processing stage of each picture, attempting to capture and correct artifacts from previous stages. Appropriate filter coefficients are determined by the encoder and explicitly signaled to the decoder. That is, ALF requires a set of parameters to be sent to the decoder, i.e., appropriate filter coefficients. These parameters are transmitted in a high-level syntax structure, such as an Adaptive Parameter Set (APS). The APS is a set of parameters sent in a bitstream before the Video Coding Layer (VCL) NAL (Network Abstraction Layer) unit, i.e., before slicing the picture. ALF is applied to the complete, reconstructed picture. Also, in the encoder, ALF estimation is one of the final steps in the coding process.

[0007] In low-latency environments, this poses a problem because the encoder wants to start transmitting the processed portion of the picture as soon as possible, especially before the picture encoding process is complete. ALF cannot be optimally used in these environments because the APS, which has estimated filter parameters for the encoded picture, must be transmitted before the first slice of the picture.

[0008] Furthermore, a set of NAL units in a specified format is called an access unit, or AU, and decoding each AU yields one decoded picture. Each AU contains a set of VCL NAL units that together constitute the primary coded picture. An access unit delimiter (AUD) may also be prefixed to the AU to help identify its starting position.

[0009] AUD is used to isolate AU within a bitstream. It can optionally include information about subsequent pictures, such as the permitted slice types (I, P, B).

[0010] In Versatile Video Coding (VVC), several different parameter sets can be referenced in a picture, namely the video parameter set (VPS), decoder parameter set (DPS), sequence parameter set (SPS), multiple picture parameter set (PPS), different type adaptive parameter set (APS), and one or more others. All parameter sets must be present for the picture to be decoded by the decoder.

[0011] Different slices of a picture may refer to different PPS and APS. Therefore, it can be difficult for the decoder to determine if all the necessary parameter sets are available, because it would need to parse all the slice headers of the picture and the decode to see which parameters are referenced. [Overview of the Initiative] [Problems that the invention aims to solve]

[0012] The subject matter of this invention is to provide a decoder that derives necessary parameters from an access unit. [Means for solving the problem]

[0013] This objective is achieved by the subject matter of the claims of the present application.

[0014] According to embodiments of the present invention, the video decoder includes a decoding core (94) configured to reconstruct a decoded picture, e.g., the currently decoded picture or a subsequent decoded picture, from one or more video coding units (100), e.g., VCL NAL units, within the access unit AU of the video data stream, using motion-compensated prediction and transform-based residual decoding, in order to obtain a reconstructed version (46a) of the decoded picture; an in-loop filter (90) configured to filter the reconstructed version (46b) of the decoded picture to obtain a version (46b) of the decoded picture to be inserted into the decoded picture buffer DPB (92) of the video decoder, e.g., an ALF; and a parameterizer, e.g., one or more parameter sets (102, 104) located within the access unit AU of the decoded picture, e.g., an ALF per CTU APS, following one or more video coding units (100) along the data stream order, e.g., individually with or following all of them, e.g., VCL NALUs; The system comprises a parameterizer configured to parameterize an in-loop filter by, for example, reading the ALF for each ALF coefficient (or parameter) and CTU (encoding tree unit) flag, and parameterizing the in-loop filter to filter the reconstructed version of the decoded picture in a manner corresponding to the in-loop filter control information, i.e., the in-loop filter control information is derived for each of the video encoding units (100), and therefore it is possible to start decoding before receiving all video encoding units of the picture. Thus, decoding delay is reduced in low-latency environments.

[0015] According to embodiments of the present invention, the in-loop filter control information includes one or more filter coefficients for parameterizing the in-loop filter with respect to its transfer function. That is, ALF is, for example, an FIR (finite impulse response) or IIR (infinite impulse response) filter and filter coefficients FIR or IIR coefficients that control the transfer function of the filter.

[0016] According to embodiments of the present invention, the in-loop filter control information includes spatially selective in-loop filter control information for causing the in-loop filter to spatially change the filtering of a decoded picture, for example, the picture currently being decoded or a reconfigured version of a subsequent decoded picture.

[0017] According to embodiments of the present invention, each video coding unit (100) is arithmetic coded sequentially along the data stream order through the data to the end of the portion (106), i.e., up to the ALF. A predetermined set of one or more parameter sets (102) follows each of the one or more video coding units (100) in the data stream order and comprises one or more filter coefficients for parameterizing an in-loop filter with respect to the transfer function.

[0018] According to embodiments of the present invention, one or more parameter sets (104) include, for each of one or more video encoding units (100), a further predetermined set of parameters following each video encoding unit (100) in data stream order, and includes spatially selective in-loop filter control information for spatially varying the filtering of a decoded picture, e.g., a reconfigured version of the currently decoded picture or a subsequent decoded picture, by an in-loop filter within the portion of the picture encoded by each video encoding unit (100).

[0019] According to embodiments of the present invention, each of one or more video encoding units (100) includes a data section (108) of each video encoding unit (100) followed in data stream order by a filter information section (106) which includes spatially selective in-loop filter control information for spatially varying the filtering of a decoded picture, e.g., a reconfigured version of the currently decoded picture or a subsequent decoded picture, by an in-loop filter within the portion of the picture into which block-based prediction parameter data and prediction residual data are encoded in the data section of each video encoding unit (100).

[0020] According to embodiments of the present invention, the parameterizer may be configured to position the one or more parameter sets (102, 104) of the decoded picture, e.g., the current decoded picture or the subsequent decoded picture, e.g., ALF for each ALF APS and CTU APS, in the access unit (AU) in the case of a predetermined instruction in the video data stream assuming a first state, such as individually with or following all of the one or more video coding units (100) along the data stream sequence, e.g., VCL NALU, in the case of a predetermined instruction in the video data stream assuming a second state, such as the current decoded picture or the subsequent decoded picture, e.g., ALF for each ALF APS and CTU APS.

[0021] According to embodiments of the present invention, a portion (106) of one or more video coding units (100), for example, an ALF for each CTU data, is located along the data stream sequence following data (108) composed of one or more video coding units (100) that carry block-based predictive parameter data and predictive residual data, in the case of a predetermined instruction in the video data stream assuming a first state, and is located at a different position within one or more video coding units where the block-based predictive parameter data and predictive residual data are scattered, in the case of a predetermined instruction in the video data stream assuming a second state.

[0022] According to embodiments of the present invention, a video decoder is configured to read a predetermined instruction from one or more video encoding units (100). The predetermined instruction, assuming a first state, indicates one or more parameter sets by one or more identifiers, and, assuming a second state, indicates one or more different in-loop filter control information parameter settings. The video decoder is configured to respond to the predetermined instruction for each access unit so as to perform different localization of the video data stream for different access units when the predetermined instructions are different for different access units. A parameterizer is configured to reconstruct the decoded picture using the in-loop filter control information contained in the previously signaled access unit AU.

[0023] According to an embodiment of the present invention, when detecting the boundary of an access unit AU, the video decoder is configured to interpret a video coding unit that carries in-loop filter control information, such as ALF filter data, as not starting an access unit in the form of one or more parameter sets (102, 104), such as the suffix APS, and from there, for example, to ignore them in AU boundary detection and thereby detect that there is no AU boundary, and to interpret a video coding unit that carries in-loop filter control information, not in the form of one or more parameter sets (102, 104), such as the suffix APS, as starting an access unit from such a video coding unit, and to detect an AU boundary from such a video coding unit.

[0024] According to embodiments of the present invention, the video decoder is configured to decode video from a video data stream by decoding a decoded picture, for example, the picture currently being decoded or a subsequent decoded picture, in a parameterized manner using one or more predetermined encoding parameters from one or more video encoding units (100) in the access unit AU of the video data stream, derive predetermined encoding parameters (122) from a plurality of parameter sets (120) scattered in the video data stream, and read an identifier (200) from a predetermined unit (124) of the access unit AU that identifies a predetermined parameter set from a plurality of parameter sets containing the predetermined encoding parameters. That is, the presence or absence of encoding parameters is indicated by the identifier, and therefore, it is efficiently recognized which parameter sets can be derived from the received video encoding unit. Furthermore, since the identifier is contained in a predetermined unit of the AU, it is easy to include different parameter sets for different video encoding units.

[0025] According to an embodiment of the present application, a predetermined unit of an AU includes a flag (204) indicating whether an identifier (200) exists within the predetermined unit. That is, it is possible to indicate with a flag which identifier is included in a predetermined unit of an AU, and for example, it is possible to indicate a delimiter of an access unit.

[0026] According to an embodiment of the present application, a plurality of parameter sets (120) are of different hierarchical levels, and one or more video encoding units include an identifier referring to a first predetermined parameter set (126) within one or more first predetermined hierarchical levels, for example, in a slice header. The first predetermined parameter set (126) within one or more first predetermined hierarchical levels includes an identifier referring to a second predetermined parameter set (128) within one or more second predetermined hierarchical levels. The first and second predetermined parameter sets are included by a predetermined parameter set (122). An identifier read from a predetermined unit (124) of an access unit AU identifies all predetermined parameter sets directly or indirectly referred to by one or more video encoding units of the access unit. Thereby, when all predetermined parameter sets identified by the identifier are available, the access unit is decodable.

[0027] According to an embodiment of the present application, a predetermined unit of an AU includes a flag (204) indicating (205) whether a predetermined identifier of a specific predetermined parameter set (126b), for example, a specific APS, exists within the predetermined unit (124), or whether a predetermined identifier referring to a specific predetermined parameter set (126b) exists within one or more video encoding units (100).

[0028] According to an embodiment of the present application, the first predetermined parameter set (126) includes a third predetermined parameter set (126a) referred to by an identifier in one or more video encoding units (100), and a fourth predetermined parameter set (126b) referred to by an identifier (200) present in a predetermined unit (124), but not referred to by any identifier in one or more video encoding units (100) or by any predetermined parameter set.

[0029] According to an embodiment of the present application, a predetermined unit of an AU includes one or more identifiers of one or more adaptation parameter sets, APS, one or more identifiers of one or more picture parameter sets, PPS, an identifier of a video parameter set, VPS, an identifier of a decoder parameter set, DPS, and one or more identifiers of one or more sequence parameter sets, SPS. The plurality of parameter sets includes a video parameter set, VPS, a decoder parameter set, DPS, a sequence parameter set, SPS, one or more picture parameter sets, PPS, and one or more adaptation parameter sets, APS.

[0030] According to embodiments of the present invention, a video decoder is configured to decode video from a video data stream by decoding a picture from one or more video encoding units (100) of an access unit (AU) of the video data stream, and to read one or more parameters from an access unit delimiter AUD positioned in the video data stream to form the beginning of an access unit AU, the one or more parameters controlling whether separate access units are defined in the video data stream for a picture relating to a different layer that is momentarily related to one of the video data streams, or whether a picture relating to a different layer that is momentarily related to one of the video data streams is encoded in one of the access units (300), and / or, if the video encoding type indication includes different video encoding units within one access unit, the encoding type of the video encoding units is included in an access unit that is assigned within a video encoding unit within one access unit (302), and / or, the access unit indicates a picture that is not referenced by other pictures (304), and / or, the picture that is not output (306). In other words, since the parameters necessary to decode the picture are indicated by the AUD, decoding of slices of the picture contained in the AU can begin before obtaining the entire parameter set necessary to decode the complete picture. That is, the required parameter set for each slice can be efficiently indicated by the AUD, thereby improving the decoding speed.

[0031] According to embodiments of the present invention, one or more parameters form a deviation from the parameters defined by the preceding AUD. The AUD includes an indication of whether the parameters defined by the preceding AUD should be adopted. The AUD includes an indication of whether one or more parameters apply to all layers of the video data stream or to only one layer thereof. The video coding type of the video coding unit is indicated by describing the random access characteristics of multiple pictures.

[0032] According to embodiments of the present invention, the video decoder is configured to decode video from a video data stream by decoding a picture from one or more video encoding units (100) of an access unit (AU) of the video data stream, and to read one or more parameters (308) from an access unit delimiter (AUD) positioned in the data stream to form the start of an access unit (AU), the one or more parameters indicating the characteristics of the access unit, and indicating whether the characteristics apply to all layers of the video data stream or to only one layer thereof.

[0033] According to embodiments of the present invention, in order to obtain a reconstructed version of a decoded picture, the following steps are taken: reconstructing the decoded picture using motion-compensated prediction and transform-based residual decoding from one or more video coding units (100) in the access unit AU of the video data stream, e.g., a VCL NAL unit; filtering the reconstructed version of the decoded picture using an in-loop filter to obtain a version of the decoded picture to be inserted into the decoded picture buffer DPB of the video decoder; filtering the reconstructed version of the decoded picture using an in-loop filter to obtain a version of the decoded picture to be inserted into the decoded picture buffer DPB of the video decoder; and parameterizing the in-loop filter, wherein the parameterizing of the in-loop filter is performed by one or more parameter sets (102, 104) located in the access unit AU of the decoded picture following one or more video coding units (100) along the data stream order, e.g., ALF APS and CTU A method is provided which involves reading in-loop filter control information for parameterizing an in-loop filter from the ALF per APS and / or from a portion (106) of one or more video coding units (100) that follows the data (108) contained in the ALF per APS and / or the ALF per CTU data of one or more video coding units (100) that carry block-based predict parameter data and predict residual data along the data stream order, and filtering the reconstructed version of the decoded picture in a manner corresponding to the in-loop filter control information.

[0034] According to embodiments of the present invention, a method is provided for decoding video from a video data stream by decoding a decoded picture, for example, the picture currently being decoded or a subsequent decoded picture, in a parameterized manner using one or more predetermined encoding parameters from one or more video encoding units (100) in an access unit AU of the video data stream; deriving predetermined encoding parameters from a plurality of parameter sets scattered in the video data stream; and reading an identifier (200) from a predetermined unit of the access unit AU that identifies a predetermined parameter set from a plurality of parameter sets containing the predetermined encoding parameters.

[0035] According to embodiments of the present invention, a method is provided for recovering video from a video data stream by decoding a picture from one or more video encoding units (100) of an access unit AU of the video data stream, and reading one or more parameters from an access unit delimiter AUD positioned in the video data stream to form the beginning of an access unit AU, wherein one or more parameters control whether separate access units are defined in the video data stream for a picture relating to a different layer that is related to one instantaneous but related to one of the video data streams, or whether a picture relating to a different layer that is related to one instantaneous but related to one of the access units is encoded in one of the access units (300); if the video encoding type indication includes different video encoding units within one access unit, the encoding type of the video encoding units is included in an access unit that is assigned within a video encoding unit within one access unit (302); and / or the access unit indicates a picture that is not referenced by other pictures (304); and / or the access unit indicates a picture that is not output (306).

[0036] According to embodiments of the present invention, a method is provided for decoding video from a video data stream by decoding a picture from one or more video encoding units (100) of an access unit (AU) of the video data stream, and for reading one or more parameters (308) from an access unit delimiter (AUD) positioned in the data stream to form the start of an access unit (AU), wherein one or more parameters indicate a characteristic of the access unit (AU), and the characteristic indicates whether it applies to all layers of the video data stream or to only one layer thereof. [Brief explanation of the drawing]

[0037] Preferred embodiments of the present invention will be described below with reference to the drawings. [Figure 1] As an example of a video encoder capable of encoding a hierarchical video data stream according to the embodiment of the present invention, a block diagram of a device for predictive encoding video is shown. [Figure 2] As an example of a video decoder capable of decoding a hierarchical video data stream according to the embodiment of the present invention, a block diagram of a device for predictive video decoding, which is compatible with the device shown in Figure 1, is shown. [Figure 3] These are schematic diagrams illustrating an example of the relationship between the predicted residual signal, the predicted signal, and the reconstructed signal, in order to show the possibility of setting subdivisions for the definition of the predicted signal and the handling of the predicted residual signal. [Figure 4] A schematic diagram of an example of a conventional encoding process is shown. [Figure 5] A schematic diagram of an example of an encoding process for signaling a picture to a decoder according to an embodiment of the present invention is shown. [Figure 6a] A schematic diagram of an example of an access unit AU according to an embodiment of the present invention is shown. [Figure 6b] A schematic diagram of an example of an access unit AU according to an embodiment of the present invention is shown. [Figure 7] A schematic diagram of another example of an AU according to an embodiment of the present invention is shown. [Figure 8] This figure shows an example of information regarding the parameter set identifier ID included in the access unit delimiter AUD according to an embodiment of the present invention. [Figure 9] This figure shows an example of information regarding a predetermined set of parameters included in the AUD according to an embodiment of the present invention. [Figure 10] This figure shows an example of information regarding a predetermined set of parameters included in the AUD according to an embodiment of the present invention. [Figure 11a] A schematic diagram illustrating an example of the relationship between predetermined coding parameter sets according to an embodiment of the present invention is shown. [Figure 11b] A schematic diagram illustrating an example of which part of the AU represents which identifier of which parameter set, according to the embodiment of this application, is shown. [Figure 11c] A schematic diagram of another example showing which part of the AU represents which identifier of which parameter set, according to the embodiment of the present application, is shown. [Figure 12a] This figure shows an example of parameters included in AUD according to an embodiment of the present invention. [Figure 12b] This figure shows an example of parameters included in AUD according to an embodiment of the present invention. [Figure 13] This figure shows another example of parameters included in AUD according to an embodiment of the present invention. [Modes for carrying out the invention]

[0038] In the following description, elements that are the same or equivalent, or elements that have the same or equivalent function, are indicated by the same or equivalent reference number.

[0039] The following description includes several details to provide a more complete description of the embodiments of the present application. However, it will be apparent to those skilled in the art that the embodiments of the present application may be carried out without these specific details. In other examples, well-known structures and apparatus are shown in block diagram form rather than in detail, so as not to obscure the embodiments of the present application. Also, unless otherwise specified, features of the different embodiments described below may be combined.

[0040] Introduction It should be noted that the individual embodiments described herein may be used individually or in combination. Therefore, further details may be added to each of the individual embodiments without adding further detail to any other embodiment.

[0041] Furthermore, note that this disclosure explicitly or implicitly describes features that can be used in a video decoder (a device for providing a decoded representation of a video signal based on an encoded representation). Therefore, any of the features described herein can be used in the context of a video decoder.

[0042] Furthermore, the features and functionalities disclosed herein in relation to the methods may also be used in apparatuses (configured to perform such functionalities). Additionally, any features and functionalities disclosed herein in relation to apparatuses may also be used in corresponding methods. In other words, the methods disclosed herein may be supplemented by any of the features and functionalities described in relation to apparatuses.

[0043] The following description of the figures begins with a description of a block-based predictive codec video encoder and video decoder for encoding video pictures, in order to form an example of an encoding framework that can incorporate embodiments for a layered video data stream codec. The video encoder and video decoder will be described in relation to Figures 1 to 3. Hereinafter, embodiments of the concept of the layered video data stream codec of the present application will be presented, along with a description of how such concepts can be incorporated into the video encoder and video decoder of Figures 1 and 2, respectively. However, the embodiments described below can also be used to form video encoders and video decoders that do not operate according to the encoding framework that forms the basis of the video encoder and video decoder of Figures 1 and 2.

[0044] Figure 1 shows a block diagram of a device for predictive coding video, as an example of a video decoder capable of performing interpredictive block motion compensation prediction according to an embodiment of the present invention. Specifically, Figure 1 shows a device that predictively codes a video 11 consisting of a series of pictures 12 into a data stream 14. For this purpose, block-based predictive coding is used. Furthermore, transform-based residual coding is used exemplary. The device, i.e., the encoder, is indicated by reference numeral 10.

[0045] Figure 2 shows a block diagram of a device for predictive decoding video, as an example of a video decoder capable of performing interpredictive block motion compensation prediction according to an embodiment of the present invention. Specifically, Figure 2 shows a device 20 configured to predictively decode, here again exemplary, a corresponding decoder 20, i.e., video 11' consisting of picture 12' in a picture block from a data stream 14, using transform-based residual decoding. The apostrophe is used to indicate that picture 12' and video 11' reconstructed by decoder 20 deviate, respectively, from the picture 12 originally encoded by the device 10 with respect to encoding loss introduced by the quantization of the predictive residual signal. While Figures 1 and 2 use transform-based predictive residual coding exemplary, embodiments of the present invention are not limited to this type of predictive residual coding. This also applies to other details described with respect to Figures 1 and 2, as outlined below.

[0046] The encoder 10 is configured to apply a spatial-to-spectral transformation to the predicted residual signal and encode the resulting predicted residual signal into the data stream 14. Similarly, the decoder 20 is configured to decode the predicted residual signal from the data stream 14 and subject the resulting predicted residual signal to a spectral-to-spatial transformation.

[0047] Internally, the encoder 10 may include a prediction residual signal formatter 22 that generates a prediction residual 24 to measure the deviation of the prediction signal 26 from the original signal, i.e., video 11 or the current picture 12. The prediction residual signal formatter 22 may be, for example, a subtractor that subtracts the prediction signal from the original signal, i.e., the current picture 12. The encoder 10 then further includes a transformer 28 that applies a spatial spectral transform to the prediction residual signal 24 and then quantizes it by a quantizer 32 configured by the encoder 10 to obtain a spectral-domain prediction residual signal 24'. The thus quantized prediction residual signal 24'' is encoded into the data stream 14. For this purpose, the encoder 10 may optionally include an entropy coder 34 that entropy encodes the prediction residual signal so that it is transformed and quantized into the data stream 14. Based on the data stream 14 and the prediction residual signal 24'' decoded from the data stream 14, the prediction stage 36 of the encoder 10 generates the prediction residual 26. For this purpose, the prediction stage 36 may internally include an inverse quantizer 38 that inversely quantizes the prediction residual signal 24'' to obtain a spectral domain prediction residual signal 24'''' corresponding to the signal 24' excluding quantization losses, and then includes an inverse converter 40 that applies an inverse transform, i.e., a spectral space transform, to the latter prediction residual signal 24'' to obtain a prediction residual signal 24'''' corresponding to the original prediction residual signal 24 excluding quantization losses. The combiner 42 of the prediction stage 36 then recombines the prediction signal 26 and the prediction residual signal 24'''' by adding them together to obtain a reconstructed signal 46a, i.e., a reconstruction (reconstructed version) of the original signal 12. The reconstructed signal 46a may correspond to signal 12'.

[0048] The in-loop filter 90 filters the reconstructed signal 46a to obtain the decoded signal 46b of a version of the decoded picture, for example, the currently decoded picture or a subsequent decoded picture, and inserts it into the decoded picture buffer DPB92.

[0049] Next, the prediction module 44 in the prediction stage 36 generates a prediction signal 26 based on signal 46b, for example, by using spatial prediction, i.e., intra-prediction, and / or temporal prediction, i.e., inter-prediction. Further details on this are described below.

[0050] The decoder 20 includes a decoding core 94 which includes an entropy decoder 50, an inverse quantizer 52, an inverse transformer 54, a combiner 56, a prediction module 58, an in-loop filter 90, and a DPB 94.

[0051] Similarly, the decoder 20 may correspond to the prediction stage 36 and be internally composed of interconnected components in a corresponding manner. In particular, the entropy decoder 50 of the decoder 20 can entropy decode the spectral domain prediction residual signal 24'' quantized from the data stream, where the inverse quantizer 52, inverse transformer 54, combiner 56 and prediction module 58 are interconnected and cooperate with respect to the module of the prediction stage 36 in the manner described above to recover a signal reconstructed based on the prediction residual signal 24'', thereby, as shown in Figure 3, the output of the combiner 56 yields the reconstructed signal, i.e., video 11' or its current picture 12'.

[0052] Although not explicitly stated above, it is readily apparent that encoder 10 can set several coding parameters, including, for example, prediction modes and motion parameters, according to some optimization scheme, such as a method for optimizing coding cost and / or using some kind of rate control, i.e., certain rate and distortion-related criteria. As will be described in more detail below, encoder 10 and decoder 20 and corresponding modules 44, 58 support different prediction modes, such as intra-coding mode and inter-coding mode, which form a set or pool of primitive prediction modes, respectively, in which the prediction of picture blocks is configured in a manner described in more detail below. The granularity at which the encoder and decoder switch between these prediction synthesis corresponds to the subdivision of picture 12 and 12' into their respective blocks. Some of these blocks may be simply intra-coded blocks, some may be simply inter-coded blocks, and optionally, further blocks may be acquired using both intra-coding and inter-coding, but note that details will be described below. According to the intra-coding mode, the prediction signal for a block is acquired based on the spatial, already coded / decoded neighborhood of each block. There may be several intra-coded submodes, among which some intra-prediction parameters may exist. There may also be directional or angular intra-coded submodes, in which the prediction signal for each block is filled in by extrapolating neighboring sample values ​​to each block along a certain direction specified for each directional intra-coded submode.The intra-coding submode may also comprise one or more further submodes, such as a DC coding mode in which the predicted signal for each block assigns DC values ​​to all samples within each block, and / or a planar intra-coding mode in which the predicted signal for each block is approximated or determined to be a spatial distribution of sample values ​​described by a two-dimensional linear function over the sample positions of each block, and the slope and offset of a plane defined by the two-dimensional linear function are derived based on adjacent samples. In comparison, according to the inter-prediction mode, a predicted signal for a block can be obtained, for example, by predicting the inside of the block in time. For the parameterization of the inter-prediction mode, a motion vector may be signaled in the data stream, and the motion vector indicates the spatial displacement of the previously encoded portion of the video 11 from which the previously encoded / decoded picture is sampled in order to obtain a predicted signal for each block. This means that, in addition to the residual signal coding constituted by the data stream 14, such as the entropy coding transformation coefficient level representing the quantized spectral domain predicted residual signal 24'', the data stream 14 may also be coded to prediction-related parameters for assigning to block prediction modes, prediction parameters for assigned prediction modes such as motion parameters for inter-prediction modes, and optionally, as will be described in more detail later, further parameters that control the construction of the final predicted signal for the blocks using the assigned prediction modes and prediction parameters. Additionally, the data stream may include parameters that control and signal the subdivision of picture 12 and 12' to blocks, respectively. The decoder 20 uses these parameters to subdivide the picture in the same way as the encoder did, assign the same prediction modes and parameters to the blocks, and make the same predictions to obtain the same predicted signals.

[0053] Figure 3 is a schematic diagram showing an example of the relationship between the predicted residual signal, the predicted signal, and the reconstructed signal, illustrating possibilities such as the setting of subdivisions defining the predicted signal and the handling of the predicted residual signal. Specifically, Figure 3 shows the relationship between the reconstructed signal, i.e., the reconstructed picture 12', on the one hand, with the predicted residual signal 24'''' which has been signaled into the data stream, and on the other hand, with the predicted signal 26. As already mentioned above, additional combinations are also possible. The predicted signal 26 is merely one example, but it is a subdivision of the picture region into blocks 80 of various sizes. The subdivision may be any subdivision that regularly divides the picture region into rows and columns of blocks, or it may be a multi-tree subdivision that subdivides the picture 12 into leaf blocks of various sizes, such as a quad-tree subdivision, or it may be a mixture of these, in which the picture region is first subdivided into rows and columns of a tree root block and then further subdivided according to a recursive multi-tree subdivision to become blocks 80.

[0054] The following describes various aspects of the present invention.

[0055] Suffix - APS According to one aspect of the present invention, the encoder allows the transmission of a portion of a picture (e.g., a slice) before the encoding process of the entire picture is completed, while still using slices. This is achieved by allowing the Adaptation Parameter Set (APS) to be transmitted after the encoded slice of the picture, moving according to the Coding Tree Unit (ALF) parameters behind the actual slice data.

[0056] Figure 4 shows a diagram of a state-of-the-art encoder. First, the entire picture is encoded (intra prediction, motion estimation, residual coding, etc.), and then the ALF estimation process begins. The ALF filter coefficients are written to the APS, and then the slice data can be written, including the ALF for each CTU parameter (with other parameters scattered throughout). The picture can only be transmitted after the ALF encoding is complete.

[0057] Figure 5 shows an example of a low-latency encoder envisioned by the present invention. Slices can be sent before the picture encoding process is completed. In particular, the APS carrying the coefficients is moved to the end of the slice data (in VCL NAL units).

[0058] In this process, the encoder collects the estimated ALF parameters (filter coefficients, filter control information) and can then send out the encoded slice of the picture first, while collecting the APS containing the ALF parameters after the encoded picture. As soon as the slice of the picture arrives, the decoder can begin syntax analysis and decoding of the slice of the picture. Since the ALF is one of the last decoding steps, the ALF parameters can arrive after the encoded picture, which is applied after the other decoding steps.

[0059] The present invention includes the following embodiments. - An APS type indicating that the APS belongs to the previous picture. - The APS type is encoded with syntax elements. Embodiments are as follows: ○ A new NAL unit type (e.g., suffix APS) encoded in the NAL unit header. 〇A new APS type encoded with APS (e.g., suffix ALF APS). A flag indicating whether APS is applied to the previous picture or the subsequent picture. - A modification of the decryption process that simply assumes the initiation of a new access unit AU by signaled prefix APS, rather than suffix APS. Alternatively, the APS type (prefix or suffix) can be determined by the surrounding access unit delimiters (AUDs) and the relative position within the bitstream to the encoded slice of the picture. If APS is located between the last VCL NAL unit and AUD in a picture, it will be applied to the preceding picture. If APS is located between AUD and the first VCL NAL unit of the picture, it will apply to the next picture. -In this case, the decoder only needs to determine the start of a new access unit based on the position of the AUD. -In one embodiment, the slice header uses a suffix APS instead of a prefix APS, and as a result, the dependency is resolved at the end of the picture decoding process, for example, by the following equation: Depending on the prefix or suffix characteristics, the APS identifier is signaled at its position within the slice header. 〇 Signals a flag indicating that the referenced APS follows a slice coded in bitstream order.

[0060] Typically, not only is the derivation of ALF parameters (filter coefficients) performed towards the end of the coding process (based on reconstructed sample values), but further ALF control information (information about whether and how a coding tree unit CTU is filtered) is also derived at this stage. The ALF control information is carried in several syntax elements for each coding_tree_unit within the slice payload, scattered with block partitions (e.g., as shown in Figure 6b), transformation coefficients, and so on. For example, the following syntax elements may be present: 〇alf_ctb_flag: Specifies whether to apply the adaptive loop filter to the coded tree block CTB. 〇alf_ctb_use_first_APS_flag: Specifies whether to use the filter information of the APS where adaptive_parameter_set_id is equal to slice_alf_APS_id_luma[0]. 〇alf_use_APS_flag: Specifies whether to apply the filter set from APS to Luma CTB. Other.

[0061] All of this ALF control information depends on the derivation of the ALF filter parameters towards the end of the picture encoding process.

[0062] In one embodiment, ALF control information is signaled in a separate loop across the CTU of the last slice of each slice payload so that the encoder can finalize the initial part of the slice payload (e.g., conversion coefficients, block structure) before ALF is performed. This embodiment is shown in Figures 6a and 6b.

[0063] As shown in Figure 6a, a video coding unit (VCL NAL unit) 100 includes a slice header, slice data 108, and a portion (ALF per CUT APS) 106, and one or more parameter sets (ALF coefficients) 102 are signaled separately, i.e., as suffix APS (in a non-VCL-NAL unit). That is, each video coding unit 100 arithmetically encodes the data sequentially across the data stream to the end of the portion.

[0064] Figure 6b shows that slice data 108 is scattered throughout section 106. That is, ALF for each CUT APS is scattered throughout the block, and one or more parameter sets 102 are signaled separately as suffix APS.

[0065] In another embodiment, the slice header indicates that the ALF control information is not shown in the syntax elements within the encoded slice payload, i.e., within the CTU loop described above, but rather is contained within the suffix APS, i.e., within a separate loop spanning all CTUs within each suffix APS, referring to an APS of, for example, a suffix APS type.

[0066] In another embodiment, the slice header indicates that the ALF control information is not shown in the syntax elements within the encoded slice payload, i.e., within the CTU loop described above, but rather in a new type of suffix APS different from the suffix APS that carries the ALF coefficients, i.e., by referring to an APS of, for example, a suffix APS type, and is contained in a separate loop across all CTUs within each reference APS. The data for each CTU can optionally be CABAC encoded. This embodiment is shown in Figure 7.

[0067] As shown in Figure 7, the data units are signaled in data stream order by a video coding unit (VCL NAL unit) 100 containing a slice header and slice data 108, a parameter set 104 (Suffix ALF CTU-data APS: non-VCL NAL unit), a further video coding unit 100, a further parameter set 104, and a parameter set (filter control information) 102 (Suffix ALF coefficient APS: non-VCL NAL unit). That is, in reverse of the data stream shown in Figures 6a and 6b, the filter coefficients do not need to be signaled following all video coding units. In other words, the filter coefficients may be transmitted collectively for two or more video coding units 100 in bitstream order, or they may be further used by further video coding units following the filter coefficients in bitstream order.

[0068] In another embodiment, it is assumed that the slice header referencing the suffix APS and all CTUs has an adaptive loop filter to which default values ​​of filter parameters and ALF control information signaled by the suffix APS are applied.

[0069] Signaling of parameter set ID references in AUD Other aspects of the present invention, namely methods for facilitating access to a list of all parameter sets referenced within a picture, are described below.

[0070] According to this aspect of the present invention, the decoder can easily determine whether all necessary parameter sets are available before starting decoding. - A list of all parameter sets used is included in the high-level syntax structure. -The list consists of the following: 〇One VPS (video parameter set) 〇One DPS (Decoder Parameter Set) One or more SPS (Sequence Parameter Sets) One or more PPS (Picture Parameter Sets) ○ One or more APS (Adaptation Parameter Sets) (ordered by APS type) -Optionally, one or more syntax elements may precede each list to indicate which parameter set type is a list (including the option to disable submission). - The syntax structure for holding information is included in the access unit delimiter (AUD).

[0071] An example of the syntax is shown in Figure 8, where a given unit, namely AUD, contains multiple identifiers 200. For example, the identifier for VPS is "aud_vps_id", the identifier for DPS is "aud_dps_id", the identifier for SPS is "aud_sps_id", the identifier for PPS is "aud_pps_id", and so on.

[0072] APS ID signaling within AUD only An APS is referenced by each slice of a picture. When combining bitstreams, it may be necessary to rewrite and / or combine different APSs.

[0073] To avoid rewriting the slice header, the APS ID is signaled in the access unit delimiter rather than the slice header. Therefore, in case of change, it is not necessary to rewrite the slice header. Rewriting the access unit delimiter is a much easier operation.

[0074] An example of the syntax is shown in Figure 9.

[0075] In another embodiment, the APS ID is sent only in an AUD that is conditional on a different syntax element. If the syntax element indicates that the APS ID is not present in the AUD, then the APS ID is present in the slice header. An example of the syntax is shown in Figure 10. That is, as shown in Figure 10, the AUD includes flag 204, for example, syntax "aps_ids_in_aud_enabled_flag".

[0076] Figures 11a to 11c are schematic diagrams illustrating an example of the relationship between predetermined coding parameters and AU according to the embodiment shown in Figures 8 to 10.

[0077] As shown in Figure 11a, the multiple parameter sets 120 include one or more first predetermined parameter sets 126, which include, for example, APA and PPS, and one or more second predetermined parameter sets 128, which include, for example, SPS, DPS and VPS. The second predetermined parameter sets 128 belong to a higher hierarchical level than the first predetermined parameter sets 126. As shown in Figure 11a, the AU includes multiple slice data, for example, VCL0 to VCLn, and the first and second predetermined parameter sets are included in a predetermined parameter set 122, for example, the parameter set for VCL0.

[0078] Multiple parameter sets 120 are stored in the AU's AUD and signaled to the decoder.

[0079] As shown in Figure 11b, if flag 204 is included in AUD, flag 204 indicates whether a predetermined identifier of identifier 200 that refers to a specific predetermined set of parameters 126b exists in a predetermined unit 124, or whether a predetermined identifier that refers to a specific predetermined set of parameters 126b exists in one or more video encoding units 100 (205). That is, flag 204 indicates whether APS126b is in AUD or VLC (as indicated by the arrow in Figure 11b).

[0080] As shown in Figure 11c, the first predetermined parameter set 126 includes a third predetermined parameter set 126a, e.g., PPS, which is referenced by an identifier in one or more video coding units 100 (e.g., AU), and a fourth predetermined parameter set 126b, e.g., APS, which is referenced by an identifier 200 (shown in Figures 8 and 9) located in a predetermined unit 124 (e.g., AUD), but is not referenced by any of the identifiers in one or more video coding units 100 (AU), nor is it referenced by any of the predetermined parameter sets.

[0081] Signaling of access unit characteristics for AUD Currently, AUD indicates whether the following slices are of type I, B, or P. In most systems, this feature is not very useful because I-pictures do not necessarily mean that they have random access points. Prioritizing AUs for when they should be dropped can usually be done in other ways, such as parsing the temporal ID or parsing whether they are discardable pictures (not referenced by others).

[0082] Instead of indicating the picture type, it is possible to indicate the NAL unit type, as well as whether they are discardable pictures, etc. Furthermore, in the case of multi-layer structures, describing the characteristics may become more difficult. The random access properties of a picture are specified by the overall NAL unit type (e.g., IDR, CRA, etc.) used for all VCL NAL units within the access unit, rather than by what is specified in the NAL unit header. A picture within a layer can be discarded in one layer, but is not considered a colocated picture in another layer. A picture within a certain layer may be marked as having no output (pic_output_flag) in one layer, and not be a collocated picture in another layer.

[0083] Therefore, in the embodiment shown in Figure 12a, AUD indicates whether the information applies to a single layer or to all layers. In other words, the AUD flag refers to information about the AU characteristics, for example, as shown below.

[0084] "layer_specific_aud_flag" 300: Controls whether separate access units are defined in the video data stream for pictures that are instantaneously related to different layers of the video data stream, or whether pictures that are instantaneously related to different layers of the video data stream are encoded in one of the access units, and / or "nal_unit_type_present_flag" 302: In the case of a video encoding type instruction, it indicates the video encoding type of a video encoding unit within an access unit, which is assigned to a video encoding unit within an access unit, and the video encoding units contained within an access unit are different from each other, i.e., it indicates the NAL unit type by indicating the presence of a syntax element of the NAL unit type, and / or "discardable_flag" 304: The access unit indicates a picture that is not referenced by any other picture, and / or "pic_output_flag" 306: Indicates a picture that will not be output. In another embodiment, AUD may indicate that AUD is a dependent AUD. This means the following: ○ Inherits parameters from the previous AUD of the dependent layer, but adds layer-specific information, and / or, 〇The new AU service has not yet launched.

[0085] Figure 12b illustrates the instructions in AUD, namely, that the video encoding type of a video encoding unit is indicated by describing the random access characteristics of multiple pictures. That is, the syntax "random_access_info_present_flag" indicates the random access characteristics of pictures, as shown in Figure 12b as "all_pics_in_au_random_access_flag", and is specified by the overall NAL unit type (e.g., IDR, CRA, etc.) used for all VCL NAL units within the access unit, instead of being specified in the NAL unit header.

[0086] Figure 13 shows an example of syntax according to one embodiment of the present invention.

[0087] In this example, parameter set 308 indicates whether the information within the AUD applies to all layers, whether the AUD initiates a new global access unit, or whether it is only a "layer-access unit". In dependent AUDs, inheritance from the base layer AUD is indicated by "aud_inheritance_flag".

[0088] While some embodiments have been described in the context of apparatus, it is clear that these embodiments also represent descriptions of corresponding methods, where a block or apparatus corresponds to a method step or a feature of a method step. Similarly, embodiments described in the context of a method step also represent descriptions of corresponding blocks, items, or features of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware device, such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such a device.

[0089] The data stream of the present invention can be stored in a digital storage medium or transmitted to a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0090] Depending on the specific implementation requirements, embodiments of the present invention may be implemented in hardware or software. These implementations may be carried out using digital storage media, such as floppy disks, DVDs, Blu-ray discs, CDs, ROMs, PROMs, EPROMs, EEPROMs, or flash memory, which store electronically readable control signals that cooperate (or can cooperate) with a programmable computer system to perform the respective methods. Thus, the digital storage media may be computer-readable.

[0091] Some embodiments of the present invention include a data carrier having an electronically readable control signal that can cooperate with a programmable computer system so that one of the methods described herein is performed.

[0092] Generally, embodiments of the present invention can be implemented as a computer program product having program code, the program code operable to perform one of the methods when the computer program product is executed on a computer. The program code may be stored, for example, on a machine-readable carrier.

[0093] Other embodiments include a computer program stored in a machine-readable carrier for performing one of the methods described herein.

[0094] In other words, one embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is executed on a computer.

[0095] Accordingly, a further embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) having a computer program for performing one of the methods described herein recorded thereon. The data carrier, digital storage medium, or recorded medium is typically tangible and / or non-temporary.

[0096] Therefore, a further embodiment of the method of the present invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals may be configured to be transmitted, for example, over a data communication connection, such as the Internet.

[0097] Further embodiments include processing means, such as a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0098] Further embodiments include a computer on which a computer program for performing one of the methods described herein is installed.

[0099] Further embodiments of the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0100] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field-programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0101] The apparatus described herein may be implemented using hardware devices, or using a computer, or using a combination of hardware devices and a computer.

[0102] The apparatus described herein, or any component of the apparatus described herein, can be implemented at least partially in hardware and / or software.

Claims

1. A method for decoding video, Reading a filter coefficient located in the suffix unit corresponding to the first picture of the video stream, wherein the filter coefficient is for the second picture following the first picture, and the reading Reading a flag from the video data corresponding to the second picture of the video stream that indicates that an adaptive filter should be applied to a specific portion of the second picture, The method, comprising applying the adaptive filter to the specific portion of the second picture using the filter coefficients.

2. The method according to claim 1, wherein the suffix unit is the last unit of the access unit (AU) of the first picture.

3. The method according to claim 1, wherein the video data corresponding to the second picture includes video coding layer (VCL) network abstraction layer (NAL) units representing one or more slices.

4. The method according to claim 1, wherein the suffix unit corresponding to the first picture includes a non-VCL NAL unit that follows all VCL NAL units corresponding to the first picture.

5. The method according to claim 1, wherein the specific portion of the second picture includes a coding tree block (CTB) of the second picture.

6. The method according to claim 1, wherein the filter coefficients include adaptive loop filter (ALF) parameters, and the flags include the ALF CTB flag.

7. A method for encoding video, Encoding a filter coefficient located in a suffix unit corresponding to the first picture of the video stream, wherein the filter coefficient is for a second picture following the first picture, The method comprising encoding a flag from video data corresponding to a second picture of the video stream, indicating that an adaptive filter is applied to a specific portion of the second picture, wherein the adaptive filter is applied to the specific portion of the second picture using the filter coefficients.

8. The method according to claim 7, wherein the suffix unit is the last unit of the access unit (AU) of the first picture.

9. The method according to claim 7, wherein the video data corresponding to the second picture includes video coding layer (VCL) network abstraction layer (NAL) units representing one or more slices.

10. The method according to claim 7, wherein the suffix unit corresponding to the first picture includes a non-VCL NAL unit that follows all VCL NAL units corresponding to the first picture.

11. The method according to claim 7, wherein the specific portion of the second picture includes a coding tree block (CTB) of the second picture.

12. The method according to claim 7, wherein the filter coefficients include adaptive loop filter (ALF) parameters, and the flags include the ALF CTB flag.

13. A non-temporary computer-readable medium that stores instructions, when executed, causing a computer to perform the method according to any one of claims 1 to 12.

14. It includes at least one processor and memory, A video coding apparatus in which the at least one processor is configured to cooperate with the memory to perform the method according to any one of claims 1 to 12.

Images