Encoder, decoder and corresponding method

Abstract

To provide a video coding mechanism.SOLUTION: A mechanism of the present invention includes receiving a bitstream that includes a slice and a plurality of adaptive parameter sets (APSs) including a plurality of APS types. Each APS includes an APS identifier (ID), and APS IDs for the APS types are assigned in sequence across a plurality of different value spaces. The mechanism further includes decoding the slice using parameters from the plurality of APSs. The mechanism further includes forwarding the slice for display as part of a decoded video sequence.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] This disclosure relates generally to video coding, and more particularly to efficient signaling of coding tool parameters used to compress video data in video coding. [Background technology]

[0002] The amount of video data required to depict even a relatively short video can be substantial, presenting challenges when the data is streamed or otherwise communicated over communications networks with limited bandwidth capacity. Therefore, video data is typically compressed before being communicated over modern telecommunications networks. Furthermore, because memory resources may be limited, video size can also be an issue when the video is stored on a storage device. Video compression devices often use software and / or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data required to represent a digital video image. The compressed data is then received at the destination by a video decompression device, which decodes the video data. With limited network resources and ever-increasing demands for higher video quality, improved compression and decompression techniques that improve compression ratios with little or no sacrifice in image quality are desirable. Summary of the Invention

[0003] In one embodiment, the present disclosure includes a method implemented in a decoder, the method including: receiving, by a receiver of the decoder, a bitstream including multiple adaptive parameter set (APS) types associated with a coded slice, each APS including an APS identifier (ID), each APS type using a separate value space for the APS ID; decoding, by a processor, the coded slice using parameters from the multiple APSs obtained based on the APS IDs; and forwarding, by the processor, the slice for display as part of a decoded video sequence. APSs are used to maintain data related to multiple slices across multiple pictures. The present disclosure describes various improvements related to APSs. In this example, each APS includes an APS ID. Furthermore, each APS type includes a separate value space for the corresponding APS ID. Such value spaces can overlap. Thus, an APS of a first type, such as an adaptive loop filter (ALF) APS, can include the same APS ID as an APS of a second type, such as a luma mapping with chroma scaling (LMCS) APS. This is achieved by identifying each APS by a combination of APS parameter type and APS ID. By allowing each APS type to contain a different value space, the codec does not need to check across APS types for ID conflicts. Furthermore, by allowing value spaces to overlap, the codec can avoid using larger ID values, resulting in bit savings. Therefore, using separate, overlapping value spaces for APSs of different types improves coding efficiency and therefore reduces the use of network, memory, and / or processing resources in the encoder and decoder.

[0004] Optionally, in any of the above aspects, another implementation of the aspect provides that the separate value spaces overlap.

[0005] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that the plurality of APS types include an ALF type including an ALF type parameter, a scaling list type including a scaling list parameter, and an LMCS type including an LMCS parameter.

[0006] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that each APS includes an APS parameter type (aps_params_type) code set to a predefined value indicating the type of parameters included in each APS.

[0007] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that the current APS includes a current APS ID selected from a predefined range across a current value space, the current APS ID relates to a previous APS ID associated with a previous APS of the same type as the current APS, and the current APS ID is not related to another previous APS ID associated with another previous APS of a different type from the current APS.

[0008] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that each of the distinct value spaces spans a predefined range, and the predefined range is determined based on the APS type.

[0009] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that each APS is identified by a combination of a current APS type and a current APS ID.

[0010] In one embodiment, the present disclosure includes a method implemented in an encoder, the method including: encoding, by a processor of the encoder, a slice as a coded slice into a bitstream; determining, by the processor, parameter types used to code the coded slice; encoding, by the processor, parameter types in the multiple APSs into the bitstream by including the parameter types in multiple APSs; assigning, by the processor, an APS ID to each of the multiple APSs such that each of the multiple APS types uses a separate value space for the APS ID; encoding, by the processor, each APS ID into the multiple APSs; and storing, by a memory coupled to the processor, the bitstream for communication to a decoder. The APSs are used to maintain data related to multiple slices across multiple pictures. The present disclosure describes various improvements related to APSs. In this example, each APS includes an APS ID. Furthermore, each APS type includes a separate value space for the corresponding APS ID. Such value spaces may overlap. Thus, an APS of a first type, such as an ALF APS, can contain the same APS ID as an APS of a second type, such as an LMCS APS. This is achieved by identifying each APS by a combination of the APS parameter type and the APS ID. By allowing each APS type to contain a different value space, the codec does not need to check across APS types for ID conflicts. Furthermore, by allowing value spaces to overlap, the codec can avoid using larger ID values, resulting in bit savings. Therefore, using separate, overlapping value spaces for APSs of different types improves coding efficiency and therefore reduces the use of network, memory, and / or processing resources in the encoder and decoder.

[0011] Optionally, in any of the above aspects, another implementation of the aspect provides that the separate value spaces overlap.

[0012] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that the plurality of APS types include an ALF type including an ALF parameter, a scaling list type including a scaling list parameter, and an LMCS type including an LMCS parameter.

[0013] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that each APS includes an aps_params_type code set to a predefined value indicating the type of parameters included in each APS.

[0014] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that the current APS includes a current APS ID selected from a predefined range across a current value space, the current APS ID relates to a previous APS ID associated with a previous APS of the same type as the current APS, and the current APS ID is not related to another previous APS ID associated with another previous APS of a different type from the current APS.

[0015] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that each of the distinct value spaces spans a predefined range, and the predefined range is determined based on the APS type.

[0016] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that each APS is identified by a combination of a current APS type and a current APS ID.

[0017] In an embodiment, the present disclosure comprises a video coding device including a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are configured to perform a method according to any of the aforementioned aspects.

[0018] In one embodiment, a non-transitory computer-readable medium including a computer program product for use by a video coding device, the computer program product including computer-executable instructions stored on the non-transitory computer-readable medium that, when executed by a processor, cause the video coding device to perform a method according to any of the above aspects.

[0019] In one embodiment, the present disclosure includes a decoder including receiving means for receiving a bitstream including a plurality of APSs including a plurality of APS types associated with a coded slice, each APS including an APS ID, each APS type using a distinct value space for the APS ID; decoding means for decoding the coded slice using parameters from the plurality of APSs obtained based on the APS IDs; and forwarding means for forwarding the decoding result for display as part of a decoded video sequence.

[0020] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that the decoder is further configured to perform the method of any of the aforementioned aspects.

[0021] In one embodiment, the present disclosure includes an encoder including a determining means for determining a plurality of parameter types to be used to decode a slice; an encoding means for encoding the slice into a bitstream as a coded slice and encoding a plurality of parameter types in the plurality of APSs into the bitstream by including the plurality of parameter types in the plurality of APSs and encoding each APS ID into the plurality of APSs; an assigning means for assigning an APS ID to each of the plurality of APSs such that each of the plurality of APS types uses a separate value space for the APS ID; and a storage means for storing the bitstream for communication to a decoder.

[0022] Optionally, in any of the above aspects, another implementation aspect of the aspect provides that the encoder is further configured to perform the method of any of the aforementioned aspects.

[0023] For clarity, any one of the above-described embodiments may be combined with any one or more of the other above-described embodiments to create new embodiments within the scope of the present disclosure.

[0024] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. [Brief explanation of the drawings]

[0025] For a more complete understanding of the present disclosure, reference is now made to the following brief description, taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0026] [Figure 1] 1 is a flowchart of an exemplary method for coding a video signal.

[0027] [Figure 2] 1 is a schematic diagram of an example of an exemplary coding and decoding (codec) system for video coding.

[0028] [Figure 3] 1 is a schematic diagram illustrating an example video encoder;

[0029] [Figure 4] 1 is a schematic diagram illustrating an exemplary video decoder.

[0030] [Figure 5] FIG. 1 is a schematic diagram illustrating an example bitstream including multiple types of adaptation parameter sets including different types of coding tool parameters.

[0031] [Figure 6] FIG. 1 is a schematic diagram illustrating an exemplary mechanism for assigning APS identifiers (IDs) to different APS types on different value spaces.

[0032] [Figure 7] 1 is a schematic diagram of an exemplary video coding device.

[0033] [Figure 8] 1 is a flowchart of an exemplary method for encoding a video sequence into a bitstream by using multiple APS types.

[0034] [Figure 9] 1 is a flowchart of an exemplary method for decoding a video sequence from a bitstream by using multiple APS types.

[0035] [Figure 10] 1 is a schematic diagram of an exemplary system for coding a video sequence of images into a bitstream by using multiple APS types. DETAILED DESCRIPTION OF THE INVENTION

[0036] Initially, exemplary implementations of one or more embodiments are provided below, but it should be understood that the disclosed systems and / or methods may be implemented using any number of technologies, whether currently known or in existence. The present disclosure should in no way be limited to the exemplary implementations, drawings, and technologies shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the appended claims, along with their full range of equivalents.

[0037] The following abbreviations are used herein: Adaptive Loop Filter (ALF), Adaptive Parameter Set (APS), Coding Tree Block (CTB), Coding Tree Unit (CTU), Coding Unit (CU), Coded Video Sequence (CVS), Dynamic Adaptive Streaming over Hypertext Transfer Protocol (DASH), Intra-Random Access Point (IRAP), Joint Video Experts Team (JVET), Motion Constrained Tile Set (MCTS), Maximum Transmission Unit (MTU), Network Abstraction Layer (NAL), Picture Order Count (POC), Raw Byte Sequence Payload (RBSP), Sample Adaptive Offset (SAO), Sequence Parameter Set (SPS), Versatile Video Coding (VVC), and Working Draft (WD).

[0038] Many video compression techniques may be used to reduce the size of video files with minimal data loss. For example, video compression techniques may include performing spatial (e.g., intra-picture) prediction and / or temporal (e.g., inter-picture) prediction to reduce or remove data redundancy in a video sequence. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may also be referred to as tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are coded using spatial prediction with respect to reference samples in neighboring blocks within the same picture. Video blocks in an inter-coded unidirectionally predicted (P) or bidirectionally predicted (B) slice of a picture may be coded using spatial prediction with respect to reference samples in neighboring blocks within the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may be referred to as a frame and / or an image, and a reference picture may be referred to as a reference frame and / or a reference image. Spatial or temporal prediction results in a prediction block, which represents an image block. Residual data represents the pixel differences between the original image block and the prediction block. Thus, inter-coded blocks are coded according to a motion vector that points to a block of reference samples forming the prediction block and the residual data indicating the difference between the coding block and the prediction block. Intra-coded blocks are coded according to an intra-coding mode and the residual data. For further compression, the residual data may be transformed from the pixel domain to a transform domain. These result in residual transform coefficients, which may be quantized. The quantized transform coefficients may initially be arranged in a two-dimensional array. The quantized transform coefficients may then be scanned to generate a one-dimensional vector of transform coefficients.Entropy coding can be applied to achieve even greater compression. Such video compression techniques are discussed in more detail below.

[0039] To ensure that the coded video can be decoded accurately, the video is coded and decoded according to corresponding video coding standards, including International Telecommunication Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization / International Electrotechnical Commission (ISO / IEC) Motion Picture Experts Group (MPEG)-1 Part 2, Advanced Video Coding (AVC), also known as ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, ITU-T H.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC), Multiview Video Coding plus Depth (MVC+D), and three-dimensional (3D) AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The ITU-T and ISO / IEC Joint Video Experts Team (JVET) has begun development of a video coding standard called Versatile Video Coding (VVC). VVC is included in working drafts (WDs), including JVET-M1001-v5 and JVET-M1002-v1, which provide algorithm descriptions, encoder-side descriptions of the VVC WD, and reference software.

[0040] A video sequence is coded using various coding tools. The encoder selects parameters for the coding tools with the goal of improving compression with minimal quality loss when the video sequence is decoded. The coding tools may relate to different parts of the video at different scopes. For example, some coding tools relate to the video sequence level, some to the picture level, and some to the slice level. An APS may be used to signal information that can be shared by multiple pictures and / or slices across different pictures. Specifically, an APS may carry adaptive loop filter (ALF) parameters. ALF information may not be suitable for signaling at the sequence level in a sequence parameter set (SPS), at the picture level in a picture parameter set (PPS) or picture header, or at the slice level in a tile group / slice header for various reasons.

[0041] If ALF information is signaled in the SPS, the encoder must generate a new SPS and a new IRAP picture every time the ALF information changes. IRAP pictures significantly reduce coding efficiency. Therefore, placing ALF information in the SPS is problematic, especially in low-latency application environments that do not use frequent IRAP pictures. Furthermore, including ALF information in the SPS can invalidate out-of-band transmission of the SPS. Out-of-band transmission refers to the transmission of corresponding data in a transport data flow separate from the video bitstream (e.g., in a media file's sample description or sample entry, in a Session Description Protocol (SDP) file, etc.). Signaling ALF information in the PPS can also be problematic for similar reasons. Specifically, including ALF information in the PPS can invalidate out-of-band transmission of the PPS. Signaling ALF information in the picture header can also be problematic. Picture headers may not be used in some cases. Furthermore, ALF information may apply to multiple pictures. Thus, signaling ALF information in picture headers causes redundant information transmission and therefore wastes bandwidth. Signaling ALF information in tile group / slice headers is also problematic because ALF information may apply to multiple pictures and therefore multiple slices / tile groups. Therefore, signaling ALF information in slice / tile group headers causes redundant information transmission and therefore wastes bandwidth.

[0042] Based on the above, APS can be used to signal ALF parameters. However, a video coding system may use APS only to signal ALF parameters. An example APS syntax and semantics are as follows: [Table 1]

[0043] The adaptation_parameter_set_id provides an identifier for the APS for reference by other syntax elements. An APS can be shared between pictures and can be different in different tile groups within a picture. The aps_extension_flag is set equal to 0 to specify that the aps_extension_data_flag syntax element is not present in the APS RBSP syntax structure. The aps_extension_flag is set equal to 1 to specify that the aps_extension_data_flag syntax element is present in the APS RBSP syntax structure. The aps_extension_data_flag may have any value. The presence and value of the aps_extension_data_flag may not affect the decoder's compliance with the VVC-specified profile. A VVC-compliant decoder may ignore all aps_extension_data_flag syntax elements.

[0044] An example tile group header syntax related to ALF parameters is as follows: [Table 2]

[0045] The tile_group_alf_enabled_flag is set equal to 1 to specify that the adaptive loop filter is enabled and may apply to the luma (Y), blue chroma (Cb), or red chroma (Cr) color components in the tile group. The tile_group_alf_enabled_flag is set equal to 0 to specify that the adaptive loop filter is disabled for all color components in the tile group. The tile_group_aps_id specifies the adaptation_parameter_set_id of the APS referenced by the tile group. The TemporalId of an APS NAL unit with adaptation_parameter_set_id equal to tile_group_aps_id shall be less than or equal to the TemporalId of the coded tile group NAL unit. When multiple APSs with the same values of adaptation_parameter_set_id are referenced by multiple tile groups of the same picture, the multiple APSs with the same values of adaptation_parameter_set_id may contain the same content.

[0046] The reshaper parameters are used in the adaptive in-loop reshaper video coding tool, also known as luma mapping with chroma scaling (LMCS). An example SPS reshaper syntax and semantics are as follows: [Table 3]

[0047] sps_reshaper_enabled_flag is set equal to 1 to specify that the reshaper is used in Coded Video Sequences (CVS). sps_reshaper_enabled_flag is set equal to 0 to specify that the reshaper is not used in CVS.

[0048] An example tile group header / slice header reshaper syntax and semantics is as follows: [Table 4]

[0049] tile_group_reshaper_model_present_flag is set equal to 1 to specify that tile_group_reshaper_model() is present in the tile group header. tile_group_reshaper_model_present_flag is set equal to 0 to specify that tile_group_reshaper_model() is not present in the tile group header. If tile_group_reshaper_model_present_flag is not present, the flag is inferred to be equal to 0. tile_group_reshaper_enabled_flag is set equal to 1 to specify that the reshaper is enabled for the current tile group. tile_group_reshaper_enabled_flag is set to 0 to specify that the reshaper is not enabled for the current tile group. If tile_group_resharper_enable_flag is not present, the flag is inferred to be equal to 0. tile_group_reshamer_chroma_residual_scale_flag is set equal to 1 to specify that chroma residual scaling is enabled for the current tile group. tile_group_reshaper_chroma_residual_scale_flag is set to 0 to specify that chroma residual scaling is not enabled for the current tile group. If tile_group_reshaper_chroma_residual_scale_flag is not present, the flag is inferred to be equal to 0.

[0050] An example tile group header / slice header reshaper model syntax and semantics is as follows: [Table 5]

[0051] reshape_model_min_bin_idx specifies the minimum bin (or piece) index used in the reshaper construction process. The value of reshape_model_min_bin_idx may range from 0 to MaxBinIdx (inclusive). The value of MaxBinIdx may be equal to 15. reshape_model_delta_max_bin_idx specifies the maximum allowed bin (or piece) index, MaxBinIdx, minus the maximum bin index used in the reshaper construction process. The value of reshape_model_max_bin_idx is set equal to MaxBinIdx minus reshape_model_delta_max_bin_idx. reshaper_model_bin_delta_abs_cw_prec_minus1 plus 1 specifies the number of bits used to represent the syntax reshape_model_bin_delta_abs_CW[i]. reshape_model_bin_delta_abs_CW[i] specifies the absolute delta codeword value for the ith bin.

[0052] reshaper_model_bin_delta_sign_CW_flag[i] specifies the sign of reshape_model_bin_delta_abs_CW[i] as follows: If reshape_model_bin_delta_sign_CW_flag[i] is equal to 0, the corresponding variable RspDeltaCW[i] is positive. Otherwise (e.g., reshape_model_bin_delta_sign_CW_flag[i] is not equal to 0), the corresponding variable RspDeltaCW[i] is negative. If reshaper_model_bin_delta_sign_CW_flag[i] is not present, the flag is inferred to be equal to 0. The variable RspDeltaCW[i] is set equal to (1 - 2 * reshape_model_bin_delta_sign_CW[i]) * reshape_model_bin_delta_abs_CW[i].

[0053] The variable RspCW[i] is derived as follows: The variable OrgCW is set equal to (1 << BitDepthY) / (MaxBinIdx + 1). If reshaper_model_min_bin_idx <= i <= reshaper_model_max_bin_idx, then RspCW[i] = OrgCW + RspDeltaCW[i]. Otherwise, RspCW[i] = 0. The value of RspCW[i] shall be in the range 32 to 2 * OrgCW-1 when the value of BitDepthY is equal to 10. The variable InputPivot[i], with i in the range 0 to MaxBinIdx + 1 (inclusive), is derived as follows: InputPivot[i] = i * OrgCW. The variables ReshapePivot[i], with i in the range 0 to MaxBinIdx + 1 inclusive, and ScaleCoef[i] and InvScaleCoeff[i], with i in the range 0 to MaxBinIdx inclusive, are derived as follows:

number

[0054] The variable ChromaScaleCoef[i], with i in the range 0 to MaxBinIdx (inclusive), is derived as follows:

number

[0055] The properties of the reshaper parameters can be characterized as follows: The size of the set of reshaper parameters contained in the tile_group_reshamer_model() syntax structure is typically around 60-100 bits. The reshaper model is typically updated by the encoder approximately once per second and encompasses many frames. Furthermore, the parameters of an updated reshaper model are unlikely to be exactly the same as the parameters of a previous instance of the reshaper model.

[0056] The aforementioned video coding systems present certain problems. First, such systems are only configured to carry ALF parameters in APS. Furthermore, the reshaper / LMCS parameters may be shared by multiple pictures and may contain many variations.

[0057] Disclosed herein are various mechanisms for modifying APSs to support improved coding efficiency. In a first example, multiple types of APSs are disclosed. Specifically, an APS of type ALF is referred to as an ALF APS and can include ALF parameters. Furthermore, an APS of type Scaling List is referred to as a Scaling List APS and can include scaling list parameters. Additionally, an APS of type LMCS is referred to as an LMCS APS and can include LMCS / reshaper parameters. The ALF APS, Scaling List APS, and LMCS APS may each be coded as separate NAL types and thus included in different NAL units. In this manner, changes to data (e.g., ALF parameters) in one type of APS do not result in redundant coding of data (e.g., LMCS parameters) of other types that do not change. Therefore, providing multiple types of APSs improves coding efficiency and, therefore, reduces the use of network, memory, and / or processing resources in encoders and decoders.

[0058] In the second example, each APS includes an APS identifier (ID). Furthermore, each APS type includes a separate value space for the corresponding APS ID. Such value spaces can overlap. Thus, an APS of a first type (e.g., an ALF APS) can include the same APS ID as an APS of a second type (e.g., an LMCS APS). This is achieved by identifying each APS by a combination of the APS parameter type and the APS ID. By allowing each APS type to include a different value space, the codec does not need to check for ID conflicts across APS types. Furthermore, by allowing value spaces to overlap, the codec can avoid using larger ID values, resulting in bit savings. Therefore, using separate, overlapping value spaces for APSs of different types improves coding efficiency and, therefore, reduces the use of network, memory, and / or processing resources in the encoder and decoder.

[0059] In a third example, the LMCS parameters are included in the LMCS APS. As noted above, the LMCS / reshaper parameters may change approximately once per second. A video sequence may display 30 to 60 pictures per second. Therefore, the LMCS parameters may remain constant for 30 to 60 frames. Including the LMCS parameters in the LMCS APS significantly reduces redundant coding of the LMCS parameters. The slice header and / or picture header associated with the slice can reference the associated LMCS APS. In this way, the LMCS parameters are coded only when the LMCS parameters for a slice change. Therefore, using the LMCS APS to code the LMCS parameters improves coding efficiency and, therefore, reduces the use of network, memory, and / or processing resources in the encoder and decoder.

[0060] FIG. 1 is a flowchart of an exemplary operational method 100 for coding a video signal. Specifically, a video signal is encoded by an encoder. The encoding process compresses the video signal by using various mechanisms to reduce the video file size. The smaller file size allows the compressed video file to be transmitted to a user while reducing the associated bandwidth overhead. A decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally closely resembles the encoding process, allowing the decoder to consistently reconstruct the video signal.

[0061] In step 101, a video signal is input to an encoder. For example, the video signal may be an uncompressed video file stored in memory. As another example, the video file may be captured by a video capture device, such as a video camera, and encoded to support live streaming of the video. The video file may include both an audio component and a video component. The video component includes a series of image frames that, when viewed in sequence, create the visual impression of motion. The frames include pixels expressed in terms of light, referred to herein as luma components (or luma samples), and in terms of color, referred to herein as chroma components (or color samples). In some examples, the frames may also include depth values to support three-dimensional viewing.

[0062] In step 103, the video is partitioned into blocks. Partitioning involves subdividing pixels within each frame into square and / or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame may first be divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64 pixels by 64 pixels). CTUs contain both luma and chroma samples. A coding tree may be used to divide the CTUs into blocks, and then recursively subdivide the blocks until a configuration that supports further encoding is achieved. For example, the luma component of a frame may be subdivided until each block contains relatively uniform illumination values. Furthermore, the chroma component of a frame may be subdivided until each block contains relatively uniform color values. Thus, the partitioning mechanism varies depending on the content of the video frame.

[0063] In step 105, various compression mechanisms are used to compress the image blocks partitioned in step 103. For example, inter-prediction and / or intra-prediction may be used. Inter-prediction is designed to take advantage of the fact that objects in a common scene tend to appear in consecutive frames. Thus, a block depicting an object in a reference frame need not be repeatedly described in adjacent frames. Specifically, an object such as a table may remain in a constant position across multiple frames. Thus, the table may be described once, and adjacent frames may reference back to the reference frame. A pattern matching mechanism may be used to match objects across multiple frames. Furthermore, moving objects may be represented across multiple frames, for example, due to object motion or camera motion. As a specific example, a video may show a car moving across the screen over multiple frames. To describe such motion, a motion vector may be used. A motion vector is a two-dimensional vector that provides an offset from the object's coordinates in a frame to the object's coordinates in a reference frame. Thus, inter-prediction may encode an image block in a current frame as a set of motion vectors indicating its offset from a corresponding block in a reference frame.

[0064] Intra prediction encodes blocks within a common frame. It takes advantage of the fact that luma and chroma components tend to be concentrated in a frame. For example, a green patch in one part of a tree tends to be located adjacent to similar green patches. Intra prediction uses multidirectional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. Directional mode indicates that the current block is similar / identical to samples in neighboring blocks in the corresponding direction. Planar mode indicates that a series of blocks along a row / column (e.g., a plane) can be interpolated based on neighboring blocks at the edge of the row. Planar mode effectively indicates a smooth transition of light / color across a row / column by using a relatively constant slope in changing values. DC mode is used for boundary smoothing, indicating that the block is similar / identical to the average value associated with samples in all neighboring blocks associated with the angular direction of the directional prediction mode. Therefore, intra-predicted blocks can represent image blocks as various related prediction mode values instead of their actual values. Additionally, inter-predicted blocks can represent image blocks as motion vector values instead of actual values. In either case, the predicted block may not accurately represent the image block in some cases. Any differences are stored in a residual block. Transforms can be applied to the residual block to further compress the file.

[0065] Various filtering techniques may be applied in step 107. In HEVC, filters are applied according to an in-loop filtering scheme. The block-based prediction described above may result in the generation of blocky images in the decoder. Furthermore, block-based prediction schemes may encode blocks and then reconstruct the encoded blocks for later use as reference blocks. In-loop filtering schemes iteratively apply noise suppression filters, deblocking filters, adaptive loop filters, and sample adaptive offset (SAO) filters to blocks / frames. These filters mitigate such blocking artifacts so that the encoded file can be accurately reconstructed. Furthermore, these filters mitigate artifacts in the reconstructed reference blocks, making them less likely to generate additional artifacts in subsequent blocks that are coded based on the reconstructed reference blocks.

[0066] Once the video signal has been segmented, compressed, and filtered, the resulting data is coded into a bitstream in step 109. The bitstream includes the data described above, as well as any signaling data desired to support proper video signal reconstruction at the decoder. For example, such data may include partition data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream may be stored in memory for transmission to the decoder upon request. The bitstream may also be broadcast and / or multicast to multiple decoders. Generating the bitstream is an iterative process. Thus, steps 101, 103, 105, 107, and 109 may occur sequentially and / or simultaneously across many frames and blocks. The order depicted in FIG. 1 is presented for clarity and ease of discussion and is not intended to limit the video coding process to any particular order.

[0067] The decoder receives the bitstream and begins the decoding process in step 111. Specifically, the decoder uses an entropy decoding scheme to convert the bitstream into corresponding syntax and video data. In step 111, the decoder uses the syntax data from the bitstream to determine the partitions for the frame. The partitioning should match the results of the block partitioning in step 103. Entropy coding / decoding as used in step 111 will now be described. The encoder makes many choices during the compression process, such as selecting a block partitioning scheme from several possible options based on the spatial location of values in the input image. Signaling the exact choice may involve the use of multiple bins. As used herein, a bin is a binary value (e.g., a bit value that can vary depending on the context) that is treated as a variable. Entropy coding allows the encoder to retain a set of acceptable options and discard any options that are clearly infeasible in a particular case. Each acceptable option is then assigned a codeword. The length of the code word is based on the number of allowable options (e.g., one bin for two options, two bins for three to four options, etc.). The encoder then encodes a code word for the selected option. This scheme reduces the size of the code word because it is only as large as desired to uniquely indicate a choice from a small subset of allowable options, as opposed to uniquely indicating a choice from a potentially large set of all possible options. The decoder then decodes the selection by determining the set of allowable options in a similar manner to the encoder. By determining the set of allowable options, the decoder can read the code word and determine the choice made by the encoder.

[0068] In step 113, the decoder performs block decoding. Specifically, the decoder uses an inverse transform to generate a residual block. The decoder then uses the residual block and a corresponding prediction block to reconstruct an image block according to the partitioning. The prediction block may include both intra-predicted and inter-predicted blocks generated by the encoder in step 105. The reconstructed image block is then positioned within a frame of the reconstructed video signal according to the partitioning data determined in step 111. The syntax for step 113 may also be signaled in the bitstream via entropy coding, as described above.

[0069] In step 115, filtering is performed on the frames of the reconstructed video signal in a manner similar to step 107 in the encoder. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and an SAO filter may be applied to the frames to remove blocking artifacts. Once the frames have been filtered, the video signal may be output to a display in step 117 for viewing by a user.

[0070] FIG. 2 is a schematic diagram of an exemplary coding and decoding (codec) system 200 for video coding. Specifically, codec system 200 provides functionality to support implementation aspects of operational method 100. Codec system 200 is generalized to depict components used in both an encoder and a decoder. Codec system 200 receives and partitions a video signal, as discussed with reference to steps 101 and 103 of operational method 100, to obtain a partitioned video signal 201. When acting as an encoder, codec system 200 then compresses partitioned video signal 201 into a coded bitstream, as discussed with reference to steps 105, 107, and 109 of method 100. When acting as a decoder, codec system 200 generates an output video signal from the bitstream, as discussed with reference to steps 111, 113, 115, and 117 of operational method 100. Codec system 200 includes a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, an intra-picture prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header formatting and context-adaptive binary arithmetic coding (CABAC) component 231. Such components are coupled as shown. In FIG. 2, black lines indicate the movement of data to be coded / decoded, and dashed lines indicate the movement of control data that controls the operation of other components. All of the components of codec system 200 may reside within an encoder. A decoder may include a subset of the components of codec system 200.For example, the decoder may include an intra-picture prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded picture buffer component 223. These components will now be described.

[0071] The partitioned video signal 201 is a captured video sequence that has been partitioned into blocks of pixels by a coding tree. The coding tree uses various partitioning modes to subdivide the blocks of pixels into smaller blocks of pixels. These blocks can then be further subdivided into smaller blocks. Blocks are sometimes referred to as nodes on the coding tree. Larger parent nodes are divided into smaller child nodes. The number of times a node is subdivided is referred to as the depth of the node / coding tree. In some cases, the partitioned blocks may be included in a coding unit (CU). For example, a CU may be a subpart of a CTU that includes a luma block, a red-difference chroma (Cr) block, and a blue-difference chroma (Cr) block, along with the corresponding CU syntax instructions. Partitioning modes may include binary tree (BT), triple tree (TT), and quad tree (QT), which are used to partition a node into two, three, or four child nodes of different shapes, depending on the partitioning mode used. The partitioned video signal 201 is forwarded to a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, a filter control analysis component 227, and a motion estimation component 221 for compression.

[0072] The general coder control component 211 is configured to make decisions related to coding images of a video sequence into a bitstream according to application constraints. For example, the general coder control component 211 manages the optimization of bitrate / bitstream size versus reconstruction quality. Such decisions may be made based on storage / bandwidth availability and image resolution requirements. The general coder control component 211 also manages buffer utilization in relation to transmission rate to mitigate buffer underrun and overrun issues. To manage these issues, the general coder control component 211 manages partitioning, prediction, and filtering by other components. For example, the general coder control component 211 can dynamically increase compression complexity to improve resolution and bandwidth usage, or decrease compression complexity to reduce resolution and bandwidth usage. Thus, the general coder control component 211 controls other components of the codec system 200 to balance bitrate concerns with video signal reconstruction quality. The general coder control component 211 generates control data that controls the operation of other components. Control data is also forwarded to the Header Formatting and CABAC component 231, which encodes it in the bitstream to signal parameters for decoding at the decoder.

[0073] The partitioned video signal 201 is also transmitted to a motion estimation component 221 and a motion compensation component 219 for inter-prediction. A frame or slice of the partitioned video signal 201 may be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-predictive coding of the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 200 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0074] The motion estimation component 221 and the motion compensation component 219 may be highly integrated but are illustrated separately for conceptual purposes. Motion estimation, performed by the motion estimation component 221, is the process of generating motion vectors, which estimate the motion of video blocks. A motion vector may indicate, for example, the displacement of a coded object relative to a predictive block. A predictive block is a block that is found to closely match a coded block in terms of pixel differences. A predictive block is sometimes referred to as a reference block. Such pixel differences may be determined by sum of absolute differences (SAD), squared differences (SSD), or other difference metrics. HEVC uses several coded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, CTUs are divided into CTBs, which may then be divided into CBs for inclusion in CUs. CUs may be coded as prediction units (PUs), which contain prediction data, and / or transform units (TUs), which contain the transform residual data of the CU. The motion estimation component 221 uses rate-distortion analysis as part of a rate-distortion optimization process to generate motion vectors, PUs, and TUs. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for a current block / frame and select the reference block, motion vector, etc. with the best rate-distortion characteristics. The best rate-distortion characteristics balance both the quality of the video reconstruction (e.g., the amount of data lost due to compression) and the coding efficiency (e.g., the size of the final encoding).

[0075] In some examples, the codec system 200 may calculate values for sub-integer pixel positions of reference pictures stored in the decoded picture buffer component 223. For example, the video codec system 200 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference picture. Thus, the motion estimation component 221 can perform motion searches for whole-pixel and fractional pixel positions and output motion vectors with fractional pixel accuracy. The motion estimation component 221 calculates motion vectors for PUs of video blocks in inter-coded slices by comparing the positions of the PUs with the positions of the predictive blocks of the reference pictures. The motion estimation component 221 outputs the calculated motion vectors for encoding to the header formatting and CABAC component 231 as motion data and outputs motion to the motion compensation component 219.

[0076] Motion compensation performed by motion compensation component 219 may involve fetching or generating a predictive block based on a motion vector determined by motion estimation component 221. Also, in some examples, motion estimation component 221 and motion compensation component 219 may be functionally integrated. Upon receiving a motion vector for a PU of a current video block, motion compensation component 219 may locate the predictive block pointed to by the motion vector. A residual video block is then formed by subtracting pixel values of the predictive block from pixel values of the currently coded video block to form pixel difference values. Generally, motion estimation component 221 performs motion estimation on the luma component, and motion compensation component 219 uses the motion vector calculated based on the luma component for both the chroma and luma components. The predictive block and residual block are forwarded to transform scaling and quantization component 213.

[0077] The partitioned video signal 201 is also sent to an intra-picture estimation component 215 and an intra-picture prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra-picture estimation component 215 and the intra-picture prediction component 217 may be highly integrated but are illustrated separately for conceptual purposes. The intra-picture estimation component 215 and the intra-picture prediction component 217 intra-predict the current block relative to blocks within the current frame, instead of the inter-prediction performed by the inter-frame motion estimation component 221 and the motion compensation component 219 described above. In particular, the intra-picture estimation component 215 determines the intra-prediction mode to use to encode the current block. In some examples, the intra-picture estimation component 215 selects an appropriate intra-prediction mode to encode the current block from multiple tested intra-prediction modes. The selected intra-prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0078] For example, the intra picture estimation component 215 calculates rate-distortion values for various tested intra prediction modes using a rate-distortion analysis and selects the intra prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the bit rate (e.g., number of bits) used to generate the coded block and the amount of distortion (or error) between the coded block and the original uncoded block coded to generate the coded block. The intra picture estimation component 215 calculates a ratio from the distortion and rate for the various coded blocks and determines which intra prediction mode exhibits the best rate-distortion value for the block. Additionally, the intra picture estimation component 215 may be configured to code the depth blocks of the depth map using a rate-distortion optimization (RDO)-based depth modeling mode (DMM).

[0079] The intra picture prediction component 217, when implemented in an encoder, can generate a residual block from the prediction block based on the selected intra prediction mode determined by the intra picture prediction component 215, or, when implemented in a decoder, can read the residual block from the bitstream. The residual block contains the value differences between the prediction block and the original block, represented as a matrix. The residual block is then forwarded to the transform scaling and quantization component 213. The intra picture estimation component 215 and the intra picture prediction component 217 can operate on both the luma component and the chroma component.

[0080] The transform scaling and quantization component 213 is configured to further compress the residual block. The transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform, to the residual block, generating a video block containing residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. The transform may convert the residual information from the pixel value domain to a transform domain, such as the frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling may involve applying a scale factor to the residual information so that different frequency information is quantized with different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also configured to quantize the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the transform scaling and quantization component 213 may then perform a scan of the matrix containing the quantized transform coefficients, which are forwarded to the header formatting and CABAC component 231 for encoding into a bitstream.

[0081] The scaling and inverse transform component 229 applies the inverse operations of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, transformation, and / or quantization to reconstruct a residual block in the pixel domain for use as a reference block that may later become a predictive block for another current block, for example. The motion estimation component 221 and / or motion compensation component 219 can calculate a reference block by converting the residual block back to the corresponding predictive block for use in motion estimation of a later block / frame. A filter is applied to the reconstructed reference block to mitigate artifacts generated during scaling, quantization, and transformation. Otherwise, such artifacts may cause inaccurate predictions (and create additional artifacts) when subsequent blocks are predicted.

[0082] The filter control analysis component 227 and the in-loop filter component 225 apply filters to residual blocks and / or reconstructed image blocks. For example, a transformed residual block from the scaling and inverse transform component 229 may be combined with a corresponding prediction block from the intra-picture prediction component 217 and / or the motion compensation component 219 to reconstruct an original image block. The filter may then be applied to the reconstructed image block. In some examples, the filter may instead be applied to the residual block. Like the other components in FIG. 2, the filter control analysis component 227 and the in-loop filter component 225 are highly integrated and may be implemented together, but are depicted separately for conceptual purposes. The filters applied to reconstructed reference blocks are applied to specific spatial regions and include multiple parameters to adjust how such filters are applied. The filter control analysis component 227 analyzes the reconstructed reference blocks to determine where such filters should be applied and set the corresponding parameters. Such data is forwarded to the header formatting and CABAC component 231 as filter control data for encoding. The in-loop filter component 225 applies such filters based on the filter control data. The filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Such filters may be applied in the spatial / pixel domain (e.g., on reconstructed pixel blocks) or the frequency domain, depending on the example.

[0083] When operating as an encoder, the filtered reconstructed image blocks, residual blocks, and / or prediction blocks are stored in the decoded picture buffer component 223 for later use in motion estimation as described above. When operating as a decoder, the decoded picture buffer component 223 stores the reconstructed and filtered blocks as part of the output video signal for forwarding towards the display. The decoded picture buffer component 223 can be any memory device capable of storing prediction blocks, residual blocks, and / or reconstructed image blocks.

[0084] The header formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes such data into a coded bitstream for transmission to a decoder. Specifically, the header formatting and CABAC component 231 generates various headers for encoding control data, such as general control data and filter control data. Additionally, prediction data, including intra-prediction and motion data, and residual data in the form of quantized transform coefficient data are all coded into the bitstream. The final bitstream contains all information required by the decoder to reconstruct the original partitioned video signal 201. Such information may also include an intra-prediction mode index table (also called a codeword mapping table), definitions of coding contexts for various blocks, an indication of the most likely intra-prediction mode, an indication of partition information, etc. Such data may be coded using entropy coding. For example, the information may be encoded using context-adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioned entropy (PIPE) coding, or another entropy coding technique. Following entropy coding, the coded bitstream may be transmitted to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0085] 3 is a block diagram illustrating an exemplary video encoder 300. Video encoder 300 may be used to implement the encoding functionality of codec system 200 and / or to implement steps 101, 103, 105, 107, and / or 109 of method of operation 100. Encoder 300 partitions an input video signal, resulting in partitioned video signal 301, which is substantially similar to partitioned video signal 201. Partitioned video signal 301 is then compressed and encoded into a bitstream by components of encoder 300.

[0086] Specifically, the partitioned video signal 301 is forwarded to an intra-picture prediction component 317 for intra prediction. The intra-picture prediction component 317 may be substantially similar to the intra-picture estimation component 215 and the intra-picture prediction component 217. The partitioned video signal 301 is also forwarded to a motion compensation component 321 for inter prediction based on reference blocks in a decoded picture buffer component 323. The motion compensation component 321 may be substantially similar to the motion estimation component 221 and the motion compensation component 219. The prediction blocks and residual blocks from the intra-picture prediction component 317 and the motion compensation component 321 are forwarded to a transform and quantization component 313 for transforming and quantizing the residual blocks. The transform and quantization component 313 may be substantially similar to the transform scaling and quantization component 213. The transformed and quantized residual blocks and corresponding prediction blocks (along with associated control data) are forwarded to an entropy coding component 331 for coding into a bitstream. The entropy coding component 331 may be substantially similar to the header formatting and CABAC component 231 .

[0087] The transformed and quantized residual block and / or the corresponding prediction block are also transferred from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into a reference block used by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially similar to the scaling and inverse transform component 229. An in-loop filter within the in-loop filter component 325 is also applied to the residual block and / or the reconstructed reference block, depending on the example. The in-loop filter component 325 may be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may include multiple filters, as discussed with respect to the in-loop filter component 225. The filtered block is then stored in the decoded picture buffer component 323 and used as a reference block by the motion compensation component 321. The decoded picture buffer component 323 may be substantially similar to the decoded picture buffer component 223.

[0088] 4 is a block diagram illustrating an exemplary video decoder 400. Video decoder 400 may be used to implement the decoding functionality of codec system 200 and / or to implement steps 111, 113, 115, and / or 117 of operating method 100. Decoder 400 receives a bitstream, for example, from encoder 300, and generates a reconstructed output video signal based on the bitstream for display to an end user.

[0089] The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to implement an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 may use header information to provide context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding the video signal, such as general control data, filter control data, partition information, motion data, prediction data, and quantized transform coefficients from the residual block. The quantized transform coefficients are forwarded to the inverse transform and quantization component 429 for reconstruction into the residual block. The inverse transform and quantization component 429 may be similar to the inverse transform and quantization component 329.

[0090] The reconstructed residual block and / or prediction block are forwarded to the intra picture prediction component 417 for reconstruction into an image block based on an intra prediction operation. The intra picture prediction component 417 may be similar to the intra picture estimation component 215 and the intra picture prediction component 217. Specifically, the intra picture prediction component 417 uses a prediction mode to locate a reference block within a frame and applies the residual block to the result to reconstruct an intra predicted image block. The reconstructed intra predicted image block and / or residual block and corresponding inter prediction data are forwarded to the decoded picture buffer component 423 via the in-loop filter component 425, which may be substantially similar to the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block, and / or prediction block, and such information is stored in the decoded picture buffer component 423. The reconstructed image blocks from the decoded picture buffer component 423 are forwarded to the motion compensation component 421 for inter-prediction. The motion compensation component 421 may be substantially similar to the motion estimation component 221 and / or the motion compensation component 219. Specifically, the motion compensation component 421 uses motion vectors from reference blocks to generate prediction blocks and applies a residual block to the result to reconstruct an image block. The resulting reconstructed blocks may also be forwarded to the decoded picture buffer component 423 via an in-loop filter component 425. The decoded picture buffer component 423 continues to store additional reconstructed image blocks that may be reconstructed into frames via the partition information. Such frames may also be arranged in a sequence. This sequence is output to a display as a reconstructed output video signal.

[0091] 5 is a schematic diagram illustrating an example bitstream 500 including multiple types of APSs containing different types of coding tool parameters. For example, bitstream 500 may be generated by codec system 200 and / or encoder 300 for decoding by codec system 200 and / or decoder 400. As another example, bitstream 500 may be generated by an encoder at step 109 of method 100 for use by a decoder at step 111.

[0092] The bitstream 500 includes a sequence parameter set (SPS) 510, multiple picture parameter sets (PPSs) 511, multiple ALF APSs 512, multiple scaling list APSs 513, multiple LMCS APSs 514, multiple slice headers 515, and image data 520. The SPS 510 includes sequence data common to all pictures in the video sequence included in the bitstream 500. Such data may include picture size, bit depth, coding tool parameters, bit rate limits, etc. The PPS 511 includes parameters that apply to the entire picture. Thus, each picture in the video sequence may reference the PPS 511. Note that while each picture references the PPS 511, a single PPS 511 can include data for multiple pictures in some instances. For example, multiple similar pictures may be coded according to similar parameters. In such cases, a single PPS 511 may include data for such similar pictures. The PPS 511 may indicate the coding tools available for the slice in the corresponding picture, quantization parameters, offsets, etc. The slice header 515 contains parameters specific to each slice in the picture. Thus, in a video sequence, there may be one slice header 515 per slice. The slice header 515 may include slice type information, a picture order count (POC), a reference picture list, prediction weights, tile entry points, deblocking parameters, etc. Note that the slice header 515 may also be referred to as a tile group header in some contexts.

[0093] An APS is a syntax structure containing syntax elements that apply to one or more pictures 521 and / or slices 523. In the illustrated example, APSs can be divided into multiple types. The ALF APS 512 is an APS of type ALF that contains ALF parameters. The ALF is an adaptive block-based filter that includes a transfer function controlled by variable parameters and uses feedback from a feedback loop to refine the transfer function. Additionally, the ALF is used to correct coding artifacts (e.g., errors) that result from block-based coding. The adaptive filter is a linear filter with a transfer function controller with variable parameters that can be controlled by an optimization algorithm, such as an RDO process, operating in the encoder. Thus, the ALF parameters included in the ALF APS 512 may include variable parameters selected by the encoder so that the filter removes block-based coding artifacts during decoding at the decoder.

[0094] The scaling list APS 513 is an APS of type scaling list that includes scaling list parameters. As described above, the current block is coded according to inter- or intra-prediction, which results in a residual. The residual is the difference between the luma and / or chroma values of the block and the values of the corresponding predicted block. A transform is then applied to the residual to convert it into transform coefficients (smaller than the residual value). Encoding high-resolution and / or ultra-high-resolution content can result in increased residual data. Simple transform processes, when applied to such data, can result in significant quantization noise. Therefore, the scaling list parameters included in the scaling list APS 513 can include weighting parameters that are applied to scale the transform matrix and can take into account changes in display resolution and / or the acceptable level of quantization noise in the resulting decoded video image.

[0095] LMCS APS514 is an APS of type LMCS that includes LMCS parameters, also known as reshaper parameters. The human visual system is less able to distinguish color differences (e.g., chromaticity) than light differences (e.g., luminance). Therefore, some video systems use a chroma subsampling mechanism to compress video data by reducing the resolution of chroma values without adjusting the corresponding luma values. One concern with such mechanisms is that the associated interpolation may produce interpolated chroma values during decoding that are incompatible with the corresponding luma values in some locations. This creates color artifacts in such locations, which should be corrected by a corresponding filter. This is complicated by the luma mapping mechanism. Luma mapping is the process of remapping coded luma components across the dynamic range of the input luma signal (e.g., according to a piecewise linear function). This compresses the luma components. The LMCS algorithm scales the compressed chroma values based on the luma mapping to remove artifacts related to chroma subsampling. Therefore, the LMCS parameters included in LMCS APS514 indicate the chroma scaling used to describe the luma mapping. The LMCS parameters are determined by the encoder and can be used by the decoder to filter out artifacts caused by chroma subsampling when luma mapping is used.

[0096] The image data 520 includes video data coded according to inter-prediction and / or intra-prediction, as well as corresponding transformed and quantized residual data. For example, a video sequence includes multiple pictures 521 coded as image data. A picture 521 is a single frame of a video sequence and is therefore generally displayed as a single unit when displaying the video sequence. However, partial pictures may be displayed to implement certain technologies, such as virtual reality and picture-in-picture. Each of the multiple pictures 521 references a PPS 511. The multiple pictures 521 are divided into multiple slices 523. A slice 523 may be defined as a horizontal section of a picture 521. For example, a slice 523 may include a portion of the height and the entire width of the picture 521. In other cases, a picture 521 may be divided into columns and rows, and a slice 523 may be included in a rectangular portion of the picture 521 generated by such columns and rows. In some systems, slices 523 are subdivided into tiles. In other systems, slices 523 are referred to as tile groups that contain tiles. Slices 523 and / or tile groups of tiles reference slice headers 515. Slices 523 are further divided into coding tree units (CTUs). CTUs are further divided into coding blocks based on the coding tree. The coding blocks can then be coded / decoded according to a prediction mechanism.

[0097] The picture 521 and / or slice 523 may directly or indirectly reference the ALF APS 512, scaling list APS 513, and / or LMCS APS 514 containing associated parameters. For example, the slice 523 may reference the slice header 515. Furthermore, the picture 521 may reference the corresponding picture header. The slice header 515 and / or picture header may reference the ALF APS 512, scaling list APS 513, and / or LMCS APS 514 containing parameters used in coding the associated slice 523 and / or picture 521. In this way, the decoder can obtain the coding tool parameters relevant to the slice 523 and / or picture 521 according to the header references related to the corresponding slice 523 and / or picture 521.

[0098] The bitstream 500 is coded into video coding layer (VCL) NAL units 535 and non-VCL NAL units 531. NAL units are coded data units sized to fit as the payload of a single packet for transmission over a network. The VCL NAL units 535 are NAL units that contain coded video data. For example, each VCL NAL unit 535 may contain one slice 523 and / or tile group of data, CTUs, and / or coding blocks. The non-VCL NAL units 531 are NAL units that contain supporting syntax but do not contain coded video data. For example, the non-VCL NAL units 531 may include an SPS 510, a PPS 511, an APS, a slice header 515, etc. Thus, the decoder receives the bitstream 500 in discrete VCL NAL units 535 and non-VCL NAL units 531. An access unit is a group of VCL NAL units 535 and / or non-VCL NAL units 531 that contains enough data to code a single picture 521.

[0099] In some examples, the ALF APS 512, scaling list APS 513, and LMCS APS 514 are each assigned to a separate non-VCL NAL unit 531 type. In such cases, the ALF APS 512, scaling list APS 513, and LMCS APS 514 are included in an ALF APS NAL unit 532, a scaling list APS NAL unit 533, and an LMCS APS NAL unit 534, respectively. Thus, the ALF APS NAL unit 532 contains ALF parameters that remain valid until another ALF APS NAL unit 532 is received. Furthermore, the scaling list APS NAL unit 533 contains scaling list parameters that remain valid until another scaling list APS NAL unit 533 is received. Additionally, the LMCS APS NAL unit 534 contains LMCS parameters that remain valid until another LMCS APS NAL unit 534 is received. In this way, it is not necessary to issue a new APS each time the APS parameters are changed. For example, a change in LMCS parameters results in an additional LMCS APS 514, but not an additional ALF APS 512 or scaling list APS 513. Therefore, separating APSs into different NAL unit types based on parameter type avoids redundant signaling of irrelevant parameters. Therefore, separating APSs into different NAL unit types improves coding efficiency and thus reduces the use of processor, memory, and / or network resources in the encoder and decoder.

[0100] Furthermore, slice 523 and / or picture 521 may directly or indirectly reference ALF APS 512, ALF APS NAL unit 532, scaling list APS 513, scaling list APS NAL unit 533, LMCS APS 514, and / or LMCS APS NAL unit 534, which contain coding tool parameters used to code slice 523 and / or picture 521. For example, each APS may include APS ID 542 and parameter type 541. APS ID 542 is a value (e.g., a number) that identifies the corresponding APS. APS ID 542 may include a predefined number of bits. Thus, APS ID 542 may increase (e.g., by 1) according to a predefined sequence and may reset to a minimum value (e.g., 0) when the sequence reaches the end of a predefined range. Parameter type 541 indicates the type of parameters (e.g., ALF, scaling list, and / or LMCS) included in the APS. For example, parameter type 541 may include an APS parameter type (aps_params_type) code set to a predefined value indicating the type of parameters included in each APS. In this manner, parameter type 541 may be used to distinguish between ALF APS 512, scaling list APS 513, and LMCS APS 514. In some examples, ALF APS 512, scaling list APS 513, and LMCS APS 514 may each be uniquely identified by the combination of parameter type 541 and APS ID 542. For example, each APS type may include a separate value space for the corresponding APS ID 542. Thus, each APS type may include an APS ID 542 that increases sequentially based on a previous APS of the same type. However, the APS ID 542 of a first APS type may be unrelated to the APS ID 542 of a previous APS of a different second APS type. Thus, the APS IDs 542 of different APS types may include overlapping value spaces.For example, a first type of APS (e.g., an ALF APS) may, in some cases, include the same APS ID 542 as a second type of APS (e.g., an LMCS APS). By allowing each APS type to include a different value space, the codec does not need to check for APS ID 542 conflicts between APS types. Furthermore, by allowing value spaces to overlap, the codec can avoid using larger APS ID 542 values, resulting in bit savings. Thus, using separate, overlapping value spaces for multiple APS IDs 542 of different APS types improves coding efficiency and thus reduces the use of network, memory, and / or processing resources in the encoder and decoder. As noted above, the APS ID 542 may span a predefined range. In some examples, the predefined range of the APS ID 542 may vary depending on the APS type indicated by the parameter type 541. This may allow different numbers of bits to be assigned to different APS types, generally depending on how frequently parameters of different types change. For example, the APS ID 542 of the ALF APS 512 may have a range of 0-7, the APS ID 542 of the scaling list APS 513 may have a range of 0-7, and the APS ID 542 of the LMCS APS 514 may have a range of 0-3.

[0101] In another example, the LMCS parameters are included in the LMCS APS 514. Some systems include the LMCS parameters in the slice header 515. However, the LMCS / reshaper parameters may change approximately once per second. A video sequence may display 30 to 60 pictures 521 per second. Therefore, the LMCS parameters may not change for 30 to 60 frames. Including the LMCS parameters in the LMCS APS 514 significantly reduces redundant coding of the LMCS parameters. In some examples, the picture header associated with each slice header 515 and / or slice 523 and / or picture 521 may reference the associated LMCS APS 514. The slice 523 and / or picture 521, in turn, references the slice header 515 and / or picture header. This allows a decoder to obtain the LMCS parameters for the associated slice 523 and / or picture 521. In this way, the LMCS parameters are coded only when the LMCS parameters for the slice 523 and / or picture 521 change. Therefore, using the LMCS APS 514 to code the LMCS parameters increases coding efficiency and, therefore, reduces the use of network, memory, and / or processing resources in the encoder and decoder. Because LMCS is not used for all video, the SPS 510 may include an LMCS enable flag 543. The LMCS enable flag 543 may be set to indicate that LMCS is enabled for the coded video sequence. Thus, the decoder may obtain the LMCS parameters from the LMCS APS 514 based on the LMCS enable flag 543 if the LMCS enable flag 543 is set (e.g., set to 1). Furthermore, the decoder may not attempt to obtain the LMCS parameters if the LMCS enable flag 543 is not set (e.g., not set to 0).

[0102] 6 is a schematic diagram illustrating an example mechanism 600 for assigning APS IDs 642 to different APS types on different value spaces. For example, mechanism 600 may be applied to bitstream 500 to assign APS IDs 542 to ALF APSs 512, scaling list APSs 513, and / or LMCS APSs 514. Furthermore, mechanism 600 may be applied to codec 200, encoder 300, and / or decoder 400 when coding video according to method 100.

[0103] Mechanism 600 assigns APS IDs 642 to ALF APSs 612, scaling list APSs 613, and LMCS APSs 614, which may be substantially similar to APS IDs 542, ALF APSs 512, scaling list APSs 513, and LMCS APSs 514, respectively. As noted above, APS IDs 642 may be sequentially assigned across multiple different value spaces, with each value space being specific to the APS type. Furthermore, each value space may span a different range, also specific to the APS type. In the illustrated example, the value space of APS ID 642 for ALF APSs 612 ranges from 0 to 7 (e.g., 3 bits). Furthermore, the value space of APS ID 642 for scaling list APSs 613 ranges from 0 to 7 (e.g., 3 bits). Furthermore, the value space of APS ID 642 for LMCS APSs 611 ranges from 0 to 3 (e.g., 2 bits). When an APS ID 642 reaches the end of the range of the value space, the APS ID 642 of the next APS of the corresponding type wraps around to the beginning of the range (e.g., 0). If a new APS receives the same APS ID 642 as a previous APS of the same type, the previous APS is no longer active and can no longer be referenced. In this way, the range of the value space can be expanded to allow more APSs of the same type to be actively referenced. Furthermore, while the range of the value space can be reduced to improve coding efficiency, such a reduction also reduces the number of APSs of the corresponding type that can simultaneously remain active and available for reference.

[0104] In the example shown, each ALF APS 612, scaling list APS 613, and LMCS APS 614 is referenced by a combination of APS ID 642 and APS type. For example, the ALF APS 612, LMCS APS 614, and scaling list APS 613 each receive an APS ID 642 of 0. When a new ALF APS 612 is received, the APS ID 642 is incremented from the value used for the previous ALF APS 612. The same sequence applies to the scaling list APS 613 and LMCS APS 614. Thus, each APS ID 642 is related to the APS ID 642 of the previous APS of the same type. However, the APS ID 642 is not related to the APS ID 642 of the previous APS of the other type. In this example, the APS ID 642 of the ALF APS 612 increments from 0 to 7, then wraps around to 0 before continuing to increment. Additionally, the APS ID 642 of the scaling list APS 613 increases in increments from 0 to 7, then wraps around to 0 before continuing to increment. Similarly, the APS ID 642 of the LMCS APS 611 increases in increments from 0 to 3, then wraps around to 0 before continuing to increment. As shown, this value space overlaps because different APSs of different APS types can share the same APS ID 642 at the same point in the video sequence. Also, note that the mechanism 600 only renders APSs. In the bitstream, the rendered APSs are interspersed among other VCL and non-VCL NAL units, such as SPSs, PPSs, slice headers, picture headers, and slices.

[0105] Thus, the present disclosure includes improvements to the design of the APS and several improvements for signaling reshaper / LMCS parameters. The APS is designed for signaling information that can be shared across multiple pictures and can include many variations. The reshaper / LMCS parameters are used in adaptive in-loop reshaper / LMCS video coding tools. The above mechanism can be implemented as follows. This includes several aspects that can be used individually and / or in combination to solve the problems enumerated herein.

[0106] The disclosed APS is modified so that multiple APSs can be used to carry different types of parameters. Each APS NAL unit is used to carry only one type of parameter. As a result, two APS NAL units are encoded when two types of information are carried for a particular tile group / slice (e.g., one for each type of information). An APS may include an APS parameter type field in the APS syntax. An APS NAL unit may only contain parameters of the type indicated by the APS parameters type field.

[0107] In some examples, different types of APS parameters are indicated by different NAL unit types. For example, two different NAL unit types are used for APS. These two types of APS may be called ALF APS and reshaper APS, respectively. In another example, the type of tool parameters carried in an APS NAL unit is specified in the NAL unit header. In VVC, the NAL unit header has reserved bits (e.g., 7 bits denoted as nuh_reserved_zero_7bit). In some examples, some of these bits (e.g., 3 of the 7 bits) may be used to specify the APS parameter type field. In some examples, APSs of a particular type may share the same value space for the APS ID. On the other hand, different types of APSs use different value spaces for the APS ID. Therefore, two APSs of different types may coexist and have the same APS ID value at the same time. Furthermore, the combination of the APS ID and the APS parameter type may be used to distinguish an APS from other APSs.

[0108] APS IDs may be included in the tile group header syntax if the corresponding coding tool is enabled for the tile group. Otherwise, APS IDs of the corresponding type may not be included in the tile group header. For example, if ALF is enabled for the tile group, APS IDs of the ALF APSs are included in the tile group header. For example, this can be achieved by setting the APS parameter type field to indicate the ALF type. Therefore, if ALF is not enabled for the tile group, APS IDs of the ALF APSs are not included in the tile group header. Furthermore, if a reshaper coding tool is enabled for the tile group, APS IDs of the reshaper APSs are included in the tile group header. For example, this can be achieved by setting the APS parameter type field to indicate the reshaper type. Furthermore, if a reshaper coding tool is not enabled for the tile group, APS IDs of the reshaper APSs may not be included in the tile group header.

[0109] In some instances, the presence of APS parameter type information in an APS may be conditioned by the use of the coding tool associated with the parameter. If only one APS-related coding tool is enabled for a bitstream (e.g., LMCS, ALF, scaling list), the APS parameter type information may not be present and may instead be inferred. For example, if an APS may contain parameters for the ALF and reshaper coding tools, but only the ALF is enabled (e.g., as specified by flags in the SPS) and the reshaper is not enabled (e.g., as specified by flags in the SPS), the APS parameter type may not be signaled and may be inferred to be equal to the ALF parameters.

[0110] In another example, APS parameter type information may be inferred from the APS ID value. For example, a predefined APS ID value range may be associated with a corresponding APS parameter type. This aspect may be implemented as follows: Instead of allocating X bits for APS ID signaling and Y bits for APS parameter type signaling, X+Y bits may be allocated for APS ID signaling. Next, different APS ID value ranges may be specified to indicate different APS parameter types. For example, instead of using 5 bits for APS ID signaling and 3 bits for APS parameter type signaling, 8 bits may be allocated for APS ID signaling (e.g., without increasing bit cost). An APS ID value range of 0 to 63 indicates that the APS includes parameters for the ALF, an ID value range of 64 to 95 indicates that the APS includes parameters for the reshaper, and 96 to 255 may be reserved for other parameter types, such as a scaling list. In another example, an APS ID value range of 0-31 could indicate that the APS contains parameters for ALF, an ID value range of 32-47 could indicate that the APS contains parameters for reshaper, and 48-255 could be reserved for other parameter types such as scaling lists. The advantage of this approach is that APS ID ranges can be assigned depending on the frequency of parameter changes for each tool. For example, ALF parameters may be expected to change more frequently than reshaper parameters. In such cases, a larger APS ID range could be used to indicate that the APS contains ALF parameters.

[0111] In a first embodiment, one or more of the aforementioned aspects may be implemented as follows: An ALF APS may be defined as an APS with aps_params_type equal to ALF_APS. A reshaper APS (or LMCS APS) may be defined as an APS with aps_params_type equal to MAP_APS. Exemplary SPS syntax and semantics are as follows: [Table 6] sps_reshaper_enabled_flag is set equal to 1 to specify that the reshaper is used in Coded Video Sequences (CVS). sps_reshaper_enabled_flag is set equal to 0 to specify that the reshaper is not used in CVS.

[0112] Example APS syntax and semantics are as follows: [Table 7]

[0113] aps_params_type specifies the type of APS parameters carried in the APS as specified in the table below. [Table 8]

[0114] An example tile group header syntax and semantics is as follows: [Table 9]

[0115] tile_group_alf_aps_id specifies the adaptation_parameter_set_id of the ALF APS referenced by the tile group. The TemporalId of an ALF APS NAL unit with adaptation_parameter_set_id equal to tile_group_alf_aps_id shall be less than or equal to the TemporalId of the coded tile group NAL unit. When multiple ALF APS with the same adaptation_parameter_set_id value are referenced by two or more tile groups of the same picture, multiple ALF APS with the same adaptation_parameter_set_id value shall have the same content.

[0116] tile_group_reshaper_enabled_flag is set equal to 1 to specify that the reshaper is enabled for the current tile group. tile_group_reshaper_enabled_flag is set to 0 to specify that the reshaper is not enabled for the current tile group. If tile_group_resharper_enable_flag is not present, the flag is inferred to be equal to 0. tile_group_reshaper_aps_id specifies the adaptation_parameter_set_id of the reshaper APS to which the tile group refers. The TemporalId of the reshaper APS NAL unit with adaptation_parameter_set_id equal to tile_group_reshaper_aps_id shall be less than or equal to the TemporalId of the coded tile group NAL unit. When multiple reshaper APSs with the same value of adaptation_parameter_set_id are referenced by two or more tile groups of the same picture, the multiple reshaper APSs with the same value of adaptation_parameter_set_id shall have the same content. tile_group_reshaper_chroma_residual_scale_flag is set equal to 1 to specify that chroma residual scaling is enabled for the current tile group. tile_group_reshaper_chroma_residual_scale_flag is set equal to 0 to specify that chroma residual scaling is not enabled for the current tile group. If tile_group_reshaper_chroma_residual_scale_flag is not present, the flag is inferred to be equal to 0.

[0117] An example reshaper data syntax and semantics is as follows: [Table 10]

[0118] reshaper_model_min_bin_idx specifies the minimum bin (or piece) index used in the reshaper construction process. The value of reshaper_model_min_bin_idx shall range from 0 to MaxBinIdx, inclusive. The value of MaxBinIdx shall be equal to 15. reshaper_model_delta_max_bin_idx specifies the maximum allowed bin (or piece) index, MaxBinIdx, minus the maximum bin index used in the reshaper construction process. The value of reshaper_model_max_bin_idx is set equal to MaxBinIdx - reshaper_model_delta_max_bin_idx. resharper_model_bin_delta_abs_cw_prec_minus1 + 1 specifies the number of bits used to represent the syntax element resharper_model_bin_delta_abs_CW[i]. reshaper_model_bin_delta_abs_CW[i] specifies the absolute delta codeword value for the i-th bin. The reshaper_model_bin_delta_abs_CW[i] syntax element is represented by reshaper_model_bin_delta_abs_cw_prec_minus1+1 bits. resharper_model_bin_delta_sign_CW_flag[i] specifies the sign of resharper_model_bin _delta_abs_CW[i].

[0119] In a second example embodiment, one or more of the above aspects may be implemented as follows: An exemplary SPS syntax and semantics is as follows: [Table 11]

[0120] The variables ALFEnabled and ReshaperEnabled are set as follows: ALFEnabled = sps_alf_enabled_flag, and ReshaperEnabled = sps_reshaper_enabled_flag.

[0121] Example APS syntax and semantics are as follows: [Table 12]

[0122] aps_params_type specifies the type of APS parameters carried in the APS as specified in the table below. [Table 13]

[0123] If not present, the value of aps_params_type is inferred as follows: If ALFEnabled, then aps_params_type is set equal to 0. Otherwise, aps_params_type is set equal to 1.

[0124] In a second example embodiment, one or more of the above aspects may be implemented as follows: An exemplary SPS syntax and semantics is as follows: [Table 14]

[0125] The adaptation_parameter_set_id provides an identifier for the APS for reference by other syntax elements. APSs can be shared between pictures and can be different for different tile groups within a picture. The values and descriptions of the variable APSParamsType are defined in the table below. [Table 15]

[0126] Exemplary tile group header semantics are as follows: tile_group_alf_aps_id specifies the adaptation_parameter_set_id of the ALF APS to which the tile group refers. The TemporalId of an ALF APS NAL unit with adaptation_parameter_set_id equal to tile_group_alf_aps_id shall be less than or equal to the TemporalId of the coded tile group NAL unit. The value of tile_group_alf_aps_id shall be in the range 0 to 63 (inclusive).

[0127] When multiple ALF APSs with the same adaptation_parameter_set_id value are referenced by multiple tile groups of the same picture, the multiple ALF APSs with the same adaptation_parameter_set_id value shall have the same content. tile_group _reshaper_aps_id specifies the adaptation_parameter_set_id of the reshaper APS referenced by the tile group. The TemporalId of the reshaper APS NAL unit with adaptation_parameter_set_id equal to tile_group_reshaper_aps_id shall be less than or equal to the TemporalId of the coded tile group NAL unit. The value of tile_group_reshaper_aps_id shall be in the range from 64 to 95 (inclusive). When multiple reshaper APSs with the same adaptation_parameter_set_id value are referenced by two or more tile groups of the same picture, the multiple reshaper APSs with the same adaptation_parameter_set_id value shall have the same content.

[0128] FIG. 7 is a schematic diagram of an exemplary video coding device 700. The video coding device 700 is suitable for implementing the disclosed examples / embodiments as described herein. The video coding device 700 includes a downstream port 720, an upstream port 750, and / or a transceiver unit (Tx / Rx) 710 including a transmitter and / or receiver for communicating data upstream and / or downstream over a network. The video coding device 700 also includes a processor 730 including a logic unit and / or central processing unit (CPU) for processing data and a memory 732 for storing data. The video coding device 700 may also include electrical, optical-electrical (OE), electro-optical (EO), and / or wireless communication components coupled to the upstream port 750 and / or downstream port 720 for communicating data over an electrical, optical, or wireless communication network. The video coding device 700 may also include input and / or output (I / O) devices 760 for communicating data to and from a user. The I / O devices 760 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices 760 may also include input devices such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0129] The processor 730 is implemented in hardware and software. The processor 730 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 730 is in communication with the downstream port 720, the Tx / Rx 710, the upstream port 750, and the memory 732. The processor 730 includes a coding module 714. The coding module 714 implements the disclosed embodiments described herein, such as methods 100, 800, and 900, which may employ the bitstream 500 and / or mechanism 600. The coding module 714 may also implement any other method / mechanism described herein. Additionally, the coding module 714 may implement the codec system 200, the encoder 300, and / or the decoder 400. For example, the coding module 714 may encode / decode a picture in a bitstream and encode / decode parameters related to slices of the picture in multiple APSs. In some examples, different types of parameters may be coded into different types of APSs. Furthermore, different types of APSs may be included in different NAL unit types. Such APS types may include ALF APSs, scaling list APSs, and / or LMCS APSs. Each APS may include an APS ID. The APS IDs of different APS types may increase sequentially over different value spaces. Furthermore, slices and / or pictures may reference corresponding slice headers and / or picture headers. Such headers may in turn reference APSs containing associated coding tools. Such APSs may be uniquely referenced by the APS ID and APS type. Such examples reduce redundant signaling of coding tool parameters and / or reduce bit usage for identifiers.Thus, coding module 714 provides video coding device 700 with additional functionality and / or coding efficiency when coding video data. Thus, coding module 714 improves the functionality of video coding device 700 and addresses problems inherent in video coding techniques. Furthermore, coding module 714 performs transformations of video coding device 700 into different states. Alternatively, coding module 714 is implemented as instructions stored in memory 732 and executed by processor 730 (e.g., as a computer program product stored on a non-transitory medium).

[0130] Memory 732 may include one or more memory types such as a disk, tape drive, solid state drive, read-only memory (ROM), random access memory, flash memory, ternary content addressable memory (TCAM), static random access memory (SRAM), etc. Memory 732 may also be used as an overflow data storage device to store programs when such programs are selected for execution and to store instructions and data retrieved during program execution.

[0131] 8 is a flowchart of an example method 800 for encoding a video sequence into a bitstream, such as bitstream 500, by using multiple APS types, such as ALF APS 512, scaling list APS 513, and / or LMCS APS 514. Method 800 may be used by an encoder, such as codec system 200, encoder 300, and / or video coding device 700, when performing method 100. Method 800 may also assign APS IDs to different types of APSs by using different value spaces according to mechanism 600.

[0132] Method 800 may begin when an encoder receives a video sequence including multiple pictures and determines, for example, based on user input, to encode the video sequence into a bitstream. The video sequence is partitioned into pictures / images / frames for further partitioning prior to encoding. In step 801, slices are encoded into the bitstream as part of the encoded video sequence. A slice may be encoded as part of a picture. Furthermore, a slice may be encoded by encoding CTUs and / or CUs included in the slice. Such CUs may be coded according to intra prediction and / or inter prediction. For example, the encoder may code the CUs of the slice. The encoder may then decode the coded slice and use a hypothetical reference decoder (HRD) to filter the decoded slice to improve the output quality of the slice.

[0133] In step 803, the encoder determines the type of parameters to be used to encode the slice. Such parameters may include parameters used by filters applied by the HRD to improve the quality of the encoded slice. These parameters may include ALF parameters, scaling list parameters, and LMCS parameters. Determining these parameters allows the encoder to determine the parameters to be used to apply to the slice at the decoder.

[0134] In step 805, the multiple parameter types are coded in multiple APSs. Specifically, the multiple parameter types are separated by type and included in the corresponding APS types. Thus, the multiple parameter types are included in the multiple APS types. For example, the multiple APS types may include an ALF type APS including ALF parameters, a scaling list type APS including scaling list parameters, and an LMCS type APS including LMCS parameters.

[0135] In step 807, an APS ID is assigned to each of the multiple APSs. This involves assigning APS IDs for APS types in order across multiple different value spaces. Thus, an APS ID is assigned to each of the multiple APSs so that each of the multiple APS types uses a separate value space for the APS ID. For example, each of the different value spaces may span a predefined range. Furthermore, the predefined range may be determined based on the APS type specified by the parameter type of the corresponding APS. In this manner, each APS may include an APS ID selected from a predefined range. Such predefined ranges may overlap, and therefore, the multiple different value spaces may also overlap. Furthermore, APS IDs may be assigned in different orders based on APS type. For example, a current APS may include a current APS ID selected from a predefined range across the current value space. The current APS ID is in turn related to a previous APS ID associated with a previous APS of the same type as the current APS. Also, the current APS ID is not related to another previous APS ID associated with an APS of a different type than the current APS, since the sequences for different APS types are different.

[0136] In step 809, the APS ID is encoded into multiple APSs. An APS type may also be encoded into each APS. For example, each APS may include an APS parameter type (e.g., aps_params_type) code set to a predefined value indicating the type of parameters included in the corresponding APS. Each APS may then be identified by a combination of the current APS type and the current APS ID. The bitstream is stored in memory. Upon request, the bitstream may then be communicated to a decoder, for example, via a transmitter.

[0137] 9 is a flowchart of an example method 900 for decoding a video sequence from a bitstream, such as bitstream 500, by using multiple APS types, such as ALF APS 512, scaling list APS 513, and / or LMCS APS 514. Method 900 may be used by a decoder, such as codec system 200, decoder 400, and / or video coding device 700, when performing method 100. Method 900 may also reference APSs based on APS IDs assigned according to mechanism 600, in which different types of APSs use APS IDs assigned according to different value spaces.

[0138] Method 900 may begin when a decoder receives a bitstream of coded data representing a video sequence, e.g., as a result of method 800. In step 901, the bitstream is received at the decoder. The bitstream includes coded slices and multiple APSs, each including multiple APS types. The multiple APS types may include an ALF type including ALF parameters, a scaling list type including scaling list parameters, and an LMCS type including LMCS parameters. For example, each APS may include an APS parameter type (e.g., aps_params_type) code set to a predefined value indicating the type of parameters included in each APS. Each APS includes an APS ID. The APS IDs for the APS types are assigned sequentially across multiple different value spaces. Thus, each of the multiple APS types uses a separate value space for the APS ID.

[0139] The different value spaces may overlap. For example, each of the different value spaces may span a predefined range. The predefined range may be determined based on the APS type. For example, the current APS may include a current APS ID selected from a predefined range that intersects the current value space. The current APS ID is related to a previous APS ID associated with a previous APS of the same type as the current APS. However, the current APS ID is not related to another previous APS ID associated with another previous APS of a different type than the current APS. Thus, each APS is identified by a combination of the current APS type and the current APS ID.

[0140] The decoder can obtain parameters from multiple APSs by using the APS ID and APS type. For example, the decoder can use a header associated with a slice that references an APS of each type. The decoder can then obtain parameters for the slice based on such reference. In step 903, the slice is decoded using parameters from the multiple APSs. In step 905, the decoder can forward the slice for display as part of the decoded video sequence.

[0141] 10 is a schematic diagram of an example system 1000 for coding a video sequence of images into a bitstream, such as bitstream 500, by using multiple APS types, such as ALF APS 512, scaling list APS 513, and / or LMCS APS 514. System 1000 may be implemented by an encoder and decoder, such as codec system 200, encoder 300, decoder 400, and / or video coding device 700. Furthermore, system 1000 may be used when implementing methods 100, 800, 900, and / or mechanism 600.

[0142] The system 1000 includes a video encoder 1002. The video encoder 1002 includes a determination module 1001 for determining multiple parameter types used to encode a slice. The video encoder 1002 further includes an encoding module 1003 for encoding the slice into a bitstream. The encoding module 1003 is further for encoding multiple parameter types in the multiple APSs into the bitstream by including the multiple parameter types in the multiple APS types. The encoding module 1003 is further for encoding APS IDs into the multiple APSs. The video encoder 1002 further includes an assignment module 1005 for assigning APS IDs to each of the APSs by assigning APS IDs for the APS types in order across multiple different value spaces. The video encoder 1002 further includes a storage module 1006 for storing a bitstream for communication to a decoder. The video encoder 1002 further includes a transmission module 1007 for transmitting a bitstream including the multiple APSs with the multiple parameter types to support decoding of the slice at the decoder. The video encoder 1002 may be further configured to perform any of the steps of the method 800.

[0143] The system 1000 also includes a video decoder 1010. The video decoder 1010 includes a receiving module 1011 for receiving a bitstream including slices and multiple APSs including multiple APS types, each APS including an APS ID, and the APS IDs for the APS types are assigned sequentially across multiple different value spaces. The video decoder 1010 further includes a decoding module 1013 for decoding slices using parameters from the multiple APSs. The video decoder 1010 further includes a transport module 1015 for transporting the slices for display as part of the decoded video sequence. The video decoder 1010 may be further configured to perform any of the steps of the method 900.

[0144] A first component is directly coupled to a second component when there are no intervening components, other than a line, trace, or another medium, between the first and second components. A first component is indirectly coupled to a second component when there are intervening components, other than a line, trace, or other medium, between the first and second components. The term "coupled" and variations thereof include both direct and indirect coupling. Use of the term "about," unless otherwise specified, means a range that includes ±10% of the subsequent number.

[0145] It should also be understood that the steps of the exemplary methods described herein do not necessarily have to be performed in the order described, and the order of the steps of such methods is merely exemplary. Similarly, additional steps may be included in such methods, and certain steps may be omitted or combined in methods consistent with various embodiments of the present disclosure.

[0146] While several embodiments have been provided in this disclosure, it will be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the disclosure. The examples are to be considered illustrative and not limiting, and the intention is not to be limited to the details provided herein. For example, various elements or components may be combined or integrated into another system, or certain features may be omitted or not implemented.

[0147] Furthermore, the techniques, systems, subsystems, and methods described and illustrated individually or separately in various embodiments may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations will be ascertainable by those skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

1. 1. A decoder-implemented method comprising: receiving a bitstream including syntax elements and video data, the video data including a plurality of pictures; Partitioning a picture of the plurality of pictures into a plurality of coded slices, wherein the syntax element includes a plurality of adaptation parameter set (APS) types associated with a coded slice of the plurality of coded slices, each APS including an APS identifier (ID) and an aps_extension_flag, and each of the plurality of APS types includes an APS identifier (ID) and an aps_extension_flag. a separate value space for a slice ID, the APS being a syntax structure including syntax elements that apply to a slice, the aps_extension_flag being set equal to 0 to specify that an aps_extension_data_flag syntax element is not present in the APS, the aps_extension_flag being set equal to 1 to specify that the aps_extension_data_flag syntax element is present in the APS, and the plurality of APS types including an adaptive loop filter (ALF) type including ALF parameters and an LMCS type including luma mapping with chroma scaling (LMCS) parameters; decoding the coded slice using the multiple APSs; A method wherein each of the distinct value spaces spans a predefined range, the predefined range being determined based on the APS type.

2. The method of claim 1 , wherein the distinct value spaces overlap.

3. The method according to any one of claims 1 to 2, wherein the plurality of APS types includes a scaling list type that includes a scaling list parameter.

4. 4. The method of claim 1, wherein each APS includes an APS parameter type (aps_params_type) code set to a predefined value indicating the type of parameters included in each APS.

5. 5. The method of claim 1, wherein the current APS comprises a current APS ID selected from a predefined range across a current space, the current APS ID relating to a previous APS ID associated with a previous APS of the same type as the current APS, and the current APS ID is not relating to another previous APS ID associated with another previous APS of a different type from the current APS.

6. The method of any one of claims 1 to 5, wherein each APS is identified by a combination of a current APS type and a current APS ID.

7. 1. A method implemented in an encoder, comprising: Partitioning a plurality of pictures into a plurality of slices; determining an application parameter set (APS) type of a plurality of parameters used to encode the coded slice; encoding a slice as a coded slice of the plurality of slices into a bitstream; encoding the APS types of the parameters into APSs of the bitstream; assigning an APS identifier (ID) to each of the plurality of APS types, wherein each of the plurality of APS types uses a separate value space for the APS ID; encoding each APS ID into the plurality of APSs; The APS is a syntax structure including syntax elements that apply to the coded slice, and each APS includes an aps_extension_flag, the aps_extension_flag being set equal to 0 to specify that an aps_extension_data_flag syntax element is not present in the APS, and the aps_extension_flag being set equal to 1 to specify that the aps_extension_data_flag syntax element is present in the APS; the plurality of APS types include an adaptive loop filter (ALF) type including ALF parameters and an LMCS type including luma mapping with chroma scaling (LMCS) parameters; A method wherein each of the distinct value spaces spans a predefined range, the predefined range being determined based on the APS type.

8. The method of claim 7 , wherein the distinct value spaces overlap.

9. The method according to any one of claims 7 to 8, wherein the plurality of APS types includes a scaling list type that includes a scaling list parameter.

10. A method according to any one of claims 7 to 9, wherein each APS includes an APS parameter type (aps_params_type) code set to a predefined value indicating the type of parameters contained in each APS.

11. 11. The method of claim 7, wherein the current APS comprises a current APS ID selected from a predefined range across a current value space, the current APS ID relating to a previous APS ID associated with a previous APS of the same type as the current APS, and the current APS ID is not relating to another previous APS ID associated with another previous APS of a different type than the current APS.

12. The method according to any one of claims 7 to 11, wherein each APS is identified by a combination of a current APS type and a current APS ID.

13. 13. A video coding device comprising: a processor; a receiver coupled to the processor; a memory coupled to the processor; and a transmitter coupled to the processor, wherein the processor, the receiver, the memory, and the transmitter are configured to perform the method of any one of claims 1 to 12.

14. 13. A non-transitory computer readable medium comprising a computer program for use by a video coding device, the computer program comprising computer executable instructions stored on the non-transitory computer readable medium such that, when executed by a processor, the computer program causes the video coding device to perform a method according to any one of claims 1 to 12.

15. receiving a bitstream including syntax elements and video data, the video data including a plurality of pictures; Partitioning a picture of the plurality of pictures into a plurality of coded slices, wherein the syntax element includes a plurality of adaptation parameter set (APS) types associated with a coded slice of the plurality of coded slices, each APS including an APS identifier (ID) and an aps_extension_flag, and each of the plurality of APS types includes an APS identifier (ID) and an aps_extension_flag. a receiving unit configured to: use a separate value space for a slice ID; the APS is a syntax structure including syntax elements that apply to a slice; the aps_extension_flag is set equal to 0 to specify that an aps_extension_data_flag syntax element is not present in the APS; the aps_extension_flag is set equal to 1 to specify that the aps_extension_data_flag syntax element is present in the APS; and the plurality of APS types include an adaptive loop filter (ALF) type that includes ALF parameters and an LMCS type that includes LMCS parameters; a decoding unit configured to decode the coded slice using the plurality of APSs; A decoder, wherein each of the distinct value spaces spans a predefined range, the predefined range being determined based on the APS type.

16. The decoder of claim 15, wherein the decoder is further configured to perform the method of any one of claims 1 to 6.

17. a partitioning unit configured to partition a plurality of pictures into a plurality of slices; a determining unit configured to determine an adaptive parameter set (APS) type of a plurality of parameters used to decode the slice; encoding the slice into a bitstream as a coded slice of the plurality of slices; encoding APS types of the plurality of parameters into a plurality of APSs in the bitstream; a coding unit configured to code each APS identifier (ID) into the plurality of APSs, the APS being a syntax structure including syntax elements that apply to the coded slice, each APS including an aps_extension_flag, the aps_extension_flag being set equal to 0 to specify that an aps_extension_data_flag syntax element is not present in the APS, and the aps_extension_flag being set equal to 1 to specify that the aps_extension_data_flag syntax element is present in the APS; an allocation unit configured to assign an APS ID to each of the plurality of APS types, wherein each of the plurality of APS types uses a separate value space for the APS ID; the plurality of APS types include an adaptive loop filter (ALF) type including ALF parameters and an LMCS type including luma mapping with chroma scaling (LMCS) parameters; An encoder wherein each of the distinct value spaces spans a predefined range, the predefined range being determined based on the APS type.

18. Encoder according to claim 17, wherein the encoder is further configured to perform the method according to any one of claims 7 to 12.

19. 1. A method for storing a bitstream, comprising: receiving the bitstream by at least one receiver; storing the bitstream in at least one memory, the bitstream including a plurality of application parameter set (APS) types associated with coded slices of a plurality of coded slices, each APS including an APS identifier (ID) and an aps_extension_flag, and each of the plurality of APS types including an APS identifier (ID) and an aps_extension_flag. aps_extension_flag is set equal to 0 to specify that an aps_extension_data_flag syntax element is not present in the APS, and the aps_extension_flag is set equal to 1 to specify that the aps_extension_data_flag syntax element is present in the APS; the plurality of APS types include an adaptive loop filter (ALF) type including ALF parameters and an LMCS type including luma mapping with chroma scaling (LMCS) parameters; each of the separate value spaces spans a predefined range, and the predefined range is determined based on the APS type.

20. A decoder comprising processing circuitry for implementing the method of any one of claims 1 to 6.

21. An encoder comprising processing circuitry for implementing the method of any one of claims 7 to 12.

22. A computer program comprising program code for performing the method according to any one of claims 1 to 12 when the computer program is run on a computer or processor.

23. A non-transitory computer readable medium carrying program code which, when executed by a computing device, causes said computing device to perform the method of any one of claims 1 to 12.

Images