Adaptive parameter set type in video coding

Abstract

A video coding mechanism is disclosed. In one embodiment, the mechanism includes receiving a code stream, the code stream including: a first adaptive parameter set (APS) including an adaptive loop filter (ALF) type, a network abstraction layer (NAL) unit, a second APS NAL unit including a zoom list type, a third APS NAL unit including a luma mapping chroma scaling (LMCS) type, and a slice, and the code stream includes a first adaptive parameter set (APS) including an adaptive loop filter (ALF) type, a second APS NAL unit including a zoom list type, a third APS NAL unit including a luma mapping chroma scaling (LMCS) type, and a slice. The mechanism also includes obtaining an ALF parameter from the first APS NAL unit, obtaining a zoom list parameter from the second APS NAL unit, and obtaining an LMCS parameter from the third APS NAL unit. The mechanism also includes decoding the slice using the ALF parameter, the scaling list parameter, and the LMCS parameter. The mechanism also includes forwarding the slice for display as part of a decoded video sequence.

Description

[0001] This application is a divisional application. The application number of the original application is 202080017445.4, and the original application date is February 26, 2020. The entire content of the original application is incorporated into this application by reference.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This patent application claims the benefit of U.S. Provisional Patent Application No. 62 / 811,358, filed by Wang Yekui et al. on February 27, 2019, entitled “Adaptation Parameter Set for Video Coding,” U.S. Provisional Patent Application No. 62 / 816,753, filed by Wang Yekui et al. on March 11, 2019, entitled “Adaptation Parameter Set for Video Coding,” and U.S. Provisional Patent Application No. 62 / 850,973, filed by Wang Yekui et al. on May 21, 2019, entitled “Adaptation Parameter Set for Video Coding,” which are incorporated herein by reference. Technical Field

[0004] The present invention generally relates to video coding, and more particularly, to efficient signaling of coding tool parameters for compressing video data in video coding. Background Art

[0005] Even if a video is relatively short, a large amount of video data is required to describe it, which can cause difficulties when the data is to be sent or otherwise transmitted across a communication network with limited bandwidth capacity. Therefore, video data is often compressed before being sent across modern telecommunications networks. Since memory resources may be limited, the size of the video can also be an issue when storing the video on a storage device. Video compression devices typically use software and / or hardware on the source side to encode the video data before sending or storing it, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination side by a video decompression device that decodes the video data. With limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase the compression ratio with little impact on image quality. Summary of the Invention

[0006] In one embodiment, the present invention includes a method implemented in a decoder, the method comprising: receiving, at a receiver of the decoder, a bitstream including a first adaptation parameter set (APS) network abstraction layer (NAL) unit of an adaptive loop filter (ALF) type associated with a coded slice; obtaining, at a processor of the decoder, ALF parameters from the first APS NAL unit; decoding the coded slice using the ALF parameters; and forwarding the decoding result for display as part of a decoded video sequence. The APS maintains data associated with multiple slices across multiple pictures. The present invention introduces the concept of multiple types of APSs containing different types of data. Specifically, an ALF-type APS (referred to as an ALF APS) may include ALF parameters. Furthermore, a scaling list APS (referred to as a scaling list APS) may include scaling list parameters. Furthermore, a luma mapping with chroma scaling (LMCS)-type APS (referred to as an LMCS APS) may include LMCS / shaper parameters. ALF APS, scaling list APS, and LMCSAPS can each be coded as a separate NAL type and therefore included in different NAL units. This allows changes in data in one type of APS (e.g., ALF parameters) to be made without redundantly decoding data in other types (e.g., LMCSA parameters). Providing multiple types of APSs improves coding efficiency, thereby reducing the use of network resources, memory resources, and / or processing resources at the encoder and decoder sides.

[0007] Optionally, in any of the above aspects, another implementation of the aspect is provided: the code stream also includes: a second APS NAL unit including a scaling list type, wherein the method also includes the processor obtaining scaling list parameters from the second APS NAL unit, and wherein the scaling list parameters are further used to decode the slice.

[0008] Optionally, in any of the above aspects, another implementation of the aspect is provided: the code stream also includes: a third APS NAL unit of LMCS type, wherein the method also includes the processor obtaining LMCS parameters from the third APS NAL unit, and wherein the LMCS parameters are further used to decode the slice.

[0009] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: the code stream further includes a slice header, wherein the slice refers to the slice header, and wherein the slice header refers to the first APS NAL unit, the second APS NAL unit and the third APS NAL unit.

[0010] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: each APS includes an APS parameter type (aps_params_type) code, and the APS parameter type (aps_params_type) code is set to a predefined value indicating the type of the parameter included in each APS.

[0011] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: the current APS includes a current APS identifier (ID) selected from a predefined range, wherein the current APS ID is associated with a previous APS ID, the previous APS ID is related to a previous APS of the same type as the current APS, and wherein the current APS ID is not associated with another previous APS ID, the other previous APS ID is related to another previous APS of a different type than the current APS.

[0012] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: each APS includes an APS ID selected from a predefined range, and wherein the predefined range is determined according to a parameter type of each APS.

[0013] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: each APS is identified by a combination of a parameter type and an APS ID.

[0014] Optionally, in any of the above aspects, another implementation of the aspect is provided: the code stream also includes a sequence parameter set (SPS), the sequence parameter set includes a flag, the flag is set to indicate that LMCS is enabled for the coded video sequence including the slice, and wherein the LMCS parameter of the third APS NAL unit is obtained based on the flag.

[0015] In one embodiment, the present invention comprises a method implemented in an encoder, the method comprising: a processor of the encoder encoding a slice into a bitstream as a coded slice as part of a coded video sequence; the processor determining adaptive loop filter (ALF) parameters associated with the coded slice; the processor encoding the ALF parameters into a first adaptation parameter set (APS) network abstraction layer (NAL) unit in the bitstream including an ALF type; and a memory coupled to the processor storing the bitstream for transmission to a decoder. The APS is configured to maintain data associated with multiple slices across multiple pictures. The present invention introduces the concept of multiple types of APSs containing different types of data. Specifically, an ALF-type APS (referred to as ALFAPS) may include ALF parameters. Furthermore, a scaling list APS (referred to as scaling list APS) may include scaling list parameters. Furthermore, a luma mapping with chromascaling (LMCS)-type APS (referred to as LMCS APS) may include LMCS / shaper parameters. ALF APS, scaling list APS, and LMCS APS can each be coded as a separate NAL type and therefore included in different NAL units. This allows changes in data in one type of APS (e.g., ALF parameters) to be made without redundantly decoding data in other types (e.g., LMCS parameters). Providing multiple types of APSs improves coding efficiency, thereby reducing the use of network resources, memory resources, and / or processing resources at the encoder and decoder sides.

[0016] Optionally, in any of the above aspects, another implementation method of the aspect is provided: further comprising: the processor determining a scaling list parameter for the slice; the processor encoding the scaling list parameter into a second APS NAL unit including a scaling list type in the code stream.

[0017] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: further comprising: the processor determining LMCS parameters for the slice; the processor encoding the LMCS parameters into a third APS NAL unit including an LMCS type in the code stream.

[0018] Optionally, in any of the above aspects, another implementation method of the aspect is provided: further including: the processor encoding a slice header into the code stream, wherein the slice references the slice header, and wherein the slice header references the first APS NAL unit, the second APS NAL unit and the third APS NAL unit.

[0019] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: each APS includes an aps_params_type code, and the aps_params_type code is set to a predefined value indicating the type of parameters included in each APS.

[0020] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: the current APS includes a current APS ID selected from a predefined range, wherein the current APS ID is associated with a previous APS ID, the previous APS ID is related to a previous APS of the same type as the current APS, and wherein the current APS ID is not associated with another previous APS ID, the other previous APS ID is related to another previous APS of a different type than the current APS.

[0021] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: each APS includes an APS ID selected from a predefined range, and wherein the predefined range is determined according to a parameter type of each APS.

[0022] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: each APS is identified by a combination of a parameter type and an APS ID.

[0023] Optionally, in any of the above aspects, another implementation of the aspect is provided: further comprising the processor encoding the SPS into the bitstream, wherein the SPS includes a flag, and the flag is set to indicate that LMCS is enabled for the encoded video sequence including the slice.

[0024] In one embodiment, the present invention includes a video decoding device, comprising: a processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, receiver, memory, and transmitter are used to perform the method according to any one of the above aspects.

[0025] In one embodiment, the present invention includes a non-transitory computer-readable medium, wherein the non-transitory computer-readable medium includes a computer program product for use by a video decoding device; the computer program product includes computer-executable instructions stored in the non-transitory computer-readable medium; when a processor executes the computer-executable instructions, the video decoding device performs the method according to any one of the above aspects.

[0026] In one embodiment, the present invention includes a decoder comprising: a receiving module for receiving a code stream comprising: a first APS NAL unit comprising an ALF type, a second APS NAL unit comprising a scaling list type, a third APS NAL unit comprising an LMCS type, and a slice; an acquisition module for acquiring ALF parameters from the first APS NAL unit, acquiring scaling list parameters from the second APS NAL unit, and acquiring LMCS parameters from the third APS NAL unit; a decoding module for decoding the slice using the ALF parameters, the scaling list parameters, and the LMCS parameters; and a forwarding module for forwarding the slice for display as part of a decoded video sequence.

[0027] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: the decoder is further used to execute the method according to any of the above aspects.

[0028] In one embodiment, the present invention includes an encoder comprising: a determination module for determining ALF parameters, scaling list parameters, and LMCS parameters for a slice; an encoding module for encoding the slice into a bitstream as part of an encoded video sequence, encoding the ALF parameters into a first APS NAL unit in the bitstream including an ALF type, encoding the scaling list parameters into a second APS NAL unit in the bitstream including a scaling list type, and encoding the LMCS parameters into a third APS NAL unit in the bitstream including an LMCS type; and a storage module for storing the bitstream for transmission to a decoder.

[0029] Optionally, in any of the above aspects, another implementation manner of the aspect is provided: the encoder is further used to execute the method provided in any of the above aspects.

[0030] For clarity, any of the above-described embodiments may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the invention.

[0031] These and other features will become more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] For a more complete understanding of the present invention, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0033] Figure 1 is a flow chart of an exemplary method of decoding a video signal;

[0034] Figure 2 is a schematic diagram of an exemplary encoding and decoding (codec) system for video coding;

[0035] Figure 3 is a schematic diagram of an exemplary video encoder;

[0036] Figure 4 is a schematic diagram of an exemplary video decoder;

[0037] Figure 5 is a schematic diagram of an exemplary code stream including multiple types of adaptation parameter sets (APS) including different types of decoding tool parameters;

[0038] Figure 6 is a schematic diagram of an exemplary mechanism for assigning APS identifiers (IDs) to different APS types over different value spaces;

[0039] Figure 7 is a schematic diagram of an exemplary video decoding apparatus;

[0040] Figure 8 is a flowchart of an exemplary method for encoding a video sequence into a bitstream by using multiple APS types;

[0041] Figure 9 is a flowchart of an exemplary method for decoding a video sequence from a bitstream by using multiple APS types;

[0042] Figure 10 is a schematic diagram of an exemplary system for decoding a video sequence of images in a codestream using multiple APS types. DETAILED DESCRIPTION

[0043] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The present invention should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0044] The following abbreviations are used in this document: adaptive loop filter (ALF), adaptation parameter set (APS), coding tree block (CTB), coding tree unit (CTU), coding unit (CU), coded video sequence (CVS), dynamic adaptive streaming over hypertext transfer protocol (DASH), intra-random access point (IRAP), joint video experts team (JVET), motion-constrained tile set (MCTS), maximum transfer unit (MTU), network abstraction layer (NAL), picture order count (POC), raw byte sequence payload (RBSP), sample adaptive offset (SAO), sequence parameter set (SPS), versatile video coding (VVC), and working draft [1]. draft, WD).

[0045] Many video compression techniques can be used to reduce video file sizes while minimizing data loss. For example, video compression techniques can include performing spatial (e.g., intra-frame) prediction and / or temporal (e.g., inter-frame) prediction to reduce or remove data redundancy in video sequences. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) can be partitioned into video blocks. Video blocks can also be referred to as treeblocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in intra-coded (I) slices within a picture are decoded using spatial prediction with reference to reference samples in neighboring blocks within the same picture. Video blocks in inter-coded unidirectionally predicted (P) or bidirectionally predicted (B) slices within a picture can be decoded using spatial prediction with reference to reference samples in neighboring blocks within the same picture, or using temporal prediction with reference to reference samples in other reference pictures. A picture (or image) can be referred to as a frame, and a reference picture can be referred to as a reference frame. Spatial prediction or temporal prediction produces a prediction block representing the image block. The residual data represents the pixel differences between the original image block and the prediction block. Thus, an inter-coded block is encoded based on a motion vector and residual data, where the motion vector points to a block of reference samples that make up the prediction block, and the residual data indicates the difference between the coded block and the prediction block. An intra-coded block is encoded based on an intra-coding mode and the residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain, thereby producing residual transform coefficients that can then be quantized. The quantized transform coefficients are initially arranged in a two-dimensional array. The quantized transform coefficients can be scanned to produce a one-dimensional vector of transform coefficients. Entropy coding can be applied to achieve further compression. This type of video compression technique is discussed in more detail below.

[0046] To ensure that coded videos can be correctly decoded, they are encoded and decoded according to corresponding video coding standards, including ITU-T H.261, ISO / IEC Motion Picture Experts Group (MPEG)-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, Advanced Video Coding (AVC) (also known as ITU-T H.264 or ISO / IEC MPEG-4 Part 10), and High Efficiency Video Coding (HEVC) (also known as ITU-T H.265 or MPEG-H Part 2). AVC includes extensions such as scalable video coding (SVC), multiview video coding (MVC), multiview video coding plus depth (MVC+D), and three-dimensional AVC (3D-AVC). HEVC includes extensions such as scalable HEVC (SHVC), multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The Joint Video Experts Team (JVET) of ITU-T and ISO / IEC has begun developing a video coding standard called Universal Video Coding (VVC).VVC is included in a working draft (WD), which includes JVET-M1001-v5 and JVET-M1002-v1, which respectively provide an algorithm description, an encoding end description, and reference software for VVC WD.

[0047] A video sequence is decoded using various decoding tools. The encoder selects parameters for the decoding tools with the goal of increasing compression with minimal quality loss when decoding the video sequence. The decoding tools can be related to different parts of the video at different scales. For example, some decoding tools are related at the video sequence level, some are related at the picture level, some are related at the slice level, and so on. The APS can be used to indicate information that can be shared by multiple pictures and / or multiple slices across different pictures. Specifically, the APS can carry adaptive loop filter (ALF) parameters. For various reasons, ALF information may not be suitable for indication at the sequence level in a sequence parameter set (SPS), at the picture level in a picture parameter set (PPS) or picture header, or at the slice level in a tile group header / slice header.

[0048] If ALF information is indicated in the SPS, the encoder must generate a new SPS and a new IRAP picture whenever the ALF information changes. IRAP pictures significantly reduce decoding efficiency. Therefore, placing ALF information in the SPS is particularly problematic for low-latency environments where IRAP pictures are not frequently used. Including ALF information in the SPS may also disable out-of-band transmission of the SPS. Out-of-band transmission refers to transmitting the corresponding data in a transport data stream different from the video code stream (for example, in the sample description or sample entry of the media file, in the Session Description Protocol (SDP) file, etc.). For similar reasons, indicating ALF information in the PPS may also be problematic. Specifically, including ALF information in the PPS may disable out-of-band transmission of the PPS. Indicating ALF information in the picture header may also be problematic. In some cases, the picture header may not be used. In addition, some ALF information can apply to multiple pictures. Therefore, indicating ALF information in the picture header will result in redundant information transmission, thereby wasting bandwidth. Indicating ALF information in a tile group header / slice header is also problematic because ALF information can apply to multiple pictures and therefore multiple slices / tile groups. Therefore, indicating ALF information in a slice / tile group header results in redundant information transmission, thus wasting bandwidth.

[0049] Based on the above, APS can be used to indicate ALF parameters. However, the video decoding system can use APS specifically for indicating ALF parameters. The syntax and semantics of APS are as follows:

[0050]

[0051] The adaptation_parameter_set_id provides an identifier for the APS for reference by other syntax elements. The APS can be shared across images and can be different in different block groups within an image. aps_extension_flag is set to 0 to indicate that the aps_extension_data_flag syntax element is not present in the APS RBSP syntax structure. aps_extension_flag is set to 1 to indicate that the aps_extension_data_flag syntax element is present in the APS RBSP syntax structure. aps_extension_data_flag can take any value. The presence and value of aps_extension_data_flag may not affect the conformance of the decoder to the profile specified in VVC. Decoders that conform to VVC may ignore all aps_extension_data_flag syntax elements.

[0052] The following is an example of the block group header syntax related to ALF parameters:

[0053]

[0054] The tile_group_alf_enabled_flag is set to 1 to indicate that the adaptive loop filter is enabled and can be applied to the luma (Y), blue chroma (Cb), or red chroma (Cr) color components in the tile group. The tile_group_alf_enabled_flag is set to 0 to indicate that the adaptive loop filter is disabled for all color components in the tile group. The tile_group_aps_id represents the adaptation_parameter_set_id of the APS referenced by the tile group. The TemporalId of the APS NAL unit with adaptation_parameter_set_id equal to tile_group_aps_id should be less than or equal to the TemporalId of the decoded tile group NAL unit. When multiple APSs with the same adaptation_parameter_set_id value are referenced by two or more tile groups of the same picture, the multiple APSs with the same adaptation_parameter_set_id value can include the same content.

[0055] The shaper parameters are used for the adaptive in-loop shaper video decoding tool, also known as luma mapping with chroma scaling (LMCS). The syntax and semantics of the SPS shaper are as follows:

[0056]

[0057] sps_reshaper_enabled_flag is set to 1 to indicate that the reshaper is used in the coded video sequence (CVS). sps_reshaper_enabled_flag is set to 0 to indicate that the reshaper is not used in the CVS.

[0058] The following are examples of the block group header / strip header shaper syntax and semantics:

[0059]

[0060] tile_group_reshaper_model_present_flag is set to 1 to indicate that tile_group_reshaper_model() is present in the tile group header. tile_group_reshaper_model_present_flag is set to 0 to indicate that tile_group_reshaper_model() is not present in the tile group header. When tile_group_reshaper_model_present_flag is not present, the inferred flag is equal to 0. tile_group_reshaper_enabled_flag is set to 1 to indicate that the reshaper is enabled for the current tile group. tile_group_reshaper_enabled_flag is set to 0 to indicate that the reshaper is not enabled for the current tile group. When tile_group_reshaper_enable_flag is not present, the inferred flag is equal to 0. tile_group_reshaper_chroma_residual_scale_flag is set to 1 to indicate that chroma residual scaling is enabled for the current tile group. tile_group_reshaper_chroma_residual_scale_flag is set to 0 to indicate that chroma residual scaling is not enabled for the current tile group. When tile_group_reshaper_chroma_residual_scale_flag is not present, the flag is inferred to be equal to 0.

[0061] The following are examples of the block group header / strip header shaper model syntax and semantics:

[0062]

[0063] reshape_model_min_bin_idx represents the minimum bin (or piece) index to be used during the reshaper construction process. The value of reshape_model_min_bin_idx can be in the range of 0 to MaxBinIdx (inclusive). The value of MaxBinIdx can be equal to 15. reshape_model_delta_max_bin_idx represents the maximum allowed bin (or piece) index MaxBinIdx minus the maximum bin index to be used during the reshaper construction process. The value of reshape_model_max_bin_idx is set to MaxBinIdx minus reshape_model_delta_max_bin_idx. reshaper_model_bin_delta_abs_cw_prec_minus1+1 represents the number of bits used to represent the syntax reshape_model_bin_delta_abs_CW[i]. reshape_model_bin_delta_abs_CW[i] represents the absolute delta codeword value of the i-th bin.

[0064] reshaper_model_bin_delta_sign_CW_flag[i] indicates the sign of reshape_model_bin_delta_abs_CW[i], as shown below. If reshape_model_bin_delta_sign_CW_flag[i] is equal to 0, then the corresponding variable RspDeltaCW[i] is positive. Otherwise (for example, if reshape_model_bin_delta_sign_CW_flag[i] is not equal to 0), the corresponding variable RspDeltaCW[i] is negative. When reshape_model_bin_delta_sign_CW_flag[i] does not exist, the flag is inferred to be 0. The variable RspDeltaCW[i] is set to (1–2*reshape_model_bin_delta_sign_CW[i])*reshape_model_bin_delta_abs_CW[i].

[0065] The variable RspCW[i] is derived as follows: The variable OrgCW is set to (1<<BitDepthY) / (MaxBinIdx+1). If reshaper_model_min_bin_idx <= i <= reshaper_model_max_bin_idx, RspCW[i] = OrgCW + RspDeltaCW[i]. Otherwise, RspCW[i] = 0. If the value of BitDepthY is equal to 10, the value of RspCW[i] should be in the range of 32 to 2*OrgCW–1. The variable InputPivot[i] is derived as follows, where i ranges from 0 to MaxBinIdx+1 (including the end values). InputPivot[i] = I*OrgCW. The variable ReshapePivot[i] (where i ranges from 0 to MaxBinIdx+1, including the end values) and the variables ScaleCoef[i] and InvScaleCoeff[i] (where i ranges from 0 to MaxBinIdx, including the end values) are derived as follows:

[0066]

[0067] }

[0068] The variable ChromaScaleCoef[i] is derived as follows, where i ranges from 0 to MaxBinIdx (including the end values):

[0069] ChromaResidualScaleLut

[64] = {16384,16384,16384,16384,16384,16384,16384,8192,8192,8192,8192,5461,5461,5461,5461,4096,4096,4096,4096,3277,3277,3277,3277,2731,2731,2731,2731,2341,2341,2341,2048,2048,2048,1820,1820,1820,1638,1638,1638,1638,1489,1489,1489,1489,1365,1365,1365,1365,1260,1260,1260,1260,1170,1170,1170,1170,1092,1092,1092,1092,1024,1024,1024,1024};

[0070] shiftC = 11

[0071] if(RspCW[i] == 0)

[0072] ChromaScaleCoef[i]=(1< <shiftC)

[0073] Otherwise (RspCW[i]!=0),

[0074] ChromaScaleCoef[i]=ChromaResidualScaleLut[RspCW[i]>>1]

[0075] The properties of the reshaper parameters can be described as follows. The size of the reshaper parameter set contained in the tile_group_reshaper_model() syntax structure is typically around 60 to 100 bits. The reshaper model is typically updated by the encoder once per second, which includes many frames. Furthermore, the parameters of an updated reshaper model are unlikely to be exactly the same as the parameters of an earlier instance of the reshaper model.

[0076] The above-mentioned video decoding system includes certain problems. First, such a system is only used to carry ALF parameters in APS. In addition, the shaper / LMCS parameters can be shared by multiple pictures and can include many variations.

[0077] Disclosed herein are various mechanisms for modifying APSs to support improved coding efficiency. In a first example, multiple types of APSs are disclosed. Specifically, an ALF type APS (referred to as ALFAPS) may include ALF parameters. Additionally, a scaling list type APS (referred to as scaling list APS) may include scaling list parameters. Additionally, an LMCS type APS (referred to as LMCS APS) may include LMCS / shaper parameters. The ALF APS, scaling list APS, and LMCS APS may each be decoded as separate NAL types and therefore included in different NAL units. In this way, changing the data in one type of APS (e.g., ALF parameters) does not result in redundant decoding of other types of data (e.g., LMCS parameters). Thus, providing multiple types of APSs improves coding efficiency, thereby reducing the use of network resources, memory resources, and / or processing resources on the encoder and decoder sides.

[0078] In a second example, each APS includes an APS identifier (ID). Furthermore, each APS type includes a separate value space for the corresponding APS ID. Such value spaces can overlap. Thus, an APS of a first type (e.g., an ALF APS) can include the same APS ID as an APS of a second type (e.g., an LMCS APS). This can be achieved by combining the APS parameter type and the APS ID to identify each APS. By allowing each APS type to include a different value space, the codec does not need to check for ID conflicts across APS types. Furthermore, by allowing the value spaces to overlap, the codec can avoid using larger ID values, thereby saving bits. Thus, using separate overlapping value spaces for different APS types improves decoding efficiency, thereby reducing the use of network resources, memory resources, and / or processing resources on the encoder and decoder sides.

[0079] In a third example, the LMCS parameters are included in the LMCSAPS. As described above, the LMCS / shaper parameters can change once per second. A video sequence may display 30 to 60 images per second. Therefore, the LMCS parameters may not change within 30 to 60 frames. Including the LMCS parameters in the LMCS APS significantly reduces redundant decoding of the LMCS parameters. The slice header and / or picture header associated with a slice can reference the associated LMCS APS. In this way, the LMCS parameters are only encoded when the LMCS parameters for the slice change. Therefore, using the LMCS APS to encode the LMCS parameters improves decoding efficiency, thereby reducing the use of network resources, memory resources, and / or processing resources on the encoder and decoder sides.

[0080] Figure 1 The following is a flow chart of an exemplary method 100 for decoding a video signal. Specifically, an encoder encodes a video signal. The encoding process uses various mechanisms to compress the video signal to reduce the video file size. The smaller file size allows the compressed video file to be delivered to the user while reducing the associated bandwidth overhead. The decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process is generally the inverse of the encoding process, ensuring that the video signal reconstructed by the decoder is consistent with the video signal on the encoder side.

[0081] In step 101, a video signal is input into an encoder. For example, the video signal can be an uncompressed video file stored in a memory. In another example, the video file can be captured by a video capture device (e.g., a camera) and encoded to support live streaming of the video. The video file can include an audio component and a video component. The video component includes a series of image frames. When these image frames are viewed in sequence, they give the visual effect of motion. These frames include pixels represented by light (referred to herein as luminance components (or luminance samples)) and pixels represented by color (referred to as chrominance components (or chrominance samples)). In some examples, these frames can also include depth values to support three-dimensional viewing.

[0082] In step 103, the video is segmented into blocks. Segmentation involves subdividing the pixels in each frame into square and / or rectangular blocks for compression. For example, in high efficiency video coding (HEVC) (also known as H.265 and MPEG-H Part 2), frames can be first divided into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64 pixels × 64 pixels). These CTUs include luma samples and chroma samples. A coding tree can be used to divide the CTU into blocks, and then recursively subdivide these blocks until a configuration that supports further encoding is obtained. For example, the luma component of a frame can be subdivided until each block includes relatively uniform luma values. Additionally, the chroma components of a frame can be subdivided until each block includes relatively uniform color values. Thus, the segmentation mechanism differs depending on the content of the video frame.

[0083] In step 105, various compression mechanisms are used to compress the image blocks segmented in step 103. For example, inter-frame prediction and / or intra-frame prediction can be used. Inter-frame prediction takes advantage of the fact that objects in a common scene often appear in consecutive frames. Therefore, blocks depicting objects in a reference frame do not need to be repeatedly depicted in adjacent frames. Specifically, an object (such as a table) can maintain a constant position across multiple frames. Therefore, the table can be depicted only once, and adjacent frames can re-reference the reference frame. Pattern matching mechanisms can be used to match objects across multiple frames. Furthermore, moving objects can be represented across multiple frames due to, for example, object movement or camera movement. In a specific example, a video can display a car moving across the screen across multiple frames. Motion vectors can be used to describe this movement. A motion vector is a two-dimensional vector that provides the offset from the coordinates of an object in a frame to the coordinates of the object in a reference frame. Therefore, inter-frame prediction can encode image blocks in a current frame as a set of motion vectors that indicate the offset between the image blocks in the current frame and the corresponding blocks in the reference frame.

[0084] Intra-frame prediction encodes blocks in a common frame. It exploits the fact that luma and chroma components tend to cluster within a frame. For example, a green patch in a part of a tree is often adjacent to similar green patches. Intra-frame prediction employs a variety of directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. These directional modes indicate that samples in the current block are similar or identical to samples in neighboring blocks in the corresponding direction. Planar mode indicates that a series of blocks in a row or column (e.g., a plane) can be interpolated based on neighboring blocks at the edge of the row. In effect, planar mode indicates smooth transitions in light or color across a row or column by using a relatively constant slope of varying values. DC mode is used for boundary smoothing, indicating that a block is similar or identical to the average of samples from all neighboring blocks relative to the angular direction of the directional prediction mode. Therefore, intra-frame prediction blocks can represent image blocks as various relative prediction mode values rather than actual values. Furthermore, inter-frame prediction blocks can represent image blocks as motion vector values rather than actual values. In either case, the prediction block may not accurately represent the image block in some cases. All differences are stored in the residual block. The residual block can be transformed to further compress the file.

[0085] In step 107, various filtering techniques can be used. In HEVC, filters are used according to an in-loop filtering scheme. The block-based prediction described above may produce blocky images on the decoder side. In addition, the block-based prediction scheme can encode the block and then reconstruct the coded block for later use as a reference block. The in-loop filtering scheme iteratively applies a noise suppression filter, a deblocking filter, an adaptive loop filter, and a sample adaptive offset (SAO) filter to the block / frame. These filters reduce such block artifacts so that the encoded file can be accurately reconstructed. In addition, these filters reduce artifacts in the reconstructed reference block, making it less likely that the artifacts will produce other artifacts in subsequent blocks encoded based on the reconstructed reference block.

[0086] In step 109, once the video signal has been segmented, compressed, and filtered, the resulting data is encoded into a bitstream. The bitstream includes the aforementioned data as well as any indicative data required to support proper reconstruction of the video signal at the decoder. For example, such data may include segmentation data, prediction data, residual blocks, and various flags that provide decoding instructions to the decoder. The bitstream can be stored in a memory so that it can be sent to the decoder upon request. The bitstream can also be broadcast and / or multicast to multiple decoders. Creating the bitstream is an iterative process. Therefore, steps 101, 103, 105, 107, and 109 can be performed continuously and / or simultaneously across multiple frames and blocks. Figure 1The shown order is presented for clarity and ease of description and is not intended to limit the video coding process to a specific order.

[0087] In step 111, the decoder receives the bitstream and begins the decoding process. Specifically, the decoder uses an entropy decoding scheme to convert the bitstream into corresponding syntax data and video data. In step 111, the decoder uses the syntax data in the bitstream to determine the frame segmentation. The segmentation should match the results of the block segmentation in step 103. The entropy encoding / decoding used in step 111 is described below. The encoder makes many choices during the compression process, such as selecting a block segmentation scheme from several possible options based on the spatial location of values in one or more input images. Indicating the exact choice can consume a large number of bits. As used herein, bits are binary values that are treated as variables (e.g., bit values that can vary depending on the context). Entropy encoding allows the encoder to discard any options that are clearly unsuitable for a particular situation, leaving a set of viable options. A codeword is then assigned to each viable option. The length of the codeword depends on the number of viable options (e.g., one bit for two options, two bits for three or four options, etc.). The encoder then encodes the codeword for the selected option. This approach reduces the size of the codeword because the codeword is as large as needed to uniquely indicate a selection from a small subset of available options, rather than a selection from a potentially large set of all possible options. The decoder then decodes the selection by determining the set of available options in a similar manner to the encoder. By determining the set of available options, the decoder can read the codeword and determine the selection made by the encoder.

[0088] In step 113, the decoder performs block decoding. Specifically, the decoder performs an inverse transform to generate a residual block. The decoder then uses the residual block and the corresponding prediction block to reconstruct the image block based on the segmentation. The prediction block may include the intra-frame prediction block and the inter-frame prediction block generated by the encoder in step 105. The reconstructed image block is then placed in the frame of the reconstructed video signal based on the segmentation data determined in step 111. The syntax used for step 113 can also be indicated in the bitstream through the entropy coding described above.

[0089] In step 115, filtering is performed on the frame of the reconstructed video signal in a manner similar to step 107 on the encoder side. For example, a noise suppression filter, a deblocking filter, an adaptive loop filter, and a SAO filter may be applied to the frame to remove blocking artifacts. Once the frame has been filtered, in step 117, the video signal may be output to a display for viewing by an end user.

[0090] Figure 21 is a schematic diagram of an exemplary encoding and decoding (CODEC) system 200 for video decoding. Specifically, CODEC system 200 is capable of implementing method 100. CODEC system 200 broadly describes the components used in an encoder and a decoder. CODEC system 200 receives a video signal and segments the video signal, as described in steps 101 and 103 of method 100, thereby generating segmented video signals 201. Then, when acting as an encoder, CODEC system 200 compresses segmented video signal 201 into an encoded bitstream, as described in steps 105, 107, and 109 of method 100. When acting as a decoder, CODEC system 200 generates an output video signal from the bitstream, as described in conjunction with steps 111, 113, 115, and 117 of method 100. The codec system 200 includes a general decoder control component 211, a transform scaling and quantization component 213, an intra-frame estimation component 215, an intra-frame prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header formatting and context adaptive binary arithmetic coding (CABAC) component 231. These components are coupled as shown. Figure 2 In FIG, black lines represent the movement of data to be encoded / decoded, while dashed lines represent the movement of control data that controls the operation of other components. The components in codec system 200 may all be present in an encoder. A decoder may include a subset of the components of codec system 200. For example, a decoder may include an intra-frame prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded image buffer component 223. These components are described below.

[0091] The segmented video signal 201 is a captured video sequence that has been segmented into pixel blocks by a coding tree. The coding tree uses various partitioning modes to subdivide the pixel blocks into smaller pixel blocks. These blocks can then be further subdivided into smaller blocks. These blocks can be called nodes on the coding tree. Larger parent nodes are divided into smaller child nodes. The number of times a node is subdivided is called the depth of the node / coding tree. In some cases, the divided blocks can be included in a coding unit (CU). For example, a CU can be a sub-part of a CTU, including a luminance block, one or more red difference chrominance (Cr) blocks and one or more blue difference chrominance (Cb) blocks and the corresponding syntax instructions of the CU. The partitioning mode can include a binary tree (BT), a triple tree (TT) and a quad tree (QT), which are used to partition the node into two, three or four child nodes of different shapes according to the partitioning mode adopted. The segmented video signal 201 is forwarded to the general decoder control component 211, the transform scaling and quantization component 213, the intra estimation component 215, the filter control analysis component 227 and the motion estimation component 221 for compression.

[0092] The universal decoder control component 211 is used to make decisions related to encoding images in a video sequence into a bitstream based on application constraints. For example, the universal decoder control component 211 manages the optimization of bitrate / bitstream size relative to reconstruction quality. These decisions can be made based on storage space / bandwidth availability and image resolution requirements. The universal decoder control component 211 also manages buffer utilization based on transmission speed to mitigate buffer underruns and overruns. To address these issues, the universal decoder control component 211 manages segmentation, prediction, and filtering performed by other components. For example, the universal decoder control component 211 can dynamically increase compression complexity to improve resolution and increase bandwidth utilization, or decrease compression complexity to reduce resolution and bandwidth utilization. Thus, the universal decoder control component 211 controls other components of the codec system 200 to balance video signal reconstruction quality with bitrate. The universal decoder control component 211 generates control data used to control the operation of other components. The control data is also forwarded to the header formatting and CABAC component 231 for encoding into the codestream, indicating parameters for decoding by the decoder.

[0093] The segmented video signal 201 is also sent to the motion estimation component 221 and the motion compensation component 219 for inter-frame prediction. The frames or slices of the segmented video signal 201 can be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter-frame prediction decoding on the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. The codec system 200 can perform multiple decoding processes to select an appropriate decoding mode for each block of video data, etc.

[0094] The motion estimation component 221 and the motion compensation component 219 can be highly integrated, but are described separately for conceptual purposes. The motion estimation performed by the motion estimation component 221 is the process of generating motion vectors, where these motion vectors are used to estimate the motion of a video block. For example, a motion vector can represent the displacement of the decoding object relative to a prediction block. A prediction block is a block that is found to closely match the block to be decoded in terms of pixel difference. A prediction block can also be referred to as a reference block. This pixel difference can be determined using the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. HEVC uses several decoding objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU can be divided into multiple CTBs, which can then be divided into multiple CBs included in a CU. A CU can be encoded as a prediction unit (PU) including prediction data and / or a transform unit (TU) including the transform residual data of the CU. The motion estimation component 221 uses rate-distortion analysis as part of the rate-distortion optimization process to generate motion vectors, PUs, and TUs. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for the current block / frame, and may select a reference block, motion vector, etc. having an optimal rate-distortion characteristic. The optimal rate-distortion characteristic may balance the quality of video reconstruction (e.g., the amount of data lost due to compression) and decoding efficiency (e.g., the size of the final encoded image).

[0095] In some examples, the codec system 200 can calculate values for sub-integer pixel positions of a reference image stored in the decoded picture buffer component 223. For example, the video codec system 200 can interpolate quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference image. Accordingly, the motion estimation component 221 can perform motion search relative to integer-pixel positions and fractional pixel positions and output motion vectors with fractional pixel precision. The motion estimation component 221 calculates motion vectors for PUs of video blocks in inter-coded slices by comparing the positions of the PUs with the positions of prediction blocks of the reference image. The motion estimation component 221 outputs the calculated motion vectors as motion data to the header formatting and CABAC component 231 for encoding and outputs the motion to the motion compensation component 219.

[0096] The motion compensation performed by the motion compensation component 219 may involve obtaining or generating a prediction block based on the motion vector determined by the motion estimation component 221. Furthermore, in some examples, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, the motion compensation component 219 may locate the prediction block to which the motion vector points. The pixel values of the prediction block are then subtracted from the pixel values of the decoded current video block to form pixel difference values, thereby forming a residual video block. Typically, the motion estimation component 221 performs motion estimation with respect to the luma component, and the motion compensation component 219 applies the motion vector calculated based on the luma component to the chroma and luma components. The prediction block and the residual block are forwarded to the transform, scaling, and quantization component 213.

[0097] The segmented video signal 201 is also sent to an intra estimation component 215 and an intra prediction component 217. Like the motion estimation component 221 and the motion compensation component 219, the intra estimation component 215 and the intra prediction component 217 can be highly integrated, but are described separately for conceptual purposes. The intra estimation component 215 and the intra prediction component 217 perform intra prediction on the current block relative to blocks in the current frame, replacing the inter prediction performed between frames by the motion estimation component 221 and the motion compensation component 219 as described above. Specifically, the intra estimation component 215 determines an intra prediction mode to encode the current block. In some examples, the intra estimation component 215 selects an appropriate intra prediction mode from a plurality of tested intra prediction modes to encode the current block. The selected intra prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

[0098] For example, the intra-frame estimation component 215 performs a rate-distortion analysis on various tested intra-frame prediction modes to calculate rate-distortion values and selects the intra-frame prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between the coded block and the original uncoded block that is encoded to generate the coded block, as well as the code rate (e.g., the number of bits) used to generate the coded block. The intra-frame estimation component 215 calculates a ratio based on the distortion and rate of the various coded blocks to determine the intra-frame prediction mode that exhibits the best rate-distortion value for the block. In addition, the intra-frame estimation component 215 can be used to decode the depth block of the depth map using a depth modeling mode (DMM) according to rate-distortion optimization (RDO).

[0099] When implemented on an encoder, the intra prediction component 217 can generate a residual block from the prediction block according to the selected intra prediction mode determined by the intra estimation component 215, or when implemented on a decoder, read the residual block from the codestream. The residual block contains the difference between the predicted block and the original block, represented as a matrix. The residual block is then forwarded to the transform, scaling, and quantization component 213. The intra estimation component 215 and the intra prediction component 217 can operate on both luma and chroma components.

[0100] The transform scaling and quantization component 213 is used to further compress the residual block. The transform scaling and quantization component 213 performs a transform such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform on the residual block to produce a video block comprising residual transform coefficient values. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used. These transforms can convert the residual information from the pixel domain to a transform domain (e.g., the frequency domain). The transform scaling and quantization component 213 is also used to scale the transformed residual information based on frequency, etc. This scaling involves applying a scaling factor to the residual information to quantize different frequency information at different granularities, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also used to quantize the transform coefficients to further reduce the bit rate. This quantization process can reduce the bit depth associated with some or all coefficients. The degree of quantization can be modified by adjusting the quantization parameter. In some examples, the transform scaling and quantization component 213 can then scan the matrix comprising the quantized transform coefficients. The quantized transform coefficients are forwarded to the header formatting and CABAC component 231 for encoding into the codestream.

[0101] The scaling and inverse transform component 229 performs the inverse operations of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 uses inverse scaling, inverse transforming, and / or inverse quantization to reconstruct a residual block in the pixel domain, for example, for later use as a reference block, which may become a prediction block for another current block. The motion estimation component 221 and / or the motion compensation component 219 can calculate a reference block for motion estimation of subsequent blocks / frames by adding the residual block back to the corresponding prediction block. Filters are applied to the reconstructed reference block to reduce artifacts generated during scaling, quantization, and transforming. Such artifacts may cause inaccurate predictions (and generate other artifacts) when predicting subsequent blocks.

[0102] The filter control analysis component 227 and the in-loop filter component 225 apply filters to the residual block and / or the reconstructed image block. For example, the transformed residual block of the scaling and inverse transform component 229 can be merged with the corresponding prediction block of the intra-frame prediction component 217 and / or the motion compensation component 219 to reconstruct the original image block. The filter can then be applied to the reconstructed image block. In some examples, the filter can be applied to the residual block in turn. Figure 2 The other components in the, filter control analysis component 227 and the in-loop filter component 225 are highly integrated and can be implemented together, but are described separately for conceptual purposes. The filters applied to the reconstructed reference block are applied to specific spatial regions, and these filters include multiple parameters to adjust the way in which these filters are used. The filter control analysis component 227 analyzes the reconstructed reference block to determine where these filters need to be used and sets the corresponding parameters. Such data is forwarded as filter control data to the header formatting and CABAC component 231 for encoding. The in-loop filter component 225 uses these filters based on the filter control data. These filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Such filters can be applied in the spatial / pixel domain (for example, for reconstructed pixel blocks) or the frequency domain according to the examples.

[0103] When operating as an encoder, the filtered reconstructed image blocks, residual blocks, and / or prediction blocks are stored in the decoded image buffer component 223 for later use in motion estimation, as described above. When operating as a decoder, the decoded image buffer component 223 stores the reconstructed and filtered blocks and forwards them to the display as part of the output video signal. The decoded image buffer component 223 can be any storage device capable of storing prediction blocks, residual blocks, and / or reconstructed image blocks.

[0104] The header formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes this data into an encoded bitstream to be sent to the decoder. Specifically, the header formatting and CABAC component 231 generates various headers to encode control data (such as overall control data and filter control data). In addition, prediction data (including intra-frame prediction) and motion data, as well as residual data in the form of quantized transform coefficient data, are encoded into the bitstream. The final bitstream includes all the information required for the decoder to reconstruct the original segmented video signal 201. This information may also include an intra-frame prediction mode index table (also called a codeword mapping table), definitions of the coding contexts of various blocks, indications of the most likely intra-frame prediction mode, indications of segmentation information, etc. This data can be encoded using entropy coding. For example, the information may be encoded using context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding techniques. After entropy coding, the encoded bitstream may be sent to another device (e.g., a video decoder) or archived for later transmission or retrieval.

[0105] Figure 3 FIG2 is a block diagram of an exemplary video encoder 300. Video encoder 300 can be used to implement the encoding functionality of codec system 200 and / or perform steps 101, 103, 105, 107, and / or 109 of method 100. Encoder 300 segments an input video signal to generate segmented video signals 301, wherein segmented video signals 301 are substantially similar to segmented video signals 201. Segmented video signals 301 are then compressed by components of encoder 300 and encoded into a bitstream.

[0106] Specifically, the segmented video signal 301 is forwarded to the intra-frame prediction component 317 for intra-frame prediction. The intra-frame prediction component 317 can be substantially similar to the intra-frame estimation component 215 and the intra-frame prediction component 217. The segmented video signal 301 is also forwarded to the motion compensation component 321 for inter-frame prediction based on the reference blocks in the decoded picture buffer component 323. The motion compensation component 321 can be substantially similar to the motion estimation component 221 and the motion compensation component 219. The prediction blocks and residual blocks from the intra-frame prediction component 317 and the motion compensation component 321 are forwarded to the transform and quantization component 313 for transform and quantization of the residual blocks. The transform and quantization component 313 can be substantially similar to the transform, scaling, and quantization component 213. The transformed and quantized residual blocks and the corresponding prediction blocks (along with related control data) are forwarded to the entropy coding component 331 for encoding into the bitstream. The entropy coding component 331 can be substantially similar to the header formatting and CABAC component 231.

[0107] The transformed and quantized residual block and / or the corresponding prediction block are also forwarded from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into a reference block for use by the motion compensation component 321. The inverse transform and quantization component 329 can be substantially similar to the scaling and inverse transform component 229. According to an example, an in-loop filter in an in-loop filter component 325 is also applied to the residual block and / or the reconstructed reference block. The in-loop filter component 325 can be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 can include multiple filters, as described in conjunction with the in-loop filter component 225. The filtered block is then stored in the decoded image buffer component 323 for use as a reference block by the motion compensation component 321. The decoded image buffer component 323 can be substantially similar to the decoded image buffer component 223.

[0108] Figure 4 FIG4 is a block diagram of an example video decoder 400. Video decoder 400 can be used to implement the decoding functionality of codec system 200 and / or perform steps 111, 113, 115, and / or 117 of method 100. For example, decoder 400 receives a bitstream from encoder 300 and generates a reconstructed output video signal based on the bitstream for display to an end user.

[0109] The bitstream is received by entropy decoding component 433. Entropy decoding component 433 is used to perform entropy decoding schemes such as CAVLC, CABAC, SBAC, PIPE encoding, or other entropy encoding techniques. For example, entropy decoding component 433 can use header information to provide context for interpreting other data encoded as codewords in the bitstream. The decoded information includes any information required to decode the video signal, such as overall control data, filter control data, segmentation information, motion data, prediction data, and quantized transform coefficients in the residual block. The quantized transform coefficients are forwarded to inverse transform and quantization component 429 for reconstruction into the residual block. Inverse transform and quantization component 429 can be similar to inverse transform and quantization component 329.

[0110] The reconstructed residual block and / or prediction block are forwarded to the intra-frame prediction component 417 for reconstruction into an image block based on the intra-frame prediction operation. The intra-frame prediction component 417 can be similar to the intra-frame estimation component 215 and the intra-frame prediction component 217. Specifically, the intra-frame prediction component 417 uses the prediction mode to locate the reference block in the frame and applies the residual block to the result to reconstruct the intra-frame predicted image block. The reconstructed intra-frame predicted image block and / or residual block and the corresponding inter-frame prediction data are forwarded to the decoded image buffer component 423 via the in-loop filter component 425. The decoded image buffer component 423 and the in-loop filter component 425 can be substantially similar to the decoded image buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, residual block, and / or prediction block and stores such information in the decoded image buffer component 423. The reconstructed image block from the decoded image buffer component 423 is forwarded to the motion compensation component 421 for inter-frame prediction. Motion compensation component 421 can be substantially similar to motion estimation component 221 and / or motion compensation component 219. Specifically, motion compensation component 421 uses the motion vector of the reference block to generate a prediction block and applies the residual block to the result to reconstruct the image block. The resulting reconstructed block can also be forwarded to decoded image buffer component 423 via in-loop filter component 425. Decoded image buffer component 423 continues to store other reconstructed image blocks. These reconstructed image blocks can be reconstructed into frames using segmentation information. These frames can also be arranged in a sequence. The sequence is output to a display as a reconstructed output video signal.

[0111] Figure 5 FIG1 is a schematic diagram of an exemplary codestream 500 including multiple types of APSs including different types of decoding tool parameters. For example, codestream 500 may be generated by codec system 200 and / or encoder 300 and decoded by codec system 200 and / or decoder 400. In another example, in step 109 of method 100, codestream 500 may be generated by the encoder and used by the decoder in step 111.

[0112] Codestream 500 includes a sequence parameter set (SPS) 510, multiple picture parameter sets (PPSs) 511, multiple ALF APSs 512, multiple scaling list APSs 513, multiple LMCSAPSs 514, multiple slice headers 515, and picture data 520. SPS 510 contains sequence data common to all pictures in the video sequence included in codestream 500. This data may include picture size, bit depth, decoding tool parameters, rate limits, and so on. PPS 511 contains parameters that apply to the entire picture. Therefore, each picture in the video sequence can reference PPS 511. It should be noted that while each picture references PPS 511, in some examples, a single PPS 511 may contain data for multiple pictures. For example, multiple similar pictures may be decoded according to similar parameters. In this case, a single PPS 511 may contain data for these similar pictures. PPS 511 may indicate decoding tools, such as slices, quantization parameters, and offsets, that may be used in the corresponding picture. The slice header 515 contains parameters specific to each slice in the image. Therefore, there can be one slice header 515 for each slice in the video sequence. The slice header 515 can include information such as slice type, picture order count (POC), reference picture list, prediction weight, partition entry point, deblocking parameters, etc. It should be noted that in some contexts, the slice header 515 can also be referred to as a partition group header.

[0113] An APS is a syntax structure that includes syntax elements that apply to one or more pictures 521 and / or slices 523. In the example shown, APSs can be categorized into multiple types. ALF APS 512 is an ALF-type APS that includes ALF parameters. The ALF is a block-based adaptive filter that includes a transfer function controlled by a variable parameter and uses feedback from a feedback loop to modify the transfer function. Furthermore, the ALF is used to correct decoding artifacts (e.g., errors) that occur due to block-based decoding. An adaptive filter is a linear filter that has a transfer function controlled by a variable parameter that can be controlled by an optimization algorithm, such as an RDO process, running on the encoder side. Therefore, the ALF parameters included in the ALF APS 512 can include variable parameters selected by the encoder so that the filter removes block-based decoding artifacts during decoding on the decoder side.

[0114] The scaling list APS 513 is a scaling list type APS that includes scaling list parameters. As described above, the current block is decoded according to an inter-frame prediction or intra-frame prediction that produces a residual. The residual is the difference between the luminance and / or chrominance values of the block and the corresponding values of the predicted block. A transform is then applied to the residual to convert the residual into a transform coefficient (which is smaller than the residual value). Encoding high-definition and / or ultra-high-definition content may result in an increase in residual data. When applied to such data, a simple transform process may generate a large amount of quantization noise. Therefore, the scaling list parameters included in the scaling list APS 513 may include weighting parameters, which can be used to scale the transform matrix to take into account changes in display resolution and / or acceptable levels of quantization noise in the resulting decoded video image.

[0115] LMCS APS 514 is an LMCS-type APS that includes LMCS parameters, also known as shaper parameters. The human visual system is less able to distinguish color differences (e.g., chroma) than light differences (e.g., luma). Therefore, some video systems use a chroma subsampling mechanism to compress video data by reducing the resolution of chroma values without adjusting the corresponding luma values. One problem with this mechanism is that the associated interpolation during decoding may produce interpolated chroma values that are incompatible with the corresponding luma values at some locations. This can cause color artifacts at these locations, which should be corrected by corresponding filters. This is complicated by the luma mapping mechanism. Luma mapping is the process of remapping the decoded luma component within the dynamic range of the input luma signal (e.g., according to a piecewise linear function). This compresses the luma component. The LMCS algorithm scales the compressed chroma values according to the luma mapping to remove artifacts associated with chroma subsampling. Therefore, the LMCS parameters included in LMCSAPS 514 represent chroma scaling that takes luma mapping into account. The LMCS parameters are determined by the encoder and can be used by the decoder to filter out artifacts caused by chroma subsampling when luma mapping is used.

[0116] Image data 520 includes video data encoded using inter-frame prediction and / or intra-frame prediction, as well as corresponding transformed and quantized residual data. For example, a video sequence includes multiple images 521 decoded as image data. Image 521 is a single frame of the video sequence and is therefore typically displayed as a single unit when the video sequence is displayed. However, partial images can be displayed to implement certain technologies, such as virtual reality and picture-in-picture. Each image 521 references a PPS 511. Image 521 is divided into strips 523, which can be defined as a horizontal portion of image 521. For example, a strip 523 may encompass a portion of the height of image 521 and the entire width of image 521. In other cases, image 521 may be divided into columns and rows, and strips 523 may be included in the rectangular portion of image 521 created by these columns and rows. In some systems, strips 523 are subdivided into blocks. In other systems, strips 523 are referred to as block groups containing blocks. The slice 523 and / or the partition group of the partition refer to the slice header 515. The slice 523 is further divided into coding tree units (CTUs). The CTU is further divided into coding blocks according to the coding tree. The coding blocks can then be encoded / decoded according to the prediction mechanism.

[0117] The picture 521 and / or slice 523 can directly or indirectly reference the ALF APS 512, scaling list APS 513, and / or LMCS APS 514 containing related parameters. For example, the slice 523 can reference the slice header 515. In addition, the picture 521 can reference the corresponding picture header. The slice header 515 and / or picture header can reference the ALF APS 512, scaling list APS 513, and / or LMCS APS 514 containing parameters used in decoding the related slice 523 and / or picture 521. In this way, the decoder can obtain decoding tool parameters related to the slice 523 and / or picture 521 based on the header reference related to the corresponding slice 523 and / or picture 521.

[0118] The codestream 500 is decoded into video coding layer (VCL) NAL units 535 and non-VCL NAL units 531. NAL units are coded data units that can be sized to fit into the payload of a single data packet for transmission over a network. VCL NAL units 535 are NAL units that contain coded video data. For example, each VCL NAL unit 535 may contain a slice 523 and / or a partitioned group of data, CTUs, and / or coded blocks. Non-VCL NAL units 531 are NAL units that contain support syntax but do not contain coded video data. For example, non-VCL NAL units 531 may contain SPS 510, PPS 511, APS, slice header 515, etc. Thus, the decoder receives the codestream 500 in discrete VCL NAL units 535 and non-VCL NAL units 531. An access unit is a group of VCL NAL units 535 and / or non-VCL NAL units 531 that includes data sufficient to decode a single picture 521 .

[0119] In some examples, the ALF APS 512, scaling list APS 513, and LMCS APS 514 are each assigned to a separate non-VCL NAL unit 531 type. In this case, the ALF APS 512, scaling list APS 513, and LMCS APS 514 are included in the ALF APS NAL unit 532, scaling list APS NAL unit 533, and LMCS APS NAL unit 534, respectively. Thus, the ALF APS NAL unit 532 contains ALF parameters, which remain valid until another ALF APS NAL unit 532 is received. The scaling list APS NAL unit 533 contains scaling list parameters, which remain valid until another scaling list APS NAL unit 533 is received. Furthermore, the LMCS APS NAL unit 534 contains LMCS parameters, which remain valid until another LMCS APS NAL unit 534 is received. This eliminates the need to send a new APS each time an APS parameter changes. For example, a change in the LMCS parameters may result in an additional LMCS APS 514, but not an additional ALF APS 512 or scaling list APS 513. Thus, by dividing the APSs into different NAL unit types based on parameter type, redundant indication of unrelated parameters is avoided. Consequently, dividing the APSs into different NAL unit types improves decoding efficiency, thereby reducing the use of processor resources, memory resources, and / or network resources at the encoder and decoder sides.

[0120] Furthermore, the slice 523 and / or the picture 521 may directly or indirectly reference an ALF APS 512, an ALF APS NAL unit 532, a scaling list APS 513, a scaling list APS NAL unit 533, an LMCS APS 514, and / or an LMCS APS NAL unit 534, which contain decoding tool parameters for decoding the slice 523 and / or the picture 521. For example, each APS may include an APS ID 542 and a parameter type 541. The APS ID 542 is a value (e.g., a number) that identifies the corresponding APS. The APS ID 542 may include a predefined number of bits. Thus, the APS ID 542 may increase (e.g., by 1) according to a predefined sequence and may be reset to a minimum value (e.g., 0) once the sequence reaches the end of a predefined range. The parameter type 541 indicates the type of parameter included in the APS (e.g., ALF, scaling list, and / or LMCS). For example, the parameter type 541 may include an APS parameter type (aps_params_type) code set to a predefined value indicating the parameter type included in each APS. Thus, the parameter type 541 may be used to distinguish between the ALF APS 512, the zoom list APS 513, and the LMCS APS 514. In some examples, each of the ALF APS 512, the zoom list APS 513, and the LMCS APS 514 may be uniquely identified by combining the parameter type 541 and the APS ID 542. For example, each APS type may include a separate value space for a corresponding APS ID 542. Thus, each APS type may include APS IDs 542 that increase sequentially based on a preceding APS of the same type. However, the APS ID 542 of a first APS type may not be related to the APS ID 542 of a preceding APS of a different second APS type. Thus, the APS IDs 542 of different APS types may include overlapping value spaces. For example, in some cases, a first type of APS (e.g., ALFAPS) may include the same APS ID 542 as a second type of APS (e.g., LMCSAPS). By allowing each APS type to include a different value space, the codec does not need to check for APS ID 542 conflicts across APS types. Furthermore, by allowing the value space to overlap, the codec can avoid using larger APS ID 542 values, thereby saving bits. Thus, using separate overlapping value spaces for APS IDs 542 of different APS types improves coding efficiency, thereby reducing the use of network resources, memory resources, and / or processing resources on the encoder and decoder sides. As described above, the APS ID 542 can be extended within a predefined range.In some examples, the predefined range of the APS ID 542 can vary depending on the APS type indicated by the parameter type 541. This can allow different numbers of bits to be allocated to different APS types depending on how often different types of parameters typically change. For example, the APS ID 542 of the ALF APS 512 can range from 0 to 7, the APS ID 542 of the zoom list APS 513 can range from 0 to 7, and the APS ID 542 of the LMCS APS 514 can range from 0 to 3.

[0121] In another example, the LMCS parameters are included in the LMCS APS 514. Some systems include the LMCS parameters in the slice header 515. However, the LMCS / shaper parameters can change once per second. A video sequence may display 30 to 60 images 521 per second. Therefore, the LMCS parameters may not change within 30 to 60 frames. Including the LMCS parameters in the LMCS APS 514 significantly reduces redundant decoding of the LMCS parameters. In some examples, the slice header 515 and / or picture header associated with a slice 523 and / or picture 521, respectively, can reference the associated LMCS APS 514. The slice 523 and / or picture 521 can then reference the slice header 515 and / or picture header. This allows the decoder to obtain the LMCS parameters for the associated slice 523 and / or picture 521. In this way, the LMCS parameters are only encoded when the LMCS parameters for the slice 523 and / or picture 521 change. Therefore, using LMCSAPS 514 to encode LMCS parameters improves decoding efficiency, thereby reducing the use of network resources, memory resources, and / or processing resources on the encoder and decoder sides. Since LMCS is not used for all videos, SPS 510 may include an LMCS enable flag 543. LMCS enable flag 543 may be set to indicate that LMCS is enabled for the encoded video sequence. Therefore, when LMCS enable flag 543 is set (e.g., to 1), the decoder may obtain LMCS parameters from LMCS APS 514 based on LMCS enable flag 543. In addition, when LMCS enable flag 543 is not set (e.g., to 0), the decoder may not attempt to obtain LMCS parameters.

[0122] Figure 6FIG6 is a diagram illustrating an exemplary mechanism 600 for assigning APS IDs 642 to different APS types in different value spaces. For example, the mechanism 600 can be applied to the codestream 500 to assign APS IDs 542 to the ALF APS 512, the scaling list APS 513, and / or the LMCS APS 514. Furthermore, the mechanism 600 can be applied to the codec 200, the encoder 300, and / or the decoder 400 when decoding a video according to the method 100.

[0123] Mechanism 600 assigns APS IDs 642 to ALF APS 612, zoom list APS 613, and LMCS APS 614, where APS IDs 642, ALF APS 612, zoom list APS 613, and LMCS APS 614 can be substantially similar to APS IDs 542, ALF APS 512, zoom list APS 513, and LMCS APS 514, respectively. As described above, APS IDs 642 can be sequentially assigned across multiple different value spaces, each specific to an APS type. Furthermore, each value space can extend within a different range specific to the APS type. In the illustrated example, the value space of APS ID 642 for ALF APS 612 ranges from 0 to 7 (e.g., 3 bits). Furthermore, the value space of APS ID 642 for zoom list APS 613 ranges from 0 to 7 (e.g., 3 bits). Furthermore, the value space of the APS ID 642 of the LMCS APS 611 ranges from 0 to 3 (e.g., 2 bits). When the APS ID 642 reaches the end of the value space, the APS ID 642 of the next APS of the corresponding type returns to the beginning of the range (e.g., 0). When a new APS receives the same APS ID 642 as a previous APS of the same type, the previous APS is no longer active and can no longer be referenced. Therefore, the range of the value space can be expanded to allow more APSs of a certain type to remain active and be referenced. The range of the value space can be reduced to improve decoding efficiency, but this reduction also reduces the number of APSs of the corresponding type that can remain active and available for reference simultaneously.

[0124] In the example shown, each ALF APS 612, zoom list APS 613, and LMCS APS 614 is referenced by a combination of an APS ID 642 and an APS type. For example, the ALF APS 612, LMCS APS 614, and zoom list APS 613 each receive an APS ID 642 of 0. When a new ALF APS 612 is received, the APS ID 642 is incremented from the value used for a previous ALF APS 612. The same sequence applies to the zoom list APS 613 and LMCS APS 614. Therefore, each APS ID 642 is correlated with the APS ID 642 of a previous APS of the same type. However, the APS ID 642 is not correlated with the APS ID 642 of previous APSs of other types. In this example, the APS ID 642 of the ALF APS 612 is incremented from 0 to 7 and then returns to 0 before continuing to increment. In addition, the APS ID 642 of the scaling list APS 613 increments from 0 to 7 and then returns to 0 before continuing to increment. Furthermore, the APS ID 642 of the LMCS APS 611 increments from 0 to 3 and then returns to 0 before continuing to increment. As shown, these value spaces overlap, as different APSs of different APS types can share the same APS ID 642 at the same point in the video sequence. It should be noted that mechanism 600 only describes APSs. The described APSs will be interspersed in the codestream among other VCL and non-VCL NAL units, such as SPSs, PPSs, slice headers, picture headers, slices, etc.

[0125] Therefore, the present invention includes improvements to the APS design and some improvements to the indication of shaper / LMCS parameters. The APS design is used to indicate information that can be shared by multiple pictures and can include many variations. The shaper / LMCS parameters are used in the adaptive in-loop shaper / LMCS video decoding tool. The above mechanism can be implemented as follows. To address the problems outlined herein, several aspects are included that can be used individually and / or in combination.

[0126] Modify the disclosed APS so that multiple APSs can be used to carry different types of parameters. Each APS NAL unit is used to carry only one type of parameter. Therefore, when two types of information (e.g., one of each type) are carried for a particular segment group / slice, two APS NAL units are encoded. The APS may include an APS parameter type field in the APS syntax. Only parameters of the type indicated by the APS parameter type field may be included in an APS NAL unit.

[0127] In some examples, different types of APS parameters are indicated by different NAL unit types. For example, two different NAL unit types are used for APS. These two types of APS can be called ALF APS and shaper APS, respectively. In another example, the tool parameter type carried in the APS NAL unit is specified in the NAL unit header. In VVC, the NAL unit header has reserved bits (for example, 7 bits represented as nuh_reserved_zero_7bits). In some examples, some of these bits (for example, 3 of the 7 bits) can be used to specify the APS parameter type field. In some examples, a particular type of APS can share the same value space for the APS ID. At the same time, different types of APS use different APS ID value spaces. Therefore, two APSs of different types can coexist and have the same APS ID value at the same time. In addition, the combination of APS ID and APS parameter type can be used to identify an APS from multiple other APSs.

[0128] When the corresponding decoding tool is enabled for a block group, the APS ID may be included in the block group header syntax. Otherwise, the APS ID of the corresponding type may not be included in the block group header. For example, when ALF is enabled for a block group, the APS ID of the ALF APS is included in the block group header. For example, this can be achieved by setting the APS parameter type field to indicate the ALF type. Therefore, when ALF is not enabled for a block group, the APS ID of the ALF APS is not included in the block group header. In addition, when the shaper decoding tool is enabled for a block group, the APS ID of the shaper APS is included in the block group header. For example, this can be achieved by setting the APS parameter type field to indicate the shaper type. In addition, when the shaper decoding tool is not enabled for a block group, the APS ID of the shaper APS may not be included in the block group header.

[0129] In some examples, the presence of APS parameter type information in an APS can be accommodated by using parameter-related coding tools. When only one APS-related coding tool (e.g., LMCS, ALF, or scaling list) is enabled for a codestream, the APS parameter type information may not be present but may be inferred. For example, when an APS may contain parameters for both the ALF and shaper coding tools, but only the ALF is enabled (e.g., as specified by a flag in the SPS) and the shaper is not enabled (e.g., as specified by a flag in the SPS), the APS parameter type may not be indicated and may be inferred to be equal to the ALF parameter.

[0130] In another example, APS parameter type information can be inferred from the APS ID value. For example, a predefined range of APS ID values can be associated with a corresponding APS parameter type. This aspect can be implemented as follows. Instead of allocating X bits to indicate the APS ID and Y bits to indicate the APS parameter type, X+Y bits can be allocated to indicate the APS ID. Different ranges of APS ID values can then be specified to represent different types of APS parameter types. For example, instead of using 5 bits to indicate the APS ID and 3 bits to indicate the APS parameter type, 8 bits can be allocated to indicate the APS ID (e.g., without increasing bit cost). An APS ID value range of 0 to 63 indicates that the APS includes parameters for the ALF, and an ID value range of 64 to 95 indicates that the APS includes parameters for the shaper. For other parameter types (e.g., scaling lists), a value range of 96 to 255 can be reserved. In another example, an APS ID value range of 0 to 31 indicates that the APS includes parameters for the ALF, and an ID value range of 32 to 47 indicates that the APS includes parameters for the shaper. For other parameter types (such as scaling lists), the value range of 48 to 255 can be reserved. The advantage of this approach is that APS ID ranges can be assigned based on the frequency of parameter changes for each tool. For example, it may be expected that ALF parameters will change more frequently than shaper parameters. In this case, a larger APS ID range can be used to indicate that the APS includes ALF parameters.

[0131] In a first example, one or more of the above aspects may be implemented as follows: ALF APS may be defined as an APS with aps_params_type equal to ALF_APS. Shaper APS (or LMCS APS) may be defined as an APS with aps_params_type equal to MAP_APS. Examples of SPS syntax and semantics are as follows.

[0132]

[0133] sps_reshaper_enabled_flag is set to 1 to indicate that the reshaper is used in the coded video sequence (CVS). sps_reshaper_enabled_flag is set to 0 to indicate that the reshaper is not used in the CVS.

[0134] Examples of APS syntax and semantics are as follows.

[0135]

[0136]

[0137] aps_params_type indicates the type of APS parameters carried in the APS, as shown in the following table.

[0138] Table 1 - APS parameter type codes and APS parameter types

[0139] aps_params_type The name of aps_params_type Types of APS parameters 0 ALF_APS ALF parameters 1 MAP_APS In-loop mapping (i.e., shaper) parameters 2..7 reserve reserve

[0140] The following are examples of the block group header syntax and semantics.

[0141]

[0142]

[0143] tile_group_alf_aps_id indicates the adaptation_parameter_set_id of the ALF APS referenced by the tile group. The TemporalId of the ALF APS NAL unit with adaptation_parameter_set_id equal to tile_group_alf_aps_id shall be less than or equal to the TemporalId of the coded tile group NAL unit. When multiple ALF APSs with the same adaptation_parameter_set_id value are referenced by two or more tile groups of the same picture, the multiple ALF APSs with the same adaptation_parameter_set_id value shall have the same content.

[0144] tile_group_reshaper_enabled_flag is set to 1 to indicate that the reshaper is enabled for the current tile group. tile_group_reshaper_enabled_flag is set to 0 to indicate that the reshaper is not enabled for the current tile group. When tile_group_reshaper_enable_flag is not present, the flag is inferred to be equal to 0. tile_group_reshaper_aps_id represents the adaptation_parameter_set_id of the reshaper APS referenced by the tile group. The TemporalId of the reshaper APS NAL unit with adaptation_parameter_set_id equal to tile_group_reshaper_aps_id shall be less than or equal to the TemporalId of the coded tile group NAL unit. When multiple reshaper APSs with the same adaptation_parameter_set_id value are referenced by two or more tile groups of the same picture, the multiple reshaper APSs with the same adaptation_parameter_set_id value shall have the same content. tile_group_reshaper_chroma_residual_scale_flag is set to 1 to indicate that chroma residual scaling is enabled for the current tile group. tile_group_reshaper_chroma_residual_scale_flag is set to 0 to indicate that chroma residual scaling is not enabled for the current tile group. When tile_group_reshaper_chroma_residual_scale_flag is not present, the flag is inferred to be equal to 0.

[0145] The following are examples of shaper data syntax and semantics.

[0146]

[0147]

[0148] reshaper_model_min_bin_idx represents the minimum bin (or segment) index to be used during the reshaper construction process. reshaper_model_min_bin_idx should be in the range of 0 to MaxBinIdx, inclusive. The value of MaxBinIdx should be 15. reshaper_model_delta_max_bin_idx represents the maximum allowed bin (or segment) index, MaxBinIdx, minus the maximum bin index to be used during the reshaper construction process. The value of reshaper_model_max_bin_idx is set to MaxBinIdx – reshaper_model_delta_max_bin_idx. reshaper_model_bin_delta_abs_cw_prec_minus1+1 represents the number of bits used to represent the syntax element reshaper_model_bin_delta_abs_CW[i]. reshaper_model_bin_delta_abs_CW[i] represents the absolute delta codeword value for the i-th bin. The reshaper_model_bin_delta_abs_CW[i] syntax element is represented by reshaper_model_bin_delta_abs_cw_prec_minus1+1 bits. reshaper_model_bin_delta_sign_CW_flag[i] represents the sign of reshaper_model_bin_delta_abs_CW[i].

[0149] In a second example, one or more of the above aspects may be implemented as follows: An example of SPS syntax and semantics is as follows.

[0150]

[0151] The variables ALFEnabled and ReshaperEnabled are set as follows. ALFEnabled=sps_alf_enabled_flag and ReshaperEnabled=sps_reshaper_enabled_flag.

[0152] Examples of APS syntax and semantics are as follows.

[0153]

[0154]

[0155] aps_params_type indicates the type of APS parameters carried in the APS, as shown in the following table.

[0156] Table 2 - APS parameter type codes and APS parameter types

[0157] aps_params_type The name of aps_params_type Types of APS parameters 0 ALF_APS ALF parameters 1 MAP_APS In-loop mapping (i.e., shaper) parameters 2..7 reserve reserve

[0158] When not present, the value of aps_params_type is inferred as follows: If ALF is enabled, aps_params_type is set to 0. Otherwise, aps_params_type is set to 1.

[0159] In a second example, one or more of the above aspects may be implemented as follows: An example of SPS syntax and semantics is as follows.

[0160]

[0161] The adaptation_parameter_set_id provides an identifier for the APS for reference by other syntax elements. The APS can be shared across pictures and can be different for different partition groups within a picture. The values and descriptions of the variable APSParamsType are defined in the following table.

[0162] Table 3 - APS parameter type codes and APS parameter types

[0163] APSID range ApsParamsType Types of APS parameters 0～63 0:ALF_APS ALF parameters 64～95 1:MAP_APS In-loop mapping (i.e., shaper) parameters 96..255 reserve reserve

[0164] The tile group header semantics are as follows. tile_group_alf_aps_id represents the adaptation_parameter_set_id of the ALF APS referenced by the tile group. The TemporalId of the ALF APS NAL unit with adaptation_parameter_set_id equal to tile_group_alf_aps_id shall be less than or equal to the TemporalId of the coded tile group NAL unit. The value of tile_group_alf_aps_id shall be in the range of 0 to 63 (inclusive).

[0165] When multiple ALF APSs with the same adaptation_parameter_set_id value are referenced by two or more tile groups of the same picture, the multiple ALF APSs with the same adaptation_parameter_set_id value should have the same content. tile_group_reshaper_aps_id represents the adaptation_parameter_set_id of the reshaper APS referenced by the tile group. The TemporalId of the reshaper APS NAL unit with adaptation_parameter_set_id equal to tile_group_reshaper_aps_id should be less than or equal to the TemporalId of the coded tile group NAL unit. The value of tile_group_reshaper_aps_id should be in the range of 64 to 95 (inclusive). When multiple reshaper APSs with the same adaptation_parameter_set_id value are referenced by two or more tile groups of the same picture, the multiple reshaper APSs with the same adaptation_parameter_set_id value should have the same content.

[0166] Figure 7 Schematic diagram of an exemplary video decoding device 700. The video decoding device 700 is suitable for implementing the disclosed examples / embodiments described herein. The video decoding device 700 includes a downlink port 720, an uplink port 750, and / or a transceiver unit (Tx / Rx) 710, wherein the transceiver unit includes a transmitter and / or a receiver for sending data uplink and / or downlink through a network. The video decoding device 700 also includes: a processor 730, including a logic unit and / or a central processing unit (CPU) for processing data; and a memory 732 for storing data. The video decoding device 700 may also include electronic components, optical-to-electrical (OE) components, electrical-to-optical (EO) components, and / or wireless communication components coupled to the uplink port 750 and / or the downlink port 720 for communicating data via an electrical, optical, or wireless communication network. The video decoding device 700 may also include input and / or output (I / O) devices 760 for transmitting data to and from a user. The I / O devices 760 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices 760 may also include input devices such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0167] Processor 730 is implemented using both hardware and software. It can be implemented as one or more CPU chips, cores (e.g., a multi-core processor), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). Processor 730 communicates with downlink port 720, Tx / Rx 710, uplink port 750, and memory 732. Processor 730 includes a decoding module 714. Decoding module 714 implements the embodiments disclosed herein, such as methods 100, 800, and 900, which may utilize codestream 500 and / or mechanism 600. Decoding module 714 may also implement any other methods / mechanisms described herein. Furthermore, decoding module 714 may implement codec system 200, encoder 300, and / or decoder 400. For example, decoding module 714 may encode / decode images in a codestream and encode / decode parameters associated with slices of images in multiple APSs. In some examples, different types of parameters can be encoded into different types of APSs. Furthermore, different types of APSs can be included in different NAL unit types. Such APS types can include ALFAPS, scaling list APSs, and / or LMCS APSs. Each APS can include an APS ID. The APS IDs of different APS types increase sequentially across different value spaces. Furthermore, slices and / or pictures can reference corresponding slice headers and / or picture headers. These headers can then reference the APS containing the associated decoding tools. Such APSs can be uniquely referenced by the APS ID and APS type. Such examples reduce redundant indication of decoding tool parameters and / or reduce bit usage in identifiers. Thus, the decoding module 714 enables the video decoding device 700 to provide additional functionality and / or decoding efficiency when decoding video data. Thus, the decoding module 714 improves the functionality of the video decoding device 700 and addresses issues unique to the field of video coding. Furthermore, the decoding module 714 can transition the video decoding device 700 to different states. Alternatively, the decoding module 714 may be implemented as instructions stored in the memory 732 and executed by the processor 730 (eg, a computer program product stored on a non-transitory medium).

[0168] The memory 732 includes one or more memory types, such as a magnetic disk, a tape drive, a solid-state drive, a read-only memory (ROM), a random access memory (RAM), a flash memory, a ternary content-addressable memory (TCAM), a static random-access memory (SRAM), etc. The memory 732 can be used as an overflow data storage device to store programs when they are selected for execution and to store instructions and data read during the execution of the programs.

[0169] Figure 8 800 is a flow chart of an exemplary method 800 for encoding a video sequence into a codestream such as codestream 500 by using multiple APS types, such as ALF APS 512, scaling list APS 513, and / or LMCS APS 514. An encoder (e.g., codec system 200, encoder 300, and / or video decoding device 700) may use method 800 when performing method 100. Method 800 may also assign APS IDs to different types of APSs using different value spaces according to mechanism 600.

[0170] Method 800 may begin with an encoder receiving a video sequence comprising a plurality of images and determining, for example based on user input, to encode the video sequence into a bitstream. The video sequence is segmented into pictures / images / frames for further segmentation before encoding. In step 801, a slice is encoded into the bitstream as part of the encoded video sequence. The slice may be encoded as part of the picture. Furthermore, the slice may be encoded by encoding the CTUs and / or CUs contained in the slice. Such CUs may be decoded based on intra-frame prediction and / or inter-frame prediction. For example, the encoder may encode the CUs of the slice. The encoder may then use a hypothetical reference decoder (HRD) to decode the encoded slice and filter the decoded slice to improve the output quality of the slice.

[0171] In step 803, the encoder determines the parameters used by the filter applied by the HRD to improve the quality of the encoded slice. These parameters may include ALF parameters, scaling list parameters, and LMCS parameters. Determining these parameters enables the encoder to determine the parameters applied to the decoder side slice.

[0172] In step 805, the ALF parameters are encoded into a first APS NAL unit in the bitstream that includes the ALF type. Furthermore, the scaling list parameters are encoded into a second APS NAL unit in the bitstream that includes the scaling list type. Furthermore, the LMCS parameters are encoded into a third APS NAL unit in the bitstream that includes the LMCS type. Thus, by using a variety of different NAL unit types, the ALF parameters, scaling list parameters, and LMCS parameters are each encoded into a different APS. The first, second, and third APS NAL units can all be non-video coding layer (non-VCL) NAL units.

[0173] In step 807, the SPS can be encoded into the bitstream. In some examples, the SPS includes a flag that can be set to indicate that LMCS is enabled for the coded video sequence including the slice. It should be noted that the flag can also be set to indicate that LMCS is not enabled for the coded video sequence. In this case, the LMCS parameter is omitted in the other steps of method 800.

[0174] In step 809, the first slice header is encoded into the codestream. The slice header may be associated with the slice. In this case, the slice references the slice header. The slice header may then reference the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit. This ensures that the decoder can determine the relevant parameters of the slice based on the reference to the slice header and the reference to various APS NAL units. In some examples, the picture header may contain such a reference. In this case, in step 809, the picture header is encoded into the codestream. In this example, the picture containing the slice references the picture header, and the picture header references the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit.

[0175] As described above, each APS is identified by a combination of a parameter type and an APS ID. For example, each APS may include an APS parameter type code (e.g., aps_params_type), which is set to a predefined value indicating the type of parameter included in each APS. For example, the aps_params_type code may be set to the ALF type for the first APS NAL unit, the scaling list type for the second APS NAL unit, and the LMCS type for the third APS NAL unit. Each APS may include an APS ID selected from a predefined range. Furthermore, the predefined range may be determined based on the parameter type of each APS. Such predefined ranges may overlap, and APS IDs may be assigned in different sequences based on the APS type. For example, a current APS may include a current APS ID selected from a predefined range. The current APS ID is sequentially associated with a previous APS ID associated with a previous APS of the same type as the current APS. Furthermore, since the sequence is different for different APS types, the current APS ID is not associated with another previous APS ID associated with another previous APS of a different APS type than the current APS. The slice header and / or picture header of step 809 can uniquely reference the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit by using a combination of the relevant APS parameter type code and the APS ID.

[0176] The code stream is stored in a memory at step 811. Upon request, the code stream may then be transmitted to a decoder, for example, via a transmitter.

[0177] Figure 9 1 is a flow chart of an exemplary method 900 for decoding a video sequence from a codestream such as codestream 500 using multiple APS types, such as ALF APS 512, scaling list APS 513, and / or LMCS APS 514. A decoder (e.g., codec system 200, decoder 400, and / or video decoding device 700) may use method 900 when performing method 100. Method 900 may also refer to APSs based on APS IDs assigned according to mechanism 600, where different types of APSs use APS IDs assigned according to different value spaces.

[0178] Method 900 may begin when a decoder begins receiving a bitstream representing decoded data of a video sequence, for example, as a result of method 800. At step 901, a bitstream is received at a decoder. The bitstream includes a first APS NAL unit including an ALF type, a second APS NAL unit including a scaling list type, and a third APS NAL unit including an LMCS type. The bitstream also includes slices. The first, second, and third APS NAL units may each be non-video coding layer (non-VCL) NAL units.

[0179] The codestream may also include an SPS. In step 902, the value of the LMCS enable flag is obtained from the SPS in the codestream. The flag may be set to indicate that LMCS is enabled for the coded video sequence including the slice. In this case, the LMCS parameters of the third APS NAL unit may be obtained in step 903 based on the flag value.

[0180] In step 903, ALF parameters are obtained from the first APS NAL unit. In addition, scaling list parameters are obtained from the second APS NAL unit. In addition, LMCS parameters are obtained from the third APS NAL unit. In some examples, LMCS parameters are obtained only when the LMCS enable flag is set to a specific value (e.g., value 1). Obtaining the parameters of APS can be accomplished through various mechanisms. In one example, the codestream also includes a slice header. The slice references the slice header. The slice header references the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit. These references can be used to locate and obtain parameters related to the slice. In some examples, the codestream also includes a picture header that can contain such references. A slice is a part of a picture. In this example, the picture containing the slice references the picture header, and the picture header references the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit.

[0181] As described above, each APS is identified by a combination of a parameter type and an APS ID. For example, each APS may include an APS parameter type code (e.g., aps_params_type), which is set to a predefined value indicating the type of parameter included in each APS. For example, the aps_params_type code may be set to the ALF type for the first APS NAL unit, the scaling list type for the second APS NAL unit, and the LMCS type for the third APS NAL unit. Each APS may include an APS ID selected from a predefined range. Furthermore, the predefined range may be determined based on the parameter type of each APS. Such predefined ranges may overlap, and APS IDs may be assigned in different sequences based on the APS type. For example, a current APS may include a current APS ID selected from a predefined range. The current APS ID is sequentially associated with a previous APS ID associated with a previous APS of the same type as the current APS. Furthermore, since the sequence is different for different APS types, the current APS ID is not associated with another previous APS ID associated with another previous APS of a different APS type than the current APS. The slice header and / or picture header of step 903 can uniquely reference the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit by using a combination of the relevant APS parameter type code and the APS ID.

[0182] In step 905, the slice is decoded by using the ALF parameters, scaling list parameters, and LMCS parameters obtained in step 903. In step 907, the decoder may forward the slice for display as part of the decoded video sequence.

[0183] Figure 10 1 is a diagram of an exemplary system 1000 for decoding a video sequence of images in a codestream such as codestream 500 by using multiple APS types, such as ALF APS 512, scaling list APS 513, and / or LMCS APS 514. System 1000 may be implemented by an encoder and a decoder (e.g., codec system 200, encoder 300, decoder 400) and / or video decoding device 700. Furthermore, system 1000 may be used when implementing methods 100, 800, 900, and / or mechanism 600.

[0184] System 1000 includes a video encoder 1002. Video encoder 1002 includes a determination module 1001 for determining ALF parameters, scaling list parameters, and LMCS parameters to be applied to a slice. Video encoder 1002 also includes an encoding module 1003 for encoding the slice into a bitstream as part of a coded video sequence. Encoding module 1003 is further configured to encode the ALF parameters into a first APS NAL unit in the bitstream that includes an ALF type. Encoding module 1003 is further configured to encode the scaling list parameters into a second APS NAL unit in the bitstream that includes a scaling list type. Encoding module 1003 is further configured to encode the LMCS parameters into a third APS NAL unit in the bitstream that includes an LMCS type. Video encoder 1002 also includes a storage module 1005 for storing the bitstream for transmission to a decoder. The video encoder 1002 further includes a sending module 1007 for sending a code stream together with the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit to support decoding of the slice at the decoder side according to corresponding decoding tool parameters. The video encoder 1002 can also be used to perform any of the steps in the method 800.

[0185] System 1000 also includes a video decoder 1010. Video decoder 1010 includes a receiving module 1011 for receiving a bitstream comprising a first APS NAL unit of an ALF type, a second APS NAL unit of a scaling list type, a third APS NAL unit of an LMCS type, and a slice. Video decoder 1010 also includes an acquisition module 1013 for acquiring ALF parameters from the first APS NAL unit. Acquisition module 1013 is further configured to acquire scaling list parameters from the second APS NAL unit. Acquisition module 1013 is further configured to acquire LMCS parameters from the third APS NAL unit. Video decoder 1010 also includes a decoding module 1015 for decoding the slice using the ALF parameters, scaling list parameters, and LMCS parameters. Video decoder 1010 also includes a forwarding module 1017 for forwarding the slice for display as part of a decoded video sequence. Video decoder 1010 may also be configured to perform any of the steps of method 900.

[0186] A first component is directly coupled to a second component when there are no intervening components between them other than a wire, trace, or other medium. A first component is indirectly coupled to a second component when there are intervening components between them other than a wire, trace, or other medium. The term "coupled" and variations thereof encompass both direct and indirect couplings. Unless otherwise specified, the use of the term "about" is intended to include a range of ±10% of the following figure.

[0187] It should also be understood that the steps of the exemplary methods set forth herein do not necessarily need to be performed in the order described, and the order of the steps of these methods should be understood to be merely exemplary. Similarly, in methods consistent with various embodiments of the present invention, these methods may include other steps, and certain steps may be omitted or combined.

[0188] Although the present invention provides several embodiments, it should be understood that the disclosed systems and methods may be embodied in a variety of other specific forms without departing from the spirit or scope of the present invention. The present examples should be considered illustrative rather than restrictive, and the present invention is not limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0189] In addition, without departing from the scope of the present invention, the techniques, systems, subsystems and methods described and illustrated as discrete or separate in the various embodiments may be combined or integrated with other systems, components, techniques or methods. Other examples of changes, substitutions and modifications can be determined by those skilled in the art and may be given without departing from the spirit and scope of the present invention.

Claims

1. A method implemented in a decoder, characterized in that The method comprises: Receive a codestream, the codestream comprising: a first adaptation parameter set (APS) network abstraction layer (NAL) unit including an adaptive loop filter (ALF) type associated with a coded slice, and a third APS NAL unit including a luma-mapped chroma scaling (LMCS) type associated with the coded slice, wherein the first APS NAL unit includes an APS APS, and the third APS NAL unit includes a LMCS APS, the codestream further comprising a slice header, the slice header including a first flag, wherein when the first flag is 1, it indicates that the ALF is enabled and applied to a luma (Y), blue chroma (Cb), or red chroma (Cr) color component in the coded slice, and when the first flag is 0, it indicates that the ALF is disabled for all color components in the coded slice; Obtaining ALF parameters from the first APS NAL unit, and obtaining LMCS parameters from the third APS NAL unit; The coded slice is decoded using the ALF parameters, the LMCS parameters, and the first flag.

2. The method according to claim 1, characterized in that The code stream further includes a second APS NAL unit of a scaling list type, wherein the method further includes obtaining scaling list parameters from the second APS NAL unit, and wherein the coded slice is further decoded using the scaling list parameters.

3. The method according to claim 2, characterized in that The coded slice references the slice header, and wherein the slice header references the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit.

4. The method according to any one of claims 1 to 3, characterized in that Each APS includes an APS parameter type (aps_params_type) code, which is set to a predefined value to indicate a type of a parameter included in each APS.

5. The method according to any one of claims 1 to 4, characterized in that The current APS includes a current APS identifier (ID) selected from a predefined range, wherein APSs of a certain type share the same APS ID value space, and APSs of different types use different APS ID value spaces.

6. The method according to any one of claims 1 to 5, characterized in that Each APS includes an APS ID selected from a predefined range, and wherein the predefined range is determined according to a parameter type of each APS.

7. The method according to any one of claims 1 to 6, characterized in that Each APS is identified by a combination of parameter type and APS ID.

8. The method according to any one of claims 1 to 7, characterized in that The code stream further includes a sequence parameter set (SPS), the sequence parameter set including a flag, the flag being set to indicate that LMCS is enabled for the coded video sequence including the coded slice, and wherein the LMCS parameter of the third APS NAL unit is obtained according to the flag.

9. A method implemented in an encoder, characterized in that The method comprises: Encoding the slice into a bitstream as a coded slice as part of a coded video sequence; determining adaptive loop filter (ALF) parameters associated with the coded slice, and determining luma-mapped chroma scaling (LMCS) parameters associated with the coded slice; Encoding the ALF parameters into a first adaptation parameter set (APS) network abstraction layer (NAL) unit including an ALF type in the codestream, and encoding the LMCS parameters into a third APS NAL unit including an LMCS type in the codestream, wherein the first APS NAL unit includes an ALF APS and the third APS NAL unit includes an LMCS APS; A slice header is encoded into the codestream, wherein the slice header includes a first flag, where when the first flag is 1, it indicates that the ALF is enabled and applied to the luma (Y), blue chroma (Cb), or red chroma (Cr) color component in the coded slice, and when the first flag is 0, it indicates that the ALF is disabled for all color components in the coded slice.

10. The method according to claim 9, characterized in that Also includes: determining scaling list parameters for the coded slice; The scaling list parameters are encoded into a second APS NAL unit including a scaling list type in the code stream.

11. The method according to claim 10, characterized in that The coded slice references the slice header, and wherein the coded slice header references the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit.

12. The method according to any one of claims 9 to 11, characterized in that Each APS includes an APS parameter type (aps_params_type) code, which is set to a predefined value to indicate a type of a parameter included in each APS.

13. The method according to any one of claims 9 to 12, characterized in that The current APS includes a current APS identifier (ID) selected from a predefined range, wherein APSs of a certain type share the same APS ID value space, and APSs of different types use different APS ID value spaces.

14. The method according to any one of claims 9 to 13, characterized in that Each APS includes an APS ID selected from a predefined range, and wherein the predefined range is determined according to a parameter type of each APS.

15. The method according to any one of claims 9 to 14, characterized in that Each APS is identified by a combination of parameter type and APSID.

16. The method according to any one of claims 9 to 15, characterized in that The processor is further included encoding a sequence parameter set (SPS) into the bitstream, wherein the SPS includes a flag, and the flag is set to indicate that the coded video sequence including the coded slice enables LMCS.

17. A video decoding device, characterized in that: include: A processor, a receiver coupled to the processor, a memory coupled to the processor, and a transmitter coupled to the processor, wherein the processor, the receiver, the memory, and the transmitter are configured to execute the method according to any one of claims 1 to 16.

18. A non-transitory computer-readable storage medium, characterized in that A computer program product for use with a video decoding device; the computer program product comprises computer-executable instructions stored in the non-transitory computer-readable storage medium; when a processor executes the computer-executable instructions, the video decoding device is caused to perform the method according to any one of claims 1 to 16.

19. A computer program product, characterized in that The method comprises computer-executable instructions stored in a non-transitory computer storage medium, which, when executed by a processor, cause a computer to perform the method according to any one of claims 1 to 16.

20. A non-transitory computer-readable storage medium, characterized in that The storage medium stores a code stream, and the code stream includes: a first adaptation parameter set (APS) network abstraction layer (NAL) unit associated with a coded slice and comprising an adaptive loop filter (ALF) type, and a third APS NAL unit associated with the coded slice and comprising a luma-mapped chroma scaling (LMCS) type, wherein the first APS NAL unit comprises an ALS APS and the third APS NAL unit comprises an LMS APS, each of the APSs comprising a current APS identifier (ID) selected from a predefined range, the predefined range being determined based on a parameter type of the each APS; a slice header, the slice header comprising a first flag, wherein when the first flag is 1, it indicates that the ALF is enabled and applied to the luma (Y), blue chroma (Cb), or red chroma (Cr) color component in the coded slice, and when the first flag is 0, it indicates that the ALF is disabled for all color components in the coded slice; The first APS NAL unit is used to obtain ALF parameters, and the third APS NAL unit is used to obtain LMCS parameters.

21. The storage medium according to claim 20, wherein The code stream further includes a second APS NAL unit of a scaling list type, wherein the second APS NAL unit is used to obtain scaling list parameters.

22. The storage medium according to claim 21, wherein The coded slice references the slice header, and wherein the slice header references the first APS NAL unit, the second APS NAL unit, and the third APS NAL unit.

23. The storage medium according to any one of claims 20 to 22, characterized in that Each APS includes an APS parameter type (aps_params_type) code, which is set to a predefined value to indicate a type of a parameter included in each APS.

24. The storage medium according to any one of claims 20 to 23, characterized in that The current APS includes a current APS identifier (ID) selected from a predefined range, wherein APSs of a certain type share the same APS ID value space, and APSs of different types use different APS ID value spaces.

25. The storage medium according to any one of claims 20 to 24, characterized in that Each APS includes an APS ID selected from a predefined range, and wherein the predefined range is determined according to a parameter type of each APS.

26. The storage medium according to any one of claims 20 to 25, characterized in that Each APS is identified by a combination of parameter type and APS ID.

27. The storage medium according to any one of claims 20 to 26, characterized in that The code stream further includes a sequence parameter set (SPS), the sequence parameter set including a flag, the flag being set to indicate that LMCS is enabled for the coded video sequence including the coded slice, and wherein the LMCS parameter of the third APS NAL unit is obtained according to the flag.

Images