Lossless mode for common video codecs

Abstract

A lossless codec mode is proposed in a video codec system including a plurality of codec tools, some of which are lossy by design and some of which can be adapted to be lossless or near-lossless. In order to enable the lossless mode in such a system, it is proposed to disable tools that are lossy by design and use only lossless tools, to adapt some tools to enable lossless codec, and to adapt some tools to enable near-lossless codec so that secondary lossless codec can be applied after residual codec. In a specific embodiment, it is proposed to determine the type of residual codec in the following manner: when information indicates that transform skip residual codec is used, a flag indicating a special mode is obtained, and when the flag is true, it is determined that conventional residual codec must be used instead of transform skip residual codec that should be used.

Description

Technical Field

[0001] The present disclosure belongs to the field of video compression, and at least one embodiment more specifically relates to a lossless mode for Versatile Video Coding (VVC). Background Art

[0002] In order to achieve high compression efficiency, image and video coding schemes usually use prediction and transformation to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlation, and then the difference between the original image block and the predicted image block (usually expressed as a prediction error or prediction residual) is transformed, quantized and entropy encoded and decoded. During encoding, the original image block is usually segmented / divided into sub-blocks (e.g., quadtree segmentation may be used). In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to prediction, transformation, quantization and entropy encoding and decoding. Summary of the invention

[0003] A lossless codec mode is proposed in a video codec system including a plurality of codec tools, some of which are lossy by design and some of which can be adapted to be lossless or near-lossless. In order to enable the lossless mode in such a video codec system, it is proposed to disable tools that are lossy by design and use only lossless tools, to adapt some tools to enable lossless codec, and to adapt some tools to enable near-lossless codec, so that secondary lossless codec can be applied after residual codec, thereby providing lossless codec.

[0004] In a specific embodiment, a method for determining a residual codec type includes, when information indicates that transform skip residual coding is used, obtaining a flag indicating a special mode, and when the flag indicating the special mode is true, determining that conventional residual coding must be used instead of transform skip residual coding that should be used.

[0005] According to a first aspect, a method for determining a residual codec type includes, when information indicates that transform skip residual codec is used, obtaining a flag indicating a special mode, and when the flag indicating the special mode is true, determining that conventional residual codec must be used instead of transform skip residual codec that should be used.

[0006] According to a second aspect, a video encoding method includes, for a video block, determining a type of residual coding according to the method of the first aspect.

[0007] According to a third aspect, a video decoding method includes, for a video block, determining a type of residual coding according to the method of the first aspect.

[0008] According to a fourth aspect, a video encoding device includes an encoder configured to determine a type of residual coding according to the method of the first aspect.

[0009] According to a fifth aspect, a video decoding apparatus includes a decoder configured to determine a type of residual coding according to the method of the first aspect.

[0010] One or more embodiments of the present invention further provide a non-transitory computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to at least a portion of any of the above methods. One or more embodiments further provide a computer program product including instructions for executing at least a portion of any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A A block diagram of a video encoder according to an embodiment is shown.

[0012] Figure 1B A block diagram of a video decoder according to an embodiment is shown.

[0013] Figure 2 A block diagram is shown of an example of a system in which various aspects and embodiments are implemented.

[0014] Figure 3 An example of a simplified block diagram of the decoding process for PCM, lossless and transform skip modes is shown.

[0015] Figure 4 The horizontal and vertical transformations for each SBT position are shown.

[0016] Figure 5 The LMCS architecture is shown from the decoder's perspective.

[0017] Figure 6 The application of the quadratic transform is shown.

[0018] Figure 7 A simplified quadratic transform (RST) is shown.

[0019] Figure 8 The forward and inverse simplification transforms are shown.

[0020] Fig. 9 An example of forward RST 8x8 processing utilizing a 16x48 matrix is ​​shown.

[0021] Fig. 10A and 10B An example flow diagram is shown in accordance with at least one embodiment.

[0022] Fig.11 The application of forward and then inverse functions for brightness reshaping is shown.

[0023] Fig.12 An example flow diagram of an embodiment including parsing of cu_transquant_bypass_flag, transform_skip_flag, and residual coding when transform skip is inferred to be true for a lossless codec block is shown.

[0024] Fig.13 An example flow diagram of an embodiment including parsing of cu_transquant_bypass_flag, transform_skip_flag, and residual coding when transform skip is inferred to be false for a lossless codec block is shown.

[0025] Fig.14 An example flow diagram of an embodiment including parsing of cu_transquant_bypass_flag, transform_skip_flag, and residual coding when transform skip is always parsed for lossless codec blocks is shown.

[0026] Fig.15 An example of a simplified block diagram of secondary lossless codec is shown.

[0027] Fig.16 An example flow chart of the parsing process of region_transquant_bypass_flag and split_cu_flag when using region level signaling is shown.

[0028] Fig.17 Different partitioning of luma and chroma in the dual-tree case is shown. DETAILED DESCRIPTION

[0029] Various embodiments relate to a post-processing method for a predicted value of a sample of an image block, the value being predicted based on an intra-frame prediction angle, wherein the value of the sample is modified after the prediction so that it is determined based on a weighted difference between a value of a left reference sample and the obtained predicted value of the sample, wherein the left reference sample is determined based on the intra-frame prediction angle. An encoding method, a decoding method, an encoding device, and a decoding device based on the post-processing method are proposed.

[0030] Furthermore, although principles related to specific drafts of VVC (Universal Video Codec) or HEVC (High Efficiency Video Codec) specifications are described, the present aspects are not limited to VVC or HEVC, and may be applied, for example, to other standards and recommendations, whether existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise stated, or technically excluded, the various aspects described in this application may be used alone or in combination.

[0031] Figure 1A A video encoder 100 is shown. Variations of the encoder 100 are contemplated, but for clarity, the encoder 100 is described below without describing all contemplated variations. Prior to being encoded, the video sequence may undergo a pre-encoding process (101), such as applying a color transform to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0032] In the encoder 100, a picture is encoded by an encoder element as described below. The picture to be encoded is segmented (102) and processed, for example, in units of CUs. Each unit is encoded using, for example, an intra-frame or inter-frame mode. When the unit is encoded in intra-frame mode, it performs intra-frame prediction (160). In inter-frame mode, motion estimation (175) and compensation (170) are performed. The encoder determines (105) which of intra-frame mode or inter-frame mode is used to encode the unit, and indicates the intra-frame / inter-frame determination by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the prediction block from the original image block.

[0033] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass the transform and quantization, i.e., encode and decode the residual directly without applying a transform or quantization process.

[0034] The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inversely transformed (150) to decode the prediction residual. The image block is reconstructed by combining (155) the decoded prediction residual and the predicted block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset), adaptive loop filter (ALF) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0035] Figure 1B A block diagram of a video decoder 200 is shown. In the decoder 200, a bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs the same Figure 1A The encoder 100 also typically performs video decoding as part of the encoded video data. In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other encoding and decoding information. The picture segmentation information indicates how the picture is segmented. Thus, the decoder can divide the picture according to the decoded picture segmentation information (235). The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The image block is reconstructed by combining (255) the decoded prediction residual and the prediction block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0036] The decoded pictures may be further subjected to post-decoding processing (285), such as an inverse color transform (e.g., from YCbCr 4:2:0 to RGB 4:4:4), or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived from the pre-encoding process and signaled in the bitstream.

[0037] Figure 2A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 can be implemented as a device including various components described below, and is configured to perform one or more of the various aspects described in this document. Examples of such equipment include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. The elements of system 1000 can be implemented in a single integrated circuit (IC), multiple ICs, and / or discrete components, either individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed on multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the various aspects described in this document.

[0038] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing various aspects described in this document, for example. The processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network accessible storage device.

[0039] The system 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a (multiple) module that can be included in a device to perform encoding and / or decoding functions. As known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software known to those skilled in the art.

[0040] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0041] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) may be used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, an operating system for a television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, while 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Codec, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Codec, a new standard being developed by the Joint Video Experts Group JVET).

[0042] As shown in block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to: (i) an RF section that receives a radio frequency (RF) signal, such as sent over the air by a broadcaster, (ii) a component (COMP) input (or a collection of COMP inputs), (iii) a universal serial bus (USB) input, and / or (iv) a high-definition multimedia interface (HDMI) input. Figure 2 Other examples not shown include composite video.

[0043] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF portion may be associated with elements suitable for the following operations: (i) selecting a desired frequency (also referred to as selecting a signal, or band limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select (for example) a signal frequency band (which may be referred to as a channel in some embodiments), (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a stream of desired data packets. The RF portion of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a frequency band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal sent by wired (for example, cable) medium, and filter to the desired frequency band to perform frequency selection by filtering, down conversion and filtering again. Various embodiments rearrange the order of above-mentioned (and other) elements, remove some in these elements, and / or add other elements of similar or different functions. Adding element can include inserting element between existing elements, such as, for example, inserting amplifier and analog-to-digital converter. In various embodiments, the RF part includes antenna.

[0044] In addition, USB and / or HDMI terminals may include corresponding interface processors for connecting system 1000 to other electronic devices via USB and / or HDMI. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, can be implemented as needed in, for example, a separate input processing IC or in processor 1010. Similarly, various aspects of USB or HDMI interface processing can be implemented as needed in a separate interface IC or in processor 1010. The stream through demodulation, error correction and demultiplexing is provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operates in combination with memory and storage element to process data streams as needed for presentation on output devices.

[0045] The various elements of system 1000 may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected and transmit data between using a suitable connection arrangement 1140 (e.g., an internal bus known in the art, including an inter-IC (I2C) bus, wiring, and printed circuit boards).

[0046] The system 1000 includes a communication interface 1050 that implements communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented in, for example, a wired and / or wireless medium.

[0047] In various embodiments, data is streamed or otherwise provided to the system 1000 using a wireless network such as a Wi-Fi network (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)). The Wi-Fi signals of these embodiments are received through a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet for applications that allow streaming and other over-the-top communications. Other embodiments provide streaming data to the system 1000 using a set-top box that transmits data through an HDMI connection of an input block 1130. Still other embodiments provide streaming data to the system 1000 using an RF connection of an input block 1130. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0048] The system 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other devices. The display 1100 can also be integrated with other components (for example, as in a smart phone), or separated (for example, an external monitor for a laptop computer). In various examples of embodiments, other peripheral devices 1120 include one or more of a stand-alone digital video disk (or digital versatile disk) (DVR for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functions based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.

[0049] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, consumer electronics control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to the system 1000 via communication interface 1050 using communication channel 1060. In an electronic device such as, for example, a television, the display 1100 and speaker 1110 may be integrated into a single unit with other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver such as, for example, a timing controller (T Con) chip.

[0050] Display 1100 and speaker 1110 may alternatively be separate from one or more of the other components, such as if the RF portion of input 1130 is part of a separate set-top box. In various embodiments where display 1100 and speaker 1110 are external components, output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0051] These embodiments may be implemented by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. As a non-limiting example, the memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as an optical storage device, a magnetic storage device, a semiconductor-based storage device, a fixed memory, and a removable memory. As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0052] Lossless mode is available in HEVC. In this mode, the transform and quantization bypass is indicated at the CU level by a flag at the beginning of the CU syntax structure. If the bypass is turned on, the transform and quantizer scaling operations are skipped, and the residual signal is directly encoded and decoded without any degradation. Therefore, this mode achieves perfect reconstruction of the lossless representation of the codec block. The sampling difference is encoded as if it is a quantized transform coefficient level, that is, the transform block codec is reused using transform sub-blocks, coefficient scanning and the last significant coefficient signaling. This mode can be useful, for example, for local coding of graphic content, where quantization artifacts may be highly visible or completely intolerable. If the rate-distortion cost of the normal transform codec (usually when using a low quantization parameter) just exceeds the rate cost of the bypass codec, the encoder can also switch to this mode. For CUs encoded and decoded in lossless mode, the post filter is disabled.

[0053] If activated in the sequence parameter set (SPS), PCM codec can be indicated at the CU level. If valid for the CU under consideration, prediction, quantization and transform are not applied. Instead, the sample values ​​of the samples in the corresponding codec block are directly encoded into the bitstream at the PCM sample bit depth configured in the SPS. The granularity of the application of the PCM codec can be configured between the luma codec tree block size and the minimum value of 32×32 on the high end and the minimum luma codec block size on the low end. If the codec unit is encoded in PCM mode, the size of other codec units in the same codec tree unit shall not be less than the size of the PCM unit. Because, by definition, the PCM codec implements a lossless representation of the corresponding block, the bits spent on the PCM codec of the codec unit can be considered as an upper limit on the amount of bits required to encode the CU. Therefore, in cases where the application of transform-based residual codecs will exceed this limit (for example, for abnormally noisy content), the encoder can switch to PCM codecs.

[0054] The lossless mode of HEVC is activated using the transquant_bypass_enabled_flag flag encoded in the picture parameter set (PPS) (see Table 1 below). This flag implements the encoding and decoding of cu_transquant_bypass_flag at the CU level (Table 2). cu_transquant_bypass_flag Specifies to bypass quantization, transform processing, and loop filters.

[0055] Table 1: PPS syntax used in HEVC to signal lossless codecs at picture level

[0056]

[0057] Table 2: Codec unit syntax for lossless codecs in HEVC

[0058]

[0059] If cu_transquant_bypass_flag is true for all CUs, lossless codec can be used for the entire frame, but it is also possible to use lossless codec for only a region. This is often the case for mixed content with natural video with overlapping text and graphics. Text and graphic areas can be losslessly coded to maximize readability, while natural content can be coded in a lossy manner.

[0060] Furthermore, since quantization is lossy by definition, it is disabled for lossless codecs. In HEVC, the transform process (DCT-2 and DST-4 are used in HEVC) is lossy due to rounding operations. If the reconstructed signal is lossless, post filters such as deblocking filters and sample adaptive offset are useless.

[0061] Figure 3 An example of a simplified block diagram of a decoding process for PCM, lossless and transform skip modes is shown. The input to the decoding process 300 is a bitstream. In step 310, data is decoded from the bitstream. In particular, this provides quantized transform coefficients and transform types. In step 320, inverse quantization is applied to the quantized transform coefficients. In step 330, an inverse transform is applied to the resulting transform coefficients. The resulting residual is added to the prediction signal from the intra-frame or inter-frame prediction of step 340 in step 350. The result is a reconstructed block. When the I_PCM mode is selected in step 305, the sample values ​​are decoded directly without entropy decoding. When the lossless codec mode is selected, steps 320 and 330 are skipped. When the transform skip mode is selected, step 330 is skipped.

[0062] In the recent Versatile Video Codec (VVC) Test Model 4 (VTM4), large transform sizes up to 64×64 are enabled, which are mainly used for higher resolution videos, such as 1080p and 4K sequences. For transform blocks with a size (width or height, or both width and height) equal to 64, the high-frequency transform coefficients are zeroed so that only the low-frequency coefficients are retained, thus losing this information. More precisely, this information is lost because it is not encoded in the resulting bitstream and is therefore unavailable to the decoder. For example, for an M×N transform block, where M is the block width and N is the block height, when M is equal to 64, only the left 32 columns of the transform coefficients are retained. Similarly, when N is equal to 64, only the top 32 rows of the transform coefficients are retained. In both examples, the remaining 32 columns or 32 rows of the transform coefficients are lost. When the transform skip mode is used for large blocks, the entire block is used without zeroing any values. In practice, this involves setting the maximum size of the transform (to 5), as shown in the syntax extracted from the current VVC specification shown in Table 3 below (the relevant lines are set in bold). Thus, the maximum actual transform size is used, which is set to 32 (2^5) in the current VVC specification, for example. In the following this parameter will be named "max_actual_transf_size".

[0063] Table 3

[0064]

[0065] In addition to the DCT-2 used in HEVC, VVC adds a multiple transform selection (MTS) scheme for residual coding of inter-frame and intra-frame codec blocks. The newly introduced transform matrices are DST-7 and DCT-8. In order to control the MTS scheme, separate enable flags are specified at the SPS level for intra-frame and inter-frame, respectively. When MTS is enabled at the SPS level, the MTS index at the CU level is signaled to indicate the separable transform pairs used for CU in DCT-2, DST-7, and DCT-8. MTS applies only to luminance. The CU-level MTS index is signaled when the following conditions are met: both the width and height are less than or equal to 32, and the CBF flag is equal to 1.

[0066] To reduce the complexity of large-sized DST-7 and DCT-8, high-frequency transform coefficients are zeroed for DST-7 and DCT-8 blocks of size (width or height, or both) equal to 32. Only coefficients within the 16x16 low-frequency region are retained.

[0067] The block size restriction for transform skip is the same as that for MTS, which states that transform skip is applicable to a CU when both block width and height are equal to or less than 32.

[0068] For inter-prediction CUs with cu_cbf equal to 1, cu_sbt_flag can be signaled to indicate whether to decode the entire residual block or a sub-part of the residual block. In the former case, the inter-MTS information is further parsed to determine the transform type of the CU. In the latter case, a portion of the residual block is encoded and decoded using the inferred adaptive transform, while the rest of the residual block is zeroed.

[0069] Figure 4 The horizontal and vertical transforms for each SBT position are shown. The sub-block transform is a position-dependent transform applied to the luma transform block. The two positions of SBT-H (horizontal) and SBT-V (vertical) are associated with different core transforms. More specifically, Figure 4 The horizontal and vertical transforms for each SBT position are specified in . For example, the horizontal and vertical transforms for SBT-V position 0 are DCT-8 and DST-7, respectively. When one side of the residual TU is greater than 32, the corresponding transform is set to DCT-2. Therefore, the sub-block transform jointly specifies the TU tiling, cbf, and horizontal and vertical transforms of the residual block, which can be considered as a syntactic shortcut in the case where the main residual of the block is on one side of the block.

[0070] In VTM4, a codec tool called Luma Mapping and Chroma Scaling (LMCS) was added as a new processing block before the loop filter. LMCS has two main parts: 1) Adaptive piecewise linear model-based in-loop mapping of the luma component; 2) For the chroma components, luma-dependent chroma residual scaling is applied.

[0071] Figure 5 The LMCS architecture is shown from the decoder's perspective. Figure 5 The medium to light grey shaded blocks indicate where processing is applied in the mapped domain; and these include inverse quantization, inverse transform, luma intra prediction, and addition of the luma prediction to the luma residual. Figure 5 The unshaded blocks in indicate where processing is applied in the original (i.e., unmapped) domain; and these include loop filters such as deblocking, ALF and SAO, motion compensated prediction, chroma intra prediction, addition of chroma prediction and chroma residual, and storage of decoded pictures as reference pictures. Figure 5 The dark grey shaded blocks in are the new LMCS functional blocks, including forward and reverse mapping of luma signals and luma-dependent chroma scaling processing. Like most other tools of VVC, LMCS can be enabled / disabled at the sequence level using the SPS flag.

[0072] Also known as Non-Separable Secondary Transform (NSST) or Reduced Secondary Transform (RST), the secondary transform is applied between the forward main transform and quantization (at the encoder) and between dequantization and inverse main transform (at the decoder side).

[0073] Figure 6 Illustrates the application of the second transformation. In JEM, as shown in the figure, for each 8×8 block, a 4×4 second transformation is applied to small blocks (i.e., min(width, height) < 8), and an 8×8 second transformation is applied to larger blocks (i.e., min(width, height) > 4).

[0074] The application of the non-separable transformation is described below by taking the input as an example. The 4x4 input block is represented as the following matrix:

[0075]

[0076] To apply the non-separable transformation, this 4x4 input block X is first represented as a vector

[0077]

[0078] The non-separable transformation is calculated as where represents the transformation coefficient vector, and T is a 16×16 transformation matrix. The 16×1 coefficient vector is then reorganized into a 4×4 block using the scan order (horizontal, vertical, or diagonal) of the block. Coefficients with smaller indices will be placed in the 4x4 coefficient block together with smaller scan indices. There are a total of 35 transformation sets, and each transformation set uses 3 non-separable transformation matrices (kernels). The mapping from the intra prediction mode to the transformation set is predefined. For each transformation set, the selected non-separable second transformation candidate is further specified by an explicitly signaled second transformation index. This index is signaled once in the bitstream after the transformation coefficients for each CU within a frame.

[0079] Figure 7 Illustrates the Reduced Second Transformation (RST). Using this technique, 16x48 and 16x16 matrices are used for 8×8 and 4×4 blocks respectively. For ease of annotation, the 16x48 transformation is denoted as RST8x8, and the 16x16 transformation is denoted as RST4x4. The main idea of the Reduced Transformation (RT) is to map an N-dimensional vector to an R-dimensional vector in a different space, where R / N (R < N) is the reduction factor.

[0080] Figure 8 Illustrates the forward and inverse reduced transformations. The RT matrix is an R×N matrix as follows:

[0081]

[0082] where the R rows of the transformation are the R bases of the N-dimensional space. The inverse transformation matrix of RT is the transpose of its forward transformation.

[0083] Fig. 9 An example of forward RST8x8 processing using a 16x48 matrix is ​​shown. In the configuration adopted, a 16x48 matrix is ​​applied instead of a 16x64 matrix with the same transform set configuration, each of which takes 48 input data from three 4x4 blocks in the upper left 8x8 block (excluding the lower right 4x4 block) ( Figure 4 ). With the help of dimensionality reduction, the memory usage for storing all RST matrices is reduced from 10KB to 8KB with a reasonable performance degradation.

[0084] Furthermore, in VTM5, a coding tool called chroma residual joint coding has been adopted. When this tool is activated, a single joint residual block is used to describe the residuals of both Cb and Cr blocks in the same transform unit, as shown in Equation 1.

[0085] res joint =(res Cb -res Cr ) / 2

[0086] Equation 1: Joint residual calculated from Cb and Cr residuals

[0087] Then, the Cb and Cr signals are reconstructed by subtracting the joint residual of Cb and adding it to Cr, as shown in Equation 2.

[0088]

[0089] Equation 2: Reconstruction of Cb and Cr signals from joint residual codec

[0090] Codec flag at TU level to enable joint codec of chroma residual, if this flag is disabled, separate codec of Cb and Cr residual is used.

[0091] The embodiments described below have been designed in consideration of the foregoing. Figure 1A Encoder 100, Figure 1B The decoder 200 and Figure 2 The system 1000 is suitable for implementing at least one of the following embodiments.

[0092] In at least one embodiment, the present application is directed to a lossless codec mode in a video codec system, which includes multiple codec tools, some of which are lossy by design and some of which can be adapted to be lossless or near-lossless. In order to enable lossless mode in a video codec system such as VVC, the following strategy is proposed:

[0093] - disable tools that are lossy by design and use only non-destructive tools,

[0094] -Adapt some tools to enable lossless codec,

[0095] - Adapt some tools to enable near-lossless codecs (limited and small differences from the original signal) so that a secondary lossless codec can be applied after the residual codec, thus providing a lossless codec.

[0096] Lossless codecs can be handled at the frame level or at the region level.

[0097] Fig. 10A and 10B An example flow chart according to at least one embodiment is shown. The figure shows an example of a general process for deciding whether a given tool should be disabled at the frame level, block level, should be adapted to be lossless, or whether a secondary lossless codec should be performed to use the tool adapted to be nearly lossless. Such a process can be performed, for example, Figure 1A is implemented in the encoder device 100.

[0098] In the first step ( Fig. 10A 400), to assess whether the tool is lossy by design.

[0099] ● If the tool is damaged, the second check ( Fig. 10B The 402) test tool can be disabled at the block level (in which case it can only be controlled at the frame level).

[0100] ○ If the tool cannot be disabled at the block level, then Fig. 10B In step 404, the value of the flag transquant_bypass_enabled_flag (usually signaled at the PPS level) is checked. If transquant_bypass_enabled_flag is true, then Fig. 10B If transquant_bypass_enabled_flag is false, then in step 410 the tool is disabled (this applies to LMCS, for example). Fig. 10B The tool is enabled in step 411. Once step 410 or 411 is applied, the process ends.

[0101] ○ If the tool can be disabled at the block level, then Fig. 10B In step 405, the value of the CU-level flag cu_transquant_bypass_flag is checked. If cu_transquant_bypass_flag is true, Fig. 10B Disable the tool at CU level in step 408 (this applies to, for example, SBT, MTS, LFNTS). If cu_transquant_bypass_flag is false, then Fig. 10BIn step 409, the tool can be enabled at the CU level. Once step 408 or 409 is applied, the process ends.

[0102] ● If the tool is non-destructive by design, then Fig. 10A In step 401, a test is performed to check whether it is lossless.

[0103] o If the tool is non-destructive, then nothing special applies and the process proceeds to the end of the treatment.

[0104] ○ If the tool is not non-destructive, Fig. 10A In step 403 it is checked whether the tool is nearly intact.

[0105] ■ If the tool is not nearly lossless, it is not lossy by design either, which means it can be adapted to become lossless ( Fig. 10A This may be applicable, for example, to an MTS that uses only lossless transforms in the MTS transform set.

[0106] ■ If the tool is nearly lossless, an additional (secondary) lossless encoding and decoding step is applied ( Fig. 10A Step 407). For example, a lossless transform without quantization may be applied.

[0107] o Once step 306 or 307 is applied, the process ends.

[0108] In the following of this disclosure, the application of this process to specific tools of the VVC specification is described.

[0109] Disable tools that are incompatible with lossless codecs

[0110] The first case corresponds to Fig. 10A Steps 410 (picture level) and 408 (CU level) of FIG. 4 are repeated for each block. The first element is related to the zeroing transform with large CUs. In an embodiment, the zeroing transform cannot be used in case of a losslessly coded CU or if transform skipping is used. This is illustrated in the syntax specification in Table 4, which highlights the proposed changes compared to the current VVC syntax in italic text. If the block is larger than max_actual_transf_size x max_actual_transf_siz (actually 32x32 in the current VVC version) and is losslessly coded, all coefficients are coded, as opposed to the case of a lossy coded block where only a portion of the coefficients are coded.

[0111] In fact, in contrast to the current VVC specification, the transform size restriction is only applied when transform_skip_flag is False and cu_transquant_bypass_flag is False.

[0112] Table 4: Modified syntax for not using zeroing transform if transform skip or lossless codec is used for the current CU

[0113]

[0114] In a variant, if the picture-level transquant_bypass_enabled_flag is enabled, and if the block is larger than max_actual_transf_size x max_actual_transf_size (e.g., 32x32 in current VVC), a quadtree partition is inferred or forced. In other words, when lossless encoding is desired, blocks larger than 64x64 are systematically partitioned so that the block size becomes 32x32 and is therefore not subject to zeroing. Table 5 shows the modified syntax.

[0115] Table 5: Syntax for quadtree partitioning inference when lossless is enabled at the picture level

[0116]

[0117]

[0118] In another variation, cu_transquant_bypass_flag is only encoded if the current CU is smaller than or equal to max_actual_transf_size x max_actual_transf_size (e.g., 32x32 in current VVC). This means that larger blocks cannot be losslessly encoded. Table 6 shows the associated syntax.

[0119] Table 6: Proposed syntax for encoding cu_transquant_bypass for blocks less than or equal to 32x32 only

[0120]

[0121] The second element is related to the sub-block transform (SBT). Since SBT uses transform tree subdivision, where one of the transform units is inferred to have no residual, the tool cannot guarantee that the CU is reconstructed without loss. In an embodiment, when lossless mode is activated (i.e., when transquant_bypass_enabled_flag is true), SBT is disabled at the CU level. The corresponding syntax changes are shown in Table 7. It is only possible to decode SBT-related syntax and activate SBT when transquant_bypass_enabled_flag is false.

[0122] Table 7: Codec unit syntax when SBT is disabled for lossless codec CU

[0123]

[0124]

[0125] In an alternative implementation, SBT can only be used as TU tiling. In this embodiment, the residual can be encoded and decoded for each sub-block, while in the initial design, some blocks are forced to have coefficients with values ​​of 0. Horizontal and vertical transforms are inferred as transform skips. The corresponding syntax changes are shown in Table 8.

[0126] Table 8: Codec unit syntax when residuals are inferred for two CUs in a CU not coded for SBT

[0127]

[0128]

[0129] Fig.11 The application of the forward then inverse function for luma shaping is shown. In fact, the third element is related to luma shaping (LMCS). Shaping is a lossy transform, since the application of the forward then inverse function is not guaranteed to give the original signal due to rounding operations. In LMCS, the reference pictures are stored in the original domain. In the intra case, the prediction process is implemented in the "shaped" domain and once the samples are reconstructed, the inverse shaping is applied before the loop filtering step. In the inter case, after motion compensation, the predicted signal is forward shaped. Then, once the samples are reconstructed, the inverse shaping is applied before the loop filtering step.

[0130] In an embodiment, if lossless codec is allowed in PPS, LMCS is disabled at the slice level. The corresponding syntax changes are shown in Table 9.

[0131] Table 9: Strip header syntax modifications for disabling lmcs if transquant bypass is enabled in PPS

[0132]

[0133]

[0134] The fourth element is related to multiple transform selection (MTS) and transform skip, and an embodiment involves inferring transform skip to be true. Even if the quantization step size is equal to 1, DCT and DST transforms are lossy because rounding errors can cause slight losses. In a first embodiment, transform skip can only be used if the CU is losslessly coded (checked by the value of the CU level flag cu_transquant_bypass_flag). transform_skip_flag is inferred to be 1 and tu_mts_idx is not coded. The corresponding syntax changes are shown in Table 10. In the case where transform_skip_flag is inferred to be 1, residual coding for transform skip is used for lossless blocks.

[0135] Table 10: Transform unit syntax for codecs disabling MTS and transform skipping when CU lossless is enabled

[0136]

[0137] In VTM-5.0, two different residual codec processing can be used, the first one is for non-transform skip residual codec and is effective for coding residual of natural content blocks, and the second one is for transform skip residual codec, which is effective for coding screen content blocks. The residual codec syntax selection is shown in Table 11.

[0138] Table 11: Residual codec syntax in VTM-5.0

[0139]

[0140] Fig.12 An example flow chart of an embodiment including parsing of cu_transquant_bypass_flag, transform_skip_flag, and residual coding when transform skip is inferred to be true for a lossless codec block is shown. In a first step 800, cu_transquant_bypass_flag is parsed, if the flag is true, then transform_skip_flag is inferred to be true in step 801, otherwise transform_skip_flag is parsed in step 802. If transform_skip_flag is equal to false, then regular residual is parsed in step 803, otherwise if the flag is true, then transform skip residual is parsed in step 804.

[0141] Fig.13An example flow chart of an embodiment including parsing of cu_transquant_bypass_flag, transform_skip_flag, and residual codec when transform skip is inferred to be false for a lossless codec block is shown. In a first step 900, cu_transquant_bypass_flag is parsed, if the flag is true, then transform_skip_flag is inferred to be false at step 901, and regular residual codec is used at step 904, otherwise transform_skip_flag is parsed at step 902. If transform_skip_flag is equal to false, then regular residual is parsed at step 904, otherwise if the flag is true, then transform skip residual is parsed at step 903.

[0142] Since the residual codec for transform skip is designed to encode the residual from the screen content codec block, it may be less efficient to encode the natural content for lossless blocks. Therefore, in another embodiment, if the CU is lossless, transform_skip_flag is inferred to be 0 and tu_mts_idx is not encoded. The residual codec for regular transform is used for lossless codec blocks. Therefore, in this special mode (i.e., when cu_transquant_bypass_flag is true), the regular codec will be used even when transform_skip_flag is true, while usually when transform_skip_flag is true, the transform skip residual codec should be used.

[0143] in other words, Fig.13 The embodiment described in proposes that, in the case where information indicates that transform skip residual codec is used, the type of residual codec is determined by: obtaining a flag indicating a special mode, and selecting a normal residual codec for the residual codec when the flag is true, instead of the transform skip residual codec that should be used. The flag indicating the special mode can be derived from other information, such as an indication that the codec is lossless (e.g., cu_transquant_bypass_flag), or an indication that quantization, transform processing, and loop filter are bypassed (not used), or an indication that the residual codec is forced to be a normal residual codec.

[0144] Fig.14 An example flow diagram of an embodiment including parsing of cu_transquant_bypass_flag, transform_skip_flag, and residual coding when transform skip is always parsed for lossless codec blocks is shown.

[0145] In such an embodiment, in order to keep the design of the lossless codec block close to the lossy codec block, conventional residual codec and transform skip codec can be used. In this case, transform_skip_flag is coded for the lossless codec block. This allows competition between two different ways of coding the residual coefficients and makes the residual codec better adapted to the content.

[0146] In a first step 1400, cu_transquant_bypass_flag is parsed, then transform_skip_flag is parsed in step 1401. If transform_skip_flag is equal to false, then regular residuals are parsed in step 1403, otherwise if the flag is true, then transform skip residuals are parsed in step 1402.

[0147] In fact, transform skip residual codec methods are optimized to codec computer generated content, where the correlation between the values ​​of spatially adjacent coefficients is strong. In general, in lossy codecs, transform skip coefficient codecs are most commonly used when the content is computer generated and is efficient for such content.

[0148] However, in this embodiment, it is proposed to use a flag to select between the selection of the normal coefficient coding method and the transform skip coefficient coding method. In practice, the transform skip coefficient coding can be used for computer generated content and the normal coefficient coding method can be used for regular content. This allows competition between two different ways of coding the residual coefficients and better adapts the residual coding method to the content. The flag only changes the coefficient coding method used to code the residual of the block.

[0149] In VVC, two methods are used to encode and decode the residual:

[0150] -When transform and quantization are used, a conventional coefficient encoding method is used. This method inherits HEVC with some improvements, and it has been designed to encode the residual after the transform, where the energy is compressed in the low-frequency part of the signal.

[0151] - When only quantization is used, a transform skip coefficient coding method is used. This method is designed for computer generated content, where the transform is often skipped due to the large amount of spatial correlation between coefficients. The main characteristics of the transform skip residual coding methods are that they do not signal the last non-zero coefficient, start from the top left corner of the block, use a reduced template of the previous coefficient to code the current coefficient, and the sign of the coefficient is Cabac context coded.

[0152] Fig.12 , 13Figures 14 and 14 show that, unlike the current VVC specification, the residual coding method can be decoupled from the fact whether the transform has been skipped. In fact, in the current VVC specification, the transform_skip_flag syntax element equal to true means that the transform is not applied to the residual of the block, and the transform skip coefficient coding method is used. In the proposed method, if the current block is losslessly coded (transform and quantization are skipped), the conventional coefficient coding method is used, and a better coefficient coding method is used in the case of coding natural content by inferring transform_skip_flag to be false. In a variant, even if cu_transquant_bypass_flag is true (in this case, transform and quantization are skipped), the flag (here transform_skip_flag is reused for this purpose) is coded to indicate only which residual coding method is used.

[0153] One embodiment relates to a low frequency non-separable transform (LFNTS). In this embodiment, if the current CU is losslessly coded, LFNTS is disabled. The corresponding syntax changes are shown in Table 12.

[0154] Table 12: Codec unit syntax for disabling lfnts when the CU is lossless coded

[0155]

[0156] One embodiment involves joint Cb-Cr encoding and decoding. The joint encoding and decoding process of Cb and Cr residuals is irreversible. By using Equation 1 and Equation 2, where resCb=resCr, we get resjoint=0, and recCb=predCb, recCr=predCr. In one embodiment, if cu_transquant_bypass_flag is true, joint Cb-Cr encoding and decoding is disabled at the CU level. The corresponding syntax changes are shown in Table 13.

[0157] Table 13: Residual Codec Syntax for Disabling Joint Cb-Cr Codec if cu_transquant_bypass_flag is true

[0158]

[0159] When Equation 1 separates resjoint_Cb from resJoint_Cr, as shown in Equation 3.

[0160] Equation 3: Modification of the joint residual calculation proposed in N0347

[0161]

[0162] With the proposed modification, the process is reversible, but lossy due to rounding errors, as shown in Equation 4.

[0163] Equation 4: Modification to reconstruct Cb and Cr signals from joint residual coding

[0164]

[0165] This variation can be used for lossless encoding and decoding when a secondary lossless encoding and decoding process is performed as described below.

[0166] Adaptation of VVC tools for lossless codec

[0167] refer to Fig. 10A , this case corresponds to step 406.

[0168] In one embodiment, lossless transforms are added to the MTS transform set. For lossless encoding and decoding, those transforms should be selected. Examples of lossless transforms are non-normalized Walsh-Hadamard transform, non-normalized Haar transform. Examples of near-lossless transforms are normalized Walsh-Hadamard transform, normalized Haar transform.

[0169] The lossless transform can be obtained by Walsh-Hadamard transform or Haar transform. The denormalized transforms of Walsh-Hadamard and Haar consist of ±1 and zero, as shown in Equation 5 and Equation 6. For example, the denormalized 4×4 Walsh-Hadamard matrix is:

[0170] Equation 5: Denormalized 4x4 Walsh-Hadamard Matrix

[0171]

[0172] The unnormalized 4x4 Haar matrix is:

[0173] Equation 6: Denormalized 4x4 Haar matrix

[0174]

[0175] Lossless reconstruction can be achieved using the Walsh-Hadamard transform or the Haar transform. To explain this, consider the example of a 2x2 transform, which uses residual samples (r0 and r1). The transform matrices for Haar and Walsh-Hadamard are:

[0176] Equation 7: Denormalized 2x2 Walsh-Hadamard or Haar Matrix

[0177]

[0178] The transformation coefficients (c0 and c1) are obtained by the following formula:

[0179] Equation 8: Transform coefficients calculated using the 2x2 Walsh-Hadamard or Haar denormalized transform

[0180]

[0181] The inverse transform is performed this way:

[0182] Equation 9: Inverse transform using 2x2 Walsh transform or Haar denormalization transform

[0183]

[0184] In another example of a 4×4 Walsh-Hadamard transform, to transform 4 residual samples (r0, r1, r2, r3), the matrix is ​​multiplied by the coefficient vector to obtain transform coefficients c0, c1, c2, and c3:

[0185] Equation 10: Transform coefficients calculated using the 4x4 Walsh-Hadamard denormalized transform

[0186]

[0187] The inverse transform is performed in this way

[0188] Equation 11: Inverse transform using the 4x4 Walsh transform denormalized transform

[0189]

[0190] Finally, to transform the 4 residual samples using the Haar transform, the following equation is used: Equation 12: The transform coefficients calculated using the 4x4 Haar transform

[0191]

[0192] The inverse transform is performed in this way

[0193] Equation 13: Inverse transform using 4x4 Haar denormalized transform

[0194]

[0195] A denormalized Haar or Walsh-Hadamard transform can sometimes increase the bitrate because the dynamics of the coefficients are increased. For example, when using a 4×4 denormalized Walsh-Hadamard matrix, the coefficient energy can be as high as 2 times the residual sample energy. Therefore, normalizing the matrix is ​​achieved by dividing the denormalized matrix by 2. However, doing so results in a loss of integer representation. Consider the same example of transforming 4 residual samples using a "normalized" Walsh-Hadamard transform:

[0196] Equation 14: Transform coefficients calculated using the 4x4 Walsh-Hadamard normalized transform

[0197]

[0198] Since integer representation is used for transform coefficients, rounding will result in an error of +-1 / 2. Therefore, an additional encoding / decoding step is required to encode / decode the hint after dividing by 2. This can be done as follows:

[0199] First, calculate the hint for the division of each coefficient:

[0200] Equation 15: Hint for the division of each coefficient calculated using the 4x4 Walsh-Hadamard normalized transform

[0201]

[0202] where the prompts (rem_c0, rem_c1, rem_c2, and rem_c3) can take values ​​(-1, 0, or 1).

[0203] The hint is then encoded into the bitstream. First, the valid hint bit is encoded, which indicates if the hint is non-zero, then, if valid, the sign of the hint is encoded (0 for negative and 1 for positive). The syntax is shown in Table 15. Differential Pulse Coded Modulation (DPCM) can also be used to exploit the correlation between hints. Once the hint is decoded, the decoder can calculate the inverse transform in this way:

[0204] Equation 16: Inverse transform using 4x4 Walsh-Hadamard normalization transform

[0205]

[0206] This technique can be used for other transform sizes, but normalization cannot always be achieved. For example, a Walsh-Hadamard transform matrix of size 2x2 needs to be divided by the square root of 2 for normalization, which will always result in a loss because the hint part is not just +-0.5. However, dividing by 2 as in the 4x4 case is beneficial because it reduces the dynamics of the increase in transform coefficients.

[0207] This technique also works for the Haar transform, although due to the non-uniform norm of the Haar transform matrix, this technique does not result in a normalized transform. For illustration, a 4x4 Haar transform is explained.

[0208] Equation 17: Transform coefficients calculated using the 4x4 Haar Normalization Transform

[0209]

[0210] Hints are calculated this way:

[0211] Equation 18: Hint for the division of each coefficient calculated using the 4x4 Haar Normalization Transform

[0212]

[0213] The hint takes values ​​of (0, -1 and 1). On the decoder side, the decoded hint is used for the inverse transform process in this way:

[0214] Equation 19: Inverse transform using 4x4 Haar normalization transform

[0215]

[0216] With Haar, Walsh-Hadamard and No Transform tools, we have multiple ways to transform the residual data in a lossless manner. In general, we have 9 options for horizontal and vertical transformation of the 2D residual signal. The following table provides their details:

[0217] Table 14

[0218]

[0219]

[0220] A straightforward way to exploit these transforms is to have the encoder select the best transform pair for the reduced rate, and encode the index of the pair used so that the decoder can deduce the pair used and perform the inverse transform.

[0221] In VTM, the current transform pairs are either the core DCT2 transform for horizontal and vertical transforms, or an additional set of DST7 and DCT8 transforms called Multi-Transform Selection (MTS). MTS can be turned on / off by a high-level flag. In order to comply with this design, the default lossless transform for both horizontal and vertical is no transform, while the "multi-transforms" are the Walsh-Hadamard transform and the Haar transform. They can also be controlled by the high-level flag "Lossless_MTS_Flag". Therefore, the transform selection table can be modified as follows:

[0222] Table 15

[0223]

[0224] Typically, the two unnormalized transforms fit the residuals well with small dimensions. For example, they can use dimensions up to 8x8 or 4x4.

[0225] Block-level with secondary lossless codec

[0226] refer to Fig. 10A , this case corresponds to step 407.

[0227] If the CU is lossy coded, but the error is bounded below a given threshold (typically if the absolute value of the error is less than or equal to 1, as shown in Equation 20), an additional codec stage can be introduced to allow lossless coding. The error is measured as the difference between the original pixel and the reconstructed sample.

[0228] Equation 20: Bounded error of a lossy codec unit to which a secondary lossless codec can be applied

[0229] -1≤error≤1

[0230] For example, if a DCT or DST transform is used, where the quantization step size is equal to 1, a secondary lossless codec can be applied after the lossy codec.

[0231] Fig.15 An example of a simplified block diagram of a secondary lossless codec is shown. The input to the process is a bitstream. In step 700, data is decoded from the bitstream. In particular, this provides quantized transform coefficients, transform types and newly introduced secondary coefficients. In step 701, inverse quantization is applied to the quantized transform coefficients. In step 702, an inverse transform is applied to the resulting transform coefficients. The resulting residual is added to the prediction signal of the intra-frame or inter-frame prediction from step 704 in step 703. The result is a reconstructed block. When the I_PCM mode is selected in step 710, the sample values ​​are directly decoded without entropy decoding. When the lossless codec mode is selected, steps 701 and 702 are skipped. When the transform skip mode is selected, step 702 is skipped. When the block level with the secondary transform mode is selected, step 701 is skipped, and the inverse transform is applied in step 720 to obtain the first residual, and the secondary lossless residual is added to the first residual. Then, in step 703, the resulting residual sum is added to the prediction signal.

[0232] Since the error is limited below a given threshold, the syntax for encoding and decoding this small residual is very simple. Basically, if the threshold is 1, only the valid flag and the sign flag may be needed to encode and decode the quadratic residual. The syntax is shown in Table 16.

[0233] Table 16: Quadratic residual syntax for threshold 1

[0234]

[0235] This secondary lossless codec can also be applied on a per-region basis instead of on a per-block basis.

[0236] Regional signaling

[0237] In the use case of lossless codec, it is possible to code losslessly for the entire region and not mix some lossless codec blocks with lossy codec blocks. To handle this case, the actual syntax requires cu_transquant_bypass_flag for each CU codec, which can be costly.

[0238] In the first embodiment, we propose to move this flag and encode it before split_cu_flag so that all child CUs from the parent CU can share region_transquant_bypass_flag, and the associated syntax is shown in Table 17.

[0239] Table 17: Proposed syntax for region_transquant_bypass_flag

[0240]

[0241] Fig.16 An example flow chart of the parsing process of region_transquant_bypass_flag and split_cu_flag when using region level signaling is shown. When parsing the syntax of the codec tree unit, the first step (500) is to initialize two variables to false, where isRegionTransquantCoded indicates whether the flag for indicating lossless codec has been encoded for the current region, and isRegionTransquant indicates whether the current region is lossless coded. Then step 501 checks whether lossless codec is enabled at the picture level (Transquant_bypass_enabled_flag is true) and region_transquant_bypass_flag has not been encoded for the current region (isRegionTransquantCoded is false). If these conditions are true, the flag region_transquant_bypass_flag is parsed in step 502. Otherwise, step 504 parses split_cu_flag. After step 502, step 503 checks whether region_transquant_bypass_flag is true. If region_transquant_bypass_flag is true, the variable isRegionTransquant is set to true in step 504. Then split_cu_flag is parsed in step 504. If split_cu_flag is true, the process returns to step 500, otherwise the tree is no longer split and the process ends (step 505). The syntax of the current codec unit may be parsed.

[0242] Lossless encoding and decoding in dual-tree case

[0243] Fig.17 Different partitioning of luma and chroma in the dual-tree case is shown. In VVC, a codec structure called dual-tree allows separate coding of the luma and chroma trees for intra-frame slices.

[0244] In an embodiment, cu_transquant_bypass_flag is parsed for each tree individually, which allows for more flexible signaling of lossless codecs.

[0245] In a variant embodiment, cu_transquant_bypass_flag is not coded for the chroma tree, but is inferred from the luma tree. For a given chroma CU, if one of the co-located luma CUs is losslessly coded, the current chroma CU is also losslessly coded. The figure shows different partitions of the luma tree and the chroma tree, for example, if the luma CU L4 is losslessly coded, the chroma CUs C3 and C4 are losslessly coded. In another example, if the luma CU L3 is losslessly coded, the chroma CU C2 is also losslessly coded, even if the co-located luma CU L2 is not losslessly coded.

[0246] The present application describes various aspects, including tools, features, embodiments, models, methods, etc. Aspects in these aspects are specifically described, and are usually described in a manner that may sound restrictive at least to show their own characteristics. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with the various aspects described in earlier files.

[0247] The aspects described and contemplated in this application can be implemented in a variety of different forms. Figure 1A , 1B Some embodiments are provided in the embodiments of the present invention, but other embodiments are contemplated and the discussion of these figures does not limit the breadth of implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to sending a generated or encoded bitstream. These and other aspects may be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the methods.

[0248] Various methods are described herein, and each of these methods includes one or more steps or actions for implementing the method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0249] The various methods and other aspects described in this application can be used to modify e.g. Figure 1A and Figure 1B Modules of the video encoder 100 and decoder 200 shown, such as motion compensation and motion estimation modules (170, 175, 275). In addition, the present aspects are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, whether existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise stated, or technically excluded, the various aspects described in this application can be used alone or in combination.

[0250] Various numerical values ​​are used in this application. These specific values ​​are for illustrative purposes, and the various aspects described are not limited to these specific values.

[0251] Various implementations involve decoding. "Decoding" as used in the present application may include, for example, all or part of the processing performed on the received coded sequence to produce a final output suitable for display. In various embodiments, such processing includes one or more of the processing typically performed by a decoder. In various embodiments, such processing also includes or alternatively includes the processing performed by the decoder of the various implementations described in the present application.

[0252] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to specifically refer to a subset of operations or to refer generally to a broader decoding process will be clear based on the context of the specific description and is considered to be fully understood by those skilled in the art.

[0253] Various implementations involve encoding. In a manner similar to the discussion above about "decoding", "encoding" as used in this application may include, for example, all or part of the processing performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processing includes one or more of the processing typically performed by an encoder. In various embodiments, such processing also includes or alternatively includes processing performed by an encoder of the various implementations described in this application.

[0254] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to refer generally to a broader encoding process will be clear based on the context of the specific description and is considered to be fully understood by those skilled in the art.

[0255] Note that the grammatical elements as used herein are descriptive terms. Therefore, they do not exclude the use of other grammatical element names.

[0256] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0257] Various embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered, usually given a computational complexity constraint. Rate-distortion optimization is usually formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all coding options, including all considered modes or codec parameter values, and a complete evaluation of their codec costs and the associated distortion of the encoded and decoded reconstructed signal. Faster approaches can also be used to save coding complexity, in particular, to calculate approximate distortion based on a predicted or predicted residual signal rather than a reconstructed signal. A mixture of these two approaches can also be used, such as by using approximate distortion for only some of the possible coding options and full distortion for other coding options. Other approaches only evaluate a subset of possible coding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of codec costs and associated distortion.

[0258] The present application describes various aspects, including tools, features, embodiments, models, methods, etc. Aspects in these aspects are specifically described, and are usually described in a manner that may sound restrictive at least to show their own characteristics. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with the various aspects described in earlier files.

[0259] The implementation and various aspects described herein can be implemented, for example, in methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the discussed features can also be implemented in other forms (e.g., devices or programs). The device can be implemented, for example, with appropriate hardware, software, and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a tablet computer, a smart phone, a mobile phone, a portable / personal digital assistant, and other devices that contribute to the information communication between end users.

[0260] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation", and other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation", and any other variations thereof, appearing in various places throughout this application do not necessarily all refer to the same embodiment.

[0261] Additionally, this application may refer to “determining” various pieces of information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0262] Additionally, this application may refer to "accessing" various pieces of information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0263] Furthermore, the present application may refer to "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include, for example, one or more of accessing information, or retrieving information (e.g., from a memory). Furthermore, "receiving" is often referred to in one way or another during operations such as, for example, storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0264] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture", "frame", "slice" and "tile" are used interchangeably. Usually, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0265] It should be understood that, for example, in the case of "A / B", "A and / or B", and "at least one of A and B", the use of any of the following " / ", "and / or", and "at least one of..." is intended to include selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As another example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such language is intended to include selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to as many items as listed.

[0266] In addition, as used herein, the term "signal" means, among other things, to indicate something to a corresponding decoder. For example, in some embodiments, the encoder signals a specific one of the illumination compensation parameters. In this way, in an embodiment, the same parameters are used on both the encoder side and the decoder side. Thus, for example, the encoder can send (explicit signaling) a specific parameter to the decoder so that the decoder can use the same specific parameter. Conversely, if the decoder already has the specific parameter as well as other parameters, signaling can be used without sending (implicit signaling) to simply allow the decoder to know and select the specific parameter. Bit savings are achieved in various embodiments by avoiding the transmission of any actual functionality. It should be understood that signaling can be accomplished in a variety of ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the term "signal" was mentioned above, the term "signal" can also be used as a noun in this article.

[0267] It will be apparent to one of ordinary skill in the art that various implementations can generate various signals that are formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for executing the method, or data generated by one of the implementations. For example, a signal can be formatted to carry a bitstream of the embodiment. Such a signal can be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be sent via a variety of different wired or wireless links. The signal can be stored on a processor readable medium.

Claims

1. A method for determining a type of residual coding for a block of an image or video, the method comprising: - obtaining first information, the first information indicating use of transform skip; - obtaining second information, the second information indicating to use conventional residual coding and decoding for the transform skip block; as well as - In response to said first information and said second information, determining that normal residual codec has to be used instead of transform skip residual codec which should be used for said block.

2. A method for decoding a block of pixels of an image or video, the method comprising: - determining the type of residual coding for the block according to claim 1; as well as - decoding the block using a prediction residual, wherein the prediction residual is decoded using the determined residual codec.

3. An apparatus for decoding a block of pixels of an image or video, the apparatus comprising a processor configured to: - determining according to claim 1 the type of residual coding for said block; and - decoding the block using a prediction residual, wherein the prediction residual is decoded using the determined residual codec.

4. A non-transitory computer-readable medium comprising program code instructions for implementing the steps of the method according to claim 1 when the program code instructions are executed by a processor.

Images