Enhancement layer encoding and decoding method and device

Abstract

The present application provides an enhancement layer encoding and decoding method and apparatus. The method includes: obtaining a reconstructed block of a base layer of an image block to be encoded, taking the difference between corresponding pixel points in the image block to be encoded and the reconstructed block of the base layer to obtain a residual block of a first enhancement layer of the image block to be encoded. Dividing the residual block of the first enhancement layer into transform blocks and then transforming them to obtain transform coefficients of the first enhancement layer. Quantizing and entropy encoding the transform coefficients of the first enhancement layer to obtain a code stream of the first enhancement layer; quantizing and inversely quantizing the transform coefficients of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer. Taking the difference between the transform coefficients of the first enhancement layer and the dequantized coefficients of the first enhancement layer to obtain a residual block of coefficients of the second enhancement layer, quantizing and entropy encoding the residual block of coefficients of the second enhancement layer to obtain a code stream of the second enhancement layer. The present application can reduce the encoder processing flow, improve the encoder coding efficiency while taking into account the complexity.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of video or image compression, and in particular to an enhancement layer encoding and decoding method and device. Background Art

[0002] Video codecs (video encoding and decoding) are widely used in digital video applications such as broadcast digital television, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blu-ray Discs, video content acquisition and editing systems, and security applications in camcorders.

[0003] Even in the case of relatively short films, large amounts of video data are required to describe them, which can create difficulties when the data is sent or otherwise transmitted across networks with limited bandwidth capacity. Therefore, video data is often compressed before being transmitted across modern telecommunications networks. Since memory resources may be limited, the size of the video can also be an issue when storing the video on a storage device. Video compression equipment typically uses software and / or hardware at the source to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received by a video decompression device at the destination. With limited network resources and the growing demand for higher video quality, there is a need for improved compression and decompression techniques that can increase compression rates with little to no impact on image quality.

[0004] Nowadays, when watching live video or making video calls, people have increasingly stringent requirements for video continuity and end-to-end transmission latency. Typically, servers need to prepare video streams with different characteristics to meet these user requirements for video stability. Servers typically address this issue by configuring multiple encoders, or by using a single encoder for multiple encodings to produce videos with different resolutions, frame rates, and bit rates. However, this requires the use of multiple encoder resources.

[0005] The scalable video coding standard provides a solution that achieves this goal with only a single encoding pass. It performs spatial scalability (resolution scalability), temporal scalability, and quality scalability within the encoder to create different stream layers. This allows the same stream to contain video streams of varying resolutions, frame rates, and bit rates.

[0006] Scalable video coding involves sending a video source into a scalable encoder and encoding it into a base layer stream and an enhancement layer stream, depending on the encoding configuration. The base layer generally encodes the lowest spatial and temporal layer, or the lowest quality stream. The enhancement layer, based on the base layer, superimposes and encodes higher spatial and temporal layers, or higher quality streams. As the number of enhancement layers increases, the spatial, temporal, or quality levels of the encoding become increasingly higher. During transmission, the base layer stream is prioritized, and when network capacity is sufficient, the enhancement layers are gradually transmitted. At the decoder, the base layer stream is first received and decoded. Then, based on the received enhancement layer streams, from lower to higher levels, the code streams of higher spatial, temporal, or quality levels are gradually decoded. By superimposing higher-level information on lower layers, higher resolution, higher frame rate, or higher quality video is obtained. However, the encoding and decoding process in existing technologies is relatively complex and inefficient. Summary of the Invention

[0007] In an embodiment of the present application, after image blocks are layered using spatial, temporal, or quality domain stratification, in addition to a base layer, they are further divided into multiple enhancement layers. This application provides an enhancement layer encoding and decoding method and apparatus that can reduce the processing flow of an encoder and improve the encoding efficiency of the encoder.

[0008] In a first aspect, the present application provides an enhancement layer encoding and decoding method, wherein the encoding method includes: an encoder obtains a reconstructed block of a base layer of an image block to be encoded, and differs corresponding pixel points in the image block to be encoded and the reconstructed block of the base layer to obtain a residual block of a first enhancement layer of the image block to be encoded; according to the residual block of the first enhancement layer, a transform block of the first enhancement layer is obtained, the transform block of the first enhancement layer is transformed to obtain a transform coefficient of the first enhancement layer, and the transform coefficient of the first enhancement layer is quantized and dequantized to obtain a dequantized coefficient of the first enhancement layer; the transform coefficient of the first enhancement layer is quantized and entropy encoded to obtain a code stream of the first enhancement layer; the transform coefficient of the first enhancement layer is differed from the dequantized coefficient of the first enhancement layer to obtain a coefficient residual block of a second enhancement layer of the image block to be encoded; and the coefficient residual block of the second enhancement layer is quantized and entropy encoded to obtain a code stream of the second enhancement layer.

[0009] In the first aspect and possible implementations thereof, the first enhancement layer corresponds to the residual pixel layer in the enhancement layer, and the second enhancement layer corresponds to the transform domain coefficient layer in the enhancement layer.

[0010] In a possible implementation, a difference is calculated between corresponding pixel points in the transform coefficients of the first enhancement layer and the dequantized coefficients of the first enhancement layer to obtain a coefficient residual block of the second enhancement layer.

[0011] In traditional methods, the encoder predicts the enhancement layer's prediction block based on the base layer's reconstructed block, then takes the difference between the corresponding pixels in the base layer's reconstructed block and the enhancement layer's prediction block to obtain the enhancement layer's residual. The present embodiment can directly take the difference between the corresponding pixels in the image block to be encoded and the base layer's reconstructed block to obtain the residual of the residual pixel layer, simplifying the process of obtaining the prediction block of the residual pixel layer, reducing the encoder's processing flow, and improving the encoder's coding efficiency.

[0012] In the embodiment of the present application, the transform block division method of the residual block of the residual pixel layer is different from the transform block division method of the residual of the base layer. That is, when the encoder processes the residual block of the residual pixel layer, the transform block division method adopted is different from the transform block division method adopted when processing the residual block of the base layer. For example, the residual block of the base layer is divided into three sub-blocks using the TT division method, but the residual block of the enhancement layer is not divided using the TT division method. The encoder can use an adaptive method to determine the transform block division method of the residual block of the enhancement layer.

[0013] The present application no longer uses the same transform unit (TU) division method as the residual of the base layer for the residual block of the residual pixel layer, but adopts an adaptive TU division method to obtain a TU division method suitable for the residual block of the residual pixel layer, and then encodes it. When the TU division of the residual of the residual pixel layer is independent of the CU division method of the base layer, the TU size of the residual pixel layer is no longer limited by the CU size, which can improve the flexibility of encoding.

[0014] When there are multiple residual pixel layers, the TU division methods between the residual pixel layers are independent of each other. Each residual pixel layer adopts an independent adaptive TU division method to obtain a TU division method of the residual block suitable for each residual pixel layer, and then encodes it.

[0015] The encoder may transform, quantize, and entropy encode the residual blocks of the residual pixel layer according to the above-mentioned transform block division method to obtain a code stream of the residual pixel layer.

[0016] In a possible implementation, the image block to be encoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the first enhancement layer) and the first transform domain coefficient layer (corresponding to the second enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The encoder calculates the difference between the corresponding pixel points in the image block to be encoded and the reconstructed block of the second enhancement layer to obtain the residual block of the third enhancement layer of the image block to be encoded; based on the residual block of the third enhancement layer, the transform block of the third enhancement layer is obtained, and the transform block of the third enhancement layer is transformed to obtain the transform coefficient of the third enhancement layer; the transform coefficient of the third enhancement layer is quantized and entropy encoded to obtain the code stream of the third enhancement layer.

[0017] In one possible implementation, the image block to be encoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the second enhancement layer) and the first residual pixel layer (corresponding to the first enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the fourth enhancement layer) and a second residual pixel layer (corresponding to the third enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The encoder quantizes and dequantizes the transform coefficients of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; calculates the difference between the transform coefficients of the third enhancement layer and the dequantized coefficients of the third enhancement layer to obtain a coefficient residual block of the fourth enhancement layer of the image block to be encoded; and quantizes and entropy encodes the coefficient residual block of the fourth enhancement layer to obtain a code stream of the fourth enhancement layer.

[0018] In one possible implementation, the image block to be encoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the first enhancement layer) and the first transform domain coefficient layer (corresponding to the second enhancement layer), the image block to be encoded may also include a second residual pixel layer (corresponding to the fourth enhancement layer) and a second transform domain coefficient layer (corresponding to the third enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The encoder quantizes and inversely quantizes the coefficient residual block of the third enhancement layer to obtain the dequantized coefficients of the third enhancement layer; sums the corresponding pixel points of the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the first enhancement layer to obtain the reconstructed dequantized coefficients of the second enhancement layer; sums the corresponding pixel points in the dequantized coefficients of the third enhancement layer and the reconstructed dequantized coefficients of the second enhancement layer and then inversely transforms to obtain the reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the base layer to obtain the reconstructed block of the third enhancement layer; takes the difference between the corresponding pixel points in the image block to be encoded and the reconstructed block of the third enhancement layer to obtain the residual block of the fourth enhancement layer; obtains the transform block of the fourth enhancement layer according to the residual block of the fourth enhancement layer, transforms the transform block to obtain the transform coefficients of the fourth enhancement layer; quantizes and entropy encodes the transform coefficients of the fourth enhancement layer to obtain the code stream of the fourth enhancement layer.

[0019] In a possible implementation, the image block to be encoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the second enhancement layer) and the first residual pixel layer (corresponding to the first enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The encoder quantizes and dequantizes the coefficient residual block of the second enhancement layer to obtain the dequantized coefficients of the second enhancement layer; sums the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the first enhancement layer to obtain the reconstructed dequantized coefficients of the second enhancement layer; takes the difference between the corresponding pixel points of the transform coefficients of the first enhancement layer and the reconstructed dequantized coefficients of the second enhancement layer to obtain the coefficient residual block of the third enhancement layer; quantizes and entropy encodes the coefficient residual block of the third enhancement layer to obtain the code stream of the third enhancement layer.

[0020] The above method can be applied to both spatial domain scalarization and quality domain scalarization. When spatial domain scalarization is used to obtain base layer image blocks and enhancement layer image blocks, the resolution of the base layer image blocks is lower than that of the enhancement layer image blocks. Therefore, before the encoder calculates the difference between the corresponding pixels in the image block to be coded and the reconstructed block of the base layer or low enhancement layer to obtain the residual pixel layer of the image block to be coded, it can downsample the original image block to be coded to obtain the image block to be coded at the first resolution, and upsample the original reconstructed block of the base layer or low enhancement layer to obtain the reconstructed block of the base layer or low enhancement layer at the first resolution. That is, the encoder can downsample the original image block to be coded and upsample the original reconstructed block of the base layer or low enhancement layer to make the resolution of the image block to be coded and the resolution of the reconstructed block of the base layer the same.

[0021] The decoding method of the first aspect includes: a decoder obtains a code stream of a first enhancement layer and a code stream of a second enhancement layer of an image block to be decoded; performs entropy decoding and inverse quantization on the code stream of the first enhancement layer to obtain a dequantized coefficient of the first enhancement layer; performs entropy decoding, inverse quantization and inverse transformation on the code stream of the first enhancement layer to obtain a reconstructed residual block of the first enhancement layer; obtains a reconstructed block of a base layer of the image block to be decoded, sums corresponding pixels in the reconstructed residual block of the first enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the first enhancement layer; performs entropy decoding and inverse quantization on the code stream of the second enhancement layer to obtain a dequantized coefficient of the second enhancement layer; sums corresponding pixels in the dequantized coefficients of the first enhancement layer and the dequantized coefficients of the second enhancement layer, and then performs inverse transformation to obtain a reconstructed residual block of the second enhancement layer; and sums corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the second enhancement layer.

[0022] In one possible implementation, the image to be decoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the first enhancement layer) and the first transform domain coefficient layer (corresponding to the second enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The decoder obtains the code stream of the third enhancement layer of the image block to be decoded; performs entropy decoding, inverse quantization and inverse transformation on the code stream of the third enhancement layer to obtain the reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the second enhancement layer to obtain the reconstructed block of the third enhancement layer.

[0023] In one possible implementation, the image block to be decoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the second enhancement layer) and the first residual pixel layer (corresponding to the first enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the fourth enhancement layer) and a second residual pixel layer (corresponding to the third enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The decoder obtains the code stream of the fourth enhancement layer of the image block to be decoded; performs entropy decoding and inverse quantization on the code stream of the fourth enhancement layer to obtain the dequantized coefficients of the fourth enhancement layer; performs entropy decoding and inverse quantization on the code stream of the third enhancement layer to obtain the dequantized coefficients of the third enhancement layer; sums the corresponding pixel points in the dequantized coefficients of the third enhancement layer and the dequantized coefficients of the fourth enhancement layer, and then performs inverse transformation to obtain a reconstructed residual block of the fourth enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the second enhancement layer to obtain a reconstructed block of the fourth enhancement layer.

[0024] In one possible implementation, the image to be decoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the first enhancement layer) and the first transform domain coefficient layer (corresponding to the second enhancement layer), it may also include a second residual pixel layer (corresponding to the fourth enhancement layer) and a second transform domain coefficient layer (corresponding to the third enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The decoder obtains the code stream of the fourth enhancement layer of the image block to be decoded; performs entropy decoding, inverse quantization and inverse transformation on the code stream of the fourth enhancement layer to obtain the reconstructed residual block of the fourth enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the third enhancement layer to obtain the reconstructed block of the fourth enhancement layer.

[0025] In one possible implementation, the image block to be decoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the second enhancement layer) and the first residual pixel layer (corresponding to the first enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The decoder obtains a code stream of the third enhancement layer of the image block to be decoded; performs entropy decoding and inverse quantization on the code stream of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; sums the corresponding pixels in the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the first enhancement layer to obtain reconstructed dequantized coefficients of the second enhancement layer; sums the corresponding pixels in the reconstructed dequantized coefficients of the second enhancement layer and the dequantized coefficients of the third enhancement layer, and then performs inverse transformation to obtain a reconstructed residual block of the third enhancement layer; sums the corresponding pixels in the reconstructed residual block of the third enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the third enhancement layer.

[0026] In the first aspect and its possible implementation methods, the implementation process of each enhancement layer is implemented in the image residual domain, which reduces the process of obtaining the prediction block of the residual pixel layer, can reduce the processing flow of the decoder, and improve the decoding efficiency of the decoder.

[0027] The residual of the current residual pixel layer no longer uses the TU division method of the residual of the base layer or other residual pixel layers. Instead, an adaptive TU division method is used to obtain a TU division method of the residual block suitable for the current residual pixel layer, and then decoding is performed. When the TU division of the residual block of the current residual pixel layer is independent of the CU division method of the base layer, the TU size of the residual pixel layer is no longer limited by the CU size, which can more effectively improve the compression efficiency of the residual.

[0028] The current transform domain coefficient layer is implemented in the transform domain, and the residual of the current transform domain coefficient layer directly inherits the TU division method of the residual of the base layer or other enhancement layers, and then decoded, avoiding the transformation process and the TU division implementation process, which can more effectively reduce the implementation complexity.

[0029] In a second aspect, the present application provides an enhancement layer encoding and decoding method, wherein the encoding method includes: the encoder transforms, quantizes and entropy encodes the residual block of the base layer of the image block to be encoded to obtain the code stream of the base layer, transforms, quantizes and inversely quantizes the residual block of the base layer of the image block to be encoded to obtain the dequantized coefficient of the base layer, transforms the transform block of the base layer to obtain the transform coefficient of the base layer, calculates the difference between the corresponding pixel points of the transform coefficient of the base layer and the dequantized coefficient of the base layer to obtain the coefficient residual block of the first enhancement layer of the image block to be encoded, quantizes and entropy encodes the coefficient residual block of the first enhancement layer to obtain the code stream of the first enhancement layer, quantizes and inversely quantizes the coefficient residual block of the first enhancement layer to obtain the dequantized coefficient of the first enhancement layer, and The dequantized coefficients of the first layer and the corresponding pixel points of the dequantized coefficients of the base layer are summed to obtain the reconstructed dequantized coefficients of the first enhancement layer, the reconstructed dequantized coefficients of the first enhancement layer are inversely transformed to obtain the reconstructed residual block of the first enhancement layer, the reconstructed residual block of the first enhancement layer and the corresponding pixel points in the prediction block of the base layer are summed to obtain the reconstructed block of the first enhancement layer, the corresponding pixel points in the reconstructed block of the image block to be encoded and the reconstructed block of the first enhancement layer are subtracted to obtain the residual block of the second enhancement layer of the image block to be encoded, the transform block of the second enhancement layer is obtained according to the residual block of the second enhancement layer, the transform block of the second enhancement layer is transformed to obtain the transform coefficients of the second enhancement layer, and the transform coefficients of the second enhancement layer are quantized and entropy encoded to obtain the code stream of the second enhancement layer.

[0030] In the second aspect and possible implementations thereof, the first enhancement layer corresponds to the transform domain coefficient layer in the enhancement layer, and the second enhancement layer corresponds to the residual pixel layer in the enhancement layer.

[0031] In the second aspect and its possible implementation methods, after the image blocks are layered using spatial domain grading, temporal domain grading or quality domain grading, in addition to the basic layer, they are also divided into multiple enhancement layers. Compared with the first aspect, this embodiment is a coding and decoding method based on the combination of transform domain coefficient layer and residual pixel layer.

[0032] In a possible implementation, obtaining the reconstructed residual block of the first enhancement layer according to the coefficient residual block of the first enhancement layer may specifically include:

[0033] quantizing and inverse-quantizing the coefficient residual block of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; summing corresponding pixel points in the dequantized coefficients of the base layer and the dequantized coefficients of the first enhancement layer to obtain reconstructed dequantized coefficients of the first enhancement layer of the image block to be encoded; and inversely transforming the reconstructed dequantized coefficients of the first enhancement layer of the image block to be encoded to obtain a reconstructed residual block of the first enhancement layer of the image block to be encoded.

[0034] In a possible implementation, the image block to be encoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the second enhancement layer) and the first transform domain coefficient layer (corresponding to the first enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The transform coefficients of the second enhancement layer are quantized, dequantized, and inversely transformed to obtain a reconstructed block of the second enhancement layer; the corresponding pixel points in the image block to be encoded and the reconstructed block of the second enhancement layer are subtracted to obtain a residual block of the third enhancement layer of the image block to be encoded; based on the residual block of the third enhancement layer, the transform block of the third enhancement layer is obtained, and the transform block is transformed to obtain the transform coefficients of the third enhancement layer; the transform coefficients of the third enhancement layer are quantized and entropy coded to obtain the code stream of the third enhancement layer.

[0035] In one possible implementation, the image block to be encoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The transform coefficients of the third enhancement layer are quantized and inversely quantized to obtain the dequantized coefficients of the third enhancement layer; the transform coefficients of the third enhancement layer of the image block to be encoded and the corresponding pixel points of the dequantized coefficients of the third enhancement layer are subtracted to obtain the coefficient residual block of the fourth enhancement layer; the coefficient residual block of the fourth enhancement layer is quantized and entropy coded to obtain the code stream of the fourth enhancement layer.

[0036] In one possible implementation, the image block to be encoded may include multiple residual pixel layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), the image block to be encoded may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The encoder quantizes and inversely quantizes the coefficient residual block of the third enhancement layer to obtain the dequantized coefficients of the third enhancement layer; sums the corresponding pixel points in the dequantized coefficients of the third enhancement layer and the dequantized coefficients of the second enhancement layer and then inversely transforms to obtain the reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the first enhancement layer to obtain the reconstructed block of the third enhancement layer; differs the corresponding pixel points in the image block to be encoded and the reconstructed block of the third enhancement layer to obtain the residual block of the fourth enhancement layer; obtains the transform block of the fourth enhancement layer according to the residual block of the fourth enhancement layer, transforms the transform block to obtain the transform coefficients of the fourth enhancement layer; quantizes and entropy encodes the transform coefficients of the fourth enhancement layer to obtain the code stream of the fourth enhancement layer.

[0037] In one possible implementation, the image block to be encoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The encoder quantizes and dequantizes the transform coefficients of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer;

[0038] Differences are taken between corresponding pixel points of the transform coefficients of the second enhancement layer of the image block to be encoded and the dequantized coefficients of the second enhancement layer to obtain a coefficient residual block of the third enhancement layer; and quantization and entropy encoding are performed on the coefficient residual block of the third enhancement layer to obtain a code stream of the third enhancement layer.

[0039] In the second aspect of the present application and its possible implementations, the transform block partitioning method of the residual block of the second residual pixel layer is different from the transform block partitioning method of the residual block of the first residual pixel layer. The encoder can also use the above-mentioned RDO processing to determine the transform block partitioning method of the residual block of the second enhancement layer, which is not further described here.

[0040] The second aspect of the present application and its possible implementation method no longer use the same TU division method for the residual blocks of the second residual pixel layer as the residual blocks of the first residual pixel layer and the residual blocks of the base layer, but adopt an adaptive TU division method to obtain a TU division method suitable for the residual blocks of the second enhancement layer, and then encode them. When the TU division of the residual blocks of the second residual pixel layer is independent of the CU or TU division method of other layers, the TU size of the second residual pixel layer is no longer limited by the CU size, or the TU size of the second residual pixel layer is no longer limited by the TU size of the first residual pixel layer, which can improve the flexibility of encoding.

[0041] The second aspect of the present application and its possible implementation method can directly calculate the difference between the corresponding pixel points in the reconstructed block of the coding layer (base layer or enhancement layer) one layer lower than the second residual pixel layer and the image block to be coded to obtain the residual block of the second residual pixel layer, thereby reducing the process of obtaining the prediction block of the second residual pixel layer, reducing the processing flow of the encoder, and improving the coding efficiency of the encoder. In addition, the residual block of the second residual pixel layer no longer uses the same TU division method as the residual block of the first residual pixel layer, but adopts an adaptive TU division method to obtain a TU division method suitable for the residual block of the second residual pixel layer, and then encodes it. When the TU division of the residual block of the second residual pixel layer is independent of the CU or TU division method of the first residual pixel layer, the TU size of the second residual pixel layer is no longer limited by the CU size, or the TU size of the second residual pixel layer is no longer limited by the TU size of the first residual pixel layer, which can more effectively improve the compression efficiency of the residual block.

[0042] The above method can be applied to both spatial domain grading and quality domain grading. When an image block adopts spatial domain grading to obtain a first residual pixel layer image block and a second residual pixel layer image block, the resolution of the first residual pixel layer image block is lower than the resolution of the second residual pixel layer image block. Therefore, before the encoder calculates the difference between the corresponding pixel points in the image block to be coded and the reconstructed block of the coding layer (base layer or enhancement layer) one layer lower than the first residual pixel layer to obtain the residual block of the second residual pixel layer of the image block to be coded, it can downsample the original image block to be coded to obtain the image block to be coded at the first resolution, and upsample the reconstructed block of the original first residual pixel layer to obtain the reconstructed block of the first residual pixel layer at the first resolution. That is, the encoder can downsample the original image block to be coded and upsample the reconstructed block of the original first enhancement layer so that the resolution of the image block to be coded and the resolution of the reconstructed block of the first residual pixel layer are the same.

[0043] The decoding method of the second aspect includes: a decoder obtaining a code stream of a base layer, a code stream of a first enhancement layer, and a code stream of a second enhancement layer of an image block to be decoded; performing entropy decoding on the code stream of the base layer to obtain quantization coefficients of the base layer; performing inverse quantization on the quantization coefficients of the base layer to obtain dequantization coefficients of the base layer; performing entropy decoding and inverse quantization on the code stream of the first enhancement layer to obtain dequantization coefficients of the first enhancement layer; summing corresponding pixels in the dequantization coefficients of the first enhancement layer and the dequantization coefficients of the base layer, and then performing inverse transformation to obtain a reconstructed residual block of the first enhancement layer; summing corresponding pixels in the reconstructed residual block of the first enhancement layer and the prediction block of the base layer to obtain a reconstructed block of the first enhancement layer; performing entropy decoding, inverse quantization, and inverse transformation on the code stream of the second enhancement layer to obtain a reconstructed residual block of the second enhancement layer; and summing corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the first enhancement layer to obtain a reconstructed block of the second enhancement layer.

[0044] In one possible implementation, the image block to be decoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the second enhancement layer) and the first transform domain coefficient layer (corresponding to the first enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The decoder obtains the code stream of the third enhancement layer of the image block to be decoded; performs entropy decoding, inverse quantization and inverse transformation on the residual code stream of the third enhancement layer to obtain the reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the second enhancement layer to obtain the reconstructed block of the third enhancement layer.

[0045] In one possible implementation, the image block to be decoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the fourth enhancement layer) and a second residual pixel layer (corresponding to the third enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The decoder obtains the code stream of the fourth enhancement layer of the image block to be decoded; performs entropy decoding and inverse quantization on the code stream of the fourth enhancement layer to obtain the dequantized coefficients of the fourth enhancement layer; performs entropy decoding and inverse quantization on the code stream of the third enhancement layer to obtain the dequantized coefficients of the third enhancement layer; sums the corresponding pixel points in the dequantized coefficients of the third enhancement layer and the dequantized coefficients of the fourth enhancement layer, and then performs inverse transformation to obtain a reconstructed residual block of the fourth enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the third enhancement layer to obtain the reconstructed block of the fourth enhancement layer.

[0046] In one possible implementation, the image block to be decoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the second enhancement layer) and the first transform domain coefficient layer (corresponding to the first enhancement layer), it may also include a second residual pixel layer (corresponding to the fourth enhancement layer) and a second transform domain coefficient layer (corresponding to the third enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The decoder obtains the code stream of the fourth enhancement layer of the image block to be decoded; performs entropy decoding, inverse quantization and inverse transformation on the residual code stream of the fourth enhancement layer to obtain the reconstructed residual block of the fourth enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the third enhancement layer to obtain the reconstructed block of the fourth enhancement layer.

[0047] In one possible implementation, the image block to be decoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The decoder obtains the code stream of the third enhancement layer of the image block to be decoded; performs entropy decoding and inverse quantization on the code stream of the third enhancement layer to obtain the dequantized coefficients of the third enhancement layer; performs entropy decoding and inverse quantization on the code stream of the second enhancement layer to obtain the dequantized coefficients of the second enhancement layer; sums the corresponding pixel points in the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the third enhancement layer, and then performs inverse transformation to obtain a reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the first enhancement layer to obtain the reconstructed block of the third enhancement layer.

[0048] In a third aspect, the present application provides an encoding device, comprising an acquisition module for obtaining a reconstructed block of a base layer of an image block to be encoded; taking the difference between corresponding pixel points in the image block to be encoded and the reconstructed block of the base layer to obtain a residual block of a first enhancement layer of the image block to be encoded; a determination module for obtaining a transform block of the first enhancement layer based on the residual block of the first enhancement layer, transforming the transform block of the first enhancement layer to obtain transform coefficients of the first enhancement layer, and quantizing and dequantizing the transform coefficients of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; the acquisition module is also used to take the difference between the transform coefficients of the first enhancement layer and the dequantized coefficients of the first enhancement layer to obtain a coefficient residual block of a second enhancement layer of the image block to be encoded; an encoding module for quantizing and entropy encoding the transform coefficients of the first enhancement layer to obtain a code stream of the first enhancement layer; and quantizing and entropy encoding the coefficient residual block of the second enhancement layer to obtain a code stream of the second enhancement layer.

[0049] In the present application, the encoding device in the third aspect can be used to implement the encoding method in the first aspect and its possible implementation methods.

[0050] In a fourth aspect, the present application provides a coding device, comprising: an acquisition module, configured to transform, quantize, and inversely quantize a residual block of a base layer of an image block to be coded to obtain dequantized coefficients of the base layer; transform the transform block of the base layer to obtain the transform coefficients of the base layer, and calculate the difference between the transform coefficients of the base layer and the corresponding pixel points of the dequantized coefficients of the base layer to obtain a coefficient residual block of a first enhancement layer of the image block to be coded; quantize and inversely quantize the coefficient residual block of the first enhancement layer to obtain the dequantized coefficients of the first enhancement layer; sum the corresponding pixel points of the dequantized coefficients of the first enhancement layer and the dequantized coefficients of the base layer to obtain the reconstructed dequantized coefficients of the first enhancement layer; and inversely transform the reconstructed dequantized coefficients of the first enhancement layer to obtain a reconstructed residual block of the first enhancement layer. ; summing the corresponding pixels in the reconstructed residual block of the first enhancement layer and the prediction block of the base layer to obtain the reconstructed block of the first enhancement layer; taking the difference between the corresponding pixels in the image block to be encoded and the reconstructed block of the first enhancement layer to obtain the residual block of the second enhancement layer of the image block to be encoded; a determination module, configured to obtain the second enhancement layer transform block according to the residual block of the second enhancement layer, and transform the transform block of the second enhancement layer to obtain the transform coefficient of the second enhancement layer; an encoding module, configured to transform, quantize and entropy encode the residual block of the base layer to obtain the code stream of the base layer; quantize and entropy encode the coefficient residual block of the first enhancement layer to obtain the code stream of the first enhancement layer; quantize and entropy encode the transform coefficient of the second enhancement layer to obtain the code stream of the second enhancement layer.

[0051] In the present application, the encoding device in the fourth aspect can be used to implement the encoding method in the second aspect and its possible implementation methods.

[0052] In a fifth aspect, the present application provides a decoding device, comprising: an obtaining module for obtaining a code stream of a first enhancement layer and a code stream of a second enhancement layer of an image block to be decoded; a decoding module for performing entropy decoding and inverse quantization on the code stream of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; performing entropy decoding, inverse quantization, and inverse transformation on the code stream of the first enhancement layer to obtain a reconstructed residual block of the first enhancement layer; performing entropy decoding and inverse quantization on the code stream of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer; summing corresponding pixels in the dequantized coefficients of the first enhancement layer and the dequantized coefficients of the second enhancement layer, and then performing inverse transformation to obtain a reconstructed residual block of the second enhancement layer; and a reconstruction module for obtaining a reconstructed block of a base layer of the image block to be decoded; summing corresponding pixels in the reconstructed residual block of the first enhancement layer and the reconstructed block of the base layer to obtain the reconstructed block of the first enhancement layer; and summing corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the base layer to obtain the reconstructed block of the second enhancement layer.

[0053] In the present application, the decoding device in the fifth aspect can be used to implement the decoding method in the first aspect and its possible implementation methods.

[0054] In a sixth aspect, the present application provides a decoding device, comprising: an acquisition module for obtaining a code stream of a base layer, a code stream of a first enhancement layer, and a code stream of a second enhancement layer of an image block to be decoded; a decoding module for entropy decoding the code stream of the base layer to obtain quantization coefficients of the base layer, and dequantizing the quantization coefficients of the base layer to obtain dequantization coefficients of the base layer; entropy decoding and dequantizing the code stream of the first enhancement layer to obtain dequantization coefficients of the first enhancement layer; summing corresponding pixels in the dequantization coefficients of the first enhancement layer and the dequantization coefficients of the base layer, and then performing an inverse transform to obtain a reconstructed residual block of the first enhancement layer; and a reconstruction module for summing corresponding pixels in the reconstructed residual block of the first enhancement layer and the prediction block of the base layer to obtain a reconstructed block of the first enhancement layer; entropy decoding, dequantization, and inverse transforming the code stream of the second enhancement layer to obtain a reconstructed residual block of the second enhancement layer; and summing corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the first enhancement layer to obtain a reconstructed block of the second enhancement layer.

[0055] In the present application, the decoding device in the sixth aspect can be used to implement the decoding method in the second aspect and its possible implementation methods.

[0056] In the seventh aspect, the present application provides an encoder comprising: a processor and a memory; the processor is coupled to the memory, and the memory stores computer-readable instructions; the processor is used to read the computer-readable instructions so that the encoder implements the method as described in any one of the first to second aspects above.

[0057] In an eighth aspect, the present application provides a decoder comprising: a processor and a memory; the processor is coupled to the memory, and the memory stores computer-readable instructions; the processor is used to read the computer-readable instructions to enable the encoder to implement the method as described in any one of the first to second aspects above.

[0058] In a ninth aspect, the present application provides a computer program product comprising program code, which, when executed on a computer or processor, is used to execute the method according to any one of the first to second aspects above.

[0059] In a tenth aspect, the present application provides a computer-readable storage medium, characterized in that it includes program code, which, when executed by a computer device, is used to execute the method according to any one of the first to second aspects above.

[0060] The enhancement layers in this embodiment of the present invention include at least a first enhancement layer and a second enhancement layer, namely, a residual pixel layer and a transform domain coefficient layer. Through the rational and effective combination of the residual pixel layer and the transform domain coefficient layer, compression efficiency can be effectively improved while achieving hardware-friendly implementation and flexible control of codec complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1A is an exemplary block diagram of a decoding system 10 according to an embodiment of the present application;

[0062] Figure 1B is an exemplary block diagram of a video decoding system 40 according to an embodiment of the present application;

[0063] Figure 2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application;

[0064] Figure 3a to Figure 3g Several exemplary schematic diagrams of the division method of the embodiments of the present application;

[0065] Figure 4 This is an exemplary schematic diagram of the QT-MTT division method of this application;

[0066] Figure 5 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application;

[0067] Figure 6 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application;

[0068] Figure 7 is an exemplary block diagram of an apparatus 500 according to an embodiment of the present application;

[0069] Figure 8 This is an exemplary flowchart of the decoding method according to an embodiment of the present application;

[0070] Figure 9 This is an exemplary hierarchical diagram of scalable video decoding in this application;

[0071] Figure 10 This is an example diagram of the decoding method according to an embodiment of the present application;

[0072] Figure 11 This is an exemplary flowchart of the decoding method according to an embodiment of the present application;

[0073] Figure 12 This is an exemplary flowchart of the layer decoding method according to an embodiment of the present application;

[0074] Figure 13 This is an exemplary flowchart of the decoding method according to an embodiment of the present application;

[0075] Figure 14a and 14b This is an example diagram of the decoding method according to an embodiment of the present application;

[0076] Figure 15a and 15b This is an example diagram of the decoding method according to an embodiment of the present application;

[0077] Figure 16 This is an example diagram of the decoding method according to an embodiment of the present application;

[0078] Figure 17 This is an example diagram of the decoding method according to an embodiment of the present application;

[0079] Figure 18 This is an example diagram of the decoding method according to an embodiment of the present application;

[0080] Figure 19a and 19b This is an example diagram of a spatially hierarchical hybrid coding and decoding framework according to an embodiment of the present application;

[0081] Figure 20 This is an exemplary structural diagram of the encoding device of this application;

[0082] Figure 21 This is a schematic diagram of an exemplary structure of the decoding device of this application. DETAILED DESCRIPTION

[0083] The embodiments of the present application provide a video image compression technology, specifically a codec architecture based on hierarchical residual coding to improve a traditional hybrid video codec system.

[0084] Video coding and decoding generally refers to the processing of a sequence of images that form a video or video sequence. In the field of video coding and decoding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding (coding, or commonly referred to as coding and decoding) includes two parts: video encoding and video decoding. Video coding is performed on the source side and generally includes processing (for example, compressing) the original video image to reduce the amount of data required to represent the video image (thereby making it more efficient to store and / or transmit). Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video image. The "coding" of the video image (or commonly referred to as the image) involved in the embodiment should be understood as the "coding" or "decoding" of the video image or video sequence. The coding part and the decoding part are also collectively referred to as coding and decoding (coding and decoding, CODEC).

[0085] In the case of lossless video coding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming there is no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed through quantization, etc. to reduce the amount of data required to represent the video image, but the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0086] Several video coding standards fall under the category of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is typically divided into a set of non-overlapping blocks, which are typically coded at the block level. In other words, the encoder typically processes, i.e., encodes, the video at the block (video block) level, for example, generating a prediction block through spatial (intra-frame) and temporal (inter-frame) prediction; subtracting the prediction block from the current block (currently processed / to-be-processed block) to obtain a residual block; transforming the residual block in the transform domain and dequantizing the coefficients to reduce the amount of data to be transmitted (compressed), while the decoder applies the inverse of the encoder's processing to the coded or compressed block to reconstruct the current block for representation. Furthermore, the encoder needs to repeat the decoder's processing steps so that the encoder and decoder generate the same predictions (e.g., intra-frame and inter-frame predictions) and / or reconstructed pixels for processing, i.e., encoding, the subsequent block.

[0087] In the following embodiment of the decoding system 10, the encoder 20 and the decoder 30 are based on Figure 1A The description proceeds to FIG3 .

[0088] Figure 1A FIG1 is an exemplary block diagram of a decoding system 10 according to an embodiment of the present application, for example, a video decoding system 10 (or simply decoding system 10) that can utilize the techniques of the present application. The video encoder 20 (or simply encoder 20) and the video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices that can be used to perform various techniques according to various examples described in this application.

[0089] like Figure 1A As shown, the decoding system 10 includes a source device 12 for providing encoded image data 21 such as an encoded image to a destination device 14 for decoding the encoded image data 21.

[0090] The source device 12 includes an encoder 20 , and optionally, may include an image source 16 , a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22 .

[0091] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for obtaining and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0092] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0093] The preprocessor 18 is configured to receive raw image data 17 and preprocess the raw image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.

[0094] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. for further description).

[0095] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0096] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0097] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0098] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0099] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0100] The communication interface 28 corresponds to the communication interface 22 , and can be used, for example, to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0101] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1A The unidirectional communication interface or the bidirectional communication interface indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0102] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (which will be further described below with reference to FIG. 3 and the like).

[0103] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image, to obtain post-processed image data 33, such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.

[0104] The display device 34 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0105] although Figure 1A The source device 12 and the destination device 14 are shown as independent devices, but the device embodiments may also include both the source device 12 and the destination device 14 or the functions of both the source device 12 and the destination device 14, that is, include both the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In these embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0106] According to the description, Figure 1A It will be apparent to the skilled person that the presence and (exact) division of the different units or functionalities in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.

[0107] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30), or both, may be configured to generate a video signal by: Figure 1BThe processing circuitry shown may be implemented, for example, by one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to include reference to Figure 2 The various modules discussed with reference to encoder 20 and / or any other encoder systems or subsystems described herein. The decoder 30 may be implemented with processing circuitry 46 to include the various modules discussed with reference to decoder 30 in FIG. 3 and / or any other decoder systems or subsystems described herein. The processing circuitry 46 may be used to perform the various operations discussed below. Figure 5 As shown, if part of the technology is implemented in software, the device can store the software instructions in a suitable computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology of the present invention. One of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as Figure 1B shown.

[0108] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook computer or laptop, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0109] In some cases, Figure 1AThe video decoding system 10 shown is merely exemplary, and the techniques provided herein may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, and so on. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve and decode data from a memory.

[0110] Figure 1B FIG. 4 is an exemplary block diagram of a video decoding system 40 according to an embodiment of the present application. Figure 1B As shown, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memory storages 44 and / or a display device 45.

[0111] like Figure 1B As shown, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44, and / or display device 45 can communicate with each other. In different embodiments, video decoding system 40 can include only video encoder 20 or only video decoder 30.

[0112] In some instances, antenna 42 can be used to transmit or receive an encoded bitstream of video data. Additionally, in some instances, display device 45 can be used to present the video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory). In a non-limiting example, memory storage 44 can be implemented as cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache memory, etc.) for implementing an image buffer, etc.

[0113] In some examples, video encoder 20 implemented by logic circuitry may include an image buffer (e.g., implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuitry 46 to implement reference Figure 2 and / or any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

[0114] In some examples, the video decoder 30 may be implemented by processing circuitry 46 in a similar manner to implement the various modules discussed with reference to the video decoder 30 of FIG. 3 and / or any other decoder systems or subsystems described herein. In some examples, the logic circuitry-implemented video decoder 30 may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented by processing circuitry 46 to implement the various modules discussed with reference to FIG. 3 and / or any other decoder systems or subsystems described herein.

[0115] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames, indicators, index values, mode selection data, etc., as discussed herein, such as data related to encoding partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and / or data defining encoding partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0116] It should be understood that for the examples described herein with reference to video encoder 20, video decoder 30 can be configured to perform the reverse process. With respect to signaling syntax elements, video decoder 30 can be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, video decoder 30 can parse such syntax elements and decode the associated video data accordingly.

[0117] For ease of description, embodiments of the present invention are described with reference to the Universal Video Coding (VVC) reference software or the High-Efficiency Video Coding (HEVC) developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will appreciate that embodiments of the present invention are not limited to HEVC or VVC.

[0118] Encoders and encoding methods

[0119] Figure 2 FIG. 2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application. Figure 2As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a hybrid video codec-based video encoder.

[0120] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208, and the mode selection unit 260 constitute the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 constitute the backward signal path of the encoder, where the backward signal path of the encoder 20 corresponds to the signal path of the decoder (see decoder 30 in Figure 3). The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer 230, the inter-frame prediction unit 244, and the intra-frame prediction unit 254 also constitute the "internal decoder" of the video encoder 20.

[0121] Images and image segmentation (images and blocks)

[0122] Encoder 20 is operable to receive, via input 201 or the like, an image (or image data) 17, for example, an image from a sequence of images forming a video or video sequence. The received image or image data may also be a pre-processed image (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 may also be referred to as a current image or image to be encoded (particularly when distinguishing the current image from other images in video encoding, such as previously encoded and / or decoded images in the same video sequence, i.e., a video sequence that also includes the current image).

[0123] A (digital) image is, or can be considered to be, a two-dimensional array or matrix of pixels with intensity values. The pixels in the array are also referred to as pixels (or pels, short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the image size and / or resolution. To represent color, three color components are typically used, meaning that an image can be represented as or include three pixel arrays. In the RBG format or color space, an image includes corresponding arrays of red, green, and blue pixels. However, in video coding, each pixel is typically represented in a luma / chroma format or color space, such as YCbCr, which includes a luma component indicated by Y (sometimes also indicated by L) and two chroma components, indicated by Cb and Cr. The luma component Y represents the brightness or grayscale level intensity (for example, in grayscale images, both are the same), while the two chroma components (abbreviated as chroma) Cb and Cr represent the chroma or color information components. Accordingly, an image in YCbCr format includes a luma pixel array of luma pixel values ​​(Y) and two chroma pixel arrays of chroma values ​​(Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format, and vice versa, a process also known as color conversion or transformation. If the image is black and white, the image may include only a luma pixel array. Accordingly, the image may be, for example, a luma pixel array in monochrome format or a luma pixel array and two corresponding chroma pixel arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0124] In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2 ), is used to partition the image 17 into a plurality of (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTBs), or coding tree units (CTUs) in the H.265 / HEVC and VVC standards. The partitioning unit may be used to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to vary the block size between images or subsets or groups of images, and to partition each image into corresponding blocks.

[0125] In other embodiments, the video encoder may be configured to directly receive a block 203 of the image 17, for example, one, several or all blocks constituting the image 17. The image block 203 may also be referred to as a current image block or an image block to be encoded.

[0126] Like image 17, image block 203 is also or can be considered to be a two-dimensional array or matrix of pixels having intensity values ​​(pixel values), but image block 203 is smaller than image 17. In other words, block 203 may include one pixel array (e.g., a luminance array in the case of monochrome image 17, or a luminance array or chrominance array in the case of a color image), or three pixel arrays (e.g., one luminance array and two chrominance arrays in the case of a color image 17), or any other number and / or type of arrays depending on the color format used. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, a block may be an M×N (M columns×N rows) pixel array, or an M×N transform coefficient array, etc.

[0127] In one embodiment, Figure 2 The video encoder 20 is shown for encoding the image 17 block by block, eg, performing encoding and prediction on each block 203 .

[0128] In one embodiment, Figure 2 The illustrated video encoder 20 may also be configured to partition and / or encode an image using slices (also referred to as video slices), where an image may be partitioned or encoded using one or more (typically non-overlapping) slices. Each slice may include one or more blocks (e.g., coding tree units (CTUs)) or one or more groups of blocks (e.g., coding tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0129] In one embodiment, Figure 2 The video encoder 20 shown can also be used to partition and / or encode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), where an image can be partitioned or encoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTUs) or one or more coding blocks, etc., where each coding block can be rectangular or other shaped and may include one or more complete or partial blocks (e.g., CTUs).

[0130] Residual calculation

[0131] The residual calculation unit 204 is used to calculate the residual block 205 (the prediction block 265 is described in detail later) based on the image block 203 and the prediction block 265 in the following manner: for example, the pixel value of the prediction block 265 is subtracted from the pixel value of the image block 203 pixel by pixel (pixel by pixel) to obtain the residual block 205 in the pixel domain.

[0132] Transform

[0133] The transform processing unit 206 is configured to perform a discrete cosine transform (DCT) or a discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be referred to as transform residual coefficients, representing the residual block 205 in the transform domain.

[0134] The transform processing unit 206 may be used to apply an integerized approximation of the DCT / DST, such as the transform specified for H.265 / HEVC. This integerized approximation is typically scaled by a factor compared to the orthogonal DCT transform. In order to maintain the norm of the residual block after the forward and inverse transforms, other scaling factors are used as part of the transform process. The scaling factor is typically selected based on certain constraints, such as whether the scaling factor is a power of 2 for the shift operation, the bit depth of the transform coefficients, a trade-off between accuracy and implementation cost, etc. For example, a specific scaling factor is specified for the inverse transform on the encoder 20 side by the inverse transform processing unit 212 (and for the corresponding inverse transform on the decoder 30 side by, for example, the inverse transform processing unit 312), and correspondingly, a corresponding scaling factor may be specified for the forward transform on the encoder 20 side by the transform processing unit 206.

[0135] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) may be configured to output transform parameters such as one or more transform types, for example, directly output or output after being encoded or compressed by the entropy coding unit 270, such that the video decoder 30 may receive and use the transform parameters for decoding.

[0136] Quantification

[0137] The quantization unit 208 is configured to quantize the transform coefficients 207 by, for example, scalar quantization or vector quantization to obtain quantized transform coefficients 209 . The quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209 .

[0138] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, during quantization, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, varying degrees of scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to finer quantization, while a larger quantization step size corresponds to coarser quantization. The appropriate quantization step size may be indicated by a quantization parameter (QP). For example, the quantization parameter may be an index into a predefined set of appropriate quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), while a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), or vice versa. Quantization may include dividing by the quantization step size, while the corresponding or inverse dequantization performed by the inverse quantization unit 210, etc., may include multiplying by the quantization step size. Embodiments according to some standards, such as HEVC, may be used to determine the quantization step size using the quantization parameter. Generally, the quantization step size may be calculated based on the quantization parameter using a fixed-point approximation of an equation involving division. Other scaling factors can be introduced for quantization and dequantization to restore the norm of the residual block that may have been modified by the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization can be combined. Alternatively, a custom quantization table can be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where larger quantization step sizes result in greater losses.

[0139] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) may be configured to output a quantization parameter (QP), for example, directly or after being encoded or compressed by the entropy coding unit 270, such that the video decoder 30 may receive and use the quantization parameter for decoding.

[0140] Dequantization

[0141] The inverse quantization unit 210 is configured to perform inverse quantization performed by the quantization unit 208 on the quantized coefficients to obtain dequantized coefficients 211, for example, by performing an inverse quantization scheme based on or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 may also be referred to as dequantized coefficients 211 or dequantized residual coefficients 211, which correspond to the transform coefficients 207. However, due to the loss caused by quantization, the dequantized coefficients 211 are generally not identical to the transform coefficients.

[0142] Inverse transform

[0143] The inverse transform processing unit 212 is configured to perform the inverse transform of the transform performed by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain a reconstructed residual block 213 (or corresponding dequantized coefficients 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 213.

[0144] reconstruction

[0145] The reconstruction unit 214 (e.g., the summer 214) is used to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 213 and the pixel point values ​​of the prediction block 265.

[0146] Filtering

[0147] The loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstructed block 215 to obtain a filter block 221, or generally to filter the reconstructed pixels to obtain filtered pixel values. For example, the loop filter unit is used to smoothly perform pixel transitions or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process can be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 2 2. The loop filter unit 220 is shown as a loop filter in FIG. 2, but in other configurations, the loop filter unit 220 can be implemented as a post-loop filter. The filtering block 221 can also be referred to as a filtering and reconstruction block 221.

[0148] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) may be configured to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly or after being entropy-encoded by the entropy coding unit 270, such that the decoder 30 may receive and use the same or different loop filter parameters for decoding.

[0149] Decoded Image Buffer

[0150] The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The decoded picture buffer 230 may be used to store one or more filter blocks 221. The decoded picture buffer 230 may also be used to store other previously filtered blocks, such as previously reconstructed and filtered blocks 221, of the same current picture or a different picture, such as a previously reconstructed block, and may provide a complete previously reconstructed, or decoded, picture (and corresponding reference blocks and pixels) and / or a partially reconstructed current picture (and corresponding reference blocks and pixels), for example, for inter-frame prediction. The decoded image buffer 230 may also be used to store one or more unfiltered reconstructed blocks 215, or generally to store unfiltered reconstructed pixels, for example, reconstructed blocks 215 that have not been filtered by the loop filtering unit 220, or reconstructed blocks or reconstructed pixels that have not undergone any other processing.

[0151] Mode selection (segmentation and prediction)

[0152] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, and is configured to receive or obtain original image data, such as an original block 203 (current block 203 of the current image 17) and reconstructed block data, from the decoded image buffer 230 or other buffer (e.g., a column buffer, not shown), such as filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed block data is used as reference image data for prediction, such as inter-frame prediction or intra-frame prediction, to obtain a prediction block 265 or a prediction value 265.

[0153] The mode selection unit 260 may be used to determine or select a partitioning mode (eg, intra-frame or inter-frame prediction mode) for the current block (including no partitioning), generate a corresponding prediction block 265 , and calculate the residual block 205 and reconstruct the reconstruction block 215 .

[0154] In one embodiment, the mode selection unit 260 may be configured to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260) that provides the best match or minimum residual (minimum residual means better compression during transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression during transmission or storage), or simultaneously considers or balances both. The mode selection unit 260 may be configured to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., select the prediction mode that provides minimum rate distortion optimization. Terms such as "best," "lowest," and "optimal" herein do not necessarily refer to "best," "lowest," or "optimal" overall, but may also refer to situations where termination or selection criteria are met, e.g., values ​​exceeding or falling below a threshold or other limit may result in a "suboptimal selection" but reduce complexity and processing time.

[0155] In other words, the partitioning unit 262 may be configured to partition an image in a video sequence into a sequence of coding tree units (CTUs), the CTU 203 being further partitioned into smaller block portions or sub-blocks (again forming blocks), e.g., by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT), or triple-tree partitioning (TT), or any combination thereof, and for, e.g., performing prediction on each of the block portions or sub-blocks, wherein the mode selection comprises selecting a tree structure for partitioning the block 203 and selecting a prediction mode to be applied to each of the block portions or sub-blocks.

[0156] The segmentation (eg, performed by segmentation unit 262) and prediction processes (eg, performed by inter-prediction unit 244 and intra-prediction unit 254) performed by video encoder 20 are described in detail below.

[0157] segmentation

[0158] The partitioning unit 262 may partition (or divide) a coding tree unit 203 into smaller parts, such as small blocks in a square or rectangular shape. For an image with three pixel arrays, a CTU consists of an N×N luma pixel block and two corresponding chroma pixel blocks.

[0159] The H.265 / HEVC video coding standard divides a frame into non-overlapping CTUs, with the CTU size set to 64×64 (the CTU size can also be set to other values, such as 128×128 or 256×256 in the JVET reference software JEM). A 64×64 CTU consists of a rectangular pixel matrix with 64 columns and 64 pixels per column, and each pixel contains a luminance component and / or a chrominance component.

[0160] H.265 uses a QT-based CTU partitioning method, taking CTU as the root node of QT. According to the QT partitioning method, CTU is recursively divided into several leaf nodes, such as Figure 3a As shown. A node corresponds to an image area. If the node is not divided, the node is called a leaf node, and the image area corresponding to the node is a CU; if the node continues to be divided, the image area corresponding to the node can be divided into four areas of the same size (their length and width are half of the divided area respectively). Each area corresponds to a node, and it is necessary to determine separately whether these nodes will be divided again. Whether a node is divided is indicated by the split flag split_cu_flag corresponding to the node in the bitstream. A node A is divided once to obtain 4 nodes Bi, i=0~3, Bi is called the child node of A, and A is called the parent node of Bi. The QT level (qtDepth) of the root node is 0, and the QT level of the node is the four QT levels of the parent node of the node plus 1. For the sake of simplicity, the size and shape of the node below refer to the size and shape of the image area corresponding to the node.

[0161] For example, a 64×64 CTU (whose QT level is 0) becomes a 64×64 CU if it is determined not to be divided according to its corresponding split_cu_flag; if it is determined to be divided into four 32×32 nodes (QT level is 1). Each of the four 32×32 nodes can be determined to continue to be divided or not according to its corresponding split_cu_flag. If one of the 32×32 nodes is further divided, four 16×16 nodes are generated (QT level is 2). And so on, until all nodes are no longer divided, such a CTU is divided into a group of CUs. The minimum size of the CU is indicated in the sequence parameter set (SPS), for example, 8×8 is the minimum size of the CU. In the above recursive partitioning process, if the size of a node is equal to the minimum CU size, this node is defaulted to no longer being divided, and its partition flag does not need to be included in the bitstream.

[0162] When a node is parsed as a leaf node, the leaf node is a CU, and the coding information corresponding to the CU is further parsed (including the prediction mode, transform coefficients and other information of the CU, for example, the coding_unit() syntax structure in H.265), and then the CU is predicted, dequantized, inversely transformed, loop filtered and other decoding processes are performed according to these coding information to obtain the reconstructed block corresponding to the CU. QT partitioning enables CTU to be divided into a group of CUs of appropriate size according to the local characteristics of the image. For example, smooth areas are divided into larger CUs, while texture-rich areas are divided into smaller CUs.

[0163] The H.266 / VVC standard adds BT division and TT division based on QT division.

[0164] The BT partitioning method divides one node into two sub-nodes. There are two specific BT partitioning methods:

[0165] 1) Horizontal bisection: Divide the area corresponding to the node into two equal-sized areas, the upper and lower areas, with the width unchanged and the height reduced to half of the area before division. Each area after division corresponds to a child node. For example, Figure 3b shown.

[0166] 2) Vertical bisection: Divide the area corresponding to the node into two equal-sized areas on the left and right, with the height remaining unchanged and the width becoming half of the area before division. Each area after division corresponds to a child node. For example, Figure 3c shown.

[0167] The TT partitioning method divides one node into three sub-nodes. There are two specific TT partitioning methods:

[0168] 1) Horizontal Trisection: Divide the area corresponding to the node into upper, middle and lower areas. The heights of the upper, middle and lower areas are 1 / 4, 1 / 2 and 1 / 4 of the area before division respectively. Each divided area corresponds to a child node. For example, Figure 3d shown.

[0169] 2) Vertical trisection: Divide the area corresponding to the node into three areas: left, middle, and right. The widths of the left, middle, and right areas are 1 / 4, 1 / 2, and 1 / 4 of the area before division, respectively. Each area after division corresponds to a child node. For example, Figure 3e shown.

[0170] H.266 uses the QT cascade BT / TT division method, referred to as the QT-MTT division method. That is, the CTU is divided through QT to generate 4 QT nodes. The QT node can be further divided into 4 QT nodes using the QT division method until the QT node is no longer divided using the QT division method and becomes a QT leaf node, or the QT node is not divided using the QT division method and becomes a QT leaf node. The QT leaf node serves as the root node of the MTT. The nodes in the MTT can be divided into child nodes using one of the four division methods: horizontal bisection, vertical bisection, horizontal trisection, and vertical trisection, or no longer divided and become one MTT leaf node. An MTT leaf node is a CU.

[0171] For example, Figure 4 This is an exemplary schematic diagram of the QT-MTT division method of this application, as shown in FIG. Figure 4 As shown in the figure, QT-MTT is used to divide a CTU into 16 CUs a to p. Each endpoint in the tree diagram represents a node. Four lines from one node represent QT division, two lines from one node represent BT division, and three lines from one node represent TT division. The solid line represents QT division, the dotted line represents the first-level division of MTT, and the dot-dash line represents the second-level division of MTT. a to p are 16 MTT leaf nodes, and each MTT leaf node corresponds to a CU. Based on the above tree diagram, we can get Figure 4 Partition diagram of CTU in .

[0172] In the QT-MTT partitioning method, each CU has a QT level (or QT depth) (quad-tree depth, QT depth) and an MTT level (MTT depth) (multi-type tree depth, MTT depth). The QT level represents the QT level of the QT leaf node to which the CU belongs, and the MTT level represents the MTT level of the MTT leaf node to which the CU belongs. For example, Figure 4 The QT level of a, b, c, d, e, f, g, i, and j is 1, and the MTT level is 2; the QT level of h is 1, and the MTT level is 1; the QT level of n, o, and p is 2, and the MTT level is 0; the QT level of l and m is 2, and the MTT level is 1. If a CTU is divided into only one CU, the QT level and MTT level of this CU are 0.

[0173] The AVS3 standard adopts another QT-MTT partitioning method, namely the QT cascade BT / EQT partitioning method, that is, the AVS3 standard replaces the TT partitioning in H.266 with extended quad-tree (EQT) partitioning. That is, the CTU is divided by QT to generate 4 QT nodes. The QT node can be further divided into 4 QT nodes using the QT partitioning method until the QT node is no longer divided using the QT partitioning method and becomes a QT leaf node, or the QT node is not divided using the QT partitioning method and becomes a QT leaf node. The QT leaf node serves as the root node of the MTT. The nodes in the MTT can be divided into child nodes using one of the four partitioning methods: horizontal binary division, vertical binary division, horizontal quartering, and vertical quartering, or no longer divided and become 1 MTT leaf node. The MTT leaf node is a CU.

[0174] The EQT partitioning method divides one node into four sub-nodes. There are two specific EQT partitioning methods:

[0175] 1) Horizontal Quartering: Divide the area corresponding to the node into four areas: upper, middle left, middle right, and lower. The heights of the upper, middle left, middle right, and lower areas are 1 / 4, 1 / 2, 1 / 2, and 1 / 4 of the area before division, respectively. The widths of the middle left and middle right areas are 1 / 2 of the area before division, respectively. Each divided area corresponds to a child node. For example, Figure 3f shown.

[0176] 2) Vertical Quartering: Divide the area corresponding to the node into four areas: left, upper middle, lower middle, and right. The widths of the left, upper middle, lower middle, and right areas are 1 / 4, 1 / 2, 1 / 2, and 1 / 4 of the area before the division, respectively. The heights of the upper middle and lower middle areas are 1 / 2 of the area before the division, respectively. Each divided area corresponds to a child node. For example, Figure 3g shown.

[0177] In the H.265 / HEVC standard, for YUV 4:2:0 images, a CTU consists of one luma block and two chroma blocks. Luma and chroma blocks can be divided in the same way, known as a joint luma-chroma coding tree. In VVC, if the current frame is an I-frame, and a CTU is a node of a preset size (e.g., 64×64) in an intra-coded frame (I-frame), the luma block contained in that node is divided into a set of coding units (CUs) consisting solely of luma blocks using the luma coding tree, while the chroma blocks contained in that node are divided into a set of coding units (CUs) consisting solely of chroma blocks using the chroma coding tree. The division of the luma and chroma coding trees is independent of each other. These separate coding trees for luma and chroma blocks are known as separate trees. In H.265, a CU contains both luma and chroma pixels. In standards like H.266 and AVS3, in addition to CUs containing both luma and chroma pixels, there are also luma and chroma CUs containing only luma pixels and only chroma pixels.

[0178] As described above, the video encoder 20 is configured to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, intra-frame prediction modes and / or inter-frame prediction modes.

[0179] Intra-frame prediction

[0180] The intra prediction mode set may include 35 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in HEVC, or may include 67 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and planar mode, or directional modes as defined in VVC. For example, several traditional angle intra prediction modes are adaptively replaced by wide-angle intra prediction modes for non-square blocks defined in VVC. For another example, in order to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of the non-square block. In addition, the intra prediction result of the planar mode can also be modified using the position-dependent intra prediction combination (PDPC) method.

[0181] The intra prediction unit 254 is configured to generate an intra prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to an intra prediction mode in the intra prediction mode set.

[0182] The intra-frame prediction unit 254 (or generally the mode selection unit 260) is also used to output intra-frame prediction parameters (or generally information indicating the selected intra-frame prediction mode for the block) in the form of syntax elements 266 to the entropy coding unit 270 for inclusion in the encoded image data 21, so that the video decoder 30 can perform operations, such as receiving and using the prediction parameters for decoding.

[0183] Inter-frame prediction

[0184] In a possible implementation, the set of inter-frame prediction modes depends on the available reference image (i.e., at least part of the previously decoded image stored in DBP230 as mentioned above) and other inter-frame prediction parameters, for example, on whether the entire reference image is used or only a part of the reference image is used, such as a search window area around the area of ​​the current block, to search for the best matching reference block, and / or on whether pixel interpolation such as half-pixel, quarter-pixel and / or 1 / 16 interpolation is performed, for example.

[0185] In addition to the above prediction modes, skip mode and / or direct mode may also be employed.

[0186] For example, extended merge prediction, the merge candidate list of this mode consists of the following five candidate types in order: spatial MVP from spatially adjacent CUs, temporal MVP from collocated CUs, history-based MVP from the FIFO table, pairwise average MVP, and zero MV. Decoder side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the MV of the merge mode. Merge mode with MVD (MMVD) is derived from the merge mode with motion vector difference. The MMVD flag is sent immediately after the skip flag and merge flag are sent to specify whether the CU uses the MMVD mode. The CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the MVD of the CU with different precisions. The MVD of the current CU is adaptively selected according to the prediction mode of the current CU. When the CU is encoded in merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted averaging the inter-frame and intra-frame prediction signals. For affine motion compensation prediction, the affine motion field of the block is described by the motion information of the motion vector of 2 control points (4 parameters) or 3 control points (6 parameters). Subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but it predicts the motion vector of the sub-CU within the current CU. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces calculations, especially in terms of the number of multiplications and the size of the multipliers. In the triangle partitioning mode, the CU is evenly divided into two triangular parts using diagonal partitioning and anti-diagonal partitioning. In addition, the bidirectional prediction mode is extended based on the simple averaging to support weighted averaging of the two prediction signals.

[0187] The inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in Figure 2(not shown in the figure). The motion estimation unit may be configured to receive or obtain the image block 203 (the current image block 203 of the current image 17) and the decoded image 231, or at least one or more previously reconstructed blocks, e.g., reconstructed blocks of one or more other / different previously decoded images 231, for motion estimation. For example, a video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form a sequence of images forming the video sequence.

[0188] For example, the encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different images in a plurality of other images, and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. This offset is also referred to as a motion vector (MV).

[0189] The motion compensation unit is configured to obtain, for example, receive, inter-frame prediction parameters and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain an inter-frame prediction block 246. The motion compensation performed by the motion compensation unit may include extracting or generating a prediction block based on a motion / block vector determined by motion estimation, and may also include performing interpolation with sub-pixel accuracy. Interpolation filtering can generate pixel points of other pixels from pixel points of known pixels, thereby potentially increasing the number of candidate prediction blocks that can be used to encode the image block. Upon receiving a motion vector corresponding to a PU of the current image block, the motion compensation unit may locate the prediction block pointed to by the motion vector in one of the reference picture lists.

[0190] The motion compensation unit may also generate syntax elements associated with blocks and video slices for use by video decoder 30 when decoding image blocks of a video slice. In addition to, or in lieu of, slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be generated or used.

[0191] Entropy Coding

[0192] The entropy coding unit 270 is configured to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC (CALVC) scheme, an arithmetic coding scheme, a binarization algorithm, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements, resulting in coded image data 21 that can be output via an output terminal 272 in the form of a coded bitstream 21, etc., so that the video decoder 30, etc. can receive and use the parameters for decoding. The coded bitstream 21 can be transmitted to the video decoder 30 or stored in a memory for later transmission or retrieval by the video decoder 30.

[0193] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another implementation, the encoder 20 may have the quantization unit 208 and the inverse quantization unit 210 combined into a single unit.

[0194] Decoder and decoding method

[0195] Figure 5 3 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application. The video decoder 30 is configured to receive coded image data 21 (e.g., coded bitstream 21), for example, encoded by the encoder 20, and generate a decoded image 331. The coded image data or bitstream includes information for decoding the coded image data, such as data representing image blocks of a coded video slice (and / or coding block group or coding block) and related syntax elements.

[0196] exist Figure 5In the example of FIG. 3 , decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. Inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same motion compensation as reference video. Figure 2 The decoding process of the video encoder 100 is the opposite of the encoding process described.

[0197] As described with respect to encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer DPB 230, inter-prediction unit 344, and intra-prediction unit 354 also constitute the "built-in decoder" of video encoder 20. Accordingly, inverse quantization unit 310 may be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 may be functionally identical to inverse transform processing unit 122, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoded picture buffer 330 may be functionally identical to decoded picture buffer 230. Therefore, the explanations of the corresponding units and functions of video encoder 20 apply accordingly to the corresponding units and functions of video decoder 30.

[0198] Entropy decoding

[0199] The entropy decoding unit 304 is used to parse the bit stream 21 (or generally the coded image data 21) and perform entropy decoding on the coded image data 21 to obtain the quantization coefficients 309 and / or the decoded coding parameters ( Figure 5 , such as any or all of inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 may be used to apply a decoding algorithm or scheme corresponding to the coding scheme of the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 may also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice and / or video block level. In addition, or in lieu of slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be received or used.

[0200] Dequantization

[0201] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or generally information related to inverse quantization) and quantization coefficients from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and inverse quantize the decoded quantization coefficients 309 based on the quantization parameter to obtain inverse quantization coefficients 311 or dequantized coefficients 311. The inverse quantization process may include using the quantization parameter calculated by the video encoder 20 for each video block in the video slice to determine a degree of quantization, and thus a degree of inverse quantization to be performed.

[0202] Inverse transform

[0203] The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311 and apply a transform to the dequantized coefficients 311 to obtain a reconstructed residual block 313 in the pixel domain. The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may also be configured to receive transform parameters or corresponding information from the coded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform to be applied to the dequantized coefficients 311.

[0204] reconstruction

[0205] The reconstruction unit 314 (eg, summer 314 ) is configured to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the pixel domain, eg, by adding the pixel values ​​of the reconstructed residual block 313 and the pixel values ​​of the prediction block 365 .

[0206] Filtering

[0207] The loop filter unit 320 (in or after the encoding loop) is used to filter the reconstructed block 315 to obtain a filter block 321, thereby smoothly performing pixel conversion or improving video quality. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process can be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process can also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Although the loop filter unit 320 is shown as a loop filter in Figure 3, in other configurations, the loop filter unit 320 can be implemented as a post-loop filter.

[0208] Decoded Image Buffer

[0209] The decoded video blocks 321 of one picture are then stored in a decoded picture buffer 330, which stores the decoded picture 331 as a reference picture for subsequent motion compensation of other pictures and / or for respective output displays.

[0210] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., for display to the user or for the user to view.

[0211] predict

[0212] The inter-frame prediction unit 344 may be functionally identical to the inter-frame prediction unit 244 (particularly the motion compensation unit), and the intra-frame prediction unit 354 may be functionally identical to the inter-frame prediction unit 254 and may determine the partitioning or segmentation and perform prediction based on the segmentation and / or prediction parameters or corresponding information received from the coded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304). The mode application unit 360 may be configured to perform prediction (intra-frame or inter-frame prediction) for each block based on the reconstructed block, block, or corresponding pixel point (filtered or unfiltered), resulting in a prediction block 365.

[0213] When the video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 354 in the mode application unit 360 is configured to generate a prediction block 365 for the image block of the current video slice based on the indicated intra-prediction mode and data from previously decoded blocks of the current image. When the video image is encoded as an inter-coded (i.e., B or P) slice, the inter-prediction unit 344 (e.g., a motion compensation unit) in the mode application unit 360 is configured to generate a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. For inter-prediction, these prediction blocks may be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct reference frame list 0 and list 1 using a default construction technique based on the reference pictures stored in DPB 330. The same or similar processes may be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks) in addition to or instead of slices (e.g., video slices), e.g., video may be encoded using I, P, or B coding block groups and / or coding blocks.

[0214] Mode application unit 360 is configured to determine prediction information for video blocks of a current video slice by parsing motion vectors and other syntax elements, and to use the prediction information to generate a prediction block for the current video block being decoded. For example, mode application unit 360 uses received syntax elements to determine the prediction mode (e.g., intra prediction or inter prediction) used to encode the video blocks of the video slice, the inter prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more reference picture lists for the slice, motion vectors for each inter-coded video block in the slice, inter prediction status for each inter-coded video block in the slice, and other information to decode the video blocks within the current video slice. In addition to or in lieu of slices (e.g., video slices), the same or similar processes may be applied to embodiments involving coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, where video may be encoded using I, P, or B coding block groups and / or coding blocks.

[0215] In one embodiment, Figure 5 The illustrated video encoder 30 may also be configured to partition and / or decode an image using slices (also referred to as video slices), where an image may be partitioned or decoded using one or more (typically non-overlapping) slices. Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coding blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0216] In one embodiment, Figure 5The video decoder 30 shown can also be used to segment and / or decode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), where an image can be segmented or decoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTUs) or one or more coding blocks, etc., where each coding block can be rectangular or other shaped and may include one or more complete or partial blocks (e.g., CTUs).

[0217] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform-based decoder 30 may directly inverse quantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 may have the inverse quantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0218] It should be understood that the processing result of the current step can be further processed in the encoder 20 and the decoder 30 and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, the processing result of interpolation filtering, motion vector derivation, or loop filtering can be further operated on, such as clipping or shifting operations.

[0219] It should be noted that further operations can be performed on the derived motion vector of the current block (including but not limited to the control point motion vector of the affine mode, the sub-block motion vector of the affine, planar, ATMVP mode, the temporal motion vector, etc.). For example, the value of the motion vector is limited to a predefined range based on the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents a power. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of four 4×4 sub-blocks in an 8×8 block) is limited so that the maximum difference between the integer parts of the four 4×4 sub-block MVs does not exceed N pixels, for example, not more than 1 pixel. Two methods of limiting the motion vector based on bitDepth are provided here.

[0220] Although the above embodiments primarily describe video coding, it should be noted that embodiments of the decoding system 10, encoder 20, and decoder 30, as well as other embodiments described herein, may also be used for still image processing or coding, i.e., processing or coding a single image in a video codec that is independent of any previous or subsequent images. In general, if image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and the inter-frame prediction unit 344 (decoder) may not be available. All other functionalities (also referred to as tools or techniques) of the video encoder 20 and video decoder 30 are also applicable to still image processing, such as residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354, and / or loop filtering 220 / 320, entropy coding 270, and entropy decoding 304.

[0221] Figure 6 FIG4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, for example, Figure 1A The video decoder 30 in , may also be an encoder, such as Figure 1A The video encoder 20 in .

[0222] Video decoding device 400 includes: an input port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. Video decoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to input port 410, receiver unit 420, transmitter unit 440, and output port 450 for output or output of optical or electrical signals.

[0223] Processor 430 is implemented using hardware and software. Processor 430 can be implemented as one or more processor chips, cores (e.g., a multi-core processor), FPGAs, ASICs, and DSPs. Processor 430 communicates with input port 410, receiving unit 420, transmitting unit 440, output port 450, and memory 460. Processor 430 includes a decoding module 470. Decoding module 470 implements the embodiments disclosed above. For example, decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, decoding module 470 provides substantial improvements to the functionality of video decoding device 400 and affects the switching of video decoding device 400 to different states. Alternatively, decoding module 470 can be implemented using instructions stored in memory 460 and executed by processor 430.

[0224] Memory 460 includes one or more disks, tape drives, and solid-state drives and can be used as overflow data storage for storing programs when such programs are selected for execution, and for storing instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0225] Figure 7 This is an exemplary block diagram of an apparatus 500 according to an embodiment of the present application. The apparatus 500 may be used as Figure 1A Either or both of the source device 12 and the destination device 14 in .

[0226] The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices, now available or developed in the future, capable of manipulating or processing information. While the disclosed implementations may be implemented using a single processor, such as the processor 502 shown, using more than one processor may provide greater speed and efficiency.

[0227] In one implementation, the memory 504 in the apparatus 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that are accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and application programs 510, which include at least one program that allows the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N, as well as a video decoding application that performs the methods described herein.

[0228] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with touch-sensitive elements that can be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0229] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. Furthermore, secondary storage may be directly coupled to other components of device 500 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 may have a variety of configurations.

[0230] Figure 8 This is an exemplary flowchart of the decoding method according to an embodiment of the present application. Process 800 may be performed by the video encoder 20 (or encoder) and the video decoder 30 (or decoder). Process 800 is described as a series of steps or operations. It should be understood that process 800 may be performed in various orders and / or occur simultaneously, not limited to Figure 8 The execution order shown. Figure 8 In the embodiment of the present invention, the first enhancement layer corresponds to the residual pixel layer in the enhancement layer, and the second enhancement layer corresponds to the transform domain coefficient layer in the enhancement layer. The residual pixel layer may also be referred to as the residual pixel layer; the transform domain coefficient layer may also be referred to as the transform domain coefficient layer.

[0231] Step 801: The encoder obtains a reconstructed block of a base layer of an image block to be encoded.

[0232] Scalable video coding, also known as scalable video coding, is an extension of current video coding standards (typically scalable video coding (SVC), an extension of advanced video coding (AVC) (H.264), or scalable high-efficiency video coding (SHVC), an extension of high-efficiency video coding (HEVC) (H.265). Scalable video coding was developed primarily to address packet loss and latency issues associated with real-time video transmission due to changes in network bandwidth.

[0233] The basic structure in scalable video coding can be called a layer. Scalable video coding technology can obtain hierarchical code streams of different resolutions by performing spatial grading (resolution grading) on ​​the original image blocks. Resolution can refer to the size of the image block in pixels, with the lower layer having a lower resolution and the higher layer having a resolution no lower than that of the lower layer; or, by performing temporal grading (frame rate grading) on ​​the original image blocks, hierarchical code streams of different frame rates can be obtained. Frame rate can refer to the number of image frames contained in a video per unit time, with the lower layer having a lower frame rate and the higher layer having a frame rate no lower than that of the lower layer; or, by performing quality domain grading on the original image blocks, hierarchical code streams of different coding qualities can be obtained. Coding quality can refer to the quality of the video, with the lower layer having a higher degree of image distortion and the higher layer having a degree of image distortion no higher than that of the lower layer.

[0234] Generally, the layer called the base layer is the lowest layer in scalable video coding. In spatial domain scalability, the base layer image blocks are encoded using the lowest resolution; in temporal domain scalability, the base layer image blocks are encoded using the lowest frame rate; in quality domain scalability, the base layer image blocks are encoded using the highest QP or the lowest bit rate. That is, the base layer is the lowest quality layer in scalable video coding. The layer called the enhancement layer is the layer above the base layer in scalable video coding, and can be divided into multiple enhancement layers from low to high. The lowest enhancement layer has a higher encoding resolution than the base layer based on the encoding information obtained from the base layer, or a higher frame rate than the base layer, or a higher bit rate than the base layer. Higher-level enhancement layers can encode higher-quality image blocks based on the encoding information of lower-level enhancement layers.

[0235] For example, Figure 9 This is an exemplary hierarchical diagram of scalable video coding in this application, as shown in Figure 9As shown, after the original image block is sent to the scalable encoder, it can be layered into base layer image blocks B and enhancement layer image blocks (E1~En, n≥1) according to different coding configurations, and then encoded separately to obtain a code stream containing the base layer code stream and the enhancement layer code stream. The base layer code stream is generally the code stream obtained by encoding the lowest spatial domain image block, the lowest temporal domain image block or the lowest quality image block. The enhancement layer code stream is based on the base layer and is superimposed with the encoding of high-level spatial domain, high-level temporal domain or high-level quality image blocks. As the number of enhancement layers increases, the spatial domain level, temporal domain level or quality level of the code will also become higher and higher. When the encoder transmits the code stream to the decoder, it gives priority to ensuring the transmission of the base layer code stream. When the network has a margin, it gradually transmits higher and higher level code streams. The decoder first receives and decodes the base layer code stream, and then decodes the code streams with increasingly higher spatial, temporal or quality levels layer by layer according to the received enhancement layer code stream, in order from low to high levels. The higher-level decoding information is then superimposed on the reconstructed blocks of the lower levels to obtain reconstructed blocks with higher resolution, higher frame rate or higher quality.

[0236] The image block to be encoded may refer to the image block currently being processed by the encoder, and the image block may refer to the largest coding unit (LCU) in the entire frame image. The entire frame image may refer to any image frame in the image sequence contained in the video being processed by the encoder. The image frame has not been divided and its size is the size of a complete image frame. In the H.265 standard, the original image frame is divided into multiple coding tree units (CTUs) before video encoding. The CTU is the largest coding unit for video encoding and can be divided into CUs of different sizes in a quadtree manner. As the largest coding unit, the CTU is also called an LCU; alternatively, the image block may also refer to the entire frame image; alternatively, the image block may also refer to a region of interest (ROI) in the entire frame image, that is, a specified image area in the image that needs to be processed.

[0237] As mentioned above, each image in a video sequence is usually divided into a set of non-overlapping blocks, and is usually encoded at the block level. In other words, the encoder usually processes, i.e. encodes, the video at the block (image block) level, for example, by generating a prediction block through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction block from the image block (currently processed / to-be-processed block) to obtain a residual block; transforming the residual block in the transform domain and dequantizing the coefficients can reduce the amount of data to be transmitted (compressed). The encoder also needs to obtain a reconstructed residual block through inverse quantization and inverse transformation, and then add the pixel values ​​of the reconstructed residual block and the pixel values ​​of the prediction block to obtain a reconstructed block. The reconstructed block of the base layer refers to the reconstructed block obtained by performing the above operations on the base layer image block obtained by layering the original image block. For example, Figure 10 is an exemplary flowchart of a coding method for a base layer, as shown in FIG. Figure 10 As shown, the encoder obtains a prediction block for the base layer based on the original image block (e.g., LCU). It then calculates the difference between the corresponding pixels in the original image block and the prediction block of the base layer to obtain a residual block for the base layer. The residual block of the base layer is then divided, transformed, and quantized. It is then entropy-encoded along with the base layer coding control information, prediction information, motion information, etc. to obtain a bitstream for the base layer. The encoder then dequantizes and inversely transforms the quantized coefficients to obtain a reconstructed residual block for the base layer. The encoder then sums the corresponding pixels in the prediction block and the reconstructed residual block of the base layer to obtain a reconstructed block for the base layer.

[0238] Step 802: The encoder calculates the difference between the corresponding pixels in the image block to be encoded and the reconstructed block of the base layer to obtain a residual block of the first enhancement layer of the image block to be encoded.

[0239] The resolution of the residual pixel layer image blocks is not lower than the resolution of the base layer image blocks, or the encoding quality of the residual pixel layer image blocks is not lower than the encoding quality of the base layer image blocks. As described above, the encoding quality may refer to the quality of the video, and the encoding quality of the residual pixel layer image blocks being not lower than the encoding quality of the base layer image blocks may mean that the image distortion of the base layer is greater, while the image distortion of the residual pixel layer is not higher than that of the base layer.

[0240] An image block includes multiple pixels arranged in an array, and each pixel's position in the image block is uniquely identified by its row and column number. Assume that the size of the image block to be coded and the reconstructed block of the base layer are both M×N, that is, the image block to be coded and the reconstructed block of the base layer each contain M×N pixels. a(i1, j1) represents the pixel located in the i1th column and j1th row of the image block to be coded, with i1 = 1 to M and j1 = 1 to N. b(i2, j2) represents the pixel located in the i1th column and j1th row of the reconstructed block of the base layer, with i2 = 1 to M and j2 = 1 to N. Corresponding pixels in the image block to be coded and the reconstructed block of the base layer mean that the row and column numbers of the pixels in their respective image blocks are equal, that is, i1 = i2 and j1 = j2. For example, the size of an image block is 16×16, which contains 16×16 pixels, with row numbers 0 to 15 and column numbers 0 to 15. The pixel identified as a(0,0) in the image block to be encoded and the pixel identified as b(0,0) in the reconstructed block of the base layer are corresponding pixels, or the pixel identified as a(6,9) in the image block to be encoded and the pixel identified as b(6,9) in the reconstructed block of the base layer are corresponding pixels, etc. The difference between the corresponding pixels can be the difference between the pixel values ​​of the corresponding pixels in the image block to be encoded and the reconstructed block of the base layer. The pixel value can be the luminance value, chrominance value, etc. of the pixel, which is not specifically limited in this application.

[0241] In the SHVC standard, the encoder predicts the residual pixel layer based on the reconstructed block of the base layer, and then takes the difference between the corresponding pixels in the reconstructed block of the base layer and the predicted block of the residual pixel layer to obtain the residual block of the residual pixel layer. However, in step 802, the residual block of the residual pixel layer can be directly obtained by taking the difference between the corresponding pixels in the image block to be encoded and the reconstructed block of the base layer. This simplifies the process of obtaining the predicted block of the residual pixel layer, reduces the encoder's processing flow, and improves the encoder's coding efficiency.

[0242] Step 803: The encoder obtains a transform block of the first enhancement layer according to the residual block of the first enhancement layer, and transforms the transform block of the first enhancement layer to obtain transform coefficients of the first enhancement layer.

[0243] The transform block division method of the residual block of the residual pixel layer is different from the transform block division method of the residual block of the base layer. That is, when the encoder processes the residual block of the residual pixel layer, the transform block division method adopted is different from the transform block division method adopted when processing the residual block of the base layer. For example, the residual block of the base layer is divided into three sub-blocks using the TT division method, but the residual block of the residual pixel layer is not divided using the TT division method. The encoder can use the following adaptive method to determine the transform block division method of the residual block of the residual pixel layer:

[0244] In one possible implementation, the encoder may first perform an iterative tree structure division on the first largest transform unit (LTU) of the residual block of the residual pixel layer to obtain transform units (TU) of multiple segmentation depths, where the maximum segmentation depth among the multiple segmentation depths is equal to Dmax, where Dmax is a positive integer. The division method of the first TU is then determined based on the loss estimate of the first TU and the sum of the loss estimates of the multiple second TUs contained in the first TU. The segmentation depth of the first TU is i, the depth of the second TU is i+1, and 0≤i≤Dmax-1.

[0245] Corresponding to the size of the image block to be encoded, the size of the residual block of the residual pixel layer can be the size of the entire frame image, or the size of the image block (such as CTU or CU) divided in the entire frame image, or the size of the ROI in the entire frame image. LTU is the largest image block in the image block used for transform processing. The size of LTU can be the same as the size of the reconstruction block of the base layer. This can maximize the efficiency of transform coding while ensuring that the residual pixel layer between different reconstruction blocks can be processed in parallel. An LTU can be divided into multiple nodes based on the configured partitioning information, and each node can be further divided based on the configured partitioning information until all nodes are no longer divided. This process can be called iterative tree structure partitioning. The tree structure partitioning can include QT partitioning, BT partitioning and / or TT partitioning, and can also include EQT partitioning. This application does not specifically limit the partitioning method of LTU. A node is divided once to obtain multiple nodes. The divided node is called the parent node, and the node obtained by the partition is called the child node. The split depth (depth) of the root node is 0, and the depth of the child node is the depth of the parent node plus 1. Therefore, the residual block of the residual pixel layer is iteratively divided into a tree structure starting from the root node LTU, and multiple depth TUs can be obtained, such as Figure 4 As shown. The maximum depth among the multiple depths (i.e. the depth of the smallest TU obtained by division) is Dmax. Assuming that the width and height of the LTU are both L and the width and height of the smallest TU are both S, the division depth of the smallest TU is in It means round down, so Dmax≤d.

[0246] Based on the TUs divided by the above transformation, the encoder performs rate distortion optimization (RDO) processing to determine the division method of the LTU. Taking the first TU with a division depth of i and the second TU with a depth of i+1 as an example, i starts from Dmax-1 and decreases in value. The second TU is obtained by dividing the first TU. The encoder first transforms and quantizes the first TU to obtain the quantization coefficient of the first TU (the quantization coefficient can be as follows Figure 10 ), then pre-encode the quantized coefficients of the first TU (pre-encoding is a process of encoding to estimate the length of the encoded codeword, or a process of processing in a manner similar to encoding) to obtain the code stream size R of the first TU, then inverse quantize and inverse transform the quantized coefficients of the first TU to obtain a reconstructed block of the TU, calculate the sum of squares of errors between the first TU and the reconstructed block of the first TU to obtain a distortion value D of the first TU, and finally obtain a loss estimate C of the first TU based on the code stream size R of the first TU and the distortion value D of the first TU.

[0247] The calculation formula of the distortion value D of the first TU is as follows:

[0248]

[0249] Among them, P rs (i, j) represents the original value of the residual pixel in the first TU at the coordinate point (i, j) within the TU range, P rc (i, j) represents the reconstructed value of the residual pixel in the first TU at the coordinate point (i, j) within the TU range.

[0250] The loss estimate C of the first TU is calculated as follows:

[0251] C=D+λR

[0252] Here, λ represents a constant value related to the quantization parameter of the current layer, which determines the pixel distortion of the current layer.

[0253] The encoder can use the same method described above to calculate the loss estimate for the second TU. After obtaining the loss estimate for the first TU and the loss estimate for the second TU, the encoder compares the loss estimate for the first TU with the sum of the loss estimates for multiple second TUs. After dividing the first TU into multiple second TUs, the encoder determines the division method corresponding to the smaller of the two as the division method for the first TU. That is, if the loss estimate for the first TU is greater than the sum of the loss estimates for the multiple second TUs, the division method for the first TU into multiple second TUs is determined as the division method for the first TU. If the loss estimate for the first TU is less than or equal to the sum of the loss estimates for the multiple second TUs, the first TU is no longer divided. After traversing all TUs of the LTU using the above method, the encoder obtains the division method for the LTU. When the residual block of the residual pixel layer contains only one LTU, the division method for the LTU is the division method for the residual block of the residual pixel layer. When the residual block of the residual pixel layer contains multiple LTUs, the division method for the residual block of the residual pixel layer and the division method for the residual block of the residual pixel layer into LTUs constitute the division method for the residual block of the residual pixel layer.

[0254] The present application no longer uses the same TU division method for the residual blocks of the residual pixel layer as that of the residual blocks of the base layer. Instead, an adaptive TU division method is adopted to obtain a TU division method suitable for the residual blocks of the residual pixel layer before encoding. When the TU division of the residual blocks of the residual pixel layer is independent of the CU division method of the base layer, the TU size of the residual pixel layer is no longer limited by the CU size, which can improve the flexibility of encoding.

[0255] Step 804: The encoder quantizes and dequantizes the transform coefficients of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer.

[0256] Step 805: The encoder calculates the difference between the transform coefficients of the first enhancement layer and the dequantized coefficients of the first enhancement layer to obtain a coefficient residual block of the second enhancement layer. In this embodiment, the second enhancement layer is a transform domain coefficient layer.

[0257] In a possible implementation, a block partitioning method of the coefficient residual block of the second enhancement layer is the same as a transform block partitioning method of the residual block of the first enhancement layer.

[0258] Step 806: The encoder quantizes and entropy codes the transform coefficients of the first enhancement layer to obtain a bitstream of the first enhancement layer. The encoder quantizes and entropy codes the coefficient residual block of the second enhancement layer to obtain a bitstream of the second enhancement layer.

[0259] In step 806, the encoder quantizes and entropy encodes the transform coefficients of the first enhancement layer to obtain the bitstream of the first enhancement layer. This can also be completed after step 804 and before step 805, and this application does not limit this.

[0260] This embodiment may also include ( Figure 8 (not shown), performing an inverse transform on the dequantized coefficients of the first enhancement layer to obtain a reconstructed residual block of the first enhancement layer; and summing corresponding pixels in the reconstructed block of the base layer and the reconstructed residual block of the first enhancement layer to obtain a reconstructed block of the first enhancement layer. The resolution of the reconstructed block of the first enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the quality of the reconstructed block of the first enhancement layer is not lower than the quality of the reconstructed block of the base layer.

[0261] This embodiment may also include ( Figure 8 (not shown), quantizing and inverse-quantizing the coefficient residual block of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer, summing the corresponding pixels of the dequantized coefficients of the second enhancement layer with those of the first enhancement layer, and then performing an inverse transform to obtain a reconstructed residual block of the second enhancement layer; and summing the corresponding pixels in the reconstructed block of the base layer and the reconstructed residual block of the second enhancement layer to obtain a reconstructed block of the second enhancement layer. The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the first enhancement layer, or the quality of the reconstructed block of the second enhancement layer is not lower than the quality of the reconstructed block of the first enhancement layer.

[0262] The encoder can transform, quantize, and entropy encode the residual blocks of the residual pixel layer according to the above-mentioned transform block division method to obtain a code stream of the residual blocks of the residual pixel layer. The methods of transformation, quantization, and entropy encoding can refer to the above description and are not repeated here.

[0263] The encoder quantizes and entropy encodes the residual block of the transform domain coefficient layer to obtain a bitstream of the residual block of the transform domain coefficient layer. The quantization and entropy encoding methods can be referred to the above description and will not be repeated here.

[0264] Step 807: The encoder sends or stores the code streams of the first enhancement layer and the second enhancement layer.

[0265] The above describes the operation at the encoding end. The following describes the operation at the decoding end.

[0266] After obtaining the bitstream of the residual pixel layer of the image block to be decoded, the decoder performs entropy decoding, inverse quantization, and inverse transformation on the residual pixel layer based on the syntax elements carried in the bitstream to obtain a reconstructed residual block of the residual pixel layer. Based on the row and column numbers of the multiple pixels contained in the reconstructed residual block of the residual pixel layer and the row and column numbers of the multiple pixels contained in the reconstructed block of the base layer or low enhancement layer of the image block to be decoded, the decoder adds the pixels with the same row and column numbers in the two image blocks to obtain the reconstructed block of the residual pixel layer. After obtaining the bitstream of the transform domain coefficient layer of the image block to be decoded, the decoder performs entropy decoding and inverse quantization on the transform domain coefficient layer based on the syntax elements carried in the bitstream to obtain the dequantized coefficients of the transform domain coefficient layer. The residual bitstream of the low enhancement layer is entropy decoded and inverse quantized to obtain the dequantized coefficients of the low enhancement layer. The corresponding pixels in the dequantized coefficients of the transform domain coefficient layer and the dequantized coefficients of the low enhancement layer are summed and then inversely transformed to obtain the reconstructed residual block of the transform domain coefficient layer. The decoder adds the pixels with the same row and column numbers in the image blocks of the transform domain coefficient layer and the base layer based on the row and column numbers of the multiple pixels in the reconstructed residual block of the transform domain coefficient layer and the row and column numbers of the multiple pixels in the reconstructed block of the base layer to obtain the reconstructed block of the transform domain coefficient layer. The specific description is as follows:

[0267] Step 811: The decoder obtains a code stream of a first enhancement layer and a code stream of a second enhancement layer of an image block to be decoded.

[0268] The encoder and decoder can transmit code streams (including the code stream of the base layer and the code stream of the enhancement layer) through a wired or wireless link between the two, which will not be described in detail.

[0269] Similar to the encoding end, the image block to be decoded refers to the largest coding unit LCU in the entire frame image; or, the image block to be decoded refers to the entire frame image; or, the image block to be decoded refers to the region of interest ROI in the entire frame image.

[0270] Step 812: The decoder performs entropy decoding and inverse quantization on the bitstream of the first enhancement layer to obtain the dequantized coefficients of the first enhancement layer, and performs entropy decoding, inverse quantization and inverse transformation on the bitstream of the first enhancement layer to obtain the reconstructed residual block of the first enhancement layer.

[0271] After the decoder obtains the bitstream of the residual block of the residual pixel layer, it performs entropy decoding, inverse quantization, and inverse transformation based on the syntax elements carried in the bitstream to obtain the reconstructed residual block of the residual pixel layer. This process can be referred to the description of the encoder above. Since the encoder also performs the above operations when obtaining the reconstructed block, it will not be repeated here.

[0272] After obtaining the bitstream of the residual block of the transform domain coefficient layer, the decoder performs entropy decoding and inverse quantization based on the syntax elements carried in the bitstream to obtain the dequantized coefficients of the transform domain coefficient layer. This process can be referred to above. The encoder also performs the above operations when obtaining the reconstructed block, so it will not be repeated here.

[0273] Step 813: The decoder obtains a reconstructed block of the base layer of the image block to be decoded.

[0274] The decoding end may refer to the description of step 801 and adopt the same method as the encoder to obtain the reconstructed block of the base layer, for example, it may include: obtaining the code stream of the base layer; performing entropy decoding on the code stream of the base layer to obtain the quantization coefficient of the base layer; decoding the code stream of the base layer to obtain the prediction block of the base layer; performing inverse quantization and inverse transformation on the quantization coefficient of the base layer to obtain the reconstructed residual block of the base layer; summing the corresponding pixel points in the reconstructed residual block of the base layer and the prediction block of the base layer to obtain the reconstructed block of the base layer.

[0275] Step 814: The decoder sums the corresponding pixels in the reconstructed residual block of the first enhancement layer and the reconstructed block of the base layer to obtain the reconstructed block of the first enhancement layer.

[0276] The decoder adds the pixel points with the same row and column numbers in the two image blocks based on the row and column numbers of the multiple pixel points contained in the reconstructed residual block of the residual pixel layer and the row and column numbers of the multiple pixel points contained in the reconstructed block of the base layer to obtain the reconstructed block of the residual pixel layer.

[0277] Optionally, when the base layer and the first enhancement layer are obtained through spatial layering, the resolution of the reconstructed block of the base layer is the same as the resolution of the reconstructed block of the first enhancement layer, and the transform block partitioning method of the residual block of the first enhancement layer and the transform block partitioning method of the residual block of the base layer are independent of each other.

[0278] Step 815: The decoder performs entropy decoding and inverse quantization on the bitstream of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer.

[0279] Optionally, a transform block division manner of the residual block of the first enhancement layer is the same as a block division manner of the coefficient residual block of the second enhancement layer.

[0280] Step 816: The decoder sums the dequantized coefficients of the second enhancement layer and the corresponding pixel points in the dequantized coefficients of the first enhancement layer and performs an inverse transform to obtain a reconstructed residual block of the second enhancement layer.

[0281] Step 817: The decoder sums the corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the second enhancement layer.

[0282] At the decoding end, a resolution of the reconstructed block of the first enhancement layer is not lower than a resolution of the reconstructed block of the base layer, or a quality of the reconstructed block of the first enhancement layer is not lower than a quality of the reconstructed block of the base layer;

[0283] The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the first enhancement layer, or the quality of the reconstructed block of the second enhancement layer is not lower than the quality of the reconstructed block of the first enhancement layer.

[0284] The decoder adds the pixel points with the same row and column numbers in the two image blocks based on the row and column numbers of the multiple pixel points contained in the reconstructed residual block of the transform domain coefficient layer and the row and column numbers of the multiple pixel points contained in the reconstructed block of the base layer to obtain the reconstructed block of the transform domain coefficient layer.

[0285] The present application can directly calculate the difference between the corresponding pixel points in the image block to be coded and the reconstructed block of the base layer to obtain the residual block of the residual pixel layer, which reduces the process of obtaining the prediction block of the residual pixel layer. The transform domain coefficient layer also does not need the prediction block acquisition process, which reduces the processing flow of the encoder and improves the encoding efficiency of the encoder. In addition, the residual block of the residual pixel layer no longer uses the same TU division method as the residual block of the base layer. Instead, an adaptive TU division method is used to obtain a TU division method suitable for the residual block of the residual pixel layer, and then encoding is performed. When the TU division of the residual block of the residual pixel layer is independent of the CU division method of the base layer, the TU size of the residual pixel layer is no longer limited by the CU size, which can more effectively improve the compression efficiency of the residual block. The transform domain coefficient layer is implemented in the transform domain, which reduces the transformation process. At the same time, the TU division method of the transform domain coefficient layer inherits the TU division method of the residual pixel layer, so no additional area is required for TU division. It can effectively reduce the implementation complexity.

[0286] Figure 8The illustrated method can be applied to both spatial domain scalarization and quality domain scalarization. When spatial domain scalarization is used to obtain a base layer image block and a residual pixel layer image block, the resolution of the base layer image block is lower than the resolution of the residual pixel layer image block. Therefore, before the encoder calculates the difference between the corresponding pixels in the image block to be coded and the reconstructed block of the base layer to obtain the residual block of the enhancement layer of the image block to be coded, it can downsample the original image block to be coded to obtain an image block to be coded at a first resolution, and upsample the original reconstructed block of the base layer to obtain a reconstructed block of the base layer at the first resolution. That is, the encoder can downsample the original image block to be coded and upsample the original reconstructed block of the base layer, respectively, so that the resolution of the downsampled image block to be coded is the same as the resolution of the upsampled reconstructed block of the base layer.

[0287] Figure 8 In the described embodiment, a single residual pixel layer and a single transform domain coefficient layer are taken as an example. In other possible implementations, the image to be encoded may include multiple residual pixel layers. When two residual pixel layers are adjacent, the encoder obtains a reconstructed block of the low residual pixel layer of the image block to be encoded. The encoder takes the difference between the corresponding pixel points in the image block to be encoded and the reconstructed block of the low residual pixel layer to obtain a residual block of the high residual pixel layer of the image block to be encoded. The encoder determines a transform block partitioning method for the residual block of the high residual pixel layer. The encoder transforms the residual block of the high residual pixel layer according to the transform block partitioning method to obtain the transform coefficient of the high residual pixel layer, and further quantizes and entropy encodes the transform coefficient of the high residual pixel layer to obtain a code stream of the high residual pixel layer.

[0288] The image to be encoded may also include multiple transform domain coefficient layers. When two transform domain coefficient layers are adjacent, if the lower enhancement layer is a base layer or a residual pixel layer, the encoder obtains the transform coefficients of the lower enhancement layer for the image block to be encoded. The transform coefficients of the lower transform domain coefficient layer of the image block to be encoded are quantized and dequantized to obtain the dequantized coefficients of the lower transform domain coefficient layer. The transform coefficients of the lower enhancement layer of the image block to be encoded are quantized and dequantized to obtain the dequantized coefficients of the lower enhancement layer. The dequantized coefficients of the lower transform domain coefficients and the corresponding pixels of the dequantized coefficients of the lower enhancement layer are summed to obtain the reconstructed dequantized coefficients of the lower transform domain coefficient layer. The encoder calculates the difference between the transform coefficients of the lower enhancement layer and the corresponding pixels of the reconstructed dequantized coefficients of the lower transform domain coefficient layer to obtain a coefficient residual block of the higher transform domain coefficient layer of the image block to be encoded. The coefficient residual block of the higher transform domain coefficient layer is quantized and entropy coded to obtain a bitstream for the higher transform domain coefficient layer. When two transform domain coefficient layers are adjacent, if the next lower enhancement layer is a transform domain coefficient layer, the encoder cyclically obtains information from the layer below the next lower enhancement layer until the next lower layer is a base layer or a residual pixel layer, obtaining the transform coefficients of the next lower layer. The process is then cyclically processed in the same manner as above until the reconstructed dequantized coefficients of the next lower enhancement layer are obtained. The transform coefficients of the lower transform domain coefficient layer of the image block to be encoded are quantized and inverse-quantized to obtain the dequantized coefficients of the lower transform domain coefficient layer. The dequantized coefficients of the lower transform domain coefficients are summed with the corresponding pixels of the reconstructed dequantized coefficients of the next lower enhancement layer to obtain the reconstructed dequantized coefficients of the lower transform domain coefficient layer. The encoder then subtracts the corresponding pixels of the transform coefficients of the next lower layer from the reconstructed dequantized coefficients of the lower transform domain coefficient layer to obtain a coefficient residual block of the higher transform domain coefficient layer of the image block to be encoded. The coefficient residual block of the higher transform domain coefficient layer is quantized and entropy coded to obtain the bitstream of the higher transform domain coefficient layer.

[0289] In a possible implementation, the image block to be encoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the first enhancement layer) and the first transform domain coefficient layer (corresponding to the second enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The encoder calculates the difference between the corresponding pixel points in the image block to be encoded and the reconstructed block of the second enhancement layer to obtain the residual block of the third enhancement layer of the image block to be encoded; based on the residual block of the third enhancement layer, the transform block of the third enhancement layer is obtained, and the transform block of the third enhancement layer is transformed to obtain the transform coefficient of the third enhancement layer; the transform coefficient of the third enhancement layer is quantized and entropy encoded to obtain the code stream of the third enhancement layer.

[0290] In a possible implementation, the image block to be encoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the second enhancement layer) and the first residual pixel layer (corresponding to the first enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the fourth enhancement layer) and a second residual pixel layer (corresponding to the third enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The encoder quantizes and dequantizes the transform coefficients of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; calculates the difference between the transform coefficients of the third enhancement layer and the corresponding pixel points of the dequantized coefficients of the third enhancement layer to obtain a coefficient residual block of the fourth enhancement layer of the image block to be encoded; and quantizes and entropy encodes the coefficient residual block of the fourth enhancement layer to obtain a code stream of the fourth enhancement layer.

[0291] In one possible implementation, the image block to be encoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the first enhancement layer) and the first transform domain coefficient layer (corresponding to the second enhancement layer), the image block to be encoded may also include a second residual pixel layer (corresponding to the fourth enhancement layer) and a second transform domain coefficient layer (corresponding to the third enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The encoder quantizes and inversely quantizes the coefficient residual block of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; quantizes and inversely quantizes the coefficient residual block of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer; sums the dequantized coefficients of the second enhancement layer and the corresponding pixel points in the dequantized coefficients of the first enhancement layer to obtain reconstructed dequantized coefficients of the second enhancement layer; sums the dequantized coefficients of the third enhancement layer and the corresponding pixel points in the reconstructed dequantized coefficients of the second enhancement layer and then inversely transforms to obtain a reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the third enhancement layer; takes the difference between the corresponding pixel points in the image block to be encoded and the reconstructed block of the third enhancement layer to obtain a residual block of the fourth enhancement layer; obtains a transform block of the fourth enhancement layer based on the residual block of the fourth enhancement layer, transforms the transform block to obtain transform coefficients of the fourth enhancement layer; quantizes and entropy encodes the transform coefficients of the fourth enhancement layer to obtain a code stream of the fourth enhancement layer.

[0292] In a possible implementation, the image block to be encoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the second enhancement layer) and the first residual pixel layer (corresponding to the first enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The encoder quantizes and dequantizes the coefficient residual block of the second enhancement layer to obtain the dequantized coefficients of the second enhancement layer; sums the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the first enhancement layer to obtain the reconstructed dequantized coefficients of the second enhancement layer; takes the difference between the corresponding pixel points of the transform coefficients of the first enhancement layer and the reconstructed dequantized coefficients of the second enhancement layer to obtain the coefficient residual block of the third enhancement layer; quantizes and entropy encodes the coefficient residual block of the third enhancement layer to obtain the code stream of the third enhancement layer.

[0293] Similarly, in other possible implementations, the image to be decoded may include multiple residual pixel layers. When two residual pixel layers are adjacent, after the decoder obtains the code stream of the low residual pixel layer of the image block to be decoded, it performs entropy decoding, inverse quantization, and inverse transformation on the code stream of the low residual pixel layer based on the syntax elements carried in the code stream to obtain a reconstructed residual block of the low residual pixel layer. The decoder adds the pixels with the same row and column numbers in the two image blocks based on the row and column numbers of the multiple pixels contained in the reconstructed residual block of the low residual pixel layer and the row and column numbers of the multiple pixels contained in the reconstructed block of the base layer or the lower enhancement layer of the image block to be decoded to obtain the reconstructed block of the low residual pixel layer. After obtaining the code stream of the high residual pixel layer of the image block to be decoded, it performs entropy decoding, inverse quantization, and inverse transformation on the code stream of the high residual pixel layer based on the syntax elements carried in the code stream to obtain a reconstructed residual block of the high residual pixel layer. The decoder adds the pixel points with the same row and column numbers in the two image blocks based on the row and column numbers of the multiple pixel points contained in the reconstructed residual block of the high residual pixel layer and the row and column numbers of the multiple pixel points contained in the reconstructed block of the low residual pixel layer to obtain the reconstructed block of the high residual pixel layer.

[0294] In other possible implementations, the image to be decoded may include multiple transform domain coefficient layers. When two transform domain coefficient layers are adjacent, the decoder obtains the bitstream of the lower transform domain coefficient layer of the block to be decoded and, based on syntax elements carried in the bitstream, performs entropy decoding and inverse quantization on the lower transform domain coefficient layer to obtain dequantized coefficients for the lower transform domain coefficient layer. The next lower layer is a decoding layer one layer lower than the lower transform domain coefficient layer. The bitstream of the next lower layer is entropy decoded and inverse quantized to obtain dequantized coefficients for the next lower layer. If the next lower layer is a base layer or a residual pixel layer, the dequantized coefficients of the lower transform domain coefficient layer and the corresponding pixels in the dequantized coefficients of the next lower layer are summed and inversely transformed to obtain a reconstructed residual block for the lower transform domain coefficient layer. Here, if the next lower layer is the base layer, the decoder, based on the row and column numbers of the multiple pixels contained in the reconstructed residual block of the lower transform domain coefficient layer and the row and column numbers of the multiple pixels contained in the prediction block of the base layer, adds the pixels with the same row and column numbers in the two image blocks to obtain a reconstructed block for the lower transform domain coefficient layer. If the lower layer is a residual pixel layer, the decoder adds the pixel points with the same row and column numbers in the image blocks of the lower layer and the reconstructed residual block of the lower transform domain coefficient layer based on the row and column numbers of the multiple pixel points contained in the reconstructed residual block of the lower transform domain coefficient layer and the row and column numbers of the multiple pixel points contained in the reconstructed block of the lower layer to obtain the reconstructed block of the lower transform domain coefficient layer. After obtaining the bitstream of the high transform domain coefficient layer of the block to be decoded, it is entropy decoded and inverse quantized based on the syntax elements carried in the bitstream to obtain the dequantized coefficients of the high transform domain coefficient layer. The dequantized coefficients of the low transform domain coefficient layer and the corresponding pixel points in the dequantized coefficients of the lower layer are summed to obtain the reconstructed dequantized coefficients of the low transform domain coefficient layer. The reconstructed dequantized coefficients of the low transform domain coefficient layer and the corresponding pixel points in the dequantized coefficients of the high transform domain coefficient layer are summed and inverse transformed to obtain the reconstructed residual block of the high transform domain coefficient layer. If the next lower layer is the base layer, the decoder adds together the pixels with the same row and column numbers in the image blocks of the high transform domain coefficient layer and the prediction block of the base layer based on the row and column numbers of the pixels in the reconstructed residual block of the high transform domain coefficient layer and the row and column numbers of the pixels in the prediction block of the base layer, thereby obtaining a reconstructed block for the high transform domain coefficient layer. If the next lower layer is the residual pixel layer, the decoder adds together the pixels with the same row and column numbers in the image blocks of the high transform domain coefficient layer and the reconstructed block of the next lower layer, thereby obtaining a reconstructed block for the high transform domain coefficient layer. If the next lower layer is the transform domain coefficient layer, the decoder cyclically obtains bitstream information for the next lower layer until the next lower layer becomes the base layer or the residual pixel layer. The reconstructed dequantized coefficients of the next lower layer are then obtained according to the above process. The dequantized coefficients of the low transform domain coefficient layer and the corresponding pixel points in the reconstructed dequantized coefficients of the lower layer are summed and then inversely transformed to obtain the reconstructed residual block of the low transform domain coefficient layer.If the lower layer is the base layer, the decoder adds together the pixels with the same row and column numbers in the image blocks of the low transform domain coefficient layer based on the row and column numbers of the multiple pixels contained in the reconstructed residual block of the low transform domain coefficient layer and the row and column numbers of the multiple pixels contained in the prediction block of the base layer, thereby obtaining a reconstructed block for the low transform domain coefficient layer. If the lower layer is the residual pixel layer, the decoder adds together the pixels with the same row and column numbers in the image blocks of the low transform domain coefficient layer based on the row and column numbers of the multiple pixels contained in the reconstructed residual block of the low transform domain coefficient layer and the row and column numbers of the multiple pixels contained in the reconstructed block of the layer lower than the lower layer, thereby obtaining a reconstructed block for the low transform domain coefficient layer. Similarly, the method for obtaining the reconstructed block of the high transform domain coefficient layer can be used to obtain the reconstructed block of the low transform domain coefficient layer, which will not be repeated here.

[0295] In one possible implementation, the image to be decoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the first enhancement layer) and the first transform domain coefficient layer (corresponding to the second enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The decoder obtains the code stream of the third enhancement layer of the image block to be decoded; performs entropy decoding, inverse quantization and inverse transformation on the code stream of the third enhancement layer to obtain the reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the second enhancement layer to obtain the reconstructed block of the third enhancement layer.

[0296] In one possible implementation, the image block to be decoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the second enhancement layer) and the first residual pixel layer (corresponding to the first enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the fourth enhancement layer) and a second residual pixel layer (corresponding to the third enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The decoder obtains the code stream of the fourth enhancement layer of the image block to be decoded; performs entropy decoding and inverse quantization on the code stream of the fourth enhancement layer to obtain the dequantized coefficients of the fourth enhancement layer; performs entropy decoding and inverse quantization on the code stream of the third enhancement layer to obtain the dequantized coefficients of the third enhancement layer; sums the corresponding pixel points in the dequantized coefficients of the third enhancement layer and the dequantized coefficients of the fourth enhancement layer, and then performs inverse transformation to obtain a reconstructed residual block of the fourth enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the second enhancement layer to obtain a reconstructed block of the fourth enhancement layer.

[0297] In one possible implementation, the image to be decoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the first enhancement layer) and the first transform domain coefficient layer (corresponding to the second enhancement layer), it may also include a second residual pixel layer (corresponding to the fourth enhancement layer) and a second transform domain coefficient layer (corresponding to the third enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The decoder obtains the code stream of the fourth enhancement layer of the image block to be decoded; performs entropy decoding, inverse quantization and inverse transformation on the code stream of the fourth enhancement layer to obtain the reconstructed residual block of the fourth enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the third enhancement layer to obtain the reconstructed block of the fourth enhancement layer.

[0298] In one possible implementation, the image block to be decoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the second enhancement layer) and the first residual pixel layer (corresponding to the first enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The decoder obtains a code stream of the third enhancement layer of the image block to be decoded; performs entropy decoding and inverse quantization on the code stream of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; sums the corresponding pixels in the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the first enhancement layer to obtain reconstructed dequantized coefficients of the second enhancement layer; sums the corresponding pixels in the reconstructed dequantized coefficients of the second enhancement layer and the dequantized coefficients of the third enhancement layer, and then performs inverse transformation to obtain a reconstructed residual block of the third enhancement layer; sums the corresponding pixels in the reconstructed residual block of the third enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the third enhancement layer.

[0299] Figure 11 900 is an exemplary flowchart of the decoding method according to an embodiment of the present application. Process 900 may be performed by the video encoder 20 (or encoder) and the video decoder 30 (or decoder). Process 900 is described as a series of steps or operations. It should be understood that process 900 may be performed in various orders and / or occur simultaneously, not limited to Figure 11 The execution order shown. Figure 11 In the embodiment, the first enhancement layer corresponds to the transform domain coefficient layer in the enhancement layer, and the second enhancement layer corresponds to the residual pixel layer in the enhancement layer.

[0300] Step 901: The encoder obtains dequantized coefficients of a base layer of an image block to be encoded.

[0301] and Figure 8Similar to the embodiment shown, the image block to be encoded refers to the largest coding unit LCU in the entire frame image; or, the image block to be encoded refers to the entire frame image; or, the image block to be encoded refers to the region of interest ROI in the entire frame image. Figure 8 The description of the embodiment will not be repeated here.

[0302] Step 902: The encoder calculates the difference between the transform coefficients of the base layer of the image block to be encoded and the corresponding pixels of the dequantized coefficients of the base layer of the image block to be encoded to obtain a coefficient residual block of the first enhancement layer of the image block to be encoded. In this example, the first enhancement layer is a transform domain coefficient layer.

[0303] Step 903: The encoder obtains a reconstructed residual block of the first enhancement layer, and sums the corresponding pixels in the reconstructed residual block of the first enhancement layer and the prediction block of the base layer to obtain the reconstructed block of the first enhancement layer. Specifically, the encoder may quantize and inverse quantize the coefficient residual block of the transform domain coefficient layer to obtain the dequantized coefficients of the transform domain coefficient layer. The encoder sums the corresponding pixels in the dequantized coefficients of the base layer and the dequantized coefficients of the transform domain coefficient layer to obtain the reconstructed dequantized coefficients of the transform domain coefficient layer. The encoder inversely transforms the reconstructed dequantized coefficients of the transform domain coefficient layer to obtain the reconstructed residual block of the transform domain coefficient layer. The encoder sums the corresponding pixels in the reconstructed residual block of the transform domain coefficient layer and the prediction block of the base layer to obtain the reconstructed block of the transform domain coefficient layer.

[0304] The resolution of the reconstructed block of the first enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the quality of the reconstructed block of the first enhancement layer is not lower than the quality of the reconstructed block of the base layer.

[0305] In a possible implementation, the size of the maximum transform unit (LTU) of the coefficient residual block of the first enhancement layer is the same as the size of the reconstructed block of the base layer; or, the size of the LTU is the same as the size of the reconstructed block of the original base layer after upsampling.

[0306] Step 904: The encoder calculates the difference between the corresponding pixels in the image block to be encoded and the reconstructed block of the first enhancement layer to obtain a residual block of the second enhancement layer. In this example, the second enhancement layer is a residual pixel layer.

[0307] The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the quality of the reconstructed block of the second enhancement layer is not lower than the quality of the reconstructed block of the first enhancement layer. In one possible implementation, the block partitioning method of the coefficient residual block of the first enhancement layer and the transform block partitioning method of the residual block of the second enhancement layer are independent of each other.

[0308] Step 905: The encoder performs entropy coding to obtain code streams of the base layer, the first enhancement layer, and the second enhancement layer.

[0309] The encoder can transform, quantize, and entropy encode the residual blocks of the residual pixel layer according to the above-mentioned transform block division method to obtain a code stream of the residual blocks of the residual pixel layer. The methods of transformation, quantization, and entropy encoding can refer to the above description and are not repeated here.

[0310] The encoder quantizes and entropy codes the coefficient residual block of the transform domain coefficient layer to obtain a bitstream of the transform domain coefficient layer residual block. The quantization and entropy coding methods can be referred to the above description and will not be repeated here.

[0311] Step 906: The encoder sends or stores the code streams of the base layer, the first enhancement layer, and the second enhancement layer.

[0312] The above is the action of the encoding end. For the specific action of the encoding end, please refer to Figure 8 The description of the embodiment will not be repeated here. The following describes the operation of the decoding end.

[0313] Step 911: The decoder obtains the code streams of the base layer, the first enhancement layer, and the second enhancement layer of the image block to be decoded.

[0314] The encoder and decoder can transmit code streams (including the code stream of the base layer and the code stream of the enhancement layer) through a wired or wireless link between the two, which will not be described in detail.

[0315] and Figure 8 Similar to the embodiment shown, the image block to be decoded refers to the largest coding unit LCU in the entire frame image; or, the image block to be decoded refers to the entire frame image; or, the image block to be decoded refers to the region of interest ROI in the entire frame image.

[0316] Step 912: The decoder performs entropy decoding and inverse quantization on the bitstream of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer, performs entropy decoding on the bitstream of the base layer to obtain quantized coefficients of the base layer, inverse quantizes the quantized coefficients of the base layer to obtain dequantized coefficients of the base layer, sums corresponding pixel points in the dequantized coefficients of the first enhancement layer and the dequantized coefficients of the base layer, and obtains a reconstructed residual block of the first enhancement layer by performing an inverse transform.

[0317] Step 913: The decoder sums the reconstructed residual block of the first enhancement layer and the corresponding pixel points in the prediction block of the base layer to obtain a reconstructed block of the first enhancement layer.

[0318] Specifically, the decoder adds the pixel points with the same row and column numbers in the two image blocks based on the row and column numbers of the multiple pixel points contained in the reconstructed residual block of the transform domain coefficient layer and the row and column numbers of the multiple pixel points contained in the prediction block of the base layer to obtain the reconstructed block of the transform domain coefficient layer.

[0319] The resolution of the reconstructed block of the first enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the decoding quality of the reconstructed block of the first enhancement layer is not lower than the decoding quality of the reconstructed block of the base layer.

[0320] Step 914: The decoder performs entropy decoding, inverse quantization, and inverse transformation on the bitstream of the second enhancement layer to obtain a reconstructed residual block of the second enhancement layer.

[0321] After the decoder obtains the bitstream of the residual block of the residual pixel layer, it performs entropy decoding, inverse quantization, and inverse transformation based on the syntax elements carried in the bitstream to obtain the reconstructed residual block of the residual pixel layer. This process can be referred to the above description. The encoder also performs the above operations when obtaining the reconstructed block, so it will not be repeated here.

[0322] Step 915: The decoder sums the corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the first enhancement layer to obtain the reconstructed block of the second enhancement layer.

[0323] Specifically, the decoder can add the pixels with the same row and column numbers in the two image blocks based on the row and column numbers of the multiple pixels contained in the reconstructed residual block of the residual pixel layer and the row and column numbers of the multiple pixels contained in the reconstructed block of the first enhancement layer to obtain the reconstructed block of the residual pixel layer.

[0324] The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the first enhancement layer, or the decoding quality of the reconstructed block of the second enhancement layer is not lower than the decoding quality of the reconstructed block of the first enhancement layer.

[0325] In a possible implementation, a transform block partitioning method of the residual block of the second enhancement layer and a transform block partitioning method of the residual block of the base layer are independent of each other.

[0326] exist Figure 11 In the embodiment shown, a single transform domain coefficient layer and a single residual pixel layer are taken as an example. Figure 8 Similar to the embodiment described above, in other possible implementations, the image to be processed (the image to be encoded or the image to be decoded) may also include multiple residual pixel layers or multiple transform domain coefficient layers.

[0327] In a possible implementation, the image block to be encoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the second enhancement layer) and the first transform domain coefficient layer (corresponding to the first enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The difference between the corresponding pixel points in the image block to be encoded and the reconstructed block of the second enhancement layer is obtained to obtain the residual block of the third enhancement layer of the image block to be encoded; based on the residual block of the third enhancement layer, the transform block of the third enhancement layer is obtained, and the transform block is transformed to obtain the transform coefficient of the third enhancement layer; the transform coefficient of the third enhancement layer is quantized and entropy coded to obtain the code stream of the third enhancement layer.

[0328] In one possible implementation, the image block to be encoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The transform coefficients of the third enhancement layer are quantized and inversely quantized to obtain the dequantized coefficients of the third enhancement layer; the transform coefficients of the third enhancement layer of the image block to be encoded and the corresponding pixel points of the dequantized coefficients of the third enhancement layer are subtracted to obtain the coefficient residual block of the fourth enhancement layer; the coefficient residual block of the fourth enhancement layer is quantized and entropy coded to obtain the code stream of the fourth enhancement layer.

[0329] In one possible implementation, the image block to be encoded may include multiple residual pixel layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), the image block to be encoded may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The encoder quantizes and inversely quantizes the coefficient residual block of the third enhancement layer to obtain the dequantized coefficients of the third enhancement layer; sums the corresponding pixel points in the dequantized coefficients of the third enhancement layer and the dequantized coefficients of the second enhancement layer and then inversely transforms to obtain the reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the first enhancement layer to obtain the reconstructed block of the third enhancement layer; differs the corresponding pixel points in the image block to be encoded and the reconstructed block of the third enhancement layer to obtain the residual block of the fourth enhancement layer; obtains the transform block of the fourth enhancement layer according to the residual block of the fourth enhancement layer, transforms the transform block to obtain the transform coefficients of the fourth enhancement layer; quantizes and entropy encodes the transform coefficients of the fourth enhancement layer to obtain the code stream of the fourth enhancement layer.

[0330] In one possible implementation, the image block to be encoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The encoder quantizes and dequantizes the transform coefficients of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer;

[0331] Differences are taken between corresponding pixel points of the transform coefficients of the second enhancement layer of the image block to be encoded and the dequantized coefficients of the second enhancement layer to obtain a coefficient residual block of the third enhancement layer; and quantization and entropy encoding are performed on the coefficient residual block of the third enhancement layer to obtain a code stream of the third enhancement layer.

[0332] In one possible implementation, the image block to be decoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the second enhancement layer) and the first transform domain coefficient layer (corresponding to the first enhancement layer), it may also include a second residual pixel layer (corresponding to the third enhancement layer) and a second transform domain coefficient layer (corresponding to the fourth enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The decoder obtains the code stream of the third enhancement layer of the image block to be decoded; performs entropy decoding, inverse quantization and inverse transformation on the residual code stream of the third enhancement layer to obtain the reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the second enhancement layer to obtain the reconstructed block of the third enhancement layer.

[0333] In a possible implementation, the image block to be decoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the fourth enhancement layer) and a second residual pixel layer (corresponding to the third enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The decoder obtains the code stream of the fourth enhancement layer of the image block to be decoded; performs entropy decoding and inverse quantization on the code stream of the fourth enhancement layer to obtain the dequantized coefficients of the fourth enhancement layer; performs entropy decoding and inverse quantization on the code stream of the third enhancement layer to obtain the dequantized coefficients of the third enhancement layer; sums the corresponding pixel points in the dequantized coefficients of the third enhancement layer and the dequantized coefficients of the fourth enhancement layer, and then performs inverse transformation to obtain a reconstructed residual block of the fourth enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the second enhancement layer to obtain the reconstructed block of the fourth enhancement layer.

[0334] In one possible implementation, the image block to be decoded may include multiple residual pixel layers. For example, in addition to the first residual pixel layer (corresponding to the second enhancement layer) and the first transform domain coefficient layer (corresponding to the first enhancement layer), it may also include a second residual pixel layer (corresponding to the fourth enhancement layer) and a second transform domain coefficient layer (corresponding to the third enhancement layer). The first residual pixel layer is a low residual pixel layer, and the second residual pixel layer is a high residual pixel layer. The decoder obtains the code stream of the fourth enhancement layer of the image block to be decoded; performs entropy decoding, inverse quantization and inverse transformation on the residual code stream of the fourth enhancement layer to obtain the reconstructed residual block of the fourth enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the third enhancement layer to obtain the reconstructed block of the fourth enhancement layer.

[0335] In one possible implementation, the image block to be decoded may include multiple transform domain coefficient layers. For example, in addition to the first transform domain coefficient layer (corresponding to the first enhancement layer) and the first residual pixel layer (corresponding to the second enhancement layer), it may also include a second transform domain coefficient layer (corresponding to the third enhancement layer) and a second residual pixel layer (corresponding to the fourth enhancement layer). The first transform domain coefficient layer is a low transform domain coefficient layer, and the second transform domain coefficient layer is a high transform domain coefficient layer. The decoder obtains the code stream of the third enhancement layer of the image block to be decoded; performs entropy decoding and inverse quantization on the code stream of the third enhancement layer to obtain the dequantized coefficients of the third enhancement layer; performs entropy decoding and inverse quantization on the code stream of the second enhancement layer to obtain the dequantized coefficients of the second enhancement layer; sums the corresponding pixel points in the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the third enhancement layer, and then performs inverse transformation to obtain a reconstructed residual block of the third enhancement layer; sums the corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the first enhancement layer to obtain the reconstructed block of the third enhancement layer.

[0336] Please refer to the description of the above embodiment for details, which will not be repeated here.

[0337] Figure 12 1 is an exemplary flowchart of the decoding method of the present invention. Process 1100 may be performed by the video encoder 20 (or encoder) and the video decoder 30 (or decoder). Process 1100 is described as a series of steps or operations. It should be understood that process 1100 may be performed in various orders and / or occur simultaneously, not limited to Figure 11 In this embodiment, there may be more enhancement layers, such as two adjacent residual pixel layers, where the first residual pixel layer is a low residual pixel layer and the second residual pixel layer is a high residual pixel layer.

[0338] Step 1101: The encoder obtains a reconstructed block of a first residual pixel layer of an image block to be encoded.

[0339] like Figure 12 As shown, after the image block is layered using spatial domain grading, temporal domain grading or quality domain grading, in addition to the basic layer and the transform domain coefficient layer, it is also divided into multiple residual pixel layers. Figure 8 The method shown is a coding and decoding method based on a base layer versus a residual pixel layer one level higher than the base layer. This embodiment is a coding and decoding method based on a lower residual pixel layer (first residual pixel layer) versus a second residual pixel layer one level higher.

[0340] The difference from the method of obtaining the reconstructed block of the base layer is that the coding block can obtain the residual block of the first enhancement layer by taking the difference between the corresponding pixel points in the reconstructed block of the third layer of the image block to be coded and the image block to be coded. The third layer is a layer lower than the first residual pixel layer, which can be a base layer, a transform domain coefficient layer, or a residual pixel layer. The residual block of the first residual pixel layer is then transformed and quantized to obtain the quantized coefficients of the residual block of the first residual pixel layer, and the quantized coefficients of the residual block of the first residual pixel layer are then inversely quantized and inversely transformed to obtain the reconstructed residual block of the first residual pixel layer. Finally, the corresponding pixel points in the reconstructed block of the third layer and the reconstructed residual block of the first residual pixel layer are summed to obtain the reconstructed block of the first residual pixel layer.

[0341] Step 1102: The encoder calculates the difference between corresponding pixels in the image block to be encoded and the reconstructed block of the first enhancement layer to obtain a residual block of the second enhancement layer of the image block to be encoded.

[0342] Step 1103: The encoder transforms, quantizes, and entropy encodes the residual block of the second enhancement layer according to the transform block division method to obtain a code stream of the second residual pixel layer.

[0343] The transform block division method of the residual block of the second residual pixel layer is different from the transform block division method of the residual block of the first residual pixel layer.

[0344] The encoder may also use RDO processing to determine the transform block division method of the residual block of the second enhancement layer, which will not be described in detail here.

[0345] The present application no longer uses the same TU division method for the residual blocks of the second residual pixel layer as the residual blocks of the first residual pixel layer and the residual blocks of the base layer. Instead, an adaptive TU division method is adopted to obtain a TU division method suitable for the residual blocks of the second residual pixel layer, and then encode them. When the TU division of the residual blocks of the second residual pixel layer is independent of the CU or TU division method of other layers, the TU size of the second residual pixel layer is no longer limited by the CU size, or the TU size of the second residual pixel layer is no longer limited by the TU size of the first residual pixel layer, which can improve the flexibility of encoding.

[0346] Step 1104: The encoder sends or stores the code stream of the second enhancement layer.

[0347] Step 1111: The decoder obtains a code stream of the second residual pixel layer of the image block to be decoded.

[0348] Step 1112: The decoder performs entropy decoding, inverse quantization, and inverse transformation on the bitstream of the residual block of the second residual pixel layer to obtain a reconstructed residual block of the second enhancement layer.

[0349] Step 1113: The decoder obtains a reconstructed block of the first enhancement layer of the image block to be decoded.

[0350] Step 1114: The decoder sums the corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the first enhancement layer to obtain a reconstructed block of the second enhancement layer.

[0351] This embodiment can directly calculate the difference between the corresponding pixel points in the image block to be encoded and the reconstructed block of the first enhancement layer to obtain the residual block of the second enhancement layer, thereby reducing the process of obtaining the prediction block of the second enhancement layer, reducing the processing flow of the encoder, and improving the encoding efficiency of the encoder. In addition, the residual block of the second enhancement layer no longer uses the same TU division method as the residual block of the first enhancement layer. Instead, an adaptive TU division method is used to obtain a TU division method suitable for the residual block of the second enhancement layer, and then encoding is performed. When the TU division of the residual block of the second enhancement layer is independent of the CU or TU division method of the first enhancement layer, the TU size of the second enhancement layer is no longer limited by the CU size, or the TU size of the second enhancement layer is no longer limited by the TU size of the first enhancement layer, which can more effectively improve the compression efficiency of the residual block.

[0352] Figure 12 The illustrated method can be applied to both spatial domain scalarization and quality domain scalarization. When spatial domain scalarization is used to obtain a first enhancement layer image block and a second enhancement layer image block, the resolution of the first enhancement layer image block is lower than that of the second enhancement layer image block. Therefore, before the encoder calculates the difference between corresponding pixels in the image block to be coded and the reconstructed block of the first enhancement layer to obtain a residual block of the second enhancement layer of the image block to be coded, it can downsample the original image block to be coded to obtain an image block to be coded at the first resolution, and upsample the original reconstructed block of the first enhancement layer to obtain a reconstructed block of the first enhancement layer at the first resolution. That is, the encoder can downsample the original image block to be coded and upsample the original reconstructed block of the first enhancement layer to ensure that the resolution of the image block to be coded and the resolution of the reconstructed block of the first enhancement layer are the same.

[0353] Figure 13 1 is an exemplary flow chart of the method for encoding the enhancement layer of the present application. Process 1200 may be performed by the video encoder 20 (or encoder) and the video decoder 30 (or decoder). Process 1200 is described as a series of steps or operations. It should be understood that process 1200 may be performed in various orders and / or occur simultaneously, not limited to Figure 13 In this embodiment, there may be more enhancement layers, such as two adjacent transform domain coefficient layers.

[0354] Step 1201: The encoder obtains reconstructed dequantized coefficients of the first transform domain coefficient layer of the image block to be encoded.

[0355] After the image block is layered using spatial domain scalarization, temporal domain scalarization, or quality domain scalarization, it is further divided into multiple transform domain coefficient layers in addition to the base layer and the residual pixel layer. This embodiment is a coding and decoding method for a higher-layer transform domain coefficient layer (second transform domain coefficient layer) based on a lower-layer transform domain coefficient layer (first transform domain coefficient layer).

[0356] The coding block can perform quantization and inverse quantization on the transform coefficients of the third layer of the image block to be coded to obtain the dequantized coefficients of the third layer. The third layer is a layer lower than the first transform domain coefficient layer, which can be a base layer or a residual pixel layer enhancement layer. If the third layer is a transform domain coefficient layer, the third layer is converted into a lower-layer transform domain coefficient layer, and the first transform domain coefficient layer is converted into a higher-layer transform domain coefficient layer, and the same processing can be performed as described in this embodiment. The transform coefficients of the third layer are subtracted from the corresponding pixels in the dequantized coefficients of the third layer to obtain a coefficient residual block of the first transform domain coefficient layer. The coefficient residual block of the first transform domain coefficient layer is quantized and inverse quantized to obtain the dequantized coefficients of the first transform domain coefficient layer. The corresponding pixels of the dequantized coefficients of the first transform domain coefficient layer and the dequantized coefficients of the third layer are summed to obtain the reconstructed dequantized coefficients of the first transform domain coefficient layer.

[0357] Step 1202: Calculate the difference between corresponding pixel points of the transform coefficients of the third layer and the reconstructed dequantized coefficients of the first transform domain coefficient layer to obtain a coefficient residual block of the second transform domain coefficient layer of the image block to be encoded.

[0358] Step 1203: The encoder quantizes and entropy encodes the transform coefficients of the third layer to obtain the bitstream of the third layer, and quantizes and entropy encodes the coefficient residual blocks of the first enhancement layer and the second enhancement layer to obtain the bitstreams of the first enhancement layer and the second enhancement layer.

[0359] Step 1204: The encoder sends or stores the code streams of the third layer, the first enhancement layer, and the second enhancement layer.

[0360] Step 1211: The decoder obtains the code stream of the third layer, the code stream of the first enhancement layer, and the code stream of the second enhancement layer of the image block to be decoded.

[0361] Step 1212: The decoder performs entropy decoding and inverse quantization on the bitstream of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer. The decoder performs entropy decoding and inverse quantization on the bitstream of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer. The decoder performs entropy decoding and inverse quantization on the bitstream of the third layer to obtain dequantized coefficients of the third layer.

[0362] Step 1213: The decoder sums the corresponding pixel points in the dequantized coefficients of the third layer and the dequantized coefficients of the first enhancement layer to obtain the reconstructed dequantized coefficients of the first enhancement layer.

[0363] Step 1214: The decoder sums the corresponding pixel points in the reconstructed dequantized coefficients of the first enhancement layer and the dequantized coefficients of the second enhancement layer and performs an inverse transform to obtain a reconstructed residual block of the second enhancement layer.

[0364] Step 1215: The decoder sums the corresponding pixels in the reconstructed residual block of the second enhancement layer and the prediction block of the base layer of the block to be decoded or the reconstructed block of the layer lower than the third layer to obtain the reconstructed block of the second enhancement layer.

[0365] In this step, if there is a residual pixel layer lower than the low transform domain coefficient layer, such as when the first residual pixel layer is the first enhancement layer, the second residual pixel layer is the second enhancement layer, the low transform domain coefficient layer is the third enhancement layer, and the high transform domain coefficient layer is the fourth enhancement layer, the second residual pixel layer is a layer lower than the low transform domain coefficient layer and is the residual pixel layer closest to the low transform domain coefficient layer. In this case, the layer lower than the residual pixel layer closest to the low transform domain coefficient layer (the second residual pixel layer) is used as the fourth layer (which is the base layer or enhancement layer). The reconstructed block of the low transform domain coefficient layer is obtained by summing the corresponding pixels of the reconstructed residual block of the low transform domain coefficient layer and the reconstructed block of the fourth layer. If there is no residual pixel layer lower than the low transform domain coefficient layer, the reconstructed block of the low transform domain coefficient layer is obtained by summing the corresponding pixels of the reconstructed residual block of the low transform domain coefficient layer and the prediction block of the base layer.

[0366] The following is combined with Figures 14a-19b , for the above Figures 8-13 The specific implementation method is described in detail.

[0367] Example 1

[0368] Figure 14a and 14b This is an exemplary flowchart of the decoding method of the enhancement layer of the present application, which may correspond to the above Figure 8 Examples of Figure 14a and 14b As shown, this embodiment uses quality domain grading to layer image blocks, where the image blocks are LCUs. In this embodiment, the residual pixel layer corresponds to the first enhancement layer, and the transform domain coefficient layer corresponds to the second enhancement layer.

[0369] Encoding side:

[0370] Step 1: Input a frame of source image data into the encoder, perform base layer (BL) encoding according to the largest coding unit (LCU), and obtain a reconstructed image of the base layer of the largest coding unit (LCU).

[0371] In this step, the encoder can use an encoder that complies with the H.264 / H.265 standard, or use other non-standard video encoders for encoding, which is not limited here. In this embodiment, the BL encoder is described by taking an encoder that complies with the H.265 encoding standard as an example.

[0372] The source image is divided into sub-images according to the size of the largest coding unit (LCU). CU partitioning is performed for each LCU to obtain the optimal prediction unit (PU) partitioning method for intra-frame prediction and inter-frame prediction of the block. Within the CU, the residual quadtree (RQT) partitioning process can be performed to obtain the optimal TU partitioning. Based on this TU partitioning method, the residual can be transformed and quantized, and entropy encoded together with the LCU control information, prediction information, motion information, etc. to obtain the BL code stream. Among them, when obtaining the optimal PU and TU partitioning process, the rate-distortion optimization (RDO) method can be used to achieve the best coding compression rate. On the other hand, the quantized residual transform coefficients are dequantized and detransformed, and then superimposed with the predicted pixels at the corresponding position in the CU to obtain the reconstructed image of the CU. By performing the above process on each CU in the LCU, the reconstructed image of all CUs in the entire LCU can be obtained, that is, the BL reconstructed image corresponding to the LCU.

[0373] Here, before executing step 2, the corresponding reconstructed image in the LCU may be further post-processed (such as loop filtering), and then used as the BL reconstructed image in the LCU after processing. Whether to perform the post-processing process is not limited here.

[0374] Step 2: Subtract the pixels of the source image corresponding to the LCU from those of the BL reconstructed image to obtain the residual image of the source image and the BL reconstructed image in the LCU. Since the difference between the current source image and the BL reconstructed image corresponding to the LCU is taken, the BL reconstructed image is used as the reference image in the LCU.

[0375] Step 3: Send the residual image obtained in step 2 to the first enhancement layer encoder for residual coding.

[0376] In this step, after obtaining the enhancement layer (EL) residual image corresponding to the LCU, the LCU can be fed into the EL encoder for processing. The EL maximum transform unit (LTU) is pre-specified as the LCU corresponding to the BL at the current position. TU adaptive partitioning is performed to obtain TUs of different sizes. Specifically, this method uses the RDO method based on the residual quadtree (RQT).

[0377] After determining the optimal TU partitioning for an LTU, the EL residual image in that LTU is transformed and quantized according to that partitioning to obtain quantized coefficients. These coefficients are then entropy-coded along with the LTU partitioning information to obtain the EL codestream for the LTU's residual pixel layer. Furthermore, the quantized coefficients undergo inverse quantization and inverse transformation to obtain a coded distorted reconstructed image of the LTU's EL residual image. This image is then superimposed with the BL reconstructed image in the LCU corresponding to the LTU to obtain the residual pixel layer reconstructed image corresponding to that LTU.

[0378] Here, the EL code stream of the LTU can be encoded using the syntax element organization and encoding method of the TU part code stream in the existing coding standard. The TU after adaptive division is independent of the TU division of the BL.

[0379] In this step, the transform coefficients of each TU are saved as reference transform coefficients; and the inverse quantized coefficients are saved as dequantized coefficients of the current EL.

[0380] Step 4: Subtract the reference transform coefficient of each TU in the LTU from the dequantized coefficient of the TU to obtain the coefficient residual of the TU.

[0381] Step 5: Send the coefficient residual of the TU obtained in step 4 to the second enhancement layer encoder for encoding.

[0382] The coefficient residuals of each TU in the LTU are quantized using the same LTU division method as the residual pixel layer (EL). Quantized coefficients are then entropy-encoded along with the LTU division information to obtain the transform domain coefficient layer (EL) codestream for that LTU. Furthermore, the quantized coefficient residuals are inversely quantized to obtain dequantized coefficients. These coefficients are then superimposed with the dequantized coefficients of the previous residual pixel layer (EL) to obtain the reconstructed dequantized coefficients of the current EL. These reconstructed dequantized coefficients are inversely transformed to obtain a reconstructed residual image for that LTU. This image is then superimposed with the reference image in the LCU corresponding to the LTU to obtain the reconstructed image of the transform domain coefficient layer (EL) corresponding to that LTU.

[0383] Step 6: Following steps 1 to 5, all LCUs of the entire source image are completed, resulting in the BL reconstructed image for the entire image and the EL reconstructed image for each layer, as well as the BL codestream for the entire image and the EL codestream for each layer. These reconstructed images can be placed in a reference frame list for reference by subsequent frame images in the time domain. After obtaining the BL or EL reconstructed image for each layer, further post-processing (such as loop filtering) can be performed, and then placed in the reference frame list after processing, which is not limited here. If post-processing is performed, the indication information for performing post-processing and placing it in the reference frame list must be encoded into the corresponding codestream.

[0384] Here, information indicating whether the current EL type is the residual pixel layer EL or the transform domain coefficient layer EL needs to be encoded into the bitstream so that the decoding end can parse and obtain the current EL type and perform correct decoding processing.

[0385] Step 7: Repeat steps 1 to 6 to encode subsequent frames of the image sequence until the encoding of the entire image sequence is completed.

[0386] The steps in this embodiment are based on LCU-level processing of each frame image. Since the LCU image is a subset of the entire frame image, all steps can also be processed in a frame-level manner, and the processing process is consistent.

[0387] Decoding end:

[0388] Step 1: Obtain the first enhancement layer code stream of the current frame, send it to the first enhancement layer decoder for decoding, and obtain the corresponding first enhancement layer reconstructed image in the LTU.

[0389] After obtaining the EL code stream of the current frame, the information that the current EL is the residual pixel layer can be parsed from the code stream, thereby instructing the current layer code stream to be sent to the decoder of the residual pixel layer for processing.

[0390] After the code stream is sent to the decoder, entropy decoding is performed to obtain the division information of the current layer LTU (the same size as the LCU at the corresponding BL position) and the residual information of the LTU. According to the division of the LTU, the residual information is dequantized and inversely transformed to obtain the reconstructed residual image corresponding to the LTU. This image is added to the BL reconstructed image corresponding to the LTU to obtain the first layer EL reconstructed image corresponding to the LTU.

[0391] The BL decoder uses a decoder that can decode the BL stream corresponding to the BL stream encoding rules. A decoder that complies with the H.264 / H.265 standard can be used here, or other non-standard video decoders can be used for decoding, which is not limited here. In this embodiment, the BL decoder is described using a decoder that complies with the H.265 encoding standard as an example.

[0392] After the base layer code stream is fed into the decoder for entropy decoding, it obtains the LCU's coding block division information, transform block division information, control information, prediction information, and motion information. This information is used for intra-frame prediction and inter-frame prediction to obtain the LCU's predicted image. Furthermore, residual information is obtained. Based on the decoded transform block division information, the LCU's residual image is obtained through inverse quantization and inverse transformation. The predicted image and residual image are superimposed to obtain the BL reconstructed image corresponding to the LCU.

[0393] If the decoding obtains information that post-processing (such as loop filtering) needs to be performed on the reconstructed image, the corresponding post-processing process is performed after the reconstructed image is obtained, so that it corresponds to the process in the encoding.

[0394] In this process, the dequantized coefficients of each TU in the LTU are saved. Since the current reconstructed residual image corresponding to the LTU is superimposed with the BL reconstructed image to generate the current layer EL reconstructed image, the BL reconstructed image is used as the reference image of the LTU.

[0395] In this step, if the decoding obtains information that post-processing (such as loop filtering) needs to be performed on the reconstructed image, the corresponding post-processing process is performed after the reconstructed image to make it correspond to the process in the encoding.

[0396] Step 2: Obtain the second enhancement layer code stream of the current frame, send it to the second enhancement layer decoder for decoding, and obtain the reconstructed image of the corresponding second enhancement layer in the LTU.

[0397] After obtaining the current frame EL code stream, the information that the current EL is the transform domain coefficient layer can be parsed from the code stream, thereby instructing the current layer code stream to be sent to the decoder of the transform domain coefficient layer for processing.

[0398] After the code stream is sent to the decoder, it is entropy decoded to obtain the LTU division information (the same size as the LCU at the corresponding position of the BL) and the residual information of the LTU. The LTU division method of this layer should be the same as the division of the LTU at the corresponding position of the previous residual pixel layer. The residual information in each TU is dequantized and superimposed with the dequantized coefficients of the TU corresponding to the previous residual pixel layer to obtain the reconstructed dequantized coefficients of the current EL. The reconstructed dequantized coefficients are inversely transformed to obtain the reconstructed residual image corresponding to the LTU. This image is added to the reference image corresponding to the LTU to obtain the reconstructed image of the second enhancement layer corresponding to the LTU.

[0399] In this step, if the decoding obtains information that post-processing (such as loop filtering) needs to be performed on the reconstructed image, the corresponding post-processing process is performed after the reconstructed image to make it correspond to the process in the encoding.

[0400] Step 3: Repeat steps 1 and 2 until the base layer, residual pixel enhancement layer, and transform domain coefficient layer processing for all LCUs corresponding to the entire frame are completed. The base layer reconstructed image, residual pixel enhancement layer reconstructed image, and transform domain coefficient layer reconstructed image of the current frame can be obtained. The base layer reconstructed image and the reconstructed image of each enhancement layer can be placed in the reference frame list for reference by subsequent frame images in the time domain.

[0401] If the code stream contains information indicating that the reconstructed image should be post-processed (such as loop filtering) before being placed in the reference frame list, this information can be obtained through entropy decoding, and the BL reconstructed image and the EL reconstructed image can be post-processed and then placed in the reference frame list to correspond to the process in the encoding.

[0402] Step 4: Repeat steps 1 to 3 until the entire image sequence is decoded.

[0403] The steps in this embodiment are based on LCU-level processing of each frame image. Since the LCU image is a subset of the entire frame image, all steps can also be processed in an entire frame manner, and the processing process is consistent.

[0404] The transform domain coefficient layer is integrated into the residual pixel layer to obtain a hybrid hierarchical coding architecture. In this architecture, the residual pixel layer no longer uses the TU division method in the BL. Instead, it adaptively selects a more appropriate TU division method for encoding, thereby more effectively improving the compression efficiency of the residual image. The transform domain coefficient layer no longer requires TU division or TU transformation. This layer directly quantizes the transform coefficients, eliminating the step of the residual pixel enhancement layer, reducing computational complexity, and significantly reducing hardware costs. The hybrid use of the two can achieve an optimal balance between image coding efficiency and hardware costs.

[0405] EL no longer performs the prediction process. The residual pixel layer directly performs adaptive division and transform coding on the residual image, and the transform domain coefficient layer directly performs quantization coding on the transform coefficients. Compared with the coding method that requires prediction, it can further improve the coding speed, reduce the hardware pipeline processing process, and save hardware processing costs.

[0406] This coding framework uses a layered architecture that combines transform domain coefficient layers with residual pixel layers. This changes the inter-layer processing between the transform domain coefficient layer and the residual pixel layer, improving coding efficiency while reducing hardware costs while maintaining the same number of layers. The decoder can effectively parallelize BL and EL processing at the LCU or frame level, saving hardware costs and achieving an optimal balance between coding efficiency and hardware costs.

[0407] In this embodiment, each image may be implemented in the form of an image block.

[0408] In this embodiment, each residual may be implemented in the form of a residual block.

[0409] The execution order of the steps in this embodiment can be adjusted. For example, the generation of the BL code stream in the encoding end can be implemented in other steps.

[0410] Example 2

[0411] Figure 15a and 15bThis is an exemplary flowchart of the encoding method of the enhancement layer of the present application, which may correspond to the above Figure 11 Examples of Figure 15a and 15b As shown, this embodiment uses quality domain grading to hierarchize image blocks, and the image blocks are LCUs. In this embodiment, the transform domain coefficient layer is the first enhancement layer, and the residual pixel layer corresponds to the second enhancement layer.

[0412] Encoding side:

[0413] Step 1: The same as step 1 of the encoding end in embodiment 1.

[0414] In this step, the transform coefficients of each TU in the LCU are saved as the reference transform coefficients; the inverse quantized coefficients are saved as the dequantized coefficients of the BL. Since the current layer residual is obtained by subtracting the source image corresponding to the LCU from the BL prediction image at the BL layer, the BL prediction image is used as the reference image in the LCU.

[0415] Step 2: Subtract the reference transform coefficient of each TU in the LCU from the dequantized coefficient of the TU to obtain the coefficient residual of the TU.

[0416] Step 3: Send the TU coefficient residual obtained in step 2 to the first enhancement layer encoder (transform domain coefficient layer encoder) for encoding.

[0417] This step is similar to step five on the encoding side of embodiment one, except that the residual pixel layer of the previous layer is changed to BL. According to the same TU division method as BL, the coefficient residual of each TU in the LCU is quantized to obtain the quantized coefficient, which is entropy encoded together with the TU division information to obtain the transform domain coefficient layer EL code stream of the TU. On the other hand, after the quantized coefficient residual is inversely quantized, the dequantized coefficient of the coefficient residual is obtained, and the coefficient is superimposed with the dequantized coefficient of BL to obtain the reconstructed dequantized coefficient of the current EL. The reconstructed dequantized coefficient is inversely transformed to obtain the reconstructed residual image of the LCU, and the image is superimposed with the corresponding reference image of the LCU to obtain the transform domain coefficient layer EL reconstructed image corresponding to the LCU.

[0418] Step 4: Subtract the corresponding pixels of the source image from those of the reconstructed image of the transform domain coefficient layer to obtain the EL residual image.

[0419] Step 5: Send the residual image obtained in step 4 to the second enhancement layer encoder (residual pixel layer encoder) for residual encoding. This step is the same as the processing method of step 3 at the encoding end in embodiment 1.

[0420] Step 6: The same as step 6 on the encoding side in embodiment 1.

[0421] Step 7: The same as step 7 on the encoding side in embodiment 1.

[0422] Similarly, the steps in this embodiment are based on LCU-level processing of each frame image. Since the LCU image is a subset of the entire frame image, all steps can also be processed in a frame-level manner, and the processing process is consistent.

[0423] Decoding end:

[0424] Step 1: Obtain the base layer code stream and send it to the base layer decoder for decoding to obtain the base layer reconstructed image corresponding to the LCU.

[0425] In this step, the inverse quantized coefficients of each TU in the LCU are saved as the dequantized coefficients of the BL. Since the BL reconstructed residual image corresponding to the LCU is superimposed with the BL prediction image to generate the BL reconstructed image, the BL prediction image is used as the reference image of the LCU.

[0426] Step 2: Obtain the first enhancement layer code stream of the current layer, send it to the first enhancement layer decoder for decoding, and obtain the corresponding first enhancement layer reconstructed image in the LCU.

[0427] After obtaining the current frame EL code stream, the information that the current EL is the transform domain coefficient layer can be parsed from the code stream, thereby instructing the current layer code stream to be sent to the decoder of the transform domain coefficient layer for processing.

[0428] The decoding steps are similar to those in step 2 of the first embodiment, except that the previous layer is the BL layer. After the code stream is sent to the decoder, entropy decoding is performed to obtain the LTU division information (the same size as the LCU at the corresponding position of the BL) and the residual information of the LTU. The LTU division of this layer should be the same as the TU division in the LCU at the corresponding position of the BL. The residual information in each TU is inversely quantized and superimposed with the dequantized coefficients of the TU corresponding to the BL to obtain the reconstructed dequantized coefficients of the current EL. The reconstructed dequantized coefficients are inversely transformed to obtain the reconstructed residual image corresponding to the LTU. This image is added to the reference image corresponding to the LTU (i.e., the BL prediction image) to obtain the first-layer EL reconstructed image corresponding to the LTU.

[0429] In this step, if the decoding obtains information that post-processing (such as loop filtering) needs to be performed on the reconstructed image, the corresponding post-processing process is performed after the reconstructed image to make it correspond to the process in the encoding.

[0430] Step 3: Obtain the second enhancement layer code stream of the current frame, send it to the second enhancement layer decoder for decoding, and obtain the corresponding second enhancement layer reconstructed image in the LTU.

[0431] After obtaining the current frame EL code stream, the information that the current EL is the residual pixel layer EL can be parsed from the code stream, thereby instructing the current layer code stream to be sent to the decoder of the residual pixel layer EL for processing.

[0432] This step is similar to step 1 on the decoder side of Example 1, except that the previous layer is the transform domain coefficient layer. After the bitstream is fed into the decoder, entropy decoding is performed to obtain the partitioning information for the current layer LTU (the same size as the LCU at the corresponding position of the BL) and the residual information of the LTU. The partitioning of this LTU may differ from the LCU at the corresponding position of the BL or the LTU at the corresponding position of the first EL layer. Based on the LTU partitioning, the residual information is dequantized and inversely transformed to obtain a reconstructed residual image corresponding to the LTU. This image is added to the reconstructed image of the transform domain coefficient layer corresponding to the LTU to obtain the reconstructed image of the second EL layer corresponding to the LTU.

[0433] In this step, if the decoding obtains information that post-processing (such as loop filtering) needs to be performed on the reconstructed image, the corresponding post-processing process is performed after the reconstructed image to make it correspond to the process in the encoding.

[0434] Step 4: The same as step 3 at the decoding end of embodiment 1.

[0435] Step 5: The same as step 4 at the decoding end of embodiment 1.

[0436] The steps in this embodiment are based on LCU-level processing of each frame image. Since the LCU image is a subset of the entire frame image, all steps can also be processed in an entire frame manner, and the processing process is consistent.

[0437] It should be noted that after receiving the EL code stream, the decoder can send the EL layer to the corresponding decoder according to the information indicated in the code stream as to whether the current layer is a residual pixel enhancement layer or a transform domain coefficient layer. Therefore, the residual pixel layer and the transform domain coefficient layer can be interchanged at any EL layer. When decoding each EL layer, the decoder needs to retain the TU dequantization coefficients or reconstructed dequantization coefficients of the current EL for use in the next transform domain coefficient layer; if the next layer is a residual pixel layer, the TU dequantization coefficients or reconstructed dequantization coefficients retained by the EL layers above this layer can be reset or eliminated.

[0438] In this embodiment, each image may be implemented in the form of an image block.

[0439] In this embodiment, each residual may be implemented in the form of a residual block.

[0440] The execution order of the steps in this embodiment can be adjusted.

[0441] Example 3

[0442] Figure 16This is an exemplary flowchart of the method for encoding the enhancement layer of the present application, as shown in FIG. Figure 16 As shown, this embodiment is a hybrid hierarchical framework based on four enhancement layers, which is an extension of embodiment 2. In this embodiment, the first and third enhancement layers are transform domain coefficient layers, and the second and fourth enhancement layers are residual pixel layers.

[0443] Encoding side:

[0444] Steps 1 to 5 are the same as Steps 1 to 5 of the encoding end in Example 2.

[0445] Before executing step 6, save the transform coefficients of each TU in the second EL layer and replace them with the reference transform coefficients in step 1. Save the dequantized coefficients of the TU. Because the difference between the current source image and the second EL layer reconstructed image corresponding to the LTU is calculated, the second EL layer reconstructed image is updated to the reference image in that LTU.

[0446] Step 6: The same as step 2 at the encoding end in embodiment 2.

[0447] Step 7: Same as step 3 in Example 2.

[0448] Step 8: Same as step 4 in embodiment 2.

[0449] Step nine: The same as step five in embodiment two.

[0450] Step 10: Following steps 1 to 9, all LCUs of the entire source image are completed to obtain the BL reconstructed image of the entire image and the EL reconstructed image of each layer, as well as the BL code stream of the entire image and the EL code stream of each layer. These reconstructed images can be placed in the reference frame list for reference by subsequent frame images in the time domain. After obtaining the BL or each layer EL reconstructed image, further post-processing (such as loop filtering) can be performed and then placed in the reference frame list after processing, which is not limited here. If post-processing is performed, the instruction information for performing post-processing and then placing it in the reference frame list must be encoded into the corresponding code stream.

[0451] Information indicating whether the current EL type is the residual pixel layer EL or the transform domain coefficient layer EL needs to be encoded into the bitstream so that the decoding end can parse and obtain the current EL type and perform correct decoding processing.

[0452] Step 11: Repeat steps 1 to 10 to encode subsequent frames of the image sequence until the encoding of the entire image sequence is completed.

[0453] The steps in this embodiment are based on LCU-level processing of each frame image. Since the LCU image is a subset of the entire frame image, all steps can also be processed in a frame-level manner, and the processing process is consistent.

[0454] Example 4

[0455] Figure 17 This is an exemplary flowchart of the method for encoding the enhancement layer of the present application, as shown in FIG. Figure 17 As shown, this embodiment is a hybrid hierarchical framework based on three enhancement layers, which is an extension of the first embodiment.

[0456] Encoding side:

[0457] Steps 1 to 5: are the same as Steps 1 to 5 of the encoding end in Example 1.

[0458] Before executing step 6, retain the reconstructed dequantization coefficient corresponding to the TU in the second layer EL (transform domain coefficient layer EL) in step 5; the reference transform coefficient of each TU remains unchanged, which is the transform coefficient of the TU in the first layer EL (residual pixel layer EL); the reference image in the LCU at the corresponding position of the LTU remains unchanged, which is the BL reconstructed image corresponding to the LCU.

[0459] Step 6: Subtract the reference transform coefficient of each TU in the LTU from the reconstructed dequantized coefficient corresponding to the TU to obtain the coefficient residual of the TU.

[0460] Step 7: The residual of the TU coefficient obtained in step 6 is sent to the transform domain coefficient layer EL (third layer EL) encoder for encoding.

[0461] This step is similar to step five in Example 1, except that the previous EL layer is the transform domain coefficient layer EL. The coefficient residuals of each TU in the LTU are quantized using the same LTU partitioning scheme as the residual pixel layer EL (first EL layer) to obtain quantized coefficients. These coefficients are entropy encoded along with the LTU partitioning information to obtain the transform domain coefficient layer EL codestream for that LTU. Furthermore, the quantized coefficient residuals are inversely quantized to obtain dequantized coefficients of the coefficient residuals. These coefficients are then superimposed with the reconstructed dequantized coefficients of the previous transform domain coefficient layer EL (second EL layer) to obtain the reconstructed dequantized coefficients of the current EL. These reconstructed dequantized coefficients are inversely transformed to obtain a reconstructed residual image for that LTU. This image is then superimposed with the reference image in the LCU corresponding to the LTU to obtain a reconstructed image for the transform domain coefficient layer EL corresponding to that LTU.

[0462] Step 8: Following steps 1 to 7, all LCUs of the entire source image are completed, resulting in the BL reconstructed image for the entire image and the EL reconstructed image for each layer, as well as the BL codestream for the entire image and the EL codestream for each layer. These reconstructed images can be placed in a reference frame list for reference by subsequent frame images in the time domain. After obtaining the BL or EL reconstructed image for each layer, further post-processing (such as loop filtering) can be performed, and then placed in the reference frame list after processing, which is not limited here. If post-processing is performed, the indication information for performing post-processing and placing it in the reference frame list must be encoded into the corresponding codestream.

[0463] Here, information indicating whether the current EL type is the residual pixel layer EL or the transform domain coefficient layer EL needs to be encoded into the bitstream so that the decoding end can parse and obtain the current EL type and perform correct decoding processing.

[0464] Step 9: Repeat steps 1 to 8 to encode subsequent frames of the image sequence until the encoding of the entire image sequence is completed.

[0465] The steps in this embodiment are based on LCU-level processing of each frame image. Since the LCU image is a subset of the entire frame image, all steps can also be processed in a frame-level manner, and the processing process is consistent.

[0466] In this embodiment, each image can be implemented in the form of an image block.

[0467] In this embodiment, each residual may be implemented in the form of a residual block.

[0468] The execution order of the steps in this embodiment can be adjusted.

[0469] Example 5

[0470] Figure 18 This is an exemplary flowchart of the method for encoding the enhancement layer of the present application, as shown in FIG. Figure 18 As shown, this embodiment is a hybrid hierarchical framework based on three enhancement layers, wherein the third enhancement layer is a residual pixel layer, which is an extension of the first embodiment.

[0471] Encoding side:

[0472] Steps 1 to 5: are the same as Steps 1 to 5 of the encoding end in Example 1.

[0473] Step 6: The same as step 4 of the encoding end in embodiment 2.

[0474] Step 7: The same as step 5 of the encoding end in embodiment 2.

[0475] Steps 8 to 9 are the same as steps 8 to 9 of the encoding end in the fourth embodiment.

[0476] Example 6

[0477] The above embodiments are all hybrid grading frameworks based on quality domain grading. Figure 19a and Figure 19b This embodiment provides a hybrid hierarchical coding framework for spatial scalability based on the BL and the residual pixel layer EL, which is an extension of Embodiment 5. In this framework, the image resolutions of the BL and the residual pixel layer EL can be different.

[0478] Before the source image is fed into the BL or residual pixel layer EL encoder, it is downsampled to generate an image with a lower resolution than the source image. The resolution of the downsampled image fed into the BL is lower than the resolution of the downsampled image fed into the residual pixel layer EL. The resolution of the downsampled image fed into the lower residual pixel layer EL is lower than the resolution of the downsampled image fed into the higher residual pixel layer EL. The highest residual pixel layer EL can also be directly input using the source image without downsampling. The resolution of the processed image in the transform domain coefficient layer EL is the same as that of the BL or the previous residual pixel layer EL. Before processing the residual pixel layer EL encoding, the reconstructed image of the previous layer is obtained and then upsampled to generate an upsampled image with the same resolution as the downsampled source image input to the next residual pixel layer EL layer. The residual image of the two is then subjected to residual pixel layer EL encoding in the EL encoder in the same manner as the residual pixel layer EL encoding in the above embodiment.

[0479] Encoding side:

[0480] Step 1: Downsample the source image to image resolution 1 (image width and height are W1xH1) to generate downsampled image 1. Here, resolution 1 is smaller than the source image resolution. The sampling method can be any method, such as linear interpolation, bicubic interpolation, etc., which is not limited here.

[0481] Step 2: Send the downsampled image 1 to the BL encoder for encoding. The steps are the same as step 1 in embodiment 1.

[0482] Step 3: Downsample the source image to image resolution 2 (image width and height are W2xH2) to generate downsampled image 2, where resolution 2 is higher than resolution 1. The scaling ratio of downsampled image 2 to image 1 is Sw1 = W2 / W1 in width and Sh1 = H2 / H1 in height.

[0483] Step 4: Upsample the BL reconstructed image in the LCU to obtain a reconstructed image of the same size as the LTU of the first layer EL, and then obtain the residual image of the first layer EL (residual pixel layer EL) corresponding to the LTU.

[0484] Set the LTU size of the first residual pixel layer EL to the LCU size of the BL multiplied by the scaling ratio Sw1xSh1 to obtain an LTU of size Lw1xLh1, where Lw1=L*Sw1, Lh1=L*Sh1. Upsample the BL reconstructed image corresponding to the LCU obtained in step 2 to the size of Lw1xLh1 to obtain a BL reconstructed image with the same size as the LTU of the first residual pixel layer EL. Take the LTU source image at the same position in the downsampled image 2. Subtract the above source image from the above BL reconstructed image to obtain the residual image in the LTU of the first layer EL. Here, the upsampling method may or may not correspond to the downsampling method in step 1, and there is no limitation here. Since the LCU corresponds to the difference between the current source image and the BL reconstructed image, the image after upsampling the BL reconstructed image is used as the reference image in the LCU.

[0485] Step 5: Send the residual image in the LTU of the first EL layer to the residual pixel layer EL encoder for residual encoding. The encoding steps are the same as step 3 of embodiment 1.

[0486] Step 6: The same as step 4 on the encoding side in embodiment 1.

[0487] Step 7: The same as step 5 on the encoding side of embodiment 1.

[0488] Step 8: Downsample the source image to image resolution 3 (image width and height are W3×H3) to generate downsampled image 3, where resolution 3 is higher than resolution 2. The scaling ratio of downsampled image 3 to downsampled image 2 is Sw2=W3 / W2 in width and Sh2=H3 / H2 in height.

[0489] Step 9: Upsample the LTU reconstructed image of the second layer EL (transform domain coefficient layer EL) obtained in step 7 to obtain a reconstructed image of the same size as the LTU of the third layer EL (residual pixel layer EL), and then obtain the residual image corresponding to the LTU of the third layer EL.

[0490] This step is similar to step 4. The LTU size of the third EL layer is set to the LTU size of the second EL layer multiplied by the scaling ratio Sw2xSh2. The reconstructed image in the LTU of the second EL layer is upsampled to obtain a reconstructed image of the same size as the LTU of the third EL layer. The LTU source image at the same position in the downsampled image 3 is taken. The residual image in the LTU of the third EL layer is obtained by subtracting the source image from the reconstructed image.

[0491] Step 10: Send the residual image in the LTU of the third EL layer to the residual pixel layer EL encoder for residual encoding. The encoding step is the same as step 3 of the encoding end in embodiment 1.

[0492] Step 11: Complete all LCUs of the entire source image according to Steps 1 through 10, similar to Step 6 on the encoder side. In addition, the resolution information of the downsampled image input at each layer must be encoded into the bitstream for decoding and processing of all layers. Furthermore, information about the upsampling method for each layer can also be encoded into the bitstream.

[0493] Step 12: Repeat steps 1 to 11 to encode subsequent frames of the image sequence until the encoding of the entire image sequence is completed.

[0494] The steps in this embodiment are based on LCU-level processing of each frame image. Since the LCU image is a subset of the entire frame image, all steps can also be processed in a frame-level manner, and the processing process is consistent.

[0495] Decoding end:

[0496] Step 1: Similar to Step 1 on the decoding side of Example 1, except that before decoding, the residual image resolution information corresponding to the current EL carried in the first-layer EL codestream is parsed, and EL decoding of the residual pixel layer is performed according to the resolution of the current EL. Based on the resolution of the EL-reconstructed residual image of that layer, the obtained BL reconstructed image corresponding to the LCU is upsampled to the same resolution as the EL-reconstructed residual image corresponding to the LTU of that layer. The two are then superimposed to obtain the first-layer EL reconstructed image corresponding to the LTU. The upsampled BL reconstructed image serves as the reference image for the LTU.

[0497] Step 2: The same as step 2 of the decoding end in embodiment 1.

[0498] Step 3: Similar to Step 1 on the decoding side of Example 1, except that before decoding, the residual image resolution information corresponding to the current EL carried in the third-layer EL codestream is parsed, and EL decoding of the residual pixel layer is performed according to the resolution of the current EL. Based on the resolution of the EL reconstructed residual image of this layer, the second-layer EL reconstructed image corresponding to the LTU obtained in Step 2 is upsampled to the same resolution as the EL reconstructed residual image corresponding to the LTU of this layer. The two are then superimposed to obtain the third-layer EL reconstructed image corresponding to the LTU.

[0499] Step 4: Repeat steps 1 through 3 until the BL, residual pixel layer (EL), and transform domain coefficient layer (EL) processing is complete for all LCUs corresponding to the entire frame. This yields the BL reconstructed image, residual pixel layer (EL) reconstructed image, and transform domain coefficient layer (EL) reconstructed image for the current frame. The BL reconstructed image and each EL layer reconstructed image can be placed in a reference frame list for reference by subsequent frames in the temporal domain.

[0500] If the code stream contains information indicating that the reconstructed image should be post-processed (such as loop filtering) before being placed in the reference frame list, this information can be obtained through entropy decoding, and the BL reconstructed image and the EL reconstructed image can be post-processed and then placed in the reference frame list to correspond to the process in the encoding.

[0501] Step 5: Repeat steps 1 to 4 until the entire image sequence is decoded.

[0502] In this embodiment, each image can be implemented in the form of an image block.

[0503] In this embodiment, each residual may be implemented in the form of a residual block.

[0504] The execution order of the steps in this embodiment can be adjusted.

[0505] Example 7

[0506] This embodiment is an extension of the first embodiment, describing a process for processing only an image region of interest (ROI) at the residual pixel layer EL or the transform domain coefficient layer EL. An image ROI refers to a specific image region that requires processing. In practice, ROI-based processing is not limited to the first embodiment; the encoding processing method can be used in the previously described embodiments. This example only illustrates the steps added to the first embodiment.

[0507] Encoding side:

[0508] Step 1: Same as step 1 of the encoding end in Example 1

[0509] Step 2: Determine the image region of interest (ROI) corresponding to the first layer EL, and process only the LCU (or LTU) image within the ROI range corresponding to the first layer EL. The processing process is the same as step 2 of the encoding end in embodiment 1.

[0510] Step 3: The same as step 3 of the encoding end of embodiment 1, except that only the LCU image within the ROI range corresponding to the first layer EL is processed. The position and size information of the ROI area corresponding to the first layer EL can be encoded into the corresponding bitstream.

[0511] Step 4: Determine the image ROI corresponding to the second layer EL, and process only the LCU (or LTU) image within the ROI range corresponding to the second layer EL. The processing process is the same as step 4 of the encoding end in embodiment 1.

[0512] Step 5: The same as step 5 of the encoding end of embodiment 1, except that only the LCU image within the ROI range corresponding to the second layer EL is processed. The position and size information of the ROI area corresponding to the second layer EL can be encoded into the corresponding bitstream.

[0513] Step 6 and Step 7: are the same as Step 6 and Step 7 at the encoding end in the first embodiment.

[0514] Decoding end:

[0515] Step 1: The same as step 1 on the decoding end of embodiment 1, except that the decoded information of the first-layer EL code stream corresponds to the ROI area corresponding to the first-layer EL, so the processing is limited to the LCU (or LTU) image in the ROI area corresponding to the first-layer EL.

[0516] If the bitstream contains the position and size information of the area corresponding to the EL image, the information is first decoded and then the area is found in the previous layer reconstructed image based on the information to obtain the reconstructed image to be processed.

[0517] Step 2: The same as step 2 at the decoding end of embodiment 1, except that the decoded information of the second-layer EL code stream corresponds to the ROI area corresponding to the second-layer EL, so the processing is limited to the LCU (or LTU) image in the ROI area corresponding to the second-layer EL.

[0518] Similarly, if the bitstream contains the position and size information of the area corresponding to the EL image, the information is first decoded to obtain the information, and the area is found in the previous layer reconstructed image based on the information to obtain the reconstructed image to be processed.

[0519] Step 3 and Step 4: are the same as Step 3 and Step 4 in the decoding end of the first embodiment.

[0520] In this embodiment, each image can be implemented in the form of an image block.

[0521] In this embodiment, each residual may be implemented in the form of a residual block.

[0522] The execution order of the steps in this embodiment can be adjusted.

[0523] Example 8

[0524] This embodiment uses quality domain grading to hierarchically classify image blocks to obtain at least three enhancement layers, where the image blocks are LCUs. A coding and decoding method is provided when the enhancement layer comprises at least two adjacent transform domain coefficient layers, where the two adjacent transform domain coefficient layers are a low transform domain coefficient layer and a high transform domain coefficient layer. In one possibility, the low transform domain coefficient layer and the high transform domain coefficient layer in this embodiment correspond to the first enhancement layer and the second enhancement layer. In another possibility, this embodiment may be an extension of Embodiment 1 or Embodiment 2.

[0525] Encoding side:

[0526] Step 1: The encoder obtains the original image frame and encodes the lower-layer image blocks according to the LCU to obtain the lower-layer code stream of the LCU. The lower-layer of the LCU can be the base layer or the residual pixel layer.

[0527] Step 2: The encoder calculates the difference between the corresponding pixel points of the transform coefficients of the next lower layer of the LCU and the dequantized coefficients of the next lower layer of the LCU to obtain a coefficient residual block of the lower transform domain coefficient layer of the LCU. The dequantized coefficients of the next lower layer of the LCU are obtained after quantization and inverse quantization of the transform coefficients of the next lower layer of the LCU.

[0528] Step 3: Obtain the reconstructed dequantized coefficients of the low transform domain coefficient layer of the LCU. The method for obtaining the reconstructed dequantized coefficients of the low transform domain coefficient layer of the LCU is as follows: after performing a quantization and dequantization process on the coefficient residual block of the low transform domain coefficient layer of the LCU, the dequantized coefficients of the low transform domain coefficient layer of the LCU are obtained; after performing a quantization and dequantization process on the transform coefficients of the lower layer of the LCU, the dequantized coefficients of the lower layer of the LCU are obtained. The dequantized coefficients of the low transform domain coefficients of the LCU and the corresponding pixel points of the dequantized coefficients of the lower layer are summed to obtain the reconstructed dequantized coefficients of the low transform domain coefficient layer of the LCU.

[0529] Step 4: The encoder calculates the difference between the corresponding pixel points of the transform coefficients of the lower layer of the LCU and the reconstructed dequantized coefficients of the low transform domain coefficient layer of the LCU to obtain a coefficient residual block of the high transform domain of the LCU.

[0530] Step 5: Send the coefficient residual block of the low transform domain coefficient layer of the LCU to the low transform domain coefficient layer encoder for residual coding.

[0531] Step 6: Send the coefficient residual block of the high transform domain coefficient layer of the LCU to the high transform domain coefficient layer encoder for residual coding.

[0532] Step 7: Repeat steps 1 to 6 to complete all LCUs of the entire source image, and obtain the lower layer code stream, the low transform domain coefficient layer code stream, and the high transform domain coefficient layer code stream of the entire image.

[0533] Step 8: Repeat steps 1 to 7 to encode subsequent frames of the image sequence until the encoding of the entire image sequence is completed.

[0534] Decoding end:

[0535] Step 1: Obtain the reconstructed residual block of the low transform domain coefficient layer.

[0536] After the decoder obtains the bitstream of the low transform domain coefficient layer, the decoder of the low transform domain coefficient layer performs entropy decoding and inverse quantization based on the syntax elements carried in the bitstream of the low transform domain coefficient layer to obtain the dequantized coefficients of the low transform domain coefficient layer. After obtaining the bitstream of the next lower layer, the decoder of the next lower layer performs entropy decoding and inverse quantization based on the syntax elements carried in the bitstream of the next lower layer to obtain the dequantized coefficients of the next lower layer. The dequantized coefficients of the low transform domain coefficient layer and the corresponding pixels in the dequantized coefficients of the next lower layer are summed and inverse transformed to obtain the reconstructed residual block of the low transform domain coefficient layer.

[0537] In this step, the lower layer decoder uses a decoder that can decode the lower layer stream corresponding to the lower layer stream encoding rules. Here, a decoder that complies with the H.264 / H.265 / H.266 standard or other non-standard video decoders can be used for decoding, which is not limited here.

[0538] Step 2: Obtain a reconstructed block of the low transform domain coefficient layer by summing corresponding pixels of the reconstructed residual block of the low transform domain coefficient layer and the prediction block of the base layer or the reconstructed block of the layer lower than the third layer.

[0539] In this step, the third layer is a base layer, residual pixel enhancement layer, or transform domain coefficient layer that is lower than the low transform domain coefficient layer. If a residual pixel layer exists that is lower than the low transform domain coefficient layer, such as when the first residual pixel layer is the first enhancement layer, the second residual pixel layer is the second enhancement layer, the low transform domain coefficient layer is the third enhancement layer, and the high transform domain coefficient layer is the fourth enhancement layer, the second residual pixel layer is a layer lower than the low transform domain coefficient layer and is the residual pixel layer closest to the low transform domain coefficient layer. In this case, the layer below the residual pixel layer closest to the low transform domain coefficient layer (the second residual pixel layer) is used as the fourth layer (which is the base layer or enhancement layer). The reconstructed block of the low transform domain coefficient layer is obtained by summing the corresponding pixels of the reconstructed residual block of the low transform domain coefficient layer and the reconstructed block of the fourth layer. If no residual pixel layer exists that is lower than the low transform domain coefficient layer, the reconstructed block of the low transform domain coefficient layer is obtained by summing the corresponding pixels of the reconstructed residual block of the low transform domain coefficient layer and the prediction block of the base layer.

[0540] In this step, if the decoding obtains information that post-processing (such as loop filtering) needs to be performed in the reconstructed block, the corresponding post-processing process is performed after the reconstructed block to make it correspond to the process in the encoding.

[0541] After completing this step, the obtained reconstructed block is a low transform domain coefficient layer reconstructed block.

[0542] Step 3: Obtain the reconstructed residual block of the high transform domain coefficient layer.

[0543] After the decoder obtains the bitstream of the high transform domain coefficient layer, the decoder of the high transform domain coefficient layer performs entropy decoding and inverse quantization based on the syntax elements carried in the bitstream of the high transform domain coefficient layer to obtain the dequantized coefficients of the high transform domain coefficient layer. The dequantized coefficients of the low transform domain coefficient layer and the corresponding pixels in the dequantized coefficients of the next lower layer are summed to obtain the reconstructed dequantized coefficients of the low transform domain coefficient layer. The reconstructed dequantized coefficients of the low transform domain coefficient layer and the corresponding pixels in the dequantized coefficients of the high transform domain coefficient layer are summed and inversely transformed to obtain the reconstructed residual block of the high transform domain coefficient layer.

[0544] In this step, if the decoding obtains information that post-processing (such as loop filtering) needs to be performed in the reconstructed block, the indicated post-processing process is performed after the reconstructed block so that it corresponds to the process in the encoding.

[0545] After completing this step, the residual pixel layer closest to the low transform domain coefficient layer is used as the fourth layer (this layer is the base layer or enhancement layer), and the reconstructed block of the high transform domain coefficient layer is obtained by summing the corresponding pixels of the reconstructed residual block of the high transform domain coefficient layer and the reconstructed block of the fourth layer. If there is no residual pixel layer lower than the low transform domain coefficient layer, the reconstructed block of the high transform domain coefficient layer is obtained by summing the corresponding pixels of the reconstructed residual block of the high transform domain coefficient layer and the prediction block of the base layer.

[0546] Step 4: Repeat steps 1 to 3 until the entire image sequence is decoded.

[0547] In this embodiment, each image can be implemented in the form of an image block.

[0548] In this embodiment, each residual may be implemented in the form of a residual block.

[0549] The execution order of the steps in this embodiment can be adjusted.

[0550] Embodiment 9

[0551] This embodiment uses quality domain grading to hierarchically classify image blocks to obtain at least three enhancement layers, where the image blocks are LCUs. A coding and decoding processing method is provided when the enhancement layer has at least two adjacent residual pixel layers, where the two adjacent residual pixel layers are a low residual pixel layer and a high residual pixel layer. In one possibility, the low residual pixel layer and the high residual pixel layer in this embodiment correspond to the first enhancement layer and the second enhancement layer. In another possibility, this embodiment may also be an extension of Embodiment 1 or Embodiment 2.

[0552] Encoding side:

[0553] Step 1: The encoder obtains the original image frame and encodes the lower-layer image blocks according to the LCU to obtain the reconstructed blocks of the lower-layer LCU and the lower-layer code stream. The lower-layer LCU can be the base layer, the residual pixel layer, or the transform domain coefficient layer.

[0554] Step 2: The encoder obtains the code stream and reconstructed block of the low residual pixel layer of the LCU.

[0555] The reconstructed block of the low residual pixel layer of the LCU is obtained by taking the difference between the image block corresponding to the LCU and the corresponding pixel points in the reconstructed block of the next lower layer to obtain the residual block of the low residual pixel layer of the LCU. The residual block of the low residual pixel layer of the LCU is transformed, quantized, inversely quantized, and inversely transformed, and then summed with the corresponding pixel points in the reconstructed block of the next lower layer to obtain the reconstructed block of the low residual pixel layer. The residual block of the low residual pixel layer of the LCU is transformed, quantized, and entropy coded to obtain the code stream of the low residual pixel layer of the LCU.

[0556] Step 3: The encoder calculates the difference between the image block corresponding to the LCU and the corresponding pixel points in the reconstructed block of the low residual pixel layer to obtain the residual block of the high residual pixel layer of the LCU.

[0557] Optionally, before executing step 2, the encoder may further post-process the reconstructed blocks of the lower layer of the LCU (e.g., loop filtering). Whether or not to perform post-processing is not specified. If post-processing is performed, information indicating that post-processing is required is added to the lower layer bitstream.

[0558] Step 4: The encoder obtains the code stream and reconstructed block of the high residual pixel layer of the LCU.

[0559] The residual block of the high residual pixel layer of the LCU is transformed, quantized, inversely quantized, and inversely transformed, and then summed with the corresponding pixel points in the reconstructed block of the low residual pixel layer to obtain the reconstructed block of the high residual pixel layer. The residual block of the high residual pixel layer of the LCU is transformed, quantized, and entropy coded to obtain the code stream of the high residual pixel layer of the LCU.

[0560] Optionally, before executing step 4, the encoder may further post-process the reconstructed blocks of the low residual pixel layer of the LCU (e.g., loop filtering). Whether or not to perform post-processing is specified. If post-processing is performed, information indicating that post-processing is required is added to the low residual pixel layer bitstream.

[0561] Here, the superimposed image can also be further post-processed (such as loop filtering) and then used as the reconstructed block of the high residual pixel layer in the LCU. There is no restriction on whether to perform post-processing. If post-processing is performed, information indicating that post-processing is required must be added to the high residual pixel layer bitstream.

[0562] Step 5: Repeat steps 1 to 4 to complete all LCUs of the entire source image, and obtain the reconstructed blocks of the continuous residual pixel layer of the entire image, as well as the code stream of the continuous residual pixel layer of the entire image. These reconstructed blocks can be placed in the reference frame list for reference by subsequent frame images in the time domain. After obtaining the reconstructed blocks of the continuous residual pixel layer, further post-processing (such as loop filtering) can be performed, and then placed in the reference frame list after processing, which is not limited here. If post-processing is performed, the instruction information for performing post-processing and then placing it in the reference frame list needs to be encoded into the corresponding code stream.

[0563] Here, information indicating that the residual pixel layer is no longer divided according to the LCU or CU of the base layer, or information indicating that the high residual pixel layer is no longer divided according to the LCU of the base layer or the LTU of the low residual pixel layer can also be encoded into the bitstream for the decoding end to perform enhancement layer processing after decoding.

[0564] Step 6: Repeat steps 1 to 5 to encode subsequent frames of the image sequence until the encoding of the entire image sequence is completed.

[0565] Decoding end:

[0566] Step 1: Obtain the reconstructed block of the low residual pixel layer.

[0567] After the decoding end obtains the code stream of the low residual pixel layer of the image block to be decoded, the decoder of the low residual pixel layer performs entropy decoding, inverse quantization and inverse transformation on the code stream of the low residual pixel layer based on the syntax elements carried in the code stream of the low residual pixel layer to obtain a reconstructed residual block of the low residual pixel layer. After the decoding end obtains the code stream of the lower layer, the decoder of the lower layer decodes the code stream of the lower layer based on the syntax elements carried in the code stream of the lower layer to obtain a reconstructed block of the lower layer. The decoder obtains the reconstructed block of the low residual pixel layer by summing the reconstructed residual block of the low residual pixel layer and the corresponding pixel points in the reconstructed block of the lower layer.

[0568] In this step, the decoder of the low residual pixel layer and the lower layer decoder use decoders that can decode the lower layer streams corresponding to the coding rules of the low residual pixel layer and the lower layer streams. A decoder that complies with the H.264 / H.265 / H.266 standard can be used here, or other non-standard video decoders can be used for decoding, which is not limited here.

[0569] In this step, if the decoding obtains information that post-processing (such as loop filtering) needs to be performed in the reconstructed block, the corresponding post-processing process is performed after the reconstructed block to make it correspond to the process in the encoding.

[0570] After completing this step, the obtained reconstructed block is a low residual pixel layer reconstructed block.

[0571] Step 2: After the decoding end obtains the code stream of the high residual pixel layer, it is sent to the high residual pixel layer decoder for decoding to obtain the reconstructed residual block of the high residual pixel layer.

[0572] This step is similar to step 1, except that the reconstructed block of the low residual pixel layer is added to the reconstructed residual block of the high residual pixel layer to obtain the reconstructed block of the high residual pixel layer.

[0573] In this step, if the decoding obtains information that post-processing (such as loop filtering) needs to be performed in the high residual pixel layer reconstruction block, the indicated post-processing process is performed after the high residual pixel layer reconstruction block to make it correspond to the process in the encoding.

[0574] After completing this step, the obtained reconstructed block is a high residual pixel layer reconstructed block.

[0575] Step 3: Repeat steps 1 and 2 until the entire image sequence is decoded.

[0576] In this embodiment, each image can be implemented in the form of an image block.

[0577] In this embodiment, each residual may be implemented in the form of a residual block.

[0578] The execution order of the steps in this embodiment can be adjusted.

[0579] Figure 20 This is an exemplary structural diagram of the encoding device of this application, as shown in FIG. Figure 20 As shown, the apparatus of this embodiment may correspond to the video encoder 20. The apparatus may include: an obtaining module 1601, a determining module 1602 and an encoding module 1603.

[0580] In one possible implementation, an obtaining module 1601 is configured to obtain a reconstructed block of a base layer of an image block to be encoded; perform a difference calculation between corresponding pixels in the image block to be encoded and the reconstructed block of the base layer to obtain a residual block of a first enhancement layer of the image block to be encoded, wherein the resolution of the first enhancement layer image block is not lower than the resolution of the base layer image block, or the coding quality of the first enhancement layer image block is not lower than the coding quality of the base layer image block; a determining module 1602 is configured to obtain a transform block of the first enhancement layer based on the residual block of the first enhancement layer, transform the transform block of the first enhancement layer to obtain transform coefficients of the enhancement layer, and quantize and dequantize the transform coefficients of the first enhancement layer to obtain dequantized coefficients of the enhancement layer; the obtaining module is further configured to perform a difference calculation between corresponding pixels of the transform coefficients of the first enhancement layer and the dequantized coefficients of the first enhancement layer to obtain a coefficient residual block of a second enhancement layer of the image block to be encoded; and an encoding module 1603 is configured to quantize and entropy encode the transform coefficients of the first enhancement layer to obtain a code stream of the first enhancement layer. The coefficient residual block of the second enhancement layer is also configured to quantize and entropy encode the coefficients of the second enhancement layer to obtain a code stream of the second enhancement layer.

[0581] In one possible implementation, the determination module 1602 is specifically used to iteratively divide the first maximum transformation unit LTU of the residual block of the enhancement layer into a tree structure to obtain transformation units TU of multiple segmentation depths, where the maximum segmentation depth among the multiple segmentation depths is equal to Dmax, and Dmax is a positive integer; the division method of the first TU is determined according to the loss estimation value of the first TU and the sum of the loss estimation values ​​of multiple second TUs contained in the first TU, the segmentation depth of the first TU is i, the depth of the second TU is i+1, and 0≤i≤Dmax-1.

[0582] In one possible implementation, the determination module 1602 is specifically used to transform and quantize the TU to obtain the dequantization coefficient of the TU, where the TU is the first TU or the second TU; pre-encode the dequantization coefficient of the TU to obtain the codeword length of the TU; inverse quantize and inverse transform the dequantization coefficient of the TU to obtain the reconstructed block of the TU; calculate the sum of squares of errors SSD of the TU and the reconstructed block of the TU to obtain the distortion value of the TU; obtain the loss estimate of the TU based on the codeword length of the TU and the distortion value of the TU; determine the smaller of the loss estimate of the first TU and the sum of the loss estimate values ​​of the multiple second TUs; and determine the division method corresponding to the smaller one as the division method of the first TU.

[0583] In a possible implementation, the size of the LTU is the same as the size of the reconstructed block of the base layer; or, the size of the LTU is the same as the size of the original reconstructed block of the base layer after upsampling.

[0584] In a possible implementation, the image block to be encoded refers to the largest coding unit LCU in the entire frame image; or, the image block to be encoded refers to the entire frame image; or, the image block to be encoded refers to the region of interest ROI in the entire frame image.

[0585] In a possible implementation manner, the base layer and the enhancement layer are obtained in layers based on resolution or coding quality.

[0586] In one possible implementation, when the base layer and the enhancement layer are obtained based on resolution layering, the acquisition module 1601 is further used to downsample the original image block to be encoded to obtain the image block to be encoded of the first resolution; and upsample the original reconstructed block of the base layer to obtain the reconstructed block of the base layer of the first resolution.

[0587] In one possible implementation, the obtaining module 1601 is specifically configured to calculate the difference between corresponding pixel points in the image block to be encoded and the prediction block of the image block to be encoded...

Claims

1. A method for encoding an image block, characterized in that: include: Obtain a reconstructed block of a base layer of an image block to be encoded; Differences are calculated between corresponding pixels in the image block to be encoded and the reconstructed block of the base layer to obtain a residual block of a first enhancement layer of the image block to be encoded; Obtaining a transform block of the first enhancement layer according to the residual block of the first enhancement layer, transforming the transform block of the first enhancement layer to obtain transform coefficients of the first enhancement layer, and quantizing and dequantizing the transform coefficients of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; quantizing and entropy encoding the transform coefficients of the first enhancement layer to obtain a bitstream of the first enhancement layer; Taking a difference between corresponding pixel points of the transform coefficients of the first enhancement layer and the dequantized coefficients of the first enhancement layer to obtain a coefficient residual block of the second enhancement layer of the image block to be encoded; as well as The coefficient residual block of the second enhancement layer is quantized and entropy encoded to obtain a code stream of the second enhancement layer.

2. The method according to claim 1, characterized in that The method further comprises: quantizing and dequantizing the coefficient residual block of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer; Summing corresponding pixel points of the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the first enhancement layer to obtain reconstructed dequantized coefficients of the second enhancement layer; Performing an inverse transformation on the reconstructed dequantized coefficients of the second enhancement layer to obtain a reconstructed residual block of the second enhancement layer; The corresponding pixels in the reconstructed block of the base layer and the reconstructed residual block of the second enhancement layer are summed to obtain a reconstructed block of the second enhancement layer.

3. The method according to claim 2, characterized in that The method further comprises: Differences are calculated between corresponding pixels in the image block to be encoded and the reconstructed block of the second enhancement layer to obtain a residual block of the image block to be encoded on a third enhancement layer; Obtaining a transform block of the third enhancement layer according to the residual block of the third enhancement layer, and transforming the transform block of the third enhancement layer to obtain transform coefficients of the third enhancement layer; The transform coefficients of the third enhancement layer are quantized and entropy encoded to obtain a code stream of the third enhancement layer.

4. The method according to claim 3, characterized in that The method further comprises: quantizing and dequantizing the transform coefficients of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; Taking a difference between corresponding pixel points of the transform coefficients of the third enhancement layer and the dequantized coefficients of the third enhancement layer to obtain a coefficient residual block of the fourth enhancement layer of the image block to be encoded; The coefficient residual block of the fourth enhancement layer is quantized and entropy encoded to obtain a code stream of the fourth enhancement layer.

5. The method according to claim 1, wherein The method further comprises: quantizing and dequantizing the coefficient residual block of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer; Summing corresponding pixel points of the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the first enhancement layer to obtain reconstructed dequantized coefficients of the second enhancement layer; Differences are taken between corresponding pixel points of the transform coefficients of the first enhancement layer and the reconstructed dequantized coefficients of the second enhancement layer to obtain a coefficient residual block of a third enhancement layer of the image block to be encoded; The coefficient residual block of the third enhancement layer is quantized and entropy encoded to obtain a code stream of the third enhancement layer.

6. The method according to claim 5, characterized in that The method further comprises: quantizing and dequantizing the coefficient residual block of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; Summing the dequantized coefficients of the third enhancement layer and the corresponding pixel points of the reconstructed dequantized coefficients of the second enhancement layer and then inversely transforming them to obtain a reconstructed residual block of the third enhancement layer; Summing corresponding pixels in the reconstructed residual block of the third enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the third enhancement layer; Differences are calculated between corresponding pixels in the image block to be encoded and the reconstructed block on the third enhancement layer to obtain a residual block on a fourth enhancement layer of the image block to be encoded; Obtaining a transform block of the fourth enhancement layer according to the residual block of the fourth enhancement layer, and transforming the transform block of the fourth enhancement layer to obtain transform coefficients of the fourth enhancement layer; Quantizing and entropy encoding the transform coefficients of the fourth enhancement layer to obtain a code stream of the fourth enhancement layer.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Performing an inverse transformation on the dequantized coefficients of the first enhancement layer to obtain a reconstructed residual block of the first enhancement layer; The corresponding pixels in the reconstructed block of the base layer and the reconstructed residual block of the first enhancement layer are summed to obtain the reconstructed block of the first enhancement layer.

8. The method according to claim 7, characterized in that The resolution of the reconstructed block of the first enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the encoding quality of the reconstructed block of the first enhancement layer is not lower than the encoding quality of the reconstructed block of the base layer.

9. The method according to claim 7, characterized in that The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the first enhancement layer, or the encoding quality of the reconstructed block of the second enhancement layer is not lower than the encoding quality of the reconstructed block of the first enhancement layer.

10. The method according to any one of claims 1-6, 8-9, characterized in that The block division method of the coefficient residual block of the second enhancement layer is the same as the transform block division method of the residual block of the first enhancement layer.

11. The method according to any one of claims 1-6, 8-9, characterized in that The size of the maximum transformation unit (LTU) of the residual block of the first enhancement layer is the same as the size of the reconstructed block of the base layer; or the size of the LTU of the residual block of the first enhancement layer is the same as the size after upsampling the reconstructed block of the original base layer.

12. The method according to any one of claims 1-6, 8-9, characterized in that The image block to be encoded refers to the largest coding unit LCU in the entire frame image; or, the image block to be encoded refers to the entire frame image; or, the image block to be encoded refers to the region of interest ROI in the entire frame image.

13. The method according to any one of claims 1-6, 8-9, characterized in that The step of obtaining a reconstructed block of a base layer of an image block to be encoded comprises: Difference is calculated between corresponding pixels in the image block to be encoded and a prediction block of the image block to be encoded to obtain a residual block of the image block to be encoded; transforming, quantizing, and inverse-quantizing the residual block of the image block to be encoded to obtain a dequantized coefficient of the image block to be encoded; Performing an inverse transformation on the dequantized coefficients of the image block to be encoded to obtain a reconstructed residual block of the image block to be encoded; The corresponding pixel points in the prediction block of the image block to be encoded and the reconstructed residual block of the image block to be encoded are summed to obtain the reconstructed block of the base layer.

14. The method according to claim 13, characterized in that Summing corresponding pixels in the prediction block of the image block to be encoded and the reconstructed residual block of the image block to be encoded to obtain the reconstructed block of the base layer includes: Corresponding pixel points in the prediction block of the image block to be encoded and the reconstructed residual block of the image block to be encoded are summed and up-sampled to obtain a reconstructed block of the base layer.

15. A method for encoding an image block, characterized in that: include: Transforming, quantizing, and entropy coding the residual block of the base layer of the image block to be coded to obtain a code stream of the base layer; Transforming, quantizing, and inverse-quantizing the residual block of the base layer to obtain dequantized coefficients of the base layer; transforming the transform block of the base layer to obtain transform coefficients of the base layer; Differences are taken between corresponding pixel points of the transform coefficients of the base layer and the dequantized coefficients of the base layer to obtain a coefficient residual block of the first enhancement layer of the image block to be encoded; quantizing and entropy encoding the coefficient residual block of the first enhancement layer to obtain a code stream of the first enhancement layer; quantizing and dequantizing the coefficient residual block of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; Summing corresponding pixel points of the dequantized coefficients of the first enhancement layer and the dequantized coefficients of the base layer to obtain reconstructed dequantized coefficients of the first enhancement layer; Performing an inverse transformation on the reconstructed dequantized coefficients of the first enhancement layer to obtain a reconstructed residual block of the first enhancement layer; Summing the reconstructed residual block of the first enhancement layer and corresponding pixels in the prediction block of the base layer to obtain a reconstructed block of the first enhancement layer; Differences are calculated between corresponding pixels in the image block to be encoded and the reconstructed block of the first enhancement layer to obtain a residual block of the second enhancement layer of the image block to be encoded; A transform block of the second enhancement layer is obtained according to the residual block of the second enhancement layer, the transform block of the second enhancement layer is transformed to obtain transform coefficients of the second enhancement layer, and the transform coefficients of the second enhancement layer are quantized and entropy encoded to obtain a code stream of the second enhancement layer.

16. The method according to claim 15, characterized in that The method further comprises: quantizing and dequantizing transform coefficients of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer; Taking a difference between corresponding pixel points of a transform coefficient of a second enhancement layer of the image block to be encoded and a dequantized coefficient of the second enhancement layer to obtain a coefficient residual block of a third enhancement layer of the image block to be encoded; The coefficient residual block of the third enhancement layer is quantized and entropy encoded to obtain a code stream of the third enhancement layer.

17. The method according to claim 16, characterized in that The method further comprises: quantizing and dequantizing the coefficient residual block of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; Summing the dequantized coefficients of the third enhancement layer and the corresponding pixel points of the dequantized coefficients of the second enhancement layer and then inversely transforming them to obtain a reconstructed residual block of the third enhancement layer; Summing corresponding pixels in the reconstructed residual block of the third enhancement layer and the reconstructed block of the first enhancement layer to obtain a reconstructed block of the third enhancement layer; Differences are calculated between corresponding pixels in the image block to be encoded and the reconstructed block on the third enhancement layer to obtain a residual block on a fourth enhancement layer of the image block to be encoded; Obtaining a transform block of the fourth enhancement layer according to the residual block of the fourth enhancement layer, and transforming the transform block of the fourth enhancement layer to obtain transform coefficients of the fourth enhancement layer; Quantizing and entropy encoding the transform coefficients of the fourth enhancement layer to obtain a code stream of the fourth enhancement layer.

18. The method according to claim 15, characterized in that The method further comprises: quantizing, dequantizing, and inversely transforming the transform coefficients of the second enhancement layer to obtain a reconstructed residual block of the second enhancement layer; Summing corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the first enhancement layer to obtain a reconstructed block of the second enhancement layer; Differences are calculated between corresponding pixels in the image block to be encoded and the reconstructed block of the second enhancement layer to obtain a residual block of the image block to be encoded on a third enhancement layer; Obtaining a transform block of the third enhancement layer according to the residual block of the third enhancement layer, and transforming the transform block of the third enhancement layer to obtain transform coefficients of the third enhancement layer; Quantizing and entropy encoding the transform coefficients of the third enhancement layer to obtain a code stream of the third enhancement layer.

19. The method according to claim 18, characterized in that The method further comprises: quantizing and dequantizing transform coefficients of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; Taking a difference between corresponding pixel points of a transform coefficient of a third enhancement layer of the image block to be encoded and a dequantized coefficient of the third enhancement layer to obtain a coefficient residual block of a fourth enhancement layer of the image block to be encoded; The coefficient residual block of the fourth enhancement layer is quantized and entropy encoded to obtain a code stream of the fourth enhancement layer.

20. The method according to any one of claims 15 to 19, characterized in that The resolution of the reconstructed block of the first enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the encoding quality of the reconstructed block of the first enhancement layer is not lower than the encoding quality of the reconstructed block of the base layer.

21. The method according to any one of claims 15 to 19, wherein: The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the encoding quality of the reconstructed block of the second enhancement layer is not lower than the encoding quality of the reconstructed block of the first enhancement layer.

22. The method according to any one of claims 15 to 19, characterized in that The block partitioning manner of the coefficient residual block of the first enhancement layer and the transform block partitioning manner of the residual block of the second enhancement layer are independent of each other.

23. The method according to any one of claims 15 to 19, characterized in that The size of the maximum transform unit (LTU) of the coefficient residual block of the first enhancement layer is the same as the size of the reconstructed block of the base layer; or the size of the LTU is the same as the size of the reconstructed block of the original base layer after upsampling.

24. The method according to any one of claims 15 to 19, characterized in that The image block to be encoded refers to the largest coding unit LCU in the entire frame image; or, the image block to be encoded refers to the entire frame image; or, the image block to be encoded refers to the region of interest ROI in the entire frame image.

25. A method for decoding an image block, characterized in that: include: Obtaining a code stream of a base layer, a code stream of a first enhancement layer, and a code stream of a second enhancement layer of an image block to be decoded, wherein the code stream of the base layer includes a code stream of a residual block of the base layer, the code stream of the first enhancement layer includes a code stream of a residual block of the first enhancement layer, and the code stream of the second enhancement layer includes a code stream of a coefficient residual block of the second enhancement layer; Performing entropy decoding and inverse quantization on the bitstream of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; Performing entropy decoding, inverse quantization, and inverse transformation on the bitstream of the first enhancement layer to obtain a reconstructed residual block of the first enhancement layer; Obtain a reconstructed block of a base layer of the image block to be decoded, and sum corresponding pixels in the reconstructed residual block of the first enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the first enhancement layer; Performing entropy decoding and inverse quantization on the bitstream of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer; Summing corresponding pixel points in the dequantized coefficients of the first enhancement layer and the dequantized coefficients of the second enhancement layer to obtain reconstructed dequantized coefficients of the second enhancement layer; Performing an inverse transformation on the reconstructed dequantized coefficients of the second enhancement layer to obtain a reconstructed residual block of the second enhancement layer; The reconstructed block of the second enhancement layer is obtained by summing the corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the base layer.

26. The method according to claim 25, characterized in that The resolution of the reconstructed block of the first enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the decoding quality of the reconstructed block of the first enhancement layer is not lower than the decoding quality of the reconstructed block of the base layer.

27. The method according to claim 25, characterized in that The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the first enhancement layer, or the decoding quality of the reconstructed block of the second enhancement layer is not lower than the decoding quality of the reconstructed block of the first enhancement layer.

28. The method according to any one of claims 25 to 27, characterized in that The method further comprises: Obtaining a code stream of a third enhancement layer of the image block to be decoded; performing entropy decoding, inverse quantization, and inverse transformation on the code stream of the third enhancement layer to obtain a reconstructed residual block of the third enhancement layer; The reconstructed block of the third enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the third enhancement layer and the reconstructed block of the second enhancement layer.

29. The method according to claim 28, characterized in that The method further comprises: Obtaining a code stream of a fourth enhancement layer of the image block to be decoded; Performing entropy decoding and inverse quantization on the bitstream of the fourth enhancement layer to obtain dequantized coefficients of the fourth enhancement layer; Performing entropy decoding and inverse quantization on the bitstream of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; Summing the dequantized coefficients of the third enhancement layer and the corresponding pixel points of the dequantized coefficients of the fourth enhancement layer and performing inverse transformation to obtain a reconstructed residual block of the fourth enhancement layer; The reconstructed block of the fourth enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the second enhancement layer.

30. The method according to any one of claims 25 to 27, characterized in that The method further comprises: Obtaining a code stream of a third enhancement layer of the image block to be decoded; Performing entropy decoding and inverse quantization on the bitstream of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; Summing corresponding pixel points of the dequantized coefficients of the second enhancement layer and the dequantized coefficients of the first enhancement layer to obtain reconstructed dequantized coefficients of the second enhancement layer; Summing the reconstructed dequantized coefficients of the second enhancement layer and corresponding pixel points of the dequantized coefficients of the third enhancement layer and performing inverse transformation to obtain a reconstructed residual block of the third enhancement layer; The reconstructed block of the third enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the third enhancement layer and the reconstructed block of the base layer.

31. The method according to claim 30, wherein The method further comprises: Obtaining a code stream of a fourth enhancement layer of the image block to be decoded; Performing entropy decoding, inverse quantization, and inverse transformation on the bitstream of the fourth enhancement layer to obtain a reconstructed residual block of the fourth enhancement layer; The reconstructed block of the fourth enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the third enhancement layer.

32. The method according to any one of claims 25 to 27 and 31, characterized in that The transform block partitioning mode of the residual block of the first enhancement layer and the transform block partitioning mode of the residual block of the base layer are independent of each other.

33. The method according to any one of claims 25 to 27 and 31, characterized in that The transform block division manner of the residual block of the first enhancement layer is the same as the block division manner of the coefficient residual block of the second enhancement layer.

34. The method according to any one of claims 25 to 27 and 31, characterized in that The image block to be decoded refers to the largest coding unit LCU in the entire frame image; or, the image block to be decoded refers to the entire frame image; or, the image block to be decoded refers to the region of interest ROI in the entire frame image.

35. The method according to any one of claims 25 to 27 and 31, characterized in that When the base layer and the first enhancement layer are obtained in a spatial domain layered manner, the reconstructed block of the base layer is upsampled so that the resolution of the reconstructed block of the base layer is the same as the resolution of the reconstructed block of the first enhancement layer.

36. The method according to any one of claims 25 to 27 and 31, characterized in that The obtaining of a reconstructed block of a base layer of the image block to be decoded comprises: Obtaining a code stream of the base layer; Performing entropy decoding on the code stream of the base layer to obtain quantization coefficients of the base layer; Decoding the code stream of the base layer to obtain a prediction block of the base layer; Dequantizing and inversely transforming the quantized coefficients of the base layer to obtain a reconstructed residual block of the base layer; The reconstructed residual block of the base layer and corresponding pixel points in the prediction block of the base layer are summed to obtain the reconstructed block of the base layer.

37. A method for decoding an image block, characterized in that: include: Obtaining a code stream of a base layer, a code stream of a first enhancement layer, and a code stream of a second enhancement layer of an image block to be decoded, wherein the code stream of the base layer includes a code stream of a residual block of the base layer, and the code stream of the second enhancement layer includes a code stream of a residual block of the second enhancement layer; Performing entropy decoding on the code stream of the base layer to obtain quantization coefficients of the base layer; Decoding the code stream of the base layer to obtain a prediction block of the base layer; Dequantizing the quantized coefficients of the base layer to obtain dequantized coefficients of the base layer; Performing entropy decoding and inverse quantization on the bitstream of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; Summing the dequantized coefficients of the first enhancement layer and the corresponding pixel points of the dequantized coefficients of the base layer and performing inverse transformation to obtain a reconstructed residual block of the first enhancement layer; Summing the reconstructed residual block of the first enhancement layer and corresponding pixels in the prediction block of the base layer to obtain a reconstructed block of the first enhancement layer; Performing entropy decoding, inverse quantization, and inverse transformation on the bitstream of the second enhancement layer to obtain a reconstructed residual block of the second enhancement layer; The reconstructed block of the second enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the second enhancement layer and the reconstructed block of the first enhancement layer.

38. The method according to claim 37, wherein The resolution of the reconstructed block of the first enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the decoding quality of the reconstructed block of the first enhancement layer is not lower than the decoding quality of the reconstructed block of the base layer.

39. The method according to claim 37, wherein The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the first enhancement layer, or the decoding quality of the reconstructed block of the second enhancement layer is not lower than the decoding quality of the reconstructed block of the first enhancement layer.

40. The method according to any one of claims 37 to 39, characterized in that The method further comprises: Obtaining a code stream of a third enhancement layer of the image block to be decoded; Performing entropy decoding, inverse quantization, and inverse transformation on the residual bitstream of the third enhancement layer to obtain a reconstructed residual block of the third enhancement layer; The reconstructed block of the third enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the third enhancement layer and the reconstructed block of the second enhancement layer.

41. The method according to claim 40, wherein The method further comprises: Obtaining a code stream of a fourth enhancement layer of the image block to be decoded; Performing entropy decoding and inverse quantization on the bitstream of the fourth enhancement layer to obtain dequantized coefficients of the fourth enhancement layer; Performing entropy decoding and inverse quantization on the bitstream of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; Summing the dequantized coefficients of the third enhancement layer and the corresponding pixel points of the dequantized coefficients of the fourth enhancement layer and performing inverse transformation to obtain a reconstructed residual block of the fourth enhancement layer; The reconstructed block of the fourth enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the second enhancement layer.

42. The method according to any one of claims 37 to 39, wherein: The method further comprises: Obtaining a code stream of a third enhancement layer of the image block to be decoded; Performing entropy decoding and inverse quantization on the bitstream of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; Performing entropy decoding and inverse quantization on the bitstream of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer; Summing the dequantized coefficients of the second enhancement layer and corresponding pixel points of the dequantized coefficients of the third enhancement layer and performing inverse transformation to obtain a reconstructed residual block of the third enhancement layer; The reconstructed block of the third enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the third enhancement layer and the reconstructed block of the first enhancement layer.

43. The method according to claim 42, wherein The method further comprises: Obtaining a code stream of a fourth enhancement layer of the image block to be decoded; Performing entropy decoding, inverse quantization, and inverse transformation on the residual bitstream of the fourth enhancement layer to obtain a reconstructed residual block of the fourth enhancement layer; The reconstructed block of the fourth enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the third enhancement layer.

44. The method according to any one of claims 37-39, 41 and 43, characterized in that The transform block partitioning mode of the residual block of the second enhancement layer and the transform block partitioning mode of the residual block of the base layer are independent of each other.

45. The method according to any one of claims 37-39, 41 and 43, characterized in that The image block to be decoded refers to the largest coding unit LCU in the entire frame image; or, the image block to be decoded refers to the entire frame image; or, the image block to be decoded refers to the region of interest ROI in the entire frame image.

46. ​​The method according to claim 40, wherein When the second enhancement layer and the third enhancement layer are obtained by spatial layering, the reconstructed block of the second enhancement layer is upsampled so that the resolution of the reconstructed block of the second enhancement layer is the same as the resolution of the reconstructed block of the third enhancement layer.

47. A coding device, characterized in that include: An obtaining module, configured to obtain a reconstructed block of a base layer of an image block to be encoded; Differences are calculated between corresponding pixels in the image block to be encoded and the reconstructed block of the base layer to obtain a residual block of a first enhancement layer of the image block to be encoded; a determination module, configured to obtain a transform block of the first enhancement layer according to the residual block of the first enhancement layer, transform the transform block of the first enhancement layer to obtain transform coefficients of the first enhancement layer, and quantize and dequantize the transform coefficients of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; The obtaining module is further configured to obtain a coefficient residual block of a second enhancement layer of the image block to be encoded by performing a difference calculation on corresponding pixel points of the transform coefficients of the first enhancement layer and the dequantized coefficients of the first enhancement layer; an encoding module, configured to quantize and entropy encode the transform coefficients of the first enhancement layer to obtain a bit stream of the first enhancement layer; The coefficient residual block of the second enhancement layer is quantized and entropy encoded to obtain a code stream of the second enhancement layer.

48. The device according to claim 47, characterized in that: The obtaining module is further configured to obtain a difference between corresponding pixels in the image block to be encoded and the reconstructed block of the second enhancement layer to obtain a residual block of the third enhancement layer of the image block to be encoded; The determining module is further configured to obtain a transform block of the third enhancement layer according to the residual block of the third enhancement layer, and transform the transform block of the third enhancement layer to obtain transform coefficients of the third enhancement layer; The encoding module is further configured to perform quantization and entropy encoding on the transform coefficients of the third enhancement layer to obtain a code stream of the third enhancement layer.

49. The device according to claim 48, characterized in that: The determining module is further configured to transform, quantize, and dequantize the residual block of the third enhancement layer to obtain a dequantized coefficient of the third enhancement layer; The obtaining module is further configured to obtain a coefficient residual block of a fourth enhancement layer of the image block to be encoded by calculating a difference between corresponding pixel points of the transform coefficients of the third enhancement layer and the dequantized coefficients of the third enhancement layer; The encoding module is further configured to perform quantization and entropy encoding on the coefficient residual block of the fourth enhancement layer to obtain a code stream of the fourth enhancement layer.

50. A decoding device, characterized in that: include: An obtaining module, configured to obtain a code stream of a first enhancement layer and a code stream of a second enhancement layer of an image block to be decoded; a decoding module, configured to perform entropy decoding and inverse quantization on the bitstream of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; and perform entropy decoding, inverse quantization, and inverse transformation on the bitstream of the first enhancement layer to obtain a reconstructed residual block of the first enhancement layer; Performing entropy decoding and inverse quantization on the bitstream of the second enhancement layer to obtain dequantized coefficients of the second enhancement layer; summing corresponding pixel points in the dequantized coefficients of the first enhancement layer and the dequantized coefficients of the second enhancement layer and performing inverse transformation to obtain a reconstructed residual block of the second enhancement layer; a reconstruction module, configured to obtain a reconstructed block of a base layer of the image block to be decoded, sum corresponding pixel points in the reconstructed residual block of the first enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the first enhancement layer; and sum corresponding pixel points in the reconstructed residual block of the second enhancement layer and the reconstructed block of the base layer to obtain a reconstructed block of the second enhancement layer.

51. The device according to claim 50, characterized in that The resolution of the reconstructed block of the first enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the decoding quality of the reconstructed block of the first enhancement layer is not lower than the decoding quality of the reconstructed block of the base layer.

52. The device according to claim 50 or 51, characterized in that The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the first enhancement layer, or the decoding quality of the reconstructed block of the second enhancement layer is not lower than the decoding quality of the reconstructed block of the first enhancement layer.

53. The device according to claim 50 or 51, characterized in that: The obtaining module is further configured to obtain a code stream of a third enhancement layer of the image block to be decoded; The decoding module is further configured to perform entropy decoding, inverse quantization, and inverse transformation on the code stream of the third enhancement layer to obtain a reconstructed residual block of the third enhancement layer; The reconstruction module is further configured to sum corresponding pixel points in the reconstructed residual block of the third enhancement layer and the reconstructed block of the second enhancement layer to obtain the reconstructed block of the third enhancement layer.

54. The device according to claim 53, characterized in that: The obtaining module is further configured to obtain a code stream of a fourth enhancement layer of the image block to be decoded; The decoding module is further configured to perform entropy decoding and inverse quantization on the bitstream of the fourth enhancement layer to obtain dequantized coefficients of the fourth enhancement layer; sum corresponding pixel points in the dequantized coefficients of the third enhancement layer and the dequantized coefficients of the fourth enhancement layer, and then perform inverse transformation to obtain a reconstructed residual block of the fourth enhancement layer; The reconstruction module is further configured to sum corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the second enhancement layer to obtain the reconstructed block of the fourth enhancement layer.

55. An encoding device, characterized in that include: An obtaining module, configured to transform, quantize, and dequantize a residual block of a base layer of an image block to be encoded to obtain a dequantized coefficient of the base layer; Transforming the transform block of the base layer to obtain transform coefficients of the base layer, and subtracting corresponding pixel points of the transform coefficients of the base layer and the dequantized coefficients of the base layer to obtain a coefficient residual block of the first enhancement layer of the image block to be encoded; quantizing and inverse-quantizing the coefficient residual block of the first enhancement layer to obtain dequantized coefficients of the first enhancement layer; summing corresponding pixel points of the dequantized coefficients of the first enhancement layer and the dequantized coefficients of the base layer to obtain reconstructed dequantized coefficients of the first enhancement layer; performing inverse transformation on the reconstructed dequantized coefficients of the first enhancement layer to obtain a reconstructed residual block of the first enhancement layer; summing corresponding pixel points in the reconstructed residual block of the first enhancement layer and the prediction block of the base layer to obtain a reconstructed block of the first enhancement layer; and subtracting corresponding pixel points in the image block to be coded and the reconstructed block of the first enhancement layer to obtain a residual block of the second enhancement layer of the image block to be coded; a determination module, configured to obtain a transform block of the second enhancement layer according to the residual block of the second enhancement layer, and transform the transform block of the second enhancement layer to obtain transform coefficients of the second enhancement layer; an encoding module, configured to transform, quantize, and entropy encode the residual block of the base layer to obtain a code stream of the base layer; quantizing and entropy encoding the coefficient residual block of the first enhancement layer to obtain a code stream of the first enhancement layer; The transform coefficients of the second enhancement layer are quantized and entropy encoded to obtain a code stream of the second enhancement layer.

56. The device according to claim 55, characterized in that: The obtaining module is further configured to quantize, dequantize, and inversely transform the transform coefficients of the second enhancement layer to obtain a reconstructed block of the second enhancement layer; and to obtain a residual block of a third enhancement layer of the image block to be encoded by subtracting corresponding pixels in the image block to be encoded and the reconstructed block of the second enhancement layer; The determining module is further configured to obtain a transform block of the third enhancement layer according to the residual block of the third enhancement layer, and transform the transform block of the third enhancement layer to obtain transform coefficients of the third enhancement layer; The encoding module is further configured to perform quantization and entropy encoding on the transform coefficients of the third enhancement layer to obtain a code stream of the third enhancement layer.

57. The device according to claim 56, characterized in that: The obtaining module is further configured to quantize and dequantize the transform coefficients of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer, and to calculate the difference between corresponding pixel points of the transform coefficients of the third enhancement layer and the dequantized coefficients of the third enhancement layer of the image block to be encoded to obtain a coefficient residual block of a fourth enhancement layer of the image block to be encoded; The encoding module is further configured to perform quantization and entropy encoding on the coefficient residual block of the fourth enhancement layer to obtain a code stream of the fourth enhancement layer.

58. A decoding device, characterized in that include: An acquisition module is configured to obtain a code stream of a base layer, a code stream of a first enhancement layer, and a code stream of a second enhancement layer of an image block to be decoded; a decoding module that performs entropy decoding on a bitstream of a base layer to obtain quantization coefficients of the base layer, dequantizes the quantization coefficients of the base layer to obtain dequantization coefficients of the base layer; performs entropy decoding and dequantization on a bitstream of the first enhancement layer to obtain dequantization coefficients of the first enhancement layer; and sums corresponding pixel points in the dequantization coefficients of the first enhancement layer and the dequantization coefficients of the base layer, and then performs an inverse transform to obtain a reconstructed residual block of the first enhancement layer; a reconstruction module, configured to sum corresponding pixels in the reconstructed residual block of the first enhancement layer and the prediction block of the base layer to obtain a reconstructed block of the first enhancement layer; Performing entropy decoding, inverse quantization, and inverse transformation on the code stream of the second enhancement layer to obtain a reconstructed residual block of the second enhancement layer; and summing corresponding pixel points in the reconstructed residual block of the second enhancement layer and the reconstructed block of the first enhancement layer to obtain a reconstructed block of the second enhancement layer.

59. The device according to claim 58, characterized in that The resolution of the reconstructed block of the first enhancement layer is not lower than the resolution of the reconstructed block of the base layer, or the decoding quality of the reconstructed block of the first enhancement layer is not lower than the decoding quality of the reconstructed block of the base layer.

60. The device according to claim 58 or 59, characterized in that The resolution of the reconstructed block of the second enhancement layer is not lower than the resolution of the reconstructed block of the first enhancement layer, or the decoding quality of the reconstructed block of the second enhancement layer is not lower than the decoding quality of the reconstructed block of the first enhancement layer.

61. The device according to claim 58 or 59, characterized in that: The obtaining module is further configured to obtain a code stream of a third enhancement layer of the image block to be decoded; The decoding module is further configured to perform entropy decoding, inverse quantization, and inverse transformation on the residual bitstream of the third enhancement layer to obtain a reconstructed residual block of the third enhancement layer; The reconstructed block of the third enhancement layer is obtained by summing corresponding pixels in the reconstructed residual block of the third enhancement layer and the reconstructed block of the second enhancement layer.

62. The device according to claim 61, characterized in that: The obtaining module is further configured to obtain a code stream of a fourth enhancement layer of the image block to be decoded; The decoding module is further configured to perform entropy decoding and inverse quantization on the bitstream of the fourth enhancement layer to obtain dequantized coefficients of the fourth enhancement layer, and perform entropy decoding and inverse quantization on the bitstream of the third enhancement layer to obtain dequantized coefficients of the third enhancement layer; and sum corresponding pixel points in the dequantized coefficients of the third enhancement layer and the dequantized coefficients of the fourth enhancement layer, and then perform inverse transformation to obtain a reconstructed residual block of the fourth enhancement layer; The reconstruction module is further configured to sum corresponding pixel points in the reconstructed residual block of the fourth enhancement layer and the reconstructed block of the second enhancement layer to obtain the reconstructed block of the fourth enhancement layer.

63. An encoder, characterized in that The method comprises a processing circuit for executing the method according to any one of claims 1 to 24.

64. A decoder, characterized in that comprising a processing circuit for executing the method of any one of claims 25-46.

65. A computer program product, characterized in that The method comprises a program code for executing the method according to any one of claims 1 to 46 when the program code is executed on a computer or a processor.

66. An encoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium is coupled to the processor and stores a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform the method of any one of claims 1 to 24.

67. A decoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method according to any one of claims 25-46.

68. An image processor, characterized in that comprising processing circuitry for performing the method according to any one of claims 1-46.

69. A non-transitory computer-readable storage medium, characterized in that The method comprises a program code for executing the method according to any one of claims 1 to 46 when the program code is executed by a computer device.

Images