LCEVC-based encoding method, decoding method, and coding device

Abstract

The present disclosure relates to an LCEVC-based encoding method, a decoding method and a coding device, and relates to the technical field of video coding. The method comprises: decoding L2 enhancement coefficients of a target transform block in an enhancement layer 2 code stream to obtain a target residual of the target transform block; decoding a temporal layer code stream to obtain prediction mode information; when the prediction mode information is inter prediction mode information, scaling a first motion vector of a target block in a current frame of a base layer to obtain a second motion vector corresponding to the target transform block in a current frame of the enhancement layer 2, the inter prediction mode information including a position of a matching block of the target transform block; determining a reconstructed residual of the matching block according to the second motion vector of the target transform block and the position of the matching block; and determining L2 residual of the target transform block according to the reconstructed residual of the matching block and the target residual. The embodiments of the present disclosure are used to solve the problem that the temporal prediction of the enhancement layer 2 ignores the motion process of the image and cannot effectively reduce the code rate.

Description

Technical Field

[0001] This disclosure relates to the field of video coding technology, and in particular to an encoding method, decoding method and decoding device based on LCEVC. Background Technology

[0002] Low Complexity Enhancement Video Coding (LCEVC) is a video coding standard released in 2021 (ISO / IEC 23094-2). LCEVC has advantages such as strong compatibility, simple implementation, and Scalable Video Coding (SVC) fault tolerance. However, in the current LCEVC encoder, the temporal prediction of enhancement layer 2 only references the co-position blocks of the reconstructed residual of the previous frame, ignoring the motion process of the image. Therefore, it cannot effectively reduce the bitrate and increases the coding difficulty. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides an encoding method, decoding method, and decoding device based on LCEVC, which can effectively reduce the code rate and reduce the encoding difficulty.

[0004] To achieve the above objectives, the technical solutions provided by the embodiments of this disclosure are as follows:

[0005] Firstly, a decoding method based on LCEVC is provided, including:

[0006] The L2 enhancement coefficients of the target transform block in the bitstream of enhancement layer 2 are decoded to obtain the target residual of the target transform block;

[0007] Decode the time-layer bitstream to obtain prediction mode information;

[0008] When the prediction mode information is inter-frame prediction mode information, the first motion vector of the target block in the current frame of the base layer is scaled to obtain the second motion vector corresponding to the target transform block in the current frame of the enhancement layer 2. The inter-frame prediction mode information includes the position of the matching block of the target transform block.

[0009] The reconstruction residual of the matching block is determined based on the second motion vector of the target transform block and the position of the matching block;

[0010] The L2 residual of the target transform block is determined based on the reconstruction residual of the matching block and the target residual.

[0011] Secondly, an encoding method based on LCEVC is provided, including:

[0012] The first motion vector of the target block in the current frame of the base layer is scaled to obtain the second motion vector corresponding to the target transform block of the enhancement layer 2;

[0013] The initial search block is determined in the reconstructed frame of the previous frame based on the second motion vector of the target transform block;

[0014] The initial search block is used as the origin to search for matching blocks;

[0015] Based on the reconstruction residual of the matching block and the L2 residual of the target transform block, determine the inter-frame residual and inter-frame cost of the target transform block;

[0016] When the inter-frame cost is less than or equal to the intra-frame cost, the inter-frame residual of the target transform block is encoded to obtain the L2 enhancement coefficient of the target transform block.

[0017] Thirdly, a decoding device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the LCEVC-based decoding method as described in any one of claims 1 to 3, or, when the computer program is executed by the processor, it implements the LCEVC-based encoding method as described in any one of claims 4 to 16.

[0018] Fourthly, a computer-readable storage medium is provided, comprising: storing a computer program on the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the LCEVC-based decoding method as described in the first aspect or any optional embodiment thereof, or the LCEVC-based encoding method as described in the second aspect or any optional embodiment thereof.

[0019] Fifthly, a computer program product is provided, wherein a computer program is stored in the computer program, and when the computer program is executed by a processor, it implements the LCEVC-based decoding method as described in the first aspect or any optional embodiment thereof, or the LCEVC-based encoding method as described in the second aspect or any optional embodiment thereof.

[0020] The LCEVC-based encoding method, decoding method, and decoding device provided in this disclosure consider that the L2 residual of enhancement layer 2 mainly consists of edge and some detail data. The motion of these contents is consistent with the motion of the corresponding scene in the image. Therefore, this application embodiment utilizes the MV of the base layer to perform motion estimation in enhancement layer 2, which can reduce the encoding difficulty, effectively reduce the bit rate, and improve the utilization rate of the bit rate of enhancement layer 2 while ensuring accurate temporal prediction. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a block diagram of a video decoding system 100 in some embodiments of this application;

[0024] Figure 2 This is a schematic diagram of the LCEVC encoder architecture;

[0025] Figure 3 A flowchart illustrating an adaptive transform kernel encoding method provided in an embodiment of this application;

[0026] Figure 4 A flowchart illustrating an LCEVC-based encoding method provided in this application embodiment;

[0027] Figure 5 A schematic diagram illustrating motion search provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of an encoding method provided in an embodiment of this application;

[0029] Figure 7 This is a schematic diagram of the structure of a decoding device provided in an embodiment of this application. Detailed Implementation

[0030] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0031] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0032] Video has always been one of the main ways people interact with information, greatly influencing their production, work, and lives. A video can be viewed as a sequence of multiple video frames (images). Video playback can be seen as displaying video frames in the order they appear in the sequence at a preset speed (e.g., 24 frames / second, 30 frames / second, 60 frames / second). Theoretically, the amount of video data is positively correlated with the resolution of the video frames; the higher the resolution of the video frames, the larger the amount of video data. If all the data of the video frames were directly saved in the video file, the amount of video data would be enormous, making it difficult to store and transmit. Video decoding was proposed to address this problem to some extent. Video decoding mainly includes video encoding and video decoding. Video encoding can be understood as the process of compressing video data, while video decoding can be understood as the process of restoring the compressed video data.

[0033] Figure 1 This is a block diagram of a video decoding system 100 in some embodiments of this application. For example... Figure 1 As shown, the video decoding system 100 includes a source device 11 and a destination device 12. The source device 11 can acquire raw video data through a video source 111, encode the raw video data using a video encoder 112 to obtain encoded video data, and provide the encoded video data to the destination device 12 through an output interface 113. The destination device 12 can acquire the encoded video data provided by the source device 11 through an input interface 121, decode the encoded video data using a video decoder 122 to obtain decoded video data, and input the decoded video data to a player 123 to play the video. The source device 11 and the destination device 12 can include any of a wide range of devices, such as personal computers, laptops, tablets, set-top boxes, mobile phones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc.

[0034] In some embodiments, the video source 111 of the source device 11 can be a video recording device, such as a camera. In other embodiments, the video source 111 can be a component capable of generating video based on computer graphics, such as a screen recording component, an animation generation component, etc.

[0035] In some embodiments, the destination device 12 may receive video encoded data provided by the source device 11 via a computer-readable medium. The computer-readable medium may include any type of media or device capable of moving video encoded data from the source device 11 to the destination device 12. In one example, the computer-readable medium may include a communication medium enabling the source device 11 to transmit video encoded data directly to the destination device 12 in real time. The video encoded data may be modulated according to a communication standard (e.g., a wireless communication protocol) and transmitted to the destination device 12. The communication medium may include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet network (e.g., a local area network, a wide area network, or a global network, such as the Internet). The communication medium may include a router, switch, base station, or any other device that can be used to facilitate communication from the source device 11 to the destination device 12.

[0036] In some instances, video encoded data can be output from output interface 113 to a storage device. Correspondingly, video encoded data can be accessed from the storage device via input interface 123. The storage device may comprise any of a variety of distributed or locally accessible data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing video encoded data.

[0037] In another example, the storage device may correspond to a file server or another intermediate storage device capable of holding the encoded video generated by the source device 11. The destination device 12 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server capable of storing and transmitting the encoded video data to the destination device 12. Exemplary examples of such file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The destination device 12 can access the encoded video data via any standard data connection, including an Internet connection. This may include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., DSL, cable modems, etc.), or a combination of both suitable for accessing the encoded video data stored on the file server. The transmission of the encoded video data from the storage device may be streaming transmission, download transmission, or a combination thereof.

[0038] As mentioned above, video encoding can be understood as the process of compressing video data, and video decoding can be understood as the process of restoring compressed video data. The video encoder 112 can be understood as a set of standard rules in the video data compression process, and the video decoder 122 can be understood as a set of standard rules in the video data restoration process. In general, the video decoder 122 needs to use a decoding method that corresponds to the encoding method used by the video encoder 121 in order to correctly restore the video data.

[0039] Currently, video coding standards have evolved from the initial ISO / IEC MPEG-1, through ISO / IEC MPEG-2, ISO / IEC MPEG-4, Advanced Video Coding (H.264 / AVC), High Efficiency Video Coding (H.265 / HEVC), to Versatile Video Coding (H.266 / VVC) and Low Complexity Enhancement Video Coding (LCEVC) standards. The adaptive transform kernel encoding method, encoding device, and storage medium provided in the embodiments of this application are all applicable to the LCEVC standard.

[0040] Low Complexity Enhancement Video Coding (LCEVC) is a video coding standard released in 2021 (ISO / IEC 23094-2). LCEVC has the advantages of strong compatibility, simple implementation, and Scalable Video Coding (SVC) fault tolerance.

[0041] Figure 2 This is a schematic diagram of the LCEVC encoder architecture.

[0042] like Figure 2 As shown, the LCEVC encoder includes: a first downsampling layer 21, a second downsampling layer 22, a basic coding layer 23, a first upsampling layer 24, a first enhancement layer 25, a restoration layer 26, a fusion layer 27, a second upsampling layer 28, and a second enhancement layer 29.

[0043] The first downsampling layer 21 is used to downsample the video to be encoded in order to reduce the resolution of the video.

[0044] The second downsampling layer 22 is used to further downsample the output of the first downsampling layer 21 to further reduce the resolution of the video frames of the video to be encoded.

[0045] The basic coding layer 23, also known as the basic layer, is used to encode the output of the second downsampling layer 22 through the basic layer encoder to generate the bitstream of the basic coding layer 23. It should be noted that the basic layer encoder can be any encoder, such as x264, x265, High Efficiency Video Coding (HEVC) encoders, Versatile Video Coding (VVC) encoders, etc.

[0046] The first upsampling layer 24 is used to upsample the coded reconstructed image of the basic coding layer 23 to improve the resolution of the coded reconstructed image of the basic coding layer 13.

[0047] The first enhancement layer 25, also known as enhancement layer 1, is used to calculate the residual between the output of the first downsampling layer 21 and the output of the first upsampling layer 24, namely the L1 residual (also known as the L-1 residual), and to perform transformation, quantization and entropy encoding on the L1 residual in sequence to obtain the L1 enhancement coefficients (also known as the L-1 coefficients).

[0048] The restoration layer 26 is used to sequentially perform inverse quantization, inverse transformation, and filtering on the data obtained by quantization from the first enhancement layer 25, as well as to obtain the output of the restoration layer 26.

[0049] The fusion layer 27 is used to add and fuse the output of the restoration layer 26 and the output of the first upsampling layer 24 to obtain the output of the fusion layer 27.

[0050] The second upsampling layer 28 is used to upsample the output of the fusion layer 27 to improve the resolution of the image output by the fusion layer 27.

[0051] The second enhancement layer 29, also known as enhancement layer 2, is used to calculate the residual between the output of the second upsampling layer 28 and the original video to be encoded, namely the L2 residual (also known as the L-2 residual), and to sequentially perform temporal prediction, transformation, quantization and entropy coding on the L2 residual to obtain the L2 enhancement coefficients (also known as L-2 coefficients).

[0052] The quantization control parameters of the basic coding layer 23, the first enhancement layer 25, and the second upsampling layer 28 together constitute the rate control parameters of the LCEVC encoder. They can be described using the following formula (1):

[0053] out=E2(E1(E0(x,BaseRate),SW1),SW2);

[0054] In formula (1), x represents the input of the LCEVC encoder; out represents the output of the LCEVC encoder; E2, E1 and E0 represent the functions of the second upsampling layer 28, the first enhancement layer 25 and the basic coding layer 23, respectively; BaseRate, SW1 and SW2 represent the rate control parameters of the basic coding layer 23, the first enhancement layer 25 and the second upsampling layer 28, respectively.

[0055] In the LCEVC encoder described above, the user has the right to choose the transform kernel used for transform and inverse transform processing during the encoding process of the enhancement layer. However, users without experience may not be able to select the appropriate transform kernel for different video sequences, resulting in low compression performance during the video encoding process.

[0056] The LCEVC encoder provides two transform kernels: Directional Decomposition (DD) and Directional Decomposition Squared (DDS). The transform kernel is configured globally, meaning that the entire video sequence uses the same transform kernel for both transform and inverse transform processing.

[0057] The transform kernel DDS has a size of 4x4 and can be represented as follows:

[0058]

[0059] The transform kernel DDS can be applied to each 4x4 residual block to calculate the transform coefficients; the formula for calculating the transform coefficients using the transform kernel DDS is shown in formula (2) below:

[0060]

[0061] In the above formula (2), C0, C1...C 15 C represents the 16 coefficients in the transformation coefficients. i R represents the i-th coefficient, where i can take the value of an integer from 0 to 15. In the above formula (2), R 0,0 R 0,1 R 0,2 ...R 3,3 R represents the residual corresponding to different coordinates in the residual block. x,y This represents the residual with coordinates (x, y) in the residual block, where x and y can take integer values ​​from 0 to 3.

[0062] The size of the transform kernel DD is 2x2, and the transform kernel DD can be represented as: The transform kernel DD can be applied to each 2x2 residual block to calculate the transform coefficients; the formula for calculating the transform coefficients using the transform kernel DD is shown in formula (3) below:

[0063]

[0064] In the above formula (3), C0, C1, C2, and C3 represent the four coefficients in the transformation coefficients, C i R represents the i-th coefficient, where i can take the value of an integer from 0 to 3. In the above formula (3), R 0,0 R 0,1 R 1,0 and R 1,1 R represents the residual corresponding to different coordinates in the residual block. x,y This represents the residual with coordinates (x, y) in the residual block, where x and y can take values ​​of 0 or 1.

[0065] In related technologies, High Efficiency Video Coding (HEVC) employs two transform methods: Integer Discrete Cosine Transform (DCT-2) and Integer Discrete Sine Transform (DST). The Integer DCT-2 transform kernel has four sizes: 32x32, 16x16, 8x8, and 4x4, while the Integer DST transform kernel has a size of 4x4. The Integer DST transform is only used for intra-frame 4x4 luma blocks. The choice of the four sizes in the Integer DCT-2 transform is determined by the Rate-Distortion Optimization (RDO) process in HEVC's partitioning process.

[0066] VVC: The transformation methods used in VVC include integer DCT-2 transformation, integer DST-7 transformation and integer DCT-8 transformation. The maximum size of integer DCT-2 is 64x64, and the maximum size of integer DST-7 and integer DCT-8 is 32x32, which is determined by the RDO (Rate Distortion Optimization) process.

[0067] In the traditional HEVC and VVC encoders mentioned above, the structure is based on block partitioning. Therefore, the size of the transform kernel is highly related to the size of the block. The final decision-making method is to select the transform kernel through RDO cost, which is usually calculated based on the information of spatially adjacent blocks. This transform kernel selection method is not applicable to LCEVC encoders.

[0068] Because the residual data to be encoded in the enhancement layer of the LCEVC encoder is sparse, the transform kernel size in LCEVC is much smaller than that in traditional encoders. This is because the residual data contains sparse information, such as edges, points, and some detailed information. Using a small transform kernel can better handle this sparse data. Based on this, in some embodiments of this application, information affecting the sparsity of the residual to be encoded (i.e., sparsity-related information of the video sequence) is considered in the transform kernel selection process for different video sequences. Therefore, an adaptive transform kernel can be determined for different video sequences based on the sparsity of the residual to be encoded corresponding to different video sequences, thereby improving the compression performance in the video encoding process.

[0069] Figure 3 This is a flowchart illustrating an adaptive transform kernel encoding method provided in an embodiment of this application.

[0070] like Figure 3 As shown, the encoding method of this adaptive transform kernel may include the following steps:

[0071] 301. Obtain information related to the sparsity of the video sequence.

[0072] The sparsity-related information includes at least one of the following: the mean absolute value of the base layer residuals, the average quantization parameter of the base layer, and the quantization step size of the enhancement layers. The enhancement layers include enhancement layer 1 and enhancement layer 2. The quantization step size of the enhancement layers includes: the quantization step size of enhancement layer 1, and / or, the quantization step size of enhancement layer 2.

[0073] The mean of the absolute values ​​of the base layer residuals (MARbase): A large mean of the absolute values ​​of the base layer residuals indicates that the prediction of the base layer is not accurate enough and the correlation between adjacent frames is low. At this time, the correlation between the L2 residual of the current frame and the reconstructed residual of the previous frame in enhancement layer 2 will also be low, thus affecting the sparsity of the residual to be encoded.

[0074] The average quantization parameter (aveQP) of the basic layer mentioned above: In video coding, the source of loss is the quantization process, and the quantization parameter reflects the magnitude of the loss. The larger the quantization parameter, the greater the loss, and vice versa. When the loss is large, the L2 residual in the enhancement layer will also be large, which will also affect the sparsity of the residual to be encoded.

[0075] The quantization step size of the enhancement layer affects the reconstructed residual, and the reconstructed residual of the current frame affects the residual to be encoded in the next frame. Therefore, the quantization step size of the enhancement layer affects the sparsity of the residual to be encoded. The quantization step size of enhancement layer 2 can be represented as SW2.

[0076] 302. Determine the adaptive transform kernel based on sparsity information of the video sequence.

[0077] In some embodiments of this application, the adaptive transform kernels described above include, but are not limited to, directional decomposition (DD) or directional decomposition squared (DDS).

[0078] The adaptive transform kernel selection problem can be understood as a binary classification problem. That is, for a sequence, if the transform kernel DD can achieve higher compression efficiency, then it is determined to be classified into the DD class; if the transform kernel DDS can achieve higher compression efficiency, then it is classified into the DDS class.

[0079] Support Vector Machines (SVMs) are a type of generalized linear classifier that performs binary classification of data using supervised learning. Their decision boundary is the maximum-margin hyperplane obtained by solving for the training samples. SVMs use a hinge loss function to calculate empirical risk and incorporate a regularization term into the solution system to optimize structural risk. Therefore, SVMs are sparsity- and robust classifiers. SVMs can perform non-linear classification using kernel methods, making them one of the common kernel learning methods.

[0080] Since Support Vector Machine (SVM) is a few-shot learning method that requires a small training set, it can solve binary classification problems well and has low inference complexity, making it suitable for the adaptive transformation kernel selection problem in the embodiments of this application.

[0081] In some embodiments of this application, determining the adaptive transform kernel based on sparsity-related information of the video sequence may include, but is not limited to: inputting sparsity-related information of the video sequence into the target support vector machine; and obtaining the adaptive transform kernel output by the target support vector machine.

[0082] The target support vector machine (SVM) is trained on a target training dataset. This dataset includes training data for multiple video sequences. Each video sequence's training data includes sparsity-related information extracted from the sequence, and a standard transform kernel corresponding to this sparsity-related information. This standard transform kernel is pre-determined and suitable for the specific video sequence; that is, encoding based on this standard transform kernel achieves higher compression efficiency. After training on the target training dataset, the resulting target SVM can quickly and accurately select and output an adaptive transform kernel based on the sparsity-related information of the input video sequences.

[0083] 303. During the encoding process of the video sequence in the enhancement layer, transformation and inverse transformation are performed based on the adaptive transform kernel.

[0084] For example, as Figure 1 Taking the enhancement layer 2 as an example, the time-domain prediction process involves transformation and inverse transformation. In this process, time-domain prediction is performed based on the adaptive transformation kernel. Furthermore, after the time-domain prediction, the enhancement layer 2 will continue to perform transformation processing, which is also performed based on the adaptive transformation kernel.

[0085] In this embodiment, since the transform kernel applicable to different videos is also related to the sparsity of the residuals to be encoded in different videos, the mean absolute value of the residuals in the base layer, the average quantization parameter of the base layer, and the quantization step size of the enhancement layer can all affect the parameters of residual sparsity. Based on these parameters, adaptive transform kernel selection can determine a suitable adaptive transform kernel for different videos. Then, the adaptive transform kernel is used to perform transform and inverse transform processing on the video sequence during the encoding process of the enhancement layer, which can improve the compression performance of video encoding.

[0086] In some embodiments of this application, after determining the adaptive transform kernel in step 302 above, encoding is performed based on the adaptive transform kernel to obtain header information. The header information typically includes prediction information (e.g., whether the prediction method is inter-frame prediction or intra-frame prediction), the transform kernel (such as the adaptive transform kernel determined in the embodiments of this application), and quantization information (such as the quantization step size).

[0087] In some embodiments of this application, after determining the adaptive transform kernel in step 302 above, the video sequence is transformed and inverse transformed based on the adaptive transform kernel during the encoding process of the enhancement layer, thereby obtaining the L2 enhancement coefficients after the enhancement layer 2 is encoded, the L1 enhancement coefficients after the enhancement layer 1 is encoded, and the time information obtained by entropy encoding the encoding information of the prediction information in the temporal prediction process in the L2 enhancement layer. Finally, the header information, time information, L2 enhancement coefficients, and L1 enhancement coefficients are all sent to the decoding device, and the basic layer bitstream is also sent to the decoding device, so that the decoding device can combine the header information, time information, L2 enhancement coefficients, L1 enhancement coefficients, and basic layer bitstream to recover the video sequence during the decoding process.

[0088] In related technologies, the temporal prediction of enhancement layer 2 in the LCEVC encoder only references the co-position blocks of the reconstruction residual of the previous frame, ignoring the motion process of the image. The temporal prediction module in enhancement layer 2 of the LCEVC encoder performs the following operations:

[0089] (1) The L2 residual of the current frame is directly transformed and quantized to obtain intra symbols.

[0090] (2) Perform inverse quantization and inverse transformation on intra_symbols to obtain the intra-reconstruction residual (intra_resi_recon);

[0091] (3) Add intra_resi_recon to the upsampled image of the current frame to obtain intra_recon;

[0092] (4) Calculate the intra cost (intra_cost) for each transform block based on intra_recon and intra_symbols;

[0093] (5) Subtract the reconstruction residual of the previous frame from the L2 residual of the current frame to obtain the inter-frame residual (inter_resi);

[0094] (6) Transform and quantize inter_resi to obtain inter_symbols;

[0095] (7) Perform inverse quantization and inverse transformation on inter_symbols to obtain the inter-frame reconstruction residual (inter_resi_recon);

[0096] (8) Inter-reconstruction is obtained by adding the reconstruction residual of the previous frame and the upsampled image of the current frame to inter_resi_recon;

[0097] (9) Calculate the inter-frame cost (inter_cost) based on inter_recon and inter_symbols;

[0098] (10) The encoding method of intra-frame mode or inter-frame mode is determined based on the intra_cost and inter_cost of each transform block.

[0099] The temporal prediction process can be understood as calculating the cost of intra-frame coding and inter-frame coding for each transform block. Inter-frame coding serves as a reference for a co-position block, and the final coding method is selected based on the cost of intra-frame coding and inter-frame coding.

[0100] Because the temporal prediction of enhancement layer 2 in the LCEVC encoder only references the co-position blocks of the reconstructed residual of the previous frame, ignoring the motion process of the image, it cannot effectively reduce the bitrate and increases the encoding difficulty. Based on the above problem, considering that the L2 residual of enhancement layer 2 mainly consists of edge and some detail data, and the motion of these contents is consistent with the motion of the corresponding scenes in the image, this embodiment utilizes the MV of the base layer to perform motion estimation in enhancement layer 2. This can reduce the encoding difficulty, effectively reduce the bitrate, and improve the bitrate utilization of enhancement layer 2 while ensuring accurate temporal prediction.

[0101] Figure 4 This is a flowchart illustrating an LCEVC-based encoding method provided in an embodiment of this application.

[0102] like Figure 4 As shown, the LCEVC-based decoding method may include the following steps:

[0103] 401. Scale the first motion vector of the target block in the current frame of the base layer to obtain the second motion vector corresponding to the target transform block of the enhancement layer 2.

[0104] The target block can be any block in the current frame, and the target transformation block is a specific block within the image region corresponding to the target block in enhancement layer 2. For example, assuming the target block is a 4x4 block, and after scaling, the corresponding image region in enhancement layer 2 is a 16x16 region, the target transformation block can be a 4x4 block within that 16x16 region.

[0105] In the base layer bitstream, each 4x4 block has an MV and a corresponding reference frame. Considering that the encoded video in the base layer is obtained by downsampling the width and height of the original video by 1 / 2, and the distance between the coordinates stored in the MV is the distance under 1 / 4 pixel, it is necessary to map the motion vector (MV) of the base layer to the transform block and scale it. The scaling formula of MV is shown in the following formula (4):

[0106]

[0107] In formula (4) above, iPOC is the playback order identifier (Picture OrderCount, POC) of the current frame, refPOC is the POC of the reference frame, and mv base mv is the first motion vector of the target block in the current frame. blk For MV base The result after scaling.

[0108] 402. Determine the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block.

[0109] The initial search block is an image block determined in the reconstructed frame of the previous frame based on the position of the target transform block and the second motion vector.

[0110] In some embodiments of this application, before determining the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block, the block average gradient of each transform block in the current frame of enhancement layer 2 and the frame-level average gradient of the current frame can be determined first; then, when determining the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block, specifically when the block average gradient of the target transform block is greater than or equal to the frame-level average gradient, the initial search block can be determined in the reconstructed frame of the previous frame based on the second motion vector of the target transform block.

[0111] Since the residuals in the enhancement layer mostly store edge information after encoding, the edge information of the residuals can be obtained by calculating the block average gradient of each transform block in the current frame of enhancement layer 2 and the frame-level average gradient of the current frame. If the block average gradient of the target transform block is less than the average gradient of the current frame, it indicates that the target transform block does not contain edge information and becomes almost entirely zero after encoding. Similarly, the corresponding reference block will also be almost entirely zero. In this case, the effect of using intra-frame residuals and inter-frame residuals is the same. Therefore, intra-frame residuals can be used directly for encoding without motion estimation to obtain the L2 enhancement coefficients of the target transform block.

[0112] Correspondingly, if the block average gradient of the target transform block is greater than or equal to the average gradient of the current frame, the magnitudes of the intra-frame residual and the inter-frame residual can be further calculated and compared, and then it can be determined whether to use the intra-frame residual for encoding or the inter-frame residual for encoding.

[0113] In some embodiments of this application, before determining the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block, the block type of each block in the current frame of the base layer can be used as the block type of each transform block in the current frame of the enhancement layer 2. Then, when determining the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block, specifically, when the block type (type_blk) of the target transform block is not type I, the initial search block can be determined in the reconstructed frame of the previous frame based on the second motion vector of the target transform block.

[0114] If the target transform block's type_blk is an I block (i.e., the block type is I), it indicates that the content corresponding to the target transform block is temporarily added to the frame and does not have a temporal reference. Therefore, the intra-frame residual can be directly used for encoding without motion estimation to obtain the L2 enhancement coefficient of the target transform block.

[0115] In this application, the block involved is the basic unit of encoding. An image must first be divided into multiple blocks (4x4 pixels) before encoding processing can be performed. A macroblock should consist of an integer number of blocks, typically 16x16 pixels in size. Macroblocks usually correspond to three different block types: I type, P type, and B type.

[0116] Specifically, for macroblocks of type I, intra-frame prediction can only be performed using the decoded pixels in the current frame as a reference; for macroblocks of type P, intra-frame prediction can be performed using the previously decoded image as a reference image; and for macroblocks of type B, intra-frame prediction is performed using forward and backward reference images.

[0117] The above embodiments of this application take into account the special characteristics of residuals, namely, the reconstructed residuals obtained after encoding and reconstruction of residual data will only retain the edge information and a small amount of detail information in the original image, while the data corresponding to other parts are all zero. Therefore, it is not necessary to perform motion estimation on all blocks. Thus, based on the above gradient and block type, the transform blocks are filtered once, and motion estimation is performed on only some blocks, which can improve the encoding efficiency.

[0118] 403. Search for matching blocks with the initial search block as the origin.

[0119] In some embodiments of this application, searching for a matching block with the initial search block as the origin includes: performing a motion search of integer pixels with the initial search block as the origin to determine the matching block.

[0120] In related technologies, motion search to determine matching blocks involves not only integer-pixel motion search but also sub-pixel motion search, such as searching at 1 / 2-pixel intervals or 1 / 4-pixel intervals. In this embodiment, considering that the residual data is simpler and less complex than the original image, sub-pixel motion estimation is omitted during motion estimation, significantly reducing the search range and improving search efficiency.

[0121] In some embodiments of this application, a matching block is searched within a 3x3 grid with the initial search block as the origin. After the search is completed and the matching block is determined, the position of the matching block can be encoded to generate inter-frame prediction mode information. The position of the matching block can be represented by a target flag, which indicates the position of the matching block within the 3x3 grid.

[0122] Figure 5 This is a schematic diagram illustrating a motion search method provided in an embodiment of this application. Figure 5 As shown, the initial search block 51a can be determined in the previous frame 51 by using the target transformation block 52a and the second motion vector MV1 in the current frame 52, and the range of motion search is determined as a 3x3 grid 51b. Then, the motion search can be performed in the 3x3 grid 51b with the initial search block 51a as the origin, that is, traversing each block in the 3x3 grid 51b to determine the block most similar to the target transformation block as the matching block.

[0123] Since mv_blk is the scaled result of mv_base, it may not be completely accurate for L2 residuals. Therefore, it is still necessary to use it as the origin for a small-scale motion search. Searching the 9-grid can minimize the amount of motion vector data that needs to be saved.

[0124] In the original LCEVC of related technologies, the TemporalMask stores a flag indicating whether the corresponding transform block uses intra-frame residuals or inter-frame residuals. That is, there are originally two flags: one for intra-frame residuals and the other for inter-frame residuals. However, in this embodiment, since mv_blk can be obtained from the base layer's bitstream, only the position of the matching block needs to be stored in the enhancement layer. For a 9-grid search range, this means only 8 flags need to be added to the TemporalMask, resulting in 10 flags. One flag can be used to represent the intra-frame residual, and another flag can be used to represent the position of each block in the 9-grid corresponding to the inter-frame residual. This representation requires fewer bits compared to storing motion vector data in related technologies.

[0125] 404. Based on the reconstruction residual of the matching block and the L2 residual of the target transform block, determine the inter-frame residual and inter-frame cost of the target transform block.

[0126] In some embodiments of this application, before determining the intra-frame cost, or before determining the inter-frame residual and inter-frame cost based on the reconstruction residual of the matching block and the L-2 residual of the target transform block, sparsity-related information of the video sequence can be obtained. The sparsity-related information includes at least one of the following: the mean absolute value of the base layer residual, the average quantization parameter of the base layer, and the quantization step size of the enhancement layer. Then, an adaptive transform kernel is determined based on the sparsity-related information of the video sequence. The adaptive transform kernel includes: directional decomposition or directional decomposition squared. Afterwards, the video sequence can be transformed and inverse transformed based on the target transform kernel during the encoding process of the enhancement layer.

[0127] In some embodiments of this application, both inter-frame cost and intra-frame cost are Sum of Absolute Transformed Difference (SATD) costs. SATD is a measure of the size of video residual signals, which is the sum of the absolute values ​​of the prediction residuals of 4×4 blocks after the residuals have undergone a Hartmann transform. It can be regarded as a simple time-frequency transform, and its value can reflect the size of the generated bitstream to a certain extent. In the embodiments of this application, SATD is used to calculate inter-frame cost and intra-frame cost. Compared with SAD, SATD can more accurately describe the relationship of the bitstream required for the corresponding parts of the encoding.

[0128] In this embodiment of the application, intra-frame cost or inter-frame cost can be calculated based on an adaptive transform kernel.

[0129] In some embodiments of this application, the target cost includes intra-frame cost or inter-frame cost. Determining the target cost includes: transforming the target residual of the target transform block based on an adaptive transform kernel to obtain adaptive transform coefficients, wherein the target residual includes intra-frame residual or inter-frame residual; quantizing the adaptive transform coefficients to obtain the target symbol; dequantizing the target symbol and performing an inverse transform based on the adaptive transform kernel to obtain the reconstruction residual (resi_recon) of the target transform block; and determining the target cost based on the reconstruction residual of the target transform block, the L2 residual of the target transform block, and the target symbol.

[0130] The calculation of intra-frame cost is as follows: the intra-frame residual of the target transform block is transformed based on the adaptive transform kernel to obtain the adaptive transform coefficients; the adaptive transform coefficients are quantized to obtain the intra-frame symbols; the intra-frame symbols are dequantized and inversely transformed based on the adaptive transform kernel to obtain the intra-frame reconstruction residual (resi_recon) of the target transform block; the target cost is determined based on the intra-frame reconstruction residual of the target transform block, the L2 residual of the target transform block, and the intra-frame symbols.

[0131] Specifically, determining the target cost based on the intra-reconstruction residual of the target transform block, the L2 residual of the target transform block, and the intra-symbol can include: obtaining the intra-reconstruction (intra_recon) by adding the upsampled image of the current frame to the intra-reconstruction residual of the target transform block, and then calculating the intra-cost of each transform block based on the intra-reconstruction and the intra-symbol.

[0132] In this embodiment of the application, intra-frame residuals or inter-frame residuals can be encoded based on an adaptive transform kernel to obtain L2 enhancement coefficients.

[0133] In some embodiments of this application, the target residual includes intra-frame residual or inter-frame residual. Encoding the target residual of the target transform block to obtain the L2 enhancement coefficients of the target transform block includes: transforming the target residual of the target transform block based on an adaptive transform kernel to obtain adaptive transform coefficients; quantizing the adaptive transform coefficients to obtain target symbols; and entropy encoding the target symbols to obtain the L2 enhancement coefficients of the target transform block.

[0134] In the enhancement layer 2 provided in this application embodiment, an adaptive transform kernel can be used to process both transformation and inverse transformation during the encoding process, which can improve encoding performance.

[0135] 405. Determine whether the inter-frame cost is less than or equal to the intra-frame cost.

[0136] If the inter-frame cost is less than or equal to the intra-frame cost, perform step 406 below to encode the inter-frame residual of the target transform block to obtain the L2 enhancement coefficients of the target transform block.

[0137] If the inter-frame cost is greater than the intra-frame cost, perform step 407 below to encode the intra-frame residual of the target transform block to obtain the L2 enhancement coefficients of the target transform block.

[0138] 406. Encode the inter-frame residuals of the target transform block to obtain the L2 enhancement coefficients of the target transform block.

[0139] 407. Encode the intra-frame residual of the target transform block to obtain the L2 enhancement coefficients of the target transform block.

[0140] In this embodiment of the application, a low-cost encoding mode can be selected for encoding. When the inter-frame cost is low, the inter-frame residual can be encoded, and when the intra-frame cost is low, the intra-frame residual can be encoded.

[0141] The LCEVC-based coding method provided in this application embodiment performs motion estimation during temporal prediction of enhancement layer 2, and utilizes the MV of the base layer to perform motion estimation in enhancement layer 2. This can reduce coding difficulty, effectively reduce the bitrate, and improve the utilization rate of the bitrate of enhancement layer 2 while ensuring accurate temporal prediction.

[0142] This application embodiment corresponds to the above-described LCEVC-based encoding method, and correspondingly also provides an LCEVC-based decoding method, which includes, but is not limited to: decoding the L-2 coefficients of the target transform block in the bitstream of enhancement layer 2 to obtain the target residual of the target transform block; decoding the temporal layer bitstream to obtain prediction mode information; when the prediction mode information is inter-frame prediction mode information, scaling the first motion vector of the target block in the current frame of the base layer to obtain the second motion vector corresponding to the target transform block in the current frame of enhancement layer 2, wherein the inter-frame prediction mode information includes the position of the matching block of the target transform block; determining the reconstruction residual of the matching block based on the second motion vector of the target transform block and the position of the matching block; and determining the L-2 residual of the target transform block based on the reconstruction residual of the matching block and the target residual.

[0143] In some embodiments of this application, corresponding to the LCEVC-based encoding method described above, after determining the matching block, the encoding device can encode the position of the matching block to generate inter-frame prediction mode information. Correspondingly, after the decoding device parses the inter-frame prediction mode information, it can obtain the position of the matching block. The position of the matching block is represented by a target flag, which is used to indicate the position of the matching block within the 3x3 grid.

[0144] In some embodiments of this application, after performing entropy decoding on the temporal layer bitstream to obtain prediction mode information, the target residual can be determined as the L-2 residual of the target transform block when the prediction mode information is intra-frame prediction mode information.

[0145] The prediction mode information includes either intra-frame prediction mode information or inter-frame prediction mode information. This prediction mode information is entropy-encoded in the encoding device to generate a temporal bitstream, i.e., temporal information. After being transmitted to the decoding device, it undergoes entropy decoding, allowing the decoding device to obtain the prediction mode information.

[0146] In some embodiments of this application, after the encoding device determines the adaptive transform kernel, it performs transform and inverse transform processing on the video sequence during the encoding process of the enhancement layer based on the adaptive transform kernel, thereby obtaining the L2 enhancement coefficients after the enhancement layer 2 is encoded, the L1 enhancement coefficients after the enhancement layer 1 is encoded, and the time information obtained by entropy encoding the encoding information of the prediction information in the temporal prediction process in the L2 enhancement layer. Finally, the header information, time information, L2 enhancement coefficients, and L1 enhancement coefficients are all sent to the decoding device, and the base layer bitstream is also sent to the decoding device, so that the decoding device can combine the header information, time information, L2 enhancement coefficients, L1 enhancement coefficients, and base layer bitstream to recover the video sequence during the decoding process.

[0147] The LCEVC-based encoding and decoding methods provided in this application, based on the existing LCEVC standard, perform motion estimation in the enhancement layer 2 based on the MV of the base layer during encoding, and use corresponding decoding methods to decode and recover the video, effectively reducing the bit rate and achieving lower bit rate video transmission.

[0148] Figure 6 This is a schematic diagram illustrating an encoding method provided in an embodiment of this application. For example... Figure 6 As shown, the encoding method may include:

[0149] 601. Extract the residuals from the base layer bitstream, as well as the quantization parameters, motion vectors, and block types from the base layer bitstream.

[0150] For the currently encoded video sequence, information such as residuals, quantization parameters, motion vectors, and block types can be extracted from the basic layer bitstream of the video sequence.

[0151] 602. Calculate the mean of the absolute values ​​of the residuals of the base layer.

[0152] Specifically, the mean of the absolute values ​​of the basic layer residuals is calculated based on the residuals in the basic layer bitstream corresponding to the video sequence.

[0153] 603. Calculate the average quantization parameter of the base layer.

[0154] Specifically, the average quantization parameter of the base layer is calculated based on the quantization parameters in the base layer bitstream corresponding to the video sequence.

[0155] 604. Input the mean absolute value of the residuals of the base layer, the average quantization parameter of the base layer, and the quantization step size of the enhancement layer 2 into the target support vector machine to obtain the adaptive transformation kernel.

[0156] The adaptive transform kernel selection problem can be understood as a binary classification problem. That is, for a sequence, if the transform kernel DD can achieve higher compression efficiency, then it is determined to be classified into the DD class; if the transform kernel DDS can achieve higher compression efficiency, then it is classified into the DDS class.

[0157] Since Support Vector Machine (SVM) is a few-shot learning method that requires a small training set, it can solve binary classification problems well and has low inference complexity, making it suitable for the adaptive transformation kernel selection problem in the embodiments of this application.

[0158] The target support vector machine (SVM) is trained on a target training dataset. This dataset includes training data for multiple video sequences. Each video sequence's training data includes sparsity-related information extracted from the sequence, and a standard transform kernel corresponding to this sparsity-related information. This standard transform kernel is pre-determined and suitable for the specific video sequence; that is, encoding based on this standard transform kernel achieves higher compression efficiency. After training on the target training dataset, the resulting target SVM can quickly and accurately select and output an adaptive transform kernel based on the sparsity-related information of the input video sequences.

[0159] In this embodiment of the application, after the adaptive transform kernel is determined, the encoding process of enhancement layer 2 can continue to be executed, that is, the subsequent 605 to 616 are executed.

[0160] 605. Scale the first motion vector of the target block in the current frame of the base layer to obtain the second motion vector corresponding to the target transform block of the enhancement layer 2.

[0161] The description of 605 above can be referred to the description of 401 above, and will not be repeated here.

[0162] 606. Use the block type of each block in the current frame of the base layer as the block type of each transform block in the current frame of the enhancement layer 2.

[0163] Map the block type of each block in the base layer to the corresponding transform block in enhancement layer 2.

[0164] 607. Calculate the gradient of the L2 residual corresponding to each pixel in the target transform block.

[0165] Based on the residuals in the base layer bitstream extracted from the video sequence in step 601 (i.e., the residuals for each pixel in the base layer bitstream), the gradient of the L2 residual corresponding to each pixel in the target transform block is calculated.

[0166] 608. Based on the gradient of the L2 residual corresponding to each pixel in the target transform block, calculate the block average gradient of the target transform block and the frame-level average gradient of the current frame.

[0167] The block-averaged gradient of the target transform block is the average gradient of the L2 residual corresponding to each pixel in the target transform block. The frame-averaged gradient of the current frame is the average gradient of the L2 residual corresponding to each pixel in the current frame.

[0168] 609. Determine whether the block average gradient of the target transform block is greater than or equal to the average gradient of the current frame.

[0169] If the average gradient of the target transform block is less than the average gradient of the current frame, it indicates that the target transform block does not contain edge information. After encoding, it becomes almost all zero. Similarly, the corresponding reference block will also be almost all zero. At this time, the effect of using intra-frame residual and inter-frame residual is the same. Therefore, intra-frame residual can be used directly for encoding without motion estimation, that is, the following step 610 is executed.

[0170] If the average gradient of the target transform block is greater than or equal to the average gradient of the current frame, then steps 611 to 617 below can be executed to further determine the block type, calculate and compare the magnitudes of the intra-frame residual and the inter-frame residual, and then determine whether to use the intra-frame residual for encoding or the inter-frame residual for encoding.

[0171] 610. Encode the intra-frame residual of the target transform block to obtain the L2 enhancement coefficients of the target transform block.

[0172] Considering the special nature of residuals, that is, the reconstructed residuals obtained after encoding and reconstruction of residual data will only retain the edge information and a small amount of detail information in the original image, while the data corresponding to other parts are zero, it is not necessary to perform motion estimation on all blocks. Therefore, based on the above gradient and block type, the transform blocks are filtered once, and motion estimation is performed on only some blocks, which can improve the encoding efficiency.

[0173] 611. Determine whether the block type of the target transform block is type I.

[0174] If the block type (type_blk) of the target transform block is not type I, perform step 612 below; if the block type (type_blk) of the target transform block is type I, use the intra-frame residual directly for coding without motion estimation, that is, perform step 610 above.

[0175] 612. Determine the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block.

[0176] The description of step 612 above can be referred to step 402 above, and will not be repeated here.

[0177] 613. Using the initial search block as the origin and SATD as the cost, perform a motion search within the 3x3 grid to determine the matching block.

[0178] In this case, after filtering out some transform blocks by gradient and block type, the number of transform blocks that need to be motion searched is small, which can improve the efficiency of subsequent motion search and computation. Furthermore, when the number of transform blocks to be processed is small, using the more complex but more accurate SATD to perform motion search to calculate intra-frame cost and inter-frame cost can improve the accuracy of the calculation.

[0179] 614. Calculate the inter-frame residual, inter-frame cost, and intra-frame cost.

[0180] 615. Determine whether the inter-frame cost is less than or equal to the intra-frame cost.

[0181] If the inter-frame cost is less than or equal to the intra-frame cost, perform step 616 below to encode the inter-frame residual of the target transform block to obtain the L2 enhancement coefficient of the target transform block; if the inter-frame cost is greater than the intra-frame cost, perform step 610 above to encode the intra-frame residual of the target transform block to obtain the L2 enhancement coefficient of the target transform block.

[0182] 616. Encode the inter-frame residuals of the target transform block to obtain the L2 enhancement coefficients of the target transform block.

[0183] The LCEVC-based encoding and decoding methods provided in this application, based on the existing LCEVC standard, perform motion estimation in the enhancement layer 2 based on the MV of the base layer during encoding, and use corresponding decoding methods to decode and recover the video, effectively reducing the bit rate and achieving lower bit rate video transmission.

[0184] Figure 7 This is a schematic diagram of the hardware structure of a decoding device provided in an embodiment of this disclosure.

[0185] like Figure 7 As shown, the decoding device provided in this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements various processes of the LCEVC-based encoding method or the LCEVC-based decoding method in the above method embodiments, and achieves the same technical effect.

[0186] The aforementioned decoding device can be a decoding device:

[0187] In some embodiments, the processor 701 is configured to: decode the L2 enhancement coefficients of the target transform block in the bitstream of enhancement layer 2 to obtain the target residual of the target transform block; and decode the temporal bitstream to obtain prediction mode information.

[0188] When the prediction mode information is inter-frame prediction mode information, the first motion vector of the target block in the current frame of the base layer is scaled to obtain the second motion vector corresponding to the target transform block in the current frame of the enhancement layer 2. The inter-frame prediction mode information includes the position of the matching block of the target transform block. Based on the second motion vector of the target transform block and the position of the matching block, the reconstruction residual of the matching block is determined. Based on the reconstruction residual of the matching block and the target residual, the L2 residual of the target transform block is determined.

[0189] In some embodiments, after performing entropy decoding on the temporal layer bitstream to obtain prediction mode information, the processor 701 is further configured to: when the prediction mode information is intra-frame prediction mode information, determine the target residual as the L2 residual of the target transform block.

[0190] In some embodiments, the position of the matching block is represented by a target flag, which is used to indicate the position of the matching block within the 3x3 grid.

[0191] The aforementioned decoding device can be an encoding device:

[0192] In some embodiments, the processor 701 is configured to scale the first motion vector of the target block in the current frame of the base layer to obtain the second motion vector corresponding to the target transform block of the enhancement layer 2.

[0193] The initial search block is determined in the reconstructed frame of the previous frame based on the second motion vector of the target transform block;

[0194] The initial search block is used as the origin to search for matching blocks;

[0195] Based on the reconstruction residual of the matching block and the L2 residual of the target transform block, determine the inter-frame residual and inter-frame cost of the target transform block;

[0196] When the inter-frame cost is less than or equal to the intra-frame cost, the inter-frame residual of the target transform block is encoded to obtain the L2 enhancement coefficient of the target transform block.

[0197] In some embodiments, the processor 701 is further configured to: after determining the inter-frame residual and inter-frame cost of the target transform block based on the reconstruction residual of the matching block and the L2 residual of the target transform block, and if the inter-frame cost is greater than the intra-frame cost, encode the intra-frame residual of the target transform block to obtain the L2 enhancement coefficient of the target transform block.

[0198] In some embodiments, the processor 701 is specifically configured to: the search for a matching block with the initial search block as the origin includes: performing a motion search of integer pixels with the initial search block as the origin to determine the matching block.

[0199] In some embodiments, both the inter-frame cost and the intra-frame cost are SATD costs.

[0200] In some embodiments, the processor 701 is specifically configured to: the search for matching blocks with the initial search block as the origin includes: searching for matching blocks within a 3x3 grid with the initial search block as the origin;

[0201] The processor 701 is also configured to encode the position of the matching block to generate inter-frame prediction mode information.

[0202] In some embodiments, the position of the matching block is represented by a target flag, which is used to indicate the position of the matching block within the 3x3 grid.

[0203] In some embodiments, the processor 701 is further configured to: before determining the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block, determine the block average gradient of each transform block in the current frame of the enhancement layer 2 and the frame-level average gradient of the current frame; and determine the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block when the block average gradient of the target transform block is greater than or equal to the frame-level average gradient.

[0204] In some embodiments, the processor 701 is further configured to: before determining the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block, use the block type of each block in the current frame of the base layer as the block type of each transform block in the current frame of the enhancement layer 2; and when the block type of the target transform block is not type I, determine the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block.

[0205] In some embodiments, the processor 701 is further configured to: determine the intra-frame cost before encoding the inter-frame residual of the target transform block to obtain the L2 enhancement coefficients of the target transform block when the inter-frame cost is less than or equal to the intra-frame cost; before determining the intra-frame cost, or before determining the inter-frame residual and inter-frame cost based on the reconstruction residual of the matching block and the L2 residual of the target transform block, obtain sparsity-related information of the video sequence, the sparsity-related information including at least one of: the mean absolute value of the base layer residuals, the average quantization parameter of the base layer, and the quantization step size of the enhancement layer; and determine an adaptive transform kernel based on the sparsity-related information of the video sequence, the adaptive transform kernel including: directional decomposition or directional decomposition squared.

[0206] The processor 701 is also configured to perform transformation and inverse transformation processing based on the target transform kernel during the encoding of the video sequence in the enhancement layer, wherein the enhancement layer includes enhancement layer 2 and enhancement layer 1.

[0207] In some embodiments, the target cost includes the intra-frame cost or the inter-frame cost, and the processor 701 is specifically configured to: determine the target cost, including: transforming the target residual of the target transform block based on the adaptive transform kernel to obtain adaptive transform coefficients, the target residual including intra-frame residual or inter-frame residual; quantizing the adaptive transform coefficients to obtain a target symbol; dequantizing the target symbol and performing an inverse transform based on the adaptive transform kernel to obtain a reconstruction residual of the target transform block; and determining the target cost based on the reconstruction residual of the target transform block, the L2 residual of the target transform block, and the target symbol.

[0208] In some embodiments, the target residual includes intra-frame residual or inter-frame residual, and the processor 701 is specifically configured to: encode the target residual of the target transform block to obtain the L2 enhancement coefficients of the target transform block, including: transforming the target residual of the target transform block based on the adaptive transform kernel to obtain adaptive transform coefficients; quantizing the adaptive transform coefficients to obtain target symbols; and entropy encoding the target symbols to obtain the L2 enhancement coefficients of the target transform block.

[0209] In some embodiments, the processor 701 is specifically configured to: determine the adaptive transform kernel based on the sparsity-related information of the video sequence, including: inputting the sparsity-related information of the video sequence into a target support vector machine; obtaining the adaptive transform kernel output by the target support vector machine; wherein the target support vector machine is a support vector machine trained based on a target training dataset, the target training dataset including: training data for multiple video sequences, the training data for each video sequence including: sparsity-related information extracted from each video sequence, and a standard transform kernel corresponding to the sparsity-related information.

[0210] In some embodiments, the quantization step size of the enhancement layer includes: the quantization step size of enhancement layer 1, and / or, the quantization step size of enhancement layer 2.

[0211] This disclosure provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the various processes of the LCEVC-based encoding method or the LCEVC-based decoding method in the above-described method embodiments, and can achieve the same technical effect.

[0212] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0213] This disclosure provides a computer program product that stores a computer program. When the computer program is executed by a processor, it implements the various processes of the LCEVC-based encoding method or the LCEVC-based decoding method described in the above method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0214] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0215] In this disclosure, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0216] In this disclosure, memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0217] In this disclosure, computer-readable media includes both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0218] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0219] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A decoding method based on LCEVC, characterized in that, include: The L2 enhancement coefficients of the target transform block in the bitstream of enhancement layer 2 are decoded to obtain the target residual of the target transform block; The target transformation block is a certain part of the image region corresponding to the target block in the enhancement layer 2; Decode the time-layer bitstream to obtain prediction mode information; When the prediction mode information is inter-frame prediction mode information, the first motion vector of the target block in the current frame of the base layer is scaled to obtain the second motion vector corresponding to the target transform block in the current frame of the enhancement layer 2. The inter-frame prediction mode information includes the position of the matching block of the target transform block. The reconstruction residual of the matching block is determined based on the second motion vector of the target transform block and the position of the matching block; the position of the matching block is represented by a target flag, which is used to indicate the position of the matching block within the motion search range; The L2 residual of the target transform block is determined based on the reconstruction residual of the matching block and the target residual.

2. The method according to claim 1, characterized in that, After performing entropy decoding on the time-layer bitstream to obtain prediction mode information, the method further includes: When the prediction mode information is intra-frame prediction mode information, the target residual is determined as the L2 residual of the target transform block.

3. An encoding method based on LCEVC, characterized in that, include: The first motion vector of the target block in the current frame of the base layer is scaled to obtain the second motion vector corresponding to the target transform block of the enhancement layer 2; The initial search block is determined in the reconstructed frame of the previous frame based on the second motion vector of the target transform block; The initial search block is used as the origin to search for matching blocks; Based on the reconstruction residual of the matching block and the L2 residual of the target transform block, determine the inter-frame residual and inter-frame cost of the target transform block; When the inter-frame cost is less than or equal to the intra-frame cost, the inter-frame residual of the target transform block is encoded to obtain the L2 enhancement coefficient of the target transform block.

4. The encoding method according to claim 3, characterized in that, include: When the inter-frame cost is greater than the intra-frame cost, the intra-frame residual of the target transform block is encoded to obtain the L2 enhancement coefficient of the target transform block.

5. The method according to claim 3, characterized in that, The step of searching for matching blocks with the initial search block as the origin includes: A whole-pixel motion search is performed within the motion search range with the initial search block as the origin to determine the matching block; The method further includes: The position of the matching block is encoded to generate inter-frame prediction mode information; The position of the matching block is represented by a target flag, which indicates the position of the matching block within the motion search range.

6. The method according to claim 3, characterized in that, Both the inter-frame cost and the intra-frame cost are SATD costs.

7. The method according to any one of claims 3 to 6, characterized in that, Before determining the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block, the method further includes: Determine the block average gradient of each transform block in the current frame of enhancement layer 2 and the frame-level average gradient of the current frame; If the block average gradient of the target transform block is greater than or equal to the frame-level average gradient, an initial search block is determined in the reconstructed frame of the previous frame based on the second motion vector of the target transform block.

8. The method according to any one of claims 3 to 6, characterized in that, Before determining the initial search block in the reconstructed frame of the previous frame based on the second motion vector of the target transform block, the method further includes: The block type of each block in the current frame of the base layer is used as the block type of each transform block in the current frame of the enhancement layer 2; When the block type of the target transform block is not type I, the initial search block is determined in the reconstructed frame of the previous frame based on the second motion vector of the target transform block.

9. The method according to claim 3, characterized in that, Before encoding the inter-frame residual of the target transform block to obtain the L2 enhancement coefficients of the target transform block when the inter-frame cost is less than or equal to the intra-frame cost, the method further includes: Determine the intra-frame cost; Before determining the intra-frame cost, or before determining the inter-frame residual and inter-frame cost based on the reconstruction residual of the matching block and the L2 residual of the target transform block, the method further includes: Obtain sparsity-related information of the video sequence, wherein the sparsity-related information includes at least one of the following: the mean absolute value of the residual of the base layer, the average quantization parameter of the base layer, and the quantization step size of the enhancement layer; The adaptive transform kernel is determined based on the sparsity information of the video sequence; The adaptive transform kernel is used to perform transform and inverse transform processing on the video sequence during the encoding process in the enhancement layer. The enhancement layer includes enhancement layer 2 and enhancement layer 1.

10. The method according to claim 9, characterized in that, The target cost includes the intra-frame cost or the inter-frame cost. Determining the target cost includes: The target residual of the target transform block is transformed based on the adaptive transform kernel to obtain adaptive transform coefficients. The target residual includes intra-frame residual or inter-frame residual. The adaptive transform coefficients are quantized to obtain the target symbol; The target symbol is dequantized and inversely transformed based on the adaptive transform kernel to obtain the reconstruction residual of the target transform block; The target cost is determined based on the reconstruction residual of the target transform block, the L2 residual of the target transform block, and the target symbol.

11. The method according to claim 9, characterized in that, The target residual includes intra-frame residual or inter-frame residual. The target residual of the target transform block is encoded to obtain the L2 enhancement coefficients of the target transform block, including: The target residual of the target transform block is transformed based on the adaptive transform kernel to obtain adaptive transform coefficients; The adaptive transform coefficients are quantized to obtain the target symbol; The target symbol is entropy encoded to obtain the L2 enhancement coefficients of the target transform block.

12. The method according to claim 9, characterized in that, The step of determining the adaptive transform kernel based on the sparsity information of the video sequence includes: The sparsity information of the video sequence is input into the target support vector machine; Obtain the adaptive transformation kernel output by the target support vector machine, wherein the adaptive transformation kernel includes directional decomposition (DD) or directional decomposition squared DDS; The target support vector machine is a support vector machine trained on a target training dataset. The target training dataset includes training data for multiple video sequences. The training data for each video sequence includes sparsity-related information extracted from each video sequence, and a standard transform kernel corresponding to the sparsity-related information.

13. The method according to claim 12, characterized in that, The quantization step size of the enhancement layer includes: the quantization step size of enhancement layer 1, and / or, the quantization step size of enhancement layer 2.

14. A decoding device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the LCEVC-based decoding method as described in any one of claims 1 to 2, or the computer program, when executed by the processor, implements the LCEVC-based encoding method as described in any one of claims 3 to 13.

Images