Video compression using adaptive iterative intra-prediction

Abstract

A method of video decoding in a video decoder may include the step of receiving one or more syntax elements associated with a current first block belonging to a plurality of first blocks partitioned from a picture, wherein the one or more syntax elements represent an optimal partition indicating how the current first block is partitioned into second blocks for intra prediction, a set of block selection signals—in which the current first block is re-partitioned into third blocks, each block selection signal corresponds to one of the third blocks and indicates whether each third block is coded using a first coding method or a second coding method—and a set of compressed representations corresponding to each of the third blocks. The current first block may be reconstructed based on one or more syntax elements to generate a reconstructed current first block.

Description

Technology Field

[0001] The present disclosure claims priority to U.S. Provisional Application No. 63 / 142,377, filed on January 27, 2021, titled "Neural Image Compression Using Adaptive Iterative Intra Prediction," and to U.S. Patent Application No. 17 / 486,533, filed on September 27, 2021, titled "Video Compression Using Adaptive Iterative Intra Prediction." The disclosure of the prior applications is incorporated herein by reference in its entirety.

[0002] The present disclosure generally describes embodiments related to video coding based on artificial neural networks. Background Technology

[0003] The background description provided herein is intended to provide the context of the present disclosure in general. The work of the inventors currently named is not recognized as prior art to the extent described in this background section, and any aspects of the description that were not qualified as prior art at the time of filing are not recognized as prior art of the present disclosure, expressly or implied.

[0004] Neural networks are based on a collection of connected nodes (also called neurons) that loosely model the neurons of the biological brain. Neurons can be composed of multiple layers. Neurons in one layer can be connected to neurons in the immediately preceding layer and the immediately following layer.

[0005] Connections between two neurons, such as synapses in the biological brain, can transmit signals from one neuron to another. The neuron receiving the signal processes it and can transmit it to other connected neurons. In some examples, to find the output of a neuron, the input to the neuron is weighted by the weights of the connections from that input to the neuron, and the weighted inputs are summed to produce a weighted sum. A bias may be added to the weighted sum. Additionally, the weighted sum is passed through an activation function to generate the output.

[0006] The International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) Moving Picture Experts Group (MPEG) (JTC 1 / SC 29 / WG 11) are actively seeking potential demand for the standardization of future video coding technologies. The ISO / IEC Joint Photographic Experts Group (JPEG) established the JPEG AI Group, focusing on AI-based end-to-end neural image compression using neural networks (NN). Several companies have funded specialized research projects on Neural Image Compression (NIC). China's Audio Video Standard (AVS) also formed the AVS-AI Special Group to research neural image and video compression technologies. The success of recent approaches has led to increased industrial interest in advanced neural image and video compression methodologies.

[0007] A aspect of the present disclosure provides a video encoding method in a video encoder. The method comprises: a step of partitioning a picture into a plurality of first blocks; a step of processing a current first block belonging to a plurality of first blocks to generate the following output: an optimal partition indicating how the current first block is partitioned into second blocks for intra prediction, a set of block selection signals—in which the current first block is re-partitioned into third blocks, each block selection signal corresponds to one of the third blocks and indicates whether each third block is coded using a first coding method or a second coding method—and a set of compressed representations corresponding to each of the third blocks; and a step of encoding the output to generate coded bits corresponding to the picture.

[0008] In one embodiment, the estimated block of the current first block partitioned by the optimal partition is generated based on an iterative prediction estimation neural network (NN). The estimated block includes an estimated second block corresponding to the second block. In one embodiment, for each third block coded using the first coding method, the estimated second block of each third block is encoded based on a neural compression NN to generate a compressed representation of each third block. For each third block coded using the second coding method, intra-prediction is performed to generate a residual block for each second block in each third block, and the residual block of each third block is compressed based on a residual neural compression NN to generate a compressed representation of each third block.

[0009] In one embodiment, the intra prediction for generating residual blocks for each second block in each third block is based on one or more estimated blocks of the first block processed prior to the estimated second block of each second block in the estimated block of the current block and the estimated second block of the current first block processed prior to the current first block.

[0010] In one embodiment, the estimated block of the current first block partitioned into optimal partitions may be generated based on an iterative prediction estimation NN in the following steps: in the current iteration, using the current estimated second block set of the current first block and optionally the estimated block of the first block processed prior to the current first block as input to an intra prediction NN to generate a predicted second block corresponding to the current estimated second block of the current first block; generating a residual second block corresponding to the predicted second block; re-partitioning the set of residual second blocks into residual third blocks corresponding to the third block; compressing the residual third block based on a residual neural compression NN to generate a compressed third block; decompressing the compressed third block based on a residual neural decompression NN to recover the residual third block; and combining the recovered third block with the corresponding predicted second block to generate a reconstructed second block.

[0011] In one embodiment, for each third block, a first rate-distortion (RD) loss corresponding to the third block encoded by the first coding method and a second RD loss corresponding to the third block encoded by the second coding method are determined. Based on the first RD loss and the second RD loss, a block selection signal of the third block may be determined to indicate which of the first coding method and the second coding method is selected to code the third block.

[0012] In one embodiment, the processing step further comprises: determining a first overall loss of the current first block corresponding to an optimal partition for partitioning the current first block into second blocks—the first overall loss is a combination of the RD loss of each third block corresponding to one selected from a first coding method and a second coding method for each third block—determining a second overall loss of the current first block corresponding to each of a plurality of different methods other than the optimal partition for partitioning the current first block into second blocks; and comparing the first overall loss and the second overall loss to determine that the first overall loss of the current first block corresponding to the optimal partition is the smallest loss among the first overall losses and the second overall losses.

[0013] A part of the present disclosure further provides a video decoding method in a video decoder. The method may include receiving one or more syntax elements associated with a current first block belonging to a plurality of first blocks partitioned from a picture, wherein the one or more syntax elements may include: an optimal partition indicating how the current first block is partitioned into second blocks for intra prediction, a set of block selection signals indicating that the current first block is re-partitioned into third blocks, each block selection signal corresponding to one of the third blocks and each third block is coded using a first coding method or a second coding method, and a set of compressed representations corresponding to each of the third blocks. The current first block may be reconstructed based on one or more syntax elements to generate a reconstructed current first block.

[0014] In one embodiment, the reconstruction comprises: generating a third block reconstructed based on a neural decompression NN, wherein for each third block coded using a first coding method indicated by each of a block selection signal set, each compressed representation of the third block is input to the neural decompression neural network (NN); and generating a residual third block based on the residual neural decompression NN, wherein for each third block coded using a second coding method indicated by each of a block selection signal set, each compressed representation of the third block is input to the residual neural decompression NN.

[0015] In one embodiment, the reconstruction further includes the step of generating a reconstructed third block based on each residual third block and each predicted third block for each third block coded using a second coding method.

[0016] In one embodiment, the reconstruction further includes the step of performing an intra prediction based on an intra prediction NN, for each third block coded using a second coding method, using previously reconstructed first blocks and previously generated reconstructed third blocks as inputs to the intra prediction NN, to generate a predicted second block set corresponding to each third block.

[0017] In one embodiment, a set of predicted second blocks corresponding to each third block forms each predicted third block. The reconstruction further includes a step of, for each third block coded using a second coding method, performing each intra prediction as an optimal partition, and indicating a manner in which the current first block is partitioned into second blocks for intra prediction as an input to the intra prediction NN. In one embodiment, the reconstruction further includes a step of merging the reconstructed third blocks corresponding to the third blocks coded using a first coding method or a second coding method to generate a reconstructed current first block.

[0018] An aspect of the present disclosure further provides a computer-readable non-transient medium that stores instructions that, when executed by a processor, cause the processor to perform a video encoding or decoding method. Brief explanation of the drawing

[0019] Additional features, characteristics, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and attached drawings. FIG. 1 illustrates an exemplary workflow (100) of an encoder according to an embodiment of the present disclosure. FIG. 2 illustrates an exemplary workflow (200) of a repeat partition selection module (102) according to an embodiment of the present disclosure. FIG. 3 illustrates an exemplary workflow (300) of a decoder according to an embodiment of the present disclosure. FIG. 4 illustrates an exemplary workflow (400) of an iterative prediction estimation module (201) in the example of FIG. 2 according to an embodiment of the present disclosure. FIG. 5 illustrates an exemplary workflow (500) of a training process according to an embodiment of the present disclosure. FIG. 6 illustrates a video encoding process (600) according to an embodiment of the present disclosure. FIG. 7 illustrates a video decoding process (700) according to an embodiment of the present disclosure. FIG. 8 illustrates a computer system (800) suitable for implementing a specific embodiment of the disclosed subject matter. Specific details for implementing the invention

[0020] I. Neural Image Compression and Block-Based Intra Prediction and Residual Coding

[0021] The present disclosure provides a Neural Image Compression (NIC) framework that compresses input images by a Neural Network (NN) (or a set of NNs) using a block-based intra-prediction mechanism having an adaptive block size. Additionally, residual coding and intra-prediction may be performed based on different block partitions. Furthermore, the NIC framework may adaptively select one of two candidate coding methods for each coding block. While image encoding and decoding examples are described for the sake of illustration of specific aspects of the present disclosure, the techniques described herein may be applied to the field of video coding and decoding directly or through adaptations based on the principles disclosed herein.

[0022] In some embodiments, given an input image x, the goal of the NIC is to use image x as input to an NN encoder to obtain a compact, compressed representation for storage and transmission. After calculating, Using as input to the NN decoder, the image It is to reconstruct. In some examples, the NIC method may take the VAE (Variational Autoencoder) structure, where the NN encoder directly uses the entire image x as input, and this is the output representation It is passed through a set of network layers acting as a black box to compute. Accordingly, the NN decoder computes the entire representation It uses as input, and this is reconstructed It is passed through another set of network layers that act like other black boxes to calculate.

[0023] In some embodiments, instead of directly encoding the entire original image, a block-based intra-prediction and residual coding mechanism may be adopted, which is highly effective for compressing image frames in a video coding system. In one example, the entire image is partitioned into blocks of various sizes, prediction blocks are generated by copying boundary pixels of previously compressed blocks along various angular directions, and then the residuals between the original blocks and the prediction blocks are compressed. Residuals can be encoded much more efficiently than original pixels, resulting in better coding performance. Different block sizes can have a direct impact on compression performance, and the optimal block size generally depends on the specific image.

[0024] II. Neural Image Compression Using Intra-Prediction Residuals

[0025] The present disclosure provides a NIC framework that uses block-based intra-prediction with an adaptive block size. The NIC framework may encode residuals between a prediction block and an original block (or an estimated block derived based on the original block) instead of encoding the original pixel, and the block size is adaptively determined based on compression quality, such as rate-distortion (RD) loss.

[0026] FIG. 1 illustrates an exemplary workflow (100) of an encoder according to an embodiment of the present disclosure. The encoder may operate as a video encoder or an image encoder. The encoder may include a partition module (101) and an iterative partition selection module (102). An input image x is first in the partition module (101) It can be partitioned into k microblocks denoted, where m i represents the i-th microblock. Each m i Is ( w m , h m It can have the size of ) and additionally blocks It can be partitioned as, Is m i It is the j-th block of. Block The size of the blocks is different It can be diverse or the same, and other micro blocks m i It can be partitioned in different ways. In one embodiment, microblocks are aligned with Coding Tree Unit (CTU) partitions in current video coding tools such as High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC). Each CTU microblock is or There may be an option to further partition into blocks having various sizes, such as pixels. The present disclosure does not place any limitations on the choice of the size of the CTU microblocks or the method by which the blocks of the CTU microblocks are partitioned.

[0027] Each micro block m i Regarding, for example, microblocks for intra-prediction m i Partitioning into blocks (referred to as intra-prediction blocks) P It is assumed that there may be several different methods. The repeating partition selection module (102) is a micro block m i The optimal method for partitioning ( Can determine (indicated as ).

[0028] In addition, for example, for residual coding or switching coding methods, microblocks m i Re-partitioning may be performed in the workflow (100) to partition into blocks (referred to as re-partitioned blocks). Due to these two partitions—partition for intra-prediction and re-partition for residual coding and switching coding methods—intra-prediction and residual coding may be performed based on blocks of different sizes. Generally, intra-prediction may have higher efficiency with a smaller acceptable block size, while residual coding may have higher efficiency with a larger acceptable block size. The coding framework disclosed herein provides the ability to separate the intra-prediction block size from the residual coding block size, thereby improving image or video coding efficiency.

[0029] In some embodiments, for each repartitioned block, the iterative partition selection module (102) may select a coding method from two candidate coding methods for encoding each repartitioned block. For example, the selection may be based on a Rate-Distortion (RD) based quality loss calculated for each of the two candidate methods for coding the same repartitioned block. The result of this selection may be indicated by a block selection signal associated with each repartitioned block. For example, a block selection signal having a first value (e.g., 1) may indicate that the first coding method is applied to each repartitioned block, while a block selection signal having a second value (e.g., 0) may indicate that the second coding method is applied to each repartitioned block.

[0030] As illustrated in FIG. 1, the output of the iterative partition selection module (102) is the optimal partition method , block selection signal Set and compressed expression of Includes a set of, indicates the number of repartitioned blocks. Each compressed representation may correspond to one repartitioned block. This output can be transmitted to a decoder side or a storage system (typically after being further compressed, for example, by quantization and entropy coding).

[0031] FIG. 2 illustrates an exemplary workflow (200) of a repeat partition selection module (102) according to an embodiment of the present disclosure. In the example of FIG. 2, the iterative partition selection module (102) includes an iterative prediction estimation module (201), an intra-prediction module (211), a residual neural compression module (213), a residual neural decompression module (214), a compute residual compression loss module (216), a neural compression module (221), a neural decompression module (222), a compute compression loss module (223), a block selection module (231), a compute partition loss module (232), a first addition module (212), and a second addition module (215). These elements are combined together as shown in FIG. 2.

[0032] the original input x Microblocks m i of p Let's assume that the intra-predicted blocks obtained by the i-th partitioning method are displayed. In one example, the total number of blocks for this partition It can be determined by the size of the microblock and the size of the intra-prediction block. As mentioned above, the microblock m i partitioning P There may be several different methods. For each partitioning method, the resulting intra-prediction block may have the same size or different sizes. pFor each partition method indicated by , the workflow (200) has a total quality loss It can be executed once to determine. Based on this overall quality loss, microblocks m i Optimal method for partitioning ( (indicated by) can be determined from P different partitioning methods.

[0033] The iterative prediction estimation module (201) uses an iterative intra-prediction mechanism to the original microblock m i It can be utilized to first compute the estimated microblock corresponding to. The estimated microblock is the original intra-predicted block Corresponding to, It may include an intra prediction block indicated by . The estimated intra prediction block may be used as a better basis for subsequent processing of the workflow (200) than the original intra prediction block in terms of coding efficiency and performance (e.g., indicated by RD loss).

[0034] Next, each estimated partitioned block Regarding, predicted block This can be calculated by the intra prediction module (211) based on this prediction network (NN for intra prediction). The prediction network Previously encoded current Estimated block in A set of image pixels from is used as input. In some embodiments, the input to the prediction network is also It may come from previously encoded estimated microblocks. In different examples, reconstructed microblocks may be used as input to the intra prediction module (211) instead of the estimated microblocks.

[0035] There can be many ways to select pixels and form the input for a prediction network. For example, in a context area The spatially nearest neighbor pixels can be arranged in any order (stacking, concatenation, spatial transformation, etc.) to form inputs for the prediction network. The intra prediction module (211) may choose to skip the prediction process of blocks on the boundary without sufficient pixel context. Alternatively, the intra prediction module (211) may pad microblocks for the prediction of blocks on the boundary. The prediction network predicts the blocks through inference computation. Prints.

[0036] The prediction network may have various architectures in various embodiments. For each partitioning scheme, the prediction network may use different NN models for prediction in some embodiments. Generally, convolutional and fully connected layers may be used. The present disclosure makes no restrictions on the size and shape of the context region for pixel selection, the manner in which pixels are converted into inputs to the prediction network, or the network architecture of the prediction network.

[0037] block Regarding, predicted block After calculating, the residual is, for example, by subtraction in the additional block (212). and It can be calculated based on. The entire i-th microblock partitioned in the p-th manner m i Let's assume we are displaying the residuals. Residuals go q dog residual blocks It can be repartitioned into sets of. Theseq The number of residual blocks is called the repartitioned residual blocks. q Repartitioning of the residual blocks is The original partitioning of the predicted blocks (or intra-predicted blocks) may be the same or different. When q=1, the entire microblock will be treated as a single piece. The corresponding estimated microblock and predicted blocks As shown in FIG. 2, each (referred to as the repartitioned estimated block) and It can be repartitioned in the same way as (referred to as the repartitioned predicted block).

[0038] Each repartitioned residual block For this, the residual neural compression module (213) is each residual block By compressing, the compressed expression It calculates, which is the recovered residual block To calculate, it is decompressed by the residual neural decompression module (214). The recovered residual block is converted into the corresponding repartitioned predicted block. In addition to, the reconstructed block You can obtain.

[0039] Residual compression quality loss In this residual compression loss calculation module, the repartitioned estimated block (including estimated intra-prediction blocks), reconstructed blocks and compressed expression It can be calculated based on. For example, in one embodiment, Rate-Distortion (RD) loss is a quality measure:

[0040] (1)

[0041] It can be calculated as, where, Is It is a distortion between. compressed It is a rate loss measured by bit consumption. λ is a trade-off hyperparameter that balances the importance of different terms. Other compression quality losses may be used herein. This disclosure makes no limitations on the specific measurements used for compression quality loss, distortion, or rate loss.

[0042] At the same time, each repartitioned estimated block (Including the original estimated intra-prediction block) is directly compressed by the neural compression module (221) to form a compressed representation It can calculate, and this is the recovered block It is decompressed by the neural decompression module (222) to directly calculate it.

[0043] Compression quality loss Residual quality loss In the same way as , and It can be calculated in the compression loss calculation module (2230) based on. and Based on, the block selection module (231) selects a selection signal By generating repartitioned residual blocks, for example, by selecting an option with less quality loss or the original repartitioned estimated block This compressed or Indicates whether it is used to generate. This is the current j-th block Optimal quality loss for compression It provides, for example, am.

[0044] The partition loss calculation module (232) is a micro block of p Total quality loss for the th partition method Next:

[0045] (2)

[0046] It can be calculated as follows, and each Is These are weights associated with each other. In one example, all weights are simply set to 1 to treat all blocks equally. In another example, some blocks may be treated with more attention than others, and an attention map (or significance map) may be used to obtain the weights.

[0047] every P If the same process (workflow (200)) is repeated for each partition method, quality loss This occurs. Optimal partitioning method For example, optimal loss: , It can be selected as a partition having. Corresponding block selection signal It can also be determined by the output of the repeat partition selection module (102).

[0048] This micro block m i Let's assume that the optimally partitioned block selected for is displayed. Block selection signal Depending on, the corresponding compressed or Ga also blocks Actual compressed representation of It can be decided to become this. Compressed expression A set of optimal partitions can also be output from the iterative partition selection module (102). , compressed expression and block selection signal Additionally, for example, it is encoded through quantization and entropy encoding to generate encoded bits in the encoded stream and transmitted to the decoder. In some examples, additionally encoded, optimal partition , compressed expression and block selection signal It is represented by some syntax elements that are encoded and also transmitted to the decoder.

[0049] The neural compression module (221), neural decompression module (222), residual neural compression module (213), and residual neural decompression module (214) may use any neural compression method. The present disclosure does not place any restrictions on the specific method or network architecture used in these modules.

[0050] FIG. 3 illustrates an exemplary workflow (300) of a decoder according to an embodiment of the present disclosure. The decoder may operate as a video decoder or an image decoder. The decoder may include a residual neural decompression module (301), a neural decompression module (302), an intra-prediction module (303), a merging module (305), and an additional module (304). These components are combined together as shown in FIG. 3.

[0051] In one embodiment, on the decoder side, the system (decoder) of FIG. 3 is an optimal partition , compressed expression and block selection signal Receives (typically recovered from the received bitstream by entropy decoding and inverse quantization). Each block selection signal Based on, the system selects one of the following methods for the recovered blocks Calculates the signal The compressed representation at the encoder side (the encoder in Fig. 1) (Repartitioned estimated block If it indicates that a recovered block is obtained from a decoded block based on (corresponding to), the neural decompression module (302) obtains the recovered block It can be used to calculate.

[0052] signal The residual representation of the recovered block compressed on the encoder side (Repartitioned residual block If it indicates that it has come as a decoded block based on (corresponding to), the residual neural decompression module (301) recovers the residual It can be used to calculate.

[0053] In one embodiment, the recovered residual In cases where is used, the previously reconstructed block Set of and previously recovered microblocks Based on this, the intra prediction module (303) uses a prediction network in the same way as the encoder side to predict the intra prediction block. Calculate. In one embodiment, the difference between the operation of the intra prediction module (303) and the intra prediction module (211) is that the input to the prediction network is formed by pixels of the estimated block and microblock on the encoder side, but comes from the corresponding recovered block and microblock on the decoder side.

[0054] Predicted intra predicted block Based on, repartitioned predicted block It can be obtained in the same way as the encoder side. Recovered residual is a predicted block that has been repartitioned (in the same way as the encoder side) in the additional module (304). Blocks restored by being added back to You can obtain.

[0055] or is the actual recovered block It provides, and the decoder moves to process the next block repartition.

[0056] Finally, micro blocks m i The recovered block of (Reconstructed The image reconstructed in the merge module (305) from the result) It can be aggregated as such. In some embodiments, the merging module (305) may further process the recovered block to remove artifacts such as deblocking, denoising, etc. The present disclosure describes an image in which the recovered block is reconstructed. No restrictions are placed on the specific method of aggregation.

[0057] FIG. 4 illustrates an exemplary workflow (400) of an iterative prediction estimation module (201) in the example of FIG. 2 according to an embodiment of the present disclosure. The iterative prediction estimation module (201) may include an intra-prediction module (401), an additional module (402), a residual neural compression module (403), a residual neural decompression module (404), and an additional module (405). These components are combined together as shown in FIG. 4.

[0058] In the workflow (400), the estimated intra-prediction block of the microblock can be determined in an iterative manner. The estimated intra-prediction block determined in this manner can provide a better basis for subsequent intra-prediction and residual block generation in the workflow (200) compared to directly using the original microblock for intra-prediction and residual block generation. For example, the resulting residual block based on the estimated block can contain fewer residuals and can be coded with fewer bits. Thus, the RD performance of the encoder in the example of FIG. 2 can be improved.

[0059] t For the nth iteration, the currently estimated block Assume there is a set of. In one embodiment, for the first iteration, (Current partition Ro) is It is initialized to. Each estimated block Regarding, predicted block This can be calculated by the intra-prediction module (401) based on the prediction network illustrated in FIG. 4. The prediction network Previously encoded estimated block A set of image pixels selected from is taken as input. As in the example of FIG. 2, in some embodiments, the selected pixels are also It can come from previously encoded estimated microblocks. The prediction network predicts blocks through inference computation. It outputs the residual in the addition module (402). Then, the residual go and It can be calculated based on.

[0060] Total residual go q 2 repartitioned blocks It can be repartitioned into a set of. Corresponding estimated microblocks and predicted blocks In the same way and Each can be repartitioned as follows. Each residual block Regarding, compressed expression To calculate, the recovered residual block can be compressed by the residual neural compression module (403). To calculate, it can be decompressed by the residual neural decompression module (404). The recovered residual block is a reconstructed block Corresponding repartitioned prediction block to obtain It can be added again to.

[0061] Finally, all reconstructed blocks are reconstructed estimated blocks that are retransmitted as input to the prediction network. It is from, and the system is the next iteration tProceed with +1. For example, a number of iterations are generally performed until the estimated block converges or the maximum number of iterations is reached. The final reconstructed estimated block generated in the last iteration is the output estimated block of the iteration prediction estimation module (201). am.

[0062] Note that the iterative prediction estimation module (201) can mimic the entire encoding and decoding process in some other embodiments. That is, each iteration t For this, the iterative prediction estimation module (201) executes the entire encoding and decoding procedure. In the encoding process described in the example of FIG. 2, silver (or, in other forms) It can be replaced with ), and the optimal partition (Current partition method) ), compressed expression and block selection signal It outputs. Then, in the decoding process described in the example of Fig. 3, the reconstructed image is the input for the next iteration. As such, it is supplied again to the iterative prediction estimation module (201).

[0063] III. Training Process

[0064] FIG. 5 illustrates an exemplary workflow (500) of a training process according to an embodiment of the present disclosure. The training process may be performed for the following components: a partition module (511), an iterative partition selection module (512), a residual neural decompression module (521), a neural decompression module (522), an addition module (523), an intra-prediction module (524), a merging module (525), an additional loss calculation module (502), and a total loss calculation (501). These components are combined together as shown in FIG. 5.

[0065] The goal of the training process is to learn or train the neural network (or neural network model) used in the examples of FIGS. 1 through 4. For example, the neural networks of modules (521, 522, 524, and 525) can be used in modules (301, 302, 303, and 305), respectively, after training. The neural network of module (512) (which may have a structure similar to that of module (102)) can be used in module (102) (which includes modules (201, 211, 213, 214, 221, 222, and 231)). Module (201) includes modules (401, 403, and 404).

[0066] In some embodiments, intra-prediction modules (211, 303, and 401) may share the same neural network; residual neural compression modules (213, and 403) may share the same neural network; residual neural decompression modules (214, 301, and 404) may share the same neural network; and neural decompression modules (222, and 302) may share the same neural network. In other embodiments, the above modules may not share the same neural network.

[0067] In some embodiments, the merging modules (305, 525) and the block selection module (231) are implemented as neural networks. For example, the NN is used to aggregate the recovered blocks into the recovered images. Corresponding trainable parameters may also be learned during the training process. In other embodiments, the merging modules (305, 525) and the block selection module (231) may not be implemented using neural networks. Therefore, during the training process, the corresponding neural networks are not trained.

[0068] In one embodiment, during the training process, the weight coefficients of the network and module to be learned may be initialized, for example, by using a pre-trained model or by setting their parameters to random numbers. Then, given an input training image x, each microblock is processed through the encoding process described in FIG. 1 (passing through modules (511 and 512) of FIG. 5). m i Optimal partition for , compressed expression and block selection signal Calculating, and the final reconstructed image To generate, the decoding process described in Fig. 3 (passing through modules (521 and 523-525 or 522 and 524-525)) follows.

[0069] In one embodiment, distortion loss ) can be calculated as a general Peak Signal-to-Noise Ratio (PSNR), Multiscale Structural Similarity (MS-SSIM), or a weighted combination of both. In one embodiment, rate loss is a compressed expression It can be calculated to measure the bit consumption of. Therefore, the total RD loss It can be calculated in the total loss calculation module (501):

[0070] (1)

[0071] Here, , is a hyperparameter that balances the importance of different terms.

[0072] In some embodiments, reconstructed residuals and the original residual Distortion loss between Other forms of loss and distortion loss such as It can also be calculated in an additional loss calculation module (502), for example, as a Mean Square Error (MSE) or Structural Similarity (SMS) measurement. and Also optionally It can be combined with ) as the final loss of the entire system.

[0073] The gradient of the total loss or final loss can be calculated and backpropagated to update learnable parameters in the system. Different components (e.g., networks or modules) can be updated at different times with different update frequencies. In some embodiments, some components or parts of the parameters of some components may be pre-trained and fixed, and the training process updates only the remaining parameters.

[0074] The technique disclosed herein provides a flexible framework for improving the performance of neural image compression. Multiple partitioning schemes can be tested for intra-prediction to select the optimal partition that adapts to the characteristics of the image being processed. Additionally, since the partition for intra-prediction residual generation and the partition for residual compression are separated, the optimal partition can be determined independently and applied to intra-prediction and residual compression. Furthermore, two coding methods can be adaptively selected for the re-partitioned blocks based on the quality loss of the two coding methods.

[0075] IV. Video Coding Process Based on an Adaptive Iterative Intra-Prediction Framework

[0076] FIG. 6 illustrates a video encoding process (600) according to an embodiment of the present disclosure. The process (600) may be performed in an encoder of the example of FIG. 1. The process (600) may proceed from (S601) to (S610).

[0077] In (S610), the picture of the video sequence may be partitioned into a plurality of first blocks. In one example, the first block may be aligned with the CTU partitioned in the figure.

[0078] In (S620), the current first block belonging to a plurality of first blocks can be processed to generate the following output: an optimal partition indicating how the current first block is partitioned into second blocks for intra prediction, a set of block selection signals - the current first block is re-partitioned into third blocks, each block selection signal corresponds to one of the third blocks and indicates whether each third block is coded using a first coding method or a second coding method - and a compressed representation set corresponding to each of the third blocks.

[0079] In one example, the estimated blocks of the current first block partitioned by the optimal partition may first be generated based on an iterative prediction estimation NN. The estimated blocks may include estimated second blocks corresponding to the second block. In one embodiment, the estimated blocks of the current first block may be generated in an iterative manner. For example, in the current iteration, a set of currently estimated second blocks of the current first block and optionally the estimated blocks of first blocks processed before the current first block may be used as input to an intra-prediction NN for first generating predicted second blocks corresponding to the currently estimated second blocks of the current first block.

[0080] Residual second blocks corresponding to the predicted second blocks can be generated. The set of residual second blocks can then be repartitioned into residual third blocks corresponding to the third blocks. Then, the residual third blocks can be compressed based on a residual neural compression NN to generate compressed third blocks. The compressed third blocks can be decompressed based on a residual neural decompression NN to recover the residual third blocks. The recovered residual third blocks can be combined with the corresponding predicted second blocks to generate reconstructed second blocks. Finally, the reconstructed second blocks can be used as input for the next iteration instead of the currently estimated set of second blocks.

[0081] In one embodiment, for each third block coded using the first coding method, the estimated second blocks of each third block may be compressed based on a neural compression NN to generate each compressed representation of each third block. For each third block coded using the second coding method, intra-prediction may be performed to generate predicted second blocks. Based on the predicted second blocks, residual blocks for each second block may be generated in each third block. The residual blocks of each third block may be compressed based on a residual neural compression NN to generate each compressed representation of each third block.

[0082] In one embodiment, the intra prediction for generating residual blocks for each second block in each third block may be based on (1) estimated second blocks processed prior to the estimated second block of each second block in the estimated block of the current first block, and (2) one or more estimated blocks of first blocks processed prior to the current first block.

[0083] In one embodiment, for each third block, a first rate-distortion (RD) loss corresponding to the third block encoded by the first coding method and a second RD loss corresponding to the third block encoded by the second coding method may be generated. Then, based on the first RD loss and the second RD loss, a block selection signal of the third block may be generated indicating which of the first coding method and the second coding method is selected to code the third block.

[0084] In one embodiment, a first total loss of the current first block corresponding to an optimal partition for partitioning the current first block into second blocks may be generated. The first total loss may be a combination of the RD loss of each third block corresponding to one selected from the first coding method and the second coding method for each third block. Additionally, to partition the current first block into second blocks, a second total loss of the current first block corresponding to each of a plurality of different methods, in addition to the optimal partition, may be generated. By comparing the first total loss and the second total loss, it may be determined that the first total loss of the current first block corresponding to the optimal partition is the smallest loss among the first total loss and the second total loss.

[0085] At (S630), the output may be encoded, for example, by quantization and / or entropy coding to generate coded bits corresponding to the picture. The coded bits may be transmitted as a bitstream corresponding to the video sequence or stored in a storage medium. The process (600) may proceed to (S699) and terminate.

[0086] FIG. 7 illustrates a video decoding process (700) according to an embodiment of the present disclosure. The process (700) may be performed in a decoder in the example of FIG. 3. The process (700) may proceed from (S701) to (S710).

[0087] In (S710), one or more syntax elements associated with the current first block may be received. The first block belongs to a plurality of first blocks partitioned from the picture. One or more syntax elements represent an optimal partition indicating how the current first block is partitioned into second blocks for intra-prediction, a set of block selection signals—where the current first block is re-partitioned into third blocks, each block selection signal corresponds to one of the third blocks and indicates whether each third block is coded using a first coding method or a second coding method—and a set of compressed representations corresponding to each of the third blocks.

[0088] In (S720), the current first block can be reconstructed based on one or more syntax elements and consequently becomes the reconstructed current first block. In one embodiment, for each third block coded using a first coding method indicated by each of the block selection signal sets, a reconstructed third block can be generated based on a neural decompression NN that takes each of the compressed representations of the third blocks as input to a neural decompression NN.

[0089] For each third block coded using a second coding method directed by each of the block selection signal sets, a residual third block can be generated based on a residual neural decompression NN that takes each compressed representation of the third blocks as input to a residual neural decompression NN. For each third block coded using the second coding method, a reconstructed third block can be generated based on each residual third block and each predicted third block.

[0090] In one embodiment, for each third block coded using the second coding method, an intra prediction can be performed based on an intra prediction NN that takes the previously reconstructed first blocks and the previously generated reconstructed third blocks as inputs to the intra prediction NN to generate a predicted second block set corresponding to each third block. The predicted second block set corresponding to each third block forms each predicted third block. In one embodiment, for each third block coded using the second coding method, each intra prediction is performed using an optimal partition as input to the intra prediction NN.

[0091] In one embodiment, a reconstructed current first block is created by merging the reconstructed third blocks corresponding to the third blocks coded using the first coding method or the second coding method. The process (700) may proceed to (S799) and terminate.

[0092] V. Computer Systems

[0093] The technology disclosed herein (e.g., functions, modules, elements, methods, processes, operations in various embodiments or examples) may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In some examples, the technology may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media.

[0094] Computer software may be coded using any suitable machine code or computer language, which can generate code containing instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., by applying assembly, compilation, linking, or similar mechanisms.

[0095] The command can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game devices, Internet of Things devices, etc.

[0096] FIG. 8 illustrates a computer system (800) suitable for implementing any embodiment of the disclosed subject. The components illustrated in FIG. 8 for the computer system (800) are essentially exemplary and are not intended to suggest any limitations on the scope of use or function of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependencies or requirements related to any one or combination of the components illustrated in the exemplary embodiments of the computer system (800).

[0097] The computer system (800) may include a specific human interface input device. This human interface input device may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, applause), visual input (e.g., gestures), and olfactory input (not shown). The human interface device may also be used to capture specific media that are not directly related to conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., 2D video, 3D video including stereoscopic video).

[0098] The input human interface device may include one or more of a keyboard (801), a mouse (802), a trackpad (803), a touch screen (810), a data glove (not shown), a joystick (805), a microphone (806), a scanner (807), and a camera (808) (only one of each shown).

[0099] The computer system (800) may also include specific human interface output devices. These human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light and smell / taste. These human interface output devices may include a tactile output device (e.g., a touch screen (810), a data glove (not shown), or a tactile feedback device that includes tactile feedback via a joystick (805) but does not function as an input device), an audio output device (e.g., a speaker (809), headphones (not shown)), a visual output device (e.g., a screen (810) including a CRT screen, an LCD screen, a plasma screen, or an OLED screen, each of which may or may not have a touch screen input function, each of which may or may not have a tactile feedback capability, some of which may or may not be capable of three-dimensional or higher output or two-dimensional visual output via stereographic output means such as virtual reality glasses (not shown), a holographic display and a smoke tank (not shown)), and a printer (not shown). These visual output devices (e.g., a screen (810)) may be connected to a system bus (848) via a graphics adapter (850).

[0100] The computer system (800) also includes human-accessible storage devices and associated media, such as optical media including CD / DVD ROM / RW (820) or similar media (821) having CD / DVD, thumb drives (822), removable hard drives or solid-state drives (823), legacy magnetic media such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0101] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transient signals.

[0102] The computer system (800) may also include a network interface (845) for one or more communication networks (855). One or more communication networks (855) may be, for example, wireless, wired, optical. One or more communication networks (855) may also be local, wide area, metropolitan, automotive and industrial, real-time, latency-tolerant, etc. Examples of one or more communication networks (855) include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV and terrestrial broadcast TV, automotive and industrial including CANBus, etc. A specific network generally requires an external network interface adapter attached to a specific general-purpose data port or peripheral bus (849) (for example, a USB port of the computer system (800)); Others are integrated into the core of the computer system (800) by being attached to the system bus (e.g., an Ethernet interface for a PC computer system or a cellular network interface for a smartphone computer system), as described below. Using any of these networks, the computer system (800) can communicate with other entities. Such communication may be unidirectional, receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., from a CANbus to a specific CANbus device), or bidirectional, to other computer systems using local or wide-area digital networks, for example. Specific protocols and protocol stacks may be available for each network and network interface as described above.

[0103] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (840) of the computer system (800).

[0104] The core (840) may include one or more central processing units (CPUs) (841), graphics processing units (GPUs) (842), special programmable processing units in the form of Field Programmable Gate Areas (FPGAs) (843), hardware accelerators for specific tasks (844), graphics adapters (850), etc. Along with read-only memory (ROM) (845), random access memory (846), and internal mass storage (847) such as internal non-user accessible hard drives, SSDs, etc., these devices may be connected via a system bus (848). In some computer systems, the system bus (848) may be accessed in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached to the core's system bus (848) directly or via a peripheral bus (849). In one example, a screen (810) may be connected to a graphics adapter (850). Architectures of peripheral buses include PCI, USB, etc.

[0105] The CPU (841), GPU (842), FPGA (843), and accelerator (844) can be combined to execute specific instructions that can construct the aforementioned computer code. This computer code may be stored in ROM (845) or RAM (846). Transitional data may be stored in RAM (846), but permanent data may be stored, for example, in internal mass storage (847). Fast storage and retrieval for all memory devices may be enabled by using a cache memory that may be closely associated with one or more CPUs (841), GPUs (842), mass storage (847), ROM (845), RAM (846), etc.

[0106] A computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the field of computer software.

[0107] As an example, not a limitation, a computer system having an architecture (800), in particular a core (840), may provide functionality as a result of processor(s) (including a CPU, GPU, FPGA, accelerator, etc.) executing software implemented on one or more types of computer-readable media. Such computer-readable media may be media related to the user-accessible mass storage introduced above, or may be specific storage of the core (840) having non-transient characteristics, such as internal mass storage (847) or ROM (845). Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (840). The computer-readable media may include one or more memory devices or chips depending on specific requirements. The software may cause the core (840) and, in particular, the processors therein (including a CPU, GPU, FPGA, etc.) to execute a specific process or part of a specific process described herein, including defining data structures stored in RAM (846) and modifying such data structures according to a process defined in the software. Additionally or alternatively, a computer system may provide functionality as a result of logic that is hardwired or implemented in a circuit (e.g., accelerator (844)) capable of working instead of or with software to execute the specific process or a specific part of the specific process described herein. References to software may include logic, and where appropriate, vice versa. References to computer-readable media may include a circuit that stores software for execution (e.g., integrated circuit (IC)), a circuit that implements logic for execution, or, where appropriate, both. The present disclosure covers any suitable combination of hardware and software.

[0108] Although the present disclosure describes some exemplary embodiments, there are modifications, permutations, and various alternative equivalents that fall within the scope of the disclosure. Accordingly, those skilled in the art will understand that numerous systems and methods can be devised that implement the principles of the present disclosure and thus fall within the spirit and scope of the invention, even though they are not expressly illustrated or described herein.

Claims

Claim 1 A video decoding method comprising the step of receiving one or more syntax elements associated with a current first block belonging to a plurality of first blocks partitioned from a picture, wherein the one or more syntax elements include: a partition indicating a manner in which the current first block is partitioned into second blocks for intra prediction, a set of block selection signals indicating that the current first block is re-partitioned into third blocks of different sizes from the second blocks, each block selection signal corresponding to one of the third blocks and indicating whether each third block is coded using a first coding method or a second coding method, and a set of compressed representations corresponding to each of the third blocks; and a step of reconstructing the current first block based on the one or more syntax elements to generate a reconstructed current first block. Claim 2 A video decoding method according to claim 1, wherein the reconstructing step comprises: generating a reconstructed third block based on a neural decompression neural network (NN), wherein for each third block coded using the first coding method indicated by each of the block selection signal sets, each compressed representation of the third block is input to the neural decompression neural network (NN); and generating a residual third block based on a residual neural decompression neural network, wherein for each third block coded using the second coding method indicated by each of the block selection signal sets, each compressed representation of the third block is input to the residual neural decompression neural network (NN). Claim 3 A video decoding method according to paragraph 2, wherein the reconstructing step further comprises the step of generating a reconstructed third block based on each residual third block and each predicted third block for each third block coded using the second coding method. Claim 4 A video decoding method according to claim 3, wherein the reconstructing step further comprises the step of, for each third block coded using the second coding method, performing intra prediction based on the intra prediction NN, using previously reconstructed first blocks and previously generated reconstructed third blocks as inputs to the intra prediction NN, thereby generating a predicted second block set corresponding to each third block. Claim 5 A video decoding method according to claim 4, wherein the predicted second block set corresponding to each of the third blocks forms each of the predicted third blocks. Claim 6 A video decoding method according to claim 5, wherein the reconstructing step further comprises the step of performing each intra prediction for each third block coded using the second coding method, as an input to the intra prediction NN, a partition representing the manner in which the current first block is partitioned into second blocks for intra prediction. Claim 7 A video decoding method according to claim 3, wherein the reconstructing step further comprises the step of merging the reconstructed third blocks corresponding to the third blocks coded using the first coding method or the second coding method to generate the reconstructed current first block. Claim 8 A video decoding device comprising a circuit, wherein the circuit receives one or more syntax elements associated with a current first block belonging to a plurality of first blocks partitioned from a picture, wherein the one or more syntax elements include: a partition indicating a manner in which the current first block is partitioned into second blocks for intra prediction, a set of block selection signals indicating that the current first block is re-partitioned into third blocks of different sizes from the second blocks, each block selection signal corresponding to one of the third blocks, and each third block is coded using a first coding method or a second coding method, and a set of compressed representations corresponding to each of the third blocks; and wherein the current first block is reconstructed based on the one or more syntax elements to generate a reconstructed current first block. Claim 9 A video decoding device according to claim 8, wherein the circuit further comprises, for each third block coded using the first coding method indicated by each of the block selection signal sets, generating a third block reconstructed based on the neural decompression NN, with each compressed representation of the third block as input to the neural decompression NN; and for each third block coded using the second coding method indicated by each of the block selection signal sets, generating a residual third block based on the residual neural decompression NN, with each compressed representation of the third block as input to the residual neural decompression NN. Claim 10 In claim 9, the video decoding device is further configured such that, for each third block coded using the second coding method, the circuit generates a reconstructed third block based on each residual third block and each predicted third block. Claim 11 A video decoding device according to claim 10, wherein the circuit is further configured to perform intra prediction based on the intra prediction NN, which takes previously reconstructed first blocks and previously generated reconstructed third blocks as inputs to the intra prediction NN, for each third block coded using the second coding method, to generate a predicted second block set corresponding to each third block. Claim 12 In claim 11, the predicted second block set corresponding to each of the third blocks forms each of the predicted third blocks, a video decoding device. Claim 13 A video decoding device according to claim 12, wherein the circuit is further configured to perform each intra prediction with a partition representing the manner in which the current first block is partitioned into second blocks for intra prediction, as an input to the intra prediction NN, for each third block coded using the second coding method. Claim 14 A video decoding device according to claim 10, wherein the circuit is further configured to merge the reconstructed third blocks corresponding to the third blocks coded using the first coding method or the second coding method to generate the reconstructed current first block. Claim 15 A computer-readable non-transient medium for storing instructions, wherein when the instructions are executed by a processor, the processor performs a method according to any one of claims 1 through 7. Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images