Video encoding and decoding method, device, and computer-readable medium

Abstract

The embodiments of the present application disclose a method, apparatus, and computer-readable medium for video encoding, the method comprising: receiving one or more syntax elements associated with a current first block, the current first block belonging to a plurality of first blocks partitioned from a picture, the one or more syntax elements being used to indicate: an optimal partition, the optimal partition indicating how to partition the current first block into a plurality of second blocks for intra-frame prediction, a set of block selection signals, wherein the current first block is re-partitioned into a plurality of third blocks, each block selection signal corresponding to one of the plurality of third blocks and indicating whether the corresponding third block is encoded using a first encoding method or a second encoding method, and a set of compressed representations, each compressed representation corresponding to one of the plurality of third blocks. The current first block can be reconstructed based on the one or more syntax elements to generate a reconstructed current first block.

Description

[0001] Incorporation by reference

[0002] This disclosure claims priority to U.S. Patent Application No. 17 / 486,533, filed on September 27, 2021, entitled “Video Compression with Adaptive Iterative Intra Prediction,” which claims priority to U.S. Provisional Application No. 63 / 142,377, filed on January 27, 2021, entitled “Neural Image Compression with Adaptive Iterative Intra Prediction.” The disclosures of the prior applications are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure describes embodiments generally related to artificial neural network-based video encoding and decoding. Background Art

[0004] The background description provided herein is intended to generally present the context of the present disclosure. To the extent that the work of the presently named inventors is performed in the background section and in various aspects of this specification, it is not intended that it be prior art at the time of filing this disclosure, and it is neither expressly nor implied that it is admitted to be prior art with respect to the present disclosure.

[0005] Neural networks are based on a set of connected nodes (also called neurons) that are loosely modeled after neurons in biological brains. Neurons can be organized into multiple layers. Neurons in one layer can be connected to neurons in the immediately previous layer and the immediately following layer.

[0006] The connection between two neurons, like a synapse in a biological brain, can transmit a signal from one neuron to another. The neuron that receives the signal then processes it and can send signals to other connected neurons. In some examples, to find the output of a neuron, the inputs to the neuron are weighted by the weights of the connections from the inputs to the neuron, and the weighted inputs are summed to generate a weighted sum. A bias can be added to the weighted sum. Further, the weighted sum is then passed through an activation function to generate the output.

[0007] The International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC) Moving Picture Experts Group (MPEG) (JTC 1 / SC29 / WG 11) have been actively looking for potential needs for standardization of future video coding and decoding technologies. The ISO / IEC Joint Photographic Experts Group (JPEG) has established a JPEG Artificial Intelligence (AI) group that focuses on AI-based end-to-end neural image compression using neural networks (NN). Some companies have funded dedicated research projects on neural image compression (NIC). The Chinese Audio and Video Standard (AVS) has also established an AVS-AI special group to work on neural image and video compression technology. The success of recent methods has brought increasing industrial interest in advanced neural image and video compression methods. Summary of the Invention

[0008] Aspects of the present disclosure provide a method for performing video encoding at a video encoder. The method may include: dividing a picture into a plurality of first blocks; processing a current first block belonging to the plurality of first blocks to generate the following outputs: an optimal partition, wherein the optimal partition indicates how to partition the current first block into a plurality of second blocks for intra-frame prediction, a set of block selection signals, wherein the current first block is re-partitioned into a plurality of third blocks, each block selection signal corresponding to one of the plurality of third blocks and indicating whether the corresponding third block is encoded using a first encoding method or a second encoding method, and a set of compressed representations, each compressed representation corresponding to one of the plurality of third blocks; and encoding the output to generate encoding bits corresponding to the picture.

[0009] In an embodiment, an estimated block of the current first block divided using the optimal partition is generated based on an iterative prediction estimation neural network (NN). The estimated block includes a plurality of estimated second blocks corresponding to the plurality of second blocks. In an embodiment, for each third block encoded using the first encoding method, the plurality of estimated second blocks in the corresponding third block are encoded based on the neural compression NN to generate a corresponding compressed representation of the corresponding third block. For each third block encoded using the second encoding method, intra-frame prediction is performed to generate a residual block for each second block in the corresponding third block, and the plurality of residual blocks of the corresponding third block are compressed based on the residual neural compression NN to generate a corresponding compressed representation of the corresponding third block.

[0010] In an embodiment, the intra-frame prediction for generating the residual block for each second block in the corresponding third block is based on a plurality of estimated second blocks and one or more estimation blocks of the plurality of first blocks, wherein the plurality of estimated second blocks are processed before the estimated second blocks of the corresponding second blocks in the estimation blocks of the current first block, and the one or more estimation blocks are processed before the current first block.

[0011] In an embodiment, an estimated block of a current first block divided using the best partitioning can be generated based on an iterative prediction estimation NN in the following steps: in the current iteration, using a set of current estimated second blocks of the current first block and optional estimated blocks of multiple first blocks processed before the current first block as inputs of an intra-frame prediction NN to generate multiple predicted second blocks corresponding to the multiple current estimated second blocks of the current first block; generating multiple residual second blocks corresponding to the multiple predicted second blocks, repartitioning the set of multiple residual second blocks into multiple residual third blocks corresponding to multiple third blocks; compressing the multiple residual third blocks based on a residual neural compression NN to generate multiple compressed third blocks; decompressing the multiple compressed third blocks based on a residual neural decompression NN to restore the multiple residual third blocks; and combining the restored multiple residual third blocks with the corresponding multiple predicted second blocks to generate multiple reconstructed second blocks. The multiple reconstructed second blocks are used instead of the set of current estimated second blocks as input for the next iteration.

[0012] In an embodiment, for each third block, a first rate-distortion (RD) loss corresponding to the third block encoded using a first encoding method and a second RD loss corresponding to the third block encoded using a second encoding method are determined. Based on the first and second RD losses, a block selection signal for the third block may be determined to indicate which of the first encoding method and the second encoding method is selected to encode the third block.

[0013] In an embodiment, the processing further includes: determining a first total loss of the current first block corresponding to an optimal partition for dividing the current first block into a plurality of second blocks, the first total loss being a combination of RD losses of corresponding third blocks corresponding to one of the first encoding method and the second encoding method selected for each third block; determining a second total loss of the current first block corresponding to each of a plurality of different ways, the plurality of different ways being ways other than the optimal partition for dividing the current first block into a plurality of second blocks; comparing the first total loss and the second total loss to determine that the first total loss of the current first block corresponding to the optimal partition is the minimum loss between the first total loss and the second total loss.

[0014] Aspects of the present disclosure also provide a method for performing video decoding at a video decoder. The method may include: receiving one or more syntax elements associated with a current first block, the current first block belonging to a plurality of first blocks partitioned from a picture, the one or more syntax elements being used to indicate: an optimal partition, the optimal partition indicating how to partition the current first block into a plurality of second blocks for intra-frame prediction, a set of block selection signals, wherein the current first block is re-partitioned into a plurality of third blocks, each block selection signal corresponding to one of the plurality of third blocks and indicating whether the corresponding third block is encoded using a first encoding method or a second encoding method, and a set of compressed representations, each compressed representation corresponding to one of the plurality of third blocks. The current first block may be reconstructed based on the one or more syntax elements to generate a reconstructed current first block.

[0015] In an embodiment, the reconstruction includes: for each third block encoded using the first encoding method, generating a reconstructed third block based on a neural decompression neural network (NN), the first encoding method being indicated by a corresponding one of the set of block selection signals, and the corresponding compressed representation of the third block serving as input to the neural decompression NN; and for each third block encoded using the second encoding method, generating a residual third block based on a residual neural decompression NN, the second encoding method being indicated by a corresponding one of the set of block selection signals, and the corresponding compressed representation of the third block serving as input to the residual neural decompression NN.

[0016] In an embodiment, the reconstructing further comprises: for each third block encoded using the second encoding method, generating a reconstructed third block based on the corresponding residual third block and the corresponding predicted third block.

[0017] In an embodiment, the reconstruction further comprises: for each third block encoded using the second encoding method, performing intra prediction based on an intra prediction NN to generate a set of predicted second blocks corresponding to the corresponding third block, a plurality of previously reconstructed first blocks and a plurality of previously generated reconstructed third blocks as inputs to the intra prediction neural network NN.

[0018] In an embodiment, the set of predicted second blocks corresponding to the corresponding third block forms the corresponding predicted third block. The reconstruction further comprises: for each third block encoded using the second encoding method, performing corresponding intra-frame prediction using the optimal partition, the optimal partition serving as input to the intra-frame prediction NN and indicating how to partition the current first block into a plurality of second blocks for intra-frame prediction. In an embodiment, the reconstruction further comprises: merging the plurality of reconstructed third blocks to generate the reconstructed current first block, the plurality of reconstructed third blocks corresponding to the plurality of third blocks encoded using the first encoding method or the second encoding method.

[0019] Aspects of the present disclosure also provide a method for video encoding, including: determining one or more syntax elements associated with a current first block, wherein the current first block belongs to multiple first blocks partitioned from a picture, the one or more syntax elements being used to indicate: an optimal partition, wherein the optimal partition indicates how to partition the current first block into multiple second blocks for intra-frame prediction, a set of block selection signals, wherein the current first block is re-partitioned into multiple third blocks, each block selection signal corresponds to one of the multiple third blocks and indicates whether the corresponding third block is encoded using a first encoding method or a second encoding method, and a set of compressed representations, each compressed representation corresponding to one of the multiple third blocks; and sending the one or more syntax elements to generate a reconstructed current first block.

[0020] Aspects of the present disclosure also provide a video decoding apparatus, including a circuit configured to perform a video decoding method.

[0021] Aspects of the present disclosure further provide a video encoding apparatus, including a circuit configured to perform a video encoding or decoding method.

[0022] Aspects of the present disclosure also provide a video decoding device, including: a receiving module, configured to receive one or more syntax elements associated with a current first block, wherein the current first block belongs to multiple first blocks partitioned from a picture, and the one or more syntax elements are used to indicate: an optimal partition, wherein the optimal partition indicates how to partition the current first block into multiple second blocks for intra-frame prediction, a set of block selection signals, wherein the current first block is re-partitioned into multiple third blocks, each block selection signal corresponds to one of the multiple third blocks and indicates whether the corresponding third block is encoded using a first encoding method or a second encoding method, and a set of compressed representations, each compressed representation corresponding to one of the multiple third blocks; and a reconstruction module, configured to reconstruct the current first block based on the one or more syntax elements to generate a reconstructed current first block.

[0023] Aspects of the present disclosure also provide a non-transitory computer-readable medium for storing instructions, which, when executed by a processor, cause the processor to perform a video encoding or decoding method. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Other features, properties, and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, in which:

[0025] Figure 1 An exemplary workflow of an encoder according to an embodiment of the present disclosure is shown.

[0026] Figure 2 An exemplary workflow of an iterative partition selection module according to an embodiment of the present disclosure is shown.

[0027] Figure 3 An exemplary workflow of a decoder according to an embodiment of the present disclosure is shown.

[0028] Figure 4 The embodiment according to the present disclosure is shown Figure 2 An example workflow of the iterative forecast estimation module in the example.

[0029] Figure 5 An exemplary workflow of a training process according to an embodiment of the present disclosure is shown.

[0030] Figure 6 The video encoding process according to an embodiment of the present disclosure is shown.

[0031] Figure 7 The video decoding process according to an embodiment of the present disclosure is shown.

[0032] Figure 8 A computer system suitable for implementing certain embodiments of the disclosed subject matter is shown. DETAILED DESCRIPTION

[0033] I. Neural Image Compression and Block-based Intra Prediction and Residual Coding

[0034] The present disclosure provides a neural image compression (NIC) framework that uses a block-based intra-frame prediction mechanism with adaptive block size and compresses the input image through a neural network (NN) (or a group of NNs). In addition, residual coding and intra-frame prediction can be performed based on different block partitions. Furthermore, the NIC framework can adaptively select one of two candidate coding methods for each coding block. Although image encoding and decoding examples are described to illustrate certain aspects of the present disclosure, the techniques described herein can be directly applied to the field of video encoding and decoding, or adjusted based on the principles disclosed herein.

[0035] In some embodiments, given an input image x, the goal of the NIC is to compute a compressed representation using the image x as input to the NN encoder This compressed representation is compact for storage and transmission, and then, using As input to the NN decoder to reconstruct the image In some examples, NIC methods can adopt a variational autoencoder (VAE) structure, where the NN encoder directly uses the entire image x as input, which is passed through a set of network layers that work like a black box to calculate the output representation Accordingly, the NN decoder converts the entire representation As input, this input is passed through another set of network layers that work like another black box to compute the reconstructed

[0036] In some embodiments, a block-based intra-frame prediction and residual coding mechanism can be used instead of directly encoding the original entire image, which is very effective for image frame compression in video coding systems. In this example, the entire image is divided into blocks of various sizes, and prediction blocks are generated by copying the boundary pixels of the previous compressed block along various angles. The residual between the original block and the predicted block is then compressed. The residual can be encoded more efficiently than the original pixels and can achieve better coding performance. Different block sizes directly affect compression performance, and the optimal block size usually depends on the specific image.

[0037] II. Neural Image Compression with Intra Prediction Residual

[0038] The present disclosure provides a NIC framework that uses block-based intra-frame prediction with adaptive block size. The NIC framework can encode the residual between the predicted block and the original block (or an estimated block derived based on the original block) instead of encoding the original pixels, and adaptively determines the block size based on compression quality (e.g., rate-distortion (RD) loss).

[0039] Figure 1 An exemplary workflow (100) of an encoder according to an embodiment of the present disclosure is shown. The encoder can be used as a video encoder or an image encoder. The encoder can include a partitioning module (101) and an iterative partition selection module (102). The input image x can first be partitioned into k microblocks in the partitioning module (101), represented as Among them, m i Represents the i-th microblock. Each m i Can have (w m , h m ) and further divided into blocks b i，1 ,…,bi，n , where b i，j It is m i For different blocks b i，j , block b i，j The sizes can be different or the same, and different microblocks m can be treated differently. i In an embodiment, the microblocks are aligned with the coding tree unit (CTU) partitions in current video coding tools, such as High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC). Each CTU microblock can be further divided into blocks of various sizes, such as 2×2, 4×4, 8×8, 16×16, 32×32, or 64×64 pixels. The present disclosure does not impose any restrictions on the size of the CTU microblock or on how to divide the blocks in the CTU microblock.

[0040] For example, for the purpose of intra prediction, assume that for each microblock m i There are P different ways to convert microblock m i It is divided into multiple blocks (called intra-frame prediction blocks). The iterative partition selection module (102) can determine the microblock m i The best way to partition (denoted as p * ).

[0041] Additionally, repartitioning can be performed in the workflow (100) to separate the microblocks m i Divided into multiple blocks (called repartition blocks), for example, for residual coding or switching coding methods. Due to the two partitions - the partition for intra prediction and the repartition for residual coding and switching coding methods, intra prediction and residual coding can be performed based on blocks of different sizes. Generally, intra prediction can have higher efficiency with smaller allowed block sizes, while residual coding can have higher efficiency with larger allowed block sizes. The coding framework disclosed in this article provides the ability to separate the intra prediction block size from the residual coding block size, thereby improving image or video coding efficiency.

[0042] In some embodiments, for each repartitioned block, the iterative partition selection module (102) may select one encoding method from two candidate encoding methods for encoding the respective repartitioned block. For example, the selection may be made based on a rate-distortion (RD)-based quality loss calculated for each of the two candidate encoding methods for encoding the same repartitioned block. The result of the selection may be indicated by a block selection signal associated with each repartitioned block. For example, a block selection signal having a first value (e.g., 1) may indicate that a first encoding method is applied to the respective repartitioned block, while a block selection signal having a second value (e.g., 0) may indicate that a second encoding method is applied to the respective repartitioned block.

[0043] like Figure 1 As shown in FIG, the output of the iterative partition selection module (102) includes the best partition mode p*, a set of block selection signals and a set of compressed representations where q represents the number of repartition blocks. Each compressed representation can correspond to one repartition block. These outputs can be sent to the decoder side or to a storage system (usually after further compression by, for example, quantization and entropy coding).

[0044] Figure 2 An exemplary workflow (200) of the iterative partition selection module (102) according to an embodiment of the present disclosure is shown. Figure 2 In the example of , the iterative partition selection module (102) may include an iterative prediction estimation module (201), an intra-frame prediction module (211), a residual neural compression module (213), a residual neural decompression module (214), a residual compression loss calculation module (216), a neural compression module (221), a neural decompression module (222), a compression loss calculation module (223), a block selection module (231), a partition loss calculation module (232), a first addition module (212) and a second addition module (215). These elements are as follows: Figure 2 shown coupled together.

[0045] make Represents the microblock m in the original input x i The intra prediction block is obtained by the p-th partitioning method. In this example, the total number of blocks in this partition is n p It can be determined by the size of the microblock and the size of the intra prediction block. As mentioned above, there are P different ways to divide the microblock m i For each partitioning method, the resulting intra prediction blocks can have the same size or different sizes. For each partitioning method denoted as p, the workflow (200) can be run once to determine the total quality loss L i，p Based on these total quality losses, the number of microblocks m can be determined from P different partitioning methods. i The best way (denoted as p * ).

[0046] The iterative prediction estimation module (201) can be used to first use the iterative intra-frame prediction mechanism to calculate the original micro-block m i The estimated microblock may include the original intra-frame prediction block. The corresponding estimated intra prediction block is expressed as In terms of coding efficiency and performance (eg, indicated by RD loss), the estimated intra-predicted block can be used as a better basis for subsequent processing in the workflow (200) than the original intra-predicted block.

[0047] Then, for each estimated partition block b i，p，j , the prediction block can be calculated by the intra prediction module (211) based on the prediction network (NN for intra prediction) The prediction network takes as input a set of image pixels from b i，p，j The current m previously encoded i Estimation block In some embodiments, the input of the prediction network can also come from i Previously coded estimated microblocks. In a different example, reconstructed microblocks may be used instead of estimated microblocks as input to the intra prediction module (211).

[0048] There are many ways to select pixels and form the input of the prediction network. For example, the pixels in the context region that are spatially closest to b can be organized in some order (stacked, concatenated, spatially transformed, etc.). i，p，j The intra prediction module 211 can choose to skip the prediction process of blocks without sufficient pixel context on the boundary. Alternatively, the intra prediction module 211 can fill the microblocks to predict the blocks on the boundary. Through inference calculation, the prediction network outputs the predicted block

[0049] In various embodiments, the prediction network can have various architectures. For each partitioning method, in some embodiments, the prediction network can use different NN models for prediction. Generally, convolutional layers and fully connected layers can be used. The present disclosure does not impose any restrictions on the size and shape of the context region used for pixel selection, how pixels are transformed into input to the prediction network, or the network architecture of the prediction network.

[0050] For block b i，p，j , in the calculation prediction block Afterwards, based on b i，p，j and The residual r is calculated by, for example, subtraction at the addition module (212) i，p，j .make Represents the entire i-th microblock m divided in the p-th way i The residual Can be re-divided into a set of q residual blocks These q residual blocks are called repartitioned residual blocks. Note that the repartitioning of the q residual blocks can be done with n pThe original partitions of the predicted blocks (or intra-frame predicted blocks) are the same or different. When q=1, the entire microblock will be processed as a whole. The corresponding estimated microblock and prediction blocks In the same way, they can be re-divided into (called the repartitioned estimation block) and (called repartitioned prediction block), such as Figure 2 shown.

[0051] For each repartitioned residual block The residual neural compression module (213) compresses the corresponding residual block Computationally compressed representation This compressed representation is decompressed by the residual neural decompression module (214) to compute the recovered residual block The recovered residual block can be added back to the corresponding repartitioned prediction block To obtain the reconstructed block

[0052] Residual compression quality loss The estimation block based on repartitioning can be used in the residual compression loss module (including estimated intra-frame prediction blocks), reconstructed blocks and compressed representation For example, in an embodiment, the rate-distortion (RD) loss may be calculated as a quality measure:

[0053]

[0054] in, yes The distortion between. It measures compression The bit consumption is the rate loss. λ is a trade-off hyperparameter that balances the importance of different terms. Of course, other compression quality losses can also be used here. This disclosure does not impose any restrictions on the specific measurement used for compression quality loss, distortion, or rate loss.

[0055] At the same time, each repartitioned estimation block (including the original estimated intra-frame prediction block) can be directly compressed by the neural compression module (221) to compute the compressed representation This compressed representation is decompressed by the neural decompression module (222) to directly compute the recovered block

[0056] Compression quality loss The compression loss calculation module (223) can be based on and and with residual quality loss Calculated in a similar way. and The block selection module (231) generates a selection signal s by, for example, selecting an option with less quality loss. i，p，j , to indicate the use of repartitioned residual blocks Or the original repartitioned estimated block To generate compression or This gives the number of blocks used to compress the current j-th block. Best quality loss For example,

[0057] The calculation partition loss module (232) can calculate the microblock m i The total mass loss L of the p-th partitioning method i，p for:

[0058]

[0059] Among them, each w i，p，j is with In this example, all weights are simply set to 1 to treat all blocks equally. In other examples, some blocks may receive more attention than others, and the weights may be obtained using an attention map (or significance map).

[0060] Repeating the same process (workflow (200)) for all P partitioning methods will result in a quality loss of L i，p , p = 1, ..., P. Then, the best partitioning method p can be selected * , for example, as the partition with the best loss: p * =argmin p L i，p 、 The corresponding block select signal It may also be determined as the output of the iterative partition selection module (102).

[0061] make Represented as microblock m i Select the best partition block. Select the signal based on the block The corresponding compression or Can also be determined as a block The actual compressed representation of A set of compressed representations It is also output from the iterative partition selection module (102). * , compressed representation and block select signals The optimal partition p is further encoded, for example, by quantization and entropy coding, to generate coded bits in a coded stream and sent to a decoder. * , compressed representation and block select signals is encoded and represented by some syntax elements that are encoded and sent to the decoder.

[0062] The neural compression module (221), the neural decompression module (222), the residual neural compression module (213), and the residual neural decompression module (214) can use any neural compression method. This disclosure does not impose any restrictions on the specific methods or network architectures used for these modules.

[0063] Figure 3 An exemplary workflow (300) of a decoder according to an embodiment of the present disclosure is shown. The decoder can be used as a video decoder or an image decoder. The decoder may include a residual neural decompression module (301), a neural decompression module (302), an intra-frame prediction module (303), a merging module (305), and an addition module (304). These components are as follows: Figure 3 shown coupled together.

[0064] In an embodiment, at the decoder side, Figure 3 The system (decoder) in receives the best partition p * , compressed representation and block select signals (Usually recovered from the received bitstream by entropy decoding and dequantization.) Select the signal on a per-block basis The system chooses one of the following methods to calculate the recovered blocks If the signal Indicates that the recovered block comes from the encoder side ( Figure 1 The compressed representation of the encoder in (corresponding to the repartitioned estimation block ), the neural decompression module (302) can be used to calculate the recovered block

[0065] If the signal Indicates that the recovered block comes from the compressed residual representation based on the encoder side (corresponding to the repartitioned residual block ) decoding block, the residual neural decompression module (301) can be used to calculate the recovered residual

[0066] In an embodiment, the recovered residual is used In the case of a set of previously recovered blocks and previously recovered microblocks The intra prediction module (303) uses the prediction network to calculate the predicted intra prediction block in the same way as the encoder side In an embodiment, the difference between the operation of the intra prediction module (303) and the intra prediction module (211) is that the input to the prediction network is formed by the pixels of the estimation blocks and microblocks on the encoder side, but comes from the corresponding restored blocks and microblocks on the decoder side.

[0067] Intra-frame prediction block based on prediction Repartitioned prediction blocks can be obtained in the same way as on the encoder side. Then, the recovered residual can be added to the summation module (304). Add back the repartitioned (in the same way as on the encoder side) prediction blocks To get the recovered blocks

[0068] or will give the actual recovered blocks The decoder continues with the next repartitioned block.

[0069] Finally, the microblock m can be merged into i The recovered block (From the recovered block generated) aggregated into the reconstructed image In some embodiments, the merging module (305) can further process the recovered blocks to remove artifacts, such as deblocking, denoising, etc. The present disclosure provides a method for aggregating the recovered blocks into the reconstructed image. There are no restrictions on the specific methods in .

[0070] Figure 4 The embodiment according to the present disclosure is shown Figure 2 The example workflow (400) of the iterative prediction estimation module (201) in the example. The iterative prediction estimation module (201) may include an intra-frame prediction module (401), an addition module (402), a residual neural compression module (403), a residual neural decompression module (404) and an addition module (405). These components are as follows: Figure 4 shown coupled together.

[0071] In the workflow (400), an estimated intra-frame prediction block of a micro-block can be determined in an iterative manner. Compared with directly using the original micro-block for intra-frame prediction and residual block generation, the estimated intra-frame prediction block determined in this manner can provide a better basis for subsequent intra-frame prediction and residual block generation in the workflow (200). For example, the residual block obtained based on the estimated block may include less residual and can be encoded with fewer bits. Therefore, the efficiency can be improved. Figure 2 RD performance of the encoder in the example.

[0072] Assume that for iteration t, there is a set of current estimation blocks In an embodiment, for the first iteration, (with current partition p) is initialized to For each estimation block Prediction Block The intra-frame prediction module (401) can be based on Figure 4 The prediction network takes as input a set of image pixels that are obtained from Previously encoded estimate block Selected in. Figure 2 The same example, in some embodiments, the selected pixels can also come from m i The estimated micro-block previously encoded. Through inference calculation, the prediction network outputs the predicted block Then, the summing module 402 can be used based on and Calculate the residual r i，p，j .

[0073] Total residual can be repartitioned into a set of q repartitioned residual blocks The corresponding estimated microblock and prediction blocks In the same way, they can be re-divided into and For each residual block It can be compressed by the residual neural compression module (403) to compute the compressed representation This compressed representation is decompressed by the residual neural decompression module (404) to compute the recovered residual block The recovered residual block can be added back to the corresponding repartitioned prediction block To obtain the reconstructed block

[0074] Finally, all reconstructed blocks form the reconstructed estimated block It is sent back to the prediction network as input and the system enters the next iteration t+1. Typically, multiple iterations are performed, for example, until the estimation block converges or the maximum number of iterations is reached. The final reconstructed estimation block generated in the last iteration is the output estimation block of the iterative prediction estimation module (201).

[0075] Note that in some other embodiments, the iterative prediction estimation module (201) can simulate the entire encoding and decoding process. That is, for each iteration t, the iterative prediction estimation module (201) runs the complete encoding and decoding process. Figure 2 In the example described in the encoding process, Can be replaced by (or in another form ), and output the best partition p * (Current partition mode p), compressed representation and block select signals Then, in Figure 3 In the decoding process described in the example, the reconstructed image is fed back to the iterative prediction estimation module (201) as the input for the next iteration.

[0076] III. Training Process

[0077] Figure 5 An exemplary workflow (500) of a training process according to an embodiment of the present disclosure is shown. The training process may be performed on the following components: a partition module (511), an iterative partition selection module (512), a residual neural decompression module (521), a neural decompression module (522), an addition module (523), an intra-frame prediction module (524), a merging module (525), a calculation of additional loss module (502), and a calculation of total loss module (501). These components are as follows: Figure 5 shown coupled together.

[0078] The goal of the training process is to learn or train Figure 1-4 The neural network (or neural network model) used in the example. For example, the neural networks in modules (521), (522), (524) and (525) can be used in modules (301), (302), (303) and (305) respectively after training. The neural network in module 512 (which can have a similar structure to module (102)) can be used in module (102) (including modules (201), (211), (213), (214), (221), (222) and (231)). Module (201) includes modules (401), (403) and (404).

[0079] In some embodiments, the intra prediction modules (211), (303), and (401) may share the same neural network; the residual neural compression modules (213) and (403) may share the same neural network; the residual neural decompression modules (214), (301), and (404) may share the same neural network; and the neural decompression modules (222) and (302) may share the same neural network. In other embodiments, the above modules may not share the same neural network.

[0080] In some embodiments, the merging modules (305) and (525) and the block selection module (231) are implemented using a neural network. For example, the NN is used to aggregate the recovered blocks into the recovered image. Corresponding learnable parameters may also be learned during the training process. In other embodiments, the merging modules (305) and (525) and the block selection module (231) may not be implemented using a neural network. Accordingly, the corresponding neural network is not trained during the training process.

[0081] In an embodiment, during the training process, the weight coefficients of the networks and modules to be learned can be initialized by, for example, using a pre-trained model or by setting their parameters to random numbers. Then, given an input training image x, it is obtained by Figure 1 The encoding process described (via Figure 5 Modules (511) and (512) in the calculation of each microblock m i The best partition p * , compressed representation and block select signals Then pass Figure 3 The decoding process described (via modules (521) and (523)-(525) or (522) and (524)-(525)) to generate the final reconstructed image

[0082] In an embodiment, the distortion loss may be calculated For example, the traditional Peak Signal-to-Noise Ratio (PSNR), Multiscale Structural Similarity (MS-SSIM), or a weighted combination of the two. In an embodiment, the rate loss can be calculated Expressed as measured compression Therefore, the total RD loss can be calculated in the calculation total loss module (501)

[0083] Among them, α, β i is a hyperparameter that balances the importance of different terms.

[0084] In some embodiments, other forms of loss, such as the recovered residual, may also be calculated in the calculate additional loss module (502) (e.g., mean squared error (MSE) or structural similarity (SSIM) measure). and the original residual Distortion loss between and distortion loss and Optionally, you can ) are combined into the final loss of the entire system.

[0085] The gradient of the total loss or final loss can be calculated and back-propagated to update the learnable parameters in the system. Note that different components (i.e., networks or modules) can be updated at different times and at different update frequencies. In some embodiments, some components or some parameters in some components can be pre-trained and fixed, and the training process only updates the remaining parameters.

[0086] The techniques disclosed herein provide a flexible framework for improving the performance of neural image compression. Multiple partitioning schemes can be tested for intra prediction to select the best partitioning to suit the characteristics of the image being processed. Furthermore, the partitions used for intra prediction residual generation and those used for residual compression can be separated so that the best partitions can be determined independently and applied to both intra prediction and residual compression. Furthermore, the choice between two encoding methods for a repartitioned block can be adaptively determined based on the quality loss of the two encoding methods.

[0087] IV. Video Coding Process Based on Adaptive Iterative Intra-frame Prediction Framework

[0088] Figure 6 FIG. 6 shows a video encoding process (600) according to an embodiment of the present disclosure. Figure 1 The process (600) is performed at the encoder in the example. The process (600) may start from (S601) and proceed to (S610).

[0089] At (S610), a picture in a video sequence may be divided into a plurality of first blocks. In an example, the first blocks may be aligned with CTUs divided from the picture.

[0090] At (S620), a current first block belonging to a plurality of first blocks may be processed to generate the following outputs: an optimal partition, wherein the optimal partition indicates how to divide the current first block into a plurality of second blocks for intra-frame prediction, a set of block selection signals, wherein the current first block is re-divided into a plurality of third blocks, each block selection signal corresponds to one of the plurality of third blocks and indicates whether the corresponding third block is encoded using the first encoding method or the second encoding method, and a set of compressed representations, each compressed representation corresponding to one of the plurality of third blocks.

[0091] In an example, an estimated block of the current first block divided using the optimal partitioning may be first generated based on an iterative prediction estimation NN. The estimated block may include a first estimated second block corresponding to a plurality of second blocks. In an embodiment, the estimated block of the current first block may be generated in an iterative manner. For example, in the current iteration, a set of current estimated second blocks of the current first block and optional estimated blocks of a plurality of first blocks processed before the current first block may be first used as inputs of the intra-frame prediction NN to generate a plurality of predicted second blocks corresponding to the plurality of current estimated second blocks of the current first block.

[0092] Then, a plurality of residual second blocks corresponding to the plurality of predicted second blocks may be generated. Then, the set of the plurality of residual second blocks may be repartitioned into a plurality of residual third blocks corresponding to the plurality of third blocks. Thereafter, the plurality of residual third blocks may be compressed based on the residual neural compression NN to generate a plurality of compressed third blocks. The plurality of compressed third blocks may then be decompressed based on the residual neural decompression NN to restore the plurality of residual third blocks. The restored plurality of residual third blocks may be combined with the corresponding plurality of predicted second blocks to generate a plurality of reconstructed second blocks. Finally, the plurality of reconstructed second blocks may be used instead of the set of currently estimated second blocks as input for the next iteration.

[0093] In an embodiment, for each third block encoded using the first encoding method, a plurality of estimated second blocks in the corresponding third block may be compressed based on a neural compression neural network to generate a corresponding compressed representation of the corresponding third block. For each third block encoded using the second encoding method, intra-frame prediction may be performed to generate a plurality of predicted second blocks. Based on the plurality of predicted second blocks, a residual block for each second block in the corresponding third block may be generated. The plurality of residual blocks of the corresponding third block may be compressed based on the residual neural compression neural network to generate a corresponding compressed representation of the corresponding third block.

[0094] In an embodiment, intra-frame prediction for generating a residual block for each second block in the corresponding third block can be based on (1) multiple estimated second blocks and (2) one or more estimated blocks of multiple first blocks, wherein the multiple estimated second blocks are processed before the estimated second blocks of the corresponding second blocks in the estimated blocks of the current first block, and the one or more estimated blocks are processed before the current first block.

[0095] In an embodiment, for each third block, a first rate-distortion (RD) loss corresponding to the third block encoded using the first encoding method and a second RD loss corresponding to the third block encoded using the second encoding method may be generated. Then, based on the first and second RD losses, a block selection signal for the third block may be generated to indicate which of the first encoding method and the second encoding method is selected to encode the third block.

[0096] In an embodiment, a first total loss of the current first block corresponding to the best partition for partitioning the current first block into a plurality of second blocks may be generated. The first total loss may be a combination of the RD losses of the corresponding third blocks corresponding to one of the first encoding method and the second encoding method selected for each third block. In addition, a second total loss of the current first block corresponding to each of a plurality of different methods may be generated, the plurality of different methods being methods other than the best partition for partitioning the current first block into a plurality of second blocks. By comparing the first total loss and the second total loss, it may be determined that the first total loss of the current first block corresponding to the best partition is the minimum loss of the first total loss and the second total loss.

[0097] At (S630), the output may be encoded, for example, by quantization and / or entropy coding, to generate coded bits corresponding to the picture. The coded bits may be transmitted in a bitstream corresponding to the video sequence or may be stored in a storage medium. The process (600) may proceed to (S699) and end.

[0098] Figure 7 FIG. 7 shows a video decoding process (700) according to an embodiment of the present disclosure. The process (700) may be performed in Figure 3 The process (700) may start at (S701) and proceed to (S710).

[0099] At (S710), one or more syntax elements associated with a current first block may be received. The current first block belongs to a plurality of first blocks partitioned from a picture. The one or more syntax elements may indicate: an optimal partition, wherein the optimal partition indicates how to partition the current first block into a plurality of second blocks for intra prediction; a set of block selection signals, wherein the current first block is re-partitioned into a plurality of third blocks, each block selection signal corresponding to one of the plurality of third blocks and indicating whether the corresponding third block is encoded using the first encoding method or the second encoding method; and a set of compressed representations, each compressed representation corresponding to one of the plurality of third blocks.

[0100] At (S720), the current first block may be reconstructed based on the one or more syntax elements to generate a reconstructed current first block. In an embodiment, for each third block encoded using a first encoding method, the reconstructed third block may be generated based on a neural decompression NN, the first encoding method being indicated by a corresponding one of a set of block selection signals, and a corresponding compressed representation of the third block being used as an input to the neural decompression NN.

[0101] For each third block encoded using the second encoding method, a residual third block can be generated based on a residual neural decompression NN, the second encoding method being indicated by a corresponding one of a set of block selection signals, and a corresponding compressed representation of the third block as input to the residual neural decompression NN. For each third block encoded using the second encoding method, a reconstructed third block can then be generated based on the corresponding residual third block and the corresponding predicted third block.

[0102] In an embodiment, for each third block encoded using the second encoding method, intra prediction may be performed based on an intra prediction NN to generate a set of predicted second blocks corresponding to the corresponding third block, with a plurality of previously reconstructed first blocks and a plurality of previously generated reconstructed third blocks serving as inputs to the intra prediction NN. The set of predicted second blocks corresponding to the corresponding third block forms the corresponding predicted third block. In an embodiment, for each third block encoded using the second encoding method, corresponding intra prediction is performed using an optimal partition, with the optimal partition serving as input to the intra prediction NN.

[0103] In an embodiment, a plurality of reconstructed third blocks corresponding to a plurality of third blocks encoded using the first encoding method or the second encoding method are merged to generate a reconstructed current first block. The process (700) may proceed to (S799) and end.

[0104] V. Computer Systems

[0105] The techniques disclosed herein (e.g., functions, modules, elements, methods, processes, operations in various embodiments or examples) can be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In some examples, these techniques can be implemented as computer software through computer-readable instructions and physically stored in one or more computer-readable media.

[0106] The computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or other mechanisms to create code comprising instructions, which may be directly executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), or the like, or executed through decoding, microcode, or the like.

[0107] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, and the like.

[0108] Figure 8 A computer system (800) suitable for implementing certain embodiments of the disclosed subject matter is shown. Figure 8The components shown for the computer system (800) are exemplary in nature and are not intended to limit the scope of use or functionality of computer software implementing embodiments of the present disclosure. Nor should the configuration of components be interpreted as having any dependency or requirement on any one or combination of components shown in the exemplary embodiment of the computer system (800).

[0109] The computer system (800) may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human users through tactile input (e.g., keyboard input, sliding, data glove movement), audio input (e.g., sound, applause), visual input (e.g., gestures), and olfactory input (not shown). The human-computer interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0110] The human-computer interface input device may include one or more of the following (only one of which is depicted): keyboard (801), mouse (802), touchpad (803), touch screen (810), data gloves (not shown), joystick (805), microphone (806), scanner (807), camera (808).

[0111] The computer system (800) may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback through a touch screen (810), a data glove (not shown), or a joystick (805), but there may also be tactile feedback devices that are not used as input devices), audio output devices (e.g., speakers (809), headphones (not shown)), visual output devices (e.g., screens (810) including cathode ray tube screens, liquid crystal screens, plasma screens, organic light emitting diode screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities - some of which may output two-dimensional visual output or output of more than three dimensions through means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and cigarette boxes (not shown)), and printers (not shown).

[0112] The computer system (800) may also include human-accessible storage devices and their associated media, such as optical media including high-density read-only / rewritable optical disks (CD / DVD ROM / RW) (820) with CD / DVD or similar media (821), thumb drives (822), removable hard drives or solid state drives (823), traditional magnetic media such as tapes and floppy disks (not shown), ROM / ASIC / PLD-based specialized devices such as security software dongles (not shown), and the like.

[0113] Those skilled in the art will also understand that the term "computer-readable media" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0114] The computer system (800) may also include an interface (854) to one or more communication networks (855). For example, the one or more networks (855) may be wireless, wired, or optical. The one or more networks (855) may also be local area networks, wide area networks, metropolitan area networks, vehicle-mounted networks, and industrial networks, real-time networks, delay-tolerant networks, and the like. Examples of the one or more networks (855) also include local area networks such as Ethernet, wireless local area networks, cellular networks (GSM, 3G, 4G, 5G, LTE, etc.), television wired or wireless wide area digital networks (including cable television, satellite television, and terrestrial broadcast television), vehicle-mounted and industrial networks (including CANBus), and the like. Some networks typically require an external network interface adapter for connecting to some universal data port or peripheral bus (849) (e.g., a USB port of the computer system (800)); other systems are typically integrated into the core of the computer system (800) by connecting to a system bus as described below (e.g., an Ethernet interface integrated into a PC computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, the computer system (800) can communicate with other entities. The communication can be one-way, for reception only (e.g., wireless television), one-way, for transmission only (e.g., a CAN bus to certain CAN bus devices), or two-way, such as to other computer systems via a local or wide area digital network. Each of the above networks and network interfaces can use certain protocols and protocol stacks.

[0115] The aforementioned human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to the core (840) of the computer system (800).

[0116] The core (840) may include one or more central processing units (CPUs) (841), graphics processing units (GPUs) (842), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (843), hardware accelerators for specific tasks (844), graphics adapters (850), and the like. These devices, as well as read-only memory (ROM) (845), random access memory (846), internal mass storage (e.g., internal non-user accessible hard disk drives, solid-state drives, etc.) (847), and the like, may be connected via a system bus (848). In some computer systems, the system bus (848) may be accessed in the form of one or more physical plugs so that it can be expanded with additional central processing units, graphics processing units, and the like. Peripheral devices may be attached directly to the core's system bus (848) or connected via a peripheral bus (849). In an example, a screen (810) may be connected to a graphics adapter (850). Peripheral bus architectures include peripheral component interconnect (PCI), universal serial bus (USB), and the like.

[0117] The CPU (841), GPU (842), FPGA (843), and accelerator (844) can execute certain instructions, which, when combined, can constitute the aforementioned computer code. The computer code can be stored in ROM (845) or RAM (846). Transient data can also be stored in RAM (846), while permanent data can be stored in, for example, internal mass storage (847). Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with one or more CPUs (841), GPUs (842), mass storage (847), ROM (845), RAM (846), etc.

[0118] The computer readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purposes of the present disclosure, or may be medium and code well known and available to those skilled in the art of computer software.

[0119] As an example and not a limitation, a computer system having architecture (800), in particular core (840), can provide functionality as a processor (including a CPU, GPU, FPGA, accelerator, etc.) to execute software contained in one or more tangible computer-readable media. Such computer-readable media can be media associated with the aforementioned user-accessible mass storage, as well as specific memory of the core (840) having non-volatile properties, such as core internal mass storage (847) or ROM (845). Software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (840). Depending on specific needs, the computer-readable medium may include one or more storage devices or chips. The software can enable the core (840), in particular the processor therein (including a CPU, GPU, FPGA, etc.) to execute a specific process or a specific part of a specific process described herein, including defining a data structure stored in RAM (846) and modifying such a data structure according to a software-defined process. Additionally or alternatively, the computer system may provide functionality hardwired in logic or otherwise contained in circuitry (e.g., accelerator (844)) that may operate in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuitry (e.g., an integrated circuit (IC)) storing the executing software, circuitry containing the executing logic, or both. The present disclosure includes any suitable combination of hardware and software.

[0120] Although this disclosure has described a number of exemplary embodiments, various modifications, permutations, and equivalent substitutions of the embodiments are within the scope of this disclosure. Therefore, it should be understood that those skilled in the art will be able to devise a variety of systems and methods that, although not explicitly shown or described herein, embody the principles of this disclosure and are therefore within the spirit and scope of this disclosure.

Claims

1. A video decoding method, characterized in that: include: One or more syntax elements associated with a current first block, where the current first block belongs to a plurality of first blocks partitioned from a picture, are received, where the one or more syntax elements are used to indicate: an optimal partition, the optimal partition indicating how to divide the current first block into a plurality of second blocks for intra prediction, a set of block selection signals, wherein the current first block is re-divided into a plurality of third blocks of different sizes from the plurality of second blocks, each block selection signal corresponding to one of the plurality of third blocks and indicating whether the corresponding third block is encoded using a first encoding method based on a neural decompression neural network NN or a second encoding method based on a residual neural decompression neural network NN, and a set of compressed representations, each compressed representation corresponding to one of the plurality of third blocks; as well as Reconstructing the current first block based on the one or more syntax elements to generate a reconstructed current first block, wherein the reconstructing comprises: generating, for each third block encoded using the first encoding method indicated by a corresponding one of the set of block selection signals, a reconstructed third block based on the neural decompression neural network NN, with a corresponding compressed representation of the third block as input to the neural decompression neural network NN; and For each third block encoded using the second encoding method, a residual third block is generated based on the residual neural decompression neural network NN, and the second encoding method is indicated by a corresponding one of the set of block selection signals, and the corresponding compressed representation of the third block is used as input to the residual neural decompression neural network NN.

2. The method according to claim 1, characterized in that The reconstruction further comprises: For each third block encoded using the second encoding method, a reconstructed third block is generated based on the corresponding residual third block and the corresponding predicted third block.

3. The method according to claim 2, characterized in that The reconstruction further comprises: For each third block encoded using the second encoding method, intra prediction is performed based on an intra prediction neural network NN to generate a set of predicted second blocks corresponding to the corresponding third block, with a plurality of previously reconstructed first blocks and a plurality of previously generated reconstructed third blocks serving as inputs to the intra prediction neural network NN.

4. The method according to claim 3, characterized in that The set of predicted second blocks corresponding to the respective third block forms the respective predicted third block.

5. The method according to claim 4, characterized in that The reconstruction further comprises: For each third block encoded using the second encoding method, corresponding intra-frame prediction is performed using the optimal partition, the optimal partition serves as input to the intra-frame prediction neural network NN, and indicates how to divide the current first block into multiple second blocks for intra-frame prediction.

6. The method according to claim 1, characterized in that The reconstruction further comprises: The plurality of reconstructed third blocks are merged to generate the reconstructed current first block, the plurality of reconstructed third blocks corresponding to the plurality of third blocks encoded using the first encoding method or the second encoding method.

7. A video encoding method, characterized in that: include: Determine one or more syntax elements associated with a current first block, where the current first block belongs to a plurality of first blocks partitioned from a picture, the one or more syntax elements being used to indicate: an optimal partition, the optimal partition indicating how to divide the current first block into a plurality of second blocks for intra prediction, a set of block selection signals, wherein the current first block is re-divided into a plurality of third blocks of different sizes from the plurality of second blocks, each block selection signal corresponding to one of the plurality of third blocks and indicating whether the corresponding third block is encoded using a first encoding method based on a neural decompression neural network NN or a second encoding method based on a residual neural decompression neural network NN, and a set of compressed representations, each compressed representation corresponding to one of the plurality of third blocks; as well as Sending the one or more syntax elements to generate a reconstructed current first block, wherein the reconstructing comprises: generating, for each third block encoded using the first encoding method indicated by a corresponding one of the set of block selection signals, a reconstructed third block based on the neural decompression neural network NN, with a corresponding compressed representation of the third block as input to the neural decompression neural network NN; and For each third block encoded using the second encoding method, a residual third block is generated based on the residual neural decompression neural network NN, and the second encoding method is indicated by a corresponding one of the set of block selection signals, and the corresponding compressed representation of the third block is used as input to the residual neural decompression neural network NN.

8. A video decoding device, characterized in that: The device comprises a circuit configured to execute the video decoding method according to any one of claims 1 to 6.

9. A video encoding device, characterized in that: comprising a circuit configured to perform the method of claim 7.

10. A video decoding device, characterized in that: include: A receiving module, configured to receive one or more syntax elements associated with a current first block, where the current first block belongs to a plurality of first blocks divided from a picture, the one or more syntax elements being used to indicate: an optimal partition, the optimal partition indicating how to divide the current first block into a plurality of second blocks for intra prediction, a set of block selection signals, wherein the current first block is re-divided into a plurality of third blocks of different sizes from the plurality of second blocks, each block selection signal corresponding to one of the plurality of third blocks and indicating whether the corresponding third block is encoded using a first encoding method based on a neural decompression neural network NN or a second encoding method based on a residual neural decompression neural network NN, and a set of compressed representations, each compressed representation corresponding to one of the plurality of third blocks; as well as a reconstruction module, configured to reconstruct the current first block based on the one or more syntax elements to generate a reconstructed current first block, wherein the reconstruction module includes: generating, for each third block encoded using the first encoding method indicated by a corresponding one of the set of block selection signals, a reconstructed third block based on the neural decompression neural network NN, with a corresponding compressed representation of the third block as input to the neural decompression neural network NN; and For each third block encoded using the second encoding method, a residual third block is generated based on the residual neural decompression neural network NN, and the second encoding method is indicated by a corresponding one of the set of block selection signals, and the corresponding compressed representation of the third block is used as input to the residual neural decompression neural network NN.

11. The device according to claim 10, characterized in that The reconstruction module is further configured to: For each third block encoded using the second encoding method, a reconstructed third block is generated based on the corresponding residual third block and the corresponding predicted third block.

12. The device according to claim 11, characterized in that The reconstruction module is further configured to: For each third block encoded using the second encoding method, intra prediction is performed based on an intra prediction neural network NN to generate a set of predicted second blocks corresponding to the corresponding third block, with a plurality of previously reconstructed first blocks and a plurality of previously generated reconstructed third blocks serving as inputs to the intra prediction neural network NN.

13. The device according to claim 12, characterized in that The set of predicted second blocks corresponding to the respective third block forms the respective predicted third block.

14. The device according to claim 13, characterized in that The reconstruction module is further configured to: For each third block encoded using the second encoding method, corresponding intra-frame prediction is performed using the optimal partition, the optimal partition serves as input to the intra-frame prediction neural network NN, and indicates how to divide the current first block into multiple second blocks for intra-frame prediction.

15. The device according to claim 10, characterized in that The reconstruction module is further configured to: The plurality of reconstructed third blocks are merged to generate the reconstructed current first block, the plurality of reconstructed third blocks corresponding to the plurality of third blocks encoded using the first encoding method or the second encoding method.

16. A non-volatile computer-readable medium, characterized in that Used to store instructions, when the instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

17. A method for storing a video stream, characterized in that: Execute the method of claim 7 to generate a video stream, and store the video stream.

18. A method for transmitting a video stream, characterized in that: Execute the method of claim 7 to generate a video stream, and transmit the video stream.

19. A computer-readable storage medium storing a computer program / instruction and a video stream, characterized in that: When the computer program / instruction is executed by a processor, the steps of the method according to claim 7 are implemented to generate the video code stream.

Images