Decoding method, coding method, training method, decoder, and coder

Abstract

Provided in the embodiments of the present application are a decoding method, a coding method, a training method, a decoder, and a coder. The decoding method comprises: decoding a code stream, so as to determine a motion parameter of the current block; on the basis of the motion parameter, determining a reference block of the current block in a reference image of the current block; performing quality enhancement on the reference block, so as to obtain an enhanced block; and on the basis of the enhanced block, determining a prediction block of the current block. In the embodiments of the present application, a decoder performs quality enhancement on a reference block to obtain an enhanced block, and then determines a prediction block of the current block on the basis of the enhanced block, which is equivalent to the decoder converting a quality enhancement process of the prediction block into a quality enhancement process of the reference block, such that a quality enhancement process of a prediction block being coupled to a determination process of the prediction block can be prevented, thereby reducing decoding complexity, and thus improving a decoding performance.

Description

Decoding methods, encoding methods, training methods, decoders, and encoders Technical Field

[0001] This application relates to the field of encoding and decoding technology, and more specifically, to decoding methods, encoding methods, training methods, decoders, and encoders. Background Technology

[0002] Digital video technology can be integrated into various video devices, such as digital televisions, smartphones, computers, e-readers, and video players. With the development of video technology, the amount of data contained in video data has become increasingly large. To facilitate the transmission of video data, video devices implement video compression technology to enable more efficient transmission or storage of video data.

[0003] Because videos contain temporal or spatial redundancy, prediction can eliminate or reduce this redundancy, thus improving compression efficiency. To improve prediction performance, interpolation filtering is commonly used for predictive compression; however, current interpolation filtering methods suffer from low prediction efficiency, resulting in poor video encoding and decoding performance.

[0004] Summary of the Invention

[0005] This application provides a decoding method, an encoding method, a training method, a decoder, and an encoder, which can improve decoding performance.

[0006] In a first aspect, embodiments of this application provide a decoding method, including:

[0007] Decode the bitstream to determine the motion parameters of the current block;

[0008] Based on the motion parameters of the current block, a reference block is determined in the reference image of the current block;

[0009] The reference block is mass-enhanced to obtain an enhanced block;

[0010] Based on the enhancement block, the prediction block of the current block is determined.

[0011] Secondly, embodiments of this application provide an encoding method, including:

[0012] Motion estimation is performed on the current block to obtain a reference block for the current block;

[0013] The reference block is mass-enhanced to obtain an enhanced block;

[0014] Based on the enhancement block, the prediction block of the current block is determined.

[0015] Thirdly, embodiments of this application provide a neural network training method, including:

[0016] The sample video is encoded to obtain a bitstream;

[0017] The bitstream is decoded to determine the prediction block for the current block;

[0018] The neural network is trained based on the label of the current block and the predicted block.

[0019] Fourthly, embodiments of this application provide a decoder, including:

[0020] The first determining unit is used to decode the bitstream and determine the motion parameters of the current block;

[0021] The second determining unit is used to determine a reference block of the current block in the reference image of the current block based on the motion parameters of the current block;

[0022] An enhancement unit is used to enhance the quality of the reference block to obtain an enhanced block;

[0023] The third determining unit is used to determine the prediction block of the current block based on the enhancement block.

[0024] Fifthly, embodiments of this application provide an encoder, including:

[0025] The first determining unit is used to perform motion estimation on the current block to obtain a reference block for the current block;

[0026] An enhancement unit is used to enhance the quality of the reference block to obtain an enhanced block;

[0027] The second determining unit is used to determine the prediction block of the current block based on the enhancement block.

[0028] Sixthly, embodiments of this application provide a neural network training apparatus, comprising:

[0029] The encoding unit is used to encode the sample video to obtain the bitstream;

[0030] A decoding unit is used to decode the bitstream and determine the prediction block of the current block;

[0031] A training unit is used to train the neural network based on the label of the current block and the prediction block.

[0032] Seventhly, embodiments of this application provide a decoder, including:

[0033] Processor, adapted to implement computer instructions; and,

[0034] A computer-readable storage medium storing computer instructions adapted for loading by a processor and executing the decoding method in the first aspect or its various implementations mentioned above.

[0035] In one implementation, there are one or more processors and one or more memories.

[0036] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be disposed separately from the processor.

[0037] Eighthly, embodiments of this application provide an encoder, including:

[0038] Processor, adapted to implement computer instructions; and,

[0039] A computer-readable storage medium storing computer instructions adapted for loading by a processor and executing the encoded methods of the second aspect or its various implementations mentioned above.

[0040] In one implementation, there are one or more processors and one or more memories.

[0041] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be disposed separately from the processor.

[0042] Ninthly, embodiments of this application provide a neural network training apparatus, comprising:

[0043] Processor, adapted to implement computer instructions; and,

[0044] A computer-readable storage medium storing computer instructions adapted for loading by a processor and executing the encoded methods of the third aspect or its implementations mentioned above.

[0045] In one implementation, there are one or more processors and one or more memories.

[0046] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be disposed separately from the processor.

[0047] In a tenth aspect, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when read and executed by a processor of a computer device, cause the computer device to perform the decoding method described in the first aspect above, the encoding method described in the second aspect above, or the training method described in the third aspect above.

[0048] Eleventhly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the decoding method described in the first aspect above, the encoding method described in the second aspect above, or the training method described in the third aspect above.

[0049] In a twelfth aspect, embodiments of this application provide a bitstream, which is the bitstream involved in the decoding method described in the first aspect above or the bitstream generated by the method described in the second aspect above.

[0050] Based on the above technical solutions, embodiments of this application provide a decoding method, which includes: decoding a bitstream to determine motion parameters of a current block; determining a reference block of the current block in a reference image of the current block based on the motion parameters of the current block; performing quality enhancement on the reference block to obtain an enhanced block; and determining a prediction block of the current block based on the enhanced block. In this embodiment, the decoder obtains the enhanced block by performing quality enhancement on the reference block, and then determines the prediction block of the current block based on the enhanced block; equivalently, the decoder converts the quality enhancement process for the prediction block into a quality enhancement process for the reference block. This avoids coupling the quality enhancement process for the prediction block to the prediction block determination process, reducing decoding complexity and improving decoding performance. Attached Figure Description

[0051] Figure 1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of this application.

[0052] Figure 2 is a schematic block diagram of a video encoder involved in an embodiment of this application.

[0053] Figure 3 is a schematic block diagram of the video decoder involved in the embodiments of this application.

[0054] Figure 4 is an example of the VVC / H.266 framework involved in the embodiments of this application.

[0055] Figure 5 is an example of a reference block involved in an embodiment of this application.

[0056] Figure 6 is an example of motion compensation involved in an embodiment of this application.

[0057] Figure 7 is a schematic flowchart of the decoding method provided in the embodiments of this application.

[0058] Figure 8 is an example of the encoding process provided in the embodiments of this application.

[0059] Figure 9 is an example of the decoding process provided in an embodiment of this application.

[0060] Figure 10 is an example of interpolation filtering based on enhancement blocks provided in the embodiments of this application.

[0061] Figure 11 is another example of interpolation filtering based on enhancement blocks provided in the embodiments of this application.

[0062] Figure 12 is a schematic flowchart of the inter-frame prediction process without introducing enhancement blocks provided in an embodiment of this application.

[0063] Figure 13 is a schematic flowchart of the inter-frame prediction process with the introduction of enhancement blocks provided in an embodiment of this application.

[0064] Figure 14 is a schematic structural diagram of Dense-RVCNN provided in an embodiment of this application.

[0065] Figure 15 is a schematic structural diagram of the DenseNet structure in Dense-RVCNN provided in the embodiments of this application.

[0066] Figure 16 is a schematic flowchart of the encoding method provided in the embodiments of this application.

[0067] Figure 17 is a schematic flowchart of the neural network training method provided in the embodiments of this application.

[0068] Figure 18 is an example of the input and label when training a neural network according to an embodiment of this application.

[0069] Figure 19 is a schematic block diagram of the decoder provided in an embodiment of this application.

[0070] Figure 20 is a schematic block diagram of the encoder provided in an embodiment of this application.

[0071] Figure 21 is a schematic block diagram of a neural network training device provided in an embodiment of this application.

[0072] Figure 22 is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0073] The solutions provided in this application can be applied to the field of digital video coding technology.

[0074] For example, the technical fields to which the solutions provided in this application are applicable include, but are not limited to: image encoding and decoding, video encoding and decoding, hardware video encoding and decoding, dedicated circuit video encoding and decoding, and real-time video encoding and decoding. Furthermore, the solutions provided in this application can be combined with Audio Video Coding Standard (AVS), AVS2, or AVS3. For example, these include, but are not limited to: H.264 / Audio Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), and H.266 / Versatile Video Coding (VVC). Additionally, the solutions provided in this application can be used for lossy compression of images, or for lossless compression of images. This lossless compression can be visually lossless compression or mathematically lossless compression.

[0075] For ease of understanding, the video encoding and decoding system involved in the embodiments of this application will be introduced first with reference to Figure 1.

[0076] Figure 1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of this application.

[0077] As shown in Figure 1, the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120.

[0078] The encoding device 110 encodes (can be understood as compressing) the video data to generate a bitstream, and transmits the bitstream to the decoding device 120. The decoding device 120 decodes the bitstream generated by the encoding device 110 to obtain the decoded video data.

[0079] Encoding device 110 can be understood as a device with video encoding function, and decoding device 120 can be understood as a device with video decoding function. That is, the embodiments of this application include encoding device 110 and decoding device 120 in a wider range of devices, such as smartphones, desktop computers, mobile computing devices, laptops (e.g., laptop computers), tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, vehicle computers, etc.

[0080] Encoding device 110 can transmit encoded video data (e.g., bitstream) to decoding device 120 via channel 130.

[0081] Channel 130 may include one or more media and / or devices capable of transmitting encoded video data from encoding device 110 to decoding device 120.

[0082] Channel 130 may include one or more communication media that enable encoding device 110 to transmit encoded video data directly to decoding device 120 in real time. Encoding device 110 may modulate the encoded video data according to a communication standard and transmit the modulated video data to decoding device 120. The communication media may include wireless communication media, such as radio frequency spectrum. The communication media may also include wired communication media, such as one or more physical transmission lines.

[0083] Channel 130 may include a storage medium that can store video data encoded by encoding device 110. The storage medium includes various local access data storage media, such as optical discs, DVDs, flash memory, etc. In this example, decoding device 120 can retrieve the encoded video data from the storage medium.

[0084] Channel 130 may include a storage server that stores the video data encoded by encoding device 110. In this example, decoding device 120 can download the stored encoded video data from the storage server. Optionally, the storage server can store and transmit the encoded video data to decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0085] Encoding device 110 includes video encoder 112 and output interface 113.

[0086] The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter. The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored on a storage medium or a storage server for later retrieval by the decoding device 120.

[0087] In addition to the video encoder 112 and the input interface 113, the encoding device 110 may also include a video source 111.

[0088] Video source 111 may include at least one of a video capture device (e.g., a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate the video data. Video encoder 112 encodes the video data from video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains encoding information for the pictures or picture sequences in the form of a bitstream. The encoding information may include encoded image data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. The SPS may contain parameters applied to one or more sequences. The PPS may contain parameters applied to one or more pictures. A syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the bitstream.

[0089] Decoding device 120 includes input interface 121 and video decoder 122. Input interface 121 may include receiver and / or modem.

[0090] In addition to the input interface 121 and the video decoder 122, the decoding device 120 may also include a display device 123.

[0091] Input interface 121 can receive encoded video data via channel 130. Video decoder 122 decodes the encoded video data to obtain decoded video data and transmits the decoded video data to display device 123. Display device 123 displays the decoded video data. Display device 123 can be integrated with decoding device 120 or external to decoding device 120. Display device 123 can include various display devices, such as liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or other types of display devices.

[0092] It should be understood that Figure 1 is merely an example of this application and should not be construed as a display of this application. That is to say, the technical solutions of the embodiments of this application are not limited to the system framework shown in Figure 1. For example, the technology of this application can also be applied to one-sided video encoding or one-sided video decoding.

[0093] The video coding framework involved in the embodiments of this application is described below.

[0094] Figure 2 is a schematic block diagram of the video encoder 200 involved in an embodiment of this application.

[0095] It should be understood that the video encoder 200 can be applied to image data in luma / chroma (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents luminance (Luma), Cb (U) represents blue chroma, Cr (V) represents red chroma, and U and V represent chroma (Chroma) used to describe color and saturation. For example, in color format, 4:2:0 means that there are 4 luminance components and 2 chroma components (YYYYCbCr) for every 4 pixels; 4:2:2 means that there are 4 luminance components and 4 chroma components (YYYYCbCrCbCr) for every 4 pixels; and 4:4:4 means full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0096] The video encoder 200 reads video data and divides each frame into several coding tree units (CTUs). In some examples, CTUs may be called "tree blocks," "largest coding units" (LCUs), or "coding tree blocks" (CTBs). Each CTU can be associated with a pixel block of equal size within the image. Each pixel corresponds to one luminance (luma) sample and two chrominance (chroma) samples. Therefore, each CTU can be associated with one luminance sample block and two chrominance sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU can be further divided into several coding units (CUs) for encoding. CUs can be rectangular or square blocks. CUs can be further divided into prediction units (PUs) and transform units (TUs), thus separating encoding, prediction, and transform for more flexible processing. In one example, CTU is partitioned into CU using a quadtree, and CU is partitioned into TU and PU using a quadtree.

[0097] The video encoder and decoder support various PU sizes. Assuming a specific CU size of 2N×2N, the video encoder and decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and also support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0098] As shown in Figure 2, the video encoder 200 may include: a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filtering unit 260, a decoded image buffer 270, and an entropy coding unit 280. It should be noted that the video encoder 200 may contain more, fewer, or different functional components. In this application, the current block may be referred to as the current coding unit (CU) or the current prediction unit (PU), etc. The prediction block may also be referred to as the predicted image block or the image prediction block, and the reconstructed image block may also be referred to as the reconstruction block or the image reconstruction block.

[0099] The prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame prediction unit 212. Because there is strong correlation between adjacent pixels in a video frame, intra-frame prediction is used in video encoding and decoding techniques to eliminate spatial redundancy between adjacent pixels. Because there is strong similarity between adjacent frames in a video, inter-frame prediction is used in video encoding and decoding techniques to eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.

[0100] Inter-frame prediction unit 211 can be used for inter-frame prediction, which can include motion estimation and motion compensation. It can reference image information from different frames. Inter-frame prediction uses motion information to find a reference block from the reference frame and generates a prediction block based on the reference block to eliminate temporal redundancy. The frames used for inter-frame prediction can be P-frames and / or B-frames, where P-frames refer to forward prediction frames and B-frames refer to bidirectional prediction frames. The motion information includes a list of reference frames, the reference frame index, and motion vectors. Motion vectors can be integer-pixel or fractional-pixel. If the motion vector is fractional-pixel, interpolation filtering needs to be used in the reference frame to create the required fractional-pixel blocks. Here, the integer-pixel or fractional-pixel blocks in the reference frame found based on the motion vectors are called reference blocks. Some techniques directly use the reference block as the prediction block, while others process the reference block further to generate the prediction block. Processing the reference block further to generate the prediction block can also be understood as using the reference block as the prediction block and then processing it to generate a new prediction block.

[0101] Intra-prediction unit 212 refers only to information from the same frame image to predict pixel information within the current code image block, thereby eliminating spatial redundancy. The frame used for intra-prediction can be an I-frame.

[0102] Intra-frame prediction has multiple prediction modes. Taking the international digital video coding standards H-series as an example, the H.264 / AVC standard has 8 angular prediction modes and 1 non-angular prediction mode, while H.265 / HEVC extends this to 33 angular prediction modes and 2 non-angular prediction modes. HEVC uses Planar, DC, and 33 angular modes for a total of 35 intra-frame prediction modes. VVC uses Planar, DC, and 65 angular modes for a total of 67 intra-frame prediction modes. It should be noted that with the increase in angular modes, intra-frame prediction becomes more accurate and better meets the needs of high-definition and ultra-high-definition digital video development.

[0103] The residual unit 220 can generate a residual block of the CU based on the pixel block of the CU and the prediction block of the PU of the CU. For example, the residual unit 220 can generate a residual block of the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel block of the CU and the corresponding sample in the prediction block of the PU of the CU.

[0104] Transform / quantization unit 230 can quantize transform coefficients. Transform / quantization unit 230 can quantize transform coefficients associated with the TU of the CU based on the quantization parameter (QP) value associated with the CU. Video encoder 200 can adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0105] The inverse transform / quantization unit 240 can apply inverse quantization and inverse transform to the quantized transform coefficients to reconstruct the residual block from the quantized transform coefficients.

[0106] The reconstruction unit 250 can add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by the prediction unit 210 to produce a reconstructed image block associated with the TU. By reconstructing the sampled blocks of each TU of the CU in this way, the video encoder 200 can reconstruct the pixel blocks of the CU.

[0107] The loop filtering unit 260 processes the pixels after inverse transform and inverse quantization to compensate for distortion information and provide a better reference for subsequent encoded pixels. For example, it can perform deblocking filtering to reduce the block effect of pixel blocks associated with the CU. In some embodiments, the loop filtering unit 260 includes a deblocking filtering unit and a sample adaptive compensation / adaptive loop filtering (SAO / ALF) unit, wherein the deblocking filtering unit is used to remove block effects, and the SAO / ALF unit is used to remove ringing effects.

[0108] The decoded image buffer 270 can store reconstructed pixel blocks. The inter-frame prediction unit 211 can use a reference image containing the reconstructed pixel blocks to perform inter-frame prediction on PUs of other images. In addition, the intra-frame prediction unit 212 can use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra-frame prediction on other PUs in the same image as the CU.

[0109] Entropy coding unit 280 can receive quantized transform coefficients from transform / quantization unit 230. Entropy coding unit 280 can perform one or more entropy coding operations on the quantized transform coefficients to produce entropy-coded data.

[0110] Figure 3 is a schematic block diagram of the video decoder involved in the embodiments of this application.

[0111] As shown in Figure 3, the video decoder 300 includes: an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transformation unit 330, a reconstruction unit 340, a loop filtering unit 350, and a decoded image buffer 360. It should be noted that the video decoder 300 may contain more, fewer, or different functional components.

[0112] The video decoder 300 can receive a bitstream. The entropy decoding unit 310 can parse the bitstream to extract syntax elements from it. As part of parsing the bitstream, the entropy decoding unit 310 can parse the entropy-encoded syntax elements in the bitstream. The prediction unit 320, the dequantization / transform unit 330, the reconstruction unit 340, and the loop filtering unit 350 can decode the video data based on the syntax elements extracted from the bitstream, i.e., generate decoded video data.

[0113] The prediction unit 320 includes an intra-frame prediction unit 322 and an inter-frame prediction unit 321.

[0114] Intra-prediction unit 322 can perform intra-prediction to generate prediction blocks for the PU. Intra-prediction unit 322 can use an intra-prediction mode to generate prediction blocks for the PU based on pixel blocks of spatially adjacent PUs. Intra-prediction unit 322 can also determine the intra-prediction mode of the PU based on one or more syntax elements parsed from the bitstream.

[0115] Inter-frame prediction unit 321 can construct a first reference image list (list 0) and a second reference image list (list 1) based on the syntax elements parsed from the bitstream. Furthermore, if the PU uses inter-frame prediction coding, the entropy decoding unit 310 can parse the motion information of the PU. Inter-frame prediction unit 321 can determine one or more reference blocks of the PU based on the motion information of the PU. Inter-frame prediction unit 321 can generate prediction blocks for the PU based on one or more reference blocks of the PU.

[0116] The dequantization / transform unit 330 reversibly quantizes (i.e., dequantizes) the transform coefficients associated with the TU. The dequantization / transform unit 330 can use the QP value associated with the CU of the TU to determine the degree of quantization. After dequantizing the transform coefficients, the dequantization / transform unit 330 can apply one or more inverse transforms to the dequantized transform coefficients to produce a residual block associated with the TU.

[0117] The reconstruction unit 340 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 can add the sample of the residual block to the corresponding sample of the prediction block to reconstruct the pixel block of the CU, thereby obtaining the reconstructed image block.

[0118] The loop filter unit 350 can perform deblocking filtering operations to reduce the block effect of pixel blocks associated with the CU.

[0119] The video decoder 300 can store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 can use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0120] The basic process of video encoding and decoding is as follows: At the encoding end, a frame image is divided into blocks. For the current block, the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual unit 220 can calculate a residual block based on the prediction block and the original block of the current block, that is, the difference between the prediction block and the original block of the current block. This residual block can also be called residual information. This residual block is transformed and quantized by the transform / quantization unit 230, which can remove information that is not sensitive to the human eye to eliminate visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization unit 230 can be called a temporal residual block, and the temporal residual block after transformation and quantization by the transform / quantization unit 230 can be called a frequency residual block or a frequency domain residual block. The entropy coding unit 280 receives the quantized change coefficients output by the change quantization unit 230, and can perform entropy coding on the quantized change coefficients to output a bitstream. For example, the entropy coding unit 280 can eliminate character redundancy based on the target context model and the probability information of the binary bitstream.

[0121] At the decoding end, the entropy decoding unit 310 can parse the bitstream to obtain the prediction information and quantization coefficient matrix of the current block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction to generate the prediction block of the current block based on the prediction information. The dequantization / transform unit 330 uses the quantization coefficient matrix obtained from the bitstream to perform dequantization and inverse transform on the quantization coefficient matrix to obtain the residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain the reconstructed block. The reconstructed blocks form the reconstructed image. The loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or based on the blocks to obtain the decoded image. The encoding end also needs to use similar operations as the decoder to obtain the decoded image. This decoded image can also be called the reconstructed image, which can be used as a reference frame for inter-frame prediction in subsequent frames.

[0122] It should be noted that the block partitioning information determined by the encoder, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, are carried in the bitstream when necessary. The decoder determines the same block partitioning information, prediction, transform, quantization, entropy coding, and loop filtering mode information or parameter information as the encoder by parsing the bitstream and analyzing existing information, thereby ensuring that the decoded image obtained by the encoder and the decoder are the same. Furthermore, in this embodiment, due to the need for parallel processing, the image can be divided into slices, etc., and slices within the same image can be processed in parallel, meaning there is no data dependency between them. A "frame" is a commonly used term, generally understood as one image. Of course, the frame mentioned in this application can also be replaced with an image or slice, etc.

[0123] It is worth noting that the above is the basic process of a video codec under a block-based hybrid coding framework. With the development of technology, some modules or steps of this framework or process may be optimized. This application is applicable to the basic process of a video codec under this block-based hybrid coding framework, but is not limited to this framework and process.

[0124] Figure 4 is an example of another coding framework involved in the embodiments of this application.

[0125] As shown in Figure 4, the encoding and decoding process under this encoding framework includes: First, the encoder divides the original video signal into many image blocks, processes each image block individually, and then passes it through the prediction module. The prediction module is mainly divided into intra-frame prediction and inter-frame prediction. Intra-frame prediction mainly infers the content of the current block based on the spatial relationship between images. Inter-frame prediction predicts the content of the current block based on the spatial and temporal relationship between consecutive frames. The difference between the predicted image (predicted block) and the original signal image (original block) is used to obtain the residual image (residual block). Then, the residual block is further compressed by operations such as change quantization and entropy encoding, and finally, the video bitstream file is output. The decoder can reconstruct the video signal based on the bitstream file output by the encoder. Most common video compression is lossy compression, which sacrifices image quality to achieve a high compression ratio. Therefore, the video signal reconstructed by the decoder is lossy compared to the original video.

[0126] Inter-frame prediction is primarily used to eliminate temporal redundancy in video signals.

[0127] In video signals, adjacent video frames often contain a large amount of similar content. As shown in Figure 5, when encoding the current block in the current frame (frame T), the most similar image block (reference block) can be found in the reconstructed reference frame (frame Tn). Since the two image blocks are extremely similar, we only need to transmit the difference between the two blocks and the motion vector (MV) between them. Compared to directly transmitting the information of the current block image, this method greatly compresses the image information that needs to be transmitted. The process of finding the reference block is called motion estimation (ME). However, due to the inherent spatial discretization characteristics of digital video, the block translation may not be perfectly aligned with the pixels.

[0128] Figure 6 is an example of motion compensation involved in an embodiment of this application.

[0129] As shown in Figure 6, the block containing the dot represents the integer pixel position. The current block can usually find the position of the reference block based on the integer pixel MV (Int MV, IMV), but this method is still not precise enough. Optionally, the reference block can be interpolated first, and then the image at the fractional pixel position can be selected, thereby further improving the prediction accuracy. The fractional pixel image is called the prediction block, and its displacement is called fractional precision displacement (FMV). The process of selecting the optimal prediction block based on the reference block is called motion compensation (MC). To derive the fractional pixel image from the integer pixel image, VVC uses discrete cosine interpolation filters with different parameters to filter the integer pixel image. The predicted block obtained after filtering may be closer to the original block than the reference block.

[0130] As shown above, the quality of the reference block directly affects the coding quality of the current block. The closer the reference block image is to the original block, the smaller the residual block information needs to be transmitted, and the lower the video bitrate. In this direction, most research focuses on optimizing interpolation filters, using more suitable interpolation filter tap coefficients to generate better prediction blocks.

[0131] With the development of deep learning, many scholars have used network models generated by deep learning to derive or refine images of predicted blocks. Most of these methods use neural network models to replace interpolation filtering in inter-frame prediction to improve the accuracy of predicted blocks. One feasible approach involves providing a neural network model with three convolutional layers and training the model on different datasets using reference blocks and original blocks corresponding to different fractional interpolation filters. Finally, the convolutional layer parameters in the network model are compressed into a matrix, which is then used to replace the interpolation filter tap coefficients in H.266 / VVC inter-frame prediction. This method, which directly replaces the interpolation filtering process with a neural network model, has the following two drawbacks:

[0132] 1. During inter-frame prediction, the current block often performs hundreds of interpolation filters to find the optimal prediction value. Calling the neural network model every time would greatly increase the complexity of the encoding.

[0133] 2. Use network models instead of interpolation filters; different network models cannot be combined.

[0134] In view of this, this application introduces a method for enhancing the quality of a reference region of a reference frame to obtain an enhanced image. Specifically, an enhancement block corresponding to the reference block is found in the enhanced image, and interpolation filtering is performed to obtain a prediction block. This technique only requires quality enhancement of the frame after the reference region reconstruction is completed, effectively reducing the complexity caused by calling the neural network, and improving the image quality of the reference region, which is beneficial for more accurate prediction. First, this application generates a dataset to train the proposed neural network; second, the neural network is combined into the H.266 / VVC framework to enhance the reconstructed reference region. Multiple enhanced images are generated for the same reference image using different network parameters of the neural network. Based on the minimum side length of the CU block, an enhancement block matching the size of the CU block is determined on different enhanced images. The enhanced block is used instead of the reference block for subsequent motion compensation and other operations. Finally, experiments verify the effectiveness of this method in improving video encoding and decoding performance.

[0135] The decoding method provided in this application embodiment will be described below with reference to Figure 7, taking the decoding end as an example.

[0136] Figure 7 is a schematic flowchart of the decoding method 400 provided in an embodiment of this application. It should be understood that the decoding method 400 can be executed by a decoder. For example, the decoding method 400 can be executed by the video decoder 122 shown in Figure 1 or the video decoder 300 shown in Figure 3. For ease of description, a decoder will be used as an example below.

[0137] As shown in Figure 7, the decoding method 400 may include:

[0138] S410, the decoder decodes the bitstream and determines the motion vector (MV) of the current block.

[0139] For example, the decoder determines the MV of the current block by decoding the bitstream.

[0140] For example, the MV of the current block may include an MV with integer pixel precision and an MV with fractional pixel precision.

[0141] S420, the decoder determines the reference block of the current block in the reference image of the current block based on the motion parameters of the current block.

[0142] For example, the reference image may be a decoded image.

[0143] For example, the reference block may be a component reference block, such as a chromaticity component reference block or a luminance component reference block.

[0144] S430, the decoder performs quality enhancement on the reference block to obtain an enhanced block.

[0145] For example, the decoder enhances the quality of the reference block using a neural network to obtain the enhanced block.

[0146] The neural network includes, but is not limited to, traditional learning models, ensemble learning models, or deep learning models. Traditional learning models include, but are not limited to, tree models (regression trees) or logistic regression (LR) models; ensemble learning models include, but are not limited to, improved gradient boosting algorithms (XGBoost) or random forest models; deep learning models include, but are not limited to, deep neural networks (DNN), convolutional neural networks (CNN), recurrent neural networks (RNN), dynamic Bayesian networks (DBN), and stacked auto-encoder networks (SAE). Of course, in other embodiments of this application, other machine learning models or even combinations of multiple models can be used.

[0147] Of course, in other alternative embodiments, the decoder may also enhance the quality of the reference block using other methods, such as enhancing the quality of the reference block based on information obtained from decoding the bitstream. This application does not specifically limit this.

[0148] S440, the decoder determines the prediction block of the current block based on the enhancement block.

[0149] For example, the decoder can perform motion compensation on the augmented block to obtain the predicted block.

[0150] This motion compensation includes, but is not limited to, the following two types:

[0151] Inter-prediction filter: To mitigate the discontinuity between predicted pixels and neighboring pixels, the predicted values ​​in the reference block are filtered. Filtering is achieved through a weighted average. The filter input includes the predicted pixel and neighboring pixels, with weighting factors determined based on the positional relationships between pixels.

[0152] Affine Motion Compensation (AMC): This approach considers only translational motion when dealing with object motion, whereas real-world scenes often involve more complex motion patterns such as rotation and scaling. To accommodate these different motion patterns, AMC divides the CU (Complex Unit) into sub-blocks and generates a MV (Motion View) for each sub-block using an affine model. Affine models include a 4-parameter (2-control-point CPMV) model and a 6-parameter (3-control-point CPMV) model. After determining reference sub-blocks based on their MVs, prediction sub-blocks are obtained by interpolating and filtering these reference sub-blocks. These prediction sub-blocks then form a prediction block.

[0153] In this embodiment, the decoder obtains an enhanced block by enhancing the quality of a reference block, and then determines the prediction block of the current block based on the enhanced block. In other words, the decoder converts the quality enhancement process of the prediction block into the quality enhancement process of the reference block. This avoids coupling the quality enhancement process of the prediction block to the prediction block determination process, reduces decoding complexity, and improves decoding performance.

[0154] The technical solution proposed in this application is implemented in the VVC testing software VTM11.0_nnvc_2.0. The test sequences used are the Class B, Class C, Class D, and Class E sequences given in the general test conditions. The results in Table 1 were obtained by encoding in LDP mode with QP22, 27, 32, and 37 settings. The values ​​in the table are BD-rates, which are a way to measure algorithm performance. The table shows the changes in Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) of the algorithm provided in this application compared to the original algorithm. A negative value overall indicates a performance improvement, and the larger the absolute value, the greater the performance improvement.

[0155] Table 1

[0156] As shown in Table 1, the test results show that for categories B to E, the solution provided in this application improves PSNR by 0.8% compared to the original algorithm, and improves SSIM by 1.22% compared to the original algorithm. This indicates that the technical solution provided in this application can improve decoding performance.

[0157] In some embodiments, S430 may include:

[0158] The decoder performs quality enhancement on the reference region in the reference image that includes the reference block to obtain an enhanced image; then, based on the position of the reference block in the reference region, the enhanced block is determined in the enhanced image.

[0159] For example, the decoder uses a neural network to enhance the quality of a reference image, resulting in an enhanced image; then, based on the position of the reference block in the reference image, it determines the enhanced block in the enhanced image. Of course, in other alternative embodiments, the decoder may also use a neural network to directly enhance the quality of the reference block.

[0160] Figure 8 is an example of the encoding process provided in the embodiments of this application.

[0161] As shown in Figure 8, at the encoder end, a reference block for the current block can be found based on motion estimation. Then, an enhancement block corresponding to the position of the reference block is found on the enhanced image. This enhanced block is then subjected to subsequent motion compensation to obtain the prediction block. The enhancement image can be an image that enhances the quality of a reference region including the reference block within the reference image. After obtaining the prediction block, the encoder calculates the difference between the prediction block and the original image of the current block to obtain the residual information of the current block. This residual information is then transformed and quantized, and entropy coding is performed on the transformed and quantized information to obtain the bitstream file.

[0162] Figure 9 is an example of the decoding process provided in an embodiment of this application.

[0163] As shown in Figure 9, at the decoding end, the reference block of the current block can be found in the reconstructed image based on the decoded MV; then, the enhancement block corresponding to the position of the reference block is found in the enhanced image, and motion compensation is performed on the enhanced block to obtain the prediction block. The enhancement image can be an image obtained by enhancing the quality of the reference region, which includes the reference block, in the reference image. After obtaining the prediction block of the current block, the decoder can determine the reconstructed video based on the prediction block and the residual information obtained from entropy decoding, inverse quantization, and inverse transformation of the current block; furthermore, loop filtering or post-processing can be performed on the reconstructed video.

[0164] In some embodiments, the decoder identifies the block in the enhanced image whose position is the same as that of the reference block in the reference region as the enhanced block.

[0165] For example, the decoder determines the position of the reference pixel of the enhancement block as the pixel whose position in the enhanced image is the same as the reference pixel of the reference block. The reference pixel in the reference block can be any pixel in the reference block, such as, but not limited to, the pixel at the top right corner, bottom right corner, top left corner, bottom left corner, and center position.

[0166] In some embodiments, S440 may include:

[0167] The decoder performs boundary expansion and interpolation filtering on the augmented block to obtain the predicted block.

[0168] For example, after the decoder performs boundary expansion on the augmentation block, it uses a filter to filter each pixel in the expanded augmentation block to obtain the prediction block. For instance, the input to the filter includes the predicted pixel and neighboring pixels, and the weight factors for the predicted pixel and neighboring pixels are determined based on the positional relationship between the pixels.

[0169] For example, if the decoder determines the position of a reference pixel in the enhanced image that is at the same position as the reference pixel of the reference block as the position of the enhanced block, then when the decoder determines the prediction block based on the enhanced block, it can directly perform boundary expansion and interpolation filtering on the enhanced block to obtain the prediction block.

[0170] Figure 10 is an example of interpolation filtering based on enhancement blocks provided in the embodiments of this application.

[0171] As shown in Figure 10, during conventional motion compensation, the decoder determines the reference block for the current block based on vectors. Since fractional interpolation filters reduce image size, the boundary of the reference pixel needs to be expanded by 4 pixels before interpolation filtering to ensure consistent image size before and after filtering. The decoder adds the filtered prediction block to the residual block to obtain the reconstructed block. Correspondingly, the encoder subtracts the filtered prediction block from the original block to obtain the residual block, and then performs transform quantization and other operations on the residual block. For interpolation filtering without introducing an enhancement block, as shown in Method 1 in the figure, subsequent operations such as interpolation filtering can be directly performed on the reference block. For the interpolation filtering process after introducing an enhancement block in this application, as shown in Method 2 in the figure, the boundary expansion image of the block with the same position as the reference block can be found first on the enhanced image as the enhancement block, and then subsequent operations such as interpolation filtering can be performed on the enhancement block.

[0172] In some embodiments, S420 may include:

[0173] The decoder divides the current block into at least one sub-block; then, based on the motion parameters of the current block, it determines the motion parameters of the at least one sub-block; based on the motion parameters of the at least one sub-block, it determines at least one reference sub-block included in the reference block; wherein, S430 may include:

[0174] The decoder determines a first region where the at least one reference sub-block is located based on the position of the at least one reference sub-block in the reference region; then, it performs boundary expansion on the first region to obtain a second region; and the block in the enhanced image whose position corresponds to the second region is determined as the enhanced block.

[0175] For example, the decoder divides the current block into at least one sub-block; then, based on the MV of the current block, it determines the MV of the at least one sub-block; based on the MV of the at least one sub-block, it determines at least one reference sub-block included in the reference block; wherein, S430 may include:

[0176] The decoder determines a first region where the at least one reference sub-block is located based on the position of the at least one reference sub-block in the reference region; then, it performs boundary expansion on the first region to obtain a second region; and the block in the enhanced image whose position corresponds to the second region is determined as the enhanced block.

[0177] For example, the decoder generates an image (MV) for each sub-block using an affine model. The affine model includes a 4-parameter (2-control-point CPMV) model and a 6-parameter (3-control-point CPMV) model. The decoder determines at least one reference sub-block based on the MV of each sub-block, and uses the region including at least one reference sub-block as a first region. Then, the first region is bounded to obtain a second region, and the region in the enhanced image that is identical to this second region is determined as the enhancement block.

[0178] In some embodiments, S440 may include:

[0179] The decoder identifies the sub-block in the enhancement block whose position is the same as that of the at least one reference sub-block in the reference region as at least one enhancement sub-block included in the enhancement block; then, it performs boundary expansion and interpolation filtering on the at least one enhancement sub-block to obtain at least one prediction sub-block included in the prediction block.

[0180] For example, the decoder obtains each prediction sub-block by interpolating and filtering each reference sub-block, and the prediction sub-blocks can be combined to form a prediction block.

[0181] In some embodiments, the first region is the smallest region including the at least one reference sub-block.

[0182] For example, the first region is a minimal rectangular region including at least one reference sub-block. Of course, the first region can also be a region of other shapes, and this application does not specifically limit it.

[0183] Figure 11 is another example of interpolation filtering based on enhancement blocks provided in the embodiments of this application.

[0184] As shown in Figure 11, in affine motion compensation, for the interpolation filtering process without introducing enhancement blocks, as shown in Method 1, the current CU is divided into several 4×4 sub-blocks, and the MV of each sub-block is calculated according to a 4-parameter or 6-parameter affine model. Each current sub-block can find its corresponding reference sub-block through motion vectors, and then perform boundary expansion, fractional interpolation, and other operations on the reference sub-block to obtain the prediction sub-block; the prediction sub-blocks are combined to obtain the prediction block. For the interpolation filtering process after introducing enhancement blocks in this application, as shown in Method 2, the smallest rectangular region containing all reference sub-blocks can be found, and the corresponding enhancement block can be found on the enhanced image; then, the corresponding expanded sub-block is found in the enhancement block and fractional interpolation is performed, and the interpolated prediction sub-blocks are combined to form the prediction block.

[0185] In some embodiments, the reference region is the entire region or a portion of the reference image.

[0186] In some embodiments, the partial region is the region containing any of the following: image patch, sub-picture, tile, slice.

[0187] For example, the image may be a decoding block, which may be a CU or a CTU.

[0188] For example, a reference image can be divided into one or more slices, or stripes, each with its own independent data encoding.

[0189] For example, a reference image can be divided into several rectangular regions, called tiles, in both horizontal and vertical directions. The tiles are not required to be uniformly distributed. Typically, each tile contains approximately the same amount of CTU data. Each tile can be encoded independently, and during encoding, the CTUs within each tile are encoded in the order they were scanned.

[0190] It should be understood that the reference image can be divided into several slices or several tiles, both for the purpose of independent encoding. Some slices can contain multiple tiles, and similarly, some tiles can contain multiple slices. The number of CTUs in a tile and the number of CTUs in a slice are independent of each other.

[0191] In some embodiments, S430 may include:

[0192] The decoder extracts features from the reference region to obtain a residual image; then, the reference region and the residual image are weighted to obtain the enhanced image.

[0193] For example, the weights of the reference region and the residual image can be predefined weight values.

[0194] For example, after the decoder obtains the residual value, it can directly add the reference region and the residual image to obtain the enhanced image.

[0195] In some embodiments, the decoder determines a first parameter for feature extraction of the reference region based on the size of the current block; then, based on the first parameter, features are extracted from the reference region to obtain the residual image.

[0196] For example, the decoder can determine a first set of parameters for feature extraction of the reference region based on the size of the current block. This first set of parameters may include at least one parameter, which may include the first parameter. The first set of parameters may be parameters involved in various operations performed on the reference region. In other words, the first parameter is related to the computational process used for feature extraction of the reference region.

[0197] In some embodiments, the decoder determines the parameter corresponding to the minimum side length of the current block as the first parameter.

[0198] For example, the decoder can determine the parameter corresponding to the minimum side length of the current block as the first parameter based on the mapping relationship between multiple side lengths and multiple parameters. This mapping relationship can be predefined information.

[0199] In some embodiments, the decoder extracts features from the reference region to obtain first feature information; then, it performs multi-scale feature extraction on the first feature information and concatenates the extracted multi-scale feature information to obtain second feature information; based on the second feature information and the feature information obtained by feature extraction from the second feature information, it determines third feature information; it performs multi-scale feature extraction on the third feature information and concatenates the extracted multi-scale feature information to obtain fourth feature information; and it converts the fourth feature information into feature information with the same number of channels as the reference region to obtain the residual image.

[0200] For example, the first feature information and the second feature information have the same number of channels.

[0201] For example, the second feature information and the third feature information have the same number of channels.

[0202] For example, the third feature information and the fourth feature information have the same number of channels.

[0203] For example, the first feature information, the second feature information, or the third feature information includes feature information on multiple channels, and the number of channels in the reference region is the feature information on one or more channels. For example, when the reference image is a component reference image, the number of channels in the reference region can be 1.

[0204] In some embodiments, S430 may include:

[0205] The decoder uses a Dense-Residual Variable-size Convolutional Neural Network (Dense-RVCNN) to enhance the quality of the reference region, thereby obtaining the enhanced image.

[0206] For example, Dense-RVCNN can be a densely connected neural network used for residual convolution.

[0207] Figure 12 is a schematic flowchart of the inter-frame prediction process 450 without introducing enhancement blocks provided in an embodiment of this application.

[0208] As shown in Figure 12, the inter-frame prediction process 450 without introducing enhancement blocks may include:

[0209] S451, Get the current block.

[0210] S452, Determine the matching reference block in the reference image.

[0211] When the encoder performs inter-frame prediction for the current block, it first finds the block most similar to the current block as a reference block through motion estimation. For example, a Tzsearch can be performed on an existing reference image, and the image block most similar to the current block (with the lowest RDcost) is the reference block. Correspondingly, when the decoder performs inter-frame prediction for the current block, it can determine the block most similar to the current block as a reference block through the motion vectors obtained from the decoded bitstream.

[0212] S453, determine the reference block after subpixel precision displacement.

[0213] After determining the reference block, the encoder performs motion compensation on the reference block to obtain a reference block with multiple sub-pixel precision displacements. Similarly, after determining the reference block, the decoder can also determine a reference block with multiple sub-pixel precision displacements by performing motion compensation on the reference block.

[0214] S454, determine the reference block after optimal subpixel precision displacement.

[0215] After the encoder obtains multiple reference blocks with sub-pixel precision shifts, it can determine the optimal sub-pixel precision shifted reference block as the reference block most similar to the current block (with the smallest RDcost). Similarly, after the decoder obtains multiple sub-pixel precision shifted reference blocks, it can also determine the optimal sub-pixel precision shifted reference block as the reference block most similar to the current block (with the smallest RDcost). This optimal sub-pixel precision shifted reference block can be used as the prediction block for the current block.

[0216] Figure 13 is a schematic flowchart of the inter-frame prediction process 460 with the introduction of enhancement blocks provided in an embodiment of this application.

[0217] As shown in Figure 13, the inter-frame prediction process 460 for introducing enhancement blocks may include:

[0218] S461, Get the current block.

[0219] S462, Determine the matching reference block in the reference image.

[0220] When the encoder performs inter-frame prediction for the current block, it first finds the block most similar to the current block as a reference block through motion estimation. For example, a Tzsearch can be performed on an existing reference image, and the image block most similar to the current block (with the lowest RDcost) is the reference block. Correspondingly, when the decoder performs inter-frame prediction for the current block, it can determine the block most similar to the current block as a reference block through the motion vectors obtained from the decoded bitstream.

[0221] S463, the block in the enhanced image that is located at the same position as the reference block is the enhanced block.

[0222] Once the reference image is constructed, a neural network is used to enhance the quality of the reference region, including the reference block, to obtain an enhanced image. The enhanced block at the same location as the reference block is then found in the enhanced image, and this enhanced block replaces the original reference block for subsequent operations.

[0223] S464, determines the reference block after subpixel precision displacement.

[0224] After determining the reference block, the encoder performs motion compensation on the reference block to obtain a reference block with multiple sub-pixel precision displacements. Similarly, after determining the reference block, the decoder can also determine a reference block with multiple sub-pixel precision displacements by performing motion compensation on the reference block.

[0225] S465, determine the reference block after optimal subpixel precision displacement.

[0226] After the encoder obtains multiple reference blocks with sub-pixel precision shifts, it can determine the optimal sub-pixel precision shifted reference block as the reference block most similar to the current block (with the smallest RDcost). Similarly, after the decoder obtains multiple sub-pixel precision shifted reference blocks, it can also determine the optimal sub-pixel precision shifted reference block as the reference block most similar to the current block (with the smallest RDcost). This optimal sub-pixel precision shifted reference block can be used as the prediction block for the current block.

[0227] For example, prior to S463, the inter-frame prediction process 460 for introducing enhancement blocks may also include a training process for a neural network (or a pre-trained neural network), which may include:

[0228] S4621, encodes the sample video.

[0229] For example, 650 videos can be randomly selected from the BVI-DVC dataset (a dataset containing a large number of scene YUV video files). Using the VTM-9.3 encoder, with the encoding mode set to LDP mode and the quantization parameter (Qp) set to 22, the first 32 images of these videos are encoded to generate a bitstream. Of course, random access (RA) and low latency (LB) configuration modes can also be used, and this application does not specifically limit this.

[0230] S4622, determine the lossless current block and the reference block with optimal subpixel precision displacement.

[0231] For example, for the current block, the bitstream contains motion vector information with integer precision and motion vector information with fractional precision. The training device can deduce the integer pixel position of the reference block from the integer pixel position of the current block. Then, the prediction block is obtained by sub-pixel displacement of the integer pixel reference block through a sub-pixel interpolation filter.

[0232] For example, the predicted blocks in the dataset after subpixel precision displacement can be used as input to the neural network, and the lossless current block can be used as the ground truth to train the neural network.

[0233] S4623, train the neural network.

[0234] For example, when training a neural network, the loss function can be the MSE loss function, the initial learning rate can be set to 0.0001, and the learning rate can be reduced to 0.1 times every 60 epochs. Each network is trained for 180 epochs. Prediction blocks of sizes 16×16, 32×32, and 64×64 correspond to the training of the three neural networks DENSE-RVCNN1, DENSE-RVCNN2, and DENSE-RVCNN3, respectively.

[0235] In some embodiments, the decoder determines the network parameters used by Dense-RVCNN based on the size of the current block; then, based on the network parameters used by Dense-RVCNN, it determines the enhanced image.

[0236] For example, the network parameters of the Dense-RVCNN may include multiple sets of network parameters. When the decoder calls the Dense-RVCNN, it can determine the network parameters used by the Dense-RVCNN based on the size of the current block. Then, the network parameters used by the Dense-RVCNN are used to enhance the quality of the reference region to obtain an enhanced image.

[0237] For example, the Dense-RVCNN network parameters include multiple sets of network parameters. When the decoder calls Dense-RVCNN, it can directly use these multiple sets of network parameters to enhance the quality of the reference region, resulting in multiple enhanced images. Then, it can determine the network parameters used by Dense-RVCNN based on the size of the current block, and determine the enhanced image from among the multiple enhanced images based on the network parameters used by Dense-RVCNN. For example, the image using the network parameters corresponding to those used by Dense-RVCNN can be determined as the enhanced image.

[0238] In this embodiment, the decoder determines the network parameters used by Dense-RVCNN based on the size of the current block, which can reduce the complexity of the decoder when calling Dense-RVCNN and thus improve the decoding performance of the decoder.

[0239] In some embodiments, the decoder determines the parameter corresponding to the minimum side length of the current block as the network parameter used by the Dense-RVCNN.

[0240] For example, if the minimum side length of the current block is greater than or equal to a preset threshold, the decoder will determine the parameter corresponding to the minimum side length of the current block as the network parameter used by Dense-RVCNN.

[0241] Assuming that an enhancement image is introduced for all rectangular blocks larger than 16×16 in the reference region, the decoder can determine which enhancement block replaces the reference block in the following way:

[0242] For a rectangular block with a minimum side length of 16, the reference block can be replaced by the enhanced block in enhanced image 1 output by the DENSE-RVCNN1 model; for a rectangular block with a minimum side length of 32, the reference block can be replaced by the enhanced block in enhanced image 2 output by the DENSE-RVCNN2 model; and for rectangular blocks with minimum side lengths of 64 and 128, the reference block can be replaced by the enhanced block in enhanced image 3 output by the DENSE-RVCNN3 model.

[0243] In some embodiments, the Dense-RVCNN includes at least one of the following:

[0244] The input layer is used for feature extraction from the reference region;

[0245] At least one residual multi-scale convolutional RVCB structure is used to perform feature extraction and feature concatenation at multiple scales on the input feature information;

[0246] The DenseNet network structure is used for feature extraction from input feature information;

[0247] The output layer is used to convert the input features into feature information with the same number of channels as the reference region, and to weight the reference region and the converted feature information.

[0248] For example, the input layer may include convolutional layers and activation layers.

[0249] For example, the RVCB structure can include a structure that performs feature extraction on input features at multiple scales and a concatenation layer that concatenates the extracted feature information. The structure for multi-scale extraction can be implemented using convolutional layers with convolutional kernels of different sizes.

[0250] For example, the DenseNet architecture is a convolutional neural network with dense connections. In a DenseNet architecture, there are direct connections between any two layers; that is, the input to each layer is the union of the outputs of all preceding layers, and the feature maps learned by that layer are directly passed to all subsequent layers as input. Each layer in a DenseNet architecture is directly connected to its preceding layers, enabling the reuse of features. Each layer in a DenseNet architecture can be designed to be particularly narrow, learning only a very small number of feature maps (in the extreme case, each layer learns only one feature map) to reduce redundancy.

[0251] For example, the output layer may include a convolutional layer that can extract feature information with the same number of channels as the reference region.

[0252] In some embodiments, the at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input of the DenseNet structure through a portion of the plurality of RVCB structures, and the output of the DenseNet structure is connected to the output layer through another portion of the plurality of RVCB structures.

[0253] For example, the input layer can be connected to the input of the DenseNet structure through half of the multiple RVCB structures, and the output of the DenseNet structure can be connected to the output layer through the other half of the multiple RVCB structures. Of course, the number of RVCB structures used to connect the input layer and the input of the DenseNet structure may be different from the number of RVCB structures used to connect the output of the DenseNet structure and the output layer, and this application does not specifically limit this.

[0254] In some embodiments, the RVCB structure includes:

[0255] Multiple first feature extraction layers connected in parallel, each first feature extraction layer including a convolutional layer, wherein the convolutional kernels of the convolutional layers in different first feature extraction layers are different, and the number of channels of the convolutional layer in the first feature extraction layer is the ratio of the number of channels of the feature information input to the first feature extraction layer to the number of at least one first feature extraction layer;

[0256] The first splicing layer is used to splice the feature information output by the multiple first feature extraction layers.

[0257] The first convolutional layer is used to extract features from the feature information output by the first splicing layer;

[0258] A skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.

[0259] For example, the first feature extraction layer may include a convolutional layer and a subsequent connected activation layer.

[0260] For example, the first convolutional layer may also be an optional convolutional layer.

[0261] For example, the skip connection layer can directly add the feature information input to the RVCB structure and the feature information output by the first convolutional layer to obtain the feature information output by the RVCB structure.

[0262] In some embodiments, the DenseNet structure includes:

[0263] Multiple second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers;

[0264] Multiple second concatenation layers, wherein two adjacent second feature extraction layers are connected through a second concatenation layer, and any one of the multiple second concatenation layers is used to: concatenate the feature information input to the DenseNet structure and the feature information output by the second feature extraction layer preceding the concatenation layer;

[0265] The second convolutional layer is connected to the last of the plurality of second feature extraction layers through the last of the plurality of second concatenation layers. The second convolutional layer is used to convert the feature information output by the last second concatenation layer into feature information with the same number of channels as the feature information input to the DenseNet structure.

[0266] For example, the second feature extraction layer may include a convolutional layer and a subsequently connected activation layer.

[0267] For example, the plurality of second splicing layers are used to: achieve dense connections of the plurality of second feature extraction layers.

[0268] Figure 14 is a schematic structural diagram of Dense-RVCNN provided in an embodiment of this application.

[0269] As shown in Figure 14, Dense-RVCNN consists of 6 RVCB structures, 1 DenseNet structure, convolutional layers at the beginning and end, and skip connection layers.

[0270] Dense-RVCNN takes an integer-pixel reference block before motion compensation as input, passes it through a first convolutional layer with a 3×3 kernel, and LeakyReLU. This process can be represented as: F1=LReLU(W0·X)=max(0,W0·X)+α·min(W0·X).

[0271] Where W0 represents the convolutional kernel of the first layer, α represents the slope of the LeakyReLU function on the negative half-axis, and X represents the input of the neural network. F1 represents the output of the first layer, and LReLU() represents the non-linear mapping function of LeakyReLU.

[0272] The RVCB structure uses a multi-size convolutional structure.

[0273] When feature data enters the RVCB structure, it is first transformed into two sets of 32-channel feature data using a 3×3 convolution kernel, a 1×1 convolution kernel, and LeakyReLU. Then, the two sets of feature data are concatenated along the channel dimension and transformed into 64-channel feature data through a 3×3 convolution. Finally, this set of data is added to the module input via skip connections to obtain the module's output. This process can be represented as: F i,1_1 =LReLU(W i,1_1 ·F i ); F i,1_3 =LReLU(W i,1_3 ·F i ); F i,2 =W i,2 ·cat(·F i,1_3 ,F i,1_3 ); F i+1 =F i +F i,2 .

[0274] Where 1≤i≤7 and i≠4, F i W is the feature information input to the i-th RVCB structure. i,1_1 Let W be the parameters of the 1×1 convolution kernel in the first layer of the i-th RVCB structure. i,1_3 Let W be the parameters of the 3×3 convolution kernel in the first layer of the i-th RVCB structure. i,2 represents the parameters of the second convolutional kernel of the i-th RVCB structure. cat() represents the concatenation of feature data.

[0275] Figure 15 is a schematic structural diagram of the DenseNet structure in Dense-RVCNN provided in the embodiments of this application.

[0276] As shown in Figure 15, the DenseNet architecture consists of a Convolutional Block (CB) and a 1×1 convolutional layer. The CB is also called the feature extraction layer. Each CB contains a 1×1 convolutional layer, a 3×3 convolutional layer, and a LeakyReLU function. The 1×1 convolutional layer is used to compress the number of channels of the input features to 64 channels to facilitate processing by the 3×3 convolution. The calculation formula for the entire DenseNet architecture is as follows: F 4,j =CB j (cat(F 4,0 ,...,F 4,j-1 ))=LReLU(W 4,j_2 ·LReLU(W 4,j_1 ·cat(F 4,0 ,...,F 4,j-1 ))); F5=W 4,5 ·F4,4 .

[0277] Where 1≤j≤4 represents the feature information input to the DenseNet structure, F 4,0 F5 represents the feature information output by the j-th CB, and F5 represents the feature information output by the DenseNet structure. 4,j_1 W represents the parameters of the first convolutional kernel in the j-th CB. 4,j_2 W represents the parameters of the second convolutional kernel in the j-th CB. 4,5 These are the parameters for the last convolutional kernel in the DenseNet architecture. In each convolutional block (CB), the 1×1 kernel has 64 channels, the 3×3 kernel has 32 channels, and the last convolutional kernel in the DenseNet architecture has 64 channels.

[0278] Finally, a convolutional layer with a kernel size of 3×3 and 1 channel is used to compress all feature maps into a residual image. The residual image is added to the input image to obtain the final output image, as follows: F9 = W9·F8; Y = F9 + X.

[0279] Where Y is the output of the neural network, F8 is the feature information output by the last RVCB structure, W9 is the parameters of the last convolution kernel, and X represents the input of the neural network.

[0280] In some embodiments, the motion parameters include at least one of the following: the motion vector of the current block, and the index of the reference image.

[0281] For example, the decoder can determine the motion vector of the current block by decoding the bitstream, and determine the reference block of the current block in a predefined reference image based on the motion vector. The predefined reference image can be a decoded image determined according to preset rules.

[0282] For example, the decoder can determine the index by decoding the bitstream and determine the reference block of the current block in the reference image indicated by the index.

[0283] For example, the decoder can determine the motion vector and index of the current block by decoding the bitstream, and determine the reference block of the current block in the reference image indicated by the index based on the motion vector. The predefined reference image can be a decoded image determined according to preset rules.

[0284] The decoding method according to the embodiments of this application has been described in detail above from the perspective of the decoder. The encoding method according to the embodiments of this application will be described below from the perspective of the encoder, with reference to FIG16.

[0285] Figure 16 is a schematic flowchart of the encoding method 500 provided in an embodiment of this application. It should be understood that the encoding method 500 can be executed by an encoder. For example, the encoding method 500 can be executed by the video encoder 112 shown in Figure 1 or the video encoder 200 shown in Figure 2. For ease of description, an encoder will be used as an example below.

[0286] As shown in Figure 16, the decoding method 500 may include:

[0287] S510, the encoder performs motion estimation on the current block to obtain a reference block for the current block;

[0288] S520, the encoder enhances the quality of the reference block to obtain an enhanced block;

[0289] S530, the encoder determines the prediction block of the current block based on the enhancement block.

[0290] In some embodiments, S520 may include:

[0291] The encoder performs quality enhancement on a reference region in the reference image that includes the reference block to obtain an enhanced image; then, based on the position of the reference block in the reference region, the enhanced block is determined in the enhanced image.

[0292] In some embodiments, the encoder identifies the block in the enhanced image whose position is the same as that of the reference block in the reference region as the enhanced block.

[0293] In some embodiments, S530 may include:

[0294] The encoder performs boundary expansion and interpolation filtering on the enhancement block to obtain the prediction block.

[0295] In some embodiments, S510 may include:

[0296] The encoder divides the current block into at least one sub-block; then, it performs motion estimation on the current block to determine the motion parameters of the current block; based on the motion parameters of the current block, it determines the motion parameters of the at least one sub-block; based on the motion parameters of the at least one sub-block, it determines at least one reference sub-block included in the reference block; wherein, S520 may include:

[0297] The encoder determines a first region where the at least one reference sub-block is located based on the position of the at least one reference sub-block in the reference region; then, it performs boundary expansion on the first region to obtain a second region; and the block in the enhanced image whose position corresponds to the second region is determined as the enhanced block.

[0298] In some embodiments, S530 may include:

[0299] The encoder identifies the sub-block in the enhancement block whose position is the same as that of the at least one reference sub-block in the reference region as at least one enhancement sub-block included in the enhancement block; then, it performs boundary expansion and interpolation filtering on the at least one enhancement sub-block to obtain at least one prediction sub-block included in the prediction block.

[0300] In some embodiments, the first region is the smallest region including the at least one reference sub-block.

[0301] In some embodiments, the reference region is the entire region or a portion of the reference image.

[0302] In some embodiments, the partial region is the region containing any of the following: image block, sub-image, rectangular region, or slice.

[0303] In some embodiments, the encoder extracts features from the reference region to obtain a residual image; then, the reference region and the residual image are weighted to obtain the enhanced image.

[0304] In some embodiments, the encoder determines a first parameter for feature extraction of the reference region based on the size of the current block; then, based on the first parameter, feature extraction is performed on the reference region to obtain the residual image.

[0305] In some embodiments, the encoder determines the parameter corresponding to the minimum side length of the current block as the first parameter.

[0306] In some embodiments, the encoder extracts features from the reference region to obtain first feature information; then, it performs multi-scale feature extraction on the first feature information and concatenates the extracted multi-scale feature information to obtain second feature information; based on the second feature information and the feature information obtained by feature extraction from the second feature information, it determines third feature information; it performs multi-scale feature extraction on the third feature information and concatenates the extracted multi-scale feature information to obtain fourth feature information; and it converts the fourth feature information into feature information with the same number of channels as the reference region to obtain the residual image.

[0307] In some embodiments, the encoder uses a dense residual multi-scale convolutional network (Dense-RVCNN) to enhance the quality of the reference region to obtain the enhanced image.

[0308] In some embodiments, the encoder determines the network parameters used by Dense-RVCNN based on the size of the current block; then, based on the network parameters used by Dense-RVCNN, it determines the enhanced image.

[0309] In some embodiments, the encoder determines the parameter corresponding to the minimum side length of the current block as the network parameter used by the Dense-RVCNN.

[0310] In some embodiments, the Dense-RVCNN includes at least one of the following:

[0311] The input layer is used for feature extraction from the reference region;

[0312] At least one residual multi-scale convolutional RVCB structure is used to perform feature extraction and feature concatenation at multiple scales on the input feature information;

[0313] The DenseNet network structure is used for feature extraction from input feature information;

[0314] The output layer is used to convert the input features into feature information with the same number of channels as the reference region, and to weight the reference region and the converted feature information.

[0315] In some embodiments, the at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input of the DenseNet structure through a portion of the plurality of RVCB structures, and the output of the DenseNet structure is connected to the output layer through another portion of the plurality of RVCB structures.

[0316] In some embodiments, the RVCB structure includes:

[0317] Multiple first feature extraction layers connected in parallel, each first feature extraction layer including a convolutional layer, wherein the convolutional kernels of the convolutional layers in different first feature extraction layers are different, and the number of channels of the convolutional layer in the first feature extraction layer is the ratio of the number of channels of the feature information input to the first feature extraction layer to the number of at least one first feature extraction layer;

[0318] The first splicing layer is used to splice the feature information output by the multiple first feature extraction layers.

[0319] The first convolutional layer is used to extract features from the feature information output by the first splicing layer;

[0320] A skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.

[0321] In some embodiments, the DenseNet structure includes:

[0322] Multiple second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers;

[0323] Multiple second concatenation layers, wherein two adjacent second feature extraction layers are connected through a second concatenation layer, and any one of the multiple second concatenation layers is used to: concatenate the feature information input to the DenseNet structure and the feature information output by the second feature extraction layer preceding the concatenation layer;

[0324] The second convolutional layer is connected to the last of the plurality of second feature extraction layers through the last of the plurality of second concatenation layers. The second convolutional layer is used to convert the feature information output by the last second concatenation layer into feature information with the same number of channels as the feature information input to the DenseNet structure.

[0325] In some embodiments, the method 500 may further include:

[0326] Encode the motion parameters of the current block;

[0327] The motion parameters of the current block include at least one of the following: the motion vector of the current block, and the index of the reference image.

[0328] It should be understood that the encoding method can be understood as the reverse process of the decoding method. Therefore, the specific scheme of the encoding method 500 can be found in the relevant content of the decoding method 400. For ease of description, this application will not elaborate further.

[0329] The above text describes in detail the encoding and decoding method according to the embodiments of this application from the perspective of decoder and encoder. The following text, with reference to Figure 17, describes the neural network training method provided according to the embodiments of this application from the perspective of neural network.

[0330] Figure 17 is a schematic flowchart of the neural network training method 600 provided in an embodiment of this application.

[0331] It should be understood that this training method 600 can be executed by any electronic device with data processing capabilities. The following explanation uses a training device as an example.

[0332] Furthermore, the training method 600 provided in this application can be incorporated into the decoding method 400 or the encoding method 500. For example, the training device uses the training method 600 to train a neural network. After the neural network is trained, the decoder can use the neural network to enhance the quality of a reference region in a reference image, obtaining an enhanced image; then, the enhancement block corresponding to the reference block is determined in the enhanced image. As another example, the training device uses the training method 600 to train a neural network. After the neural network is trained, the encoder can use the neural network to enhance the quality of a reference region in a reference image, obtaining an enhanced image; then, the enhancement block corresponding to the reference block is determined in the enhanced image.

[0333] Furthermore, this application does not specifically limit the dataset used when training the neural network.

[0334] As shown in Figure 17, the neural network training method 600 may include:

[0335] S610, the training device encodes the sample video to obtain the bitstream.

[0336] For example, 650 videos can be randomly selected from the BVI-DVC dataset (a dataset containing a large number of scene YUV video files). Using the VTM-9.3 encoder, with the encoding mode set to LDP mode and the quantization parameter (Qp) set to 22, the first 32 images of these videos are encoded to generate a bitstream. Of course, random access (RA) and low latency (LB) configuration modes can also be used, and this application does not specifically limit this.

[0337] S620, the training device decodes the bitstream to determine the prediction block of the current block;

[0338] For example, for the current block, the bitstream contains motion vector information with integer precision and motion vector information with fractional precision. The training device can deduce the integer pixel position of the reference block from the integer pixel position of the current block. Then, the prediction block is obtained by sub-pixel displacement of the integer pixel reference block through a sub-pixel interpolation filter.

[0339] For example, the predicted blocks in the dataset after subpixel precision displacement can be used as input to the neural network, and the lossless current block can be used as the ground truth to train the neural network.

[0340] S630, the training device trains the neural network based on the label of the current block and the prediction block.

[0341] For example, when training a neural network, the loss function can be the MSE loss function, the initial learning rate can be set to 0.0001, and the learning rate can be reduced to 0.1 times every 60 epochs. Each network is trained for 180 epochs. Prediction blocks of sizes 16×16, 32×32, and 64×64 correspond to the training of the three neural networks DENSE-RVCNN1, DENSE-RVCNN2, and DENSE-RVCNN3, respectively.

[0342] Figure 18 is an example of the input and label when training a neural network according to an embodiment of this application.

[0343] As shown in Figure 18, for the current block, the bitstream contains motion vector information with integer precision and motion vector information with fractional precision. The integer precision motion vector information includes Imvx and Imvy; the fractional precision motion vector information includes Fmvx and Fmvy. The training device can combine the integer pixel position of the current block with the integer precision motion vector information to derive the integer pixel position of the reference block from the reference image. That is, the integer pixel position of the reference block is (X+Imvx, Y+Imvy). Then, the sub-pixel displacement of the integer pixel reference block is performed using a sub-pixel interpolation filter to obtain the position of the predicted block. That is, the position of the predicted block is (X+Imvx+Fmvx, Y+Imvy+Fmvy).

[0344] In some embodiments, S620 may include;

[0345] The training device determines prediction blocks of multiple sizes; wherein, S630 may include:

[0346] The training device trains the neural network based on the label of the current block and the multiple sizes of the prediction blocks to obtain the network parameters corresponding to the multiple sizes.

[0347] Since the reference block sizes vary from 4×4 to 128×128 and are rectangular, this application divides the reference blocks into large, medium, and small categories based on their size. Small blocks are rectangular blocks with a minimum side length of 16, including 16×16, 16×32, 32×16, 16×64, and 64×16 rectangular blocks. Medium blocks are rectangular blocks with a minimum side length of 32, including 32×32, 32×64, and 64×32 rectangular blocks. Large blocks are rectangular blocks with a minimum side length of 64, including 64×64, 64×128, 128×64, and 128×128 rectangular blocks. Three different datasets are created for large, medium, and small rectangles to train three sets of network parameters. Due to the large number of segmented rectangular blocks in the video, only square prediction blocks of sizes 16×16, 32×32, and 64×64 and the original current block are selected to create the three datasets. When using a neural network model trained on these three datasets, quality enhancement can be performed on the same reference region in the same reference image to generate three different enhanced images. Large, medium, and small blocks can be identified as enhancement blocks in different enhanced images.

[0348] In some embodiments, the plurality of prediction blocks of various sizes include a plurality of prediction blocks with different minimum side lengths.

[0349] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the embodiments mentioned above. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the specific embodiments mentioned above can be combined in any suitable way without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application. It should also be understood that in the various method embodiments of this application, the order of the processes mentioned above does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, in the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. Specifically, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, the character " / " in this application generally indicates that the objects before and after it are in an "or" relationship.

[0350] The method embodiments of this application have been described in detail above. The apparatus embodiments of this application will be described in detail below with reference to Figures 19 to 22.

[0351] Figure 19 is a schematic block diagram of the decoder 710 according to an embodiment of this application.

[0352] As shown in Figure 19, the decoder 710 may include:

[0353] The first determining unit 711 is used to decode the bitstream and determine the motion parameters of the current block;

[0354] The second determining unit 712 is used to determine a reference block of the current block in the reference image of the current block based on the motion parameters of the current block;

[0355] Enhancement unit 713 is used to enhance the quality of the reference block to obtain an enhanced block;

[0356] The third determining unit 714 is used to determine the prediction block of the current block based on the enhancement block.

[0357] In some embodiments, the enhancement unit 713 is specifically used for:

[0358] The reference region including the reference block in the reference image is enhanced to obtain an enhanced image;

[0359] The enhancement block is determined in the enhancement image based on its position in the reference region.

[0360] In some embodiments, the enhancement unit 713 is specifically used for:

[0361] The block in the enhanced image whose position is the same as that of the reference block in the reference region is identified as the enhanced block.

[0362] In some embodiments, the third determining unit 714 is specifically used for:

[0363] The enhancement block is subjected to boundary expansion and interpolation filtering to obtain the prediction block.

[0364] In some embodiments, the second determining unit 712 is specifically used for:

[0365] Divide the current block into at least one sub-block;

[0366] Based on the motion parameters of the current block, determine the motion parameters of the at least one sub-block;

[0367] Based on the motion parameters of the at least one sub-block, determine at least one reference sub-block included in the reference block;

[0368] Specifically, the enhancement unit 713 is used for:

[0369] Based on the position of the at least one reference sub-block in the reference region, the first region where the at least one reference sub-block is located is determined;

[0370] The first region is expanded to obtain the second region;

[0371] The block in the enhanced image whose position corresponds to the second region is determined as the enhanced block.

[0372] In some embodiments, the third determining unit 714 is specifically used for:

[0373] The sub-block whose position is the same as that of the at least one reference sub-block in the reference region is determined as at least one enhanced sub-block included in the enhanced block;

[0374] Boundary expansion and interpolation filtering are performed on the at least one enhanced sub-block to obtain at least one prediction sub-block included in the prediction block.

[0375] In some embodiments, the first region is the smallest region including the at least one reference sub-block.

[0376] In some embodiments, the reference region is the entire region or a portion of the reference image.

[0377] In some embodiments, the partial region is the region containing any of the following: image block, sub-image, rectangular region, or slice.

[0378] In some embodiments, the enhancement unit 713 is specifically used for:

[0379] Feature extraction is performed on the reference region to obtain a residual image;

[0380] The enhanced image is obtained by weighting the reference region and the residual image.

[0381] In some embodiments, the enhancement unit 713 is specifically used for:

[0382] Based on the size of the current block, a first parameter is determined for feature extraction of the reference region;

[0383] Based on the first parameter, feature extraction is performed on the reference region to obtain the residual image.

[0384] In some embodiments, the enhancement unit 713 is specifically used for:

[0385] The parameter corresponding to the minimum side length of the current block is determined as the first parameter.

[0386] In some embodiments, the enhancement unit 713 is specifically used for:

[0387] Feature extraction is performed on the reference region to obtain first feature information;

[0388] The first feature information is subjected to multi-scale feature extraction, and the extracted multi-scale feature information is concatenated to obtain the second feature information;

[0389] Based on the second feature information and the feature information obtained by feature extraction from the second feature information, the third feature information is determined;

[0390] Multi-scale feature extraction is performed on the third feature information, and the extracted multi-scale feature information is concatenated to obtain the fourth feature information;

[0391] The residual image is obtained by converting the fourth feature information into feature information with the same number of channels as the reference region.

[0392] In some embodiments, the enhancement unit 713 is specifically used for:

[0393] The reference region is enhanced using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image.

[0394] In some embodiments, the enhancement unit 713 is specifically used for:

[0395] Based on the size of the current block, determine the network parameters used by Dense-RVCNN;

[0396] The enhanced image is determined based on the network parameters used by the Dense-RVCNN.

[0397] In some embodiments, the enhancement unit 713 is specifically used for:

[0398] The parameter corresponding to the minimum side length of the current block is determined as the network parameter used by the Dense-RVCNN.

[0399] In some embodiments, the Dense-RVCNN includes at least one of the following:

[0400] The input layer is used for feature extraction from the reference region;

[0401] At least one residual multi-scale convolutional RVCB structure is used to perform feature extraction and feature concatenation at multiple scales on the input feature information;

[0402] The DenseNet network structure is used for feature extraction from input feature information;

[0403] The output layer is used to convert the input features into feature information with the same number of channels as the reference region, and to weight the reference region and the converted feature information.

[0404] In some embodiments, the at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input of the DenseNet structure through a portion of the plurality of RVCB structures, and the output of the DenseNet structure is connected to the output layer through another portion of the plurality of RVCB structures.

[0405] In some embodiments, the RVCB structure includes:

[0406] Multiple first feature extraction layers connected in parallel, each first feature extraction layer including a convolutional layer, wherein the convolutional kernels of the convolutional layers in different first feature extraction layers are different, and the number of channels of the convolutional layer in the first feature extraction layer is the ratio of the number of channels of the feature information input to the first feature extraction layer to the number of at least one first feature extraction layer;

[0407] The first splicing layer is used to splice the feature information output by the multiple first feature extraction layers.

[0408] The first convolutional layer is used to extract features from the feature information output by the first splicing layer;

[0409] A skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.

[0410] In some embodiments, the DenseNet structure includes:

[0411] Multiple second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers;

[0412] Multiple second concatenation layers, wherein two adjacent second feature extraction layers are connected through a second concatenation layer, and any one of the multiple second concatenation layers is used to: concatenate the feature information input to the DenseNet structure and the feature information output by the second feature extraction layer preceding the concatenation layer;

[0413] The second convolutional layer is connected to the last of the plurality of second feature extraction layers through the last of the plurality of second concatenation layers. The second convolutional layer is used to convert the feature information output by the last second concatenation layer into feature information with the same number of channels as the feature information input to the DenseNet structure.

[0414] In some embodiments, the motion parameters of the current block include at least one of the following: the motion vector of the current block, and the index of the reference image.

[0415] Figure 20 is a schematic block diagram of the encoder 720 according to an embodiment of this application.

[0416] As shown in Figure 20, the encoder 720 may include:

[0417] The first determining unit 721 is used to perform motion estimation on the current block to obtain a reference block for the current block;

[0418] Enhancement unit 722 is used to enhance the quality of the reference block to obtain an enhanced block;

[0419] The second determining unit 723 is used to determine the prediction block of the current block based on the enhancement block.

[0420] In some embodiments, the enhancement unit 722 is specifically used for:

[0421] The reference region including the reference block in the reference image is enhanced to obtain an enhanced image;

[0422] The enhancement block is determined in the enhancement image based on its position in the reference region.

[0423] In some embodiments, the enhancement unit 722 is specifically used for:

[0424] The block in the enhanced image whose position is the same as that of the reference block in the reference region is identified as the enhanced block.

[0425] In some embodiments, the second determining unit 723 is specifically used for:

[0426] The enhancement block is subjected to boundary expansion and interpolation filtering to obtain the prediction block.

[0427] In some embodiments, the first determining unit 721 is specifically used for:

[0428] Divide the current block into at least one sub-block;

[0429] Motion estimation is performed on the current block to determine the motion parameters of the current block;

[0430] Based on the motion parameters of the at least one sub-block, determine at least one reference sub-block included in the reference block;

[0431] Specifically, the enhancement unit 722 is used for:

[0432] Based on the position of the at least one reference sub-block in the reference region, the first region where the at least one reference sub-block is located is determined;

[0433] The first region is expanded to obtain the second region;

[0434] The block in the enhanced image whose position corresponds to the second region is determined as the enhanced block.

[0435] In some embodiments, the second determining unit 723 is specifically used for:

[0436] The sub-block whose position is the same as that of the at least one reference sub-block in the reference region is determined as at least one enhanced sub-block included in the enhanced block;

[0437] Boundary expansion and interpolation filtering are performed on the at least one enhanced sub-block to obtain at least one prediction sub-block included in the prediction block.

[0438] In some embodiments, the first region is the smallest region including the at least one reference sub-block.

[0439] In some embodiments, the reference region is the entire region or a portion of the reference image.

[0440] In some embodiments, the partial region is the region containing any of the following: image block, sub-image, rectangular region, or slice.

[0441] In some embodiments, the enhancement unit 722 is specifically used for:

[0442] Feature extraction is performed on the reference region to obtain a residual image;

[0443] The enhanced image is obtained by weighting the reference region and the residual image.

[0444] In some embodiments, the enhancement unit 722 is specifically used for:

[0445] Based on the size of the current block, a first parameter is determined for feature extraction of the reference region;

[0446] Based on the first parameter, feature extraction is performed on the reference region to obtain the residual image.

[0447] In some embodiments, the enhancement unit 722 is specifically used for:

[0448] The parameter corresponding to the minimum side length of the current block is determined as the first parameter.

[0449] In some embodiments, the enhancement unit 722 is specifically used for:

[0450] Feature extraction is performed on the reference region to obtain first feature information;

[0451] The first feature information is subjected to multi-scale feature extraction, and the extracted multi-scale feature information is concatenated to obtain the second feature information;

[0452] Based on the second feature information and the feature information obtained by feature extraction from the second feature information, the third feature information is determined;

[0453] Multi-scale feature extraction is performed on the third feature information, and the extracted multi-scale feature information is concatenated to obtain the fourth feature information;

[0454] The residual image is obtained by converting the fourth feature information into feature information with the same number of channels as the reference region.

[0455] In some embodiments, the enhancement unit 722 is specifically used for:

[0456] The reference region is enhanced using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image.

[0457] In some embodiments, the enhancement unit 722 is specifically used for:

[0458] Based on the size of the current block, determine the network parameters used by Dense-RVCNN;

[0459] The enhanced image is determined based on the network parameters used by the Dense-RVCNN.

[0460] In some embodiments, the enhancement unit 722 is specifically used for:

[0461] The parameter corresponding to the minimum side length of the current block is determined as the network parameter used by the Dense-RVCNN.

[0462] In some embodiments, the Dense-RVCNN includes at least one of the following:

[0463] The input layer is used for feature extraction from the reference region;

[0464] At least one residual multi-scale convolutional RVCB structure is used to perform feature extraction and feature concatenation at multiple scales on the input feature information;

[0465] The DenseNet network structure is used for feature extraction from input feature information;

[0466] The output layer is used to convert the input features into feature information with the same number of channels as the reference region, and to weight the reference region and the converted feature information.

[0467] In some embodiments, the at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input of the DenseNet structure through a portion of the plurality of RVCB structures, and the output of the DenseNet structure is connected to the output layer through another portion of the plurality of RVCB structures.

[0468] In some embodiments, the RVCB structure includes:

[0469] Multiple first feature extraction layers connected in parallel, each first feature extraction layer including a convolutional layer, wherein the convolutional kernels of the convolutional layers in different first feature extraction layers are different, and the number of channels of the convolutional layer in the first feature extraction layer is the ratio of the number of channels of the feature information input to the first feature extraction layer to the number of at least one first feature extraction layer;

[0470] The first splicing layer is used to splice the feature information output by the multiple first feature extraction layers.

[0471] The first convolutional layer is used to extract features from the feature information output by the first splicing layer;

[0472] A skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.

[0473] In some embodiments, the DenseNet structure includes:

[0474] Multiple second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers;

[0475] Multiple second concatenation layers, wherein two adjacent second feature extraction layers are connected through a second concatenation layer, and any one of the multiple second concatenation layers is used to: concatenate the feature information input to the DenseNet structure and the feature information output by the second feature extraction layer preceding the concatenation layer;

[0476] The second convolutional layer is connected to the last of the plurality of second feature extraction layers through the last of the plurality of second concatenation layers. The second convolutional layer is used to convert the feature information output by the last second concatenation layer into feature information with the same number of channels as the feature information input to the DenseNet structure.

[0477] In some embodiments, the second determining unit 723 is further configured to:

[0478] Encode the motion parameters of the current block;

[0479] The motion parameters of the current block include at least one of the following: the motion vector of the current block, and the index of the reference image.

[0480] Figure 21 is a schematic block diagram of a neural network training device 730 according to an embodiment of this application.

[0481] As shown in Figure 21, the neural network training device 730 may include:

[0482] Encoding unit 731 is used to encode the sample video to obtain a bitstream;

[0483] Decoding unit 732 is used to decode the bitstream and determine the prediction block of the current block;

[0484] Training unit 733 is used to train the neural network based on the label of the current block and the prediction block.

[0485] In some embodiments, the decoding unit 732 is specifically used for;

[0486] Determine prediction blocks of multiple sizes;

[0487] The step of training the neural network based on the label of the current block and the predicted block includes:

[0488] Based on the label of the current block and the multiple sizes of the predicted blocks, the neural network is trained to obtain the network parameters corresponding to the multiple sizes.

[0489] In some embodiments, the plurality of prediction blocks of various sizes include a plurality of prediction blocks with different minimum side lengths.

[0490] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details are omitted here. Specifically, the decoder 710 shown in FIG19 can correspond to the corresponding subject in executing the method 400 of the embodiments of this application, and the aforementioned and other operations and / or functions of each unit in the decoder 710 are respectively for implementing the corresponding processes in the various methods such as method 400. Similarly, the encoder 720 shown in FIG20 can correspond to the corresponding subject in executing the method 500 of the embodiments of this application, that is, the aforementioned and other operations and / or functions of each unit in the encoder 720 are respectively for implementing the corresponding processes in the various methods such as method 500. The neural network training device 730 shown in FIG21 can correspond to the corresponding subject in executing the method 600 of the embodiments of this application, that is, the aforementioned and other operations and / or functions of each unit in the neural network training device 730 are respectively for implementing the corresponding processes in the various methods such as method 600.

[0491] It should also be understood that the various units in the decoder 710, encoder 720, or neural network training device 730 involved in the embodiments of this application can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This can achieve the same operation without affecting the technical effect of the embodiments of this application. The units mentioned above are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the decoder 710, encoder 720, or neural network training device 730 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units. According to another embodiment of this application, the decoder 710, encoder 720, or neural network training device 730 involved in the embodiments of this application can be constructed and the encoding or decoding method of the embodiments of this application can be implemented by running a computer program (including program code) capable of performing the steps involved in the corresponding method on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), random access storage medium (RAM), and read-only storage medium (ROM). The computer program can be recorded on, for example, a computer-readable storage medium, loaded into an electronic device through the computer-readable storage medium, and run therein to implement the corresponding method of the embodiments of this application.

[0492] In other words, the units mentioned above can be implemented in hardware, in software instructions, or in a combination of hardware and software. Specifically, the steps of the method embodiments in this application can be completed by the integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software in the decoding processor. Optionally, the software can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps in the method embodiments mentioned above.

[0493] Figure 22 is a schematic structural diagram of the electronic device 800 provided in an embodiment of this application.

[0494] As shown in Figure 22, the electronic device 800 includes at least a processor 810 and a computer-readable storage medium 820. The processor 810 and the computer-readable storage medium 820 can be connected via a bus or other means. The computer-readable storage medium 820 stores a computer program 821, which includes computer instructions. The processor 810 executes the computer instructions stored in the computer-readable storage medium 820. The processor 810 is the computing and control core of the electronic device 800, and is suitable for implementing one or more computer instructions, specifically for loading and executing one or more computer instructions to achieve corresponding method flows or corresponding functions.

[0495] Processor 810 can also be referred to as a Central Processing Unit (CPU). Processor 810 may include, but is not limited to: general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, discrete hardware components, etc.

[0496] Computer-readable storage media 820 includes, but is not limited to, volatile memory and / or non-volatile memory.

[0497] The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. Exemplarily, RAM includes, but is not limited to: static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM).

[0498] For example, the electronic device 800 may be an encoder or encoding framework involved in the embodiments of this application; the computer-readable storage medium 820 stores first computer instructions; the processor 810 loads and executes the first computer instructions stored in the computer-readable storage medium 820 to implement the corresponding steps in the encoding method provided in the embodiments of this application; in other words, the first computer instructions in the computer-readable storage medium 820 are loaded and executed by the processor 810 to implement the corresponding steps, which will not be described again here to avoid repetition.

[0499] For example, the electronic device 800 may be a decoder or decoding framework involved in the embodiments of this application; the computer-readable storage medium 820 stores second computer instructions; the processor 810 loads and executes the second computer instructions stored in the computer-readable storage medium 820 to implement the corresponding steps in the decoding method provided in the embodiments of this application; in other words, the second computer instructions in the computer-readable storage medium 820 are loaded and executed by the processor 810 to implement the corresponding steps, which will not be described again here to avoid repetition.

[0500] For example, the electronic device 800 may be a training device involved in the embodiments of this application; the computer-readable storage medium 820 stores third computer instructions; the processor 810 loads and executes the third computer instructions stored in the computer-readable storage medium 820 to implement the corresponding steps in the training method provided in the embodiments of this application; in other words, the third computer instructions in the computer-readable storage medium 820 are loaded and executed by the processor 810 to implement the corresponding steps, which will not be described again here to avoid repetition.

[0501] According to another aspect of this application, this application also provides an encoding and decoding system, including the encoder and decoder mentioned above.

[0502] According to another aspect of this application, this application also provides a computer-readable storage medium (Memory), which is a memory device in an electronic device 800 for storing programs and data. For example, a computer-readable storage medium 820. It is understood that the computer-readable storage medium 820 here may include both the built-in storage medium in the electronic device 800 and extended storage media supported by the electronic device 800. The computer-readable storage medium provides storage space that stores the operating system of the electronic device 800. Furthermore, the storage space also stores one or more computer instructions (e.g., at least one of the aforementioned first, second, and third computer instructions) suitable for loading and execution by a processor 810. These computer instructions may be one or more computer programs 821 (including program code).

[0503] According to another aspect of this application, this application also provides a computer program product or computer program that includes computer instructions (e.g., any one of the aforementioned first, second, and third computer instructions) stored in a computer-readable storage medium. For example, computer program 821. In this case, the data processing device 800 may be a computer, and the processor 810 reads the computer instructions from the computer-readable storage medium 820, executes the computer instructions, causing the computer to perform the methods provided in the various alternative methods described above.

[0504] In other words, when implemented using software, the methods provided in the various optional approaches mentioned above can be implemented, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes of the embodiments of this application are run or the functions of the embodiments of this application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0505] Those skilled in the art will recognize that the units and process steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0506] Finally, it should be noted that the above content is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A decoding method, characterized in that, include: Decode the bitstream to determine the motion parameters of the current block; Based on the motion parameters of the current block, a reference block is determined in the reference image of the current block; The reference block is mass-enhanced to obtain an enhanced block; Based on the enhancement block, the prediction block of the current block is determined.

2. The method according to claim 1, characterized in that, The process of enhancing the quality of the reference block to obtain an enhanced block includes: The reference region including the reference block in the reference image is enhanced to obtain an enhanced image; The enhancement block is determined in the enhancement image based on its position in the reference region.

3. The method according to claim 2, characterized in that, Determining the enhancement block in the enhancement image based on the position of the reference block in the reference region includes: The block in the enhanced image whose position is the same as that of the reference block in the reference region is identified as the enhanced block.

4. The method according to claim 3, characterized in that, The step of determining the prediction block for the current block based on the enhancement block includes: The enhancement block is subjected to boundary expansion and interpolation filtering to obtain the prediction block.

5. The method according to claim 2, characterized in that, Determining a reference block for the current block in a reference image based on the motion parameters of the current block includes: Divide the current block into at least one sub-block; Based on the motion parameters of the current block, determine the motion parameters of the at least one sub-block; Based on the motion parameters of the at least one sub-block, determine at least one reference sub-block included in the reference block; The step of determining the enhancement block in the enhancement image based on the position of the reference block in the reference region includes: Based on the position of the at least one reference sub-block in the reference region, the first region where the at least one reference sub-block is located is determined; The first region is expanded to obtain the second region; The block in the enhanced image whose position corresponds to the second region is determined as the enhanced block.

6. The method according to claim 5, characterized in that, The step of determining the prediction block for the current block based on the enhancement block includes: The sub-block whose position is the same as that of the at least one reference sub-block in the reference region is determined as at least one enhanced sub-block included in the enhanced block; Boundary expansion and interpolation filtering are performed on the at least one enhanced sub-block to obtain at least one prediction sub-block included in the prediction block.

7. The method according to claim 5, characterized in that, The first region is the smallest region that includes the at least one reference sub-block.

8. The method according to any one of claims 2 to 7, characterized in that, The reference region is the entire region or a portion of the reference image.

9. The method according to claim 8, characterized in that, The specified region is any of the following: image block, sub-image, rectangular region, or slice.

10. The method according to any one of claims 2 to 9, characterized in that, The step of enhancing the quality of a reference region in the reference image that includes the reference block to obtain an enhanced image includes: Feature extraction is performed on the reference region to obtain a residual image; The enhanced image is obtained by weighting the reference region and the residual image.

11. The method according to claim 10, characterized in that, The step of extracting features from the reference region to obtain a residual image includes: Based on the size of the current block, a first parameter is determined for feature extraction of the reference region; Based on the first parameter, feature extraction is performed on the reference region to obtain the residual image.

12. The method according to claim 11, characterized in that, The step of determining the first parameter for feature extraction of the reference region based on the size of the current block includes: The parameter corresponding to the minimum side length of the current block is determined as the first parameter.

13. The method according to any one of claims 10 to 12, characterized in that, The step of extracting features from the reference region to obtain a residual image includes: Feature extraction is performed on the reference region to obtain first feature information; The first feature information is subjected to multi-scale feature extraction, and the extracted multi-scale feature information is concatenated to obtain the second feature information; Based on the second feature information and the feature information obtained by feature extraction from the second feature information, the third feature information is determined; Multi-scale feature extraction is performed on the third feature information, and the extracted multi-scale feature information is concatenated to obtain the fourth feature information; The residual image is obtained by converting the fourth feature information into feature information with the same number of channels as the reference region.

14. The method according to any one of claims 2 to 9, characterized in that, The step of enhancing the quality of a reference region in the reference image that includes the reference block to obtain an enhanced image includes: The reference region is enhanced using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image.

15. The method according to claim 14, characterized in that, The process of enhancing the quality of the reference region using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image includes: Based on the size of the current block, determine the network parameters used by Dense-RVCNN; The enhanced image is determined based on the network parameters used by the Dense-RVCNN.

16. The method according to claim 15, characterized in that, The process of determining the network parameters used by Dense-RVCNN based on the size of the current block includes: The parameter corresponding to the minimum side length of the current block is determined as the network parameter used by the Dense-RVCNN.

17. The method according to any one of claims 14 to 16, characterized in that, The Dense-RVCNN includes at least one of the following: The input layer is used for feature extraction from the reference region; At least one residual multi-scale convolutional RVCB structure is used to perform feature extraction and feature concatenation at multiple scales on the input feature information; The DenseNet network structure is used for feature extraction from input feature information; The output layer is used to convert the input features into feature information with the same number of channels as the reference region, and to weight the reference region and the converted feature information.

18. The method according to claim 17, characterized in that, The at least one RVCB structure is a plurality of RVCB structures. The input layer is connected to the input end of the DenseNet structure through a portion of the plurality of RVCB structures. The output end of the DenseNet structure is connected to the output layer through another portion of the plurality of RVCB structures.

19. The method according to claim 17, characterized in that, The RVCB structure includes: Multiple first feature extraction layers connected in parallel, each first feature extraction layer including a convolutional layer, wherein the convolutional kernels of the convolutional layers in different first feature extraction layers are different, and the number of channels of the convolutional layer in the first feature extraction layer is the ratio of the number of channels of the feature information input to the first feature extraction layer to the number of at least one first feature extraction layer; The first splicing layer is used to splice the feature information output by the multiple first feature extraction layers. The first convolutional layer is used to extract features from the feature information output by the first splicing layer; A skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.

20. The method according to claim 17, characterized in that, The DenseNet structure includes: Multiple second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers; Multiple second concatenation layers, wherein two adjacent second feature extraction layers are connected through a second concatenation layer, and any one of the multiple second concatenation layers is used to: concatenate the feature information input to the DenseNet structure and the feature information output by the second feature extraction layer preceding the concatenation layer; The second convolutional layer is connected to the last of the plurality of second feature extraction layers through the last of the plurality of second concatenation layers. The second convolutional layer is used to convert the feature information output by the last second concatenation layer into feature information with the same number of channels as the feature information input to the DenseNet structure.

21. The method according to any one of claims 1 to 20, characterized in that, The motion parameters of the current block include at least one of the following: the motion vector of the current block, and the index of the reference image.

22. An encoding method, characterized in that, include: Motion estimation is performed on the current block to obtain a reference block for the current block; The reference block is mass-enhanced to obtain an enhanced block; Based on the enhancement block, the prediction block of the current block is determined.

23. The method according to claim 22, characterized in that, The process of enhancing the quality of the reference block to obtain an enhanced block includes: The reference region including the reference block in the reference image is enhanced to obtain an enhanced image; The enhancement block is determined in the enhancement image based on its position in the reference region.

24. The method according to claim 23, characterized in that, Determining the enhancement block in the enhancement image based on the position of the reference block in the reference region includes: The block in the enhanced image whose position is the same as that of the reference block in the reference region is identified as the enhanced block.

25. The method according to claim 24, characterized in that, The step of determining the prediction block for the current block based on the enhancement block includes: The enhancement block is subjected to boundary expansion and interpolation filtering to obtain the prediction block.

26. The method according to claim 23, characterized in that, The step of performing motion estimation on the current block to obtain a reference block for the current block includes: Divide the current block into at least one sub-block; Motion estimation is performed on the current block to determine the motion parameters of the current block; Based on the motion parameters of the current block, determine the motion parameters of the at least one sub-block; Based on the motion parameters of the at least one sub-block, determine at least one reference sub-block included in the reference block; The step of determining the enhancement block in the enhancement image based on the position of the reference block in the reference region includes: Based on the position of the at least one reference sub-block in the reference region, the first region where the at least one reference sub-block is located is determined; The first region is expanded to obtain the second region; The block in the enhanced image whose position corresponds to the second region is determined as the enhanced block.

27. The method according to claim 26, characterized in that, The step of determining the prediction block for the current block based on the enhancement block includes: The sub-block whose position is the same as that of the at least one reference sub-block in the reference region is determined as at least one enhanced sub-block included in the enhanced block; Boundary expansion and interpolation filtering are performed on the at least one enhanced sub-block to obtain at least one prediction sub-block included in the prediction block.

28. The method according to claim 26, characterized in that, The first region is the smallest region that includes the at least one reference sub-block.

29. The method according to any one of claims 23 to 28, characterized in that, The reference region is the entire region or a portion of the reference image.

30. The method according to claim 29, characterized in that, The specified region is any of the following: image block, sub-image, rectangular region, or slice.

31. The method according to any one of claims 23 to 30, characterized in that, The step of enhancing the quality of a reference region in the reference image that includes the reference block to obtain an enhanced image includes: Feature extraction is performed on the reference region to obtain a residual image; The enhanced image is obtained by weighting the reference region and the residual image.

32. The method according to claim 31, characterized in that, The step of extracting features from the reference region to obtain a residual image includes: Based on the size of the current block, a first parameter is determined for feature extraction of the reference region; Based on the first parameter, feature extraction is performed on the reference region to obtain the residual image.

33. The method according to claim 32, characterized in that, The step of determining the first parameter for feature extraction of the reference region based on the size of the current block includes: The parameter corresponding to the minimum side length of the current block is determined as the first parameter.

34. The method according to any one of claims 31 to 33, characterized in that, The step of extracting features from the reference region to obtain a residual image includes: Feature extraction is performed on the reference region to obtain first feature information; The first feature information is subjected to multi-scale feature extraction, and the extracted multi-scale feature information is concatenated to obtain the second feature information; Based on the second feature information and the feature information obtained by feature extraction from the second feature information, the third feature information is determined; Multi-scale feature extraction is performed on the third feature information, and the extracted multi-scale feature information is concatenated to obtain the fourth feature information; The residual image is obtained by converting the fourth feature information into feature information with the same number of channels as the reference region.

35. The method according to any one of claims 23 to 30, characterized in that, The step of enhancing the quality of a reference region in the reference image that includes the reference block to obtain an enhanced image includes: The reference region is enhanced using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image.

36. The method according to claim 35, characterized in that, The process of enhancing the quality of the reference region using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image includes: Based on the size of the current block, determine the network parameters used by Dense-RVCNN; The enhanced image is determined based on the network parameters used by the Dense-RVCNN.

37. The method according to claim 36, characterized in that, The process of determining the network parameters used by Dense-RVCNN based on the size of the current block includes: The parameter corresponding to the minimum side length of the current block is determined as the network parameter used by the Dense-RVCNN.

38. The method according to any one of claims 35 to 37, characterized in that, The Dense-RVCNN includes at least one of the following: The input layer is used for feature extraction from the reference region; At least one residual multi-scale convolutional RVCB structure is used to perform feature extraction and feature concatenation at multiple scales on the input feature information; The DenseNet network structure is used for feature extraction from input feature information; The output layer is used to convert the input features into feature information with the same number of channels as the reference region, and to weight the reference region and the converted feature information.

39. The method according to claim 38, characterized in that, The at least one RVCB structure is a plurality of RVCB structures. The input layer is connected to the input end of the DenseNet structure through a portion of the plurality of RVCB structures. The output end of the DenseNet structure is connected to the output layer through another portion of the plurality of RVCB structures.

40. The method according to claim 38, characterized in that, The RVCB structure includes: Multiple first feature extraction layers connected in parallel, each first feature extraction layer including a convolutional layer, wherein the convolutional kernels of the convolutional layers in different first feature extraction layers are different, and the number of channels of the convolutional layer in the first feature extraction layer is the ratio of the number of channels of the feature information input to the first feature extraction layer to the number of at least one first feature extraction layer; The first splicing layer is used to splice the feature information output by the multiple first feature extraction layers. The first convolutional layer is used to extract features from the feature information output by the first splicing layer; A skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.

41. The method according to claim 38, characterized in that, The DenseNet structure includes: Multiple second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers; Multiple second concatenation layers, wherein two adjacent second feature extraction layers are connected through a second concatenation layer, and any one of the multiple second concatenation layers is used to: concatenate the feature information input to the DenseNet structure and the feature information output by the second feature extraction layer preceding the concatenation layer; The second convolutional layer is connected to the last of the plurality of second feature extraction layers through the last of the plurality of second concatenation layers. The second convolutional layer is used to convert the feature information output by the last second concatenation layer into feature information with the same number of channels as the feature information input to the DenseNet structure.

42. The method according to any one of claims 22 to 41, characterized in that, The method further includes: Encode the motion parameters of the current block; The motion parameters of the current block include at least one of the following: the motion vector of the current block, and the index of the reference image.

43. A neural network training method, characterized in that, include: The sample video is encoded to obtain a bitstream; The bitstream is decoded to determine the prediction block for the current block; The neural network is trained based on the label of the current block and the predicted block.

44. The method according to claim 43, characterized in that, Decoding the bitstream to determine the prediction block of the current block includes: Determine prediction blocks of multiple sizes; The step of training the neural network based on the label of the current block and the predicted block includes: Based on the label of the current block and the multiple sizes of the predicted blocks, the neural network is trained to obtain the network parameters corresponding to the multiple sizes.

45. The method according to claim 44, characterized in that, The prediction blocks of multiple sizes include multiple prediction blocks with different minimum side lengths.

46. ​​A decoder, characterized in that, include: The first determining unit is used to decode the bitstream and determine the motion parameters of the current block; The second determining unit is used to determine a reference block of the current block in the reference image of the current block based on the motion parameters of the current block; An enhancement unit is used to enhance the quality of the reference block to obtain an enhanced block; The third determining unit is used to determine the prediction block of the current block based on the enhancement block.

47. An encoder, characterized in that, include: The first determining unit is used to perform motion estimation on the current block to obtain a reference block for the current block; An enhancement unit is used to enhance the quality of the reference block to obtain an enhanced block; The second determining unit is used to determine the prediction block of the current block based on the enhancement block.

48. A neural network training device, characterized in that, include: The encoding unit is used to encode the sample video to obtain the bitstream; A decoding unit is used to decode the bitstream and determine the prediction block of the current block; A training unit is used to train the neural network based on the label of the current block and the prediction block.

49. An electronic device, characterized in that, include: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, performs at least one of the following methods: the method according to any one of claims 1 to 21, the method according to any one of claims 22 to 42, and the method according to any one of claims 43 to 45.

50. A computer-readable storage medium, characterized in that, Used to store a computer program that, when the computer program is run on a computer, causes the computer to perform at least one of the following methods: the method according to any one of claims 1 to 21, the method according to any one of claims 22 to 42, and the method according to any one of claims 43 to 45.

51. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement at least one of the following methods: the method according to any one of claims 1 to 21, the method according to any one of claims 22 to 42, and the method according to any one of claims 43 to 45.

52. A bitstream, characterized in that, The bitstream is a bitstream decoded by the method according to any one of claims 1 to 21, or the bitstream is a bitstream generated by the method according to any one of claims 22 to 42.

Images