Decoding method, coding method, training method, decoder, and coder
By enhancing the quality of the reference block, generating enhancement blocks to determine the prediction blocks in video encoding and decoding, the existing interpolation filter prediction efficiency is solved and the video encoding and decoding performance is improved.
Patent Information
- Application Number
- PCT/CN2023/095934
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-28
AI Technical Summary
The existing interpolation filter prediction methods have low prediction efficiency in video encoding and decoding, resulting in poor video encoding and decoding performance.
By performing quality enhancement on the reference block, the enhanced block is obtained and the prediction block is determined based on the enhancement block, avoiding the determination process of coupling the quality enhancement process to the prediction block, thereby reducing the decoding complexity and improving decoding performance.
Improves video encoding and codec performance, improves the accuracy and decoding efficiency of prediction blocks.
Smart Images

Figure CN2023095934_28112024_PF_FP_ABST
Abstract
Description
Decoding method, encoding method, training method, decoder and encoder Technical Field
[0001] The embodiments of the present application relate to the field of coding and decoding technology, and more specifically, to a decoding method, an encoding method, a training method, a decoder, and an encoder. Background Art
[0002] Digital video technology can be incorporated into a variety of video devices, such as digital televisions, smartphones, computers, e-readers, or video players. With the development of video technology, the amount of video data included has increased. To facilitate the transmission of video data, video devices implement video compression technology to enable more efficient transmission or storage of video data.
[0003] Since temporal or spatial redundancy exists in videos, prediction can eliminate or reduce it, improving compression efficiency. To improve prediction, interpolation filtering prediction methods are often used for predictive compression. However, current interpolation filtering prediction methods suffer from low prediction efficiency, resulting in poor video encoding and decoding performance.
[0004] Summary of the Invention
[0005] The embodiments of the present application provide a decoding method, an encoding method, a training method, a decoder, and an encoder, which can improve decoding performance.
[0006] In a first aspect, an embodiment of the present application provides a decoding method, comprising:
[0007] Decode the code stream and determine the motion parameters of the current block;
[0008] determining, based on the motion parameter of the current block, a reference block for the current block in a reference image of the current block;
[0009] Performing quality enhancement on the reference block to obtain an enhanced block;
[0010] Based on the enhancement block, a prediction block of the current block is determined.
[0011] In a second aspect, an embodiment of the present application provides an encoding method, including:
[0012] Performing motion estimation on the current block to obtain a reference block of the current block;
[0013] Performing quality enhancement on the reference block to obtain an enhanced block;
[0014] Based on the enhancement block, a prediction block of the current block is determined.
[0015] In a third aspect, an embodiment of the present application provides a neural network training method, comprising:
[0016] Encode the sample video to obtain a bit stream;
[0017] Decoding the code stream to determine a prediction block of a current block;
[0018] The neural network is trained based on the label of the current block and the predicted block.
[0019] In a fourth aspect, an embodiment of the present application provides a decoder, including:
[0020] A first determining unit, configured to decode a bitstream and determine motion parameters of a current block;
[0021] a second determining unit, configured to determine a reference block of the current block in a reference image of the current block based on a motion parameter of the current block;
[0022] an enhancement unit, configured to perform quality enhancement on the reference block to obtain an enhanced block;
[0023] The third determining unit is configured to determine a prediction block of the current block based on the enhanced block.
[0024] In a fifth aspect, an embodiment of the present application provides an encoder, including:
[0025] A first determining unit, configured to perform motion estimation on a current block to obtain a reference block of the current block;
[0026] an enhancement unit, configured to perform quality enhancement on the reference block to obtain an enhanced block;
[0027] The second determining unit is configured to determine a prediction block of the current block based on the enhanced block.
[0028] In a sixth aspect, an embodiment of the present application provides a neural network training device, comprising:
[0029] The encoding unit is used to encode the sample video to obtain a code stream;
[0030] A decoding unit, configured to decode the code stream and determine a prediction block of a current block;
[0031] A training unit is configured to train the neural network based on the label of the current block and the predicted block.
[0032] In a seventh aspect, an embodiment of the present application provides a decoder, including:
[0033] a processor adapted to implement computer instructions; and,
[0034] A computer-readable storage medium stores computer instructions, wherein the computer instructions are suitable for being loaded by a processor and executing the decoding method in the first aspect or its various implementations mentioned above.
[0035] In one implementation, there are one or more processors and one or more memories.
[0036] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.
[0037] In an eighth aspect, an embodiment of the present application provides an encoder, including:
[0038] a processor adapted to implement computer instructions; and,
[0039] A computer-readable storage medium stores computer instructions, wherein the computer instructions are suitable for being loaded by a processor and executing the encoding method in the second aspect or its various implementations mentioned above.
[0040] In one implementation, there are one or more processors and one or more memories.
[0041] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.
[0042] In a ninth aspect, an embodiment of the present application provides a neural network training device, comprising:
[0043] a processor adapted to implement computer instructions; and,
[0044] A computer-readable storage medium stores computer instructions, wherein the computer instructions are suitable for being loaded by a processor and executing the encoding method in the third aspect or its various implementations involved above.
[0045] In one implementation, there are one or more processors and one or more memories.
[0046] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.
[0047] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are read and executed by a processor of a computer device, the computer device executes the decoding method described in the first aspect above, the encoding method described in the second aspect above, or the training method described in the third aspect above.
[0048] In an eleventh aspect, embodiments of the present application provide a computer program product or computer program, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the decoding method described in the first aspect, the encoding method described in the second aspect, or the training method described in the third aspect.
[0049] In a twelfth aspect, an embodiment of the present application provides a code stream, which is the code stream involved in the decoding method described in the first aspect above or the code stream generated by the method described in the second aspect above.
[0050] Based on the above technical solution, an embodiment of the present application provides a decoding method, which includes: decoding a code stream to determine the motion parameters of a current block; based on the motion parameters of the current block, determining a reference block of the current block in a reference image of the current block; performing quality enhancement on the reference block to obtain an enhanced block; and determining a prediction block of the current block based on the enhanced block. In an embodiment of the present application, the decoder performs quality enhancement on the reference block to obtain an enhanced block, and then determines the prediction block of the current block based on the enhanced block; this is equivalent to the decoder converting the quality enhancement process for the prediction block into a quality enhancement process for the reference block, thereby avoiding coupling the quality enhancement process for the prediction block to the determination process of the prediction block, reducing decoding complexity and improving decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] FIG1 is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application.
[0052] FIG2 is a schematic block diagram of a video encoder according to an embodiment of the present application.
[0053] FIG3 is a schematic block diagram of a video decoder according to an embodiment of the present application.
[0054] FIG4 is an example of a VVC / H.266 framework involved in an embodiment of the present application.
[0055] FIG5 is an example of a reference block involved in an embodiment of the present application.
[0056] FIG6 is an example of motion compensation involved in an embodiment of the present application.
[0057] FIG7 is a schematic flowchart of a decoding method provided in an embodiment of the present application.
[0058] FIG8 is an example of the encoding process provided by an embodiment of the present application.
[0059] FIG9 is an example of a decoding process provided by an embodiment of the present application.
[0060] FIG10 is an example of a decoder providing an embodiment of the present application performing interpolation filtering based on an enhancement block.
[0061] FIG11 is another example of a decoder providing an embodiment of the present application performing interpolation filtering based on an enhancement block.
[0062] FIG12 is a schematic flowchart of an inter-frame prediction process without introducing an enhancement block provided in an embodiment of the present application.
[0063] FIG13 is a schematic flowchart of an inter-frame prediction process with an enhancement block introduced according to an embodiment of the present application.
[0064] Figure 14 is a schematic structural diagram of the Dense-RVCNN provided in an embodiment of the present application.
[0065] Figure 15 is a schematic structural diagram of the DenseNet structure in the Dense-RVCNN provided in an embodiment of the present application.
[0066] FIG16 is a schematic flowchart of the encoding method provided in an embodiment of the present application.
[0067] FIG17 is a schematic flowchart of a neural network training method provided in an embodiment of the present application.
[0068] FIG18 is an example of input and labels for training a neural network according to an embodiment of the present application.
[0069] Figure 19 is a schematic block diagram of the decoder provided in an embodiment of the present application.
[0070] Figure 20 is a schematic block diagram of the encoder provided in an embodiment of the present application.
[0071] FIG21 is a schematic block diagram of a neural network training device provided in an embodiment of the present application.
[0072] Figure 22 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0073] The solution provided by the embodiment of the present application can be applied to the field of digital video coding technology.
[0074] For example, the technical fields to which the solutions provided in the embodiments of the present application can be applied include but are not limited to: image coding and decoding fields, video coding and decoding fields, hardware video coding and decoding fields, dedicated circuit video coding and decoding fields, and real-time video coding and decoding fields. In addition, the solutions provided in the embodiments of the present application can be combined with the Audio Video Coding Standard (AVS), the second-generation AVS standard (AVS2), or the third-generation AVS standard (AVS3). For example, including but not limited to: H.264 / Audio Video Coding (AVC) standard, H.265 / High Efficiency Video Coding (HEVC) standard, and H.266 / Versatile Video Coding (VVC) standard. In addition, the solutions provided in the embodiments of the present application can be used for lossy compression of images, or for lossless compression of images. Among them, the lossless compression can be visually lossless compression or mathematically lossless compression.
[0075] For ease of understanding, the video encoding and decoding system involved in the embodiment of the present application is first introduced with reference to FIG1 .
[0076] FIG1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present application.
[0077] As shown in FIG. 1 , the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120 .
[0078] The encoding device 110 is used to encode (which can be understood as compressing) the video data to generate a code stream, and transmit the code stream to the decoding device 120. The decoding device 120 decodes the code stream generated by the encoding device 110 to obtain decoded video data.
[0079] The encoding device 110 can be understood as a device with a video encoding function, and the decoding device 120 can be understood as a device with a video decoding function, that is, the embodiments of the present application include a wider range of devices for the encoding device 110 and the decoding device 120, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, car computers, etc.
[0080] The encoding device 110 may transmit the encoded video data (eg, a code stream) to the decoding device 120 via the channel 130 .
[0081] Channel 130 may include one or more media and / or devices capable of transmitting encoded video data from encoding device 110 to decoding device 120 .
[0082] Channel 130 may include one or more communication media that enable encoding device 110 to transmit encoded video data directly to decoding device 120 in real time. Encoding device 110 may modulate the encoded video data according to a communication standard and transmit the modulated video data to decoding device 120. The communication media may include wireless communication media, such as radio frequency spectrum. The communication media may also include wired communication media, such as one or more physical transmission lines.
[0083] Channel 130 may include a storage medium that can store the video data encoded by encoding device 110. The storage medium includes various locally accessible data storage media, such as optical disks, DVDs, flash memories, etc. In this example, decoding device 120 may obtain the encoded video data from the storage medium.
[0084] Channel 130 may include a storage server that can store the video data encoded by encoding device 110. In this example, decoding device 120 can download the stored encoded video data from the storage server. Alternatively, the storage server can store the encoded video data and transmit the encoded video data to decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.
[0085] The encoding device 110 includes a video encoder 112 and an output interface 113 .
[0086] The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter. The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored on a storage medium or storage server for subsequent reading by the decoding device 120.
[0087] In addition to the video encoder 112 and the input interface 113 , the encoding device 110 may also include a video source 111 .
[0088] Video source 111 may include at least one of a video capture device (e.g., a video camera), a video archive, a video input interface, and a computer graphics system. The video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data. Video encoder 112 encodes the video data from video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains encoding information for the picture or picture sequence in the form of a bitstream. The encoding information may include the encoded picture data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. An SPS may contain parameters applicable to one or more sequences. A PPS may contain parameters applicable to one or more pictures. A syntax structure is a set of zero or more syntax elements arranged in a specified order within the bitstream.
[0089] The decoding apparatus 120 includes an input interface 121 and a video decoder 122. The input interface 121 may include a receiver and / or a modem.
[0090] The decoding device 120 may include a display device 123 in addition to the input interface 121 and the video decoder 122 .
[0091] The input interface 121 can receive the encoded video data via the channel 130. The video decoder 122 is configured to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to the display device 123. The display device 123 displays the decoded video data. The display device 123 can be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 can include various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.
[0092] It should be understood that Figure 1 is only an example of the present application and should not be understood as a display of the present application. That is to say, the technical solution of the embodiment of the present application is not limited to the system framework shown in Figure 1. For example, the technology of the present application can also be applied to unilateral video encoding or unilateral video decoding.
[0093] The following is an introduction to the video encoding framework involved in the embodiments of the present application.
[0094] FIG2 is a schematic block diagram of a video encoder 200 according to an embodiment of the present application.
[0095] It should be understood that the video encoder 200 can be applied to image data in luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in terms of color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).
[0096] The video encoder 200 reads video data and, for each frame of the video data, divides the frame into a number of coding tree units (CTUs). In some examples, a CTB may be referred to as a "tree block," "largest coding unit" (LCU), or "coding tree block" (CTB). Each CTU may be associated with a block of pixels of equal size within the image. Each pixel may correspond to one luminance (luma) sample and two chrominance (chroma) samples. Therefore, each CTU may be associated with one luminance sample block and two chrominance sample blocks. The size of a CTU may be, for example, 128×128, 64×64, 32×32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding. A CU may be a rectangular block or a square block. A CU may be further divided into a prediction unit (PU) and a transform unit (TU), thereby separating encoding, prediction, and transform, making processing more flexible. In an example, a CTU is divided into CUs in a quadtree manner, and a CU is divided into TUs and PUs in a quadtree manner.
[0097] The video encoder and video decoder can support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and video decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.
[0098] As shown in Figure 2, the video encoder 200 may include: a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filter unit 260, a decoded image buffer 270, and an entropy coding unit 280. It should be noted that the video encoder 200 may include more, fewer, or different functional components. In this application, the current block may be referred to as the current coding unit (CU) or the current prediction unit (PU), etc. The prediction block may also be referred to as a predicted image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstructed block or an image reconstructed image block.
[0099] The prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame prediction unit 212. Because there is a strong correlation between adjacent pixels in a video frame, intra-frame prediction is used in video coding and decoding to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent frames in a video, inter-frame prediction is used in video coding and decoding to eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.
[0100] The inter-frame prediction unit 211 is used for inter-frame prediction. Inter-frame prediction includes motion estimation and motion compensation. It can refer to image information from different frames. Inter-frame prediction uses motion information to find a reference block from a reference frame and generates a prediction block based on the reference block to eliminate temporal redundancy. The frames used for inter-frame prediction can be P-frames and / or B-frames. P-frames refer to forward-predicted frames, while B-frames refer to bidirectionally predicted frames. Inter-frame prediction uses motion information to find a reference block from a reference frame and generates a prediction block based on the reference block. Motion information includes the reference frame list in which the reference frame is located, the reference frame index, and the motion vector. The motion vector can be integer pixel or fractional pixel. If the motion vector is fractional pixel, interpolation filtering is required to generate the required fractional pixel block in the reference frame. Here, the integer pixel or fractional pixel block in the reference frame found based on the motion vector is called a reference block. Some technologies directly use the reference block as the prediction block, while others further process the reference block to generate a prediction block. Reprocessing the reference block to generate a prediction block can also be understood as using the reference block as the prediction block and then processing the prediction block to generate a new prediction block.
[0101] The intra-frame prediction unit 212 only refers to the information of the same frame image to predict the pixel information in the current code image block to eliminate spatial redundancy. The frame used for intra-frame prediction can be an I frame.
[0102] There are multiple prediction modes for intra-frame prediction. Taking the H series of international digital video coding standards as an example, the H.264 / AVC standard has 8 angular prediction modes and 1 non-angular prediction mode, and H.265 / HEVC has expanded to 33 angular prediction modes and 2 non-angular prediction modes. The intra-frame prediction modes used by HEVC are planar mode (Planar), DC, and 33 angular modes, for a total of 35 prediction modes. The intra-frame modes used by VVC are Planar, DC, and 65 angular modes, for a total of 67 prediction modes. It should be noted that with the increase in angular modes, intra-frame prediction will become more accurate and more in line with the needs of the development of high-definition and ultra-high-definition digital video.
[0103] The residual unit 220 may generate a residual block for the CU based on the pixel blocks of the CU and the prediction blocks of the PUs of the CU. For example, the residual unit 220 may generate the residual block for the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel blocks of the CU and the corresponding sample in the prediction blocks of the PUs of the CU.
[0104] The transform / quantization unit 230 may quantize the transform coefficients. The transform / quantization unit 230 may quantize the transform coefficients associated with the TUs of a CU based on a quantization parameter (QP) value associated with the CU. The video encoder 200 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.
[0105] The inverse transform / quantization unit 240 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.
[0106] Reconstruction unit 250 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of a CU in this manner, video encoder 200 can reconstruct the pixel blocks of the CU.
[0107] The loop filter unit 260 processes the inverse-transformed and inverse-quantized pixels to compensate for distortion and provide a better reference for subsequent pixel encoding. For example, it can perform a deblocking filter (DF) operation to reduce the blocking artifacts of pixel blocks associated with the CU. In some embodiments, the loop filter unit 260 includes a deblocking filter (DF) unit and a sample adaptive offset / adaptive loop filter (SAO / ALF) unit. The DF unit is used to remove blocking artifacts, while the SAO / ALF unit is used to remove ringing artifacts.
[0108] The decoded image buffer 270 may store the reconstructed pixel blocks. The inter-prediction unit 211 may use a reference image containing the reconstructed pixel blocks to perform inter-prediction on PUs of other images. In addition, the intra-prediction unit 212 may use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra-prediction on other PUs in the same image as the CU.
[0109] The entropy coding unit 280 may receive the quantized transform coefficients from the transform / quantization unit 230. The entropy coding unit 280 may perform one or more entropy coding operations on the quantized transform coefficients to generate entropy-coded data.
[0110] FIG3 is a schematic block diagram of a video decoder according to an embodiment of the present application.
[0111] 3 , the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transformation unit 330, a reconstruction unit 340, a loop filter unit 350, and a decoded picture buffer 360. It should be noted that the video decoder 300 may include more, fewer, or different functional components.
[0112] The video decoder 300 may receive a bitstream. The entropy decoding unit 310 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding unit 310 may parse the entropy-encoded syntax elements in the bitstream. The prediction unit 320, the inverse quantization / transform unit 330, the reconstruction unit 340, and the loop filter unit 350 may decode the video data based on the syntax elements extracted from the bitstream, thereby generating decoded video data.
[0113] The prediction unit 320 includes an intra prediction unit 322 and an inter prediction unit 321 .
[0114] The intra-prediction unit 322 may perform intra-prediction to generate a prediction block for the PU. The intra-prediction unit 322 may use an intra-prediction mode to generate a prediction block for the PU based on pixel blocks of spatially neighboring PUs. The intra-prediction unit 322 may also determine the intra-prediction mode for the PU based on one or more syntax elements parsed from the codestream.
[0115] The inter-frame prediction unit 321 may construct a first reference picture list (List 0) and a second reference picture list (List 1) based on syntax elements parsed from the codestream. In addition, if a PU is encoded using inter-frame prediction, the entropy decoding unit 310 may parse the motion information of the PU. The inter-frame prediction unit 321 may determine one or more reference blocks for the PU based on the motion information of the PU. The inter-frame prediction unit 321 may generate a prediction block for the PU based on the one or more reference blocks of the PU.
[0116] The inverse quantization / transform unit 330 may inversely quantize (i.e., dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 330 may use the QP value associated with the CU of the TU to determine the degree of quantization. After inverse quantizing the transform coefficients, the inverse quantization / transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients to generate a residual block associated with the TU.
[0117] The reconstruction unit 340 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.
[0118] The loop filtering unit 350 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with a CU.
[0119] The video decoder 300 may store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 may use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.
[0120] The basic process of video encoding and decoding is as follows: At the encoder end, a frame of image is divided into blocks. For the current block, the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual unit 220 calculates a residual block based on the predicted block and the original block of the current block. This residual block is the difference between the predicted block and the original block of the current block. This residual block is also called residual information. This residual block undergoes transformation and quantization by the transform / quantization unit 230, removing information that is insensitive to the human eye and eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization unit 230 is called a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization unit 230 is called a frequency residual block or a frequency-domain residual block. The entropy coding unit 280 receives the quantized change coefficients output by the change quantization unit 230 and performs entropy coding on these quantized change coefficients to output a bitstream. For example, the entropy coding unit 280 can eliminate character redundancy based on the target context model and probability information of the binary bitstream.
[0121] At the decoding end, the entropy decoding unit 310 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction for the current block based on the prediction information to generate a prediction block for the current block. The inverse quantization / transformation unit 330 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image. The encoding end also needs to use operations similar to those of the decoder to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be a subsequent frame, which serves as a reference frame for inter-frame prediction.
[0122] It should be noted that the block division information determined by the encoder, as well as mode information or parameter information such as prediction, transformation, quantization, entropy coding, and loop filtering, etc., are carried in the code stream when necessary. The decoding end determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoding end by parsing the code stream and analyzing the existing information, thereby ensuring that the decoded image obtained by the encoder and the decoder are the same. In addition, in an embodiment of the present application, due to the need for parallel processing, the image can be divided into slices, etc., and the slices in the same image can be processed in parallel, that is, there is no data dependency between them. "Frame" is a commonly used term, and it can generally be understood that one frame is an image. Of course, the frame described in this application can also be replaced by an image or a slice, etc.
[0123] It is worth noting that the above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. This application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.
[0124] FIG4 is an example of another coding framework involved in an embodiment of the present application.
[0125] As shown in Figure 4, the encoding and decoding process within this coding framework includes the following: First, the encoder divides the original video signal into multiple image blocks, processes each block individually, and then passes it through the prediction module. The prediction module is mainly divided into intra-frame prediction and inter-frame prediction. Intra-frame prediction primarily infers the content of the current block based on spatial relationships between images. Inter-frame prediction, on the other hand, predicts the content of the current block based on spatial and temporal relationships between consecutive frames. The predicted image (prediction block) is subtracted from the original signal image (original block) to obtain a residual image (residual block). The residual block is then subjected to operations such as variable quantization and entropy coding to further compress the video data, ultimately outputting a video bitstream file. The decoder reconstructs the video signal based on the bitstream file output by the encoder. Most common video compression methods are lossy, sacrificing image quality in exchange for a high compression ratio. Therefore, the video signal reconstructed by the decoder is lossy compared to the original video.
[0126] Inter-frame prediction is mainly used to eliminate temporal redundancy in video signals.
[0127] In a video signal, adjacent video frame images have a large amount of similar content. As shown in Figure 5, when encoding the current block in the current frame (T frame), the image block (reference block) that is most similar to the current block can be found from the reconstructed reference frame (Tn frame). Since the two image blocks are very similar, we only need to transmit the difference between the two blocks and the motion vector (MV) between the two blocks. Compared to directly transmitting the information of the current block image, this method greatly compresses the image information that needs to be transmitted. The process of finding the reference block is called motion estimation (ME). However, due to the inherent spatial discretization characteristics of digital video, block translation may not be exactly aligned with the pixels.
[0128] FIG6 is an example of motion compensation involved in an embodiment of the present application.
[0129] As shown in Figure 6, the block where the dots are located represents the integer pixel position. The current block can usually find the position of the reference block based on the integer pixel MV (Int MV, IMV), but this method is still not accurate enough. Optionally, the reference block can be interpolated first, and then the image of the fractional pixel position can be selected to further improve the accuracy of the prediction. The fractional pixel image is called a prediction block, and its displacement is called fractional precision displacement (FMV). The process of selecting the optimal prediction block based on the reference block is called motion compensation (MC). How to derive the fractional pixel image from the integer pixel image, VVC uses discrete cosine interpolation filters with different parameters to filter the integer pixel image. The prediction block obtained after filtering may be closer to the original block than the reference block.
[0130] As can be seen above, the quality of the reference block directly affects the encoding quality of the current block. The closer the reference block image is to the original block, the less residual block information needs to be transmitted, and the lower the video bitrate. In this area, most research focuses on optimizing interpolation filters, using more appropriate interpolation filter tap coefficients to generate better prediction blocks.
[0131] With the development of the field of deep learning, many scholars use network models generated by deep learning to derive or refine images of prediction blocks. Most of them use neural network models to replace the interpolation filtering process in inter-frame prediction to improve the accuracy of the prediction block. In one feasible solution, a neural network model containing three layers of convolution can be provided, and different data sets are made using reference blocks and original blocks corresponding to different fractional interpolation filters to train the network model. Finally, the convolution layer parameters in the network model are compressed into a matrix, and the matrix is used to replace the interpolation filter tap coefficients in H.266 / VVC inter-frame prediction. This method uses a neural network model to directly replace the interpolation filtering process, which has the following two disadvantages:
[0132] 1. During inter-frame prediction, hundreds of interpolation filters are often performed to find the optimal prediction value for the current block. Calling the neural network model each time significantly increases encoding complexity.
[0133] 2. Use network models instead of interpolation filters. Different network models cannot be used in combination.
[0134] In view of this, the present application introduces a method for enhancing the quality of a reference area of a reference frame to obtain an enhanced image; specifically, an enhanced block corresponding to the reference block is found on the enhanced image to perform interpolation filtering to obtain a prediction block. This technology only needs to enhance the quality of the frame when the reconstruction of the reference area is completed, which effectively reduces the complexity caused by calling the neural network, and can improve the image quality of the reference area, which is conducive to more accurate prediction. First, the present application creates a data set to train the proposed neural network; secondly, the neural network is integrated into the H.266 / VVC framework to enhance the reconstructed reference area. The same reference image uses different network parameters of the neural network to generate multiple enhanced images, and the enhanced block that matches the size of the CU block is determined on different enhanced images based on the minimum side length of the CU block. The enhanced block is used instead of the reference block for subsequent operations such as motion compensation. Finally, the effectiveness of this method in improving video encoding and decoding performance was verified through experiments.
[0135] 7 , the decoding method provided in the embodiment of the present application will be introduced by taking the decoding end as an example.
[0136] FIG7 is a schematic flow chart of a decoding method 400 provided in an embodiment of the present application. It should be understood that the decoding method 400 can be performed by a decoder. For example, the decoding method 400 can be performed by the video decoder 122 shown in FIG1 or the video decoder 300 shown in FIG3 . For ease of description, the following description uses a decoder as an example.
[0137] As shown in FIG. 7 , the decoding method 400 may include:
[0138] S410 , the decoder decodes the code stream and determines the motion vector (MV) of the current block.
[0139] Exemplarily, the decoder determines the MV of the current block by decoding the bitstream.
[0140] Exemplarily, the MV of the current block may include an MV with integer-pixel precision and an MV with fractional-pixel precision.
[0141] S420: The decoder determines a reference block for the current block in a reference image of the current block based on the motion parameter of the current block.
[0142] Exemplarily, the reference image may be a decoded image.
[0143] Exemplarily, the reference block may be a component reference block, such as a chroma component reference block or a luminance component reference block.
[0144] S430: The decoder performs quality enhancement on the reference block to obtain an enhanced block.
[0145] Exemplarily, the decoder performs quality enhancement on the reference block through a neural network and obtains an enhanced block.
[0146] Among them, the neural network includes but is not limited to: traditional learning models, integrated learning models or deep learning models. Traditional learning models include but are not limited to: tree models (regression trees) or logistic regression (LR) models; integrated learning models include but are not limited to: improved models of gradient boosting algorithms (XGBoost) or random forest models; deep learning models include but are not limited to: deep neural networks (DNN), convolutional neural networks (CNN) and recurrent neural networks (RNN), dynamic Bayesian networks (DBN) and stacked auto-encoder networks (SAE) models. Of course, in other embodiments of the present application, other machine learning models or even models combined from multiple models can also be used.
[0147] Of course, in other alternative embodiments, the decoder may also enhance the quality of the reference block by other methods. For example, the decoder may enhance the quality of the reference block based on information obtained by decoding the bitstream. This application does not make any specific limitation on this.
[0148] S440: The decoder determines a prediction block for the current block based on the enhancement block.
[0149] Exemplarily, the decoder may perform motion compensation on the enhanced block to obtain the predicted block.
[0150] The motion compensation includes but is not limited to the following two types:
[0151] Inter prediction filter: To mitigate discontinuities between predicted pixels and adjacent pixels, each predicted value in the reference block is filtered. This filtering is achieved through weighted averaging. The filter inputs include the predicted pixel and adjacent pixels, with weights determined based on the positional relationships between the pixels.
[0152] Affine Motion Compensation (AMC): Only translational motion is considered for the movement of objects, while real scenes often have more complex motion patterns such as rotation and scaling. In order to adapt to different motion patterns, AMC divides the CU into different sub-blocks and generates MV for each sub-block through an affine model. The affine model includes a 4-parameter (2 control point CPMV) model and a 6-parameter model (3 control point CPMV). After determining each reference sub-block based on the MV of each sub-block, each predicted sub-block is obtained by interpolation filtering of each reference sub-block, and each predicted sub-block constitutes a prediction block.
[0153] In an embodiment of the present application, the decoder obtains an enhanced block by performing quality enhancement on the reference block, and then determines the prediction block of the current block based on the enhanced block; this is equivalent to the decoder converting the quality enhancement process of the prediction block into a quality enhancement process of the reference block, thereby avoiding coupling the quality enhancement process of the prediction block to the determination process of the prediction block, reducing the decoding complexity and thus improving the decoding performance.
[0154] The technical solution proposed in this application is implemented in the VVC test software VTM11.0_nnvc_2.0, and the test sequences used are the Class B, Class C, Class D, and Class E sequences given in the general test conditions. The results in Table 1 are obtained by encoding in LDP mode with QP22, 27, 32, and 37 settings. The values in the table are BD-rate, which is a way to measure the performance of the algorithm. The changes in Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) of the solution algorithm provided in this application compared to the original algorithm are generally negative, indicating that the performance has improved, and the larger the absolute value, the greater the performance improvement.
[0155] Table 1
[0156] As shown in Table 1, the test results show that for categories B to E, the solution provided by this application improves PSNR by 0.8% and SSIM by 1.22% compared to the original algorithm. This shows that the technical solution provided by this application can improve decoding performance.
[0157] In some embodiments, the S430 may include:
[0158] The decoder performs quality enhancement on a reference area in the reference image including the reference block to obtain an enhanced image; and then determines the enhanced block in the enhanced image based on a position of the reference block in the reference area.
[0159] For example, the decoder uses a neural network to enhance the quality of a reference image to obtain an enhanced image. The decoder then determines the enhanced block in the enhanced image based on the position of the reference block in the reference image. Of course, in other alternative embodiments, the decoder may also use a neural network to directly enhance the quality of the reference block.
[0160] FIG8 is an example of the encoding process provided by an embodiment of the present application.
[0161] As shown in Figure 8, at the encoder end, motion estimation can be used to find the reference block for the current block. Then, an enhancement block corresponding to the reference block's position is found on the enhanced image. Motion compensation is then performed on the enhancement block to obtain a predicted block. The enhanced image can be an image obtained by enhancing the quality of the reference region in the reference image, including the reference block. After obtaining the predicted block, the encoder performs a subtraction between the predicted block and the original image of the current block to obtain residual information for the current block. This residual information is then transformed and quantized, and entropy coding is performed on the transformed and quantized information to produce a bitstream file.
[0162] FIG9 is an example of a decoding process provided by an embodiment of the present application.
[0163] As shown in Figure 9, at the decoder end, the decoded MV can be used to find the reference block of the current block in the reconstructed image. Then, the enhanced block corresponding to the reference block is found in the enhanced image, and motion compensation is performed on the enhanced block to obtain the predicted block. The enhanced image can be an image obtained by enhancing the quality of the reference region in the reference image, including the reference block. After obtaining the predicted block for the current block, the decoder can determine the reconstructed video based on the predicted block and the residual information obtained through entropy decoding, inverse quantization, and inverse transformation of the current block. Furthermore, the reconstructed video can be subjected to loop filtering or post-processing.
[0164] In some embodiments, the decoder determines a block in the enhanced image that has the same position as the reference block in the reference region as the enhanced block.
[0165] Exemplarily, the decoder determines the position of the reference pixel of the enhanced block as the pixel in the enhanced image that is at the same position as the reference pixel of the reference block. The reference pixel in the reference block can be any pixel in the reference block, including but not limited to the pixel at the upper right corner, lower right corner, upper left corner, lower left corner, or center position.
[0166] In some embodiments, in some embodiments, the S440 may include:
[0167] The decoder performs boundary extension and interpolation filtering on the enhanced block to obtain the prediction block.
[0168] Exemplarily, after the decoder extends the boundary of the enhanced block, it uses a filter to filter each pixel in the enhanced block after the boundary extension to obtain a predicted block. For example, the filter input includes the predicted pixel and adjacent pixels, and the weight factors of the predicted pixel and the adjacent pixels are determined based on the positional relationship between the pixels.
[0169] For example, if the decoder determines the pixel in the enhanced image that is at the same position as the reference pixel of the reference block as the position of the reference pixel of the enhanced block, then when the decoder determines the prediction block based on the enhanced block, it can directly perform boundary extension and interpolation filtering on the enhanced block to obtain the prediction block.
[0170] FIG10 is an example of a decoder providing an embodiment of the present application performing interpolation filtering based on an enhancement block.
[0171] As shown in Figure 10, during conventional motion compensation, the decoder determines the reference block of the current block based on the vector. Since the fractional interpolation filter will cause the image size to be reduced, it is necessary to expand the boundary of the reference pixel by 4 pixels before performing interpolation filtering to ensure that the image size before and after filtering is consistent. The decoder adds the filtered prediction block and the residual block to obtain a reconstructed block. Correspondingly, the encoder subtracts the filtered prediction block from the original block to obtain a residual block, and then performs operations such as transform and quantization on the residual block. For the process of interpolation filtering without introducing an enhanced block, as shown in method 1 in the figure, subsequent operations such as interpolation filtering can be performed directly on the reference block. For the interpolation filtering process after the enhanced block is introduced in this application, as shown in method 2 in the figure, the boundary extension image of the block with the same position as the reference block can be found on the enhanced image as an enhanced block, and then subsequent operations such as interpolation filtering can be performed on the enhanced block.
[0172] In some embodiments, the S420 may include:
[0173] The decoder divides the current block into at least one sub-block; then, based on the motion parameter of the current block, determines the motion parameter of the at least one sub-block; based on the motion parameter of the at least one sub-block, determines at least one reference sub-block included in the reference block; wherein S430 may include:
[0174] The decoder determines a first region where the at least one reference sub-block is located based on the position of the at least one reference sub-block in the reference region; then, performs boundary extension on the first region to obtain a second region; and determines a block in the enhanced image whose position corresponds to the second region as the enhanced block.
[0175] Exemplarily, the decoder divides the current block into at least one sub-block; then, based on the MV of the current block, determines the MV of the at least one sub-block; based on the MV of the at least one sub-block, determines at least one reference sub-block included in the reference block; wherein S430 may include:
[0176] The decoder determines a first region where the at least one reference sub-block is located based on the position of the at least one reference sub-block in the reference region; then, performs boundary extension on the first region to obtain a second region; and determines a block in the enhanced image whose position corresponds to the second region as the enhanced block.
[0177] Exemplarily, the decoder generates an MV for each sub-block using an affine model. Affine models include 4-parameter (2 control point CPMV) models and 6-parameter models (3 control point CPMV). The decoder determines at least one reference sub-block based on the MV of each sub-block, and uses the area including the at least one reference sub-block as the first area. The first area is then boundary-extended to obtain a second area, and the area in the enhanced image that is identical to the second area is determined as the enhanced block.
[0178] In some embodiments, the S440 may include:
[0179] The decoder determines a subblock in the enhanced block that has the same position as the at least one reference subblock in the reference region as the at least one enhanced subblock included in the enhanced block; and then performs boundary extension and interpolation filtering on the at least one enhanced subblock to obtain the at least one predicted subblock included in the predicted block.
[0180] Exemplarily, the decoder obtains each prediction sub-block by performing interpolation filtering on each reference sub-block, and each prediction sub-block can form a prediction block.
[0181] In some embodiments, the first region is a minimum region including the at least one reference sub-block.
[0182] Exemplarily, the first region is a minimum rectangular region including at least one reference sub-block. Of course, the first region may also be a region of other shapes, which is not specifically limited in this application.
[0183] FIG11 is another example of a decoder providing an embodiment of the present application performing interpolation filtering based on an enhancement block.
[0184] As shown in Figure 11, in affine motion compensation, for the interpolation filtering process without introducing an enhancement block, as shown in Method 1 in the figure, the current CU is split into several 4×4 sub-blocks, and the MV of each sub-block is calculated according to a 4-parameter or 6-parameter affine model. Each current sub-block can find the corresponding reference sub-block through the motion vector, and then perform boundary extension, fractional interpolation and other operations on the reference sub-block to obtain a predicted sub-block; the predicted sub-blocks are combined to obtain a predicted block. For the interpolation filtering process after the enhancement block is introduced in this application, as shown in Method 2 in the figure, the minimum rectangular area containing all reference sub-blocks can be found, and the enhancement block at the corresponding position can be found on the enhanced image; then the corresponding extended sub-block is found in the enhancement block and fractional interpolation is performed, and the interpolated prediction sub-blocks are combined into a prediction block.
[0185] In some embodiments, the reference area is the entire area or a partial area of the reference image.
[0186] In some embodiments, the partial area is an area where any one of the following items is located: an image block, a sub-picture, a rectangular area (tile), or a slice.
[0187] Exemplarily, the image may be a decoded block, and the decoded block may be a CU or a CTU.
[0188] For example, the reference image may be divided into one or more slices or referred to as slices, and the data encoding of each slice is independent.
[0189] For example, the reference image can be divided horizontally and vertically into several rectangular regions, called tiles. Tiles are not required to be evenly distributed. Typically, each tile contains approximately equal CTU data. Each tile can be encoded independently, and during encoding, the CTUs contained in each tile are encoded in the scan order.
[0190] It should be understood that the reference image can be divided into multiple slices or tiles, both for independent encoding. Some slices can contain multiple tiles, and similarly, some tiles can contain multiple slices. The number of CTUs contained in a tile does not affect the number of CTUs in a slice.
[0191] In some embodiments, the S430 may include:
[0192] The decoder extracts features from the reference region to obtain a residual image; and then weights the reference region and the residual image to obtain the enhanced image.
[0193] Exemplarily, the weight of the reference region and the weight of the residual image may be predefined weight values.
[0194] Exemplarily, after obtaining the residual value, the decoder may directly add the reference region and the residual image to obtain an enhanced image.
[0195] In some embodiments, the decoder determines a first parameter for performing feature extraction on the reference area based on the size of the current block; and then, based on the first parameter, performs feature extraction on the reference area to obtain the residual image.
[0196] For example, the decoder may determine a first parameter set for feature extraction of the reference region based on the size of the current block. The first parameter set may include at least one parameter, including the first parameter. The first parameter set may be parameters involved in performing various operations on the reference region. In other words, the first parameter is related to the operation process used for feature extraction of the reference region.
[0197] In some embodiments, the decoder determines the parameter corresponding to the minimum side length of the current block as the first parameter.
[0198] Exemplarily, the decoder may determine the parameter corresponding to the minimum side length of the current block as the first parameter based on a mapping relationship between multiple side lengths and multiple parameters. The mapping relationship may be predefined information.
[0199] In some embodiments, the decoder performs feature extraction on the reference area to obtain first feature information; then, performs multi-scale feature extraction on the first feature information and splices the extracted multi-scale feature information to obtain second feature information; determines third feature information based on the second feature information and feature information obtained by feature extraction of the second feature information; performs multi-scale feature extraction on the third feature information and splices the extracted multi-scale feature information to obtain fourth feature information; and converts the fourth feature information into feature information having the same number of channels as the number of channels of the reference area to obtain the residual image.
[0200] Exemplarily, the first feature information and the second feature information have the same number of channels.
[0201] Exemplarily, the second feature information and the third feature information have the same number of channels.
[0202] Exemplarily, the third feature information and the fourth feature information have the same number of channels.
[0203] Exemplarily, the first feature information, the second feature information, or the third feature information includes feature information on multiple channels, and the number of channels of the reference region is feature information on one or more channels. For example, when the reference image is a component reference image, the number of channels of the reference region may be 1.
[0204] In some embodiments, in some embodiments, the S430 may include:
[0205] The decoder uses a dense residual variable-size convolutional neural network (Dense-RVCNN) to enhance the quality of the reference area to obtain the enhanced image.
[0206] For example, Dense-RVCNN may be a densely connected neural network for performing residual convolution.
[0207] FIG12 is a schematic flowchart of an inter-frame prediction process 450 without introducing an enhancement block provided in an embodiment of the present application.
[0208] As shown in FIG. 12 , the inter-frame prediction process 450 without introducing an enhancement block may include:
[0209] S451, obtain the current block.
[0210] S452: Determine a matching reference block in the reference image.
[0211] When the encoder performs inter-frame prediction on the current block, it first uses motion estimation to find the block most similar to the current block as a reference block. For example, a Tzsearch search can be performed on a constructed reference image. The image block that is most similar to the current block (with the lowest RDcost) is the reference block. Accordingly, when the decoder performs inter-frame prediction on the current block, it uses the motion vector obtained from the decoded bitstream to determine the block that is similar to the current block as the reference block.
[0212] S453, determining a reference block after sub-pixel precision displacement.
[0213] After the encoder determines the reference block, it performs motion compensation on the reference block to obtain multiple reference blocks with sub-pixel precision. Correspondingly, after the decoder encoder determines the reference block, it can also perform motion compensation on the reference block to determine multiple reference blocks with sub-pixel precision.
[0214] S454, determining the reference block after optimal sub-pixel precision shifting.
[0215] After the encoder obtains multiple sub-pixel precision shifted reference blocks, it can determine the reference block that is most similar to the current block (with the lowest RDcost) among the multiple sub-pixel precision shifted reference blocks as the optimal sub-pixel precision shifted reference block. Correspondingly, after the decoder obtains multiple sub-pixel precision shifted reference blocks, it can also determine the reference block that is most similar to the current block (with the lowest RDcost) among the multiple sub-pixel precision shifted reference blocks as the optimal sub-pixel precision shifted reference block. This optimal sub-pixel precision shifted reference block can be used as the prediction block for the current block.
[0216] FIG13 is a schematic flowchart of an inter-frame prediction process 460 with an enhancement block introduced according to an embodiment of the present application.
[0217] As shown in FIG. 13 , the inter-frame prediction process 460 with the enhancement block may include:
[0218] S461, obtain the current block.
[0219] S462: Determine a matching reference block in the reference image.
[0220] When the encoder performs inter-frame prediction on the current block, it first uses motion estimation to find the block most similar to the current block as a reference block. For example, a Tzsearch search can be performed on a constructed reference image. The image block that is most similar to the current block (with the lowest RDcost) is the reference block. Accordingly, when the decoder performs inter-frame prediction on the current block, it uses the motion vector obtained from the decoded bitstream to determine the block that is similar to the current block as the reference block.
[0221] S463: Determine in the enhanced image that the block having the same position as the reference block is the enhanced block.
[0222] Once the reference image is constructed, a neural network is used to enhance the quality of the reference area in the reference image, including the reference block, to obtain an enhanced image. The enhanced block at the same position as the reference block is found in the enhanced image and replaced with the original reference block for subsequent operations.
[0223] S464, determining a reference block after sub-pixel precision displacement.
[0224] After the encoder determines the reference block, it performs motion compensation on the reference block to obtain multiple reference blocks with sub-pixel precision. Correspondingly, after the decoder encoder determines the reference block, it can also perform motion compensation on the reference block to determine multiple reference blocks with sub-pixel precision.
[0225] S465, determining the reference block after optimal sub-pixel precision shifting.
[0226] After the encoder obtains multiple sub-pixel precision shifted reference blocks, it can determine the reference block that is most similar to the current block (with the lowest RDcost) among the multiple sub-pixel precision shifted reference blocks as the optimal sub-pixel precision shifted reference block. Correspondingly, after the decoder obtains multiple sub-pixel precision shifted reference blocks, it can also determine the reference block that is most similar to the current block (with the lowest RDcost) among the multiple sub-pixel precision shifted reference blocks as the optimal sub-pixel precision shifted reference block. This optimal sub-pixel precision shifted reference block can be used as the prediction block for the current block.
[0227] For example, before S463, the inter-frame prediction process 460 introducing the enhancement block may further include a neural network training process (or a pre-trained neural network), and the neural network training process may include:
[0228] S4621: Encode the sample video.
[0229] For example, 650 videos can be randomly selected from the BVI-DVC dataset (a dataset containing a large number of scene YUV video files). Using the VTM-9.3 encoder, with the encoding mode set to LDP and the quantization parameter (Qp) set to 22, the first 32 images of these videos are encoded to generate a bitstream. Alternatively, random access (RA) or low latency (LB) configurations can be used, but this application does not specifically limit this.
[0230] S4622, determining a lossless current block and a reference block with optimal sub-pixel precision displacement.
[0231] For example, for the current block, the bitstream contains integer-precision motion vector information and fractional-precision motion vector information. The training device can derive the integer-pixel positions of the reference block from the integer-pixel positions of the current block. Then, a sub-pixel interpolation filter is used to perform sub-pixel shifting on the integer-pixel reference block to obtain the predicted block.
[0232] For example, the predicted block after sub-pixel precision displacement in the dataset is used as the input of the neural network, and the lossless current block is used as the true value to train the neural network.
[0233] S4623, training the neural network.
[0234] For example, when training a neural network, the MSE loss function can be used. The initial learning rate can be set to 0.0001, and the learning rate can be increased by 0.1 every 60 cycles. Each network is trained for 180 cycles. Prediction blocks of 16×16, 32×32, and 64×64 sizes are used to train the three neural networks: DENSE-RVCNN1, DENSE-RVCNN2, and DENSE-RVCNN3, respectively.
[0235] In some embodiments, the decoder determines network parameters used by Dense-RVCNN based on the size of the current block; and then determines the enhanced image based on the network parameters used by the Dense-RVCNN.
[0236] Exemplarily, the network parameters of the Dense-RVCNN may include multiple sets of network parameters. When the decoder calls the Dense-RVCNN, it can determine the network parameters used by the Dense-RVCNN according to the size of the current block; then, the network parameters used by the Dense-RVCNN are used to enhance the quality of the reference area to obtain an enhanced image.
[0237] Exemplarily, the network parameters of the Dense-RVCNN include multiple sets of network parameters. When the decoder calls the Dense-RVCNN, it can directly call the multiple sets of network parameters of the Dense-RVCNN to enhance the quality of the reference area to obtain multiple images after quality enhancement. Then, the network parameters used by the Dense-RVCNN can be determined based on the size of the current block, and an enhanced image can be determined based on the network parameters used by the Dense-RVCNN among the multiple images after quality enhancement. For example, an image corresponding to the network parameters used by the Dense-RVCNN can be determined as an enhanced image.
[0238] In this embodiment, the decoder determines the network parameters used by Dense-RVCNN based on the size of the current block, which can reduce the complexity of the decoder when calling Dense-RVCNN, thereby improving the decoding performance of the decoder.
[0239] In some embodiments, the decoder determines the parameter corresponding to the minimum side length of the current block as the network parameter used by the Dense-RVCNN.
[0240] Exemplarily, if the minimum side length of the current block is greater than or equal to a preset threshold, the decoder determines the parameters corresponding to the minimum side length of the current block as the network parameters used by Dense-RVCNN.
[0241] Assuming that enhanced images are introduced for all rectangular blocks larger than 16×16 in the reference region, the decoder can determine the enhanced block to replace the reference block in the following way:
[0242] For rectangular blocks with a minimum side length of 16, the enhanced blocks in the enhanced image 1 output by the DENSE-RVCNN1 model can be used to replace the reference blocks; for rectangular blocks with a minimum side length of 32, the enhanced blocks on the enhanced image 2 output by the DENSE-RVCNN2 model can be used to replace the reference blocks; for rectangular blocks with a minimum side length of 64 and 128, the enhanced blocks on the enhanced image 3 output by the DENSE-RVCNN3 model can be used to replace the reference blocks.
[0243] In some embodiments, the Dense-RVCNN includes at least one of the following:
[0244] An input layer, used for extracting features from the reference area;
[0245] At least one residual multi-scale convolutional RVCB structure, used to extract and stitch features at multiple scales on input feature information;
[0246] Dense network DenseNet structure is used to extract features from input feature information;
[0247] The output layer is used to convert the input features into feature information having the same number of channels as the reference region and to weight the reference region and the converted feature information.
[0248] Exemplarily, the input layer may include a convolutional layer and an activation layer.
[0249] For example, the RVCB structure may include a structure for extracting features at multiple scales from input features and a concatenation layer for concatenating the extracted feature information. The structure for performing multi-scale extraction may be implemented by convolution layers corresponding to convolution kernels of different sizes.
[0250] For example, the DenseNet structure is a convolutional neural network with dense connections. There is a direct connection between any two layers in the DenseNet structure, that is, the input of each layer is the union of the outputs of all previous layers, and the feature map learned by the layer will also be directly passed to all subsequent layers as input. Each layer of the DenseNet structure is directly connected to the previous layer, which enables the reuse of features. Each layer of the DenseNet structure can be designed to be particularly narrow, that is, only a very small number of feature maps are learned (the most extreme case is that each layer only learns one feature map) to achieve the purpose of reducing redundancy.
[0251] Exemplarily, the output layer may include a convolutional layer capable of extracting feature information having the same number of channels as that of the reference region.
[0252] In some embodiments, the at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input end of the DenseNet structure through some of the RVCB structures in the plurality of RVCB structures, and the output end of the DenseNet structure is connected to the output layer through another portion of the RVCB structures in the plurality of RVCB structures.
[0253] For example, the input layer can be connected to the input end of the DenseNet structure through half of the RVCB structures in the multiple RVCB structures, and the output end of the DenseNet structure can be connected to the output layer through the other half of the RVCB structures in the multiple RVCB structures. Of course, the number of RVCB structures used to connect the input layer and the input end of the DenseNet structure may also be different from the number of RVCB structures used to connect the output end of the DenseNet structure and the output layer, and this application does not specifically limit this.
[0254] In some embodiments, the RVCB structure includes:
[0255] A plurality of first feature extraction layers connected in parallel, wherein the first feature extraction layers include convolutional layers, convolution kernels of different first feature extraction layers among the plurality of first feature extraction layers are different, and the number of channels of the convolutional layers in the first feature extraction layers is a ratio of the number of channels of feature information input to the first feature extraction layer to the number of the at least one first feature extraction layer;
[0256] A first splicing layer, configured to perform feature splicing on the feature information output by the plurality of first feature extraction layers;
[0257] A first convolutional layer is used to extract feature information output by the first concatenation layer;
[0258] The skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.
[0259] Exemplarily, the first feature extraction layer may include a convolutional layer and a subsequent activation layer.
[0260] Exemplarily, the first convolutional layer may also be an optional convolutional layer.
[0261] Exemplarily, the skip connection layer may directly add the feature information input to the RVCB structure and the feature information output by the first convolutional layer to obtain the feature information output by the RVCB structure.
[0262] In some embodiments, the DenseNet structure includes:
[0263] A plurality of second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers;
[0264] A plurality of second splicing layers, wherein two adjacent second feature extraction layers in the plurality of second splicing layers are connected via the second splicing layer, and any one of the plurality of second splicing layers is used to: perform feature splicing on feature information input to the DenseNet structure and feature information output by the second feature extraction layer preceding the any one second splicing layer;
[0265] A second convolutional layer, wherein the last second feature extraction layer among the multiple second feature extraction layers is connected to the second convolutional layer through the last second splicing layer among the multiple second splicing layers, and the second convolutional layer is used to: convert the feature information output by the last second splicing layer into feature information having the same number of channels as the feature information input for the DenseNet structure.
[0266] Exemplarily, the second feature extraction layer may include a convolutional layer and a subsequent activation layer.
[0267] Exemplarily, the multiple second splicing layers are used to: achieve dense connection of the multiple second feature extraction layers.
[0268] Figure 14 is a schematic structural diagram of the Dense-RVCNN provided in an embodiment of the present application.
[0269] As shown in Figure 14, the Dense-RVCNN consists of 6 RVCB structures, 1 DenseNet structure, convolutional layers at the beginning and end, and skip connection layers.
[0270] Dense-RVCNN takes the whole pixel reference block before motion compensation as input and passes it through the first convolution layer with a 3×3 kernel and LeakyReLU. This process can be expressed as: F1 = LReLU(W0·X) = max(0,W0·X) + α·min(W0·X).
[0271] Where W0 represents the convolution kernel of the first layer, α represents the slope of the LeakyReLU function on the negative half axis, X represents the input of the neural network, F1 represents the output of the first layer, and LReLU() represents the nonlinear mapping function of LeakyReLU.
[0272] The RVCB structure uses a multi-size convolution structure.
[0273] When feature data enters the RVCB structure, it is first converted into two sets of 32-channel feature data using a 3×3 convolution kernel, a 1×1 convolution kernel, and a LeakyReLU. Subsequently, the two sets of feature data are concatenated in the channel dimension and converted into 64-channel feature data through a 3×3 convolution. Finally, this set of data is added to the module input through a skip connection to obtain the module output. This process can be expressed as: i,1_1 =LReLU(W i,1_1 ·F i ); F i,1_3 =LReLU(W i,1_3 ·F i ); F i,2 =W i,2 ·cat(·F i,1_3 ,F i,1_3 ); F i+1 =F i +F i,2 .
[0274] Where, 1≤i≤7 and i≠4, F i is the characteristic information of the i-th RVCB structure input, W i,1_1 is the convolution kernel parameter of the first layer 1×1 of the i-th RVCB structure, W i,1_3 is the convolution kernel parameter of the first layer 3×3 of the i-th RVCB structure, W i,2 is the parameter of the second convolution kernel of the i-th RVCB structure. cat() represents the concatenation of feature data.
[0275] Figure 15 is a schematic structural diagram of the DenseNet structure in the Dense-RVCNN provided in an embodiment of the present application.
[0276] As shown in Figure 15, the DenseNet structure consists of a convolutional block (CB) and a 1×1 convolution layer. CB is also called a feature extraction layer. Each CB contains a 1×1 convolution layer, a 3×3 convolution layer, and a LeakyReLU function. The 1×1 convolution layer is used to compress the number of channels of the input feature to 64 channels to facilitate processing by the 3×3 convolution. The calculation formula for the entire DenseNet structure is as follows: 4,j =CB j (cat(F 4,0 ,...,F 4,j-1 ))=LReLU(W 4,j_2 LReLU(W 4,j_1 ·cat(F 4,0 ,...,F 4,j-1 ))); F5=W 4,5 ·F4,4 .
[0277] Among them, 1≤j≤4 is the feature information input for the DenseNet structure, F 4,0 is the feature information output by the jth CB, and F5 is the feature information output by the DenseNet structure. 4,j_1 is the parameter of the first convolution kernel in the jth CB, W 4,j_2 is the parameter of the second convolution kernel in the jth CB, W 4,5 are the parameters of the convolution kernel in the last layer of the DenseNet structure. In each CB, the number of channels of the 1×1 convolution kernel is 64, the number of channels of the 3×3 convolution kernel is 32, and the number of channels of the last convolution layer of the DenseNet structure is 64.
[0278] Finally, a convolutional layer with a kernel size of 3×3 and a channel number of 1 is used to compress all feature maps into a residual image. The residual image is added to the input image to obtain the final output image. The process is as follows: F9 = W9·F8; Y = F9 + X.
[0279] Among them, Y is the output of the neural network, F8 is the feature information output by the last RVCB structure, W9 is the final convolution kernel parameter, and X represents the input of the neural network.
[0280] In some embodiments, the motion parameter includes at least one of the following: a motion vector of the current block, an index of the reference image.
[0281] Exemplarily, the decoder may determine the motion vector of the current block by decoding the code stream, and determine a reference block for the current block in a predefined reference image based on the motion vector. The predefined reference image may be a decoded image determined according to a preset rule.
[0282] Exemplarily, the decoder may determine the index by decoding the code stream, and determine the reference block of the current block in the reference image indicated by the index.
[0283] Exemplarily, the decoder may determine the motion vector and index of the current block by decoding the code stream, and determine a reference block for the current block in the reference image indicated by the index based on the motion vector. The predefined reference image may be a decoded image determined according to a preset rule.
[0284] The decoding method according to the embodiment of the present application is described in detail above from the perspective of the decoder. The encoding method according to the embodiment of the present application will be described below from the perspective of the encoder in conjunction with Figure 16.
[0285] FIG16 is a schematic flow chart of an encoding method 500 provided in an embodiment of the present application. It should be understood that the encoding method 500 may be performed by an encoder. For example, the encoding method 500 may be performed by the video encoder 112 shown in FIG1 or the video encoder 200 shown in FIG2 . For ease of description, the following description uses an encoder as an example.
[0286] As shown in FIG. 16 , the decoding method 500 may include:
[0287] S510, the encoder performs motion estimation on the current block to obtain a reference block of the current block;
[0288] S520, the encoder performs quality enhancement on the reference block to obtain an enhanced block;
[0289] S530: The encoder determines a prediction block for the current block based on the enhancement block.
[0290] In some embodiments, the S520 may include:
[0291] The encoder performs quality enhancement on a reference region including the reference block in the reference image to obtain an enhanced image; and then determines the enhanced block in the enhanced image based on a position of the reference block in the reference region.
[0292] In some embodiments, the encoder determines a block in the enhanced image that has the same position as the reference block in the reference region as the enhanced block.
[0293] In some embodiments, the S530 may include:
[0294] The encoder performs boundary extension and interpolation filtering on the enhanced block to obtain the prediction block.
[0295] In some embodiments, the S510 may include:
[0296] The encoder divides the current block into at least one sub-block; then, performs motion estimation on the current block to determine motion parameters of the current block; determines motion parameters of the at least one sub-block based on the motion parameters of the current block; and determines at least one reference sub-block included in the reference block based on the motion parameters of the at least one sub-block; wherein S520 may include:
[0297] The encoder determines a first region where the at least one reference sub-block is located based on the position of the at least one reference sub-block in the reference region; then, performs boundary extension on the first region to obtain a second region; and determines a block in the enhanced image whose position corresponds to the second region as the enhanced block.
[0298] In some embodiments, the S530 may include:
[0299] The encoder determines a subblock in the enhancement block that has the same position as the at least one reference subblock in the reference region as the at least one enhancement subblock included in the enhancement block; and then performs boundary extension and interpolation filtering on the at least one enhancement subblock to obtain the at least one prediction subblock included in the prediction block.
[0300] In some embodiments, the first region is a minimum region including the at least one reference sub-block.
[0301] In some embodiments, the reference area is the entire area or a partial area of the reference image.
[0302] In some embodiments, the partial area is an area where any one of the following items is located: an image block, a sub-image, a rectangular area, or a slice.
[0303] In some embodiments, the encoder performs feature extraction on the reference region to obtain a residual image; then, the reference region and the residual image are weighted to obtain the enhanced image.
[0304] In some embodiments, the encoder determines a first parameter for performing feature extraction on the reference area based on the size of the current block; and then performs feature extraction on the reference area based on the first parameter to obtain the residual image.
[0305] In some embodiments, the encoder determines the parameter corresponding to the minimum side length of the current block as the first parameter.
[0306] In some embodiments, the encoder performs feature extraction on the reference area to obtain first feature information; then, performs multi-scale feature extraction on the first feature information and splices the extracted multi-scale feature information to obtain second feature information; determines third feature information based on the second feature information and feature information obtained by feature extraction of the second feature information; performs multi-scale feature extraction on the third feature information and splices the extracted multi-scale feature information to obtain fourth feature information; and converts the fourth feature information into feature information having the same number of channels as the number of channels of the reference area to obtain the residual image.
[0307] In some embodiments, the encoder uses a dense residual multi-scale convolutional network (Dense-RVCNN) to perform quality enhancement on the reference area to obtain the enhanced image.
[0308] In some embodiments, the encoder determines network parameters used by Dense-RVCNN based on the size of the current block; and then determines the enhanced image based on the network parameters used by the Dense-RVCNN.
[0309] In some embodiments, the encoder determines the parameter corresponding to the minimum side length of the current block as the network parameter used by the Dense-RVCNN.
[0310] In some embodiments, the Dense-RVCNN includes at least one of the following:
[0311] An input layer, used for extracting features from the reference area;
[0312] At least one residual multi-scale convolutional RVCB structure, used to extract and stitch features at multiple scales on input feature information;
[0313] Dense network DenseNet structure is used to extract features from input feature information;
[0314] The output layer is used to convert the input features into feature information having the same number of channels as the reference region and to weight the reference region and the converted feature information.
[0315] In some embodiments, the at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input end of the DenseNet structure through some of the RVCB structures in the plurality of RVCB structures, and the output end of the DenseNet structure is connected to the output layer through another portion of the RVCB structures in the plurality of RVCB structures.
[0316] In some embodiments, the RVCB structure includes:
[0317] A plurality of first feature extraction layers connected in parallel, wherein the first feature extraction layers include convolutional layers, convolution kernels of different first feature extraction layers among the plurality of first feature extraction layers are different, and the number of channels of the convolutional layers in the first feature extraction layers is a ratio of the number of channels of feature information input to the first feature extraction layer to the number of the at least one first feature extraction layer;
[0318] A first splicing layer, configured to perform feature splicing on the feature information output by the plurality of first feature extraction layers;
[0319] A first convolutional layer, configured to extract features from the feature information output by the first concatenation layer;
[0320] The skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.
[0321] In some embodiments, the DenseNet structure includes:
[0322] A plurality of second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers;
[0323] A plurality of second splicing layers, wherein two adjacent second feature extraction layers in the plurality of second splicing layers are connected via the second splicing layer, and any one of the plurality of second splicing layers is used to: perform feature splicing on feature information input to the DenseNet structure and feature information output by the second feature extraction layer preceding the any one second splicing layer;
[0324] A second convolutional layer, wherein the last second feature extraction layer among the multiple second feature extraction layers is connected to the second convolutional layer through the last second splicing layer among the multiple second splicing layers, and the second convolutional layer is used to: convert the feature information output by the last second splicing layer into feature information having the same number of channels as the feature information input for the DenseNet structure.
[0325] In some embodiments, the method 500 may further include:
[0326] encoding motion parameters of the current block;
[0327] The motion parameter of the current block includes at least one of the following: a motion vector of the current block and an index of the reference image.
[0328] It should be understood that the encoding method can be understood as the inverse process of the decoding method. Therefore, the specific scheme of the encoding method 500 can refer to the relevant content of the decoding method 400. For the sake of ease of description, this application will not go into details.
[0329] The above describes in detail the encoding and decoding method according to the embodiment of the present application from the perspective of the decoder and the encoder. The following will describe the neural network training method provided according to the embodiment of the present application from the perspective of the neural network in combination with Figure 17.
[0330] FIG17 is a schematic flowchart of a neural network training method 600 provided in an embodiment of the present application.
[0331] It should be understood that the training method 600 can be executed by any electronic device with data processing capability.
[0332] Furthermore, the training method 600 provided herein can be incorporated into the decoding method 400 or the encoding method 500. For example, a training device may train a neural network using the training method 600. After the neural network is trained, the decoder may use the neural network to perform quality enhancement on a reference region in a reference image, thereby obtaining an enhanced image with enhanced quality; and then determine, within the enhanced image, the enhanced block corresponding to the reference block. For another example, a training device may train a neural network using the training method 600. After the neural network is trained, the encoder may use the neural network to perform quality enhancement on a reference region in a reference image, thereby obtaining an enhanced image with enhanced quality; and then determine, within the enhanced image, the enhanced block corresponding to the reference block.
[0333] In addition, this application does not specifically limit the data set used when training the neural network.
[0334] As shown in FIG17 , the neural network training method 600 may include:
[0335] S610: The training device encodes the sample video to obtain a bit stream.
[0336] For example, 650 videos can be randomly selected from the BVI-DVC dataset (a dataset containing a large number of scene YUV video files). Using the VTM-9.3 encoder, with the encoding mode set to LDP and the quantization parameter (Qp) set to 22, the first 32 images of these videos are encoded to generate a bitstream. Alternatively, random access (RA) or low latency (LB) configurations can be used, but this application does not specifically limit this.
[0337] S620, the training device decodes the bitstream to determine a prediction block for the current block;
[0338] For example, for the current block, the bitstream contains integer-precision motion vector information and fractional-precision motion vector information. The training device can derive the integer-pixel positions of the reference block from the integer-pixel positions of the current block. Then, a sub-pixel interpolation filter is used to perform sub-pixel shifting on the integer-pixel reference block to obtain the predicted block.
[0339] For example, the predicted block after sub-pixel precision displacement in the dataset is used as the input of the neural network, and the lossless current block is used as the true value to train the neural network.
[0340] S630: The training device trains the neural network based on the label of the current block and the predicted block.
[0341] For example, when training a neural network, the MSE loss function can be used. The initial learning rate can be set to 0.0001, and the learning rate can be increased by 0.1 every 60 cycles. Each network is trained for 180 cycles. Prediction blocks of 16×16, 32×32, and 64×64 sizes are used to train the three neural networks: DENSE-RVCNN1, DENSE-RVCNN2, and DENSE-RVCNN3, respectively.
[0342] FIG18 is an example of input and labels for training a neural network according to an embodiment of the present application.
[0343] As shown in Figure 18, for the current block, the code stream contains integer-precision motion vector information and fractional-precision motion vector information. Integer-precision motion vector information includes Imvx and Imvy; fractional-precision motion vector information includes Fmvx and Fmvy. The training device can combine the integer-pixel position of the current block and the integer-precision motion vector information to derive the integer-pixel position of the reference block from the reference image, that is, the integer-pixel position of the reference block is: (X+Imvx, Y+Imvy), and then use a sub-pixel interpolation filter to perform sub-pixel shifting on the integer-pixel reference block to obtain the position of the predicted block, that is, the position of the predicted block is: (X+Imvx+Fmvx, Y+Imvy+Fmvy).
[0344] In some embodiments, the S620 may include:
[0345] The training device determines prediction blocks of multiple sizes; wherein S630 may include:
[0346] The training device trains the neural network based on the label of the current block and the prediction blocks of the multiple sizes to obtain network parameters corresponding to the multiple sizes.
[0347] Since the size of the reference blocks ranges from 4×4 to 128×128 and they are rectangular blocks, the present application can divide the reference blocks into large, medium and small according to the size of the reference blocks. Small blocks are rectangular blocks with a minimum side length of 16, including 16×16, 16×32, 32×16, 16×64, and 64×16 rectangular blocks. Medium blocks are rectangular blocks with a minimum side length of 32, including 32×32, 32×64, and 64×32 rectangular blocks. Large blocks are rectangular blocks with a minimum side length of 64, including 64×64, 64×128, 128×64, and 128×128 rectangular blocks. Three different data sets are made for large, medium, and small rectangles for training three sets of network parameters. Since there are too many rectangular blocks segmented in the video, only square prediction blocks with sizes of 16×16, 32×32, and 64×64 and the original current blocks are selected to make three data sets. When using the neural network model trained with these three data sets, the quality of the same reference area in the same reference image can be enhanced to generate three different enhanced images, and large, medium and small blocks can be respectively determined as enhanced blocks in different enhanced images.
[0348] In some embodiments, the prediction blocks of multiple sizes include multiple prediction blocks with different minimum side lengths.
[0349] The preferred embodiments of the present application have been described in detail above with reference to the accompanying drawings. However, the present application is not limited to the specific details of the embodiments described above. Within the scope of the technical concept of the present application, various simple variations of the technical solutions of the present application may be made, and these simple variations all fall within the scope of protection of the present application. For example, the specific technical features described in the specific embodiments described above may be combined in any suitable manner, unless there is any contradiction. To avoid unnecessary repetition, the present application will not further describe the various possible combinations. For another example, the various different embodiments of the present application may also be combined in any manner, as long as they do not violate the principles of the present application and should also be considered as the content disclosed in the present application. It should also be understood that in the various method embodiments of the present application, the order of execution of the processes described above does not necessarily indicate the order of execution. The order of execution of the processes should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in the embodiments of the present application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. Specifically, A and / or B can represent: A exists alone, A and B exists simultaneously, or B exists alone. In addition, the character “ / ” in this application generally indicates that the preceding and following related objects are in an “or” relationship.
[0350] The method embodiment of the present application is described in detail above. The following describes in detail an apparatus embodiment of the present application in conjunction with Figures 19 to 22.
[0351] FIG19 is a schematic block diagram of a decoder 710 according to an embodiment of the present application.
[0352] As shown in FIG19 , the decoder 710 may include:
[0353] A first determining unit 711 is configured to decode a bitstream and determine motion parameters of a current block;
[0354] A second determining unit 712 is configured to determine a reference block of the current block in a reference image of the current block based on a motion parameter of the current block;
[0355] an enhancing unit 713, configured to perform quality enhancement on the reference block to obtain an enhanced block;
[0356] The third determining unit 714 is configured to determine a prediction block of the current block based on the enhanced block.
[0357] In some embodiments, the enhancement unit 713 is specifically configured to:
[0358] Performing quality enhancement on a reference region including the reference block in the reference image to obtain an enhanced image;
[0359] The enhanced block is determined in the enhanced image based on a position of the reference block in the reference area.
[0360] In some embodiments, the enhancement unit 713 is specifically configured to:
[0361] A block in the enhanced image having the same position as the reference block in the reference area is determined as the enhanced block.
[0362] In some embodiments, the third determining unit 714 is specifically configured to:
[0363] Boundary extension and interpolation filtering are performed on the enhanced block to obtain the predicted block.
[0364] In some embodiments, the second determining unit 712 is specifically configured to:
[0365] Dividing the current block into at least one sub-block;
[0366] determining a motion parameter of the at least one sub-block based on a motion parameter of the current block;
[0367] Determining at least one reference sub-block included in the reference block based on a motion parameter of the at least one sub-block;
[0368] The enhancement unit 713 is specifically configured to:
[0369] determining, based on a position of the at least one reference sub-block in the reference region, a first region where the at least one reference sub-block is located;
[0370] Expanding the boundary of the first region to obtain a second region;
[0371] A block in the enhanced image whose position corresponds to the second area is determined as the enhanced block.
[0372] In some embodiments, the third determining unit 714 is specifically configured to:
[0373] Determine a subblock in the enhanced block having the same position as the at least one reference subblock in the reference region as the at least one enhanced subblock included in the enhanced block;
[0374] Boundary extension and interpolation filtering are performed on the at least one enhanced sub-block to obtain at least one prediction sub-block included in the prediction block.
[0375] In some embodiments, the first region is a minimum region including the at least one reference sub-block.
[0376] In some embodiments, the reference area is the entire area or a partial area of the reference image.
[0377] In some embodiments, the partial area is an area where any one of the following items is located: an image block, a sub-image, a rectangular area, or a slice.
[0378] In some embodiments, the enhancement unit 713 is specifically configured to:
[0379] Performing feature extraction on the reference area to obtain a residual image;
[0380] The reference region and the residual image are weighted to obtain the enhanced image.
[0381] In some embodiments, the enhancement unit 713 is specifically configured to:
[0382] determining, based on the size of the current block, a first parameter for performing feature extraction on the reference area;
[0383] Based on the first parameter, feature extraction is performed on the reference area to obtain the residual image.
[0384] In some embodiments, the enhancement unit 713 is specifically configured to:
[0385] The parameter corresponding to the minimum side length of the current block is determined as the first parameter.
[0386] In some embodiments, the enhancement unit 713 is specifically configured to:
[0387] Performing feature extraction on the reference area to obtain first feature information;
[0388] performing multi-scale feature extraction on the first feature information and splicing the extracted multi-scale feature information to obtain second feature information;
[0389] determining third feature information based on the second feature information and feature information obtained by extracting features from the second feature information;
[0390] performing multi-scale feature extraction on the third feature information and splicing the extracted multi-scale feature information to obtain fourth feature information;
[0391] The fourth feature information is converted into feature information having the same number of channels as the number of channels of the reference region to obtain the residual image.
[0392] In some embodiments, the enhancement unit 713 is specifically configured to:
[0393] The reference area is quality enhanced by using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image.
[0394] In some embodiments, the enhancement unit 713 is specifically configured to:
[0395] Determine network parameters used by Dense-RVCNN based on the size of the current block;
[0396] The enhanced image is determined based on the network parameters used by the Dense-RVCNN.
[0397] In some embodiments, the enhancement unit 713 is specifically configured to:
[0398] The parameters corresponding to the minimum side length of the current block are determined as the network parameters used by the Dense-RVCNN.
[0399] In some embodiments, the Dense-RVCNN includes at least one of the following:
[0400] An input layer, used for extracting features from the reference area;
[0401] At least one residual multi-scale convolutional RVCB structure, used to extract and stitch features at multiple scales on input feature information;
[0402] Dense network DenseNet structure is used to extract features from input feature information;
[0403] The output layer is used to convert the input features into feature information having the same number of channels as the reference region and to weight the reference region and the converted feature information.
[0404] In some embodiments, the at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input end of the DenseNet structure through some of the RVCB structures in the plurality of RVCB structures, and the output end of the DenseNet structure is connected to the output layer through another portion of the RVCB structures in the plurality of RVCB structures.
[0405] In some embodiments, the RVCB structure includes:
[0406] A plurality of first feature extraction layers connected in parallel, wherein the first feature extraction layers include convolutional layers, convolution kernels of different first feature extraction layers among the plurality of first feature extraction layers are different, and the number of channels of the convolutional layers in the first feature extraction layers is a ratio of the number of channels of feature information input to the first feature extraction layer to the number of the at least one first feature extraction layer;
[0407] A first splicing layer, configured to perform feature splicing on the feature information output by the plurality of first feature extraction layers;
[0408] A first convolutional layer, configured to extract features from the feature information output by the first concatenation layer;
[0409] The skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.
[0410] In some embodiments, the DenseNet structure includes:
[0411] A plurality of second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers;
[0412] A plurality of second splicing layers, wherein two adjacent second feature extraction layers in the plurality of second splicing layers are connected via the second splicing layer, and any one of the plurality of second splicing layers is used to: perform feature splicing on feature information input to the DenseNet structure and feature information output by the second feature extraction layer preceding the any one second splicing layer;
[0413] A second convolutional layer, wherein the last second feature extraction layer among the multiple second feature extraction layers is connected to the second convolutional layer through the last second splicing layer among the multiple second splicing layers, and the second convolutional layer is used to: convert the feature information output by the last second splicing layer into feature information having the same number of channels as the feature information input for the DenseNet structure.
[0414] In some embodiments, the motion parameter of the current block includes at least one of the following: a motion vector of the current block, an index of the reference image.
[0415] FIG20 is a schematic block diagram of the encoder 720 according to an embodiment of the present application.
[0416] As shown in FIG. 20 , the encoder 720 may include:
[0417] A first determining unit 721 is configured to perform motion estimation on a current block to obtain a reference block of the current block;
[0418] an enhancing unit 722, configured to perform quality enhancement on the reference block to obtain an enhanced block;
[0419] The second determining unit 723 is configured to determine a prediction block of the current block based on the enhanced block.
[0420] In some embodiments, the enhancement unit 722 is specifically configured to:
[0421] Performing quality enhancement on a reference region including the reference block in the reference image to obtain an enhanced image;
[0422] The enhanced block is determined in the enhanced image based on a position of the reference block in the reference area.
[0423] In some embodiments, the enhancement unit 722 is specifically configured to:
[0424] A block in the enhanced image having the same position as the reference block in the reference area is determined as the enhanced block.
[0425] In some embodiments, the second determining unit 723 is specifically configured to:
[0426] Boundary extension and interpolation filtering are performed on the enhanced block to obtain the predicted block.
[0427] In some embodiments, the first determining unit 721 is specifically configured to:
[0428] Dividing the current block into at least one sub-block;
[0429] Performing motion estimation on the current block to determine motion parameters of the current block;
[0430] Determining at least one reference sub-block included in the reference block based on a motion parameter of the at least one sub-block;
[0431] The enhancement unit 722 is specifically configured to:
[0432] determining, based on a position of the at least one reference sub-block in the reference region, a first region where the at least one reference sub-block is located;
[0433] Expanding the boundary of the first region to obtain a second region;
[0434] A block in the enhanced image whose position corresponds to the second area is determined as the enhanced block.
[0435] In some embodiments, the second determining unit 723 is specifically configured to:
[0436] Determine a subblock in the enhanced block having the same position as the at least one reference subblock in the reference region as the at least one enhanced subblock included in the enhanced block;
[0437] Boundary extension and interpolation filtering are performed on the at least one enhanced sub-block to obtain at least one prediction sub-block included in the prediction block.
[0438] In some embodiments, the first region is a minimum region including the at least one reference sub-block.
[0439] In some embodiments, the reference area is the entire area or a partial area of the reference image.
[0440] In some embodiments, the partial area is an area where any one of the following items is located: an image block, a sub-image, a rectangular area, or a slice.
[0441] In some embodiments, the enhancement unit 722 is specifically configured to:
[0442] Performing feature extraction on the reference area to obtain a residual image;
[0443] The reference region and the residual image are weighted to obtain the enhanced image.
[0444] In some embodiments, the enhancement unit 722 is specifically configured to:
[0445] determining, based on the size of the current block, a first parameter for performing feature extraction on the reference area;
[0446] Based on the first parameter, feature extraction is performed on the reference area to obtain the residual image.
[0447] In some embodiments, the enhancement unit 722 is specifically configured to:
[0448] The parameter corresponding to the minimum side length of the current block is determined as the first parameter.
[0449] In some embodiments, the enhancement unit 722 is specifically configured to:
[0450] Performing feature extraction on the reference area to obtain first feature information;
[0451] performing multi-scale feature extraction on the first feature information and splicing the extracted multi-scale feature information to obtain second feature information;
[0452] determining third feature information based on the second feature information and feature information obtained by extracting features from the second feature information;
[0453] performing multi-scale feature extraction on the third feature information and splicing the extracted multi-scale feature information to obtain fourth feature information;
[0454] The fourth feature information is converted into feature information having the same number of channels as the number of channels of the reference region to obtain the residual image.
[0455] In some embodiments, the enhancement unit 722 is specifically configured to:
[0456] The reference area is quality enhanced by using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image.
[0457] In some embodiments, the enhancement unit 722 is specifically configured to:
[0458] Determine network parameters used by Dense-RVCNN based on the size of the current block;
[0459] The enhanced image is determined based on the network parameters used by the Dense-RVCNN.
[0460] In some embodiments, the enhancement unit 722 is specifically configured to:
[0461] The parameters corresponding to the minimum side length of the current block are determined as the network parameters used by the Dense-RVCNN.
[0462] In some embodiments, the Dense-RVCNN includes at least one of the following:
[0463] An input layer, used for extracting features from the reference area;
[0464] At least one residual multi-scale convolutional RVCB structure, used to extract and stitch features at multiple scales on input feature information;
[0465] Dense network DenseNet structure is used to extract features from input feature information;
[0466] The output layer is used to convert the input features into feature information having the same number of channels as the reference region and to weight the reference region and the converted feature information.
[0467] In some embodiments, the at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input end of the DenseNet structure through some of the RVCB structures in the plurality of RVCB structures, and the output end of the DenseNet structure is connected to the output layer through another portion of the RVCB structures in the plurality of RVCB structures.
[0468] In some embodiments, the RVCB structure includes:
[0469] A plurality of first feature extraction layers connected in parallel, wherein the first feature extraction layers include convolutional layers, convolution kernels of different first feature extraction layers among the plurality of first feature extraction layers are different, and the number of channels of the convolutional layers in the first feature extraction layers is a ratio of the number of channels of feature information input to the first feature extraction layer to the number of the at least one first feature extraction layer;
[0470] A first splicing layer, configured to perform feature splicing on the feature information output by the plurality of first feature extraction layers;
[0471] A first convolutional layer is used to extract feature information output by the first concatenation layer;
[0472] The skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.
[0473] In some embodiments, the DenseNet structure includes:
[0474] A plurality of second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers;
[0475] A plurality of second splicing layers, wherein two adjacent second feature extraction layers in the plurality of second splicing layers are connected via the second splicing layer, and any one of the plurality of second splicing layers is used to: perform feature splicing on feature information input to the DenseNet structure and feature information output by the second feature extraction layer preceding the any one second splicing layer;
[0476] A second convolutional layer, wherein the last second feature extraction layer among the multiple second feature extraction layers is connected to the second convolutional layer through the last second splicing layer among the multiple second splicing layers, and the second convolutional layer is used to: convert the feature information output by the last second splicing layer into feature information having the same number of channels as the feature information input for the DenseNet structure.
[0477] In some embodiments, the second determining unit 723 is further configured to:
[0478] encoding motion parameters of the current block;
[0479] The motion parameter of the current block includes at least one of the following: a motion vector of the current block and an index of the reference image.
[0480] FIG21 is a schematic block diagram of a neural network training device 730 according to an embodiment of the present application.
[0481] As shown in FIG21 , the neural network training device 730 may include:
[0482] The encoding unit 731 is used to encode the sample video to obtain a code stream;
[0483] A decoding unit 732, configured to decode the code stream and determine a prediction block for a current block;
[0484] The training unit 733 is configured to train the neural network based on the label of the current block and the predicted block.
[0485] In some embodiments, the decoding unit 732 is specifically configured to:
[0486] determining prediction blocks of multiple sizes;
[0487] The step of training the neural network based on the label of the current block and the predicted block includes:
[0488] The neural network is trained based on the label of the current block and the prediction blocks of the multiple sizes to obtain network parameters corresponding to the multiple sizes.
[0489] In some embodiments, the prediction blocks of multiple sizes include multiple prediction blocks with different minimum side lengths.
[0490] It should be understood that the device embodiment and the method embodiment can correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, no further details will be given here. Specifically, the decoder 710 shown in Figure 19 can correspond to the corresponding subject in the method 400 of the embodiment of the present application, and the aforementioned and other operations and / or functions of the various units in the decoder 710 are respectively for implementing the corresponding processes in the various methods such as method 400. Similarly, the encoder 720 shown in Figure 20 can correspond to the corresponding subject in the method 500 of the embodiment of the present application, that is, the aforementioned and other operations and / or functions of the various units in the encoder 720 are respectively for implementing the corresponding processes in the various methods such as method 500. The neural network training device 730 shown in Figure 21 can correspond to the corresponding subject in the method 600 of the embodiment of the present application, that is, the aforementioned and other operations and / or functions of the various units in the neural network training device 730 are respectively for implementing the corresponding processes in the various methods such as method 600.
[0491] It should also be understood that the various units in the decoder 710, encoder 720 or neural network training device 730 involved in the embodiments of the present application can be separately or all merged into one or several other units to constitute, or one (some) of the units can be further split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The units involved above are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the decoder 710, encoder 720 or neural network training device 730 may also include other units. In actual applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, a computer program (including program code) capable of executing each step involved in the corresponding method can be run on a general-purpose computing device of a general-purpose computer including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), and a read-only storage medium (ROM). To construct the decoder 710, the encoder 720, or the neural network training device 730 involved in the embodiment of the present application, and to implement the encoding method or decoding method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, and loaded into an electronic device via a computer-readable storage medium, and run therein to implement the corresponding method of the embodiment of the present application.
[0492] In other words, the units mentioned above can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented in the form of a combination of hardware and software. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software in a decoding processor. Optionally, the software can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps in the method embodiments mentioned above in combination with its hardware.
[0493] FIG22 is a schematic structural diagram of an electronic device 800 provided in an embodiment of the present application.
[0494] As shown in Figure 22, the electronic device 800 includes at least a processor 810 and a computer-readable storage medium 820. The processor 810 and the computer-readable storage medium 820 may be connected via a bus or other means. The computer-readable storage medium 820 is used to store a computer program 821, which includes computer instructions. The processor 810 is used to execute the computer instructions stored in the computer-readable storage medium 820. The processor 810 is the computing core and control core of the electronic device 800 and is suitable for implementing one or more computer instructions, specifically loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.
[0495] The processor 810 may also be referred to as a central processing unit (CPU). The processor 810 may include, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, discrete hardware components, and the like.
[0496] Computer-readable storage media 820 includes, but is not limited to, volatile memory and / or non-volatile memory.
[0497] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. Exemplarily, RAM includes but is not limited to: static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM).
[0498] Exemplarily, the electronic device 800 may be an encoder or encoding framework involved in an embodiment of the present application; a first computer instruction is stored in the computer-readable storage medium 820; the processor 810 loads and executes the first computer instruction stored in the computer-readable storage medium 820 to implement the corresponding steps in the encoding method provided in an embodiment of the present application; in other words, the first computer instruction in the computer-readable storage medium 820 is loaded by the processor 810 and the corresponding steps are executed. To avoid repetition, it will not be repeated here.
[0499] Exemplarily, the electronic device 800 may be a decoder or decoding framework involved in an embodiment of the present application; a second computer instruction is stored in the computer-readable storage medium 820; the processor 810 loads and executes the second computer instruction stored in the computer-readable storage medium 820 to implement the corresponding steps in the decoding method provided in an embodiment of the present application; in other words, the second computer instruction in the computer-readable storage medium 820 is loaded by the processor 810 and the corresponding steps are executed. To avoid repetition, they will not be repeated here.
[0500] Exemplarily, the electronic device 800 may be the training device involved in the embodiment of the present application; the computer-readable storage medium 820 stores a third computer instruction; the processor 810 loads and executes the third computer instruction stored in the computer-readable storage medium 820 to implement the corresponding steps in the training method provided in the embodiment of the present application; in other words, the third computer instruction in the computer-readable storage medium 820 is loaded by the processor 810 and executes the corresponding steps, which will not be repeated here to avoid repetition.
[0501] According to another aspect of the present application, the present application also provides a coding and decoding system, including the encoder and decoder mentioned above.
[0502] According to another aspect of the present application, the present application also provides a computer-readable storage medium (Memory), which is a memory device in the electronic device 800 for storing programs and data. For example, a computer-readable storage medium 820. It can be understood that the computer-readable storage medium 820 here can include both the built-in storage medium in the electronic device 800 and, of course, the extended storage medium supported by the electronic device 800. The computer-readable storage medium provides a storage space, which stores the operating system of the electronic device 800. In addition, one or more computer instructions suitable for being loaded and executed by the processor 810 (for example, at least one of the aforementioned first computer instruction, second computer instruction, and third computer instruction) are also stored in the storage space. These computer instructions can be one or more computer programs 821 (including program codes).
[0503] According to another aspect of the present application, the present application further provides a computer program product or computer program, which includes computer instructions (e.g., any one of the aforementioned first computer instructions, second computer instructions, and third computer instructions), which are stored in a computer-readable storage medium. For example, computer program 821. In this case, the data processing device 800 can be a computer, and the processor 810 reads the computer instructions from the computer-readable storage medium 820. The processor 810 executes the computer instructions, causing the computer to perform the methods provided in the various optional embodiments mentioned above.
[0504] That is to say, when implemented using software, the methods provided in the various optional embodiments mentioned above can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process of the embodiment of the present application is run in whole or in part or the functions of the embodiment of the present application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method.
[0505] Those skilled in the art will appreciate that the units and process steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0506] Finally, it should be noted that the above content is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.
Claims
1. A decoding method, characterized in that: include: Decode the bitstream and determine the motion parameters of the current block; Determining, based on the motion parameter of the current block, a reference block of the current block in a reference image of the current block; Performing quality enhancement on the reference block to obtain an enhanced block; Based on the enhancement block, a prediction block of the current block is determined.
2. The method according to claim 1, characterized in that The step of performing quality enhancement on the reference block to obtain an enhanced block comprises: Performing quality enhancement on a reference area including the reference block in the reference image to obtain an enhanced image; The enhanced block is determined in the enhanced image based on the position of the reference block in the reference area.
3. The method according to claim 2, characterized in that The determining the enhanced block in the enhanced image based on the position of the reference block in the reference area comprises: A block in the enhanced image whose position is the same as that of the reference block in the reference area is determined as the enhanced block.
4. The method according to claim 3, characterized in that The step of determining a prediction block of the current block based on the enhancement block comprises: Boundary extension and interpolation filtering are performed on the enhanced block to obtain the predicted block.
5. The method according to claim 2, characterized in that: The determining, based on the motion parameter of the current block, a reference block of the current block in a reference image of the current block comprises: Dividing the current block into at least one sub-block; Determining a motion parameter of the at least one sub-block based on a motion parameter of the current block; Determining at least one reference sub-block included in the reference block based on the motion parameter of the at least one sub-block; Wherein, determining the enhanced block in the enhanced image based on the position of the reference block in the reference area includes: Determine, based on a position of the at least one reference sub-block in the reference region, a first region where the at least one reference sub-block is located; Expanding the boundary of the first region to obtain a second region; A block in the enhanced image whose position corresponds to the second region is determined as the enhanced block.
6. The method according to claim 5, characterized in that The step of determining a prediction block of the current block based on the enhancement block comprises: Determine a subblock in the enhanced block having the same position as the at least one reference subblock in the reference region as at least one enhanced subblock included in the enhanced block; Boundary extension and interpolation filtering are performed on the at least one enhanced sub-block to obtain at least one predicted sub-block included in the predicted block.
7. The method according to claim 5, characterized in that The first region is a minimum region including the at least one reference sub-block.
8. The method according to any one of claims 2 to 7, characterized in that The reference area is the entire area or a partial area of the reference image.
9. The method according to claim 8, characterized in that The partial area is an area where any one of the following items is located: an image block, a sub-image, a rectangular area, and a slice.
10. The method according to any one of claims 2 to 9, characterized in that The step of performing quality enhancement on a reference area including the reference block in the reference image to obtain an enhanced image comprises: Performing feature extraction on the reference area to obtain a residual image; The reference region and the residual image are weighted to obtain the enhanced image.
11. The method according to claim 10, characterized in that The step of extracting features from the reference area to obtain a residual image includes: Determining, based on the size of the current block, a first parameter for extracting features from the reference area; Based on the first parameter, feature extraction is performed on the reference area to obtain the residual image.
12. The method according to claim 11, characterized in that The determining, based on the size of the current block, a first parameter for extracting features from the reference area comprises: A parameter corresponding to the minimum side length of the current block is determined as the first parameter.
13. The method according to any one of claims 10 to 12, characterized in that The step of extracting features from the reference area to obtain a residual image includes: Extracting features from the reference area to obtain first feature information; Performing multi-scale feature extraction on the first feature information and splicing the extracted multi-scale feature information to obtain second feature information; determining third feature information based on the second feature information and feature information obtained by extracting features from the second feature information; Performing multi-scale feature extraction on the third feature information and splicing the extracted multi-scale feature information to obtain fourth feature information; The fourth feature information is converted into feature information having the same number of channels as the number of channels of the reference area to obtain the residual image.
14. The method according to any one of claims 2 to 9, characterized in that The step of performing quality enhancement on a reference area including the reference block in the reference image to obtain an enhanced image comprises: The reference area is quality enhanced by using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image.
15. The method according to claim 14, characterized in that The method of using a dense residual multi-scale convolutional network (Dense-RVCNN) to enhance the quality of the reference area to obtain the enhanced image includes: Based on the size of the current block, determine the network parameters used by Dense-RVCNN; The enhanced image is determined based on the network parameters used by the Dense-RVCNN.
16. The method according to claim 15, characterized in that Determining the network parameters used by Dense-RVCNN based on the size of the current block includes: The parameters corresponding to the minimum side length of the current block are determined as the network parameters used by the Dense-RVCNN.
17. The method according to any one of claims 14 to 16, characterized in that The Dense-RVCNN includes at least one of the following: An input layer, used for extracting features from the reference area; At least one residual multi-scale convolution RVCB structure, used to extract and concatenate features at multiple scales on input feature information; Dense network DenseNet structure is used to extract features from input feature information; The output layer is used to convert the input features into feature information having the same number of channels as the number of channels of the reference region and to weight the reference region and the converted feature information.
18. The method according to claim 17, characterized in that The at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input end of the DenseNet structure through some of the RVCB structures among the plurality of RVCB structures, and the output end of the DenseNet structure is connected to the output layer through another portion of the RVCB structures among the plurality of RVCB structures.
19. The method according to claim 17, characterized in that The RVCB structure includes: A plurality of first feature extraction layers connected in parallel, wherein the first feature extraction layers include convolution layers, convolution kernels of convolution layers in different first feature extraction layers among the plurality of first feature extraction layers are different, and the number of channels of the convolution layers in the first feature extraction layers is the ratio of the number of channels of feature information input to the first feature extraction layer to the number of the at least one first feature extraction layer; A first splicing layer, used for performing feature splicing on the feature information output by the plurality of first feature extraction layers; A first convolutional layer, used for extracting features from feature information output by the first concatenation layer; The skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.
20. The method according to claim 17, characterized in that The DenseNet structure includes: A plurality of second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers; A plurality of second splicing layers, wherein two adjacent second feature extraction layers in the plurality of second feature extraction layers are connected through the second splicing layer, and any second splicing layer in the plurality of second splicing layers is used to: perform feature splicing on feature information input to the DenseNet structure and feature information output by the second feature extraction layer before any second splicing layer; A second convolutional layer, wherein the last second feature extraction layer among the multiple second feature extraction layers is connected to the second convolutional layer through the last second splicing layer among the multiple second splicing layers, and the second convolutional layer is used to: convert the feature information output by the last second splicing layer into feature information having the same number of channels as the number of channels of the feature information input for the DenseNet structure.
21. The method according to any one of claims 1 to 20, characterized in that The motion parameter of the current block includes at least one of the following: a motion vector of the current block and an index of the reference image.
22. A coding method, characterized in that: include: Performing motion estimation on the current block to obtain a reference block of the current block; Performing quality enhancement on the reference block to obtain an enhanced block; Based on the enhancement block, a prediction block of the current block is determined.
23. The method according to claim 22, characterized in that The step of performing quality enhancement on the reference block to obtain an enhanced block comprises: Performing quality enhancement on a reference area including the reference block in the reference image to obtain an enhanced image; The enhanced block is determined in the enhanced image based on the position of the reference block in the reference area.
24. The method according to claim 23, characterized in that The determining the enhanced block in the enhanced image based on the position of the reference block in the reference area comprises: A block in the enhanced image whose position is the same as that of the reference block in the reference area is determined as the enhanced block.
25. The method according to claim 24, characterized in that The step of determining a prediction block of the current block based on the enhancement block comprises: Boundary extension and interpolation filtering are performed on the enhanced block to obtain the predicted block.
26. The method according to claim 23, characterized in that The performing motion estimation on the current block to obtain a reference block of the current block includes: Dividing the current block into at least one sub-block; Performing motion estimation on the current block to determine motion parameters of the current block; Determining a motion parameter of the at least one sub-block based on a motion parameter of the current block; Determining at least one reference sub-block included in the reference block based on the motion parameter of the at least one sub-block; The step of determining the enhanced block in the enhanced image based on the position of the reference block in the reference area includes: Determine, based on a position of the at least one reference sub-block in the reference region, a first region where the at least one reference sub-block is located; Expanding the boundary of the first region to obtain a second region; A block in the enhanced image whose position corresponds to the second region is determined as the enhanced block.
27. The method according to claim 26, characterized in that The step of determining a prediction block of the current block based on the enhancement block comprises: Determine a subblock in the enhanced block having the same position as the at least one reference subblock in the reference region as at least one enhanced subblock included in the enhanced block; Boundary extension and interpolation filtering are performed on the at least one enhanced sub-block to obtain at least one predicted sub-block included in the predicted block.
28. The method according to claim 26, characterized in that The first region is a minimum region including the at least one reference sub-block.
29. The method according to any one of claims 23 to 28, characterized in that The reference area is the entire area or a partial area of the reference image.
30. The method according to claim 29, characterized in that The partial area is an area where any one of the following items is located: an image block, a sub-image, a rectangular area, and a slice.
31. The method according to any one of claims 23 to 30, characterized in that The step of performing quality enhancement on a reference area including the reference block in the reference image to obtain an enhanced image comprises: Performing feature extraction on the reference area to obtain a residual image; The reference region and the residual image are weighted to obtain the enhanced image.
32. The method according to claim 31, characterized in that The step of extracting features from the reference area to obtain a residual image includes: Determining, based on the size of the current block, a first parameter for extracting features from the reference area; Based on the first parameter, feature extraction is performed on the reference area to obtain the residual image.
33. The method according to claim 32, characterized in that The determining, based on the size of the current block, a first parameter for extracting features from the reference area comprises: A parameter corresponding to the minimum side length of the current block is determined as the first parameter.
34. The method according to any one of claims 31 to 33, characterized in that The step of extracting features from the reference area to obtain a residual image includes: Extracting features from the reference area to obtain first feature information; Performing multi-scale feature extraction on the first feature information and splicing the extracted multi-scale feature information to obtain second feature information; determining third feature information based on the second feature information and feature information obtained by extracting features from the second feature information; Performing multi-scale feature extraction on the third feature information and splicing the extracted multi-scale feature information to obtain fourth feature information; The fourth feature information is converted into feature information having the same number of channels as the number of channels of the reference area to obtain the residual image.
35. The method according to any one of claims 23 to 30, characterized in that The step of performing quality enhancement on a reference area including the reference block in the reference image to obtain an enhanced image comprises: The reference area is quality enhanced by using a dense residual multi-scale convolutional network (Dense-RVCNN) to obtain the enhanced image.
36. The method according to claim 35, characterized in that The method of using a dense residual multi-scale convolutional network (Dense-RVCNN) to enhance the quality of the reference area to obtain the enhanced image includes: Based on the size of the current block, determine the network parameters used by Dense-RVCNN; The enhanced image is determined based on the network parameters used by the Dense-RVCNN.
37. The method according to claim 36, characterized in that Determining the network parameters used by Dense-RVCNN based on the size of the current block includes: The parameters corresponding to the minimum side length of the current block are determined as the network parameters used by the Dense-RVCNN.
38. The method according to any one of claims 35 to 37, characterized in that The Dense-RVCNN includes at least one of the following: An input layer, used for extracting features from the reference area; At least one residual multi-scale convolution RVCB structure, used to extract and concatenate features at multiple scales on input feature information; Dense network DenseNet structure is used to extract features from input feature information; The output layer is used to convert the input features into feature information having the same number of channels as the number of channels of the reference region and to weight the reference region and the converted feature information.
39. The method according to claim 38, characterized in that The at least one RVCB structure is a plurality of RVCB structures, the input layer is connected to the input end of the DenseNet structure through some of the RVCB structures among the plurality of RVCB structures, and the output end of the DenseNet structure is connected to the output layer through another portion of the RVCB structures among the plurality of RVCB structures.
40. The method according to claim 38, characterized in that The RVCB structure includes: A plurality of first feature extraction layers connected in parallel, wherein the first feature extraction layers include convolution layers, convolution kernels of convolution layers in different first feature extraction layers among the plurality of first feature extraction layers are different, and the number of channels of the convolution layers in the first feature extraction layers is the ratio of the number of channels of feature information input to the first feature extraction layer to the number of the at least one first feature extraction layer; A first splicing layer, used for performing feature splicing on the feature information output by the plurality of first feature extraction layers; A first convolutional layer, used for extracting features from feature information output by the first concatenation layer; The skip connection layer is used to weight the feature information input to the RVCB structure and the feature information output by the first convolutional layer.
41. The method according to claim 38, characterized in that The DenseNet structure includes: A plurality of second feature extraction layers connected in series, wherein the second feature extraction layers include convolutional layers; A plurality of second splicing layers, wherein two adjacent second feature extraction layers in the plurality of second feature extraction layers are connected through the second splicing layer, and any second splicing layer in the plurality of second splicing layers is used to: perform feature splicing on feature information input to the DenseNet structure and feature information output by the second feature extraction layer before any second splicing layer; A second convolutional layer, wherein the last second feature extraction layer among the multiple second feature extraction layers is connected to the second convolutional layer through the last second splicing layer among the multiple second splicing layers, and the second convolutional layer is used to: convert the feature information output by the last second splicing layer into feature information having the same number of channels as the number of channels of the feature information input for the DenseNet structure.
42. The method according to any one of claims 22 to 41, characterized in that The method further comprises: encoding motion parameters of the current block; The motion parameter of the current block includes at least one of the following: a motion vector of the current block and an index of the reference image.
43. A neural network training method, characterized in that: include: Encode the sample video to obtain a bit stream; Decoding the code stream to determine a prediction block of a current block; The neural network is trained based on the label of the current block and the predicted block.
44. The method according to claim 43, characterized in that The decoding of the code stream to determine a prediction block of the current block includes: determining prediction blocks of multiple sizes; The step of training the neural network based on the label of the current block and the predicted block includes: Based on the label of the current block and the predicted blocks of the multiple sizes, the neural network is trained to obtain network parameters corresponding to the multiple sizes.
45. The method according to claim 44, characterized in that The prediction blocks of multiple sizes include a plurality of prediction blocks having different minimum side lengths.
46. A decoder, characterized in that include: A first determining unit, configured to decode a bitstream and determine a motion parameter of a current block; A second determining unit, configured to determine a reference block of the current block in a reference image of the current block based on a motion parameter of the current block; an enhancement unit, configured to perform quality enhancement on the reference block to obtain an enhanced block; The third determining unit is configured to determine a prediction block of the current block based on the enhanced block.
47. An encoder, characterized in that include: A first determining unit, configured to perform motion estimation on a current block to obtain a reference block of the current block; an enhancement unit, configured to perform quality enhancement on the reference block to obtain an enhanced block; The second determining unit is configured to determine a prediction block of the current block based on the enhanced block.
48. A neural network training device, characterized in that: include: The encoding unit is used to encode the sample video to obtain a bit stream; A decoding unit, used for decoding the bit stream to determine a prediction block of a current block; A training unit is used to train the neural network based on the label of the current block and the predicted block.
49. An electronic device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, at least one of the following methods is performed: the method according to any one of claims 1 to 21, the method according to any one of claims 22 to 42, and the method according to any one of claims 43 to 45.
50. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed on a computer, causes the computer to perform at least one of the following methods: the method according to any one of claims 1 to 21, the method according to any one of claims 22 to 42, or the method according to any one of claims 43 to 45.
51. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, at least one of the following methods is implemented: the method according to any one of claims 1 to 21, the method according to any one of claims 22 to 42, and the method according to any one of claims 43 to 45.
52. A code stream, characterized in that The code stream is a code stream decoded according to the method according to any one of claims 1 to 21, or the code stream is a code stream generated by the method according to any one of claims 22 to 42.
Citation Information
Patent Citations
Video frame prediction method and device, and automatic compensation neural network
CN108289224A
Inter-frame prediction method based on deep neural network in hybrid video coding and decoding system
CN108833925A
Image encoder using machine learning and data processing method thereof
CN109688406A
Reference frame selection method and device for video coding
CN110062239A
Motion compensation using machine learning
US20180124418A1
