Image processing method, device, equipment, system and storage medium

CN118901239APending Publication Date: 2024-11-05GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202280093978.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-03-28
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

Existing image processing methods are not ideal in reducing video distortion, resulting in insufficient quality of reconstructed images.

Method used

Image enhancement is performed by inputting reconstructed image patches and quantization parameters into the enhancement model. The quantization parameters are then used to enhance the quality of the reconstructed image patches, thereby improving the enhancement effect of the image patches.

Benefits of technology

It improves the quality of reconstructed image patches, provides more accurate reference patches for intra-frame prediction, and enhances the accuracy of intra-frame prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118901239A_ABST
    Figure CN118901239A_ABST
Patent Text Reader

Abstract

The invention provides an image processing method, apparatus, device and system, and a storage medium, and performs quality enhancement on a reconstructed image block based on a quantization parameter to obtain an enhanced image block. The quantization coefficients corresponding to different image blocks may be different, so that in order to improve the enhancement accuracy of the image blocks, the quality of the reconstructed image blocks is enhanced based on the quantization coefficients, and the enhancement effect can be improved. Besides, image quality enhancement is carried out by taking the image block as a unit, so that when the enhanced image block is used as a reference block of other image blocks in the intra-frame prediction, more accurate reference can be provided, and the accuracy of the intra-frame prediction can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Image processing method, device, equipment, system, and storage medium Technical Field

[0001] The present application relates to the field of video coding and decoding technology, and in particular to an image processing method, apparatus, device, system, and storage medium. Background Art

[0002] Digital video technology can be incorporated into a variety of video devices, such as digital televisions, smartphones, computers, e-readers, or video players. With the development of video technology, the amount of video data included has increased. To facilitate the transmission of video data, video devices implement video compression technology to enable more efficient transmission or storage of video data.

[0003] Video compression will cause video distortion. In order to reduce video distortion, the reconstructed image needs to be processed. However, the current image processing methods have unsatisfactory processing effects.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device, system, and storage medium, which perform image enhancement by inputting a reconstructed image block and a quantization parameter into an enhancement model to obtain an enhanced image block, thereby improving the quality enhancement effect of the reconstructed image block.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0007] Decode the code stream to obtain the quantization coefficient of the current image block;

[0008] Determining a quantization parameter corresponding to a current image block, and performing inverse quantization on the quantization coefficient based on the quantization parameter to obtain a transform coefficient of the current image block;

[0009] determining a reconstructed image block of the current image block according to the transformation coefficient;

[0010] Based on the quantization parameter, quality enhancement is performed on the reconstructed image block to obtain an enhanced image block.

[0011] In a second aspect, an embodiment of the present application provides an image processing method, comprising:

[0012] Determining a quantization parameter of a current image block, and encoding the current image block based on the quantization parameter to obtain a quantization coefficient of the current image block;

[0013] Dequantizing the quantization coefficients based on the quantization parameters of the current image block to obtain a residual block of the current image block;

[0014] Obtaining a reconstructed image block of the current image block according to the residual block;

[0015] Based on the quantization parameter, quality enhancement is performed on the reconstructed image block to obtain an enhanced image block.

[0016] In a third aspect, the present application provides an image processing apparatus for executing the method of the first aspect or its respective implementations. Specifically, the encoder includes a functional unit for executing the method of the first aspect or its respective implementations.

[0017] In a fourth aspect, the present application provides an image processing apparatus for executing the method of the second aspect or its respective implementations. Specifically, the decoder includes a functional unit for executing the method of the second aspect or its respective implementations.

[0018] In a fifth aspect, a video encoder is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to execute the method of the first aspect or its respective implementations.

[0019] In a sixth aspect, a video decoder is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to perform the method of the second aspect or its respective implementations.

[0020] In a seventh aspect, a video encoding and decoding system is provided, comprising a video encoder and a video decoder. The video encoder is configured to execute the method of the first aspect or its respective implementations, and the video decoder is configured to execute the method of the second aspect or its respective implementations.

[0021] In an eighth aspect, a chip is provided for implementing the method described in any one of the first and second aspects above, or their respective implementations. Specifically, the chip includes a processor configured to load and execute a computer program from a memory, causing a device equipped with the chip to perform the method described in any one of the first and second aspects above, or their respective implementations.

[0022] In a ninth aspect, a computer-readable storage medium is provided for storing a computer program, which enables a computer to execute the method of any one of the first to second aspects or their respective implementations.

[0023] In a tenth aspect, a computer program product is provided, comprising computer program instructions, which enable a computer to execute the method of any one of the first to second aspects or their respective implementations.

[0024] In an eleventh aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method in any one of the first to second aspects or their respective implementations.

[0025] In a twelfth aspect, a code stream is provided, which is generated by any aspect or its implementation methods of the above-mentioned first aspect.

[0026] Based on the above technical solution, image enhancement is performed on the reconstructed image block based on quantization parameters to obtain an enhanced image block. Since different image blocks may correspond to different quantization coefficients, in order to improve the accuracy of image block enhancement, the present application performs quality enhancement on the reconstructed image block of the current image block based on the quantization coefficient, which can improve the enhancement effect. In addition, the present application performs image quality enhancement on an image block basis. In this way, when the enhanced image block is used as a reference block for other image blocks in intra-frame prediction, it can provide a more accurate reference, thereby improving the accuracy of intra-frame prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] FIG1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present application;

[0028] FIG2 is a schematic block diagram of a video encoder according to an embodiment of the present application;

[0029] FIG3 is a schematic block diagram of a video decoder according to an embodiment of the present application;

[0030] FIG4 is a flow chart of an image processing method according to an embodiment of the present application;

[0031] FIG5 is a schematic diagram of image processing involved in this application;

[0032] FIG6 is a schematic diagram of an enhancement model involved in this application;

[0033] FIG7 is a schematic diagram of an enhancement model involved in this application;

[0034] FIG8 is a schematic structural diagram of the first feature extraction layer involved in this application;

[0035] FIG9 is another structural diagram of the first feature extraction layer involved in this application;

[0036] FIG10 is another structural diagram of the first feature extraction layer involved in this application;

[0037] FIG11 is another structural diagram of the first feature extraction layer involved in this application;

[0038] FIG12 is a schematic diagram of a structure of a weighted processing layer involved in this application;

[0039] FIG13 is a schematic structural diagram of an enhancement model involved in this application;

[0040] FIG14 is another structural diagram of the enhancement model involved in this application;

[0041] FIG15 is another structural diagram of the enhancement model involved in this application;

[0042] FIG16 is another structural diagram of the enhancement model involved in this application;

[0043] FIG17 is another structural diagram of the enhancement model involved in this application;

[0044] FIG18 is another structural diagram of the enhancement model involved in this application;

[0045] FIG19 is a flow chart of an image processing method according to an embodiment of the present application;

[0046] FIG20 is a flow chart of an image processing method according to an embodiment of the present application;

[0047] FIG21 is a flow chart of an image processing method according to an embodiment of the present application;

[0048] FIG22 is a schematic block diagram of an image processing device provided in an embodiment of the present application;

[0049] FIG23 is a schematic block diagram of an image processing device provided in an embodiment of the present application;

[0050] FIG24 is a schematic block diagram of an electronic device provided in an embodiment of the present application;

[0051] Figure 25 is a schematic block diagram of a video encoding and decoding system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The present application can be applied to the field of image coding and decoding, the field of video coding and decoding, the field of hardware video coding and decoding, the field of dedicated circuit video coding and decoding, the field of real-time video coding and decoding, etc. For example, the solution of the present application can be combined with an audio and video coding standard (AVS), such as the H.264 / audio and video coding (AVC) standard, the H.265 / high efficiency video coding (HEVC) standard, and the H.266 / versatile video coding (VVC) standard. Alternatively, the solution of the present application can be combined with other proprietary or industry standards and operated, and the standards include ITU-TH.261, ISO / IEC MPEG-1 Visual, ITU-TH.262 or ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC), including scalable video coding (SVC) and multi-view video coding (MVC) extensions. It should be understood that the technology of this application is not limited to any specific coding standard or technology.

[0053] For ease of understanding, the video encoding and decoding system involved in the embodiment of the present application is first introduced with reference to FIG1 .

[0054] FIG1 is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application. It should be noted that FIG1 is only an example, and the video encoding and decoding system of the embodiment of the present application includes but is not limited to that shown in FIG1. ​​As shown in FIG1, the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device is used to encode (which can be understood as compressing) the video data to generate a code stream, and transmit the code stream to the decoding device. The decoding device decodes the code stream generated by the encoding device to obtain decoded video data.

[0055] The encoding device 110 of the embodiment of the present application can be understood as a device with a video encoding function, and the decoding device 120 can be understood as a device with a video decoding function, that is, the embodiment of the present application includes a wider range of devices for the encoding device 110 and the decoding device 120, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, car computers, etc.

[0056] In some embodiments, the encoding device 110 may transmit the encoded video data (eg, a code stream) to the decoding device 120 via a channel 130. The channel 130 may include one or more media and / or devices capable of transmitting the encoded video data from the encoding device 110 to the decoding device 120.

[0057] In one example, the channel 130 includes one or more communication media that enable the encoding device 110 to transmit the encoded video data directly to the decoding device 120 in real time. In this example, the encoding device 110 can modulate the encoded video data according to a communication standard and transmit the modulated video data to the decoding device 120. The communication media includes wireless communication media, such as radio frequency spectrum. Optionally, the communication media may also include wired communication media, such as one or more physical transmission lines.

[0058] In another example, channel 130 includes a storage medium that can store the video data encoded by encoding device 110. The storage medium includes various locally accessible data storage media, such as optical disks, DVDs, and flash memories. In this example, decoding device 120 can retrieve the encoded video data from the storage medium.

[0059] In another example, the channel 130 may include a storage server that can store the video data encoded by the encoding device 110. In this example, the decoding device 120 can download the stored encoded video data from the storage server. Alternatively, the storage server can store the encoded video data and transmit the encoded video data to the decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0060] In some embodiments, the encoding device 110 includes a video encoder 112 and an output interface 113. The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0061] In some embodiments, the encoding device 110 may further include a video source 111 in addition to the video encoder 112 and the input interface 113 .

[0062] The video source 111 may include at least one of a video acquisition device (eg, a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data.

[0063] The video encoder 112 encodes the video data from the video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains the coding information of the picture or picture sequence in the form of a bitstream. The coding information may include the coded picture data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. The SPS may contain parameters that apply to one or more sequences. The PPS may contain parameters that apply to one or more pictures. The syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the bitstream.

[0064] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored in a storage medium or a storage server for subsequent reading by the decoding device 120.

[0065] In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122 .

[0066] In some embodiments, the decoding device 120 may further include a display device 123 in addition to the input interface 121 and the video decoder 122 .

[0067] The input interface 121 includes a receiver and / or a modem and can receive the encoded video data via the channel 130 .

[0068] The video decoder 122 is configured to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to the display device 123 .

[0069] The decoded video data is displayed on the display device 123. The display device 123 may be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 may include various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0070] In addition, Figure 1 is only an example, and the technical solution of the embodiment of the present application is not limited to Figure 1. For example, the technology of the present application can also be applied to unilateral video encoding or unilateral video decoding.

[0071] The following is an introduction to the video encoding framework involved in the embodiments of the present application.

[0072] FIG2 is a schematic block diagram of a video encoder according to an embodiment of the present application. It should be understood that the video encoder 200 can be used to perform lossy compression or lossless compression on an image. The lossless compression can be visually lossless or mathematically lossless.

[0073] The video encoder 200 can be applied to image data in a luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in terms of color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 means full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0074] For example, the video encoder 200 reads video data, and for each frame of the video data, divides the frame into a number of coding tree units (CTUs). In some examples, CTB may be referred to as a "tree block", "largest coding unit" (LCU) or "coding tree block" (CTB). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to a luminance (luminance or luma) sample and two chrominance (chroma) samples. Therefore, each CTU may be associated with a luminance sample block and two chroma sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU can be further divided into a number of coding units (CUs) for encoding. The CU may be a rectangular block or a square block. A CU can be further divided into prediction units (PUs) and transform units (TUs), allowing for separation of coding, prediction, and transform, and greater flexibility in processing. In one example, a CTU is divided into CUs using a quadtree, and a CU is divided into TUs and PUs using a quadtree.

[0075] The video encoder and video decoder can support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and video decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0076] In some embodiments, as shown in FIG2 , the video encoder 200 may include a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filter unit 260, a decoded image buffer 270, and an entropy coding unit 280. It should be noted that the video encoder 200 may include more, fewer, or different functional components.

[0077] Optionally, in this application, the current block may be referred to as the current coding unit (CU) or the current prediction unit (PU), etc. The prediction block may also be referred to as a predicted image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstructed block or an image reconstructed image block.

[0078] In some embodiments, the prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame estimation unit 212. Because there is a strong correlation between adjacent pixels in a video frame, intra-frame prediction is used in video coding and decoding technologies to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent frames in a video, inter-frame prediction is used in video coding and decoding technologies to eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.

[0079] The inter-frame prediction unit 211 is used for inter-frame prediction. Inter-frame prediction includes motion estimation and motion compensation. It can refer to image information from different frames. Inter-frame prediction uses motion information to find a reference block from a reference frame and generates a prediction block based on the reference block to eliminate temporal redundancy. The frames used for inter-frame prediction can be P-frames and / or B-frames. P-frames refer to forward-predicted frames, while B-frames refer to bidirectionally predicted frames. Inter-frame prediction uses motion information to find a reference block from a reference frame and generates a prediction block based on the reference block. Motion information includes the reference frame list in which the reference frame is located, the reference frame index, and the motion vector. The motion vector can be integer pixel or fractional pixel. If the motion vector is fractional pixel, interpolation filtering is required to generate the required fractional pixel block in the reference frame. Here, the integer pixel or fractional pixel block in the reference frame found based on the motion vector is called a reference block. Some technologies directly use the reference block as the prediction block, while others further process the reference block to generate a prediction block. Reprocessing the reference block to generate a prediction block can also be understood as using the reference block as the prediction block and then processing the prediction block to generate a new prediction block.

[0080] The intra-frame estimation unit 212 only refers to the information of the same frame image to predict the pixel information in the current code image block to eliminate spatial redundancy. The frame used for intra-frame prediction can be an I frame.

[0081] Intra-frame prediction has multiple prediction modes. For example, the H-series international digital video coding standard H.264 / AVC has eight angular prediction modes and one non-angular prediction mode. H.265 / HEVC expands this to 33 angular prediction modes and two non-angular prediction modes. HEVC uses planar, DC, and 33 angular modes for a total of 35 intra-frame prediction modes. VVC uses planar, DC, and 65 angular modes for a total of 67 intra-frame prediction modes.

[0082] It should be noted that with the increase of angle modes, intra-frame prediction will be more accurate and more in line with the needs of high-definition and ultra-high-definition digital video development.

[0083] The residual unit 220 may generate a residual block for the CU based on the pixel block of the CU and the prediction block of the PU. For example, the residual unit 220 may generate the residual block for the CU so that each sample in the residual block has a value equal to the difference between the sample in the pixel block of the CU and the corresponding sample in the prediction block of the PU of the CU.

[0084] The transform / quantization unit 230 may quantize the transform coefficients. The transform / quantization unit 230 may quantize the transform coefficients associated with the TUs of a CU based on a quantization parameter (QP) value associated with the CU. The video encoder 200 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0085] The inverse transform / quantization unit 240 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.

[0086] Reconstruction unit 250 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of a CU in this manner, video encoder 200 can reconstruct the pixel blocks of the CU.

[0087] The loop filter unit 260 is used to process the inverse transformed and inverse quantized pixels to compensate for distortion information and provide a better reference for subsequent coded pixels. For example, it can perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with the CU.

[0088] In some embodiments, the loop filtering unit 260 includes a deblocking filtering unit and a sample adaptive offset / adaptive loop filtering (SAO / ALF) unit, wherein the deblocking filtering unit is used to remove blocking effects, and the SAO / ALF unit is used to remove ringing effects.

[0089] The decoded image buffer 270 may store the reconstructed pixel blocks. The inter prediction unit 211 may use a reference image containing the reconstructed pixel blocks to perform inter prediction on PUs of other images. In addition, the intra estimation unit 212 may use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra prediction on other PUs in the same image as the CU.

[0090] The entropy coding unit 280 may receive the quantized transform coefficients from the transform / quantization unit 230. The entropy coding unit 280 may perform one or more entropy coding operations on the quantized transform coefficients to generate entropy-coded data.

[0091] FIG3 is a schematic block diagram of a video decoder according to an embodiment of the present application.

[0092] 3 , the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transformation unit 330, a reconstruction unit 340, a loop filter unit 350, and a decoded picture buffer 360. It should be noted that the video decoder 300 may include more, fewer, or different functional components.

[0093] The video decoder 300 may receive a bitstream. The entropy decoding unit 310 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding unit 310 may parse the entropy-encoded syntax elements in the bitstream. The prediction unit 320, the inverse quantization / transform unit 330, the reconstruction unit 340, and the loop filter unit 350 may decode the video data based on the syntax elements extracted from the bitstream, thereby generating decoded video data.

[0094] In some embodiments, the prediction unit 320 includes an inter-frame prediction unit 321 and an intra-frame estimation unit 322 .

[0095] Intra estimation unit 322 may perform intra prediction to generate a prediction block for a PU. Intra estimation unit 322 may use an intra prediction mode to generate a prediction block for the PU based on pixel blocks of spatially neighboring PUs. Intra estimation unit 322 may also determine the intra prediction mode for the PU based on one or more syntax elements parsed from a codestream.

[0096] The inter-frame prediction unit 321 may construct a first reference picture list (List 0) and a second reference picture list (List 1) based on syntax elements parsed from the codestream. In addition, if a PU is encoded using inter-frame prediction, the entropy decoding unit 310 may parse the motion information of the PU. The inter-frame prediction unit 321 may determine one or more reference blocks for the PU based on the motion information of the PU. The inter-frame prediction unit 321 may generate a prediction block for the PU based on the one or more reference blocks of the PU.

[0097] The inverse quantization / transform unit 330 may inversely quantize (ie, dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 330 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0098] After inverse quantizing the transform coefficients, the inverse quantization / transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.

[0099] The reconstruction unit 340 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0100] The loop filtering unit 350 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with a CU.

[0101] The video decoder 300 may store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 may use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0102] The basic process of video encoding and decoding is as follows: At the encoder end, a frame of image is divided into CUs. For the current block (i.e., the current CU), the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual unit 220 calculates a residual block based on the predicted block and the original block of the current block. This residual block is the difference between the predicted block and the original block of the current block. This residual block can also be referred to as residual information. This residual block undergoes transformation and quantization by the transform / quantization unit 230, removing information that is insensitive to the human eye and eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization unit 230 can be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization unit 230 can be referred to as a frequency residual block or a frequency-domain residual block. The entropy coding unit 280 receives the quantized change coefficients output by the change quantization unit 230 and performs entropy coding on these quantized change coefficients to output a bitstream. For example, the entropy coding unit 280 can eliminate character redundancy based on the target context model and probability information of the binary bitstream.

[0103] At the decoding end, the entropy decoding unit 310 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction for the current block based on the prediction information to generate a prediction block for the current block. The inverse quantization / transformation unit 330 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be used as a reference frame for inter-frame prediction for subsequent frames.

[0104] It should be noted that the block division information determined by the encoder, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and analyzes the existing information to determine the same block division information, prediction, transform, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical.

[0105] The current block may be a current coding unit (CU) or a current prediction unit (PU), etc.

[0106] The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. This application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0107] The following describes the image encoding and decoding method provided in the embodiments of the present application in conjunction with specific embodiments.

[0108] First, with reference to FIG4 , the video decoding method provided in the embodiment of the present application is introduced by taking the decoding end as an example.

[0109] FIG4 is a flow chart of a video decoding method provided by an embodiment of the present application, which is applied to the video decoders shown in FIG1 and FIG3 .

[0110] As shown in FIG4 , the method of the embodiment of the present application includes:

[0111] S401: Decode the code stream to obtain the quantization coefficient of the current image block.

[0112] The embodiment of the present application does not limit the specific size of the current image block.

[0113] In some embodiments, the current image block of the embodiment of the present application is a CTU. For example, a frame of image is divided into several CTUs, and the present application does not limit the size of the CTU. For example, the size of a CTU is 128×128, 64×64, 32×32, etc.

[0114] In some embodiments, the current image block of the embodiments of the present application is a CU, for example, a CTU is divided into one or more CUs.

[0115] In some embodiments, the current image block of the embodiments of the present application is a TU or a PU, for example, a CU is divided into one or more TUs or PUs.

[0116] In some embodiments, the current image block of the embodiment of the present application only includes a chrominance component and can be understood as a chrominance block.

[0117] In some embodiments, the current image block of the embodiment of the present application only includes a brightness component and can be understood as a brightness block.

[0118] In some embodiments, the current image block includes both a luminance component and a chrominance component.

[0119] It should be noted that, if the current image block includes multiple CUs, the quantization coefficients of the current image block include the quantization coefficients corresponding to the multiple CUs.

[0120] As shown in Figure 3, at the decoding end, the entropy decoding unit 310 decodes the bitstream to obtain prediction information, quantization information, and other information for the current image block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction based on the prediction information to generate a predicted value for the current image block. The inverse quantization / transformation unit 330 uses the quantization coefficient matrix obtained from the bitstream to inversely quantize and inversely transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed image block for the current image block.

[0121] Video distortion may occur during the video encoding process. In order to reduce the distortion, the embodiment of the present application performs post-processing on the reconstructed image block, that is, performing image quality enhancement using the image block as the enhancement unit to improve the quality of the reconstructed image block.

[0122] It should be noted that, if the current image block includes multiple CUs, each CU may be decoded separately to obtain a reconstructed block of each CU, and the reconstructed blocks of each CU may be combined to obtain a reconstructed image block of the current image block.

[0123] S402 : Determine a quantization parameter corresponding to the current image block, and dequantize the quantization coefficient based on the quantization parameter to obtain a transform coefficient of the current image block.

[0124] It should be noted that, if the current image block includes multiple CUs, the quantization parameter of the current image block includes the quantization parameters corresponding to the multiple CUs. Optionally, the quantization parameters corresponding to the multiple CUs may be the same or different.

[0125] In one possible implementation, the quantization parameter of the current image block in the embodiment of the present application can be expressed in the form of a matrix. For example, if the size of the current image block is 16X16, the quantization parameter of the current image block is a 16X16 matrix, and each element in the matrix is ​​the quantization parameter of the pixel point at the corresponding position in the current image block.

[0126] The embodiment of the present application does not limit the specific process of determining the quantization parameter corresponding to the current image block.

[0127] In some embodiments, the codec uses a default quantization parameter as the quantization parameter corresponding to the current image block. In this case, the decoding end can directly determine the default quantization coefficient as the quantization parameter corresponding to the current image block.

[0128] In some embodiments, the encoder writes the quantization parameter corresponding to the current image block determined during the encoding process into the bitstream, so that the decoder can determine the quantization parameter corresponding to the current image block by decoding the bitstream.

[0129] In some embodiments, the decoding end may use the same calculation method as the encoding end to determine the quantization parameter corresponding to the current image block through calculation.

[0130] When the decoding end determines the quantization parameter corresponding to the current image block, it uses the quantization parameter to dequantize the quantization coefficient of the current image block to obtain the transformation coefficient of the current image block.

[0131] For example, if the current image block includes multiple CUs, for each CU, the quantization coefficient of the CU is dequantized using the quantization parameter corresponding to the CU to obtain the transform coefficient of the CU.

[0132] S403: Determine a reconstructed image block of the current image block according to the transformation coefficient.

[0133] Specifically, the transform coefficients of the current image block are inversely transformed to obtain the residual block of the current image block. At the same time, a prediction method such as intra-frame or inter-frame prediction is used to obtain the prediction block of the current image block. The prediction value of the current image block is added to the residual block to obtain the reconstructed image block of the current image block.

[0134] It should be noted that if the current image block includes multiple CUs, the quantization parameters corresponding to the current image block are the quantization parameters of these multiple CUs. During decoding, each CU in the current image block is decoded separately to obtain the reconstructed block of each CU, and the reconstructed blocks of each CU are combined to obtain the reconstructed image block of the current image block.

[0135] S404 : Based on the quantization parameter, perform quality enhancement on the reconstructed image block to obtain an enhanced image block.

[0136] During the video encoding process, different image blocks may have different corresponding quantization parameters QP. In some embodiments, the quantization parameter QP includes a quantization step size. During the video encoding process, the transform coefficients of the image block are quantized, wherein the larger the quantization step size, the greater the image loss, and the smaller the quantization step size, the smaller the image loss. Therefore, in order to improve the enhancement effect of the current image block, the embodiment of the present application considers the impact of the quantization parameter QP corresponding to the current image block on the quality enhancement during the quality enhancement process of the current image block, thereby improving the quality enhancement effect of the current image block.

[0137] Since different quantization coefficients result in different losses when inverse quantizing an image block, in order to improve the enhancement effect of the image block, the embodiment of the present application performs quality enhancement on the reconstructed image block of the current image block based on the quantization coefficient corresponding to the current image block, thereby improving the enhancement effect of the reconstructed image block.

[0138] In the embodiment of the present application, image quality enhancement is performed in units of image blocks. In this way, when the enhanced image blocks are used as reference blocks for other image blocks in intra-frame prediction, more accurate references can be provided, thereby improving the accuracy of intra-frame prediction.

[0139] In addition, the embodiment of the present application performs image quality enhancement in units of image blocks. Compared with image quality enhancement of the entire frame image, it can focus more on enhancing finer features in the image blocks, further improving the enhancement quality of the image blocks.

[0140] In the embodiment of the present application, there is no limitation on the method of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain the enhanced image block.

[0141] In one possible implementation, different enhancement models are pre-trained for different quantization parameters. These enhancement models are used to enhance the quality of image blocks with those quantization parameters. In this way, the decoder can select a target enhancement model corresponding to the quantization parameter of the current image block from among multiple enhancement models corresponding to different quantization parameters, and use this target enhancement model to enhance the quality of the reconstructed image block of the current image block.

[0142] In another possible implementation, the decoding end obtains a universal enhancement model, which is trained based on different image blocks and their corresponding quantization parameters. The model fully learns the impact of different quantization parameters on the quality enhancement of image blocks, and can then perform high-quality enhancement on the reconstructed image blocks obtained after inverse quantization with different quantization parameters based on different quantization parameters. In this way, the decoding end can perform quality enhancement on the reconstructed image blocks based on the quantization parameters using the universal enhancement model to obtain enhanced image blocks. Specifically, after the decoding end determines the reconstructed image block and quantization coefficient of the current image block according to the above steps, in order to reduce the distortion of the reconstructed image block and improve the quality of the reconstructed image block, as shown in Figure 5, the reconstructed image block and the corresponding quantization coefficient are input into the pre-trained enhancement model to perform image enhancement, and finally obtain the enhanced image block of the current image block.

[0143] The following embodiments all take the general enhancement model as an example to introduce the quality enhancement of the reconstructed image block.

[0144] In some embodiments, the decoding end fuses the reconstructed image block and the quantization parameter and inputs the result into the enhancement model. The fusion method of the reconstructed image block and the quantization parameter includes at least the following examples:

[0145] In Example 1, assume that the size of the reconstructed image block is N1*N2, where N1 and N2 can be the same or different. After multiplying the reconstructed image block by the quantization parameter, the block is input into the enhancement model. Specifically, each pixel in the reconstructed image block is multiplied by the quantization parameter to obtain an N1*N2 matrix, which is then input into the enhancement model.

[0146] Example 2: After concatenating the reconstructed image and the quantization parameters, the image is fed into the enhancement model. Specifically, the quantization coefficients are set to a matrix of size N1*N2, and the reconstructed image of size N1*N2 and the quantization parameters of size N1*N2 are concatenated and fed into the enhancement model.

[0147] It should be noted that, in addition to the fusion methods shown in Examples 1 and 2 above, the decoding end may also adopt other fusion methods to fuse the reconstructed image and the corresponding quantization parameters and then input them into the enhancement model for quality enhancement.

[0148] In some embodiments, to prevent features with smaller absolute values ​​from being overwritten by features with larger absolute values, the decoder normalizes the reconstructed image block and quantization parameters before inputting them into the enhancement model, ensuring that all features are treated equally. Next, based on the normalized reconstructed image block and quantization parameters, an enhanced image block of the reconstructed image block is obtained. For example, the normalized reconstructed image block and quantization parameters are concatenated and then input into the enhancement model for quality enhancement to improve the quality enhancement effect.

[0149] The reconstructed image block in the embodiment of the present application is a reconstructed image block of the current image block under the first component.

[0150] The first component may be a luminance component or a chrominance component.

[0151] In some embodiments, the above S404 includes the following steps S404-A and S404-B:

[0152] S404-A, performing feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image;

[0153] S404-B. Determine an enhanced image block according to the first feature information.

[0154] The embodiment of the present application does not limit the specific implementation of S404-A.

[0155] In one possible implementation, feature information of the reconstructed image block is extracted based on the quantization parameter. For example, the quantization parameter and the reconstructed image block are input into a neural network layer to extract the feature information of the reconstructed image block. Then, the feature information is analyzed and different weights are assigned to different features. For example, a larger weight is assigned to an important feature in the feature information to highlight the influence of the feature, and a smaller weight is assigned to a relatively unimportant feature to weaken the influence of the feature. Then, the feature information of the reconstructed image block is weighted according to the weight corresponding to the feature to obtain the first feature information of the reconstructed image block.

[0156] In another possible implementation, as shown in Figure 6, the enhancement model of the embodiment of the present application includes a first feature extraction module, which is used to extract at least one feature of the input information (i.e., the reconstructed image block and the quantization parameter) and assign different weights to the at least one extracted feature to perform feature weighted processing.

[0157] As shown in Figure 6, the enhancement model of the embodiment of the present application includes a first feature extraction module, which is used to perform weighted processing on the extracted features. Specifically, the reconstructed image block and the quantization parameter (optionally, after being fused by the method of Example 1 or Example 2 above) are input into the first feature extraction module, and the first feature extraction module performs feature extraction to extract at least one feature, and assigns a larger weight to the important feature of the at least one feature to highlight the influence of the feature, and assigns a smaller weight to the relatively unimportant feature to weaken the influence of the feature, and then performs weighted processing according to the weight corresponding to the at least one feature to obtain the first feature information of the reconstructed image block. Finally, based on the first feature information of the reconstructed image block, the enhanced image block of the reconstructed image block is determined.

[0158] The embodiment of the present application can assign different weights to different features to highlight the influence of important features and weaken the influence of unimportant features, thereby further improving the quality enhancement effect of the reconstructed image block.

[0159] The embodiment of the present application does not limit the network model of the first feature extraction module. For example, the first feature extraction module includes multiple convolutional layers and an attention mechanism.

[0160] In another possible implementation, the above S404-A includes the following steps:

[0161] S404-A1, based on the quantization parameter, perform feature weighting processing on the i-1th feature information of the reconstructed image block to obtain the i-th feature information of the reconstructed image block, where i is a positive integer from 1 to N. Repeat the process to obtain the Nth feature information of the reconstructed image block. If i is 1, the i-1th feature information is the reconstructed image block.

[0162] S404-A2: Determine first feature information of the reconstructed image block according to the Nth feature information.

[0163] In this implementation, the decoding end performs N feature-weighted iterative processing on the reconstructed image block based on the quantization parameter to obtain the Nth feature information of the reconstructed image block. Specifically, based on the quantization parameter, the reconstructed image block is subjected to feature-weighted processing to obtain the first feature information of the reconstructed image block. Then, based on the quantization parameter, the first feature information is subjected to feature-weighted processing to obtain the second feature information of the reconstructed image block. The process is repeated and, based on the quantization parameter, the N-1th feature information is subjected to feature-weighted processing to obtain the Nth feature information of the reconstructed image block. It should be noted that the embodiment of the present application does not limit the specific method of feature weighting processing. For example, the quantization parameter and the i-1th feature information are input into a neural network with a feature weighting processing function to obtain the i-th feature information of the reconstructed image block.

[0164] In some embodiments, as shown in FIG7 , the first feature extraction module of the embodiment of the present application includes N first feature extraction units. If N is greater than 1, these N first feature extraction units are connected in series, that is, the output of the previous first feature extraction unit serves as the input of the next first feature extraction unit. Based on the quantization parameter, these N first feature extraction units perform feature weighting processing on the reconstructed image block to obtain the Nth feature information of the reconstructed image block. As shown in FIG7 , each of the N first feature extraction units is configured to extract at least one feature and assign a larger weight to important features in the at least one feature to highlight the influence of the feature, and assign a smaller weight to relatively unimportant features to weaken the influence of the feature. The subsequent first feature extraction unit further extracts and assigns weights to the feature information output by the previous first feature extraction unit to further highlight important features. After processing by the N first feature extraction units, important features of the reconstructed image block can be primarily enhanced, while less important features in the reconstructed image block can be less strongly enhanced, thereby improving the enhancement effect of the reconstructed image block.

[0165] In one example, if N is 1, i.e., the first feature extraction module includes a first feature extraction unit, then S404-A1 includes: the decoder fuses the reconstructed image and the quantization coefficients, and then inputs the fused image into the first feature extraction unit. The first feature extraction unit performs feature extraction to extract at least one feature, assigns different weights to the features based on their importance, and then performs weighted processing on the at least one feature based on the different weights to obtain first feature information. Finally, first feature information of the reconstructed image block is determined based on the first feature information, for example, the first feature information is determined as the first feature information of the reconstructed image block.

[0166] In another example, if N is greater than 1, that is, the first feature extraction module includes multiple first feature extraction units, then the above S404-A1 includes: the decoding end fuses the reconstructed image and the quantization coefficient, and inputs them into the first first feature extraction unit. The first first feature extraction unit performs feature weighting processing, that is, extracts at least one feature, and determines a weight for each feature of the at least one feature, and then weights the at least one feature according to the weight to obtain feature information output by the first first feature extraction unit. For ease of description, the feature information is recorded as the first feature information M1. Then, the first feature information M1 is input into the second first feature extraction unit for feature weighting processing to obtain the second feature information M2. And so on. For the i-th first feature extraction unit among the N first feature extraction units, the i-1th feature information M output by the i-1th first feature extraction unit is input into the second first feature extraction unit for feature weighting processing to obtain the second feature information M2. i-1 , input into the i-th first feature extraction unit for feature weighting processing, and obtain the i-th feature information M i Finally, the Nth feature information M output by the Nth first feature extraction unit is obtained N According to the Nth feature information M N , determine the first feature information of the reconstructed image block, for example, determine the Nth feature information as the first feature information of the reconstructed image block.

[0167] The embodiment of the present application does not limit the specific network structure of the first feature extraction unit. For example, the first feature extraction unit includes at least one convolutional layer and an attention mechanism.

[0168] In some embodiments, the step of performing feature weighting processing on the i-1th feature information of the reconstructed image block based on the quantization parameter in S404-A1 to obtain the i-th feature information of the reconstructed image block includes the following steps:

[0169] S404-A11, extracting M feature information of different scales from the (i-1)th feature information, where M is a positive integer greater than 1;

[0170] S404-A12, performing weighted processing on the feature information of M different scales to obtain the i-th weighted feature information;

[0171] S404-A13. Determine the i-th feature information according to the i-th weighted feature information.

[0172] Specifically, the decoding end performs multi-scale feature extraction on the i-1th feature information to obtain M feature information of different scales for the i-1th feature information. Then, the feature information of M different scales is weighted to obtain the i-th weighted feature information. For example, based on the importance of the feature, a larger weight is assigned to important features and a smaller weight is assigned to unimportant features. Then, based on the weight of each feature, the feature information of M different scales is weighted to obtain the i-th weighted feature information of the reconstructed image block. Finally, based on the i-th weighted feature information, the i-th feature information is determined. For example, the i-th weighted feature information is determined as the i-th feature information.

[0173] In one example, if the decoding end extracts the i-th feature information of the reconstructed image block through the first feature extraction unit, as shown in FIG8 , the first feature extraction unit includes a multi-scale extraction layer, wherein the multi-scale extraction layer is used to extract features at multiple scales.

[0174] For example, as shown in FIG8 , taking the i-th first feature extraction unit among the N first feature extraction units as an example, the input of the i-th first feature extraction unit is the output of the i-1th first feature extraction unit. The i-1th feature information M output by the i-1th first feature extraction unit is converted into i-1 The multi-scale extraction layer in the i-th first feature extraction unit is input, and the multi-scale extraction layer is used to extract multi-scale features, for example, to extract feature information at M different scales. Then, the feature information at M different scales (D1, D2...D M ) are assigned different weights to different features, and weighted operations are performed to obtain the i-th weighted feature information G1. Finally, according to the i-th weighted feature information G1, the i-th feature information M is determined. i For example, the i-th weighted feature information G1 is used as the i-th feature information M output by the i-th first feature extraction unit. i .

[0175] From the above, it can be seen that the first feature extraction unit of the embodiment of the present application performs multi-scale feature extraction to better explore the relationship between the input reconstructed image block and the real image block, so as to further improve the enhancement effect of the reconstructed image block.

[0176] In some examples, the multi-scale extraction layer is composed of a convolutional layer and a downsampling layer. For example, the convolutional layer is used to output feature information, and the downsampling layer is used to downsample the feature information output by the convolutional layer to obtain feature information at M different scales.

[0177] In another example, as shown in FIG9 , the multi-scale extraction layer includes M first feature extraction layers of different scales, and each first feature extraction layer can extract feature information at a corresponding scale.

[0178] Based on FIG9 , the above S404 - A11 includes: extracting feature information D1, D2, ..., D of M different scales of the i-1th feature information through the first feature extraction layer of M different scales. M .

[0179] The embodiment of the present application does not limit the specific network structure of the first feature extraction layer.

[0180] In some embodiments, the first feature extraction layer includes a convolution layer, and different first feature extraction layers include convolution layers with different sums and sizes of convolutions.

[0181] For example, assume that M=2, that is, the first feature extraction unit includes two first feature extraction layers. Assume that the size of the convolution kernel of one first feature extraction layer is 3X3, and the size of the convolution kernel of the other first feature extraction layer is 5X5. Use the 3X3 and 5X5 convolution kernels to input the i-1th feature information. Perform feature extraction to obtain feature information and feature information

[0182] In some embodiments, among the M first feature extraction layers of different scales, at least one first feature extraction layer includes an activation function.

[0183] According to the above steps, the i-1th feature information is input into the multi-scale extraction layer to obtain feature information D1, D2...D of M different scales. M Then, execute S404-A12 to analyze the feature information D1, D2...D of M different scales. M Perform weighted processing to obtain the i-th weighted feature information G1.

[0184] The decoding end can fuse M feature information of different scales and perform weighted processing to obtain the i-th weighted feature information.

[0185] The embodiment of the present application does not limit the specific manner of fusing the feature information of M different scales, for example, the feature information of M different scales may be added or multiplied.

[0186] In some embodiments, the above S404-A12 includes:

[0187] S404-A12-1. Concatenate M pieces of feature information at different scales to obtain first concatenated feature information; and perform weighted processing on the first concatenated feature information to obtain i-th weighted feature information.

[0188] Specifically, M feature information of different scales D1, D2...D MAfter splicing on the channel, the first spliced ​​feature information X is obtained, and X is weighted to obtain the i-th weighted feature information G1. For example, a larger weight is assigned to the important features in X, and a smaller weight is assigned to the unimportant features. Then, according to the weight of each feature, the features in X are weighted to obtain the i-th weighted feature information G1.

[0189] In the embodiment of the present application, the specific implementation method of performing weighted processing on the feature information of M different scales in the above S404-A12 to obtain the i-th weighted feature information is not limited.

[0190] In a possible implementation, the weighted processing of the splicing feature information in S404-A12-1 to obtain the i-th weighted feature information includes:

[0191] S404-A12-11. Perform weighted processing on the first splicing feature information through a weighted processing layer to obtain weighted feature information having a first number of channels;

[0192] S404-A12-12. Obtain the i-th weighted feature information according to the weighted feature information having the first number of channels.

[0193] For example, as shown in FIG8 , the first feature extraction unit further includes a weighted processing layer, which is used to perform weighted processing on features at multiple scales. In this way, the decoding end outputs the i-1th feature information M output by the i-1th first feature extraction unit. i-1 Input a multi-scale extraction layer, for example, input M first feature extraction layers of different scales, and these M first feature extraction layers of different scales output M feature information D1, D2...D M , these M feature information D1, D2...D M After splicing, the weighted processing layer is input, and the weighted processing layer performs feature weighting processing on the spliced ​​feature information to obtain weighted feature information with the first number of channels. Next, according to the weighted feature information with the first number of channels Get the i-th weighted feature information.

[0194] In some embodiments, since the feature information of M different scales is spliced ​​and input into the weighted processing layer, the number of channels of the feature information output by the weighted processing layer may be different from the number of channels of the i-1th feature information. Therefore, as shown in FIG10 , the first feature extraction unit of the embodiment of the present application further includes a second feature extraction layer, which is used to change the number of channels. At this time, the decoding end can convert the weighted feature information with the first number of channels into The second feature extraction layer is input to change the number of channels and outputs the i-th weighted feature information G1.

[0195] In some embodiments, the number of channels of the feature information output by each of the N first feature extraction units may be the same. For example, the number of channels of the i-th weighted feature information is the same as the number of channels of the i-1-th feature information.

[0196] The embodiment of the present application does not limit the specific network structure of the above-mentioned weighted processing layer. Exemplarily, the weighted processing layer includes a neuron attention mechanism.

[0197] The embodiment of the present application does not limit the network structure of the second feature extraction layer, for example, it includes a 1X1 convolution layer.

[0198] After obtaining the i-th weighted feature information according to the above steps, S404 - A13 is executed to determine the i-th feature information according to the i-th weighted feature information.

[0199] In one example, the i-th weighted feature information is determined as the i-th feature information.

[0200] In another example, the sum of the i-th weighted feature information and the i-1-th feature information is determined as the i-th feature information.

[0201] The following introduces the network structure of the i-th first feature extraction unit in the embodiment of the present application by giving examples.

[0202] As shown in Figure 11, the i-th first feature extraction unit of the embodiment of the present application includes a multi-scale extraction layer, a weighted processing layer and a second feature extraction layer, wherein the multi-scale extraction layer includes two first feature extraction layers, and the network structures of the two first feature extraction layers are basically the same, both including a convolution layer and an activation function, but the convolution kernels included in the two first feature extraction layers are the same size, wherein the same convolution kernel size is 3X3, and the other convolution kernel size is 5X5. Exemplarily, the activation function included in the two first feature extraction layers is ReLu. It should be noted that the activation function can also be an activation function of other forms. The second feature extraction layer includes a convolution layer with a convolution kernel size of 1X1 to reduce the number of feature channels.

[0203] Specifically, as shown in FIG11 , the i-1th feature information output by the i-1th first feature extraction unit is Input 2 first feature extraction layers respectively, perform multi-scale feature extraction on 2 first feature extraction layers, and output feature information and feature information Next, the feature information and feature information After splicing, the first splicing feature information is obtained

[0204] Exemplarily, the above-mentioned characteristic information C1, C2 and X are determined by the following formula (1):

[0205]

[0206] Here, σ represents the ReLU activation function, W1 and W2 represent the 3x3 and 5x5 convolution kernels, and * represents the convolution operation. It should be noted that the size of the convolution kernel and the type of activation function can be changed according to actual needs.

[0207] Next, the first splicing feature information The input weighted processing layer performs feature weighting processing, specifically assigning larger weights to important features to highlight the influence of the feature, and assigning smaller weights to relatively unimportant features to weaken the influence of the feature. The weighted processing layer outputs weighted feature information with the first number of channels Then the feature information Input the second feature extraction layer to reduce the number of feature channels, specifically, The i-th weighted feature information is obtained through the 1X1 convolution operation To achieve the reduction of the number of features on the channel, D3 and input M i-1 Add up to get the i-th feature information output by the i-th first feature extraction unit

[0208] For example, the above feature information D3 and M i-1 , determined by the following formula (2):

[0209]

[0210] Here, σ represents the ReLU activation function, W3 represents the 1x1 convolution kernel, and * represents the convolution operation. It should be noted that the size of the convolution kernel and the type of activation function can be changed according to actual needs.

[0211] The embodiment of the present application does not limit the specific network structure of the weighted processing layer.

[0212] In some embodiments, the weighted processing layer includes a neuron attention mechanism.

[0213] For example, the network structure of the neuron attention mechanism is shown in Figure 12, which includes depthwise convolution, pointwise convolution and activation function. Specifically, through the Depthwise convolution operation, the spliced ​​feature X is convolved on different channels, then through the ReLU activation function, through the Pointwise convolution operation, the information of different feature maps is fused. Then, through the Sigmoid activation function, the weight feature map is obtained. Multiply the corresponding elements of V and X to obtain weighted feature information with the first number of channels

[0214] Exemplarily, the weighted feature information having the first number of channels It can be determined by the following formula (3):

[0215]

[0216] Among them, Y c Represents the feature information Y on a channel, X c Represents the characteristics of feature information X on a channel, σ represents the ReLU activation function, δ represents the Sigmoid activation function, W d and W p Respectively represent Depthwise convolution and Pointwise convolution, Represents the operation of multiplying corresponding elements. It should be noted that the type of activation function can be changed according to actual needs.

[0217] According to the above formula (3), the weighted feature information with the first number of channels is determined After that, the Substitute into the above formula (2) to determine the i-th feature information M i .

[0218] The above example takes the extraction of the i-th feature information by the i-th first feature extraction unit among the N first feature extraction units. For other first feature extraction units among the N first feature extraction units, the process of extracting the i-th feature information by the i-th first feature extraction unit can be referred to, and the N-th feature information extracted by the final N-th first feature extraction unit can be obtained.

[0219] Next, the above step S404 - A2 is executed to determine the first feature information of the reconstructed image block according to the Nth feature information.

[0220] The implementation process of S404-A2 includes but is not limited to the following methods:

[0221] In method 1, the Nth feature information is determined as the first feature information of the reconstructed image block.

[0222] Mode 2, the above S404-A2 includes S404-A2-1: obtaining first feature information of the reconstructed image block based on at least one of the first N-1 feature information of the Nth feature information and the Nth feature information.

[0223] As can be seen from the above, the decoding end performs N iterative feature weighting processing on the reconstructed image block based on the quantization parameter. For example, based on the quantization parameter, feature weighting processing is performed on the i-1th feature information of the reconstructed image block to obtain the i-th feature information of the reconstructed image block, where i is a positive integer from 1 to N. This process is repeated to obtain the N-th feature information of the reconstructed image block. To improve the accuracy of obtaining the first feature information, the decoding end can obtain the first feature information of the reconstructed image block based on at least one of the N-1 feature information before the N-th feature information and the N-th feature information. For example, at least one of the N-1 feature information before the N-th feature information is concatenated with the N-th feature information, and then feature extraction is performed to obtain the first feature information of the reconstructed image block.

[0224] In some embodiments, as shown in FIG13 , the first feature extraction module includes, in addition to N first feature extraction units, a second feature extraction unit, which is used to re-extract features of the feature information output by at least one first feature extraction unit in the first N-1 first feature extraction units and the Nth feature information output by the Nth first feature extraction unit, so as to obtain deeper feature information. Specifically, in order to prevent the features of the reconstructed image block of the current image block from being lost as the network goes deeper, the decoding end combines the feature information output by at least one first feature extraction module in the first N-1 first feature extraction modules among the N first feature extraction modules with the Nth feature information output by the Nth first feature extraction module. N Input the second feature extraction unit, for example, the feature information output by at least one first feature extraction module in the first N-1 first feature extraction modules is combined with the Nth feature information M N After splicing, the images are input into the second feature extraction unit, which performs deeper feature extraction to obtain first feature information F1 of the reconstructed image block.

[0225] In some embodiments, the above S404-A2-1 also includes: splicing the feature information output by at least one first feature extraction module among the first N-1 first feature extraction modules, the Nth feature information, the reconstructed image block, and the quantization parameter, and inputting them into the second feature extraction unit to obtain the first feature information of the reconstructed image block.

[0226] In this embodiment, in order to further generate first feature information that meets the requirements, the reconstructed image block and quantization parameter, together with the feature information output by at least one first feature extraction module and the Nth feature information, are input into the second feature extraction unit for feature extraction, so that the feature extraction process is supervised by the reconstructed image block and quantization parameter, and the first feature information that better meets the requirements is output.

[0227] In some embodiments, in order to improve the accuracy of determining the first feature information of the reconstructed image block, the embodiments of the present application first extract the shallow feature information (i.e., the second feature information) of the reconstructed image block, and then determine the first feature information of the reconstructed image block based on the second feature information.

[0228] Based on this, S404-A includes: extracting second feature information of the reconstructed image block based on the quantization parameter; and performing feature weighting processing on the second feature information to obtain first feature information of the reconstructed image block.

[0229] Specifically, based on the quantization parameter, shallow feature extraction is performed on the reconstructed image block to obtain second feature information of the reconstructed image block. For example, the reconstructed image block and the quantization parameter are spliced ​​to obtain splicing information, and shallow feature extraction is performed on the splicing information to obtain the second feature information of the reconstructed image block. Then, based on the second feature information, first feature information of the reconstructed image block is determined. For example, deep feature extraction is performed based on the second feature information to obtain the first feature information of the reconstructed image block.

[0230] In the embodiment of the present application, the specific method of extracting features from the splicing information to obtain the second feature information is not limited.

[0231] In a possible implementation, a second feature extraction module is used to extract features from the splicing information to obtain second feature information.

[0232] For example, as shown in Figure 14, the enhancement model of the embodiment of the present application includes, in addition to the first feature extraction module, a second feature extraction module. The second feature extraction module is used to extract shallow feature information of the reconstructed image block and input the extracted shallow feature information into the first feature extraction module for deep feature extraction. Specifically, as shown in Figure 14, the decoding end first splices the reconstructed image and the quantization parameter to obtain splicing information, and then inputs the splicing information into the second feature extraction module for shallow feature extraction to obtain second feature information C2 of the reconstructed image block. Then, the shallow second feature information C2 is input into the first feature extraction module to obtain first feature information F1 of the reconstructed image block.

[0233] At this time, in some embodiments, the above S404-A2-1 includes: splicing at least one of the first N-1 feature information, the Nth feature information and the second feature information to obtain second spliced ​​feature information; performing feature re-extraction on the second spliced ​​feature information to obtain first feature information of the reconstructed image block.

[0234] For example, as shown in FIG15, the feature information output by at least one first feature extraction module in the first N-1 first feature extraction modules, the Nth feature information M N and the second feature information C2 are spliced ​​together and input into the second feature extraction unit to obtain the first feature information F1 of the reconstructed image block.

[0235] The embodiment of the present application does not limit the specific network structure of the second feature extraction module.

[0236] In one example, the second feature extraction module includes at least one convolutional layer.

[0237] Exemplarily, if the second feature extraction module includes two convolutional layers, the decoding end obtains the second feature information of the reconstructed image block through the two convolutional layers.

[0238] The following describes the process of determining the second feature information by taking the second feature extraction module including two convolutional layers as an example.

[0239] For example, as shown in FIG16 , the decoding end normalizes the reconstructed image block to obtain the normalized reconstructed image block Similarly, the quantization parameter QP is normalized to obtain the normalized quantization parameter Next, and After splicing, the second feature extraction module is input, and the first convolution layer outputs the feature The feature C1 is input into the second convolutional layer, and the second convolutional layer outputs the second feature information

[0240] In one example, the second characteristic information C2 can be determined by the following formula (4):

[0241]

[0242] Here, σ represents the ReLU activation function, W1 and W2 represent 3x3 convolution kernels, * represents the convolution operation, and · represents the concatenation operation. It should be noted that the size of the convolution kernel and the type of activation function can be changed according to actual needs.

[0243] According to the above method, after determining the second feature information of the reconstructed image block, feature weighting processing is performed on the second feature information to obtain the first feature information of the reconstructed image block, and the enhanced image block of the reconstructed image block is determined based on the first feature information of the reconstructed image block.

[0244] The embodiment of the present application does not limit the specific method of determining the enhanced image block of the reconstructed image block according to the first feature information of the reconstructed image block in the above S404-B.

[0245] In some embodiments, if the first characteristic information of the reconstructed image block is consistent with the size of the reconstructed image block, the first characteristic information may be determined as the enhanced image block.

[0246] In some embodiments, the above S404-B includes: performing nonlinear mapping on the first feature information of the reconstructed image block to obtain an enhanced image block.

[0247] In the embodiment of the present application, the specific method of performing nonlinear mapping on the first feature information of the reconstructed image block in S404-B to obtain the enhanced image block is not limited.

[0248] For example, a nonlinear mapping method is used to process the first feature information of the reconstructed image block so that the size of the processed first feature information of the reconstructed image block is consistent with the size of the reconstructed image block, and then the processed first feature information of the reconstructed image block is used as the enhanced image block.

[0249] In some embodiments, as shown in FIG17 , the enhancement model of the embodiment of the present application further includes a reconstruction module, which is used to perform further nonlinear mapping on the first feature information extracted by the first feature extraction module.

[0250] Based on FIG. 17 , the above S404 -B includes: performing nonlinear mapping on the first feature information of the reconstructed image block through the reconstruction module to obtain an enhanced image block.

[0251] The embodiment of the present application does not limit the network model of the reconstruction module.

[0252] In some embodiments, the reconstruction module includes at least one convolutional layer.

[0253] Exemplarily, as shown in FIG18 , the reconstruction module includes two convolutional layers. The encoding end inputs the first feature information F1 of the reconstructed image block output by the first feature extraction module into the reconstruction module. After the convolution operation of the two convolutional layers, the enhanced image block O1 of the reconstructed image block is obtained.

[0254] In one example, the enhanced image block O1 can be determined by the following formula (5):

[0255]

[0256] Here, F1 represents the first feature information of the reconstructed image, σ represents the ReLU activation function, W3, W4, and W5 represent 3x3 convolution kernels, and * represents the convolution operation. It should be noted that the size of the convolution kernel and the type of activation function can be changed according to actual needs.

[0257] In the embodiment of the present application, the decoding end performs quality enhancement on the reconstructed image block based on the quantization parameter through the above steps to obtain an enhanced image block of the reconstructed image block.

[0258] In some embodiments, before performing quality enhancement on a reconstructed image block, the decoder first needs to determine whether quality enhancement of the reconstructed image block is permitted. In other words, the decoder performs quality enhancement on the reconstructed image block if it determines that the effect of quality enhancement on the reconstructed image block is greater than the effect of not enhancing the quality.

[0259] The decoding end determines whether to allow quality enhancement of the reconstructed image block in the following ways, but is not limited to:

[0260] Mode 1: Decode the code stream to obtain a first flag, where the first flag is used to indicate whether quality enhancement of the reconstructed image block of the current image block is allowed.

[0261] In this method 1, the encoder determines whether to perform quality enhancement on the reconstructed image block of the current image block and notifies the decoder of the encoder's determination result via the first flag, so that the decoder and encoder adopt a consistent image enhancement strategy. Specifically, if the encoder performs quality enhancement on the reconstructed image block of the current image block, the value of the first flag is set to a first value, such as 1. If the encoder does not perform quality enhancement on the reconstructed image block of the current image block, the value of the first flag is set to a second value, such as 0. Thus, the decoder first obtains the first flag by decoding the bitstream and, based on the first flag, determines whether quality enhancement is permitted on the reconstructed image block of the current image block. For example, if the value of the first flag is 1, the decoder determines to perform quality enhancement on the reconstructed image block using the method of the embodiment of the present application, i.e., to perform quality enhancement on the reconstructed image block based on the quantization coefficient. If the value of the first flag is 0, the decoder determines not to perform quality enhancement on the reconstructed image block and instead to filter the reconstructed image block using an existing loop filtering method.

[0262] Optionally, the first flag may be a sequence-level flag.

[0263] Optionally, the first flag may be a frame-level flag.

[0264] Optionally, the first flag may be a film-level flag.

[0265] Optionally, the first flag may be a block-level flag, such as a CTU-level flag or a CU-level flag.

[0266] In method 2, the decoding end independently determines whether to perform quality enhancement on the reconstructed image block.

[0267] Specifically, the decoding end first enhances the quality of the reconstructed image block based on the quantization parameter to obtain a test enhanced image block. Then, the decoding end determines the image quality corresponding to the test enhanced image block and the unenhanced reconstructed image block. If the image quality of the test enhanced image block is greater than the image quality of the reconstructed image block, it means that the enhancement method of the embodiment of the present application can achieve a significant enhancement effect. At this time, the decoding end determines the above-mentioned test enhanced image block as the enhanced image block of the reconstructed image block and directly outputs it for display, and / or saves the above-mentioned test enhanced image block in the decoded image cache as an intra-frame reference for subsequent image blocks.

[0268] If the image quality of the enhanced image block tested above is less than or equal to the image quality of the reconstructed image block, it indicates that the enhancement method according to the embodiment of the present application cannot achieve a significant enhancement effect. In this case, the reconstructed image block is directly output for display after loop filtering, and / or the reconstructed image block after loop filtering is saved in the decoded image cache as an intra-frame reference for subsequent image blocks.

[0269] The embodiment of the present application does not limit the method for determining image quality. For example, Peak Signal-to-Noise Ratio (PSNR) or Structural Similarity (SSIN) is used as an indicator for evaluating image quality.

[0270] In some embodiments, the reconstructed image block is a reconstructed image block that has undergone loop filtering. For example, the decoding end determines a prediction block for the current image block and a residual block for the current image block, and adds the residual block to the prediction block to obtain a reconstructed image block. The reconstructed image block is then filtered using a loop filter and input into an enhancement model for quality enhancement.

[0271] In some embodiments, the embodiments of the present application may use an enhancement model to enhance the quality of the reconstructed image block, and then perform loop filtering.

[0272] In some embodiments, after the quality of the reconstructed image block is enhanced by the enhancement model, the embodiment of the present application no longer performs loop filtering processing.

[0273] In some embodiments, after the embodiment of the present application uses the enhancement model to enhance the quality of the reconstructed image block, the enhanced image block can be displayed and stored in the decoded image cache as a reference for other image blocks.

[0274] Optionally, the decoding end may further display the enhanced image block and store the unenhanced reconstructed image block in a decoded image buffer as a reference for other image blocks.

[0275] Optionally, the decoding end may further display the reconstructed image block and store the enhanced image block in a decoded image buffer as a reference for other image blocks.

[0276] It should be noted that before using the above enhancement model for quality enhancement, the enhancement model needs to be trained first.

[0277] In some embodiments, the training of the enhancement model can be completed by other devices, and the decoding end directly uses the trained enhancement model for quality enhancement.

[0278] In some embodiments, the training of the enhancement model can be completed by the decoding end. For example, the decoding end uses training data to train the enhancement model and uses the trained enhancement model to perform quality enhancement.

[0279] In some embodiments, the training set of the enhanced model consists of 800 images used for training from the super-reconstructed DIV2K dataset. Using VTM8.2 (Versatile Video Coding and Test Model 8.2, VVC test platform version 8.2), these 800 images are encoded with the quantization parameters QP set to 22, 27, 32, 37, AI mode, and the configuration of closing the loop filter LMCS, DB, SAO, and ALF, resulting in a total of 3200 encoded images. These 3200 encoded images are used as input to the network model, and the corresponding unencoded original images are used as true values ​​to constitute the final training set.

[0280] Optionally, to expand the diversity of the dataset, random cropping is used when loading the training set. Image blocks of size 128X128 are randomly cropped from each image as input to the enhanced model.

[0281] Optionally, the initial learning rate of the enhanced model is set to 1X10 -2 , the learning rate is reduced to 1 / 2 of the original rate every 30 epochs. For example, the training was finally completed on the PyTorch 1.6.0 platform.

[0282] The image processing method provided in the embodiment of the present application decodes the code stream to obtain the quantization coefficient of the current image block; determines the quantization parameter corresponding to the current image block, and based on the quantization parameter, dequantizes the quantization coefficient to obtain the transformation coefficient of the current image block; determines the reconstructed image block of the current image block based on the transformation coefficient; and performs quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block. Since the quantization coefficients corresponding to different image blocks may be different, in order to improve the enhancement accuracy of the image blocks, the present application performs quality enhancement on the reconstructed image blocks based on the quantization coefficients, which can improve the enhancement effect. In addition, the present application performs image quality enhancement on an image block basis, so that when the enhanced image block is used as a reference block for other image blocks in intra-frame prediction, a more accurate reference can be provided, thereby improving the accuracy of intra-frame prediction.

[0283] FIG19 is a flow chart of an image processing method provided by an embodiment of the present application. FIG19 can be understood as a more specific embodiment of the image processing method shown in FIG4. As shown in FIG19, the image processing method of the embodiment of the present application includes the following steps:

[0284] S501, decoding the code stream to obtain the quantization coefficient of the current image block;

[0285] S502: Determine a quantization parameter corresponding to the current image block, and dequantize the quantization coefficient based on the quantization parameter to obtain a transform coefficient of the current image block;

[0286] S503: Determine a reconstructed image block of the current image block according to the transformation coefficient.

[0287] The specific implementation process of the above S501 to S503 can refer to the description of the above S401 to S403, which will not be repeated here.

[0288] S504: Determine whether to perform quality enhancement on the reconstructed image block of the current image block.

[0289] Method 1: decode the code stream to obtain a first flag, and determine whether to allow quality enhancement of the reconstructed image block of the current image block according to the first flag.

[0290] Method 2: The decoding end performs quality enhancement on the reconstructed image block based on the quantization parameter to obtain a test enhanced image block; determines a first image quality of the test enhanced image block and a second image quality of the reconstructed image block, and determines whether to perform quality enhancement on the reconstructed image block of the current image block based on the first image quality and the second image quality.

[0291] If the decoding end determines to perform quality enhancement on the reconstructed image block of the current image block, the following step S505 is executed.

[0292] If the decoding end determines not to perform quality enhancement on the reconstructed image block of the current image block, the following step S508 is executed.

[0293] S505 : Perform feature extraction on the reconstructed image block based on the quantization parameter to obtain second feature information of the reconstructed image block.

[0294] S506: Perform feature weighting processing on the second feature information to obtain first feature information of the reconstructed image block.

[0295] S507 : Perform nonlinear mapping on the first feature information of the reconstructed image block to obtain an enhanced image block.

[0296] In some embodiments, the quality of the reconstructed image block is enhanced using an enhancement model to obtain an enhanced image block.

[0297] Exemplarily, as shown in Figure 18, the enhancement model of an embodiment of the present application includes a second feature extraction module, a first feature extraction module and a reconstruction module, wherein the second feature extraction module is used to extract shallow features (i.e., the second feature information of the reconstructed image block), the first feature extraction module is used to extract deep features (i.e., the first feature information of the reconstructed image block), and the reconstruction module is used to perform nonlinear mapping on the deep features to obtain the final enhanced image block.

[0298] Exemplarily, the second feature extraction module shown in FIG18 includes two convolutional layers. However, the structure of the second feature extraction module provided in the embodiment of the present application includes but is not limited to that shown in FIG18 .

[0299] For example, FIG18 shows that the first feature extraction module includes N first feature extraction layers and one second feature extraction layer, wherein the N first feature extraction layers are connected in series, and the outputs of the N first feature extraction layers and the output of the second feature extraction module are both input into the second feature extraction layer. However, the network structure of the first feature extraction module provided in the embodiments of the present application includes but is not limited to that shown in FIG18.

[0300] Exemplarily, FIG18 outputs a reconstruction module including two convolutional layers, but the network structure of the reconstruction module provided in the embodiment of the present application includes but is not limited to that shown in FIG18 .

[0301] In some embodiments, the first feature extraction layer of the embodiment of the present application includes a multi-scale extraction layer and a neuron attention mechanism, wherein the multi-scale extraction layer is used to perform multi-scale feature extraction, and the neuron attention mechanism is used to weight features. In this case, the first feature extraction layer of the embodiment of the present application can also be referred to as a multi-scale neuron attention (MSNA) unit.

[0302] In some embodiments, the enhanced model of the embodiment of the present application is also referred to as a neural network model based on a neuron attention mechanism (Neuron Attention-based CNN, abbreviated as NACNN).

[0303] Specifically, as shown in Figure 18, the decoder normalizes the reconstructed image block and quantization parameters, then splices them together and inputs the spliced ​​result into the enhancement model. The first convolutional layer in the second feature extraction module performs a convolution operation on the reconstructed image block and the quantization parameter to obtain feature information C1. Feature information C is then input into the second convolutional layer for convolution operation to obtain second feature information C2. The decoder inputs the second feature information C2 into the first feature extraction module. The first first feature extraction unit in the first feature extraction module performs multi-scale feature extraction and feature weighting on C2 to obtain first feature information M1. Next, the first feature information M1 is input into the second first feature extraction unit for multi-scale feature extraction and feature weighting to obtain second feature information M2. And so on, the Nth feature information MN output by the Nth first feature extraction unit is obtained. At the decoding end, the N feature information M1, M2…..MN extracted by each layer in the first feature extraction module and the second feature information C2 output by the second feature module are concatenated and input into the second feature extraction layer in the first feature extraction module to change the number of channels, outputting the first feature information F1 of the reconstructed image block. The decoding end then inputs the first feature information F1 into the reconstruction module for reconstruction. Specifically, a convolution operation is performed on the first convolution layer in the reconstruction module to obtain feature information C3. The second convolution layer in the reconstruction module then convolutions the feature information C3 to obtain feature information C4. Finally, based on the feature information C4, an enhanced image block O1 of the reconstructed image block is obtained. For example, the feature information C4 is output as the enhanced image block O1 of the reconstructed image block.

[0304] It should be noted that the specific implementation process of the above S505 to S507 can refer to the specific description of the above S404, and will not be repeated here.

[0305] S508: Perform loop filtering on the reconstructed image block.

[0306] In some embodiments, if the decoding end determines not to use the enhancement model to enhance the quality of the reconstructed image block of the current image block, step S508 is executed to perform loop filtering on the reconstructed image block.

[0307] In the image processing method provided in the embodiment of the present application, the decoding end first determines whether to perform quality enhancement on the reconstructed image block before performing quality enhancement on the reconstructed image block, thereby improving the reliability of image processing.

[0308] The above describes the image processing method of the embodiment of the present application by taking the decoding end as an example. On this basis, the following describes the image processing method provided by the embodiment of the present application by taking the encoding end as an example.

[0309] FIG20 is a flow chart of an image processing method according to an embodiment of the present application, which is applied to the encoders shown in FIG1 and FIG2. As shown in FIG20, the method according to the embodiment of the present application includes:

[0310] S601, determining a quantization parameter of a current image block, and encoding the current image block based on the quantization parameter to obtain a quantization coefficient of the current image block;

[0311] S602, dequantizing the quantization coefficients based on the quantization parameters of the current image block to obtain a residual block of the current image block;

[0312] S603: Obtain a reconstructed image block of the current image block according to the residual block.

[0313] During the image encoding process, the encoder receives a video stream consisting of a series of image frames, performs video encoding on each frame of the video stream, and divides the image frame into blocks to obtain the current encoding block.

[0314] The embodiment of the present application does not limit the specific size of the current image block.

[0315] In some embodiments, the current image block of the embodiment of the present application is a CTU. For example, a frame of image is divided into several CTUs, and the present application does not limit the size of the CTU. For example, the size of a CTU is 128×128, 64×64, 32×32, etc.

[0316] In some embodiments, the current image block of the embodiments of the present application is a CU, for example, a CTU is divided into one or more CUs.

[0317] In some embodiments, the current image block of the embodiments of the present application is a TU or a PU, for example, a CU is divided into one or more TUs or PUs.

[0318] In some embodiments, the current image block of the embodiment of the present application only includes a chrominance component and can be understood as a chrominance block.

[0319] In some embodiments, the current image block of the embodiment of the present application only includes a brightness component and can be understood as a brightness block.

[0320] In some embodiments, the current image block includes both a luminance component and a chrominance component.

[0321] In some embodiments, if the current image block includes a CU, the encoding process of the embodiment of the present application is to divide the current image frame into blocks to obtain the current image block, use the intra-frame or inter-frame prediction method to predict the current image block, and obtain the predicted value of the current image block. The original value of the current image block is subtracted from the predicted value to obtain the residual value of the current image block. The transformation method corresponding to the current image block is determined, and the residual value is transformed using the transformation method to obtain the transformation coefficient. The transformation coefficient is quantized using the determined quantization parameter to obtain the quantization coefficient, and the quantization coefficient is encoded to obtain a code stream.

[0322] At the same time, the encoding end also includes a decoding process. Specifically, as shown in Figure 2, the dequantization / transformation unit 240 dequantizes the quantization coefficients based on the determined quantization parameters to obtain transform coefficients, and then inversely transforms the transform coefficients to obtain a residual block. The reconstruction unit 250 adds the prediction block and the residual block to obtain a reconstructed image block of the current image block.

[0323] In some embodiments, if the current image block includes CUs, the encoder divides the current image frame into blocks to obtain multiple CUs, and for each CU, reconstructs a block of each CU according to the above method. In this way, the reconstructed blocks of the CUs included in the current image block are combined to obtain a reconstructed image block of the current image block.

[0324] It should be noted that, if the current image block includes multiple CUs, the quantization parameter of the current image block includes the quantization parameters corresponding to the multiple CUs. Optionally, the quantization parameters corresponding to the multiple CUs may be the same or different.

[0325] In one possible implementation, the quantization parameter of the current image block in the embodiment of the present application can be expressed in the form of a matrix. For example, if the size of the current image block is 16X16, the quantization parameter of the current image block can be a 16X16 matrix, and each element in the matrix is ​​the quantization parameter of the pixel point at the corresponding position in the current image block.

[0326] The embodiment of the present application does not limit the specific process of determining the quantization parameter corresponding to the current image block.

[0327] In some embodiments, the codec uses a default quantization parameter as the quantization parameter corresponding to the current image block.

[0328] In some embodiments, the encoder determines the quantization parameter corresponding to the current image block through calculation. Optionally, the encoder can write the determined quantization parameter into the bitstream, so that the decoder can determine the quantization parameter corresponding to the current image block by decoding the bitstream.

[0329] S604 : Based on the quantization parameter, perform quality enhancement on the reconstructed image block to obtain an enhanced image block.

[0330] During the video encoding process, different image blocks may have different corresponding quantization parameters (QP). In some embodiments, the quantization parameter (QP) includes a quantization step size. During the video encoding process, the transform coefficients of the image block are quantized. A larger quantization step size results in greater image loss, while a smaller quantization step size results in less image loss. Therefore, in order to improve the enhancement effect of the current image block, embodiments of the present application consider the impact of the quantization parameter (QP) corresponding to the current image block during the quality enhancement process of the current image block.

[0331] Since different quantization coefficients result in different losses when inverse quantizing an image block, in order to improve the enhancement effect of the image block, the embodiment of the present application performs quality enhancement on the reconstructed image block of the current image block based on the quantization coefficient corresponding to the current image block, thereby improving the enhancement effect of the reconstructed image block.

[0332] In the embodiment of the present application, image quality enhancement is performed in units of image blocks. In this way, when the enhanced image blocks are used as reference blocks for other image blocks in intra-frame prediction, more accurate references can be provided, thereby improving the accuracy of intra-frame prediction.

[0333] In addition, the embodiment of the present application performs image quality enhancement in units of image blocks. Compared with image quality enhancement of the entire frame image, it can focus more on enhancing finer features in the image blocks, further improving the enhancement quality of the image blocks.

[0334] In the embodiment of the present application, there is no limitation on the method of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain the enhanced image block.

[0335] In one embodiment, the encoding end enhances the quality of the reconstructed image block through an enhancement model based on the quantization parameters to obtain an enhanced image block. Specifically, after the encoding end determines the reconstructed image block of the current image block according to the above steps, in order to reduce the distortion of the reconstructed image block and improve the quality of the reconstructed image block, as shown in Figure 5, the reconstructed image block and the corresponding quantization coefficient are input into a pre-trained enhancement model to perform image enhancement, and finally obtain the enhanced image block of the current image block. It should be noted that the enhancement model is obtained by training based on different image blocks and their corresponding quantization parameters, and has fully learned the influence of the quantization parameters on the quality enhancement of image blocks. Then, based on different quantization parameters, the reconstructed image block obtained after dequantization with different quantization parameters can be enhanced with high quality. The enhancement model of the embodiment of the present application can be any neural network model that can enhance the quality of image blocks, and the embodiment of the present application does not limit the specific network model of the enhancement model.

[0336] In some embodiments, the encoder fuses the reconstructed image block and the quantization parameter and inputs the result into the enhancement model. The fusing method of the reconstructed image block and the quantization parameter includes at least the following examples:

[0337] In Example 1, assume that the size of the reconstructed image block is N1*N2, where N1 and N2 can be the same or different. After multiplying the reconstructed image block by the quantization parameter, the block is input into the enhancement model. Specifically, each pixel in the reconstructed image block is multiplied by the quantization parameter to obtain an N1*N2 matrix, which is then input into the enhancement model.

[0338] Example 2: After concatenating the reconstructed image and the quantization parameters, the image is fed into the enhancement model. Specifically, the quantization coefficients are set to a matrix of size N1*N2, and the reconstructed image of size N1*N2 and the quantization parameters of size N1*N2 are concatenated and fed into the enhancement model.

[0339] It should be noted that, in addition to the fusion methods shown in Examples 1 and 2 above, the encoding end may also adopt other fusion methods to fuse the reconstructed image and the corresponding quantization parameters and then input them into the enhancement model for quality enhancement.

[0340] In some embodiments, to prevent features with smaller absolute values ​​from being overwritten by features with larger absolute values, the encoder normalizes the reconstructed image block and quantization parameters before inputting them into the enhancement model, ensuring that all features are treated equally. Next, based on the normalized reconstructed image block and quantization parameters, an enhanced image block of the reconstructed image block is obtained. For example, the normalized reconstructed image block and quantization parameters are concatenated and then input into the enhancement model for quality enhancement to improve the quality enhancement effect.

[0341] The reconstructed image block in the embodiment of the present application is a reconstructed image block of the current image block under the first component.

[0342] The first component may be a luminance component or a chrominance component.

[0343] In some embodiments, the above S604 includes the following steps S604-A and S604-B:

[0344] S604-A, performing feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image;

[0345] S604-B. Determine an enhanced image block according to the first feature information.

[0346] The embodiment of the present application does not limit the specific implementation of S604-A.

[0347] In one possible implementation, feature information of the reconstructed image block is extracted based on the quantization parameter. For example, the quantization parameter and the reconstructed image block are input into a neural network layer to extract the feature information of the reconstructed image block. Then, the feature information is analyzed and different weights are assigned to different features. For example, a larger weight is assigned to an important feature in the feature information to highlight the influence of the feature, and a smaller weight is assigned to a relatively unimportant feature to weaken the influence of the feature. Then, the feature information of the reconstructed image block is weighted according to the weight corresponding to the feature to obtain the first feature information of the reconstructed image block.

[0348] In another possible implementation, as shown in Figure 6, the enhancement model of the embodiment of the present application includes a first feature extraction module, which is used to extract at least one feature of the input information (i.e., the reconstructed image block and the quantization parameter) and assign different weights to the at least one extracted feature to perform feature weighted processing.

[0349] As shown in Figure 6, the enhancement model of the embodiment of the present application includes a first feature extraction module, which is used to perform weighted processing on the extracted features. Specifically, the reconstructed image block and the quantization parameter (optionally, after being fused by the method of Example 1 or Example 2 above) are input into the first feature extraction module, and the first feature extraction module performs feature extraction to extract at least one feature, and assigns a larger weight to the important feature of the at least one feature to highlight the influence of the feature, and assigns a smaller weight to the relatively unimportant feature to weaken the influence of the feature, and then performs weighted processing according to the weight corresponding to the at least one feature to obtain the first feature information of the reconstructed image block. Finally, based on the first feature information of the reconstructed image block, the enhanced image block of the reconstructed image block is determined.

[0350] The embodiment of the present application can assign different weights to different features to highlight the influence of important features and weaken the influence of unimportant features, thereby further improving the quality enhancement effect of the reconstructed image block.

[0351] The embodiment of the present application does not limit the network model of the first feature extraction module. For example, the first feature extraction module includes multiple convolutional layers and an attention mechanism.

[0352] In another possible implementation, the above S604-A includes the following steps:

[0353] S604-A1, based on the quantization parameter, perform feature weighting processing on the i-1th feature information of the reconstructed image block to obtain the i-th feature information of the reconstructed image block, where i is a positive integer from 1 to N. Repeat the process to obtain the Nth feature information of the reconstructed image block. If i is 1, the i-1th feature information is the reconstructed image block.

[0354] S604-A2: Determine first feature information of the reconstructed image block according to the Nth feature information.

[0355] In this implementation, the encoding end performs N feature-weighted iterative processing on the reconstructed image block based on the quantization parameter to obtain the Nth feature information of the reconstructed image block. Specifically, based on the quantization parameter, feature-weighted processing is performed on the reconstructed image block to obtain the first feature information of the reconstructed image block. Then, based on the quantization parameter, feature-weighted processing is performed on the first feature information to obtain the second feature information of the reconstructed image block. The process is repeated and, based on the quantization parameter, feature-weighted processing is performed on the N-1th feature information to obtain the Nth feature information of the reconstructed image block. It should be noted that the embodiment of the present application does not limit the specific method of feature weighting processing. For example, the quantization parameter and the i-1th feature information are input into a neural network with a feature weighting processing function to obtain the i-th feature information of the reconstructed image block.

[0356] In some embodiments, as shown in FIG7 , the first feature extraction module of the embodiment of the present application includes N first feature extraction units. If N is greater than 1, these N first feature extraction units are connected in series, that is, the output of the previous first feature extraction unit serves as the input of the next first feature extraction unit. Based on the quantization parameter, these N first feature extraction units perform feature weighting processing on the reconstructed image block to obtain the Nth feature information of the reconstructed image block. As shown in FIG7 , each of the N first feature extraction units is configured to extract at least one feature and assign a larger weight to important features in the at least one feature to highlight the influence of the feature, and assign a smaller weight to relatively unimportant features to weaken the influence of the feature. The subsequent first feature extraction unit further extracts and assigns weights to the feature information output by the previous first feature extraction unit to further highlight important features. After processing by the N first feature extraction units, important features of the reconstructed image block can be primarily enhanced, while less important features in the reconstructed image block can be less strongly enhanced, thereby improving the enhancement effect of the reconstructed image block.

[0357] In one example, if N is 1, that is, the first feature extraction module includes a first feature extraction unit, then S603-A1 includes: the encoder fuses the reconstructed image and the quantization coefficients, and inputs the fused data into the first feature extraction unit. The first feature extraction unit performs feature extraction to extract at least one feature, and assigns different weights based on the importance of the features in the feature. Then, weighted processing is performed on the at least one feature based on the different weights to obtain first feature information. Finally, first feature information of the reconstructed image block is determined based on the first feature information. For example, the first feature information is determined as the first feature information of the reconstructed image block.

[0358] In another example, if N is greater than 1, that is, the first feature extraction module includes multiple first feature extraction units, then the above S603-A1 includes: the encoding end fuses the reconstructed image and the quantization coefficient, and inputs them into the first first feature extraction unit. The first first feature extraction unit performs feature weighting processing, that is, extracts at least one feature, and determines a weight for each feature of the at least one feature, and then weights the at least one feature according to the weight to obtain feature information output by the first first feature extraction unit. For ease of description, the feature information is recorded as the first feature information M1. Then, the first feature information M1 is input into the second first feature extraction unit for feature weighting processing to obtain the second feature information M2. And so on. For the i-th first feature extraction unit among the N first feature extraction units, the i-1th feature information M output by the i-1th first feature extraction unit is input into the second first feature extraction unit for feature weighting processing to obtain the second feature information M2. i-1 , input into the i-th first feature extraction unit for feature weighting processing, and obtain the i-th feature information M i Finally, the Nth feature information M output by the Nth first feature extraction unit is obtained N According to the Nth feature information M N , determine the first feature information of the reconstructed image block, for example, determine the Nth feature information as the first feature information of the reconstructed image block.

[0359] The embodiment of the present application does not limit the specific network structure of the first feature extraction unit. For example, the first feature extraction unit includes at least one convolutional layer and an attention mechanism.

[0360] In some embodiments, in S604-A1, performing feature weighting processing on the i-1th feature information of the reconstructed image block based on the quantization parameter to obtain the i-th feature information of the reconstructed image block includes the following steps:

[0361] S604-A11, extracting M feature information of different scales from the (i-1)th feature information, where M is a positive integer greater than 1;

[0362] S604-A12, performing weighted processing on the feature information of M different scales to obtain the i-th weighted feature information;

[0363] S604-A13. Determine the i-th feature information according to the i-th weighted feature information.

[0364] Specifically, the encoder performs multi-scale feature extraction on the i-1th feature information to obtain M feature information at different scales for the i-1th feature information. Next, the M feature information at different scales is weighted to obtain the i-th weighted feature information. For example, based on the importance of the features, a larger weight is assigned to important features and a smaller weight is assigned to unimportant features. The M feature information at different scales is then weighted based on the weights of each feature to obtain the i-th weighted feature information of the reconstructed image block. Finally, the i-th feature information is determined based on the i-th weighted feature information. For example, the i-th weighted feature information is determined as the i-th feature information.

[0365] In one example, if the encoding end extracts the i-th feature information of the reconstructed image block through the first feature extraction unit, as shown in FIG8 , the first feature extraction unit includes a multi-scale extraction layer, wherein the multi-scale extraction layer is used to extract features at multiple scales.

[0366] For example, as shown in FIG8 , taking the i-th first feature extraction unit among the N first feature extraction units as an example, the input of the i-th first feature extraction unit is the output of the i-1th first feature extraction unit. The i-1th feature information M output by the i-1th first feature extraction unit is converted into i-1 The multi-scale extraction layer in the i-th first feature extraction unit is input, and the multi-scale extraction layer is used to extract multi-scale features, for example, to extract feature information at M different scales. Then, the feature information at M different scales (D1, D2...D M ) are assigned different weights to different features, and weighted operations are performed to obtain the i-th weighted feature information G1. Finally, according to the i-th weighted feature information G1, the i-th feature information M is determined. i For example, the i-th weighted feature information G1 is used as the i-th feature information M output by the i-th first feature extraction unit. i .

[0367] From the above, it can be seen that the first feature extraction unit of the embodiment of the present application performs multi-scale feature extraction to better explore the relationship between the input reconstructed image block and the real image block, so as to further improve the enhancement effect of the reconstructed image block.

[0368] In some examples, the multi-scale extraction layer is composed of a convolutional layer and a downsampling layer. For example, the convolutional layer is used to output feature information, and the downsampling layer is used to downsample the feature information output by the convolutional layer to obtain feature information at M different scales.

[0369] In another example, as shown in FIG9 , the multi-scale extraction layer includes M first feature extraction layers of different scales, and each first feature extraction layer can extract feature information at a corresponding scale.

[0370] Based on FIG9 , the above S604 - A11 includes: extracting feature information D1, D2, ..., D of M different scales of the i-1th feature information through the first feature extraction layer of M different scales. M .

[0371] The embodiment of the present application does not limit the specific network structure of the first feature extraction layer.

[0372] In some embodiments, the first feature extraction layer includes a convolution layer, and different first feature extraction layers include convolution layers with different sums and sizes of convolutions.

[0373] For example, assume that M=2, that is, the first feature extraction unit includes two first feature extraction layers. Assume that the size of the convolution kernel of one first feature extraction layer is 3X3, and the size of the convolution kernel of the other first feature extraction layer is 5X5. Use the 3X3 and 5X5 convolution kernels to input the i-1th feature information. Perform feature extraction to obtain feature information and feature information

[0374] In some embodiments, among the M first feature extraction layers of different scales, at least one first feature extraction layer includes an activation function.

[0375] According to the above steps, the i-1th feature information is input into the multi-scale extraction layer to obtain feature information D1, D2...D of M different scales. M Then, execute S604-A12 to analyze the M feature information D1, D2...D of different scales. M Perform weighted processing to obtain the i-th weighted feature information G1.

[0376] The encoding end can fuse M feature information of different scales and perform weighted processing to obtain the i-th weighted feature information.

[0377] The embodiment of the present application does not limit the specific manner of fusing the feature information of M different scales, for example, the feature information of M different scales may be added or multiplied.

[0378] In some embodiments, the above S604-A12 includes:

[0379] S604-A12-1. Concatenate M pieces of feature information at different scales to obtain first concatenated feature information; and perform weighted processing on the first concatenated feature information to obtain i-th weighted feature information.

[0380] Specifically, M feature information of different scales D1, D2...D MAfter splicing on the channel, the first spliced ​​feature information X is obtained, and X is weighted to obtain the i-th weighted feature information G1. For example, a larger weight is assigned to the important features in X, and a smaller weight is assigned to the unimportant features. Then, according to the weight of each feature, the features in X are weighted to obtain the i-th weighted feature information G1.

[0381] In the embodiment of the present application, the specific implementation method of performing weighted processing on the feature information of M different scales in the above S604-A12 to obtain the i-th weighted feature information is not limited.

[0382] In a possible implementation, weighted processing is performed on the splicing feature information in S604-A12-1 to obtain the i-th weighted feature information, including:

[0383] S604-A12-11. Perform weighted processing on the first splicing feature information through a weighted processing layer to obtain weighted feature information having a first number of channels;

[0384] S604-A12-12. Obtain the i-th weighted feature information according to the weighted feature information having the first number of channels.

[0385] For example, as shown in FIG8 , the first feature extraction unit further includes a weighted processing layer, which is used to perform weighted processing on features at multiple scales. In this way, the encoder outputs the i-1th feature information M output by the i-1th first feature extraction unit. i-1 Input a multi-scale extraction layer, for example, input M first feature extraction layers of different scales, and these M first feature extraction layers of different scales output M feature information D1, D2...D M , these M feature information D1, D2...D M After splicing, the weighted processing layer is input, and the weighted processing layer performs feature weighting processing on the spliced ​​feature information to obtain weighted feature information with the first number of channels. Next, according to the weighted feature information with the first number of channels Get the i-th weighted feature information.

[0386] In some embodiments, since the feature information of M different scales is spliced ​​and input into the weighted processing layer, the number of channels of the feature information output by the weighted processing layer may be different from the number of channels of the i-1th feature information. Therefore, as shown in FIG10 , the first feature extraction unit of the embodiment of the present application further includes a second feature extraction layer, which is used to change the number of channels. At this time, the encoder can convert the weighted feature information with the first number of channels into The second feature extraction layer is input to change the number of channels and outputs the i-th weighted feature information G1.

[0387] In some embodiments, the number of channels of the feature information output by each of the N first feature extraction units may be the same. For example, the number of channels of the i-th weighted feature information is the same as the number of channels of the i-1-th feature information.

[0388] The embodiment of the present application does not limit the specific network structure of the above-mentioned weighted processing layer. Exemplarily, the weighted processing layer includes a neuron attention mechanism.

[0389] The embodiment of the present application does not limit the network structure of the second feature extraction layer, for example, it includes a 1X1 convolution layer.

[0390] After obtaining the i-th weighted feature information according to the above steps, S604 - A13 is executed to determine the i-th feature information according to the i-th weighted feature information.

[0391] In one example, the i-th weighted feature information is determined as the i-th feature information.

[0392] In another example, the sum of the i-th weighted feature information and the i-1-th feature information is determined as the i-th feature information.

[0393] The following introduces the network structure of the i-th first feature extraction unit in the embodiment of the present application by giving examples.

[0394] As shown in Figure 11, the i-th first feature extraction unit of the embodiment of the present application includes a multi-scale extraction layer, a weighted processing layer and a second feature extraction layer, wherein the multi-scale extraction layer includes two first feature extraction layers, and the network structures of the two first feature extraction layers are basically the same, both including a convolution layer and an activation function, but the convolution kernels included in the two first feature extraction layers are the same size, wherein the same convolution kernel size is 3X3, and the other convolution kernel size is 5X5. Exemplarily, the activation function included in the two first feature extraction layers is ReLu. It should be noted that the activation function can also be an activation function of other forms. The second feature extraction layer includes a convolution layer with a convolution kernel size of 1X1 to reduce the number of feature channels.

[0395] The following introduces the network structure of the i-th first feature extraction unit in the embodiment of the present application by giving examples.

[0396] As shown in Figure 11, the i-th first feature extraction unit of the embodiment of the present application includes a multi-scale extraction layer, a weighted processing layer and a second feature extraction layer, wherein the multi-scale extraction layer includes two first feature extraction layers, and the network structures of the two first feature extraction layers are basically the same, both including a convolution layer and an activation function, but the convolution kernels included in the two first feature extraction layers are the same size, wherein the same convolution kernel size is 3X3, and the other convolution kernel size is 5X5. Exemplarily, the activation function included in the two first feature extraction layers is ReLu. It should be noted that the activation function can also be an activation function of other forms. The second feature extraction layer includes a convolution layer with a convolution kernel size of 1X1 to reduce the number of feature channels.

[0397] Specifically, as shown in FIG11 , the i-1th feature information output by the i-1th first feature extraction unit is Input 2 first feature extraction layers respectively, perform multi-scale feature extraction on 2 first feature extraction layers, and output feature information and feature information Next, the feature information and feature information After splicing, the first splicing feature information is obtained

[0398] Exemplarily, the above-mentioned characteristic information C1, C2 and X are determined by the following formula (1).

[0399] Next, the first splicing feature information The input weighted processing layer performs feature weighting processing, specifically assigning larger weights to important features to highlight the influence of the feature, and assigning smaller weights to relatively unimportant features to weaken the influence of the feature. The weighted processing layer outputs weighted feature information with the first number of channels Then the feature information Input the second feature extraction layer to reduce the number of feature channels, specifically, The i-th weighted feature information is obtained through the 1X1 convolution operation To achieve the reduction of the number of features on the channel, D3 and input M i-1 Add up to get the i-th feature information output by the i-th first feature extraction unit

[0400] For example, the above feature information D3 and M i-1 , is determined by the following formula (2).

[0401] The embodiment of the present application does not limit the specific network structure of the weighted processing layer.

[0402] In some embodiments, the weighted processing layer includes a neuron attention mechanism.

[0403] For example, the network structure of the neuron attention mechanism is shown in Figure 12, which includes depthwise convolution, pointwise convolution and activation function. Specifically, through the Depthwise convolution operation, the spliced ​​feature X is convolved on different channels, then through the ReLU activation function, through the Pointwise convolution operation, the information of different feature maps is fused. Then, through the Sigmoid activation function, the weight feature map is obtained. Multiply the corresponding elements of V and X to obtain weighted feature information with the first number of channels

[0404] Exemplarily, the weighted feature information having the first number of channels It can be determined by the above formula (3).

[0405] According to the above formula (3), the weighted feature information with the first number of channels is determined After that, the Substitute into the above formula (2) to determine the i-th feature information M i .

[0406] The above example takes the extraction of the i-th feature information by the i-th first feature extraction unit among the N first feature extraction units. For other first feature extraction units among the N first feature extraction units, the process of extracting the i-th feature information by the i-th first feature extraction unit can be referred to, and the N-th feature information extracted by the final N-th first feature extraction unit can be obtained.

[0407] Next, the above step S603 - A2 is executed to determine the first feature information of the reconstructed image block according to the Nth feature information output by the Nth first feature extraction unit.

[0408] The implementation process of S604-A2 includes but is not limited to the following methods:

[0409] In method 1, the Nth feature information is determined as the first feature information of the reconstructed image block.

[0410] Mode 2, the above S604-A2 includes S604-A2-1: obtaining first feature information of the reconstructed image block according to at least one of the first N-1 feature information of the Nth feature information and the Nth feature information.

[0411] As can be seen from the above, the encoding end performs N iterative feature weighting processing on the reconstructed image block based on the quantization parameter. For example, based on the quantization parameter, the i-1th feature information of the reconstructed image block is subjected to feature weighting processing to obtain the i-th feature information of the reconstructed image block, where i is a positive integer from 1 to N. The process is repeated to obtain the N-th feature information of the reconstructed image block. In this way, the first feature information of the reconstructed image block can be obtained based on at least one of the first N-1 feature information of the N-th feature information and the N-th feature information. For example, at least one of the first N-1 feature information is spliced ​​with the N-th feature information, and then feature extraction is performed to obtain the first feature information of the reconstructed image block.

[0412] In some embodiments, as shown in FIG13 , the first feature extraction module includes, in addition to N first feature extraction units, a second feature extraction unit, which is used to re-extract features from the feature information output by at least one first feature extraction unit in the first N-1 first feature extraction units and the Nth feature information output by the Nth first feature extraction unit, so as to obtain deeper feature information. Specifically, the feature information output by at least one first feature extraction module in the first N-1 first feature extraction modules among the N first feature extraction modules is re-extracted from the Nth feature information output by the Nth first feature extraction module. N Input the second feature extraction unit, for example, the feature information output by at least one first feature extraction module in the first N-1 first feature extraction modules is combined with the Nth feature information M N After splicing, the images are input into the second feature extraction unit, which performs deeper feature extraction to obtain first feature information F1 of the reconstructed image block.

[0413] In some embodiments, the above S604-A2-1 also includes: splicing the feature information output by at least one first feature extraction module among the first N-1 first feature extraction modules, the Nth feature information, the reconstructed image block, and the quantization parameter, and inputting them into the second feature extraction unit to obtain the first feature information of the reconstructed image block.

[0414] In this embodiment, in order to further generate first feature information that meets the requirements, the reconstructed image block and quantization parameter, together with the feature information output by at least one first feature extraction module and the Nth feature information, are input into the second feature extraction unit for feature extraction, so that the feature extraction process is supervised by the reconstructed image block and quantization parameter, and the first feature information that better meets the requirements is output.

[0415] In some embodiments, in order to improve the accuracy of determining the first feature information of the reconstructed image block, the embodiments of the present application first extract the shallow feature information (i.e., the second feature information) of the reconstructed image block, and then determine the first feature information of the reconstructed image block based on the second feature information.

[0416] Based on this, S604-A includes: extracting second feature information of the reconstructed image block based on the quantization parameter; and performing feature weighting processing on the second feature information to obtain first feature information of the reconstructed image block.

[0417] Specifically, based on the quantization parameter, shallow feature extraction is performed on the reconstructed image block to obtain second feature information of the reconstructed image block. For example, the reconstructed image block and the quantization parameter are spliced ​​to obtain splicing information, and shallow feature extraction is performed on the splicing information to obtain the second feature information of the reconstructed image block. Then, based on the second feature information, first feature information of the reconstructed image block is determined. For example, deep feature extraction is performed based on the second feature information to obtain the first feature information of the reconstructed image block.

[0418] In the embodiment of the present application, the specific method of extracting features from the splicing information to obtain the second feature information is not limited.

[0419] In a possible implementation, a second feature extraction module is used to extract features from the splicing information to obtain second feature information.

[0420] For example, as shown in Figure 14, the enhancement model of the embodiment of the present application includes, in addition to the first feature extraction module, a second feature extraction module. The second feature extraction module is used to extract shallow feature information of the reconstructed image block and input the extracted shallow feature information into the first feature extraction module for deep feature extraction. Specifically, as shown in Figure 14, the encoding end first splices the reconstructed image and the quantization parameter to obtain splicing information, and then inputs the splicing information into the second feature extraction module for shallow feature extraction to obtain second feature information C2 of the reconstructed image block. Then, the shallow second feature information C2 is input into the first feature extraction module to obtain the first feature information F1 of the reconstructed image block.

[0421] At this time, in some embodiments, the above S604-A2-1 includes: splicing at least one of the first N-1 feature information, the Nth feature information and the second feature information to obtain second spliced ​​feature information; performing feature re-extraction on the second spliced ​​feature information to obtain first feature information of the reconstructed image block.

[0422] For example, as shown in FIG15, the feature information output by at least one first feature extraction module in the first N-1 first feature extraction modules, the Nth feature information M Nand the second feature information C2 are spliced ​​and input into the second feature extraction unit to obtain the first feature information F1 of the reconstructed image block.

[0423] The embodiment of the present application does not limit the specific network structure of the second feature extraction module.

[0424] In one example, the second feature extraction module includes at least one convolutional layer.

[0425] Exemplarily, if the second feature extraction module includes two convolutional layers, the encoding end obtains the second feature information of the reconstructed image block through the two convolutional layers.

[0426] In one example, the second characteristic information C2 can be determined by the following formula (4).

[0427] According to the above method, after determining the second feature information of the reconstructed image block, the second feature information is input into the first feature extraction module for deep feature extraction to obtain the first feature information of the reconstructed image block. Based on the first feature information of the reconstructed image block, an enhanced image block of the reconstructed image block is determined.

[0428] The embodiment of the present application does not limit the specific method of determining the enhanced image block of the reconstructed image block according to the first feature information of the reconstructed image block in the above S603-B.

[0429] According to the above method, after determining the second feature information of the reconstructed image block, feature weighting processing is performed on the second feature information to obtain the first feature information of the reconstructed image block, and the enhanced image block of the reconstructed image block is determined based on the first feature information of the reconstructed image block.

[0430] The embodiment of the present application does not limit the specific method of determining the enhanced image block of the reconstructed image block according to the first feature information of the reconstructed image block in the above S604-B.

[0431] In some embodiments, if the first characteristic information of the reconstructed image block is consistent with the size of the reconstructed image block, the first characteristic information may be determined as the enhanced image block.

[0432] In some embodiments, the above S604-B includes: performing nonlinear mapping on the first feature information of the reconstructed image block to obtain an enhanced image block.

[0433] The embodiment of the present application does not limit the specific method of performing nonlinear mapping on the first feature information of the reconstructed image block in S604-B to obtain the enhanced image block.

[0434] For example, a nonlinear mapping method is used to process the first feature information of the reconstructed image block so that the size of the processed first feature information of the reconstructed image block is consistent with the size of the reconstructed image block, and then the processed first feature information of the reconstructed image block is used as the enhanced image block.

[0435] In some embodiments, as shown in FIG17 , the enhancement model of the embodiment of the present application further includes a reconstruction module, which is used to perform further nonlinear mapping on the first feature information extracted by the first feature extraction module.

[0436] Based on FIG. 17 , the above S604 -B includes: performing nonlinear mapping on the first feature information of the reconstructed image block through the reconstruction module to obtain an enhanced image block.

[0437] The embodiment of the present application does not limit the network model of the reconstruction module.

[0438] In some embodiments, the reconstruction module includes at least one convolutional layer.

[0439] Exemplarily, as shown in FIG18 , the reconstruction module includes two convolutional layers. The encoding end inputs the first feature information F1 of the reconstructed image block output by the first feature extraction module into the reconstruction module. After the convolution operation of the two convolutional layers, the enhanced image block O1 of the reconstructed image block is obtained.

[0440] In the embodiment of the present application, the encoding end performs quality enhancement on the reconstructed image block based on the quantization parameter through the above steps to obtain an enhanced image block of the reconstructed image block.

[0441] In some embodiments, before performing quality enhancement on a reconstructed image block, the encoder first needs to determine whether to perform quality enhancement on the reconstructed image block. In other words, if the encoder determines that the effect of performing quality enhancement on the reconstructed image block is greater than the effect of not performing quality enhancement, the encoder performs quality enhancement on the reconstructed image block.

[0442] The encoder determines whether to perform quality enhancement on the reconstructed image block in the following ways, but is not limited to:

[0443] In method 1, the configuration file includes a first flag that indicates whether to perform quality enhancement on the reconstructed image block of the current image block. Thus, the encoder can determine whether to perform quality enhancement on the reconstructed image block of the current image block based on the first flag. For example, if the value of the first flag is a first numerical value, such as 1, the encoder determines to perform quality enhancement on the reconstructed image block of the current image block and executes the method of the above embodiment. If the value of the first flag is a second numerical value, such as 0, the encoder determines not to perform quality enhancement on the reconstructed image block of the current image block and instead uses an existing loop filtering method to filter the reconstructed image block.

[0444] In method 2, the encoder independently determines whether to perform quality enhancement on the reconstructed image block.

[0445] Specifically, the encoding end first enhances the quality of the reconstructed image block based on the quantization parameter to obtain a test enhanced image block. Then, the image quality corresponding to the test enhanced image block and the unenhanced reconstructed image block is determined. If the image quality of the test enhanced image block is greater than the image quality of the reconstructed image block, it means that the enhancement method of the embodiment of the present application can achieve a significant enhancement effect. At this time, the above-mentioned determined test enhanced image block is determined as the enhanced image block of the reconstructed image block and is directly output for display, and / or the above-mentioned determined test enhanced image block is saved in the decoded image cache as an intra-frame reference for subsequent image blocks.

[0446] If the image quality of the enhanced image block in the above test is less than or equal to the image quality of the reconstructed image block, it means that the use of the enhancement model cannot achieve a significant enhancement effect. At this time, the reconstructed image block is directly output for display after loop filtering, and / or the reconstructed image block after loop filtering is saved in the decoded image cache as an intra-frame reference for subsequent image blocks.

[0447] In some embodiments, the encoding end writes a first flag in the code stream, and the first flag is used to indicate whether to perform quality enhancement on the reconstructed image block of the current image block, so that the decoding end determines whether to perform quality enhancement on the reconstructed image block of the current image block based on the first flag, thereby ensuring consistency between the encoding and decoding ends.

[0448] Specifically, if the encoder determines to perform quality enhancement on the reconstructed image block of the current image block, the value of the first flag is set to a first value, for example, 1. If the encoder determines not to perform quality enhancement on the reconstructed image block of the current image block, the value of the first flag is set to a second value, for example, 0. In this way, the decoder first obtains the first flag by decoding the bitstream, and determines whether to perform quality enhancement on the reconstructed image block of the current image block based on the first flag.

[0449] Optionally, the first flag may be a sequence-level flag.

[0450] Optionally, the first flag may be a frame-level flag.

[0451] Optionally, the first flag may be a film-level flag.

[0452] Optionally, the first flag may be a block-level flag, such as a CTU-level flag.

[0453] In some embodiments, the reconstructed image block is a reconstructed image block that has undergone loop filtering. For example, the encoder determines a prediction block for the current image block and a residual block for the current image block, adds the residual block to the prediction block, and obtains a reconstructed image block. The reconstructed image block is then filtered using a loop filter and input into an enhancement model for quality enhancement.

[0454] In some embodiments, the embodiments of the present application may use an enhancement model to enhance the quality of the reconstructed image block, and then perform loop filtering.

[0455] In some embodiments, after the quality of the reconstructed image block is enhanced by the enhancement model, the embodiment of the present application no longer performs loop filtering processing.

[0456] In some embodiments, after the embodiment of the present application uses the enhancement model to enhance the quality of the reconstructed image block, the enhanced image block can be displayed and stored in the decoded image cache as a reference for other image blocks.

[0457] Optionally, the encoding end may further display the enhanced image block and store the unenhanced reconstructed image block in a decoded image buffer as a reference for other image blocks.

[0458] Optionally, the encoding end may further display the reconstructed image block and store the enhanced image block in a decoded image buffer as a reference for other image blocks.

[0459] It should be noted that before using the above enhancement model for quality enhancement, the enhancement model needs to be trained first.

[0460] Furthermore, to illustrate the beneficial effects of the image processing methods provided by the embodiments of the present application, the solutions of the embodiments of the present application were tested, for example, in the VVC test software VTM8.2, using the Class A, Class B, Class C, and Class E sequences specified in the general test conditions. The results in Table 1 were obtained with QP settings of 32, 37, 42, and 47, and encoding in AI mode.

[0461] Table 1 BD-rate test results of the proposed method in VTM8.2

[0462]

[0463] Explanation of the parameters in Table 1 above:

[0464] Class represents the video category, Sequence represents the specific test sequence, and Y, Cb, and Cr represent the performance of the three video components, luminance and chrominance. The values ​​in Table 1 represent BD-rate, a measure of algorithm performance that indicates the change in bitrate and PSNR compared to the original encoding algorithm. Overall negative values ​​indicate improved performance, and larger absolute values ​​indicate greater performance improvement.

[0465] In the image processing method provided by the embodiment of the present application, the encoding end determines the quantization parameter of the current image block, and encodes the current image block based on the quantization parameter to obtain the quantization coefficient of the current image block; based on the quantization parameter of the current image block, the quantization coefficient is inversely quantized to obtain the residual block of the current image block, and based on the residual block, the quality of the reconstructed image block is enhanced based on the quantization parameter to obtain an enhanced image block. Since the quantization coefficients corresponding to different image blocks may be different, in order to improve the enhancement accuracy of the image blocks, the present application performs quality enhancement on the reconstructed image blocks based on the quantization coefficients, which can improve the enhancement effect. In addition, the present application performs image quality enhancement on an image block basis, so that when the enhanced image block is used as a reference block for other image blocks in intra-frame prediction, a more accurate reference can be provided, thereby improving the accuracy of intra-frame prediction.

[0466] FIG21 is a flow chart of an image processing method according to an embodiment of the present application. FIG21 can be understood as a more specific embodiment of the image processing method shown in FIG20. As shown in FIG21, the image processing method according to the embodiment of the present application includes the following steps:

[0467] S701, determining a quantization parameter of a current image block, and encoding the current image block based on the quantization parameter to obtain a quantization coefficient of the current image block;

[0468] S702, dequantizing the quantization coefficients based on the quantization parameters of the current image block to obtain a residual block of the current image block;

[0469] S703: Obtain a reconstructed image block of the current image block according to the residual block.

[0470] The specific implementation process of the above S701 and S703 can refer to the description of the above S601 and S603, which will not be repeated here.

[0471] S704: Determine whether to perform quality enhancement on the reconstructed image block of the current image block.

[0472] Method 1: obtaining a first flag, and determining whether to allow quality enhancement of the reconstructed image block of the current image block according to the first flag.

[0473] Method 2: Based on the quantization parameter, the reconstructed image block is quality enhanced to obtain a test enhanced image block; the first image quality of the test enhanced image block and the second image quality of the reconstructed image block are determined, and according to the first image quality and the second image quality, it is determined whether to perform quality enhancement on the reconstructed image block of the current image block.

[0474] If it is determined to perform quality enhancement on the reconstructed image block of the current image block, the following step S705 is executed.

[0475] If it is determined that the quality of the reconstructed image block of the current image block is not to be enhanced, the following step S708 is executed.

[0476] S705 : Perform feature extraction on the reconstructed image block based on the quantization parameter to obtain second feature information of the reconstructed image block.

[0477] S706: Perform feature weighting processing on the second feature information to obtain first feature information of the reconstructed image block.

[0478] S707 : Perform nonlinear mapping on the first feature information of the reconstructed image block to obtain an enhanced image block.

[0479] In some embodiments, the quality of the reconstructed image block is enhanced using an enhancement model to obtain an enhanced image block.

[0480] Exemplarily, as shown in Figure 18, the enhancement model of an embodiment of the present application includes a second feature extraction module, a first feature extraction module and a reconstruction module, wherein the second feature extraction module is used to extract shallow features (i.e., the second feature information of the reconstructed image block), the first feature extraction module is used to extract deep features (i.e., the first feature information of the reconstructed image block), and the reconstruction module is used to perform nonlinear mapping on the deep features to obtain the final enhanced image block.

[0481] Exemplarily, the second feature extraction module shown in FIG18 includes two convolutional layers. However, the structure of the second feature extraction module provided in the embodiment of the present application includes but is not limited to that shown in FIG18 .

[0482] For example, FIG18 shows that the first feature extraction module includes N first feature extraction layers and one second feature extraction layer, wherein the N first feature extraction layers are connected in series, and the outputs of the N first feature extraction layers and the output of the second feature extraction module are both input into the second feature extraction layer. However, the network structure of the first feature extraction module provided in the embodiments of the present application includes but is not limited to that shown in FIG18.

[0483] Exemplarily, FIG18 outputs a reconstruction module including two convolutional layers, but the network structure of the reconstruction module provided in the embodiment of the present application includes but is not limited to that shown in FIG18 .

[0484] In some embodiments, the first feature extraction layer of the embodiment of the present application includes a multi-scale extraction layer and a neuron attention mechanism, wherein the multi-scale extraction layer is used to perform multi-scale feature extraction, and the neuron attention mechanism is used to weight features. In this case, the first feature extraction layer of the embodiment of the present application can also be referred to as a multi-scale neuron attention (MSNA) unit.

[0485] In some embodiments, the enhanced model of the embodiment of the present application is also referred to as a neural network model based on a neuron attention mechanism (Neuron Attention-based CNN, abbreviated as NACNN).

[0486] Specifically, as shown in Figure 18, the encoder normalizes the reconstructed image block and quantization parameters, then splices them together and inputs the spliced ​​result into the enhancement model. The first convolutional layer in the second feature extraction module performs a convolution operation on the reconstructed image block and the quantization parameter to obtain feature information C1. Feature information C is then input into the second convolutional layer for convolution operation to obtain second feature information C2. The encoder inputs the second feature information C2 into the first feature extraction module. The first first feature extraction unit in the first feature extraction module performs multi-scale feature extraction and feature weighting on C2 to obtain first feature information M1. Next, the first feature information M1 is input into the second first feature extraction unit for multi-scale feature extraction and feature weighting to obtain second feature information M2. And so on, the Nth feature information MN output by the Nth first feature extraction unit is obtained. On the encoding side, the N feature information M1, M2…..MN extracted by each layer in the first feature extraction module and the second feature information C2 output by the second feature module are concatenated and input into the second feature extraction layer in the first feature extraction module to change the number of channels, outputting the first feature information F1 of the reconstructed image block. The encoding side then inputs the first feature information F1 into the reconstruction module for reconstruction. Specifically, the first convolutional layer in the reconstruction module performs a convolution operation on the feature information F1 to obtain feature information C3. The second convolutional layer in the reconstruction module performs a convolution operation on the feature information C3 to obtain feature information C4. Finally, based on the feature information C4, an enhanced image block O1 of the reconstructed image block is obtained. For example, the feature information C4 is output as the enhanced image block O1 of the reconstructed image block.

[0487] It should be noted that the specific implementation process of the above S705 to S707 can refer to the specific description of the above S604, which will not be repeated here.

[0488] S708: Perform loop filtering on the reconstructed image block.

[0489] In some embodiments, if the encoder determines not to use the enhancement model to enhance the quality of the reconstructed image block of the current image block, step S708 is executed to perform loop filtering on the reconstructed image block.

[0490] In the image processing method provided by the embodiment of the present application, the encoder first determines whether to perform quality enhancement on the reconstructed image block before performing quality enhancement on the reconstructed image block, thereby improving the reliability of image processing. In the image processing method provided by the embodiment of the present application, the encoder first determines whether to perform quality enhancement on the reconstructed image block using the enhancement model before performing quality enhancement on the reconstructed image block using the enhancement model, thereby improving the reliability of image processing.

[0491] It should be understood that Figures 4 to 21 are merely examples of the present application and should not be understood as limiting the present application.

[0492] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, a variety of simple modifications can be made to the technical solution of the present application, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0493] It should also be understood that in the various method embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three types of relationships can exist. Specifically, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship.

[0494] The above text describes in detail the method embodiment of the present application in conjunction with Figures 4 to 21, and the following text describes in detail the device embodiment of the present application in conjunction with Figures 22 to 24.

[0495] FIG22 is a schematic block diagram of an image processing device provided in an embodiment of the present application.

[0496] As shown in FIG22 , the image processing apparatus 10 includes:

[0497] The decoding unit 11 is used to decode the code stream to obtain the quantization coefficient of the current image block;

[0498] a determination unit 12, configured to determine a quantization parameter corresponding to a current image block, and dequantize the quantization coefficient based on the quantization parameter to obtain a transform coefficient of the current image block;

[0499] A reconstruction unit 13, configured to determine a reconstructed image block of the current image block according to the transformation coefficient;

[0500] The enhancement unit 14 is configured to input the reconstructed image block and the quantization parameter into an enhancement model to perform image enhancement to obtain an enhanced image block.

[0501] In some embodiments, the enhancement unit 14 is specifically configured to perform feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image block; and determine the enhanced image block according to the first feature information.

[0502] In some embodiments, the enhancement unit 14 is specifically used to perform feature weighting processing on the i-1th feature information of the reconstructed image block based on the quantization parameter to obtain the i-th feature information of the reconstructed image block, where i is a positive integer from 1 to N. Repeat the execution to obtain the Nth feature information of the reconstructed image block. If i is 1, the i-1th feature information is the reconstructed image block; and determine the first feature information of the reconstructed image block based on the Nth feature information.

[0503] In some embodiments, the enhancement unit 14 is specifically used to extract feature information of M different scales of the i-1th feature information, where M is a positive integer greater than 1; perform weighted processing on the M feature information of different scales to obtain the i-th weighted feature information; and determine the i-th feature information based on the i-th weighted feature information.

[0504] In some embodiments, the enhancing unit 14 is specifically configured to extract feature information of M different scales of the (i-1)th feature information through M first feature extraction layers of different scales.

[0505] In some embodiments, the first feature extraction layer includes a convolution layer, and the convolution kernels of the convolution layers included in different first feature extraction layers are different in size.

[0506] In some embodiments, among the M first feature extraction layers of different scales, at least one first feature extraction layer includes an activation function.

[0507] In some embodiments, the enhancement unit 14 is specifically configured to splice the M pieces of feature information at different scales to obtain first spliced ​​feature information; and perform weighted processing on the first spliced ​​feature information to obtain the i-th weighted feature information.

[0508] In some embodiments, the enhancement unit 14 is specifically configured to perform weighted processing on the first splicing feature information through a weighted processing layer to obtain weighted feature information having a first number of channels; and obtain the i-th weighted feature information based on the weighted feature information having the first number of channels.

[0509] In some embodiments, the number of channels of the i-th weighted feature information is the same as the number of channels of the i-1-th feature information.

[0510] Optionally, the weighted processing layer includes a neuron attention mechanism.

[0511] In some embodiments, the enhancing unit 14 is specifically configured to determine the sum of the i-th weighted feature information and the (i-1)-th feature information as the i-th feature information.

[0512] In some embodiments, the enhancing unit 14 is specifically configured to determine the i-th weighted feature information as the i-th feature information.

[0513] In some embodiments, the enhancement unit 14 is specifically configured to extract second feature information of the reconstructed image block based on the quantization parameter; and perform feature weighting processing on the second feature information to obtain first feature information of the reconstructed image block.

[0514] In some embodiments, the enhancement unit 14 is specifically configured to splice the reconstructed image block and the quantization parameter to obtain splicing information; and perform feature extraction on the splicing information to obtain the second feature information.

[0515] In some embodiments, the enhancement unit 14 is specifically configured to perform feature extraction on the splicing information through a second feature extraction module to obtain the second feature information.

[0516] Optionally, the second feature extraction module includes at least one convolutional layer.

[0517] In some embodiments, the enhancing unit 14 is specifically configured to obtain the first feature information of the reconstructed image block according to at least one of the first N-1 feature information of the Nth feature information and the Nth feature information.

[0518] In some embodiments, the enhancement unit 14 is specifically used to splice at least one of the first N-1 feature information, the Nth feature information and the second feature information to obtain second spliced ​​feature information; and perform feature re-extraction on the second spliced ​​feature information to obtain the first feature information of the reconstructed image block.

[0519] In some embodiments, the enhancement unit 14 is specifically configured to perform nonlinear mapping on the first feature information of the reconstructed image block to obtain the enhanced image block.

[0520] In some embodiments, the enhancement unit 14 is specifically configured to perform nonlinear mapping on the first feature information of the reconstructed image block through a reconstruction module to obtain the enhanced image block.

[0521] Optionally, the reconstruction module includes at least one convolutional layer.

[0522] In some embodiments, the decoding unit 11 is further used to decode the code stream to obtain a first flag, where the first flag is used to indicate whether to perform quality enhancement on the reconstructed image block of the current image block; the enhancement unit 14 is further used to, based on the first flag, perform quality enhancement on the reconstructed image block based on the quantization parameter to obtain the enhanced image block when it is determined that quality enhancement of the reconstructed image block is allowed.

[0523] In some embodiments, the enhancement unit 14 is further used to perform quality enhancement on the reconstructed image block based on the quantization parameter to obtain a test enhanced image block; determine a first image quality of the test enhanced image block and a second image quality of the reconstructed image block; if the first image quality is greater than the second image quality, determine the test enhanced image block as an enhanced image block of the reconstructed image block.

[0524] Optionally, the reconstructed image block is a reconstructed image block of the current image block under the first component.

[0525] Optionally, the first component is a luminance component or a chrominance component.

[0526] In some embodiments, the enhancement unit 14 is specifically configured to perform normalization processing on the reconstructed image block and the quantization parameter; and obtain the enhanced image block based on the normalized reconstructed image block and the quantization parameter.

[0527] It should be understood that the device embodiments and method embodiments may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, no further description is given here. Specifically, the device 10 shown in FIG22 can execute the image processing method of the decoding end of the embodiment of the present application, and the aforementioned and other operations and / or functions of the various units in the device 10 are respectively for implementing the corresponding processes in each method such as the above-mentioned image processing method of the decoding end. For the sake of brevity, no further description is given here.

[0528] FIG23 is a schematic block diagram of an image processing device provided in an embodiment of the present application.

[0529] As shown in FIG23 , the image processing apparatus 20 includes:

[0530] a determination unit 21, configured to determine a quantization parameter of a current image block, and encode the current image block based on the quantization parameter to obtain a quantization coefficient of the current image block;

[0531] an inverse quantization unit 22, configured to inverse quantize the quantization coefficients based on the quantization parameters of the current image block to obtain a residual block of the current image block;

[0532] A reconstruction unit 23, configured to obtain a reconstructed image block of the current image block according to the residual block;

[0533] The enhancement unit 24 is configured to perform quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block.

[0534] In some embodiments, the enhancement unit 24 is specifically configured to perform feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image block; and determine the enhanced image block according to the first feature information.

[0535] In some embodiments, the enhancement unit 24 is specifically used to perform feature weighting processing on the i-1th feature information of the reconstructed image block based on the quantization parameter to obtain the i-th feature information of the reconstructed image block, where i is a positive integer from 1 to N. The process is repeated to obtain the N-th feature information of the reconstructed image block. If i is 1, the i-1th feature information is the reconstructed image block; and the first feature information of the reconstructed image block is determined based on the N-th feature information.

[0536] In some embodiments, the enhancement unit 24 is specifically used to extract feature information of M different scales of the i-1th feature information, where M is a positive integer greater than 1; perform weighted processing on the feature information of M different scales to obtain i-th weighted feature information; and determine the i-th feature information based on the i-th weighted feature information.

[0537] In some embodiments, the enhancement unit 24 is specifically configured to extract feature information of M different scales of the (i-1)th feature information through M first feature extraction layers of different scales.

[0538] In some embodiments, the first feature extraction layer includes a convolution layer, and the convolution kernels of the convolution layers included in different first feature extraction layers are different in size.

[0539] In some embodiments, among the M first feature extraction layers of different scales, at least one first feature extraction layer includes an activation function.

[0540] In some embodiments, the enhancement unit 24 is specifically configured to splice the M pieces of feature information at different scales to obtain first spliced ​​feature information; and perform weighted processing on the first spliced ​​feature information to obtain the i-th weighted feature information.

[0541] In some embodiments, the enhancement unit 24 is specifically configured to perform weighted processing on the first splicing feature information through a weighted processing layer to obtain weighted feature information having a first number of channels; and obtain the i-th weighted feature information based on the weighted feature information having the first number of channels.

[0542] In some embodiments, the number of channels of the i-th weighted feature information is the same as the number of channels of the i-1-th feature information.

[0543] Optionally, the weighted processing layer includes a neuron attention mechanism.

[0544] In some embodiments, the enhancing unit 24 is specifically configured to determine the sum of the i-th weighted feature information and the (i-1)-th feature information as the i-th feature information.

[0545] In some embodiments, the enhancing unit 24 is specifically configured to determine the i-th weighted feature information as the i-th feature information.

[0546] In some embodiments, the enhancement unit 24 is specifically configured to extract second feature information of the reconstructed image block based on the quantization parameter; and perform feature weighting processing on the second feature information to obtain first feature information of the reconstructed image block.

[0547] In some embodiments, the enhancement unit 24 is specifically configured to splice the reconstructed image block and the quantization parameter to obtain splicing information; and perform feature extraction on the splicing information to obtain the second feature information.

[0548] In some embodiments, the enhancement unit 24 is specifically configured to perform feature extraction on the splicing information through a second feature extraction module to obtain the second feature information.

[0549] Optionally, the second feature extraction module includes at least one convolutional layer.

[0550] In some embodiments, the enhancing unit 24 is specifically configured to obtain the first feature information of the reconstructed image block according to at least one of the first N-1 feature information of the Nth feature information and the Nth feature information.

[0551] In some embodiments, the enhancement unit 24 is specifically used to splice at least one of the first N-1 feature information, the Nth feature information and the second feature information to obtain second spliced ​​feature information; and perform feature re-extraction on the second spliced ​​feature information to obtain the first feature information of the reconstructed image block.

[0552] In some embodiments, the enhancement unit 24 is specifically configured to perform nonlinear mapping on the first feature information of the reconstructed image block to obtain the enhanced image block.

[0553] In some embodiments, the enhancement unit 24 is specifically configured to obtain the enhanced image block by performing nonlinear mapping on the first feature information of the reconstructed image block through a reconstruction module, wherein the reconstruction module includes at least one convolutional layer.

[0554] Optionally, the reconstruction module includes at least one convolutional layer.

[0555] In some embodiments, the encoding unit 22 is further configured to write a first flag into the bitstream, where the first flag is configured to indicate whether to perform quality enhancement on the reconstructed image block of the current image block.

[0556] Optionally, the reconstructed image block is a reconstructed image block of the current image block under the first component.

[0557] Optionally, the first component is a luminance component or a chrominance component.

[0558] In some embodiments, the enhancement unit 24 is specifically configured to perform normalization processing on the reconstructed image block and the quantization parameter; and obtain the enhanced image block based on the normalized reconstructed image block and the quantization parameter.

[0559] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further details will be given here. Specifically, the device 20 shown in Figure 23 may correspond to the corresponding subject in the image processing method for encoding points in the embodiment of the present application, and the aforementioned and other operations and / or functions of the various units in the device 20 are respectively for implementing the corresponding processes in various methods such as the image processing method of the encoding end. For the sake of brevity, no further details will be given here.

[0560] The above describes the apparatus and system of the embodiment of the present application from the perspective of functional units in conjunction with the accompanying drawings. It should be understood that the functional unit can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented by a combination of hardware and software units. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software units in the decoding processor. Optionally, the software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0561] Figure 24 is a schematic block diagram of an electronic device provided in an embodiment of the present application.

[0562] As shown in FIG32 , the electronic device 30 may be a video encoder or a video decoder as described in an embodiment of the present application. The electronic device 30 may include:

[0563] The memory 33 and the processor 32 are configured to store a computer program 34 and transmit the program code 34 to the processor 32. In other words, the processor 32 can call and run the computer program 34 from the memory 33 to implement the method in the embodiment of the present application.

[0564] For example, the processor 32 may be configured to execute the steps of the method 200 according to the instructions in the computer program 34 .

[0565] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0566] General-purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0567] In some embodiments of the present application, the memory 33 includes but is not limited to:

[0568] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0569] In some embodiments of the present application, the computer program 34 may be divided into one or more units, which are stored in the memory 33 and executed by the processor 32 to implement the method provided by the present application. The one or more units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 34 in the electronic device 30.

[0570] As shown in FIG24 , the electronic device 30 may further include:

[0571] The transceiver 33 may be connected to the processor 32 or the memory 33 .

[0572] The processor 32 can control the transceiver 33 to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more.

[0573] It should be understood that the various components in the electronic device 30 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0574] Figure 25 is a schematic block diagram of a video encoding and decoding system provided in an embodiment of the present application.

[0575] As shown in Figure 25, the video encoding and decoding system 40 may include: a video encoder 41 and a video decoder 42, wherein the video encoder 41 is used to execute the video encoding method involved in the embodiment of the present application, and the video decoder 42 is used to execute the video decoding method involved in the embodiment of the present application.

[0576] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment. In other words, the present application also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment.

[0577] The present application also provides a code stream, which is generated by the above encoding method. Optionally, the code stream includes a first flag.

[0578] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0579] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0580] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0581] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0582] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: Decode the code stream to obtain the quantization coefficient of the current image block; Determining a quantization parameter corresponding to a current image block, and performing inverse quantization on the quantization coefficient based on the quantization parameter to obtain a transform coefficient of the current image block; determining a reconstructed image block of the current image block according to the transformation coefficient; Based on the quantization parameter, quality enhancement is performed on the reconstructed image block to obtain an enhanced image block.

2. The method according to claim 1, characterized in that The step of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block includes: performing feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image block; The enhanced image block is determined according to the first feature information.

3. The method according to claim 2, characterized in that The performing feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image block includes: Based on the quantization parameter, perform feature weighting processing on the i-1th feature information of the reconstructed image block to obtain the i-th feature information of the reconstructed image block, where i is a positive integer from 1 to N, and repeat the process to obtain the Nth feature information of the reconstructed image block. If i is 1, the i-1th feature information is the reconstructed image block; Determine first feature information of the reconstructed image block according to the Nth feature information.

4. The method according to claim 3, characterized in that The step of performing feature weighting processing on the i-1th feature information of the reconstructed image block based on the quantization parameter to obtain the i-th feature information of the reconstructed image block includes: Extracting M pieces of feature information of different scales from the (i-1)th feature information, where M is a positive integer greater than 1; Performing weighted processing on the M feature information of different scales to obtain i-th weighted feature information; The i-th feature information is determined according to the i-th weighted feature information.

5. The method according to claim 4, characterized in that The extracting M pieces of feature information of different scales from the (i-1)th feature information includes: Feature information of M different scales of the (i-1)th feature information is extracted through the first feature extraction layers of M different scales.

6. The method according to claim 5, characterized in that The first feature extraction layer includes a convolution layer, and different first feature extraction layers include convolution kernels of different sizes.

7. The method according to claim 6, characterized in that Among the M first feature extraction layers of different scales, at least one first feature extraction layer includes an activation function.

8. The method according to claim 4, characterized in that The weighted processing of the M feature information of different scales to obtain the i-th weighted feature information includes: Splicing the M pieces of feature information at different scales to obtain first spliced ​​feature information; The first splicing feature information is weighted to obtain the i-th weighted feature information.

9. The method according to claim 8, characterized in that The weighted processing of the splicing feature information to obtain the i-th weighted feature information includes: Performing weighted processing on the first splicing feature information through a weighted processing layer to obtain weighted feature information having a first number of channels; The i-th weighted feature information is obtained according to the weighted feature information having the first number of channels.

10. The method according to claim 9, characterized in that The number of channels of the i-th weighted feature information is the same as the number of channels of the i-1-th feature information.

11. The method according to claim 9, characterized in that The weighted processing layer includes a neuron attention mechanism.

12. The method according to any one of claims 4 to 11, characterized in that: The determining the i-th feature information according to the i-th weighted feature information includes: The sum of the i-th weighted feature information and the i-1-th feature information is determined as the i-th feature information.

13. The method according to any one of claims 4 to 11, characterized in that: The obtaining the i-th feature information according to the i-th weighted feature information includes: The i-th weighted feature information is determined as the i-th feature information.

14. The method according to any one of claims 3 to 11, characterized in that The performing feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image block includes: extracting second feature information of the reconstructed image block based on the quantization parameter; Perform feature weighting processing on the second feature information to obtain first feature information of the reconstructed image block.

15. The method according to claim 14, characterized in that The extracting the second feature information of the reconstructed image block based on the quantization parameter includes: splicing the reconstructed image block and the quantization parameter to obtain splicing information; Feature extraction is performed on the splicing information to obtain the second feature information.

16. The method according to claim 15, characterized in that The extracting features from the splicing information to obtain the second feature information includes: The second feature information is obtained by performing feature extraction on the splicing information through a second feature extraction module.

17. The method according to claim 14, characterized in that The determining, according to the Nth feature information, the first feature information of the reconstructed image block includes: The first feature information of the reconstructed image block is obtained according to at least one of the first N-1 feature information of the Nth feature information and the Nth feature information.

18. The method according to claim 17, characterized in that The obtaining, based on at least one of the first N-1 pieces of feature information of the Nth feature information and the Nth feature information, the first feature information of the reconstructed image block includes: splicing at least one of the first N-1 feature information, the Nth feature information, and the second feature information to obtain second spliced ​​feature information; Feature re-extraction is performed on the second splicing feature information to obtain first feature information of the reconstructed image block.

19. The method according to any one of claims 2 to 11, characterized in that: The obtaining the enhanced image block according to the first feature information includes: Nonlinear mapping is performed on the first feature information of the reconstructed image block to obtain the enhanced image block.

20. The method according to claim 19, characterized in that The non-linear mapping of the first feature information of the reconstructed image block to obtain the enhanced image block includes: The first feature information of the reconstructed image block is nonlinearly mapped through a reconstruction module to obtain the enhanced image block.

21. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Decoding a code stream to obtain a first flag, where the first flag is used to indicate whether quality enhancement of a reconstructed image block of the current image block is allowed; The step of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block includes: According to the first flag, when it is determined that quality enhancement of the reconstructed image block is allowed, quality enhancement of the reconstructed image block is performed based on the quantization parameter to obtain the enhanced image block.

22. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Based on the quantization parameter, the reconstructed image block is quality enhanced to obtain a test enhanced image block; determining a first image quality of the test enhanced image block and a second image quality of the reconstructed image block; The step of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block includes: If the first image quality is greater than the second image quality, the test enhanced image block is determined as the enhanced image block of the reconstructed image block.

23. The method according to any one of claims 1 to 11, characterized in that The reconstructed image block is a reconstructed image block of the current image block under the first component.

24. The method according to claim 23, wherein The first component is a luminance component or a chrominance component.

25. The method according to any one of claims 1 to 11, characterized in that The step of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block includes: performing normalization processing on the reconstructed image block and the quantization parameter; The enhanced image block is obtained based on the normalized reconstructed image block and the quantization parameter.

26. An image processing method, characterized in that: include: Determining a quantization parameter of a current image block, and encoding the current image block based on the quantization parameter to obtain a quantization coefficient of the current image block; Dequantizing the quantization coefficients based on the quantization parameters of the current image block to obtain a residual block of the current image block; Obtaining a reconstructed image block of the current image block according to the residual block; Based on the quantization parameter, quality enhancement is performed on the reconstructed image block to obtain an enhanced image block.

27. The method according to claim 26, characterized in that The step of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block includes: performing feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image block; The enhanced image block is determined according to the first feature information.

28. The method according to claim 27, characterized in that The performing feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image block includes: Based on the quantization parameter, perform feature weighting processing on the i-1th feature information of the reconstructed image block to obtain the i-th feature information of the reconstructed image block, where i is a positive integer from 1 to N, and repeat the process to obtain the Nth feature information of the reconstructed image block. If i is 1, the i-1th feature information is the reconstructed image block; Determine first feature information of the reconstructed image block according to the Nth feature information.

29. The method according to claim 28, characterized in that The step of performing feature weighting processing on the i-1th feature information of the reconstructed image block based on the quantization parameter to obtain the i-th feature information of the reconstructed image block includes: Extracting M pieces of feature information of different scales from the (i-1)th feature information, where M is a positive integer greater than 1; Performing weighted processing on the M feature information of different scales to obtain i-th weighted feature information; The i-th feature information is determined according to the i-th weighted feature information.

30. The method according to claim 29, wherein The extracting M pieces of feature information of different scales from the (i-1)th feature information includes: Feature information of M different scales of the (i-1)th feature information is extracted through the first feature extraction layers of M different scales.

31. The method according to claim 30, wherein The first feature extraction layer includes a convolution layer, and different first feature extraction layers include convolution kernels of different sizes.

32. The method according to claim 31, characterized in that Among the M first feature extraction layers of different scales, at least one first feature extraction layer includes an activation function.

33. The method according to claim 29, wherein The weighted processing of the M feature information of different scales to obtain the i-th weighted feature information includes: Splicing the M pieces of feature information at different scales to obtain first spliced ​​feature information; The first splicing feature information is weighted to obtain the i-th weighted feature information.

34. The method according to claim 33, wherein The weighted processing of the splicing feature information to obtain the i-th weighted feature information includes: Performing weighted processing on the first splicing feature information through a weighted processing layer to obtain weighted feature information having a first number of channels; The i-th weighted feature information is obtained according to the weighted feature information having the first number of channels.

35. The method according to claim 34, wherein The number of channels of the i-th weighted feature information is the same as the number of channels of the i-1-th feature information.

36. The method according to claim 34, wherein The weighted processing layer includes a neuron attention mechanism.

37. The method according to any one of claims 29 to 36, characterized in that The determining the i-th feature information according to the i-th weighted feature information includes: The sum of the i-th weighted feature information and the i-1-th feature information is determined as the i-th feature information.

38. The method according to any one of claims 29 to 36, characterized in that The obtaining the i-th feature information according to the i-th weighted feature information includes: The i-th weighted feature information is determined as the i-th feature information.

39. The method according to any one of claims 28 to 36, characterized in that The performing feature weighting processing on the reconstructed image block based on the quantization parameter to obtain first feature information of the reconstructed image block includes: extracting second feature information of the reconstructed image block based on the quantization parameter; Perform feature weighting processing on the second feature information to obtain first feature information of the reconstructed image block.

40. The method according to claim 39, wherein The extracting the second feature information of the reconstructed image block based on the quantization parameter includes: splicing the reconstructed image block and the quantization parameter to obtain splicing information; Feature extraction is performed on the splicing information to obtain the second feature information.

41. The method according to claim 40, wherein The extracting features from the splicing information to obtain the second feature information includes: The second feature information is obtained by performing feature extraction on the splicing information through a second feature extraction module.

42. The method according to claim 39, wherein The determining, according to the Nth feature information, the first feature information of the reconstructed image block includes: The first feature information of the reconstructed image block is obtained according to at least one of the first N-1 feature information of the Nth feature information and the Nth feature information.

43. The method according to claim 42, characterized in that The obtaining, based on at least one of the first N-1 pieces of feature information of the Nth feature information and the Nth feature information, the first feature information of the reconstructed image block includes: splicing at least one of the first N-1 feature information, the Nth feature information, and the second feature information to obtain second spliced ​​feature information; Feature re-extraction is performed on the second splicing feature information to obtain first feature information of the reconstructed image block.

44. The method according to any one of claims 27 to 36, wherein: The obtaining the enhanced image block according to the first feature information includes: Nonlinear mapping is performed on the first feature information of the reconstructed image block to obtain the enhanced image block.

45. The method according to claim 44, wherein The non-linear mapping of the first feature information of the reconstructed image block to obtain the enhanced image block includes: The first feature information of the reconstructed image block is nonlinearly mapped by a reconstruction module to obtain the enhanced image block, and the reconstruction module includes at least one convolution layer.

46. ​​The method according to any one of claims 26 to 36, wherein: The method further comprises: Obtaining a first flag, where the first flag is used to indicate whether quality enhancement of a reconstructed image block of the current image block is allowed; The step of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block includes: According to the first flag, when it is determined that quality enhancement of the reconstructed image block is allowed, quality enhancement of the reconstructed image block is performed based on the quantization parameter to obtain the enhanced image block.

47. The method according to any one of claims 26 to 36, wherein: The method further comprises: Based on the quantization parameter, the reconstructed image block is quality enhanced to obtain a test enhanced image block; determining a first image quality of the test enhanced image block and a second image quality of the reconstructed image block; The step of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block includes: If the first image quality is greater than the second image quality, the test enhanced image block is determined as the enhanced image block of the reconstructed image block.

48. The method according to any one of claims 26 to 36, wherein: The method further comprises: A first flag is written into the bitstream, where the first flag is used to indicate whether quality enhancement of the reconstructed image block of the current image block is allowed.

49. The method according to any one of claims 26 to 36, wherein: The reconstructed image block is a reconstructed image block of the current image block under the first component.

50. The method according to claim 49, wherein The first component is a luminance component or a chrominance component.

51. The method according to any one of claims 26 to 36, wherein: The step of performing quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block includes: performing normalization processing on the reconstructed image block and the quantization parameter; The enhanced image block is obtained based on the normalized reconstructed image block and the quantization parameter.

52. An image processing device, characterized in that include: A decoding unit, used to decode the code stream and obtain the quantization coefficient of the current image block; a determining unit, configured to determine a quantization parameter corresponding to a current image block, and dequantize the quantization coefficient based on the quantization parameter to obtain a transform coefficient of the current image block; a reconstruction unit, configured to determine a reconstructed image block of the current image block according to the transformation coefficient; The enhancement unit is used to input the reconstructed image block and the quantization parameter into the enhancement model to perform image enhancement to obtain an enhanced image block.

53. An image processing device, characterized in that include: a determining unit, configured to determine a quantization parameter of a current image block, and encode the current image block based on the quantization parameter to obtain a quantization coefficient of the current image block; an inverse quantization unit, configured to inverse quantize the quantization coefficients based on the quantization parameters of the current image block to obtain a residual block of the current image block; a reconstruction unit, configured to obtain a reconstructed image block of the current image block according to the residual block; The enhancement unit is configured to perform quality enhancement on the reconstructed image block based on the quantization parameter to obtain an enhanced image block.

54. A video decoder, characterized in that including processor and memory; The memory shown is used to store computer programs; The processor is configured to call and execute the computer program stored in the memory to implement the method according to any one of claims 1 to 25.

55. A video encoder, characterized in that including processor and memory; The memory shown is used to store computer programs; The processor is configured to call and execute the computer program stored in the memory to implement the method according to any one of claims 26 to 51.

56. A coding and decoding system, characterized in that include: The video decoder according to claim 54; And a video encoder according to claim 55.

57. A computer-readable storage medium, characterized in that For storing computer programs; The computer program enables a computer to execute the method according to any one of claims 1 to 25 or 26 to 51 above.

58. A code stream, characterized in that The code stream is generated by the method according to any one of claims 26 to 51.

Citation Information

Cited By

  • Image processing method, processing equipment and storage medium

    CN120568062A

  • Image processing method, processing equipment and storage medium

    CN120583237A

  • Image data processing method based on depth nonlinear model prediction

    CN120783174A