Method, device and equipment for repairing compression distortion of coded video image and storage medium

Abstract

The application provides a method, device and equipment for repairing compression distortion of an encoded video image and a storage medium, the method comprising: obtaining an encoded code stream of a current frame of a video image; decoding the encoded code stream of the current frame of the video image to obtain a decoded image; parsing a code stream parameter set in the encoded code stream to obtain parameter structured data, using the parameter structured data as code stream prior information, and performing data preprocessing on the code stream prior information to obtain prior information data; fusing the prior information data and the decoded image, inputting fusion information into a pre-trained neural network, and using the pre-trained neural network to repair distortion of the decoded image. Through the application, the performance of repairing compression distortion of an encoded video image can be optimized.

Description

Technical Field

[0001] This application relates to the field of video image post-processing technology, and in particular to a method, apparatus, device and storage medium for repairing compression distortion in coded video images. Background Technology

[0002] To save space, video images are encoded before transmission. Complete video encoding can include processes such as prediction, transformation, quantization, entropy coding, and filtering.

[0003] Current mainstream video image encoders operate based on blocks (such as coding units, prediction units, transform units, filtering units, etc.), and encoding distortion will occur in the above process.

[0004] For example, in the intra-frame prediction stage, inaccurate prediction can lead to artifact noise; in the inter-frame prediction stage, inaccurate motion estimation can lead to trailing; in the transform quantization stage, residual coefficient quantization loss can lead to loss of detail (blurring), color distortion, and checkerboard patterns; in the filtering stage, the original pixel similarity can be destroyed, resulting in mosaic and color gradation; and the coding quality of different inter-frames varies greatly, resulting in the breathing effect. Summary of the Invention

[0005] In view of this, this application provides a method, apparatus, device and storage medium for repairing compression distortion of encoded video images.

[0006] Specifically, this application is implemented through the following technical solution:

[0007] According to a first aspect of the embodiments of this application, a method for restoring compression distortion of encoded video images is provided, comprising:

[0008] Obtain the encoded bitstream of the current frame video image;

[0009] Decode the encoded bitstream of the current frame video image to obtain a decoded image; and,

[0010] The set of parameters in the encoded bitstream is parsed to obtain parameter-type structured data. The parameter-type structured data is used as the prior information of the bitstream, and the prior information of the bitstream is preprocessed to obtain prior information data.

[0011] The prior information data is fused with the decoded image, and the fused information is input into a pre-trained neural network to repair the distortion of the decoded image.

[0012] According to a second aspect of the embodiments of this application, an apparatus for repairing compression distortion of encoded video images is provided, comprising:

[0013] The acquisition unit is used to acquire the encoded bitstream of the current frame video image;

[0014] A decoding unit is configured to decode the encoded bitstream of the current frame video image to obtain a decoded image; and,

[0015] The preprocessing unit is used to parse the bitstream parameter set in the encoded bitstream to obtain parameter class structured data, use the parameter class structured data as bitstream prior information, and perform data preprocessing on the bitstream prior information to obtain prior information data.

[0016] The repair unit is used to fuse the prior information data with the decoded image, and input the fused information into a pre-trained neural network to repair the distortion of the decoded image using the pre-trained neural network.

[0017] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor;

[0018] The processor is used to execute machine-executable instructions to implement the method provided in the first aspect.

[0019] According to a fourth aspect of the embodiments of this application, a machine-readable storage medium is provided, wherein a plurality of computer instructions are stored on the machine-readable storage medium, and when the computer instructions are executed by a processor, the method provided in the first aspect is implemented.

[0020] As can be seen from the above technical solutions, in this embodiment, by acquiring the encoded bitstream of the current frame video image, on the one hand, the encoded bitstream of the current frame video image is decoded to obtain a decoded image; on the other hand, the bitstream parameter set in the encoded bitstream is parsed to obtain parameter-type structured data. The parameter-type structured data is used as prior information of the bitstream, and the prior information of the bitstream is preprocessed to obtain prior information data. Then, the prior information data and the decoded image can be fused, and the fused information is input into a pre-trained neural network. The pre-trained neural network is used to repair the distortion of the decoded image. By introducing prior information data in the process of using the neural network for distortion repair, and using the prior information data to assist the neural network in distortion repair, the neural network's ability to repair encoded distortion and its generalization ability are improved, thus optimizing the distortion repair effect. Attached Figure Description

[0021] Figure 1 This is a schematic flowchart illustrating an exemplary embodiment of a method for restoring compression distortion in encoded video images.

[0022] Figure 2 This is a schematic diagram illustrating the implementation process of a neural network-based coded video image compression distortion repair scheme according to an exemplary embodiment of this application;

[0023] Figure 3 This is a schematic diagram illustrating a priori information data in an exemplary embodiment of this application;

[0024] Figure 4A This is a schematic diagram illustrating the direct fusion of prior information data and LQ according to an exemplary embodiment of this application;

[0025] Figure 4B This is a schematic diagram illustrating the direct fusion of prior information data and LQ in a U-Net network under an exemplary embodiment of this application;

[0026] Figure 5 This is a schematic diagram illustrating feature fusion of prior information data and LQ according to an exemplary embodiment of this application;

[0027] Figure 6 This is a schematic diagram illustrating a priori information data and LQ multi-scale fusion in an exemplary embodiment of this application;

[0028] Figure 7 This is a schematic diagram illustrating the fusion of prior information data and LQ spatial domain attention in an exemplary embodiment of this application;

[0029] Figure 8 This is a schematic diagram illustrating the fusion of prior information data and LQ hierarchical classification according to an exemplary embodiment of this application;

[0030] Figure 9 This is a schematic diagram of the structure of an coded video image compression distortion repair device according to an exemplary embodiment of this application;

[0031] Figure 10 This is a schematic diagram of the hardware structure of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, some technical terms involved in the embodiments of this application will be explained below.

[0033] Post-processing: Video image post-processing refers to the further processing of video image data after encoding, transmission, and decoding, including de-coding distortion, noise reduction, and super-resolution. Post-processing can be implemented through software or hardware devices and is typically used to improve subjective quality.

[0034] Prior information: Information that can be obtained after the encoded video image is decoded by the decoder, including the predicted value in the prediction stage, the division of transform units (TU), the division of coding units (CU), the QP value of each CU unit, the number of bits required for encoding each CU unit, the value of whether each CU unit is skipped during inter-frame prediction (SKIP), the reconstructed value before filtering (the predicted value plus the residual coefficient after inverse transform), the boundary strength value (BS) of deblocking filtering, etc.

[0035] To make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0036] Please see Figure 1 This is a flowchart illustrating a method for repairing compression distortion in encoded video images according to an embodiment of this application. This method can be applied to decoding devices, such as... Figure 1 As shown, the method for restoring compression distortion in encoded video images may include the following steps:

[0037] Step S100: Obtain the encoded bitstream of the current frame video image.

[0038] In this embodiment of the application, the current frame video image refers to the video frame image that the decoding device is currently decoding.

[0039] Step S110: Decode the encoded bitstream of the current frame video image to obtain the decoded image.

[0040] Step S120: Parse the bitstream parameter set in the encoded bitstream to obtain parameter-type structured data. Use the parameter-type structured data as the bitstream prior information and perform data preprocessing on the bitstream prior information to obtain prior information data.

[0041] In this embodiment of the application, for the current frame video image, when the decoding device obtains the current frame video image, it can decode the encoded bitstream of the current frame video image to obtain a decoded image; wherein, the decoded image is a low-quality image (LQ), which needs to be improved by distortion repair.

[0042] On the other hand, in order to optimize the distortion repair effect and improve the distortion repair performance, the decoding device can also obtain the prior information of the bitstream based on the encoded bitstream of the current frame video image, and perform data preprocessing on the prior information of the bitstream to obtain prior information data.

[0043] For example, prior information data can be used to assist neural networks in repairing distortion in decoded images, so that the neural networks can perform distortion repair better.

[0044] For example, parameter class structured data can be obtained by parsing the set of parameters in the encoded bitstream, and this parameter class structured data can be used as prior information of the bitstream.

[0045] In one example, parameter-type structured data may include some or all of the following parameters:

[0046] Structured data including video image-level parameters, coding unit parameters, transform unit parameters, prediction mode parameters, quantization parameters, number of bits, and filtering parameters.

[0047] Step S130: Fuse the prior information data with the decoded image, and input the fused information into a pre-trained neural network to repair the distortion of the decoded image using the pre-trained neural network.

[0048] In this embodiment of the application, when the decoded image and prior information data can be obtained, the input features of the pre-trained neural network (a neural network for encoding video image compression distortion repair) can be determined based on the prior information data and the decoded image. Based on the determined input features, the pre-trained neural network is used to repair the distortion of the decoded image.

[0049] For example, prior information data can be incorporated into the input features of a neural network at different locations, at different scales, and / or using different fusion methods to improve the neural network's ability to repair coding distortion.

[0050] For example, the prior information data obtained in step S120 can be fused with the decoded image to obtain fused information.

[0051] For example, the above fusion may include channel-dimensional stitching, which may include stitching prior information data and decoded image in the channel dimension; and / or, performing feature extraction on prior information data and stitching the obtained prior information features and decoded image in the channel dimension.

[0052] For example, once the fused information is obtained, it can be input into a pre-trained neural network to perform distortion repair on the decoded image.

[0053] For example, prior information may include mask-type prior information and / or non-mask-type prior information.

[0054] Among them, mask-type prior information can guide the network to focus on distorted areas in the decoded image; non-mask-type prior information can provide additional detailed information, such as unfiltered prior information, which can provide relatively original detailed information, thereby optimizing the distortion repair effect of the neural network.

[0055] For example, different fusion methods can be used to fuse different types of prior information data with the decoded image.

[0056] For example, the width and height of the prior information data can be the same as the width and height of the decoded image, respectively.

[0057] Since prior information needs to be fed into the network to be learned along with the decoded image, it is usually necessary to ensure that the width / height of the prior information is consistent with the width / height of the decoded image.

[0058] In one example, the number of channels in the prior information data can also be the same as the number of channels in the decoded image.

[0059] It can be seen that, in Figure 1 In the illustrated method flow, by acquiring the encoded bitstream of the current frame video image, on the one hand, the encoded bitstream of the current frame video image is decoded to obtain the decoded image; on the other hand, the bitstream parameter set in the encoded bitstream is parsed to obtain parameter-type structured data. The parameter-type structured data is used as prior information of the bitstream, and the prior information of the bitstream is preprocessed to obtain prior information data. Then, the prior information data and the decoded image can be fused, and the fused information is input into a pre-trained neural network. The pre-trained neural network is used to perform distortion repair on the decoded image. By introducing prior information data in the process of using the neural network for distortion repair, the neural network's ability to repair encoded distortion and its generalization ability are improved, thus optimizing the distortion repair effect.

[0060] In some embodiments, the above-described fusion of prior information data and decoded image may include:

[0061] The prior information data, as well as the decoded image, are stitched together along the channel dimension.

[0062] For example, after obtaining prior information data in the manner described above, the prior information data and the decoded image can be stitched together in the channel dimension to obtain stitching features. These stitching features are then used as fusion information and input into a pre-trained neural network to perform distortion repair on the decoded image.

[0063] In another example, the above-mentioned fusion of prior information data and decoded image can include:

[0064] Feature extraction is performed on the prior information data to obtain prior information features. The prior information features and the decoded image are then stitched together along the channel dimension, and the stitched features are subjected to channel dimensionality reduction processing.

[0065] For example, after obtaining prior information data in the manner described above, feature extraction can be performed on the obtained prior information data. For instance, feature extraction can be performed on the prior information data using convolution to obtain prior information features. The prior information features, along with the decoded image, can then be stitched together along the channel dimension to obtain stitched features.

[0066] For example, in order to reduce the processing complexity of the neural network, the above-mentioned splicing features can also be aligned and subjected to channel dimensionality reduction processing. The dimensionality-reduced splicing features are then input into a pre-trained neural network, which is used to repair the distortion of the decoded image.

[0067] For example, assuming the prior information data has k types (k≥1) and the number of channels in the decoded image is C, then when the spliced ​​features are obtained through the above feature splicing method, the channel dimension of the spliced ​​features can be reduced to (k+1)*C channels.

[0068] It should be noted that, in the embodiments of this application, during the process of fusing prior information data and decoded images, feature extraction can also be performed on the decoded images. For example, feature extraction can be performed on the decoded images using convolution, and the features of the decoded images and prior information features can be concatenated to obtain concatenated features. The specific implementation of this is not described in detail here.

[0069] Furthermore, in the embodiments of this application, when the splicing features are obtained, it is not necessary to reduce the channel dimension of the splicing features; instead, the splicing features can be used as the input features of the pre-trained neural network.

[0070] As an example, the above-mentioned input of fused information into a pre-trained neural network, and the use of the pre-trained neural network to perform distortion repair on the decoded image, can include:

[0071] The fused information is downsampled at least one scale to obtain fused information at at least one downsampled scale;

[0072] The fused information, and the fused information at at least one downsampling scale, are used as input features of the corresponding scale of the pre-trained neural network, and the pre-trained neural network is used to repair the distortion of the decoded image.

[0073] For example, in order to enhance feature diversity, improve the receptive field of the network, increase the network's sensitivity to details, and improve the network's robustness, the fused information obtained by fusing prior information data and decoded images can be downsampled at least one scale, such as downsampling the fused information by x2 or x4 to obtain fused information at at least one downsampled scale (in the case of including the fused information at the original scale, there are at least two fused information at different scales).

[0074] The original scale fusion information, as well as at least one downsampling scale mentioned above, can be input into different scales of a pre-trained neural network as input features of the corresponding scale of the pre-trained neural network, and the pre-trained neural network can be used to repair the distortion of the decoded image.

[0075] As an example, prior information data may include at least one target prior information data, which is used to indicate areas that require special attention during distortion repair.

[0076] The above-mentioned inputting fused information into a pre-trained neural network and using the pre-trained neural network to perform distortion restoration on the decoded image may include:

[0077] The fused information is input into a pre-trained neural network, and at the decoding end of the pre-trained neural network, the target prior information data is incorporated into the network in a spatial attention manner, and the pre-trained neural network is used to repair the distortion of the decoded image.

[0078] For example, the prior information data includes at least one target prior information data, which is used to indicate areas that require special attention during the distortion repair process.

[0079] For example, the target prior information data may include deblocking filter boundary strength prior information (BS), which includes the boundary strength value of each deblocking filter unit saved when decoding the video image.

[0080] Based on the boundary intensity values ​​of each deblocking filter unit in the decoded image, regions with larger boundary intensity values ​​indicate that these regions have undergone strong filtering in the decoded image and require special attention during the distortion restoration process.

[0081] For example, target prior information data may include quantization parameter prior information (QP) and bit prior information required for coding units (BIT). QP includes the mapped quantization parameters of each coding unit saved when decoding the video image, and BIT includes the bits required to encode each pixel under each coding unit saved when decoding the video image.

[0082] For any coding unit in the decoded image, if the quantization parameter of the coding unit is large, it indicates that the distortion level of the region corresponding to the coding unit is large. At the same time, if the number of bits required for encoding the coding unit is also large, it indicates that the region needs to be focused on.

[0083] Accordingly, in the process of using a pre-trained neural network to repair the distortion of the decoded image, on the one hand, the prior information data and the decoded image can be fused in the manner described above to obtain fused information, and the fused information can be input into the pre-trained neural network; on the other hand, the target prior information data can be integrated into the network in a spatial attention manner, and the pre-trained neural network can be used to repair the distortion of the decoded image.

[0084] In some embodiments, the prior information data may include mask-type prior information data and non-mask-type prior information data; the width, height, and number of channels of the prior information data are consistent with the width, height, and number of channels of the decoded image, respectively.

[0085] The above-mentioned fusion of prior information data and decoded images may include:

[0086] For non-mask-type prior information data and decoded images, stitch them together in the channel dimension; or, extract features from non-mask-type prior information data to obtain prior information features, and stitch the prior information features and decoded images together in the channel dimension.

[0087] By using pixel-by-pixel multiplication and / or pixel-by-pixel addition, the stitching features and mask-type prior information data are fused.

[0088] For example, prior information data may include mask-type prior information data and non-mask-type prior information data.

[0089] For example, the visualization effect of non-masked prior information data is close to that of the visualization effect of decoded images; that is, the visual effect of non-masked prior information data is closer to that of images.

[0090] For example, non-mask-type prior information may include some or all of the prior information such as prediction prior information (PRED), reconstruction prior information before filtering (REC), and transform unit partitioning prior information (TU).

[0091] The visualization effect of mask-type prior information data is similar to that of a mask in a decoded image.

[0092] For example, mask-type prior information may include some or all of the prior information such as deblocking filter boundary strength prior information (BS), quantization parameter prior information (QP), bit prior information required by the coding unit (BIT), and whether the coding unit skips prior information (SKIP).

[0093] For example, in the process of fusing prior information data with the decoded image, for non-mask-type prior information data, it can be fused with the decoded image by concatenating the channel dimensions; or, feature extraction can be performed first, and then the extracted features can be concatenated with the decoded image by channel dimensions to obtain the concatenated features.

[0094] For mask-type prior information data, pixel-by-pixel multiplication and / or pixel-by-pixel addition can be used to fuse the stitched features with the mask-type prior information data.

[0095] In some embodiments, the prior information data may include some or all of the following prior information:

[0096] Prior information on quantization parameters at the image level, image block level, or pixel level; bit prior information.

[0097] Prior information for image block-level prediction, prior information for deblocking filter boundary strength, prior information for transform unit partitioning, and prior information for whether inter-frame prediction skips.

[0098] Pixel-level prior information for reconstructed pixels before filtering.

[0099] In some embodiments, the prior information data includes some or all of the following prior information:

[0100] Predictive Prior Information (PRED) includes the predicted values ​​for each transform unit saved during the decoding of video images.

[0101] Reconstruction prior information before filtering (REC); wherein, the reconstruction prior information before filtering includes the reconstruction value before filtering for each coding unit saved when decoding the video image.

[0102] For example, the reconstructed value before filtering is the predicted pixel plus inverse quantization and inverse transform to obtain the residual coefficients.

[0103] Deblocking filter boundary strength prior information (BS); wherein, the deblocking filter boundary strength prior information includes the boundary strength value of each deblocking filter unit saved when decoding the video image.

[0104] For example, one filtering unit stores one boundary strength value.

[0105] For example, assuming the boundary strength values ​​include 0, 1, and 2, corresponding to no filtering, weak filtering, and strong filtering respectively, then during the decoding of video images, the saved boundary strength values ​​can be mapped to 0~255. The corresponding mapped values ​​(i.e., the saved boundary strength values) can be 0, 127, and 255, representing no filtering, weak filtering, and strong filtering respectively.

[0106] Quantization parameter prior information (QP) includes the mapped quantization parameters of each coding unit saved during video image decoding.

[0107] For example, one filtering unit stores one quantization parameter.

[0108] For example, the quantization parameter range of the encoder can be uniformly mapped to 0~255 to obtain the mapped quantization parameters.

[0109] Transform Unit Partition Prior Information (TU for short); wherein, the transform unit partition prior information includes the average value of the predicted pixel values ​​under each transform unit stored when decoding the video image, or the size of each transform unit.

[0110] For example, the average value of the predicted pixel values ​​under the transform unit can be the sum of the predicted pixel values ​​of each pixel in the transform unit, divided by the area of ​​the transform unit (i.e., the number of pixels in the transform unit).

[0111] For example, the size of the transformation unit corresponding to the prior information of the transformation unit partition can be the width / height of the transformation unit.

[0112] For example, for an 8*8 transformation unit, the value corresponding to the transformation unit in the prior information of the transformation unit partition is 8.

[0113] Bit Prior Information Required for Encoding Units (BIT); wherein, the Bit Prior Information Required for Encoding Units includes the bits required to encode each pixel under each encoding unit, which are stored when decoding the video image.

[0114] For example, the number of bits required to encode each pixel under each coding unit can be the bit value used to encode the coding unit divided by the area of ​​the coding unit, and then the value is mapped to 0~255.

[0115] Whether the encoding unit skips prior information (SKIP); wherein, whether the encoding unit skips prior information includes saving whether each encoding unit skips prior information when decoding video images.

[0116] For example, the skip mode exists only in inter-frame prediction. During the decoding of video images, each coding unit has only one case: skip or not skip. This is mapped to 0 or 255, and whether each coding unit skips is stored.

[0117] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0118] This application provides a neural network-based scheme for restoring compression distortion in encoded video images, such as... Figure 2 As shown, the implementation process of this neural network-based coded video image compression distortion restoration scheme can be as follows:

[0119] S1. By modifying the decoder in the decoding device, the prior information of the bitstream is preprocessed to obtain multiple prior information data that are consistent with the width, height and number of channels of the encoded video image.

[0120] For example, the input to data preprocessing may include structured data such as video image-level parameters, coding unit parameters, transform unit parameters, prediction mode parameters, quantization parameters, bit count, and filtering parameters obtained by the decoder, and the output data is prior information data with a size consistent with the width, height, and number of channels of the encoded video image.

[0121] For example, the scope of prior information data may include:

[0122] Prior information on quantization parameters at the image level, image block level, or pixel level; bit prior information.

[0123] Prior information for image block-level prediction, prior information for deblocking filter boundary strength, prior information for transform unit partitioning, and prior information for whether inter-frame prediction skips.

[0124] Pixel-level prior information for reconstructed pixels before filtering.

[0125] S2. Prior information data is incorporated into the neural network at different locations and scales using addition, multiplication, connection, or cross-product methods to improve the neural network's ability to repair encoding distortion.

[0126] The following sections will explain prior information data and the integration of prior information data.

[0127] I. Prior information data.

[0128] like Figure 3 As shown, the prior information data includes the following prior information:

[0129] Predictive Prior Information (PRED) includes the predicted values ​​for each transform unit saved during the decoding of video images.

[0130] Reconstruction prior information before filtering (REC); wherein, the reconstruction prior information before filtering includes the reconstruction value before filtering for each coding unit saved when decoding the video image.

[0131] For example, the reconstructed value before filtering is the predicted pixel plus inverse quantization and inverse transform to obtain the residual coefficients.

[0132] Deblocking filter boundary strength prior information (BS); wherein, the deblocking filter boundary strength prior information includes the boundary strength value of each deblocking filter unit saved when decoding the video image.

[0133] For example, one filtering unit stores one boundary strength value.

[0134] For example, assuming the boundary strength values ​​include 0, 1, and 2, corresponding to no filtering, weak filtering, and strong filtering respectively, then during the decoding of video images, the saved boundary strength values ​​can be mapped to 0~255. The corresponding mapped values ​​(i.e., the saved boundary strength values) can be 0, 127, and 255, representing no filtering, weak filtering, and strong filtering respectively.

[0135] Quantization parameter prior information (QP) includes the mapped quantization parameters of each coding unit saved during video image decoding.

[0136] For example, one filtering unit stores one quantization parameter.

[0137] For example, the quantization parameter range of the encoder can be uniformly mapped to 0~255 to obtain the mapped quantization parameters.

[0138] Transform Unit Partition Prior Information (TU for short); wherein, the transform unit partition prior information includes the average value of the predicted pixel values ​​under each transform unit stored when decoding the video image, or the size of each transform unit.

[0139] For example, the average value of the predicted pixel values ​​under the transform unit can be the sum of the predicted pixel values ​​of each pixel in the transform unit, divided by the area of ​​the transform unit (i.e., the number of pixels in the transform unit).

[0140] For example, the size of the transformation unit corresponding to the prior information of the transformation unit partition can be the width / height of the transformation unit.

[0141] For example, for an 8*8 transformation unit, the value corresponding to the transformation unit in the prior information of the transformation unit partition is 8.

[0142] Bit Prior Information Required for Encoding Units (BIT); wherein, the Bit Prior Information Required for Encoding Units includes the bits required to encode each pixel under each encoding unit, which are stored when decoding the video image.

[0143] For example, the number of bits required to encode each pixel under each coding unit can be the bit value used to encode the coding unit divided by the area of ​​the coding unit, and then the value is mapped to 0~255.

[0144] Whether the encoding unit skips prior information (SKIP); wherein, whether the encoding unit skips prior information includes saving whether each encoding unit skips prior information when decoding video images.

[0145] For example, the skip mode exists only in inter-frame prediction. During the decoding of video images, each coding unit has only one case: skip or not skip. This is mapped to 0 or 255, and whether each coding unit skips is stored.

[0146] II. Integration of prior information data.

[0147] For example, prior information data can be input into a neural network using different strategies to improve the neural network's ability to repair encoding distortions.

[0148] For example, taking the U-Net neural network as an example, the way prior information data is incorporated can be determined at the loss function level by combining the main sources of encoding distortion.

[0149] 2.1 Direct fusion.

[0150] For example, such as Figure 4A As shown, all prior information data is concatenated with LQ (i.e., decoded image) in the channel dimension (CONCAT) to become 8C channels, and the concatenated features are input into the neural network; where C represents the number of channels of LQ.

[0151] For example, a schematic diagram of the case where the neural network is a U-Net network can be shown as follows: Figure 4B As shown.

[0152] like Figure 4B As shown, H and W are the width and height of the decoded image, and F is the number of channels for the stitching feature.

[0153] 2.2 Feature fusion.

[0154] like Figure 5 As shown, convolution is used to extract features from prior information data, and the extracted prior information features are concatenated with LQ in the channel dimension. Then, convolution is used to process the channel dimension change, resulting in 8C channels, which are then input into the neural network.

[0155] 2.3 Multi-scale fusion.

[0156] For example, all prior information data can be first processed by convolution to extract features, and then concatenated with LQ in the channel dimension; or, all prior information data can be concatenated with LQ in the channel dimension.

[0157] The concatenated features are downsampled at multiple scales and used as inputs to the neural network at different scales.

[0158] like Figure 6 As shown, taking the U-Net neural network as an example, the concatenated features can be downsampled by x2 and x4, and used as the input of the corresponding scale of the neural network.

[0159] 2.4 Spatial attention fusion.

[0160] For example, all prior information data can be first processed by convolution to extract features, and then concatenated with LQ in the channel dimension; or, all prior information data can be concatenated with LQ in the channel dimension, and the concatenated features can be input into the neural network.

[0161] At the neural network decoding end, QP, BIT and BS (i.e., the aforementioned target prior information data) are integrated into the neural network in a spatial attention manner to guide the neural network to pay more attention to the restoration of distorted areas of video images.

[0162] For example, such as Figure 7 As shown, taking the U-Net neural network as an example, the schematic diagram of integrating QP, BIT, and BS into the neural network using spatial attention can be illustrated as follows. Figure 7 As shown.

[0163] 2.5. Hierarchical integration.

[0164] For example, non-mask-type prior information data (such as PRED, REC, and TU) is used for feature extraction through convolution, and then concatenated with LQ in the channel dimension. Then, it is fused with mask-type prior information data (such as BS, SKIP, QP, and BIT) by pixel-by-pixel multiplication and / or pixel-by-pixel addition, and then input into the neural network.

[0165] like Figure 8 As shown, PRED, REC, and TU are extracted using convolution, and then concatenated with LQ along the channel dimension; then multiplied pixel-wise with BS and SKIP; then added pixel-wise with QP and BIT; and finally input into the neural network.

[0166] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0167] like Figure 9 The diagram shown is a flowchart illustrating a video image compression distortion repair device according to an embodiment of this application. This device can be deployed on a decoding device, such as... Figure 9 As shown, the encoded video image compression distortion repair device may include:

[0168] The acquisition unit 910 is used to acquire the encoded bitstream of the current frame video image;

[0169] Decoding unit 920 is used to decode the encoded bitstream of the current frame video image to obtain a decoded image; and,

[0170] The preprocessing unit 930 is used to parse the bitstream parameter set in the encoded bitstream to obtain parameter-type structured data, use the parameter-type structured data as bitstream prior information, and perform data preprocessing on the bitstream prior information to obtain prior information data.

[0171] The repair unit 940 is used to fuse the prior information data with the decoded image, and input the fused information into a pre-trained neural network to repair the distortion of the decoded image using the pre-trained neural network.

[0172] For example, the specific process of obtaining unit 910, decoding unit 920, preprocessing unit 930 and repair unit 940 to repair the compression distortion of encoded video images can be found in the relevant descriptions in the above embodiments, and will not be repeated here in the embodiments of this application.

[0173] Please see Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 1001 and a machine-readable storage medium 1002 storing machine-executable instructions. The processor 1001 and the machine-readable storage medium 1002 can communicate via a system bus 1003. Furthermore, by reading and executing the machine-executable instructions in the machine-readable storage medium 1002 corresponding to the encoded video image compression distortion repair logic, the processor 1001 can execute the encoded video image compression distortion repair method described above.

[0174] The machine-readable storage medium 1002 mentioned herein can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, a machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid-state drive, any type of storage disk (such as optical disc, DVD, etc.), or similar storage media, or combinations thereof.

[0175] In some embodiments, a machine-readable storage medium is also provided, which stores machine-executable instructions that, when executed by a processor, implement the coded video image compression distortion repair method described above. For example, the machine-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0176] This application also provides a computer application that, when executed by a processor, can implement the encoded video image compression distortion repair method disclosed in the above examples of this application.

Claims

1. A method of encoding video image compression distortion repair, characterized by, The method comprises the following steps: obtaining an encoded code stream of a current frame of video image; decoding the encoded code stream of the current frame of video image to obtain a decoded image; and parsing a code stream parameter set in the encoded code stream to obtain parameter type structured data, taking the parameter type structured data as code stream priori information, and performing data preprocessing on the code stream priori information to obtain priori information data; fusing the priori information data and the decoded image, and inputting the fused information into a pre-trained neural network to repair distortion of the decoded image by using the pre-trained neural network; wherein the priori information data comprises mask type priori information data and non-mask type priori information data; the mask type priori information data is used to guide the network to focus on the distortion area in the decoded image, and the non-mask type priori information data is used to provide additional detailed information; the width and height of the priori information data are consistent with the width and height of the decoded image respectively; the mask type priori information data comprises deblocking filter boundary strength priori information and coding unit required bit priori information; the non-mask type priori information data comprises pre-filtering reconstruction priori information; the fusing of the priori information data and the decoded image comprises: splicing the non-mask type priori information data and the decoded image in the channel dimension, or extracting features of the non-mask type priori information data to obtain priori information features, and splicing the priori information features and the decoded image in the channel dimension; fusing the spliced features and the mask type priori information data by using pixel-by-pixel multiplication and / or pixel-by-pixel addition; wherein the fusing of the spliced features and the mask type priori information data by using pixel-by-pixel multiplication and / or pixel-by-pixel addition comprises: performing pixel-by-pixel multiplication on the deblocking filter boundary strength priori information and the spliced features, and performing pixel-by-pixel addition on the coding unit required bit priori information; the priori information data comprises at least one target priori information data used to indicate an area that needs to be focused on in the distortion repair process; the target priori information data comprises deblocking filter boundary strength priori information and coding unit required bit priori information; the inputting of the fused information into the pre-trained neural network and the repairing of distortion of the decoded image by using the pre-trained neural network comprises: inputting the fused information into the pre-trained neural network, and fusing the target priori information data into the network in a spatial attention manner at the decoding end of the pre-trained neural network, and repairing distortion of the decoded image by using the pre-trained neural network; wherein the deblocking filter boundary strength priori information comprises boundary strength values of each deblocking filter unit saved when decoding the video image; the coding unit required bit priori information comprises bits required for coding each pixel under each coding unit saved when decoding the video image; and the pre-filtering reconstruction priori information comprises pre-filtering reconstruction values under each coding unit saved when decoding the video image.

2. The method of claim 1, wherein, The width, height and channel number of the prior information data are consistent with the width, height and channel number of the decoded image respectively; The fusing of the prior information data and the decoded image comprises: splicing the prior information data and the decoded image in a channel dimension; or, extracting features from the prior information data to obtain prior information features, splicing the prior information features and the decoded image in a channel dimension, and performing channel dimension reduction processing on the spliced features.

3. The method of claim 2, wherein, The inputting of the fused information into a pre-trained neural network and the distortion repair of the decoded image by the pre-trained neural network comprise: down-sampling the fused information in at least one scale to obtain fused information of at least one down-sampling scale; inputting the fused information and the fused information of at least one down-sampling scale as input features of corresponding scales of the pre-trained neural network respectively, and performing distortion repair on the decoded image by the pre-trained neural network.

4. The method according to any one of claims 1 to 3, characterized in that, The parameter type structured data comprises part or all of the following parameters: video image level parameters, coding unit parameters, transform unit parameters, prediction mode parameters, quantization parameters, bit numbers, and structured data of filtering parameters.

5. The method according to any one of claims 1 to 3, characterized in that, The prior information data comprises part or all of the following prior information: image level or image block level or pixel level quantization parameter prior information and bit prior information; image block level prediction prior information, deblocking filtering boundary strength prior information, transform unit division prior information, and inter-frame prediction skip prior information; pixel level reconstructed pixel prior information before filtering.

6. The method according to any one of claims 1 to 3, characterized in that, The non-mask type prior information data further comprises: prediction prior information, wherein the prediction prior information comprises prediction values under each transform unit saved when decoding a video image; transform unit division prior information, wherein the transform unit division prior information comprises an average value of prediction pixel values under each transform unit saved when decoding a video image, or a size of each transform unit; The mask type prior information further comprises: quantization parameter prior information, wherein the quantization parameter prior information comprises mapped quantization parameters of each coding unit saved when decoding a video image; coding unit skip prior information, wherein the coding unit skip prior information comprises whether each coding unit is skipped saved when decoding a video image.

7. An apparatus for encoding video image compression distortion repair, characterized by comprising: The method comprises: an acquisition unit configured to acquire a coding bitstream of a current frame of video image; a decoding unit configured to decode the coding bitstream of the current frame of video image to obtain a decoded image; and a preprocessing unit configured to parse a parameter set in the coding bitstream to obtain parameter type structured data, use the parameter type structured data as bitstream prior information, and perform data preprocessing on the bitstream prior information to obtain prior information data; a repair unit configured to fuse the prior information data and the decoded image, input fused information into a pre-trained neural network, and perform distortion repair on the decoded image by the pre-trained neural network. The prior information data includes mask type prior information data and non-mask type prior information data; the mask type prior information data is used to guide the network to focus on a distortion region in a decoded image, and the non-mask type prior information data is used to provide additional detailed information; the width and height of the prior information data are consistent with the width and height of the decoded image; the mask type prior information data includes deblocking filtering boundary strength prior information and coding unit required bit prior information; and the non-mask type prior information data includes pre-filtering reconstruction prior information. The repairing unit fuses the prior information data and the decoded image, including: splicing the non-mask type prior information data and the decoded image in a channel dimension, or extracting features of the non-mask type prior information data to obtain prior information features, and splicing the prior information features and the decoded image in a channel dimension; fusing the spliced features and the mask type prior information data by using a pixel-by-pixel multiplication and / or a pixel-by-pixel addition, wherein the fusing includes performing pixel-by-pixel multiplication on the deblocking filtering boundary strength prior information and the spliced features, and performing pixel-by-pixel addition on the coding unit required bit prior information; The prior information data includes at least one target prior information data used to indicate a region that needs to be focused on in a distortion repairing process; the target prior information data includes deblocking filtering boundary strength prior information and coding unit required bit prior information. The repairing unit inputs the fused information into a pre-trained neural network, and uses the pre-trained neural network to perform distortion repairing on the decoded image, including: inputting the fused information into the pre-trained neural network, and fusing the target prior information data into the network in a spatial attention manner at a decoding end of the pre-trained neural network, and using the pre-trained neural network to perform distortion repairing on the decoded image; The deblocking filtering boundary strength prior information includes boundary strength values of each deblocking filtering unit saved when decoding a video image; the coding unit required bit prior information includes bits required for coding each pixel under each coding unit saved when decoding the video image; and the pre-filtering reconstruction prior information includes pre-filtering reconstruction values under each coding unit saved when decoding the video image.

8. An electronic device, comprising: The electronic device includes a processor and a machine readable storage medium, and the machine readable storage medium stores machine executable instructions that can be executed by the processor; The processor is configured to execute the machine executable instructions to implement the method in any one of claims 1-6.

9. A machine-readable storage medium, characterized in that, The machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1-6.

Images