A video encoding preprocessing method, device and storage medium

Abstract

The application relates to a video coding preprocessing method and device and a storage medium, wherein the method comprises the following steps: S1, converting an RGB color image in an input original video into a YUV color image y; S2, constructing an encoder noise estimation model, inputting the YUV color image y into the encoder noise estimation model to simulate an encoder to generate a loss image y codec ; S3, constructing a multi-scale modular preprocessing model, inputting the loss image y codec into the multi-scale modular preprocessing model to obtain a noise image S4, and obtaining a preprocessed image x based on the noise image. Compared with the prior art, the noise estimation model close to real noise is designed, the quality loss caused by a video encoder can be effectively measured, and the video compression coding efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of video coding preprocessing, and in particular to a video coding preprocessing method, apparatus and storage medium. Background Technology

[0002] Viewing images and videos has become one of the main ways humans obtain information from the outside world. With the increasing demand for high-quality video services, the pressure on video storage and transmission is constantly intensifying. To achieve more efficient video storage and transmission, video encoding and decoding technologies are continuously innovating. However, traditional hybrid coding video standards (264, HEVC, VVC) have reached their limit in balancing computational complexity and compression ratio. In recent years, with deep learning demonstrating superior performance in image processing, intelligent coding technology is bringing new vitality to the field of video coding.

[0003] To address the aforementioned issues, preprocessing is typically performed before encoding to remove stray noise or attenuate irrelevant perceptual details, thereby increasing the encoder's encoding rate. Therefore, preprocessing techniques can improve the encoder's compression ratio while maintaining or even enhancing video perceptual quality. Since any preprocessing model and encoder are connected in series, and the encoder's loss primarily stems from quantization loss, preprocessing must be performed while considering the quantization loss incurred by the encoder.

[0004] Encoding preprocessing models can be broadly categorized into two types: traditional methods and deep learning methods. The former mainly employs certain filtering techniques, considering factors such as perceived brightness, pixel spatial distance, and visual masking effects to achieve effective noise reduction, while the latter primarily uses convolutional neural networks (CNNs) to learn image features to improve the accuracy of the preprocessing model.

[0005] Existing coding preprocessing models mainly suffer from the following shortcomings:

[0006] 1. When processing real data, the denoising capabilities of existing preprocessors need improvement because the training data used by the model differs significantly from the real noise data. Simply using Gaussian noise, additive noise, etc., for mixing and superimposing can lead to overfitting. Therefore, a noise estimation model that closely approximates real noise plays a crucial role in the denoising performance of the denoiser.

[0007] 2. Existing preprocessors are unable to measure the quality loss caused by video encoders. Summary of the Invention

[0008] The purpose of this invention is to overcome the defects of the prior art and provide a video encoding preprocessing method, apparatus and storage medium.

[0009] The objective of this invention can be achieved through the following technical solutions:

[0010] A video encoding preprocessing method includes the following steps:

[0011] S1. Convert the RGB color image in the input raw video into a YUV color image y;

[0012] S2. Construct an encoder noise estimation model, input the YUV color image y into the encoder noise estimation model, and simulate the encoder to generate a loss image y. codec ;

[0013] S3. Construct a multi-scale modular preprocessing model for the lost image y. codec The noise image is obtained by inputting it into the multi-scale modular preprocessing model.

[0014] S4, Based on the noise image The preprocessed image x is obtained.

[0015] Further, step S1 includes the following sub-steps:

[0016] S101, Read the input raw video;

[0017] S102. Convert the original video into a continuous sequence of image frames;

[0018] S103. Convert the RGB storage format of the images in the image frame sequence to the YUV storage format.

[0019] Furthermore, in step S1, the conversion between the RGB color image and the YUV color image conforms to the following relationship:

[0020]

[0021] In the formula, Y, U, and V represent the parameters in the YUV color image, and R, G, and B represent the parameters in the RGB color image.

[0022] Further, step S2 includes the following sub-steps:

[0023] S201. Construct an encoder-based noisy video dataset;

[0024] S202. Based on the encoder noise video dataset, construct an encoder noise estimation model, including five fully convolutional layers, to simulate the encoder generating noisy images y. codec .

[0025] Furthermore, the loss function of the encoder noise estimation model for:

[0026]

[0027]

[0028] In the formula, SSIM represents the loss due to the mean square error of the input and output images, while SSIM represents the structural similarity between the input and output images.

[0029] Furthermore, the multi-scale modular preprocessing model includes a multi-scale convolutional kernel with parallel branches, a residual learning module, and a channel attention module;

[0030] The multi-scale convolution kernel is used to extract multi-scale features from the noisy image, thereby improving the limitations of single-scale feature extraction.

[0031] The residual learning module is used to learn noise features;

[0032] The channel attention module is a squeeze and excitation-based channel attention module used to adaptively allocate channel weights and improve the accuracy of extracted features.

[0033] Furthermore, the loss function of the multi-scale modular preprocessing model for:

[0034]

[0035] In the formula, To minimize the absolute value of the deviation loss, In order to perceive loss, This is the structural similarity loss.

[0036] Furthermore, the expression for the preprocessed image x is:

[0037]

[0038] Wherein, the noise image The expression is:

[0039]

[0040] In the formula, M MCEM (·) indicates that the multi-scale modular preprocessing model applies the loss image y to the multi-scale modular preprocessing model. codec Perform convolution;

[0041] The lost image y codec The expression is:

[0042] y codec =M c (y)

[0043] In the formula, M c (·) indicates that the multi-scale modular preprocessing model convolves the YUV color image y.

[0044] A video encoding preprocessing apparatus includes a memory and a processor. The memory stores a computer program, and the processor can execute a video encoding preprocessing method as described above by calling the computer program instructions in the memory.

[0045] A computer-readable storage medium includes a computer program that can be executed by a processor to implement a video coding preprocessing method as described above.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) The present invention designs a two-stage processing method. First, the encoder noise estimation model is used to simulate the encoder to generate a loss image. Then, a noise image is obtained through a multi-scale modular preprocessing model, thereby obtaining a preprocessed image. This method effectively measures the loss brought to the video by the encoder and controls the overall distortion (preprocessing distortion + encoder quantization distortion) within the range that is not perceptible.

[0048] (2) The present invention constructs a multi-scale modular preprocessing model. The image passes through the multi-scale convolution kernel, residual learning module and channel attention module in parallel branches in sequence, which can obtain a wider receptive field and inter-pixel correlation features.

[0049] (3) The present invention introduces a channel attention mechanism to adaptively assign weights to each channel, which can improve the accuracy of the extracted features.

[0050] (4) The present invention introduces skip connections into the convolutional layer, which can improve the reuse rate of information and avoid overfitting caused by excessive network depth, thereby leading to the loss of shallow features. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0052] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0053] The technical problem to be solved by the present invention is to provide an encoding preprocessing method to help improve encoding compression efficiency.

[0054] Specifically, this invention proposes a video encoding preprocessing method, apparatus, and storage medium that can effectively measure the loss caused by the encoder and remove stray noise that the encoder cannot perceive. Before being input into the encoder, the original video is preprocessed to filter out redundant information in the original image, thereby improving the encoding compression rate.

[0055] The video encoding preprocessing method proposed in this invention can be directly embedded before any encoder, achieving plug-and-play functionality.

[0056] like Figure 1 As shown, the video coding preprocessing method based on multi-scale feature extraction provided in this invention includes the following steps:

[0057] S1. Convert the RGB color image in the input raw video into a YUV color image y;

[0058] S2. Construct an encoder noise estimation model. Input the YUV color image y into the encoder noise estimation model to simulate the encoder generating the loss image y. codec ;

[0059] S3. Construct a multi-scale modular preprocessing model to process the lost image y. codec The input is used in a multi-scale modular preprocessing model to obtain a noisy image.

[0060] S4, Noise-based image The preprocessed image x is obtained.

[0061] Step S1 includes the following sub-steps:

[0062] S101, Read the input raw video;

[0063] S102. Convert the original video into a continuous sequence of image frames;

[0064] S103. Convert the RGB storage format of the images in the image frame sequence to the YUV storage format.

[0065] The YUV color space is another popular color model besides the RGB color space. RGB is widely used in computers, while YUV is used in video encoding and decoding, focusing on color representation. It is closer to people's perceptual experience of color than RGB.

[0066] The conversion between RGB color images and YUV color images follows the following relationship:

[0067]

[0068] In the formula, Y represents the luminance component of the pixel, U and V represent the chrominance components of the pixel; R represents the red component of the pixel, G represents the green component of the pixel, and B represents the blue component of the pixel.

[0069] Step S2 includes the following sub-steps:

[0070] S201. Construct an encoder noise video dataset; In this embodiment, a real encoder noise video dataset with four quantization parameters (22, 27, 32 and 37) is constructed, wherein the value range of the quantization parameter is [0, 51]. The quantization parameter reflects different compression intensities, and the larger the value of the quantization parameter, the greater the video compression intensity.

[0071] S202. Based on the encoder noise video dataset, construct an encoder noise estimation model. The model includes five fully convolutional layers to simulate the encoder generating noisy images y. codec :

[0072] y codec =M c (y)

[0073] M c (·) indicates that the multi-scale modular preprocessing model convolves the YUV color image y.

[0074] In this embodiment, the convolutional layers in the model use the Dncnn network, which is a network model based on residual learning and has shown good performance in denoising.

[0075] In this embodiment, the loss function of the encoder noise estimation model for:

[0076]

[0077]

[0078] In the formula, SSIM represents the loss function for the mean squared error of the input and output images, and SSIM represents the structural similarity between the input and output images. To ensure the accuracy of the model's output in terms of low-frequency information, use Improve the structural similarity between the model's output and the real data.

[0079] In step S3, the multi-scale modular preprocessing model (MCEM) consists of three parts. First, the feature map is processed by multi-scale convolution kernels with parallel branches to improve the limitations of single-scale feature extraction and facilitate the extraction of inter-pixel correlation features. Then, residual learning is performed, and feature learning on the residuals is more conducive to the learning of noisy features. Finally, the channel attention part based on squeezing and excitation is entered, and the channel weights are adaptively allocated to improve the accuracy of extracting important features.

[0080] The convolutional layers use skip connections to improve the problem of shallow feature loss and visual artifacts caused by excessively deep network layers.

[0081] Noise image obtained through multi-scale modular preprocessing model The expression is:

[0082]

[0083] M MCEM (·) indicates that the multi-scale modular preprocessing model applies the loss image y to the multi-scale modular preprocessing model. codec Perform convolution;

[0084] Obtain a noisy image Afterwards, the expression for the preprocessed image x is:

[0085]

[0086] In this embodiment, the loss function of the multi-scale modular preprocessing model for:

[0087]

[0088] In the formula, To minimize the absolute value of the deviation loss, In order to perceive loss, To address structural similarity loss, a perceptual loss component is introduced, which allows the video to retain more details after preprocessing, thereby improving the perceptual quality of the video.

[0089] In summary, this invention utilizes a two-stage processing method: first, it simulates the encoder to generate a loss image, and then it obtains a noisy image, which can more accurately measure the quality loss caused by the video encoder. Secondly, it designs a parallel multi-scale feature extraction module to improve the limitations of single-scale feature extraction methods and retain more detailed information in the video. Finally, by introducing a channel attention mechanism and skip connections, it improves the problems of shallow feature loss and visual artifacts caused by excessively deep network layers.

[0090] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A video coding preprocessing method, characterized in that, Includes the following steps: S1. Convert the RGB color image in the input raw video to a YUV color image. ; S2. Construct an encoder noise estimation model for the YUV color image. The input to the encoder noise estimation model simulates the encoder generating a loss image. ; S3. Construct a multi-scale modular preprocessing model for the lost image. The noise image is obtained by inputting it into the multi-scale modular preprocessing model. ; S4, Based on the noise image The preprocessed image is obtained. ; The multi-scale modular preprocessing model includes a multi-scale convolutional kernel with parallel branches, a residual learning module, and a channel attention module. The multi-scale convolution kernel is used to extract multi-scale features from the noisy image, thereby improving the limitations of single-scale feature extraction. The residual learning module is used to learn noise features; The channel attention module is a squeeze and excitation-based channel attention module used to adaptively allocate channel weights and improve the accuracy of extracted features.

2. The video coding preprocessing method according to claim 1, characterized in that, Step S1 includes the following sub-steps: S101, Read the input raw video; S102. Convert the original video into a continuous sequence of image frames; S103. Convert the RGB storage format of the images in the image frame sequence to the YUV storage format.

3. The video coding preprocessing method according to claim 1, characterized in that, In step S1, the conversion between the RGB color image and the YUV color image conforms to the following relationship: In the formula, Y , U , V These represent the parameters in the YUV color image. R , G , B These represent the parameters in the RGB color image.

4. The video coding preprocessing method according to claim 1, characterized in that, Step S2 includes the following sub-steps: S201. Construct an encoder-based noisy video dataset; S202. Based on the encoder noise video dataset, construct an encoder noise estimation model, including five fully convolutional layers, to simulate the encoder generating noisy images. .

5. The video coding preprocessing method according to claim 1, characterized in that, The loss function of the encoder noise estimation model for: In the formula, The loss is the mean square error of the input and output images. This refers to the structural similarity between the input and output images.

6. The video coding preprocessing method according to claim 1, characterized in that, The loss function of the multi-scale modular preprocessing model for: In the formula, To minimize the absolute value of the deviation loss, In order to perceive loss, This is the structural similarity loss.

7. The video coding preprocessing method according to claim 1, characterized in that, The preprocessed image The expression is: Wherein, the noise image The expression is: In the formula, This indicates that the multi-scale modular preprocessing model applies to the lost image. Perform convolution; The lost image The expression is: In the formula, This indicates that the multi-scale modular preprocessing model is applied to the YUV color image. Perform convolution.

8. A video encoding preprocessing apparatus, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor can execute a video encoding preprocessing method as described in any one of claims 1-7 by calling the computer program instructions in the memory.

9. A computer-readable storage medium, characterized in that, It includes a computer program that can be executed by a processor to implement a video coding preprocessing method as described in any one of claims 1-7.

Images