Image perception pre-processing coding method, system and device and storage medium

Abstract

The embodiment of the invention discloses an image perception pre-processing coding method, system and device and a storage medium. According to the technical scheme provided by the embodiment of the invention, the to-be-coded video image is acquired, the to-be-coded video image is input into the pre-trained pre-processing network model, and the pre-processing network model calculates the loss function based on the regression loss, the structural similarity loss and the discrete cosine transform loss in the training process; iteratively adjusting model parameters of the pre-processing network model based on a loss function until the pre-processing network model converges, and determining discrete cosine transform loss based on frequency domain information of a predicted image of the pre-processing network model; and outputting a pre-processing image of the to-be-coded video image based on the pre-processing network model so as to carry out video image coding based on the pre-processing image. By adopting the technical means, the image quality fidelity of the video image can be ensured while the coding efficiency is ensured, the pre-processing effect of the video image is improved, and the video coding and display effects are further optimized.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video coding technology, and in particular to an image perception pre-processing coding method, system, device and storage medium. Background Art

[0002] Currently, in video encoding scenarios, pre-processing operations are performed on video frames before encoding to remove redundant or high-frequency information that is not easily perceived by the human eye in the video image, in order to improve video encoding efficiency. For example, by transforming the image from the spatial domain to the frequency domain via DCT (Discrete Cosine Transform, offline cosine transform) and then performing low-rank reconstruction, the image is reconstructed with as few frequency components as possible to achieve a perceptual closeness to the original image.

[0003] However, in the related pre-coding processing scheme, the method of transforming the image from the spatial domain to the frequency domain via DCT and then performing low-rank reconstruction can easily lead to excessive erasure of the high-frequency information of the video image, resulting in poor subjective image quality of the video image, which in turn affects the encoding and display effects of the video image. Summary of the invention

[0004] The embodiments of the present application provide an image perception pre-processing encoding method, system, device and storage medium, which can ensure the quality fidelity of video images while ensuring encoding efficiency, and solve the technical problem of poor subjective image quality caused by video image pre-processing.

[0005] In a first aspect, an embodiment of the present application provides an image perception pre-processing encoding method, comprising:

[0006] Obtaining a video image to be encoded, inputting the video image to be encoded into a pre-trained pre-processing network model, calculating a loss function based on regression loss, structural similarity loss, and discrete cosine transform loss during the training process of the pre-processing network model, iteratively adjusting model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges, and the discrete cosine transform loss is determined based on frequency domain information of the predicted image of the pre-processing network model;

[0007] A pre-processed image of the video image to be encoded is output based on the pre-processing network model, so as to encode the video image based on the pre-processed image.

[0008] In a second aspect, an embodiment of the present application provides an image perception pre-processing coding system, comprising:

[0009] An input module is configured to obtain a video image to be encoded, input the video image to be encoded into a pre-trained pre-processing network model, calculate a loss function based on regression loss, structural similarity loss and discrete cosine transform loss during the training process of the pre-processing network model, iteratively adjust the model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges, and the discrete cosine transform loss is determined based on frequency domain information of the predicted image of the pre-processing network model;

[0010] The output module is configured to output a pre-processed image of the video image to be encoded based on the pre-processing network model, so as to encode the video image based on the pre-processed image.

[0011] In a third aspect, an embodiment of the present application provides an image perception pre-processing encoding device, including:

[0012] memory and one or more processors;

[0013] The memory is configured to store one or more programs;

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the image perception pre-processing encoding method as described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions, when executed by a computer processor, are configured to execute the image perception pre-processing encoding method as described in the first aspect.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed on a computer or a processor, the computer or the processor executes the image perception pre-processing encoding method as described in the first aspect.

[0017] The embodiment of the present application obtains a video image to be encoded, inputs the video image to be encoded into a pre-trained pre-processing network model, and during the training process, the pre-processing network model calculates a loss function based on regression loss, structural similarity loss and discrete cosine transform loss, and iteratively adjusts the model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges, and the discrete cosine transform loss is determined based on the frequency domain information of the predicted image of the pre-processing network model; the pre-processed image of the video image to be encoded is output based on the pre-processing network model, so as to encode the video image based on the pre-processed image. The above-mentioned technical means are adopted, and the pre-processing network model is trained by combining regression loss, structural similarity loss and discrete cosine transform loss, so as to pre-process the video image to be encoded based on the pre-processing network model, and the optimized loss function is used to help the pre-processing network eliminate the weak texture and noise in the flat area, and retain most of the important texture details in the image, so as to ensure the quality fidelity of the video image while ensuring the encoding efficiency, improve the pre-processing effect of the video image, and then optimize the video encoding and display effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of an image perception pre-processing encoding method provided by an embodiment of the present application;

[0019] Figure 2 It is a training flow chart of the pre-processing network model in the embodiment of the present application;

[0020] Figure 3 It is a model training schematic diagram of the pre-processing network model in the embodiment of the present application;

[0021] Figure 4 This is a flowchart of the pre-processing network model image pre-processing in the embodiment of the present application;

[0022] Figure 5 It is a structural schematic diagram of an image perception pre-processing coding system provided in an embodiment of the present application;

[0023] Figure 6 It is a structural diagram of an image perception pre-processing encoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0025] The image perception pre-processing coding method provided in this application aims to train a pre-processing network model by combining regression loss, structural similarity loss and discrete cosine transform loss, so as to pre-process the video image to be encoded based on the pre-processing network model, and use the optimized loss function to help the pre-processing network eliminate weak texture and noise in flat areas.

[0026] In the relevant video coding field, there are two compression viewpoints. First, the traditional coding viewpoint actually artificially creates precision defects in the frequency domain to achieve the purpose of bit rate saving. That is, the coefficients after DCT transformation are moderately quantized, and specific frequency components are erased from high frequency to low frequency. The encoder itself does not pay attention to the content in the coding unit, and all content is lossy encoded equally. Secondly, there is the denoising viewpoint, that is, compression is performed by denoising enhancement. Different types of noise are usually introduced in this part of the link from video data acquisition to encoding. Therefore, the signal-to-noise ratio of the coded stream can be enhanced by denoising to improve the coding efficiency. However, this method is very dependent on denoising capabilities, and denoising often erases effective information, which ultimately causes image quality loss.

[0027] To address the above issues, before the video image is formally encoded, the video frame will be pre-processed to remove redundant or high-frequency information that is not easily perceived by the human eye, making it easier to encode. Generally speaking, based on AI technology, most of the existing video pre-processing solutions use the powerful nonlinear mapping capabilities of the AI ​​network to perform video pre-processing from two perspectives: frequency domain low-rank reconstruction and spatial domain perception.

[0028] Among them, in the frequency domain, the most advanced solution is to transform the image from the spatial domain to the frequency domain via DCT and then perform low-rank reconstruction. That is, in the frequency domain, the low-rank goal is followed, the frequency domain is redistributed through the DCT-loss constraint, and the image is reconstructed with as few frequency components as possible to achieve the perceptual closeness to the original image, thereby achieving the purpose of erasing the unimportant non-DC components in the frequency domain, that is, the burrs in the high-frequency components, and removing the corresponding redundant information in the spatial domain. In the AI-based framework, the video frame is used as the network input, and the network outputs the predicted reconstructed frame after calculation and transformation in the feature domain. The PSNR-loss (PSNR, Peak Signal-to-Noise Ratio) of the reconstructed frame and the input video frame in the spatial domain and the DCT-loss in the frequency domain are calculated as the loss constraint for updating the network parameters. Among them, PSNR-loss is used for image quality fidelity, and DCT-loss is used to characterize the degree of erasure of redundant information. Then this method is prone to excessive erasure of high-frequency information of the image, that is, enhancing the denoising ability of the network, but insufficient fidelity of subjective image quality.

[0029] In the spatial domain, the most advanced solution improves coding efficiency by embedding the perception model and adding a virtual encoder during the training phase, while ensuring good subjective image quality. The perception model is used to calculate the perception loss and maintain the subjective image quality. The virtual encoder is added to obtain the coding rate information bpp (bit per pixel) as the coding loss to constrain the network to lean towards a lower bit rate with the same quality. Although this method considers the actual coding information bpp as feedback, the limited performance of the virtual encoder (coding parameters can only be determined under a fixed λ parameter) will cause the pre-processing network to not be effectively applied to the actual coding link.

[0030] Obviously, the existing pre-coding processing schemes still face the problem of imbalance between improving coding efficiency and maintaining image quality, and there is still much room for improvement in the design of the pre-processing network. On the one hand, the lack of constraints on perceptual information will lead to excessive denoising in the pre-processing effect, resulting in poor subjective experience. On the other hand, it is worth considering how to effectively introduce coding information to obtain compressed perception. Therefore, in the face of the stringent requirements of actual application scenarios on coding bit rate and subjective quality, it is still necessary to further explore more efficient video pre-processing solutions.

[0031] Based on this, an image perception pre-processing encoding method of the real-time example of the present application is provided to solve the technical problem of poor subjective image quality caused by pre-processing of video images. By optimizing the loss function to help the pre-processing network eliminate weak textures and noise in flat areas and retain the ability of most important texture details in the image, the encoding BD-rate (Bjontegaarddelta rate, an indicator for evaluating video coding compression efficiency) gain on VMAF (VideoMuitimethod Assessment Fusion) is finally achieved.

[0032] Example:

[0033] Figure 1 A flowchart of an image perception pre-processing coding method provided in an embodiment of the present application is given. The image perception pre-processing coding method provided in this embodiment can be executed by an image perception pre-processing coding device, which can be implemented by software and / or hardware. The image perception pre-processing coding device can be composed of two or more physical entities, or can be composed of one physical entity. Generally speaking, the image perception pre-processing coding device can be a video image processing device such as a computer, a mobile phone, a tablet or a video coding server.

[0034] The following description is made by taking the image perception pre-processing encoding device as an example of the subject of the image perception pre-processing encoding method. Figure 1 , the image perception pre-processing encoding method specifically includes:

[0035] S110, obtaining a video image to be encoded, and inputting the video image to be encoded into a pre-trained pre-processing network model. During the training process, the pre-processing network model calculates a loss function based on regression loss, structural similarity loss, and discrete cosine transform loss, and iteratively adjusts model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges. The discrete cosine transform loss is determined based on frequency domain information of the predicted image of the pre-processing network model.

[0036] When performing pre-processing encoding of video images, the present application combines a pre-processing network model and introduces perceptual loss and DCT-loss (discrete cosine transform loss) in the design of the loss function of the pre-processing network model to improve the image quality preservation capability of the pre-processing network and obtain a certain coding efficiency gain.

[0037] In an actual end-to-end coding link, the collected video images are usually sent to the encoder for encoding after a series of degradation processes such as noise reduction, sharpening, and downsampling. After the encoding is completed, the encoded image is transmitted to the receiving end to meet the corresponding video transmission requirements. Before encoding the video image, the video image is processed based on the image perception pre-processing coding method of the present application to optimize the video image. The video image to be encoded is defined as the video image to be encoded, and the video image to be encoded is input into a pre-trained pre-processing network model to optimize the processing of the video image to be encoded with the help of the pre-trained pre-processing network model. Prior to this, the model is trained with pre-constructed training samples so that the model has the ability to optimize the processing of the video image to be encoded.

[0038] Optionally, before the model training is performed based on the training sample input pre-processing network model, the method further includes:

[0039] Performing degradation processing on the video image of the training sample, where the degradation processing includes at least one of image denoising, sharpening and downsampling;

[0040] The video image to be encoded corresponds to a video image that has been degraded.

[0041] Since video images need to undergo a series of degradation processing operations such as noise reduction, sharpening, and downsampling before pre-processing, the impact of these operators on image characteristics needs to be considered throughout the entire end-to-end link, that is, the input of the pre-processing network model also needs to be an image that has undergone these degradation processing operations. The subsequent encoder receives the output image from the pre-processing network model as input for encoding, and finally transmits it over the network and decodes and plays it at the viewer end, thereby realizing the encoded transmission of video images.

[0042] Based on the above-mentioned video images after degradation processing, they can be used as training samples to train the pre-processing network model. Figure 2 ,The training process of the pre-processing network model includes:

[0043] S1001, constructing training samples with video images, inputting the training samples into a pre-processing network model for model training, and outputting corresponding predicted images;

[0044] S1002, calculating the regression loss and structural similarity loss of the video image and the corresponding predicted image, calculating the discrete cosine transform loss of the predicted image, and calculating the loss function based on the regression loss, structural similarity loss and discrete cosine transform loss;

[0045] S1003. Iteratively adjust the model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges.

[0046] In the process of pre-processing network model training, a large number of high-quality video image datasets are prepared. These images can cover various scenes, lighting conditions and motion complexity to ensure the generalization ability of the model. Then, the images are pre-processed as necessary, such as size normalization, color space conversion, etc., to facilitate model training. A pre-processing network model architecture is designed. The model can receive the original video image as input and output the optimized pre-processed image. According to actual needs and network complexity, the appropriate number of layers and structures can be selected, such as using advanced structures such as residual blocks and attention mechanisms to improve model performance.

[0047] Based on the pre-processing network model architecture designed above, the training samples are input into the model, the corresponding predicted images are output, and then the corresponding loss function is calculated. Based on the loss function, the model parameters can be adaptively adjusted. The above process is repeated until the model converges, that is, the loss function value no longer decreases significantly. After that, the trained pre-processing network model can be used to pre-process the video image to be encoded and generate an optimized pre-processed image.

[0048] Specifically, Figure 3 A training diagram of the pre-processing network model is given. During the network training process, in order to help the model learn the possible degradation methods of images in real scenes and thus better restore the image quality, the input image X1 first undergoes a series of degradation processes to obtain the input X2 of the pre-processing network. Then the predicted image y is output through network learning. At this time, the L1-loss regression loss and the perceived MSSIM-loss structural similarity loss between the predicted image y and the input image X1, as well as the DCT-loss discrete cosine transform loss of the predicted image y itself are calculated to guide the update of the network model parameters. In the actual application link, the trained perceptual pre-processing network model is placed before the encoder, and the predicted image obtained after the encoded image passes through the pre-processing network is sent to the encoder for encoding, and finally decoded and played on the user side.

[0049] Optionally, calculating the regression loss and structural similarity loss of the video image and the corresponding predicted image includes:

[0050] Calculate the regression loss based on the mean absolute error between the video image and the corresponding predicted image;

[0051] The structural similarity loss is calculated based on the average structural similarity between the video image and the corresponding prediction image.

[0052] Regression loss is usually used to measure the pixel-level difference between the predicted image and the original video image. This application uses the mean absolute error (MAE) to calculate the regression loss. For each pixel position (i, j) in the video image, the difference between the predicted image and the pixel corresponding to that position is calculated. Then the differences of all pixel positions are averaged to obtain the regression loss.

[0053] Structural Similarity Index (SSIM) is a measure of the structural similarity between two images. SSIM takes into account the brightness, contrast, and structural information of the image. In order to calculate the structural similarity loss, you first need to calculate the SSIM value and then subtract this value from 1 (because you want to minimize the loss, and the larger the SSIM value, the more similar it is). For video images and predicted images, calculate the mean, variance, and covariance of the local window, calculate the SSIM value of the local window based on the calculation of the mean, variance, and covariance of the local window, calculate the average of the SSIM values, and the structural similarity loss is 1 minus the average of the SSIM values.

[0054] Calculate the discrete cosine transform loss of the predicted image, including:

[0055] Perform DCT transformation on each image block of the predicted image, and scan to obtain the discrete cosine transform frequency domain coefficients of the image block;

[0056] Determine the specified high-frequency component interval coefficient from the discrete cosine transform frequency domain coefficient, and calculate the coefficient mean of the high-frequency component interval coefficient;

[0057] The coefficient amplitudes of the discrete cosine transform frequency domain coefficients that are lower than the coefficient mean are accumulated to obtain the discrete cosine transform loss of the predicted image.

[0058] DCT-loss Discrete Cosine Transform loss characterizes the model's ability to optimize frequency domain information, and indirectly reflects the model's ability to save bit rate. When calculating the discrete cosine transform loss, first perform DCT transform on the image block of the video image size N*N, obtain the DCT frequency domain coefficient of the image block through zigzag scanning, and then calculate the mean of the high-frequency component interval coefficients in the DCT frequency domain coefficients (DCT frequency domain coefficients above the specified threshold) as the threshold for frequency domain erasure, and accumulate the coefficient amplitudes below the threshold to obtain the discrete cosine transform loss.

[0059] Furthermore, the loss function is calculated based on regression loss, structural similarity loss and discrete cosine transform loss, including:

[0060] The first weight coefficient of the structural similarity loss and the second weight coefficient of the discrete cosine transform loss are set based on the regression loss, and the loss function is weightedly calculated based on the regression loss, the structural similarity loss, the first weight coefficient, the discrete cosine transform loss and the second weight coefficient.

[0061] Each component of the loss function needs to adapt different weight parameters according to the actual data level to prevent some loss components from being too heavy and causing other losses to fail or the model to fail to converge. Therefore, in the loss function calculation formula, it is necessary to use the L1-loss regression loss as the baseline to adjust the weight of the MSSIM-loss structural similarity loss and the DCT-loss discrete cosine transform loss to ensure that the model can optimize the perceptual information and frequency domain information on the basis of providing stable spatial reconstruction capabilities. The loss function loss calculation formula is as follows:

[0062] loss = L1 loss +MSSIM loss *w1+DCT loss *w2

[0063] Among them, the loss function loss consists of three parts: L1 loss is the regression loss, which characterizes the reconstruction ability of the model in the spatial domain and is obtained by calculating the mean absolute error (MAE) between the input image and the predicted image; MSSIM loss It is the structural similarity loss, which represents the model's ability to reconstruct perceptual information. It is obtained by calculating the average structural similarity between the input image and the predicted image. loss The discrete cosine transform loss is calculated only by the predicted image itself, which represents the model's ability to optimize frequency domain information.

[0064] In this way, by using the loss function to introduce perceptual information and compression information into the network training process, and through reasonable weight design to control the balance between the network's fidelity to subjective quality and coding benefits, the optimized loss function is used to help the pre-processing network eliminate weak textures and noise in flat areas and retain most of the important texture details in the image, thereby ensuring the quality fidelity of the video image while ensuring coding efficiency.

[0065] Based on the above-trained pre-processing network model, during the model inference process, the degraded video image to be encoded is input into the pre-processing network to optimize the video image to be encoded. By introducing the perceptual MSSIM-loss, perceptual constraints are provided to the network to prevent the network from over-smoothing and denoising; on the other hand, by introducing the DCT-loss constraint, compressed perceptual information is provided to the network, and ultimately, the fidelity of subjective image quality is achieved while obtaining coding gain. In this way, redundant information that is not easily perceived by the human eye can be removed from the image without losing too much subjective quality of the image, making it easy to encode, thereby obtaining coding BD-rate benefits.

[0066] S120, outputting a pre-processed image of the video image to be encoded based on the pre-processing network model, so as to encode the video image based on the pre-processed image.

[0067] The pre-processing network model performs perceptual pre-processing on the video image to be encoded that is input to the model, and outputs the corresponding pre-processed image for use in the subsequent video encoding process. This can effectively improve the performance of video encoding pre-processing, and significantly improve the subjective image quality of the video image while ensuring encoding efficiency.

[0068] Specifically, outputting a pre-processed image of a video image to be encoded based on the pre-processing network model includes:

[0069] The first convolution module based on the pre-processing network model transforms the video image to be encoded from the pixel domain to the feature domain, inputs the image features transformed by the first convolution module into the grouped convolution module for feature learning, inputs the image features output by the grouped convolution module into the second convolution module to perform a convolution skip operation, and outputs a pre-processed image of the video image to be encoded based on the image features output by the second convolution module.

[0070] Before the first convolution module based on the pre-processing network model transforms the video image to be encoded from the pixel domain to the feature domain, it also includes:

[0071] Down-sampling the video image to be encoded and inputting it into the first convolution module;

[0072] Outputting a pre-processed image of the video image to be encoded based on the image features output by the second convolution module includes:

[0073] The image features output by the second convolution module are upsampled to obtain a pre-processed image of the video image to be encoded, and the pre-processed image is output from the pre-processing network model.

[0074] Reference Figure 4When the pre-processing network model processes the video image to be encoded, in order to reduce the feature domain space of the image and reduce the subsequent computing overhead, the video image to be encoded will first be downsampled through the downsampling module pixel Unshuffle, and then transformed from the pixel domain to the feature domain through a 3x3 convolution, and then feature learning is performed through the SCB group convolution module, and finally a series of convolution jump operations and upsampling module pixel Shuffle upsampling are performed to obtain the final predicted image. When upsampling and downsampling images or feature maps, this application selects the pixel Unshuffle / Shuffle method, which can avoid the checkerboard effect and will not introduce additional computing overhead, thereby improving the practicality of the model in the actual link.

[0075] In addition, considering the low latency requirements of the actual end-to-end coding link, this application adopts a lightweight network design to adapt to the end-to-end coding link, that is, using the CNN network and SCB block cascade as the network backbone, and combining up and down sampling modules to enhance the network's learning efficiency in the feature domain.

[0076] In the above, by obtaining the video image to be encoded, the video image to be encoded is input into the pre-trained pre-processing network model. During the training process, the pre-processing network model calculates the loss function based on the regression loss, structural similarity loss and discrete cosine transform loss, and iteratively adjusts the model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges. The discrete cosine transform loss is determined based on the frequency domain information of the predicted image of the pre-processing network model; the pre-processed image of the video image to be encoded is output based on the pre-processing network model, so as to encode the video image based on the pre-processed image. By adopting the above technical means, the pre-processing network model is trained by combining the regression loss, structural similarity loss and discrete cosine transform loss, so as to pre-process the video image to be encoded based on the pre-processing network model, and the optimized loss function is used to help the pre-processing network eliminate the weak texture and noise in the flat area, and retain most of the important texture details in the image, so as to ensure the quality fidelity of the video image while ensuring the encoding efficiency, improve the pre-processing effect of the video image, and then optimize the video encoding and display effect.

[0077] Based on the above embodiments, Figure 5 This is a schematic diagram of the structure of an image perception pre-processing coding system provided by this application. Figure 5 The image perception pre-processing encoding system provided in this embodiment specifically includes: an input module 21 and an output module 22.

[0078] The input module 21 is configured to obtain a video image to be encoded, input the video image to be encoded into a pre-trained pre-processing network model, and during the training process, the pre-processing network model calculates a loss function based on regression loss, structural similarity loss, and discrete cosine transform loss, and iteratively adjusts the model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges, and the discrete cosine transform loss is determined based on the frequency domain information of the predicted image of the pre-processing network model;

[0079] The output module 22 is configured to output a pre-processed image of the video image to be encoded based on the pre-processing network model, so as to perform video image encoding based on the pre-processed image.

[0080] Specifically, the training process of the pre-processing network model includes:

[0081] Construct training samples with video images, input the training samples into the pre-processing network model for model training, and output the corresponding predicted images;

[0082] Calculate the regression loss and structural similarity loss of the video image and the corresponding predicted image, calculate the discrete cosine transform loss of the predicted image, and calculate the loss function based on the regression loss, structural similarity loss and discrete cosine transform loss;

[0083] The model parameters of the pre-processing network model are iteratively adjusted based on the loss function until the pre-processing network model converges.

[0084] The calculation of the regression loss and structural similarity loss between the video image and the corresponding predicted image includes:

[0085] Calculate the regression loss based on the mean absolute error between the video image and the corresponding predicted image;

[0086] The structural similarity loss is calculated based on the average structural similarity between the video image and the corresponding prediction image.

[0087] Calculate the discrete cosine transform loss of the predicted image, including:

[0088] Perform DCT transformation on each image block of the predicted image, and scan to obtain the discrete cosine transform frequency domain coefficients of the image block;

[0089] Determine the specified high-frequency component interval coefficient from the discrete cosine transform frequency domain coefficient, and calculate the coefficient mean of the high-frequency component interval coefficient;

[0090] The coefficient amplitudes of the discrete cosine transform frequency domain coefficients that are lower than the coefficient mean are accumulated to obtain the discrete cosine transform loss of the predicted image.

[0091] The loss function is calculated based on regression loss, structural similarity loss and discrete cosine transform loss, including:

[0092] The first weight coefficient of the structural similarity loss and the second weight coefficient of the discrete cosine transform loss are set based on the regression loss, and the loss function is weightedly calculated based on the regression loss, the structural similarity loss, the first weight coefficient, the discrete cosine transform loss and the second weight coefficient.

[0093] Specifically, outputting a pre-processed image of a video image to be encoded based on the pre-processing network model includes:

[0094] The first convolution module based on the pre-processing network model transforms the video image to be encoded from the pixel domain to the feature domain, inputs the image features transformed by the first convolution module into the grouped convolution module for feature learning, inputs the image features output by the grouped convolution module into the second convolution module to perform a convolution skip operation, and outputs a pre-processed image of the video image to be encoded based on the image features output by the second convolution module.

[0095] Specifically, before the first convolution module based on the pre-processing network model transforms the video image to be encoded from the pixel domain to the feature domain, it also includes:

[0096] Down-sampling the video image to be encoded and inputting it into the first convolution module;

[0097] Outputting a pre-processed image of the video image to be encoded based on the image features output by the second convolution module includes:

[0098] The image features output by the second convolution module are upsampled to obtain a pre-processed image of the video image to be encoded, and the pre-processed image is output from the pre-processing network model.

[0099] In the above, by obtaining the video image to be encoded, the video image to be encoded is input into the pre-trained pre-processing network model. During the training process, the pre-processing network model calculates the loss function based on the regression loss, structural similarity loss and discrete cosine transform loss, and iteratively adjusts the model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges. The discrete cosine transform loss is determined based on the frequency domain information of the predicted image of the pre-processing network model; the pre-processed image of the video image to be encoded is output based on the pre-processing network model, so as to encode the video image based on the pre-processed image. By adopting the above technical means, the pre-processing network model is trained by combining the regression loss, structural similarity loss and discrete cosine transform loss, so as to pre-process the video image to be encoded based on the pre-processing network model, and the optimized loss function is used to help the pre-processing network eliminate the weak texture and noise in the flat area, and retain most of the important texture details in the image, so as to ensure the quality fidelity of the video image while ensuring the encoding efficiency, improve the pre-processing effect of the video image, and then optimize the video encoding and display effect.

[0100] The image perception pre-processing coding system provided in the embodiment of the present application can be configured to execute the image perception pre-processing coding method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0101] Based on the above practical example, the present application embodiment also provides an image perception pre-processing encoding device, referring to Figure 6 , the image perception pre-processing encoding device includes: a processor 31, a memory 32, a communication module 33, an input device 34 and an output device 35. The memory, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the image perception pre-processing encoding method described in any embodiment of the present application (for example, the input module and output module in the image perception pre-processing encoding system). The communication module is configured to perform data transmission. The processor executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory, that is, realizes the above-mentioned image perception pre-processing encoding method. The input device can be configured to receive input digital or character information, and generate key signal input related to the user settings and function control of the device. The output device may include a display device such as a display screen. The above-mentioned image perception pre-processing encoding device provided can be configured to execute the image perception pre-processing encoding method provided in the above-mentioned embodiment, and has corresponding functions and beneficial effects.

[0102] On the basis of the above embodiments, the embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are configured to execute an image perception pre-processing encoding method when executed by a computer processor, and the storage medium may be any of various types of memory devices or storage devices. Of course, the computer-executable instructions of the computer-readable storage medium provided in the embodiments of the present application are not limited to the image perception pre-processing encoding method described above, and may also execute related operations in the image perception pre-processing encoding method provided in any embodiment of the present application.

[0103] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The essence of the technical solution of the present application or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, and includes a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the image perception pre-processing encoding method described in each embodiment of the present application.

Claims

1. An image perceptual pre-processing encoding method, characterized in that: include: Acquire a video image to be encoded, input the video image to be encoded into a pre-trained pre-processing network model, calculate a loss function based on regression loss, structural similarity loss and discrete cosine transform loss during the training process of the pre-processing network model, iteratively adjust the model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges, and the discrete cosine transform loss is determined based on frequency domain information of the predicted image of the pre-processing network model; A pre-processed image of the video image to be encoded is output based on the pre-processing network model, so as to perform video image encoding based on the pre-processed image.

2. The image perceptual pre-processing encoding method according to claim 1, characterized in that: The training process of the pre-processing network model includes: Constructing training samples with video images, inputting the training samples into a pre-processing network model for model training, and outputting corresponding predicted images; Calculating the regression loss and the structural similarity loss of the video image and the corresponding predicted image, calculating the discrete cosine transform loss of the predicted image, and calculating the loss function based on the regression loss, the structural similarity loss and the discrete cosine transform loss; The model parameters of the pre-processing network model are iteratively adjusted based on the loss function until the pre-processing network model converges.

3. The image perceptual pre-processing encoding method according to claim 2, characterized in that: The calculating the regression loss and the structural similarity loss of the video image and the corresponding predicted image includes: Calculating regression loss based on the mean absolute error between the video image and the corresponding predicted image; The structural similarity loss is calculated based on the average structural similarity between the video image and the corresponding predicted image.

4. The image perceptual pre-processing encoding method according to claim 2, characterized in that: The calculating the discrete cosine transform loss of the predicted image comprises: Performing DCT transformation on each image block of the predicted image, and scanning to obtain discrete cosine transform frequency domain coefficients of the image block; Determine a specified high-frequency component interval coefficient from the discrete cosine transform frequency domain coefficient, and calculate a coefficient mean of the high-frequency component interval coefficient; The coefficient amplitudes of the discrete cosine transform frequency domain coefficients that are lower than the coefficient mean are accumulated to obtain the discrete cosine transform loss of the predicted image.

5. The image perceptual pre-processing encoding method according to claim 2, characterized in that: The loss function is calculated based on the regression loss, the structural similarity loss and the discrete cosine transform loss, comprising: The first weight coefficient of the structural similarity loss and the second weight coefficient of the discrete cosine transform loss are set based on the regression loss, and the loss function is weightedly calculated based on the regression loss, the structural similarity loss, the first weight coefficient, the discrete cosine transform loss and the second weight coefficient.

6. The image perceptual pre-processing encoding method according to any one of claims 1 to 5, characterized in that: The outputting the pre-processed image of the video image to be encoded based on the pre-processing network model includes: The first convolution module based on the pre-processing network model transforms the video image to be encoded from the pixel domain to the feature domain, inputs the image features obtained by the first convolution module into the grouped convolution module for feature learning, inputs the image features output by the grouped convolution module into the second convolution module to perform a convolution skip operation, and outputs the pre-processed image of the video image to be encoded based on the image features output by the second convolution module.

7. The image perceptual pre-processing encoding method according to claim 6, characterized in that: Before the first convolution module based on the pre-processing network model transforms the to-be-encoded video image from the pixel domain to the feature domain, it also includes: Downsampling the video image to be encoded and inputting it into the first convolution module; The step of outputting the pre-processed image of the video image to be encoded based on the image features output by the second convolution module includes: The image features output by the second convolution module are upsampled to obtain a pre-processed image of the video image to be encoded, and the pre-processed image is output from the pre-processing network model.

8. An image perception pre-processing coding system, characterized in that: include: An input module is configured to obtain a video image to be encoded, input the video image to be encoded into a pre-trained pre-processing network model, wherein during the training process, the pre-processing network model calculates a loss function based on regression loss, structural similarity loss and discrete cosine transform loss, and iteratively adjusts model parameters of the pre-processing network model based on the loss function until the pre-processing network model converges, wherein the discrete cosine transform loss is determined based on frequency domain information of a predicted image of the pre-processing network model; The output module is configured to output the pre-processed image of the video image to be encoded based on the pre-processing network model, so as to perform video image encoding based on the pre-processed image.

9. An image perception pre-processing encoding device, characterized in that: include: memory and one or more processors; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image perception pre-processing encoding method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a computer processor, are configured to execute the image perception pre-processing encoding method as described in any one of claims 1-7.

11. A computer program product, characterized in that The computer program product includes instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes the image perception pre-processing encoding method as described in any one of claims 1-7.

Images