Image compression artifact removal method inspired by decomposition model

Through alternating iterative updates of the decomposition model and deep reconstruction subnetwork, the problem of complex texture area removal problems and lack of interpretability in the image compression artifact removal method is solved, and image quality improvement and network interpretability are achieved.

CN118982476AActive Publication Date: 2024-11-19TAIYUAN UNIVERSITY OF SCIENCE AND TECHNOLOGY
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202411128734.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-19
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

The prior art is difficult to effectively remove compression artifacts during image compression, especially in complex texture areas, and deep learning methods lack interpretability and cannot fully utilize image structure and texture complementary characteristics.

Method used

A decomposition model-inspired image compression artifact removal method is proposed. The image is reconstructed through the structural feature extraction module and the texture feature extraction module, and the structure and texture features are updated alternately using the proximal gradient descent algorithm and the deep reconstruction subnetwork to construct Interpretable image compression artifact removal network STNet.

Benefits of technology

It has achieved significant improvement in image quality, has the interpretability of network topology, can better remove compression artifacts, and improve the structure and detail clarity of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982476A_ABST
    Figure CN118982476A_ABST
Patent Text Reader

Abstract

The invention relates to an image compression artifact removal method inspired by a decomposition model, which belongs to the field of image quality enhancement, aims to solve the problem of low-quality compressed image restoration, and comprises the following steps of: firstly, constructing an image compression artifact removal optimization model based on structure texture decomposition; the model is expanded into an interpretable image compression artifact removal network STNet, the network mainly comprises a structural feature extraction module and a texture feature extraction module, and the two modules are alternately iteratively updated and optimize the structural features and the texture features of the image in a progressive enhancement mode; and finally, performing pixel-by-pixel addition operation on a structure image and a texture image which are respectively output by the structure feature extraction module and the texture feature extraction module to obtain an output enhanced image. The reconstruction performance of the method provided by the invention is superior to that of a plurality of existing image compression artifact removal methods in image quality objective evaluation indexes PSNR, SSIM and PSNR-B.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention belongs to the field of image quality enhancement, and in particular relates to an image compression artifact removal method inspired by a decomposition model. Background Art

[0002] With the popularity of multimedia social software such as Wechat, Facebook, Tiktok and Instagram for posting photos and videos, the number of photos posted on the Internet has increased significantly, making it difficult for existing network communication equipment to bear the transmission of massive data. In order to cope with this problem, lossy image compression standards and video compression standards such as JPEG, AVC, HEVC, etc. are usually used to encode images, and the increasing demand for network bandwidth caused by the sharp increase in transmission data is alleviated by reducing the number of bits of transmitted data. It is worth noting that JPEG is still widely used in smartphones and smart computers. The main steps of JPEG lossy compression include: first, the input image is divided into several 8×8 blocks, and each 8×8 block is transformed by DCT to obtain DCT coefficients; then, the DCT coefficients of each block are quantized using a non-uniform quantization table to reduce storage space; finally, the quantized coefficients are losslessly encoded. However, in the image compression process, the quantization operation is irreversible. At the same time, the coarse quantization operation will cause the loss of high-frequency details of the image and cause discontinuity between pixels due to block processing. These factors often lead to obvious visual artifacts in the compressed image, such as discontinuous block artifacts, ringing artifacts, and color distortion artifacts between blocks. Since these artifacts not only affect the user's visual experience, but also are not conducive to subsequent image processing and various computer vision tasks, more and more scholars are trying to use image compression artifact removal technology to solve this problem. Many advanced image enhancement methods have been proposed to improve the quality of JPEG compressed images. Generally speaking, image compression artifact removal methods can be classified into three types: image filtering-based methods, prior regularization-based optimization methods, and deep learning-based methods.

[0003] In early image filtering-based methods, researchers usually manually designed a filter operator in the pixel domain or frequency domain to remove compression artifacts. For example, Reeve et al. performed Gaussian filtering on pixels around the boundaries of coding blocks in the pixel domain to smooth block artifacts. Buades et al. proposed to predict the filter output by weighted average of pixels in the adjacent area. Chen et al. performed a low-pass filter on the DCT coefficients of adjacent coding blocks in the frequency domain to remove high-frequency noise. Kauff et al. proposed a shape-adaptive discrete cosine transform algorithm that can adaptively determine the optimal shape of each block. However, since most compression artifacts are introduced by highly nonlinear irreversible quantization operations, and these filtering algorithms are highly complex and lack adaptability, this manually designed method cannot effectively deal with the problem of compression artifact removal in complex texture areas. In order to overcome the above problems, many researchers regard image compression artifact removal as an optimization problem and try to add various prior constraints to the optimization model. For example, some typical compression artifact removal methods use sparse representation, low-rank prior, and non-local similarity to construct regularization constraints. Although this prior regularization optimization method can achieve better artifact removal effects, most algorithms based on prior regularization require complex iterative optimization, which makes these methods often time-consuming.

[0004] In recent years, with the continuous development of deep learning, deep learning-based methods can directly learn the end-to-end nonlinear mapping from compressed images to corresponding clean images. For example, Dong et al. proposed a four-layer shallow network ARCNN to learn the mapping function. Inspired by deep residual learning, Zhang et al. proposed a layer convolutional neural network DnCNN based on residual learning. There are also many researchers exploring deep learning models based on dual domains. However, methods based on deep learning still have some defects. For example, as the network becomes deeper and deeper, the role of the internal modules of the network is difficult to understand and analyze, and the network lacks interpretability. Unlike black-box deep learning networks, the deep unfolding network DUN has good network topology structure interpretability. Therefore, this research has received increasing attention in the field of image processing. These methods usually use degradation models to build optimization models and unfold them into interpretable networks. The deep unfolding network DUN can effectively combine the advantages of good interpretability of model-based methods and powerful learning ability of CNN. For example, Fu et al. used sparse convolutional dictionary learning to build a JPEG artifact removal model, solved the model through an iterative optimization algorithm, and unfolded it into an interpretable network. Although the existing deep unrolling network (DUN) methods take into account the prior characteristics of the image and use these characteristics to optimize the unrolling network, these networks do not fully utilize the structural and texture complementary characteristics of the image itself, making these methods unable to better improve the quality of compressed images. Summary of the invention

[0005] The present invention is a method for removing image compression artifacts inspired by a decomposition model, which is dedicated to solving the problem of compressed image restoration. The network includes two modules: a structural feature extraction module SFEM and a texture feature extraction module TFEM. The initialized structural image and texture image are reconstructed by these two modules, so that the structure and details of the reconstructed image can be clearer.

[0006] In order to achieve the above object, the present invention is implemented by the following technical scheme: a method for removing image compression artifacts inspired by a decomposition model, comprising the following steps:

[0007] Step 1) constructing training data for an image compression artifact removal model;

[0008] Use the JPEG standard encoder to compress all images in the DIV2K dataset to obtain a compressed image training dataset paired with the original input image;

[0009] Step 2) constructing an image compression artifact removal optimization model based on structure texture decomposition;

[0010] Firstly, the JPEG standard encoder compressed image is filtered using the L0 gradient minimization smoothing method to obtain a structure map. The texture map is obtained by subtracting the structure map from the compressed image. Then, the structure map and the texture map are respectively passed through the structure initialization subnetwork ISN-T and the texture structure initialization subnetwork ISN-T to obtain the initialized feature map. Then, the maximum a posteriori probability model is used to mathematically model the relationship between the input image X, the structure map S and the texture map T. Finally, the probability model is transformed into an image compression artifact removal optimization model based on structure-texture decomposition.

[0011] Step 3) expanding the image compression artifact removal optimization model based on structure texture decomposition in step 2) into an interpretable network;

[0012] The proximal gradient descent algorithm is used to solve the image compression artifact removal optimization model obtained in step 2), and it is expanded into an interpretable image compression artifact removal network STNet, which includes an initialization subnetwork and a deep reconstruction subnetwork.

[0013] Step 4) Build the initialization subnetwork

[0014] The structure map and texture map obtained in step 2) are respectively subjected to image smoothing filtering using 3×3 convolution kernels with all weights of 1 to obtain preliminary denoised structure map and texture map, and then the preliminary denoised structure map and texture map are stacked with the structure map and texture map obtained in step 2) along the channel dimension and sent to the initialization subnetwork. Next, the first three feature maps are extracted from the output features of the initialization subnetwork of the structure map and the initialization subnetwork of the texture map as the initialization S0 and T0, and the remaining feature maps and It is sent to the next stage as supplementary information;

[0015] Step 5) Build a deep reconstruction subnetwork

[0016] Using the structural feature extraction module and the texture feature extraction module of the deep reconstruction subnetwork obtained in step 3), the structural features and the texture features are continuously updated in an alternating iterative manner respectively;

[0017] Step 6) Loss Function

[0018] The image enhanced by the interpretable image compression artifact removal network STNet and the original undistorted image are taken as input of the loss function, and the reconstruction loss constrained by the L1 norm is used to supervise the learning of the network.

[0019] Furthermore, the specific method for constructing the training data of the compression artifact removal model in step 1) is:

[0020] First, the DIV2K dataset containing 1000 images is compressed using the JPEG standard encoder. Here, 800 images are used for STNet training, 100 images are used for STNet verification, and 100 images are used for STNet testing. Each image in the 800-image dataset is cropped by random cropping, and it is cropped into image blocks of size 128×128 and used as the training dataset.

[0021] Furthermore, the calculation process of the image compression artifact removal optimization model based on structure texture decomposition in step 2) is:

[0022] The input compressed image X is filtered by the L0 gradient minimization smoothing method to obtain the structure graph. The prior distribution of the low-quality compressed image Y is P(Y)∝e -αg(Y) , the implicit prior distribution of X guided by the low-quality compressed graph Y is P(X|Y)∝e -μΩ(X|Y) , where g(Y) and Ω(X|Y) are energy functions about X and Y, α and μ are two hyperparameters. The following formula can be obtained by using the Bayesian formula:

[0023]

[0024] Among them, P(Y) is the probability distribution of Y, which is independent of X, and P(Y|X) is a marginal distribution of Y. Here, the required X is decomposed into S and T, X=S+T, where S is the structure map and T is the texture map, and the prior distribution of S and T is Assuming that S and T are independent of each other, substituting X=S+T into equation (1), we obtain the following equation:

[0025]

[0026] Use the maximum a posteriori probability estimation MAP formula to solve the posterior probability P(S, T|Y). The above formula can be transformed into a maximization optimization problem:

[0027]

[0028] By performing a negative logarithmic transformation on equation (3), the above equation can be transformed into an energy minimization model, which can be expressed as:

[0029]

[0030] This model is an image compression artifact removal optimization model based on structural texture decomposition. The first term of formula (4) is the data term, and the second and third terms of formula (4) are two regularization constraint terms.

[0031] Furthermore, the specific steps of step 3) are as follows: first, the proximal gradient descent algorithm is used to transform the optimization problem of formula (4) into two sub-optimization problems; then, the gradient descent algorithm and the proximal mapping algorithm are used to optimize the two sub-problems respectively; finally, they are iteratively optimized in sequence, and the optimization solution formulas of the two sub-problems can be written as:

[0032] s (k) =S (k-1) -η S (S (k-1 )-(YT (k-1) )) (5)

[0033]

[0034] t (k) =T (k-1) -η T (T (k-1) -(YS k )) (7)

[0035]

[0036] Among them, k represents the index value of the iteration, ηS and η T are the learnable step size parameters for updating S and T respectively, and are the proximal operators of the implicit priors f1(S) and f2(T), respectively. By solving formula (6) and formula (8) alternately and iteratively, we can obtain the iteratively updated structure graph S: k and texture map T k , and finally the last updated S K and T K Perform element-by-element addition to obtain the enhanced image X.

[0037] Furthermore, the deep reconstruction subnetwork in step 5) is composed of K stages, each stage is composed of a structural feature extraction module SFEM and a texture feature extraction module TFEM;

[0038] Structural feature extraction module SFEM: The initialized structure map and texture map input to this module are processed by the gradient descent algorithm shown in formula (5) to obtain a 3-channel image; for the approximate mapping of the proximal mapping algorithm shown in formula (6), the 3-channel image obtained by the gradient descent algorithm of formula (5) is first stacked with the remaining feature map of the previous stage in the channel dimension, and then passed through five residual blocks, with the number of channels remaining unchanged; finally, the output feature map with 67 channels takes the first 3 feature maps as the updated, and the remaining 64 feature maps Entered into the next stage as supplementary information;

[0039] Texture feature extraction module TFEM: The initial texture map input to this module and the structure map updated by the structural feature enhancement module SFEM are used to obtain a 3-channel image through the gradient descent algorithm shown in formula (7). The proximal mapping algorithm of formula (8) is constructed by cascading 5 residual blocks. The first 3 feature maps are used as the updated T, and the remaining 64 feature maps are used as the updated T. Entered into the next stage as supplementary information.

[0040] Furthermore, in step 6), the image enhanced by STNet and the original undistorted image X GT As the input of the loss function, the reconstruction loss with L1 norm constraint is used to supervise the learning of the interpretable image compression artifact removal network STNet. The overall loss function can be written as:

[0041]

[0042] Where N is the image X GT The total number of pixels, K is the total number of stages of STNet, X0 is the initial enhanced image, X i is the enhanced image of the i-th stage.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The present invention first proposes an image compression artifact removal method inspired by a decomposition model, then selects a proximal gradient descent algorithm to solve the model, and finally expands the algorithm model into an image compression artifact removal optimization model based on structural texture decomposition. Compared with other compression artifact removal methods based on black box networks, the present invention has the interpretability of network topology structure.

[0045] 2. The deep reconstruction subnetwork of the present invention has two submodules: the structural feature extraction module SFEM and the texture feature extraction module TFEM, which are used to optimize the structural features and texture features of the image respectively. These two modules are updated alternately and iteratively and optimize the structural features and texture features of the image in a progressively enhanced manner, so that the structural features and texture features can complement each other, and ultimately the reconstructed image has higher quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of an interpretable image compression artifact removal network STNet of the present invention.

[0047] Figure 2 It is a schematic diagram of the alternating iteration of the structural feature extraction module and the texture feature extraction module of the present invention.

[0048] Figure 3 Schematic diagram of the structural graph and texture graph initialization network of the present invention.

[0049] Figure 4 It is a schematic diagram of the structure feature enhancement module SFEM and the texture feature enhancement module TFEM of the kth stage of the present invention.

[0050] Figure 5 Schematic diagram of the proximal operator of the present invention.

[0051] Figure 6 This is a visual quality comparison chart of the method of the present invention and three existing compression artifact removal methods on the "Barbara" image of the dataset Classic5. DETAILED DESCRIPTION

[0052] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. In order to better understand the image enhancement method of the present invention, the network structure of the present invention is first described in detail.

[0053] The present invention provides a method for removing image compression artifacts inspired by a decomposition model, wherein the network structure is as follows: Figure 1 The specific implementation steps are as follows:

[0054] Step 1) construct training data for the image compression artifact removal model; use the JPEG standard encoder to compress all images of the DIV2K dataset to obtain a compressed image training dataset paired with the original input image.

[0055] Step 2) constructing an image compression artifact removal optimization model based on structure texture decomposition;

[0056] First, the JPEG standard encoder compressed image is filtered using the L0 gradient minimization smoothing method to obtain a structure map. The texture map is obtained by subtracting the structure map from the compressed image. Then, the maximum a posteriori probability model is used to mathematically model the relationship between the input image X, the structure map S, and the texture map T. The specific process of mathematical modeling is as follows:

[0057] The input compressed image X is processed by L o The gradient minimization smoothing method is used to filter and obtain the structure graph; the prior distribution of the low-quality compressed graph Y is P(Y)∝e -αg(Y) , the implicit prior distribution of X guided by the low-quality compressed graph Y is P(X|Y)∝e -μΩ(X|Y) , where g(Y) and Ω(X|Y) are energy functions related to X and Y, and α and μ are two hyperparameters. The following formula can be obtained using the Bayesian formula:

[0058]

[0059] Among them, P(Y) is the probability distribution of Y, which is independent of X, and P(Y|X) is a marginal distribution of Y. Here, we decompose the required X into S and T, that is, X=S+T, where S is the structure map and T is the texture map, and the prior distribution of S and T is Assuming that S and T are independent of each other, substituting X=S+T into equation (1), we get the following equation:

[0060]

[0061] Here, the maximum a posteriori probability estimation MAP formula is used to solve the posterior probability P(S, T|Y). Formula (2) can be transformed into the following maximization optimization problem:

[0062]

[0063] By performing a negative logarithmic transformation on equation (3), the above equation can be transformed into an energy minimization model, which can be expressed as:

[0064]

[0065] This model is an optimization model for removing image compression artifacts based on structural texture decomposition.

[0066] Step 3) Expand the image compression artifact removal optimization model based on structure-texture decomposition into an interpretable network.

[0067] The proximal gradient descent algorithm is used to solve the optimization model obtained in step 2) and expand it into an interpretable image compression artifact removal network STNet, which consists of two parts: an initialization subnetwork and a deep reconstruction subnetwork.

[0068] The proximal gradient descent algorithm is used to transform the optimization problem of formula (4) into two sub-optimization problems; then, the gradient descent algorithm and the proximal mapping algorithm are used to optimize the two sub-problems respectively; finally, they are iteratively optimized and solved in sequence. The optimization solution formulas for the two sub-problems can be written as:

[0069] s (k) =S (k-1) -η S (S (k-1) -(YT (k-1) )) (5)

[0070]

[0071] t (k) =T (k-1) -η T (T (k-1) -(YS k )) (7)

[0072]

[0073] Among them, k represents the index value of the iteration, η S and η T are the learnable step size parameters for updating S and T respectively, and are the proximal operators corresponding to the implicit priors f1(S) and f2(T), respectively. By solving formula (6) and formula (8) alternately and iteratively, we can get the iteratively updated structure graph S k and texture map T k , and finally the last updated S K and T K Perform element-by-element addition to obtain the enhanced image X.

[0074] Step 4) Build the initialization subnetwork

[0075] The structure map and texture map obtained in step 2) are respectively subjected to image smoothing filtering using 3×3 convolution kernels with all weights of 1 to obtain preliminary denoised structure map and texture map, and then the preliminary denoised structure map and texture map are stacked with the structure map and texture map obtained in step 2) along the channel dimension and sent to the initialization module, and the outputs of the structure map initialization module and the texture map initialization module are respectively taken as the initialized S0 and T0, and the remaining feature maps are sent to the next stage as supplementary information.

[0076] Step 5) Build a deep reconstruction subnetwork

[0077] The deep reconstruction sub-network is a network that expands formula (6) and formula (8). In this algorithm, due to the two proximal operators and Due to the presence of regularization, f1(S) and f2(T) cannot be derived explicitly, so a deep convolutional neural network is used to learn to update s. (k) and t (k) ; The deep reconstruction subnetwork consists of K stages, each stage consists of a structural feature enhancement module SFEM and a texture feature enhancement module TFEM.

[0078] Structural feature extraction module SFEM: The network that expands formula (5) and formula (6) is called SFEM. This module is mainly used to update the structural features of the image. The initial structure map and texture map input to this module are processed by the gradient descent algorithm shown in formula (5) to obtain a 3-channel image. For the approximate mapping of the proximal mapping algorithm shown in formula (6), the 3-channel image obtained by the gradient descent algorithm of formula (5) is first stacked with the remaining feature map of the previous stage in the channel dimension, and then passed through five residual blocks, and the number of channels remains unchanged. Finally, the output feature map with 67 channels takes the first 3 feature maps as the updated S, and the remaining 64 feature maps Entered into the next stage as supplementary information;

[0079] Texture feature extraction module TFEM: The network that expands formula (7) and formula (8) is called TFEM. This module is mainly used to update the texture features of the image. The initialization texture map input to this module and the structure map updated by the structural feature enhancement module SFEM are used through the gradient descent algorithm shown in formula (7) to obtain a 3-channel image. The proximal mapping algorithm of formula (8) is constructed by cascading 5 residual blocks. The first 3 feature maps are used as the updated T, and the remaining 64 feature maps are used as the updated T. Entered into the next stage as supplementary information.

[0080] Step 6) Loss Function

[0081] The image fused with the structure map and the texture map and the original undistorted image are taken as the input of the loss function, and the reconstruction loss constrained by the L1 norm is used to supervise the learning of an interpretable deep learning image compression artifact removal network driven by the structure-texture decomposition task.

[0082] Network training parameter configuration

[0083] First, the DIV2K dataset containing 1000 images is compressed using the JPEG standard encoder. Here, 800 images are used for STNet training, 100 images are used for STNet verification, and 100 images are used for STNet testing. Each image in the 800-image dataset is cropped by random cropping, cropped into image blocks of size 128×128 and used as the training dataset. For data augmentation, random horizontal flipping and 90-degree rotation are performed. The model is trained for a total of 100 epochs, and the Adam optimizer is used to train the network. The initial learning rate is 1e-3, and the learning rate is divided by 5 every 25 epochs. The number of channels of the convolution filter of the STNet network is 64, the size of the convolution kernel is set to 3×3, the number of iterative optimization stages of the STNet network is 10, and the batch size is set to 64. The present invention is implemented using the PyTorch deep learning framework, and all experiments are trained and tested using the graphics card NVIDIARTXA6000. In order to conduct fair testing, five widely used test data sets are used here, including LIVE1, Classic5, BSD68, Set14, and Set5.

[0084] Network training

[0085] The STNet-enhanced image and the original undistorted image X GT As the input of the loss function, the reconstruction loss with L1 norm constraint is used to supervise the learning of the interpretable image compression artifact removal network STNet. The overall loss function can be written as:

[0086]

[0087] Where N is the image X GT The total number of pixels, K is the total number of stages of STNet. X0 is the initial enhanced image, X i is the enhanced image of the i-th stage.

[0088] The above is a detailed review of the method of the present invention. The following is an evaluation of the enhancement method proposed in the present invention using three objective indicators, namely, peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and PSNR-B, as well as a subjective indicator.

[0089] 1. Comparison of objective indicators of network test results

[0090] In order to verify the feasibility of the method of this patent, it is compared with the JPEG compression algorithm and three representative methods for removing compressed image artifacts, namely DNCNN-15, FBCNN, and RCDNet. Table 1 compares the average PSNR / SSIM / PSNR-B of the above methods on the LIVE1, Classic5, BSD68, Set14 and data sets, where the best results are in bold and the second best results are underlined. DNCNN-15 is trained using the BCDS500 training set, and FBCNN is trained using the DIV2K and Flick2K training sets. For a fairer comparison, the RCDNet method is retrained on the NVIDIA RTXA6000 device using the same training data set as this patent. It can be observed from Table 1 that the average PSNR / SSIM / PSNR-B of the method of this patent is better than that of the other five methods in these five data sets. The main contribution of the method of this patent is to propose a decomposition model-inspired image compression artifact removal method. The structure map enhancement module and the texture map enhancement module of the deep reconstruction sub-network are used to continuously enhance the structural features and texture features in an alternating iterative manner, respectively, so that the quality of the reconstructed image is higher. Therefore, the method of this patent is better than these five representative compressed image artifact removal methods.

[0091]

[0092]

[0093] 2. Comparison of subjective indicators of network test results

[0094] In order to verify the effectiveness of the method of the present invention, the method of the present invention is compared with DNCNN-15, FBCNN, and RCDNe. Figure 6 The following are visualization comparisons of different enhancement methods on the "Barbara" image of the Classic5 dataset. From these three figures, it can be observed that the visual quality of the restored image by the patented method is higher, which fully verifies the superiority of the method of the present invention.

[0095] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A decomposition model-inspired image compression artifact removal method, Features: The following steps are involved: Step 1) constructing training data for an image compression artifact removal model; Use the JPEG standard encoder to compress all images in the DIV2K dataset to obtain a compressed image training dataset paired with the original input image; Step 2) constructing an image compression artifact removal optimization model based on structure texture decomposition; First, using L 0 The gradient minimization smoothing method filters the JPEG standard encoder compressed image to obtain a structure map, and the texture map is obtained by subtracting the structure map from the compressed image; then, the structure map and the texture map are respectively passed through the structure initialization sub-network ISN-S and the texture structure initialization sub-network ISN-T to obtain the initialized feature map, and then the maximum a posteriori probability model is used to mathematically model the relationship between the input image X, the structure map S and the texture map T; finally, the probability model is transformed into an image compression artifact removal optimization model based on structure texture decomposition; Step 3) expanding the image compression artifact removal optimization model based on structure texture decomposition in step 2) into an interpretable network; The proximal gradient descent algorithm is used to solve the image compression artifact removal optimization model obtained in step 2), and it is expanded into an interpretable image compression artifact removal network STNet, which includes an initialization subnetwork and a deep reconstruction subnetwork. Step 4) Build the initialization subnetwork The structure map and texture map obtained in step 2) are respectively subjected to image smoothing filtering using 3×3 convolution kernels with all weights of 1 to obtain a preliminary denoised structure map and texture map. The preliminary denoised structure map and texture map are then stacked with the structure map and texture map obtained in step 2) along the channel dimension and sent to the initialization subnetwork. Next, the first three feature maps are extracted from the output features of the initialization subnetwork of the structure map and the initialization subnetwork of the texture map as the initialized structure map S. 0 and texture map T 0 , the remaining feature maps and It is sent to the next stage as supplementary information; Step 5) Build a deep reconstruction subnetwork Using the structural feature extraction module and the texture feature extraction module of the deep reconstruction subnetwork obtained in step 3), the structural features and the texture features are continuously updated in an alternating iterative manner respectively; Step 6) Loss Function The image enhanced by the interpretable image compression artifact removal network STNet and the original undistorted image are used as the input of the loss function, and L 1 The learning of the network is supervised by a reconstruction loss with a norm constraint.

2. The method for removing image compression artifacts inspired by a decomposition model according to claim 1, Features: The specific method for constructing the training data of the compression artifact removal model in step 1) is: First, the DIV2K dataset containing 1000 images is compressed using the JPEG standard encoder. Here, 800 images are used for STNet network training, 100 images are used for STNet network verification, and 100 images are used for STNet network testing. Each image in the 800 image dataset is cropped by random cropping, and it is cropped into image blocks of size 128×128 and these image blocks are used as training datasets.

3. The method for removing image compression artifacts inspired by a decomposition model according to claim 1, Features: The calculation process of the image compression artifact removal optimization model based on structure texture decomposition in step 2) is: By L 0 The gradient minimization smoothing method filters the input compressed image X to obtain the structure graph. The prior distribution of the low-quality compressed image Y is P(Y)∝e -αg(Y) , the implicit prior distribution of X guided by the low-quality compressed graph Y is P(X|Y)∝e -μΩ(X|Y) , where g(Y) and Ω(X|Y) are energy functions about X and Y, α and μ are two hyperparameters. The following formula can be obtained by using the Bayesian formula: Among them, P(Y) is the probability distribution of Y, which is independent of X, and P(Y|X) is a marginal distribution of Y. Here, the required X is decomposed into S and T, x = S + T, where S is the structure map and T is the texture map, and the prior distribution of S and T is Assuming that S and T are independent of each other, substituting X=S+T into equation (1), we obtain the following equation: Use the maximum a posteriori probability estimation MAP formula to solve the posterior probability P(S,T|Y). The above formula can be transformed into a maximization optimization problem: By performing a negative logarithmic transformation on equation (3), the above equation can be transformed into an energy minimization model, which can be expressed as: This model is an image compression artifact removal optimization model based on structural texture decomposition. The first term of formula (4) is the data term, and the second and third terms of formula (4) are two regularization constraint terms.

4. The method for removing image compression artifacts inspired by a decomposition model according to claim 1, Features: The specific steps of step 3) are as follows: first, the proximal gradient descent algorithm is used to transform the optimization problem of formula (4) into two sub-optimization problems; then, the gradient descent algorithm and the proximal mapping algorithm are used to optimize the two sub-problems respectively; finally, they are iteratively optimized in sequence, and the optimization solution formulas of the two sub-problems can be written as: s (k) =S (k-1) -η S (S (k-1) -(Y-T (k-1) )) (5) t (k) =T (k-1) -η T (T (k-1) -(Y-S k )) (7) Among them, k represents the index value of the iteration, η S and η T are the learnable step size parameters for updating S and T respectively, and They are respectively about the implicit prior f 1 (S) and F 2 (T), and by alternately iterating and solving formula (6) and formula (8), we can obtain the iteratively updated structure graph S k and texture map T k , and finally the last updated S K and T K Perform element-by-element addition to obtain the enhanced image X.

5. The method for removing image compression artifacts inspired by a decomposition model according to claim 1, Features: In step 5), the deep reconstruction subnetwork consists of K stages, each stage consists of a structural feature extraction module SFEM and a texture feature extraction module TFEM; Structural feature extraction module SFEM: The initialized structure map and texture map input to this module are processed by the gradient descent algorithm shown in formula (5) to obtain a 3-channel image; for the approximate mapping of the proximal mapping algorithm shown in formula (6), the 3-channel image obtained by the gradient descent algorithm of formula (5) is first stacked with the remaining feature map of the previous stage in the channel dimension, and then passed through five residual blocks, and the number of channels remains unchanged; finally, the output feature map with 67 channels takes the first 3 feature maps as the updated S, and the remaining 64 feature maps are input into the next stage as supplementary information; Texture feature extraction module TFEM: The initialization texture map input to this module and the structure map updated by the structural feature enhancement module SFEM are used through the gradient descent algorithm shown in formula (7) to obtain a three-channel image. The proximal mapping algorithm of formula (8) is constructed by cascading 5 residual blocks. The first 3 feature maps are used as the updated T, and the remaining 64 feature maps are input into the next stage as supplementary information.

6. The method for removing image compression artifacts inspired by a decomposition model according to claim 1, Features: In step 6), the image enhanced by the STNet network and the original undistorted image X GT As the input of the loss function, we use L 1 The norm-constrained reconstruction loss is used to supervise the learning of the interpretable image compression artifact removal network STNet. The overall loss function can be written as: Where N is the image X GT The total number of pixels, K is the total number of STNet stages, X 0 is the initial enhanced image, X i is the enhanced image of the i-th stage.

Citation Information

Patent Citations

  • Digital restoration method for flaking diseases of ancient wall paintings based on compressed sensing, and intelligent terminal system

    CN107481204A

  • Artifact removing method based on machine learning and an artifact removing model training method and device

    CN110677649A

  • JPEG image compression artifact elimination algorithm based on cascade residual coding and decoding network

    CN112509094A

  • JPEG image deblocking method based on singular value residual constraint

    CN118195946A

  • Compression artifacts reduction method based on dual-stream multi-path recursive residual network

    WO2021042270A1