A hybrid distortion image restoration method based on convolution sparse residual learning

Abstract

The application belongs to the field of image processing, and discloses a hybrid distortion image restoration method based on convolution sparse residual learning. The distortion type and intensity handled by the application are more random in spatial distribution, and are more in line with the image distortion in real scenes. The application uses a learning iterative threshold shrinkage algorithm based on a transpose attention mechanism to solve a multi-scale convolution residual sparse coding optimization problem, and embeds a deep unfolding network into a U-Transformer network, which has the advantages of strong learning ability and good interpretability. Compared with the convolution operation often used in traditional restoration networks, the application introduces a transpose attention mechanism into a restoration network model, models the dependency between channels, and improves the adaptive ability and performance of the network while keeping the parameter quantity small.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a hybrid distortion image restoration method based on convolutional sparse residual learning. Background Technology

[0002] Imaging systems are often affected by various degradation factors, such as atmospheric disturbances, optical system aberrations, and relative motion between the camera and the object, resulting in varying degrees of image degradation. Image restoration refers to the process of recovering a potentially high-quality image from one or more low-quality images. It is an important task in the field of digital image processing, involving numerous signal processing methods and fundamental mathematical theories, such as estimation theory, digital filtering, numerical analysis, and linear algebra. Commonly used image restoration methods include filtering methods and spatial domain iterative methods. The former usually requires knowledge of the image's distortion type and even certain important distortion parameters, and the filtered image will lose some edge and detail information. The latter requires manually selecting regularization parameters to obtain the optimal model, lacks adaptability, and has high computational complexity and time cost. With the development of computer technology, image restoration methods based on machine learning and neural networks have been widely used and are currently the mainstream methods in the field of image restoration. Image restoration technology can be widely used in many fields such as medical image processing, monitoring systems, facial recognition, and satellite remote sensing. It can also be used to improve the image quality of ordinary photographic equipment, enabling it to achieve the same or similar shooting effects as high-performance photographic equipment. This reduces the requirements of users or manufacturers for high performance and high precision of certain key electronic components in photographic equipment, ultimately reducing production costs. Therefore, it has important research value and significance. Summary of the Invention

[0003] The purpose of this invention is to overcome the problems of modeling, representing and removing spatially non-uniform mixed distortion artifacts, and to provide a mixed distortion image restoration method based on convolutional sparse residual learning.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] In a first aspect, the present invention provides a hybrid distortion image restoration method based on convolutional sparse residual learning, comprising the following steps:

[0006] Based on the lossless images of the DIV2K dataset, a spatially non-uniformly mixed distortion image dataset is constructed, which simultaneously contains three distortion types: Gaussian blur, Gaussian noise, and JPEG compression. An end-to-end aggregation excitation network is trained using the spatially non-uniformly mixed distortion image dataset.

[0007] The mixed distortion image to be tested is input into the trained aggregation firing network to estimate the distortion parameters and obtain the corresponding spatial distortion parameter map.

[0008] A multi-scale convolutional residual sparse coding network was trained using a spatially non-uniformly mixed distortion image dataset.

[0009] The mixed-distortion image to be tested and the estimated spatial distortion parameter map are input into the trained multi-scale convolutional residual sparse coding network to obtain the restored image.

[0010] A further improvement of this invention is that the aggregation excitation network adopts a three-branch, four-scale U-Net structure. The encoder output features at each scale are concatenated with the upsampled features of the decoder module through residual connections, and finally output distortion parameter maps after two layers of convolution.

[0011] A further improvement of the present invention is that the encoder of the aggregation excitation network is composed of aggregation excitation modules. The aggregation excitation modules are used to fuse features from different branches to model the interaction between different distortion types, and to concatenate the fused features with features from different branches, thereby modeling the impact of mixed distortion on the parameter estimation of a single distortion type.

[0012] A further improvement of this invention is that the multi-scale convolutional residual sparse coding network adopts a multi-scale U-Transformer network architecture, and the encoder and decoder of the multi-scale convolutional residual sparse coding network are connected through the convolutional residual sparse coding module.

[0013] A further improvement of the present invention is that the output image features of the encoder are processed by the convolutional residual sparse coding module to obtain the restored features, and finally the restored image is reconstructed by the decoder.

[0014] A further improvement of this invention is that the convolutional residual sparse coding module is a deep unfolded network.

[0015] A further improvement of this invention is that the network structure of the deep unfolded network is determined through the optimization solution process of the multi-scale convolutional residual sparse coding problem.

[0016] A further improvement of this invention is that the optimization problem of multi-scale convolutional residual sparse coding can be expressed by formulas (1) to (4):

[0017]

[0018]

[0019]

[0020]

[0021] Among them, e i =x i -x′ i(i = 1, 2, 3, 4) represents the residual characteristics at each scale; and represent common and independent convolutional sparse representations, respectively, where N c and Let c and z represent respectively. i The channel dimension, and satisfying and Let N represent the corresponding convolution filter, where N is the number of convolutional filters. i ={48,72,96,120}; ↓n indicates downsampling the image to its original size. * indicates a convolution operation; and The number of convolutional filters in the middle is equal to e i Channel dimension;

[0022] Solving formulas (1) to (4) is equivalent to solving the following optimization problem:

[0023]

[0024] Where λ is the penalty parameter, formula (5) uses the method of fixing other variables and solving for the remaining variable in turn to solve each variable in the optimization problem. First, fix c and any three z. i Solve for the remaining z i ,get:

[0025]

[0026] Rewrite formula (6) as follows:

[0027]

[0028] in

[0029] Solving formula (7) yields:

[0030]

[0031] Where E i D i θ and θ are independent learnable variables; TB represents the Transformer module; all z are fixed. i Solving for the variable c in formula (5) is equivalent to solving the following optimization problem:

[0032]

[0033] in Rewrite formula (9) as follows:

[0034]

[0035] Where ↑n represents the upsampled image to n times its original size, combining the first four norms in formula (10) yields:

[0036]

[0037] in This represents the concatenation of different feature maps along the channel dimension. Using formula (8), the variable c in formula (11) is solved to obtain:

[0038]

[0039] E and D are independent learnable variables; the definition of TB is the same as that in formula (8).

[0040] Finally, the optimal solution can be obtained by iteratively solving formulas (8) and (12). Therefore, the restored image features can be represented as:

[0041]

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The distortion types and intensities processed by this invention are more random in spatial distribution, which is more consistent with the image distortion situation in real-world scenarios. This invention uses a learning iterative threshold shrinkage algorithm based on transposed attention mechanism to solve the problem of multi-scale convolution residual sparse coding optimization, and embeds a deep unfolded network into a U-Transformer network, which has the advantages of strong learning ability and good interpretability. Compared with the convolution operation often used in traditional restoration networks, this invention introduces the transposed attention mechanism into the restoration network model, models the dependencies between channels, and improves the network's adaptability and performance while keeping the number of parameters small. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the aggregation excitation network (FENet) model framework of the present invention;

[0045] Figure 2 This is a model framework diagram of the Multi-Scale Convolutional Residual Sparse Coding Network (MS-CRSCNet) of the present invention;

[0046] Figure 3 This is a diagram of the Transformer module framework of the present invention;

[0047] Figure 4 This is a framework diagram of the Convolutional Residual Sparse Coding (CRSC) module in MS-CRSCNet of the present invention;

[0048] Figure 5 z in the CRSC module of this invention i Prediction network framework diagram;

[0049] Figure 6 This is a diagram of the c prediction network framework in the CRSC module of the present invention. Detailed Implementation

[0050] To further understand the content of this invention, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments are merely illustrative and not limiting of the invention.

[0051] The hybrid distortion image restoration method based on convolutional sparse residual learning described in this invention comprises two stages of image processing: a first stage of distortion parameter prediction and a second stage of distortion parameter-guided image restoration. The detailed algorithm flow for each stage is as follows:

[0052] Phase 1: This phase uses FENet to predict the mixed distortion parameters of the image, namely Gaussian blur, Gaussian white noise, and JPEG compression. The FENet model is attached. Figure 1 As shown, this network takes the Y channel of a distorted image as input and predicts three distortion parameters using three branches. Each branch employs a U-Net network architecture, and the number of channels is kept the same across all scales to reduce model complexity. To model the joint effect of multiple distortions, the FENet encoder calculates the mean of the feature maps of each branch. Considering the impact of multiple distortions on the prediction of a single distortion parameter, this invention designs a FEB module. This module uses convolutional layers to parse the averaged feature maps of each branch and then concatenates the parsed results to each branch. The FEB module is repeated three times, each time halving the spatial size of the feature map to simulate multi-scale features related to distortion. Subsequently, the feature residuals at each scale are connected to the FENet decoder to perform feature concatenation. Finally, FENet outputs three normalized distortion parameters, namely Gaussian blur parameters. Gaussian noise parameters and JPEG compression parameters

[0053] Phase Two: This phase uses the MS-CRSCNet network to perform image restoration based on distortion parameters. The MS-CRSCNet model is attached. Figure 2As shown in the figure, "TB" represents the Transformer Block. The encoder and decoder of the MS-CRSCNet model contain multiple cascaded Transformer Blocks. The encoder and decoder are connected through the CRSC module, which is mainly used for convolutional sparse coding modeling and representation of multi-scale features. The restored features are then sent to the decoder. The entire network uses bilinear interpolation plus 1×1 convolution to perform upsampling and downsampling operations. The design idea of ​​the CRSC module comes from the assumption proposed in this invention: residual features are sparser than the image, and therefore can be represented by a linear combination of an overcomplete dictionary. Using x... i and x′ i (i = 1, 2, 3, 4) represent the four scale distortion features and the corresponding distortion-free features extracted by the encoder, respectively. Assuming that all dictionaries are known, given an input image of size H × W × 3, the scale features have the following relationship:

[0054]

[0055]

[0056]

[0057]

[0058] Among them, e i =x i -x′ i Represents the residual characteristics at each scale; and represent common and independent convolutional sparse representations, respectively, where N c and Let c and z represent respectively. i The channel dimension, and satisfying and Let N represent the corresponding convolutional filters, where N is the number of filters. i ={48,72,96,120}; ↓n indicates downsampling the image to its original size. * indicates a convolution operation; and The number of convolutional filters in the middle is equal to e i The channel dimension.

[0059] Solving formulas (1) to (4) is equivalent to solving the following optimization problem:

[0060]

[0061] Where λ is the penalty parameter. Formula (5) can be used to solve each variable in the optimization problem sequentially by fixing other variables and solving for the remaining variable. First, fix c and any three z. i Solve for the remaining z i ,get:

[0062]

[0063] Formula (6) can be rewritten as:

[0064]

[0065] in Solving formula (7), we can obtain

[0066]

[0067] Where E i D i θ and θ are learnable and independent variables; S θ It is a soft threshold operator:

[0068] S θ (x)=sign(x)·max(|x|-θ,0)#(9)

[0069] To add nonlinearity and channel attention mechanisms to the optimization process, this invention uses a Transformer module to replace S. θ Therefore, formula (8) can be expressed as:

[0070]

[0071] The network structure of the TB (Transformer module) is shown in the attached figure. Figure 3 As shown, this module mainly consists of a transposed attention module and a gated depthwise separable convolutional feedforward network cascaded together. The transposed attention module first normalizes the input features using LayerNorm, then recombines the features from each channel using transposed attention and a 1×1 convolution, and finally adds the output of the entire module to the original input via residual connections. In the gated depthwise separable convolutional feedforward network, the input features, after layer normalization, pass through two branches in parallel. Each branch contains a 1×1 convolution that maps the features to a high-dimensional space and a 3×3 depthwise separable convolution that encodes spatial neighborhood information. One branch is activated by GELU to provide non-linear representation capabilities. The outputs of the two branches are multiplied element-wise and projected onto the original feature dimension by a 1×1 convolution, then added to the original input to obtain the module's output.

[0072] Then, fix all z iSolving for the variable c in formula (5) is equivalent to solving the following optimization problem:

[0073]

[0074] in Formula (11) can be rewritten as:

[0075]

[0076] Where ↑n represents the upsampled image size being n times the original size. Combining the first four norms in equation (12), we get:

[0077]

[0078] in Here This represents the concatenation of different feature maps along the channel dimension. The variable c in formula (13) can be solved using formula (10) to obtain:

[0079]

[0080] E and D are independent learnable variables; the definition of TB is the same as that in formula (10).

[0081] Finally, the optimal solution can be obtained by iteratively solving formulas (10) and (14). The restored image features can be represented as follows:

[0082]

[0083] The above CRSC optimization method can be implemented by a deep unfolded neural network, the structure of which is shown in the attached figure. Figure 4 As shown in the figure, z i The architectures of the prediction network and the c prediction network are respectively in the appendix. Figure 5 and attached Figure 6 The details are given below. Since the original features are unknown during the testing phase, this invention includes an appendix during implementation. Figure 4 e i =x i and will Initialize to x i Through experimental analysis, this invention sets the iteration count of formulas (10) and (14) to 1, and adds the appended... Figure 4 The number of CRSC modules is set to 3, and different CRSC modules share the same network parameters. For D, E, D... i E i , and The implementation uses a 3×3 pixel convolution kernel.

[0084] To obtain the training and testing image datasets, this invention designs three different spatial degradation operations to generate mixed distortion images:

[0085] 1) Gaussian blur: This invention considers 28 Gaussian blur levels (σ). b ∈[0.5:0.25:4.5,5:0.5:10]), where σ b This represents the standard deviation of the Gaussian blur kernel. For each pixel in the original image, one of these 28 levels is randomly selected as the distortion parameter. The pixel value in the degraded image is equal to the value of the corresponding pixel in the original image after blurring using the randomly selected distortion parameter.

[0086] 2) Gaussian noise: In this invention, the noise value is set to a random scaling factor multiplied by a standard normal distribution sample value. This random scaling factor is spatially variable, therefore the final generated noise is also spatially variable. In this invention, the maximum value of the noise is set to 55.

[0087] 3) JPEG Compression: The distortion is generated in a similar way to Gaussian blur. This invention considers 31 JPEG quality factors (q∈[10:20,22:2:40,45:5:90]). To achieve JPEG compression, an image is first divided into non-overlapping 8×8 pixel image blocks. Each block is JPEG compressed using a randomly selected q value. Finally, all the compressed image blocks are stitched together to obtain the distorted image.

[0088] This invention requires training two network models: FENet and MS-CRSCNet. For FENet, training is performed using 128×128 pixel image patches (grayscale images), and its loss function is:

[0089]

[0090] in, and ξ m (i,j) represent the predicted and actual distortion parameters at spatial location (i,j), respectively; H and W represent the pixel size of the image; m = 1, 2, 3 represent the three distortion types. For MS-CRSCNet, it is trained using 128×128×3 pixel image patches (RGB images), and its loss function is a weighted combination of Mean Squared Error (MSE) loss and Structural Similarity Index (SSIM) loss:

[0091]

[0092] Where λ = 103 Used to adjust the weights of the two loss functions. The MSE and SSIM loss functions are expressed as follows:

[0093]

[0094]

[0095] Among them, I and I r These represent the original image and the restored image, respectively. represents SSIM(I,I r The average value of () is calculated using the following formula:

[0096]

[0097] in, and Representing image I / I respectively r The local mean and local standard deviation; C1 and C2 are two constants.

[0098] The training data includes 800 images from the DIV2K dataset, and spatial non-uniform quality degradation is performed on each original image at three scales, while the corresponding distortion parameter maps are saved. The degradation operations used in this invention include four combinations: (1) Gaussian blur + Gaussian noise (BN); (2) Gaussian blur + JPEG compression (BJ); (3) Gaussian noise + JPEG compression (NJ); (4) Gaussian blur + Gaussian noise + JPEG compression (BNJ). To evaluate the performance of the proposed algorithm, the public datasets LIVE, CSIQ, BSD500, CIDIQ, and SDIVL were selected as the test set, and the corresponding mixed distortion images were generated in the same way as the training set. PSNR, SSIM, and LPIPS were used as objective evaluation metrics. The performance of various algorithms on the test set is shown in Table 1, where DRSCNet (distortion-guided residual sparse coding network) represents the method of this invention. To compare the efficiency of different algorithms, the number of network parameters and floating-point operations (FLOPs) were selected as evaluation metrics, and the test results are shown in Table 2. To further examine network performance, this invention also tested the accuracy of the FENet network in predicting distortion parameters, using the estimated and actual SROCC values ​​as evaluation metrics, and the experimental results are shown in Table 3. The experimental results demonstrate that the method of this invention can achieve superior image restoration performance with relatively lightweight network parameters and computational complexity.

[0099] In summary, the method of this invention extends the traditional convolutional sparse coding model into a multi-scale convolutional residual sparse coding model, unfolds the optimization solution process of this model into a deep network, and embeds it into a U-Transformer network to achieve non-uniform mixed distortion image restoration guided by distortion parameters.

[0100] Table 1 shows the experimental results of the proposed method (DRSCNet) and other image restoration methods on different image databases. Bold numbers indicate the best performance metrics, and underlined numbers indicate the second-best performance metrics.

[0101]

[0102]

[0103] Table 2 compares the number of network parameters and floating-point operations of the method of the present invention (DRSCNet) with other image restoration methods.

[0104]

[0105] Table 3 shows the accuracy of the FENet network prediction distortion parameters in the method of this invention.

[0106]

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A hybrid distortion image restoration method based on convolutional sparse residual learning, characterized in that, Includes the following steps: Based on the lossless images of the DIV2K dataset, a spatially non-uniformly mixed distortion image dataset is constructed, which simultaneously contains three distortion types: Gaussian blur, Gaussian noise, and JPEG compression. An end-to-end aggregation excitation network is trained using the spatially non-uniformly mixed distortion image dataset. The mixed distortion image to be tested is input into the trained aggregation firing network to estimate the distortion parameters and obtain the corresponding spatial distortion parameter map. A multi-scale convolutional residual sparse coding network was trained using a spatially non-uniformly mixed distortion image dataset. The mixed-distortion image to be tested and the estimated spatial distortion parameter map are input into the trained multi-scale convolutional residual sparse coding network to obtain the restored image.

2. The hybrid distortion image restoration method based on convolutional sparse residual learning according to claim 1, characterized in that, The aggregation excitation network adopts a three-branch, four-scale U-Net structure. The encoder output features at each scale are concatenated with the upsampled features of the decoder module through residual connections, and finally output distortion parameter maps after two convolutional layers.

3. The hybrid distortion image restoration method based on convolutional sparse residual learning according to claim 2, characterized in that, The encoder of the aggregation excitation network consists of aggregation excitation modules, which are used to fuse features from different branches to model the interaction between different distortion types, and to concatenate the fused features with features from different branches to model the impact of mixed distortion on the parameter estimation of a single distortion type.

4. The hybrid distortion image restoration method based on convolutional sparse residual learning according to claim 1, characterized in that, The multi-scale convolutional residual sparse coding network adopts a multi-scale U-Transformer network architecture, and the encoder and decoder of the multi-scale convolutional residual sparse coding network are connected through the convolutional residual sparse coding module.

5. The hybrid distortion image restoration method based on convolutional sparse residual learning according to claim 4, characterized in that, The output image features of the encoder are processed by the convolutional residual sparse coding module to obtain the restored features, and finally the decoder reconstructs the restored image.

6. The hybrid distortion image restoration method based on convolutional sparse residual learning according to claim 4, characterized in that, The convolutional residual sparse coding module is a deep unfolded network.

7. The hybrid distortion image restoration method based on convolutional sparse residual learning according to claim 4, characterized in that, The network structure of deep unfolded networks is determined through the optimization process of solving the multi-scale convolutional residual sparse coding problem.

8. The hybrid distortion image restoration method based on convolutional sparse residual learning according to claim 7, characterized in that, The optimization problem of multi-scale convolutional residual sparse coding can be expressed by formulas (1) to (4): Among them, e i =x i -x′ i (i = 1, 2, 3, 4) represents the residual characteristics at each scale; and represent common and independent convolutional sparse representations, respectively, where N c and Let c and z represent respectively. i The channel dimension, and satisfying and Let N represent the corresponding convolution filter, where N is the number of convolutional filters. i ={48,72,96,120}; ↓n indicates downsampling the image to its original size. * indicates a convolution operation; and The number of convolutional filters in the middle is equal to e i Channel dimension; Solving formulas (1) to (4) is equivalent to solving the following optimization problem: Where λ is the penalty parameter, Equation (5) uses the method of fixing other variables and solving for the remaining variable to solve each variable in the optimization problem in turn. First, c and any three z are fixed. i Solve for the remaining z i ,get: Rewrite formula (6) as follows: in Solving formula (7) yields: Where E i D i θ and θ are independent learnable variables; TB represents the Transformer module; all z are fixed. i Solving for the variable c in formula (5) is equivalent to solving the following optimization problem: in Rewrite formula (9) as follows: Where ↑n represents the upsampled image to n times its original size, combining the first four norms in formula (10) yields: in This represents the concatenation of different feature maps along the channel dimension. Using formula (8), the variable x in formula (11) is solved to obtain: Among them, E and D are independent learnable variables; the definition of TB is the same as that of formula (8); Finally, the optimal c and z can be obtained by iteratively solving formulas (8) and (12). i , Therefore, the restored image features can be represented as:

Images