A Fast Denoising Method for Ultra-Large Images Based on Image Complexity Evaluation

Abstract

This invention discloses a fast denoising method for ultra-large images based on image complexity evaluation. The method includes the following steps: (1) extracting complexity features of image color, texture, and content dimensions, and constructing an image complexity classification model that fuses multiple features; (2) constructing a multi-scale residual dense denoising network; (3) constructing a dataset and setting the training parameters of the multi-scale residual dense denoising network; (4) setting a loss function, training the multi-scale residual dense denoising network with the goal of minimizing the loss function, and updating the network parameters; (5) dividing the image to be processed into blocks to obtain multiple image patches; classifying the image patches using the image complexity classification model, and guiding the image patches to output from different depth levels based on the classification results to obtain denoised image patches; (6) stitching all image patches together to restore the original image. This method significantly saves computational resources, shortens runtime, and enables fast denoising of ultra-large images.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and in particular to a fast denoising method for ultra-large images based on image complexity evaluation. Background Technology

[0002] During image acquisition and transmission, images are subject to degradation and distortion to some extent due to environmental factors, sensor limitations, and natural noise pollution. Since image noise generation is a systemic problem and extremely difficult to avoid, research on image denoising is of great significance for improving the direct perception of images by the human eye and for subsequent image processing.

[0003] Image denoising algorithms, a classic problem in image processing, have been extensively studied for decades. Early traditional algorithms can be broadly categorized into filtering-based denoising methods, sparse representation-based denoising methods, and low-rank matrix-based denoising methods. Among these, block-matching and 3D filtering (BM3D) is one of the most advanced traditional algorithms. With the improvement of computing power and the rapid development of deep learning, researchers have begun to explore deep learning methods. Multi-layer perceptron (MLP) networks, through their powerful fitting and non-linear mapping capabilities, learn the mapping relationship between noisy images and noise-free reference images. Deep convolutional residual network (DnCNN) denoising models combine residual learning and batch normalization to learn noise in images; experimental results show that their denoising performance surpasses traditional algorithms. In recent years, deep learning-based denoising algorithms have developed rapidly; however, improvements in model performance are often accompanied by increases in network depth and algorithm complexity, limiting their practical applications.

[0004] With the rapid development of digital imaging technology, imaging devices are acquiring images with increasingly higher resolutions, such as 10000×10000 pixels or higher, and data sizes reaching hundreds of MB or even several GB. Such ultra-large images typically cannot be loaded into memory all at once; therefore, the image must be divided into blocks, and each block is then input into the network model separately. However, a single image may contain vastly different regions of complexity. Regions with less content and smoother textures are inherently easier to recover than other regions. Therefore, using the same network path for all image regions adds unnecessary computation and is inefficient. Summary of the Invention

[0005] To address the shortcomings of existing methods, the present invention aims to propose a fast denoising method for ultra-large images based on image complexity evaluation.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A fast denoising method for ultra-large images based on image complexity evaluation, characterized by the following steps:

[0008] Step 1: Extract complexity features of image color, texture, and content dimensions, and construct an image complexity classification model that fuses multiple feature information;

[0009] Step 2: Construct a multi-scale residual dense denoising network. This network adopts a U-shaped network structure with a depth of M layers, including multiple convolutional units, downsampling modules, and upsampling modules. Each convolutional unit fuses contextual information through dense connections. The multi-scale residual dense denoising network has M exit points at different depth levels.

[0010] Step 3: Construct the dataset and set the training parameters of the multi-scale residual dense denoising network;

[0011] Step 4: Set the loss function and train the multi-scale residual dense denoising network with the goal of minimizing the loss function, and update the network parameters;

[0012] Step 5: Divide the image to be processed into blocks to obtain multiple blocks; use the image complexity classification model established in Step 1 to classify the blocks, and guide the blocks to be output from the exit of different depth levels according to the classification results to obtain the denoised blocks.

[0013] Step 6: Piece together all the tiles obtained in Step 5 to restore the original image.

[0014] Furthermore, in step 1, the extracted complexity features include:

[0015] 1) Color Entropy H

[0016]

[0017] Where K is the total number of color levels after image quantization under the HSV color model, and P... i Let be the frequency of the i-th color in the graph;

[0018] 2) Color distribution uniformity D

[0019]

[0020]

[0021]

[0022]

[0023] Where C H C S C V These represent the uniformity of the image across the three components of the HSV color space, where n is the number of color channels. These are the statistical frequencies of pixels with color level i in the color channel; This represents the maximum possible variance of the channel.

[0024] 3) Energy ASM

[0025]

[0026] Where p(i, j, d, θ) is the gray-level co-occurrence matrix, i and j are the gray values ​​of two pixels, d is the distance between two pixels, and θ is defined as the angle between two pixels; L is the number of gray levels after image quantization.

[0027] 4) Information Entropy ENT

[0028] ;

[0029] 5) Edge ratio R edge

[0030]

[0031] Where P canny M×N represents the number of edge pixels obtained after Canny edge detection of the image, and M×N represents the total number of pixels in the image.

[0032] Further, in step 1, constructing the image complexity classification model through multi-feature information fusion specifically includes: extracting five feature parameters describing different dimensions: color entropy, color distribution uniformity, energy, information entropy, and edge ratio, and combining them into a complexity feature vector C, specifically represented as:

[0033] C = [Color] H Color D ASM, ENT, R edge ]

[0034] 1500 images with different complexity distributions were selected, and their complexity feature vectors were extracted. Fuzzy K-means clustering was then performed to obtain three centroids, representing different levels of complexity: low complexity V... L Medium complexity V M High complexity V H For a new image, calculate the Euclidean distance between its complexity feature vector and the three center points. The category of the closest point is the complexity category of the image.

[0035] Furthermore, the convolutional unit includes a dimension transformation module, a feature extraction module, and a channel attention module; the dimension transformation module is used to change the number of channels in the input image; the feature extraction module is composed of a depthwise separable convolutional layer and a ReLU activation function connected in series, used to decompose the convolution operation into two parts: channel-wise convolution and pointwise convolution; the channel attention module is used to calculate the weights of the input feature map and fuse the weights and the feature map to obtain the attention-enhanced feature map.

[0036] Furthermore, the multi-scale residual dense denoising network also includes a depth supervision module for supervising the output of each depth level.

[0037] Furthermore, the upsampling module is a transposed convolution with a kernel size of 2×2 and a stride of 2, and the downsampling module is a convolutional layer with a kernel size of 3×3 and a stride of 2; each exit of the multi-scale residual dense denoising network uses residual learning to predict noise.

[0038] Furthermore, in step 4, the loss function L is:

[0039]

[0040]

[0041] in, For the estimated noise residual, For each output, L i (θ i ) represents the mean square error of the predicted noise residual and the true noise value, w i L represents the process of deep supervision i (θ i The weight of ).

[0042] Furthermore, in step 4, the network training process uses the Adam optimizer to add random Gaussian noise with a noise level of [0, 55] to the training set images. The training is carried out for a total of 300 iterations. The learning rate for the first 100 iterations is 1e-3, the learning rate for the 101st to 200th iterations is 1e-4, and the learning rate for the last 100 iterations is 1e-5.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] (1) An effective image complexity classification model is proposed, which can classify the patches of the original image into three categories: simple, medium and complex according to the difficulty of noise removal. The patch to be tested can be output from the shallow layer and the complex sample from the deeper layer according to the classification result. This effectively removes noise while greatly reducing the amount of computation and the total running time.

[0045] (2) In the proposed multi-scale residual dense denoising network structure, a multi-scale U-shaped network is used to fully extract features at different scales, and dense connections effectively restore the information loss caused by the encoding process. Features at different levels are fused, and the deep supervision mechanism enables gradient propagation effectively. Residual learning is more efficient in predicting noise distribution than directly predicting clean images.

[0046] (3) Compared with existing denoising methods, the method proposed in this invention can save computing resources and shorten running time while achieving comparable denoising performance, and can achieve fast denoising of ultra-large images. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention;

[0048] Figure 2 This is a diagram of the multi-scale residual dense denoising network structure in an embodiment of the present invention;

[0049] Figure 3 This is a structural diagram of the feature extraction module in an embodiment of the present invention;

[0050] Figure 4 This is a structural diagram of the channel attention module in an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the classification results in an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram showing the effect comparison in an embodiment of the present invention;

[0053] Figure 7 This is a comparison chart of runtime in an embodiment of the present invention. Detailed Implementation

[0054] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0055] Reference Figure 1 This embodiment presents a fast denoising method for ultra-large images based on image complexity evaluation, with the following specific steps:

[0056] Step 1: Extract complexity features of image color, texture, and content dimensions, and construct an image complexity classification model that integrates multiple feature information.

[0057] Step 11, extract complexity features, including:

[0058] 1) Color Entropy H

[0059]

[0060] Where K is the total number of color levels after image quantization under the HSV color model, and P... i Let be the frequency of the i-th color in the graph;

[0061] 2) Color distribution uniformity D

[0062]

[0063]

[0064]

[0065]

[0066] Where C H C S C V These represent the uniformity of the image across the three components of the HSV color space, where n is the level of that color channel. These are the statistical frequencies of pixels with color level i in that color channel; This indicates the maximum possible variance of the channel;

[0067] 3) Energy ASM

[0068]

[0069] Where L is the number of gray levels after image quantization, and takes a value of 8; p(i, j, d, θ) is the gray-level co-occurrence matrix, where i and j are the gray values ​​of two pixels, d is the distance between the two pixels, and θ is the angle between the two pixels. Typically, θ∈{0°, 45°, 90°, 135°}, and d∈{(0, d), (d, d), (d, 0), (-d, d)}. The more uniform the gray-level distribution, the greater the corresponding energy value.

[0070] 4) Information Entropy ENT

[0071]

[0072] The more fine textures an image has, the higher its complexity and the greater its entropy.

[0073] 5) Edge ratio R edge

[0074]

[0075] Where P canny M×N represents the number of edge pixels obtained after Canny edge detection of the image, and M×N represents the total number of pixels in the image.

[0076] Step 12: Establish a complexity classification model. Collect 1500 images of different types and complexities, including 500 simple, 500 medium, and 500 complex images. Based on Step 11, extract 5 feature parameters from each image to form a complexity vector C, specifically represented as follows:

[0077] C = [Color] H Color D ASM, ENT, R edge ]

[0078] Normalization is performed. The complexity feature vectors are clustered using the fuzzy K-means algorithm to obtain three centroids representing different levels of complexity: V L (Low Complexity), V M (Middle Complexity), V H (High Complexity).

[0079] Step 13, determine the complexity category. Extract the complexity vector C from the image to be tested, and calculate its Euclidean distance to the cluster centers in Step 12:

[0080]

[0081] Its complexity level Consistent with the nearest cluster center:

[0082]

[0083] Step 2: Construct a multi-scale residual dense denoising network. The network structure is as follows: Figure 2 As shown. In the specific implementation, the network depth M is set to 5, and the number of channels in the convolutional layer C is... N The value is set to 32. The network has five exits corresponding to different depths. The output of each exit is directly added to the input, and residual learning is used to predict noise. The upsampling module is a transposed convolution with a kernel size of 2×2 and a stride of 2, while the downsampling module is a convolutional layer with a kernel size of 3×3 and a stride of 2. Convolutional units at the same level use dense connections to fuse contextual information.

[0084] The specific implementation process of the multi-scale residual dense denoising network in this embodiment is as follows:

[0085] The input image ILQ, originally of size (C, H, W), is transformed into C by a dimension transformation module consisting of convolutional layers. 0，0 Size is (C N (H, W), only the number of channels is changed, which facilitates subsequent feature extraction and fusion. C is obtained after M downsampling operations.m，0 Let m ∈ {1, 2, ..., M}. Each downsampling reduces the size of the feature map by half, thus obtaining information about features at different scales. Let C be the nth feature map in the m-th layer. m，n The calculation process can be represented as follows:

[0086] C m，n =H EF (H cA ([H up (C m+1，n-1 ), C m，n-1 C m，n-2 , ...C m，0 ])), n∈{1,…Mm}

[0087] Where H EF H represents the feature extraction module. CA H represents the channel attention module. up This indicates an upsampling operation, [...] indicates a cat operation. The resulting C... m，n Size is

[0088] Feature extraction module H EF Structure such as Figure 3 As shown, it is composed of two depthwise separable convolutions and a ReLU activation function concatenated together. The difference between it and a general feature extraction module is that it uses depthwise separable convolution (DSC) instead of conventional convolution, thereby decomposing the conventional convolution operation into two parts: channel-wise convolution and point-wise convolution, which can effectively reduce the computational cost of the network.

[0089] The structure of the channel attention module HCA is as follows: Figure 4 As shown, the algorithm consists of global average pooling, fully connected layers, ReLU activation function, and sigmoid activation function. It calculates a weight on the input feature map and then fuses this weight with the feature map to obtain the attention-enhanced feature map. This process can be represented as:

[0090]

[0091] Among them, F sq F represents the compression process of global pooling of feature maps. ex This represents the activation process of applying the sigmoid function, resulting in the weight matrix F. scale This means assigning the weight matrix to the input feature map to obtain the attention-enhanced feature map. This operation can effectively learn the importance of different channels, compensating for the performance degradation caused by the reduced computational cost after applying DSC.

[0092] The network architecture also includes a deep supervision module, which can supervise the output of each depth level. The feature map C of the first layer... 0，n After a one-dimensional transformation, the output of each subnetwork (n∈{1, 2, ..., M} is an image of the original size; that is, the output of each subnetwork is already the result of the network's image processing. The gradients from backpropagation flow directly into the subnetworks through the deep supervision module, thereby maximizing the performance of each subnetwork to the optimal level for its corresponding depth.

[0093] Step 3: Construct the dataset and set the training parameters for the multi-scale residual dense denoising network model. Select 300 images from the BSD500 dataset. Perform data augmentation operations on the original images, including scaling, horizontal mirroring, vertical flipping, and rotation, with scale factors [1.0, 0.9, 0.8]. Randomly crop the images into 96×96 patches and add Gaussian white noise to the original images, ultimately obtaining 57,000 noisy image pairs. The training parameters are as follows: batch size is 64; the Adam algorithm is used to optimize the loss function, and the optimizer parameters are set to default values. The training phase iterates 300 times. The learning rate is 1e-3 for the first 100 iterations, 1e-4 for the 101st to 200th iterations, and 1e-5 for the last 100 iterations.

[0094] Step 4: Based on the multi-scale dense residual denoising network model and training parameters, update the network parameters with the objective of minimizing the loss function L:

[0095]

[0096]

[0097] in For the estimated noise residual, For each output, L i (θ i ) represents the mean square error (MSE) of the predicted noise residual and the true noise value, w i L represents the process of deep supervision i (θ i The weight of ).

[0098] Step 5: Divide the large image to be processed into blocks. Using the image complexity evaluation model established in Step 1, classify the blocks into three categories: Simple (LC), Medium (ML), and Complex (HC). Based on the classification results of the blocks, guide the samples to output from exits at different depths to obtain the denoised image. The difference between this step and Step 4 is that the batch size is 1, and instead of weighted summation at each exit, the output of each exit is directly obtained.

[0099] Step 6: Reassemble the denoised image blocks obtained in Step 5.

[0100] In one example, the ultra-large image to be processed is 6000×4000 pixels in size. It is divided into 100 tiles of 600×400 pixels each. After classification using a complexity evaluation model, 60% of the tiles belong to the low complexity (LC) category, 18% belong to the medium complexity (ML) category, and 22% belong to the high complexity (HC) category. The classification results are as follows: Figure 5 As shown.

[0101] During the testing phase, the five output branches were numbered S1, S2, S3, S4, and S5, guiding tiles of three complexity categories to be output from S1, S3, and S5 respectively. In terms of network performance, the three categories of tiles achieved PSNR improvements of 18.14dB, 13.30dB, and 12.07dB respectively, resulting in satisfactory visual effects. Regarding network efficiency, by guiding simple samples to be output from shallow layers, the runtime per simple sample was reduced by 75.9% compared to DnCNN with a single network path, while the PSNR only decreased by 0.37dB. The runtime for tiles belonging to the ML category was also reduced by 40.8%, while the runtime and PSNR for tiles belonging to the HC category were comparable. Specific results are as follows... Figure 6 , Figure 7 As shown. In summary, the method proposed in this invention significantly reduces runtime and improves computational efficiency while achieving comparable performance.

Claims

1. A fast denoising method for ultra-large images based on image complexity evaluation, characterized in that, Includes the following steps: Step 1: Extract complexity features from the image's color, texture, and content dimensions, and construct a multi-feature information fusion image complexity classification model. Specifically, this involves extracting five feature parameters describing different dimensions: color entropy, color distribution uniformity, energy, information entropy, and edge ratio, and combining them into a complexity feature vector C, specifically represented as: C=[Color H ,Color D ,ASM,ENT,R edge ] 1500 images with different complexity distributions were selected, and their complexity feature vectors were extracted. Fuzzy K-means clustering was then performed to obtain three centroids, representing different levels of complexity: low complexity V... L Medium complexity V M High complexity V H For a new image, calculate the Euclidean distance between its complexity feature vector and the three center points. The category of the closest point is the complexity category of the image. Step 2: Construct a multi-scale residual dense denoising network. This network adopts a U-shaped network structure with a depth of M layers, including multiple convolutional units, downsampling modules, and upsampling modules. Each convolutional unit fuses contextual information using dense connections. The multi-scale residual dense denoising network has M outputs at different depth levels. The multi-scale residual dense denoising network also includes a depth supervision module to supervise the output of each depth level. Step 3: Construct the dataset and set the training parameters of the multi-scale residual dense denoising network; Step 4: Set the loss function and train the multi-scale residual dense denoising network with the goal of minimizing the loss function, and update the network parameters; Step 5: Divide the image to be processed into blocks to obtain multiple blocks; use the image complexity classification model established in Step 1 to classify the blocks, and guide the blocks to be output from the exit of different depth levels according to the classification results to obtain the denoised blocks. Step 6: Piece together all the tiles obtained in Step 5 to restore the original image.

2. The fast denoising method for ultra-large images based on image complexity evaluation according to claim 1, characterized in that, In step 1, the extracted complexity features include: 1) Color Entropy H Where K is the total number of color levels after image quantization under the HSV color model, and P... i Let be the frequency of the i-th color in the graph; 2) Color distribution uniformity D Where C H C S C V These represent the uniformity of the image across the three components of the HSV color space, where n is the number of color channels and f is the uniformity. i H f i S f i V These are the statistical frequencies of pixels with color level i in the color channel; This represents the maximum possible variance of the channel. 3) Energy ASM Where p(i,j,d,θ) is the gray-level co-occurrence matrix, i and j are the gray values ​​of two pixels, d is the distance between two pixels, and θ is defined as the angle between two pixels; L is the number of gray levels after image quantization. 4) Information Entropy ENT ； 5) Edge ratio R edge Where P canny M×N represents the number of edge pixels obtained after Canny edge detection of the image, and M×N represents the total number of pixels in the image.

3. The fast denoising method for ultra-large images based on image complexity evaluation according to claim 1, characterized in that, The convolutional unit includes a dimension transformation module, a feature extraction module, and a channel attention module; The dimension transformation module is used to change the number of channels in the input image; the feature extraction module is composed of a depthwise separable convolutional layer and a ReLU activation function connected in series, which is used to decompose the convolution operation into two parts: channel-wise convolution and point-wise convolution; the channel attention module is used to calculate the weights of the input feature map and fuse the weights and the feature map to obtain the attention-enhanced feature map.

4. The fast denoising method for ultra-large images based on image complexity evaluation according to claim 1, characterized in that, The upsampling module is a transposed convolution with a kernel size of 2×2 and a stride of 2, and the downsampling module is a convolutional layer with a kernel size of 3×3 and a stride of 2; each exit of the multi-scale residual dense denoising network uses residual learning to predict noise.

5. The fast denoising method for ultra-large images based on image complexity evaluation according to claim 1, characterized in that, In step 4, the loss function L is: in, For the estimated noise residual, For each output, L i (θ i ) represents the mean square error of the predicted noise residual and the true noise value, w i L represents the process of deep supervision i (θ i The weight of ).

6. The fast denoising method for ultra-large images based on image complexity evaluation according to claim 1, characterized in that, In step 4, the network training process uses the Adam optimizer to add random Gaussian noise with a noise level of [0, 55] to the training set images. The training is carried out for a total of 300 iterations. The learning rate for the first 100 iterations is 1e-3, the learning rate for the 101st to 200th iterations is 1e-4, and the learning rate for the last 100 iterations is 1e-5.

Images