Image denoising method based on cooperative non-local contextual network

Abstract

The application discloses an image denoising method based on a cooperative non-local context network, and steps are as follows: 1, pre-processing the data set to construct input sample data; 2, constructing a cooperative non-local context network model, including a shallow feature extraction module, a deep feature extraction module and an image reconstruction module; 3, training the network, adjusting the network parameters and obtaining an optimal model. The application fully excavates the global dependence relationship in the image by combining the pixel-level non-local feature and the image block-level non-local feature, improves the quality of image recovery, solves the problem that the existing convolutional neural network cannot capture the non-local context dependence relationship and the effect is poor in actual application, so that the image denoising task can be used, and the texture detail information of the image is well recovered.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image denoising method based on a cooperative non-local context network. Background Technology

[0002] Image denoising is the process of recovering a clean, high-quality image from a noisy, low-quality image. In the real world, images are often contaminated by various signals or noise during acquisition, leading to a decline in image quality. To address this problem, image denoising, as a fundamental issue in image restoration, has been extensively studied over the past few decades and plays a crucial role in various real-world computer vision tasks, such as medical imaging, surveillance, and security. It not only improves image quality but also serves as an auxiliary task for other computer vision applications.

[0003] Existing image denoising methods can be categorized into prior-based and learning-based methods. Prior-based methods rely on priors found in natural images, such as local smoothing and self-similarity. While achieving significant performance, they can lead to inaccurate denoising results if the priors do not align with the actual data distribution. Unlike prior-dependent methods, learning-based methods learn the mapping from noisy to clean images in an end-to-end manner. In recent years, with the widespread adoption of deep neural networks, many image denoising architectures based on convolutional networks have been designed.

[0004] However, traditional convolutional neural network methods can only operate in a small local neighborhood, cannot take advantage of the contextual dependencies of features, cannot fully utilize all the features of the image, and have poor denoising performance. Summary of the Invention

[0005] The present invention addresses the shortcomings of the existing technology by proposing an image denoising method based on a cooperative nonlocal context network. This method combines pixel-level nonlocal features and image block-level nonlocal features to fully exploit global dependencies within the image, thereby improving the quality of image restoration. As a result, it can be used for image denoising tasks, enabling the excellent restoration of image texture details.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] The present invention provides an image denoising method based on a cooperative non-local context network, characterized by the following steps:

[0008] Step 1: Preprocess the dataset and construct the input sample data;

[0009] In the high-quality image dataset obtained Additive white Gaussian noise at different noise levels is added to obtain a noisy image dataset. in, This represents the i-th high-quality image. Let represent the i-th noisy image, and T be the total number of high-quality images;

[0010] Step 2: Construct a collaborative nonlocal context network model, including: a shallow feature extraction module, a deep feature extraction module, and an image reconstruction module;

[0011] Step 2.1: The shallow feature extraction module is a single-layer convolutional layer, used for the i-th noisy image. Processing is performed to obtain the i-th shallow feature.

[0012] Step 2.2: The deep feature extraction module consists of N non-local feature extraction modules. Any nth non-local feature extraction module comprises one pixel-level non-local module and one image patch-level non-local module, and is used to extract the i-th shallow feature. Processing is performed to obtain the Nth deep feature.

[0013] Step 2.3: The reconstruction module reconstructs the Nth deep feature. Perform a convolution operation, and the result is combined with the i-th shallow feature. Add them together to get the i-th high-quality image.

[0014] Step 3: Train the network, adjust the network parameters, and obtain the optimal model:

[0015] The noisy image dataset I LQ The model is input into a cooperative nonlocal context network model and trained using gradient descent. Simultaneously, the loss function L, as shown in equation (2), is calculated to update the model parameters until the loss function L converges, thus obtaining a trained cooperative nonlocal context network model for high-quality restoration of any noisy image.

[0016]

[0017] The image denoising method based on cooperative non-local context networks described in this invention is also characterized in that step 2.2 includes:

[0018] Step 2.2.1: The pixel-level nonlocal module is composed of M attention-guided residual blocks and one convolutional layer PIConv; wherein, any m-th attention-guided residual block is composed of a convolutional layer, an activation function layer ReLU, a convolutional layer, a pair of parallel spatial awareness context modules and channel awareness context modules, and residual branches.

[0019] When n=1, m=1, the shallow features The input is fed into the pixel-level nonlocal module of the nth nonlocal feature extraction module, and then sequentially processed through the first convolutional layer, the ReLU activation function layer, and the second convolutional layer of the mth attention-guided residual block before outputting the feature map. Where C, H, and W represent the number of channels, height, and width of the image, respectively;

[0020] The feature map The m-th attention-guided residual block is processed in the spatial awareness context module and the channel awareness context module, respectively. The spatial awareness context module... After passing through convolutional layers and the softmax function, the spatial context attention weights are obtained. and Perform matrix multiplication to obtain the context set.

[0021] The spatial perception context module The transformed feature vector is obtained after performing the convolution operation. and A weighted average ensemble is performed to obtain the spatially perceptual contextual features.

[0022] The channel sensing context module will After passing through a global average pooling layer (GAP), a first convolutional layer, a ReLU activation function layer, a second convolutional layer, and a sigmoid activation function, channel context attention weights are generated. and After performing matrix multiplication, the channel-aware context features are obtained.

[0023] Will and After addition, the input is passed through the residual branch and added to the input of the m-th attention-guided residual block to obtain the m-th context feature.

[0024] When n = 1, m = 2, 3, ..., M, the (m-1)th context feature The input is processed in the m-th attention-guided residual block of the pixel-level nonlocal module of the n-th nonlocal feature extraction module to obtain the m-th context feature. Then, the Mth attention guides the residual block to output the Mth context feature. After processing by the convolutional layer PIConv, the input is added to the pixel-level non-local module of the nth non-local feature extraction module to obtain the pixel-level features output by the pixel-level non-local module in the nth non-local feature extraction module.

[0025] Step 2.2.2: The image block-level non-local module in the nth non-local module feature extraction module adopts a multi-scale self-attention module with S layers, and consists of a shared Query value generation branch, S Key value generation branches, S Value value generation branches, and S attention generation modules.

[0026] The query value generation branch consists of a convolutional layer with a 1×1 kernel, an Unfold function layer, and a Reshape layer, in sequence; the s-th key value generation branch and the s-th value value generation branch each consist of a convolutional layer, an Unfold function layer, and a Reshape layer; s = 1, ..., S; wherein, the convolutional layer in the s-th key value generation branch has a 2×1 kernel. s-1 ×2 s-1 The stride is 2 s-1 The convolutional kernel of the convolutional layer in the s-th value generation branch is 2. s-1 ×2 s-1 The stride is 2 s-1 ;

[0027] When n=1, the pixel-level features output by the pixel-level nonlocal module in the nth nonlocal feature extraction module are... The image block-level nonlocal module, which is input into the nth nonlocal feature extraction module, and processed by the Query value generation branch, generates shared Query features.

[0028] Pixel-level features The S key values ​​are input into branches that generate branches and the S value values ​​that generate branches, respectively. These branches are then processed to generate the corresponding key values ​​for layer S. and S-layer Value Then, the enhanced features at multiple scales are obtained using equation (1). in, This represents the s-th Key value of the image block-level non-local feature extraction module in the n-th non-local module feature extraction module. This represents the s-th Value of the image block-level nonlocal feature extraction module in the n-th nonlocal module feature extraction module;

[0029]

[0030] In equation (1), softmax represents the activation function. This represents the s-th enhanced feature of the image block-level nonlocal module in the n-th nonlocal module feature extraction module;

[0031] Enhanced features using the Fold function Transforming back to the original space yields the expanded enhanced features, denoted as... in, This represents the s-th expanded enhanced feature of the image block-level nonlocal module in the n-th nonlocal module feature extraction module;

[0032] Will The data is sequentially concatenated and processed through one convolutional layer before being combined with pixel-level features. The features are added together to obtain the nth deep feature output by the nth nonlocal feature extraction module.

[0033] When n = 2, 3, ..., N, the (n-1)th deep feature The input is processed by the nth nonlocal feature extraction module, and the output features of the nth nonlocal feature extraction module are obtained. Thus, the Nth deep feature is obtained from the Nth nonlocal feature extraction module.

[0034] The present invention provides an electronic device, including a memory and a processor, wherein the memory is used to store a program that supports the processor in executing the image denoising method, and the processor is configured to execute the program stored in the memory.

[0035] The present invention discloses a computer-readable storage medium on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of the image denoising method.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. This invention integrates non-local attention into the network to mine globally relevant information. Traditional convolutional neural networks can only operate within a small local neighborhood, thus limiting their ability to capture long-term contextual dependencies. In contrast, this invention flexibly integrates non-local attention into the convolutional neural network, which can fully utilize global dependencies to enhance feature representation, solving the problem that existing convolutional neural networks cannot capture non-local contextual dependencies and therefore perform poorly in practical applications. Experimental results on synthetic image denoising tasks demonstrate that this method is superior in both performance and computational complexity.

[0038] 2. This invention designs a pixel-level non-local module containing multiple attention-guided residual blocks, utilizing spatial awareness attention and channel awareness attention to fully explore fine-grained contextual relationships in order to recover more accurate details.

[0039] 3. This invention designs an image block-level nonlocal module. Traditional image block-level nonlocal modules ignore the correlation of multi-scale image block features. It can make full use of the self-scale and cross-scale similarity between features of different sizes to recover similar textures, capture more reliable dependencies, and improve the quality of image restoration. Attached Figure Description

[0040] Figure 1 This is a diagram of the cooperative nonlocal contextual image denoising network structure of the present invention;

[0041] Figure 2 This is a pixel-level non-local module structure diagram of the present invention;

[0042] Figure 3 This is a structural diagram of the attention-guided residual block (AGRB) of the present invention;

[0043] Figure 4 This is a structural diagram of the image block-level nonlocal module of the present invention. Detailed Implementation

[0044] In this embodiment, an image denoising method based on a cooperative non-local context network includes the following steps:

[0045] Step 1: Preprocess the dataset and construct the input sample data;

[0046] In the high-quality image dataset obtained Additive white Gaussian noise at different noise levels is added to obtain a noisy image dataset. in, This represents the i-th high-quality image. Let represent the i-th noisy image, and T be the total number of high-quality images. In this embodiment, the DIV2K dataset is used, which contains 1000 high-resolution images with different scenes, split into 800 images for training, 100 images for validation, and 100 images for testing. Additive white Gaussian noise with a noise level in the range [0, 70] is added to each high-resolution image to obtain all noisy images.

[0047] Step 2: Construct a collaborative nonlocal context-based network model, including: a shallow feature extraction module, a deep feature extraction module, and an image reconstruction module. The network structure is as follows: Figure 1 As shown;

[0048] Step 2.1: The shallow feature extraction module is a single-layer convolutional layer, used for the i-th noisy image. Processing is performed to obtain the i-th shallow feature.

[0049] Step 2.2: The deep feature extraction module consists of N non-local feature extraction modules. Any nth non-local feature extraction module comprises one pixel-level non-local module and one image patch-level non-local module, and is used to extract the i-th shallow feature. Processing is performed to obtain the Nth deep feature. In this embodiment, N=4, and the non-local feature extraction module is as follows: Figure 1 As shown;

[0050] Step 2.2.1, as follows Figure 2 As shown, the pixel-level nonlocal module consists of M attention-guided residual blocks and one convolutional layer PIConv; in this embodiment, M=8, and the pixel-level nonlocal module is as follows: Figure 3 As shown, the PIConv convolutional layer has a 3×3 kernel;

[0051] In this embodiment, the m-th attention-guided residual block is composed of a first convolutional layer, a ReLU activation function layer, a second convolutional layer, a pair of parallel spatial awareness context modules and channel awareness context modules, and a residual branch; in this embodiment, the m-th attention-guided residual block is as follows: Figure 3 As shown, both convolutional layers have 3×3 kernels;

[0052] When n=1, m=1, shallow features The input is fed into the pixel-level nonlocal module of the nth nonlocal feature extraction module, and then sequentially processed by the convolutional layer of the mth attention-guided residual block, the ReLU activation function layer, and another convolutional layer to output the feature map. Where C, H, and W represent the number of channels, height, and width of the image, respectively;

[0053] Feature map The m-th attention-guided residual block is processed in the spatial awareness context module and the channel awareness context module, respectively.

[0054] Spatial Awareness Context Module After passing through convolutional layers and the softmax function, the spatial context attention weights are obtained. To obtain a more complex global context description, spatial context attention weights are added. and Performing matrix multiplication yields a more compact and accurate set of contexts. Again The transformed feature vector is obtained after performing the convolution operation. and A weighted average ensemble is performed to obtain the spatially perceptual contextual features.

[0055] The channel-aware context module uses Squeeze-and-Excitation Networks to explore relationships between channels and applies global pooling across spatial dimensions to encode the global context. After passing through a global average pooling layer (GAP), a first convolutional layer, a ReLU activation function layer, a second convolutional layer, and a sigmoid activation function, channel context attention weights are generated. and After performing matrix multiplication, the channel-aware context features are obtained.

[0056] Will and After addition, it is added to the input of the m-th attention-guided residual block. This allows the module to bypass more low-frequency information, thereby obtaining the m-th context feature.

[0057] When n = 1, m = 2, 3, ..., M, the (m-1)th context feature The input is processed in the m-th attention-guided residual block of the pixel-level nonlocal module of the n-th nonlocal feature extraction module to obtain the m-th context feature. Then, the Mth attention guides the residual block to output the Mth context feature. After processing by the PIConv convolutional layer, the input is added to the pixel-level non-local module of the nth non-local feature extraction module to prevent network degradation, thus obtaining the pixel-level features output by the pixel-level non-local module in the nth non-local feature extraction module.

[0058] Step 2.2.2: The image patch-level nonlocal module in the nth nonlocal module feature extraction module uses a multi-scale self-attention module with S layers to learn the similarity between self-scale feature blocks and cross-scale feature blocks. It consists of a shared Query value generation branch, S Key value generation branches, S Value value generation branches, and S attention generation modules. In this embodiment, S = 3. The image patch-level nonlocal module is as follows: Figure 4 As shown;

[0059] The query value generation branch consists of a convolutional layer with a 1×1 kernel, an Unfold function layer, and a Reshape layer, in sequence; the s-th key value generation branch and the s-th value value generation branch each consist of a convolutional layer, an Unfold function layer, and a Reshape layer; s = 1, ..., S; where the convolutional layer in the s-th key value generation branch has a 2×1 kernel. s-1 ×2 s-1 The stride is 2 s-1 The convolutional kernel of the convolutional layer in the s-th value generation branch is 2. s-1 ×2 s-1 The stride is 2 s-1 ;

[0060] When n=1, the pixel-level features output by the pixel-level nonlocal module in the nth nonlocal feature extraction module are... The image block-level nonlocal module, which is input into the nth nonlocal feature extraction module, and processed by the Query value generation branch, generates shared Query features.

[0061] Pixel-level features The S key values ​​are input into a branch to generate a key, and the S value values ​​are input into a branch to generate a key. and S-layer Value Then, the enhanced features at multiple scales are obtained using equation (1). in, This represents the s-th Key value of the image block-level non-local feature extraction module in the n-th non-local module feature extraction module. This represents the s-th Value of the image block-level nonlocal feature extraction module in the n-th nonlocal module feature extraction module;

[0062] The s-th attention generation will be in as well as As input, calculate according to equation (1):

[0063]

[0064] In equation (1), softmax represents the activation function. This represents the s-th enhanced feature of the image block-level nonlocal module in the n-th nonlocal module feature extraction module;

[0065] Enhanced features using the Fold function Transforming back to the original space yields the expanded enhanced features, denoted as... in, This represents the s-th expanded enhanced feature of the image block-level nonlocal module in the n-th nonlocal module feature extraction module;

[0066] Will The data is sequentially concatenated and processed through one convolutional layer before being combined with pixel-level features. The features are added together to obtain the nth deep feature output by the nth nonlocal feature extraction module.

[0067] When n = 2, 3, ..., N, the (n-1)th deep feature The input is processed by the nth nonlocal feature extraction module, and the output features of the nth nonlocal feature extraction module are obtained. Thus, the Nth deep feature is obtained from the Nth nonlocal feature extraction module.

[0068] Step 2.3: The reconstruction module reconstructs the Nth deep feature. Perform a convolution operation, and the result is combined with the i-th shallow feature. Add them together to get the i-th high-quality image.

[0069] Step 3: Train the network, adjust the network parameters, and obtain the optimal model. Noisy Image Dataset I LQ The model is input into a cooperative nonlocal context network model and trained using gradient descent. Simultaneously, the loss function L, as shown in equation (2), is calculated to update the model parameters until the loss function L converges, thus obtaining a trained cooperative nonlocal context network model for high-quality restoration of any noisy image.

[0070]

[0071] In this example, the ADAM optimizer is used to learn the optimal parameters. The hyperparameters of ADAM are set to β1 = 0.9, β2 = 0.999, and ε = 1 × 10⁻⁶. -8 Initialize the learning rate to 1×10. -4 Then every 2×10 5 The learning rate drops to half in the next iteration.

[0072] In an embodiment, an electronic device includes a memory and a processor, the memory being used to store a program that supports the processor in performing the methods described above, the processor being configured to execute the program stored in the memory.

[0073] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0074] In summary, this invention proposes an image denoising method based on a cooperative nonlocal context network, which integrates the advantages of image patch and pixel-level nonlocal self-similarity to capture more robust and reliable long-term dependencies. A pyramid-shaped nonlocal module is designed, which utilizes self-scale and cross-scale image patches to recover more reliable similar details at a coarse-grained level. For complex or finer texture features, this invention further develops a pixel-level nonlocal module containing multiple attention-guided residual blocks to fully explore the relationships between pixel-level global contexts to recover finer-grained structural features. Comprehensive experimental results demonstrate that this method outperforms existing methods in both quantitative and visual aspects.

Claims

1. An image denoising method based on cooperative nonlocal context networks, characterized in that, Includes the following steps: Step 1: Preprocess the dataset and construct the input sample data; In the high-quality image dataset obtained Additive white Gaussian noise at different noise levels is added to obtain a noisy image dataset. ,in, Indicates the first A high-quality image, Indicates the first A noisy image, The total number of high-quality images; Step 2: Construct a collaborative nonlocal context network model, including: a shallow feature extraction module, a deep feature extraction module, and an image reconstruction module; Step 2.1: The shallow feature extraction module is a single-layer convolutional layer, used for the first... Zhang Noise Image Processing is performed to obtain the first... shallow features ; Step 2.2, the deep feature extraction module consists of... It consists of a non-local feature extraction module, and any nth module... The first nonlocal feature extraction module consists of one pixel-level nonlocal module and one image patch-level nonlocal module, and is used to extract the first nonlocal feature. shallow features Processing is performed to obtain the first... Deep features ; Step 2.2.1: The pixel-level non-local module is sequentially composed of... It consists of an attention-guided residual block and a PIConv convolutional layer; wherein, any _th_ attention-guided residual block and a PIConv convolutional layer; Each attention-guided residual block consists of a convolutional layer, a ReLU activation function layer, another convolutional layer, a pair of parallel spatial awareness context modules and channel awareness context modules, and a residual branch. when At that time, the shallow features Enter the number The pixel-level nonlocal modules in the nonlocal feature extraction module, and sequentially through the first nonlocal feature extraction module. The feature map is output after processing the residual block by the first convolutional layer, the ReLU activation function layer, and the second convolutional layer. Where C, H, and W represent the number of channels, height, and width of the image, respectively; The feature map Enter the number respectively The residual block is processed in the spatial awareness context module and the channel awareness context module, whereby the spatial awareness context module... After passing through convolutional layers and the softmax function, the spatial context attention weights are obtained. and with Perform matrix multiplication to obtain the context set. ; The spatial perception context module The transformed feature vector is obtained after performing the convolution operation. and with A weighted average ensemble is performed to obtain the spatially perceptual contextual features. ; The channel sensing context module will After passing through a global average pooling layer (GAP), a first convolutional layer, a ReLU activation function layer, a second convolutional layer, and a sigmoid activation function, channel context attention weights are generated. and with After performing matrix multiplication, the channel-aware context features are obtained. ; Will and After addition, and then through the residual branch, it is combined with the first... The inputs of the attention-guided residual blocks are added together to obtain the first... Context features ; when At that time, the first Context features Enter the number The pixel-level nonlocal module of the nonlocal feature extraction module. The attention guides the processing of the residual block, thus obtaining the first... Context features ; and then from the first The first point to note is guiding the output of the residual block. Context features After being processed by the convolutional layer PIConv, it is then combined with the first... The inputs of the pixel-level nonlocal modules of the nth nonlocal feature extraction module are added together to obtain the nth nonlocal feature extraction module. Pixel-level features output by the pixel-level nonlocal module in the block nonlocal feature extraction module ; Step 2.2.2, the first The image block-level nonlocal modules in the nonlocal feature extraction module employ a multi-scale self-attention module with S layers, and generate branches using a shared query value. Each key value generates a branch and Each Value generates a branch, and One attention generation module; The query value generation branch consists of convolutional kernels... It consists of convolutional layers, Unfold function layers, and Reshape layers; the first The first key value generates the branch and the first key value generates the branch. Each Value generation branch consists of one convolutional layer, an Unfold function layer, and a Reshape layer; Among them, the first The convolutional kernel of the convolutional layer in the branch generated by each key value is... stride is ;No. The convolutional kernel of the convolutional layer in the branch that generates each value is... stride is ; when At that time, the first Pixel-level features output by the pixel-level nonlocal module in the block nonlocal feature extraction module Enter the number In the image block-level nonlocal module of the block nonlocal feature extraction module, and after processing by the Query value generation branch, shared Query features are generated. ; Pixel-level features Enter them separately Each key value generates a branch and The value generation branch processes and generates corresponding values. Layer Key Value and Layer Value Then, the enhanced features at multiple scales are obtained using equation (1). ,in, Indicates the first The image block-level nonlocal feature extraction module in the nonlocal module feature extraction module is the first Each key value, Indicates the first The image block-level nonlocal feature extraction module in the nonlocal module feature extraction module is the first One Value; (1) In equation (1), This represents the activation function. Indicates the first The first nonlocal module feature extraction module in the image block-level nonlocal module One enhanced feature; Enhanced features using the Fold function Transforming back to the original space yields the expanded enhanced features, denoted as... ,in, Indicates the first The first nonlocal module feature extraction module in the image block-level nonlocal module An expanded enhanced feature; Will The data is sequentially concatenated and processed through one convolutional layer before being combined with pixel-level features. Add them together to get the first one. The nth deep feature output by the nonlocal feature extraction module ; when At that time, the first Deep features Enter the number The process is performed in the non-local feature extraction module to obtain the first... The output features of each nonlocal feature extraction module ; and thus by the first The nonlocal feature extraction module obtains the Nth deep feature. ; Step 2.3, the reconstruction module for the first... Deep features Performing a convolution operation yields a result similar to the first... shallow features Add them together to get the first one. Zhang high-quality image ; Step 3: Train the network, adjust the network parameters, and obtain the optimal model: The noisy image dataset The input is given to a cooperative nonlocal context network model, and the model is trained using gradient descent. Simultaneously, the loss function shown in equation (2) is calculated. To update the model parameters until the loss function The process continues until convergence, resulting in a well-trained collaborative nonlocal context network model, which can be used to achieve high-quality recovery of any noisy image. (2)。 2. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the image denoising method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to perform the steps of the image denoising method of claim 1.

Images