A garbage image denoising method based on multi-dimensional image information fusion

Abstract

The application provides a garbage image denoising method based on multi-dimensional image information fusion, two sub-networks are used to extract complementary information of two-dimensional images and three-dimensional images respectively, more image detail features are obtained, a dynamic self-attention module is introduced to reduce noise of the two-dimensional images, wherein the dynamic convolution adaptively learns dynamic noise features of different input garbage images, and the self-attention mechanism is used to locate significant foreground information in the dynamic noise features and suppress unimportant background information; a Transformer based on an adaptive sparse mechanism is introduced to model a long-distance dependency relationship of three-dimensional garbage image information and construct an adaptive sparse mechanism. The application overcomes the receptive field limitation of the convolutional neural network, reduces the computation amount of the self-attention of the Transformer, and through the fusion of the two-dimensional and three-dimensional image features, the garbage image denoising effect can be improved, and the accuracy of subsequent garbage classification, detection and other tasks is improved.

Description

Technical Field

[0001] This invention relates to the fields of image processing and computer vision, and in particular to an image denoising method based on fused convolutional neural networks. Background Technology

[0002] Image denoising techniques are used to remove noise from images, restore clean images, and promote the widespread application of image processing. Classic image denoising methods utilize redundant information in the image or image domain transformations to suppress noise and restore the image. For example, nonlocal image similarity methods use the entire image for denoising, finding similar regions in the image on a patch-by-pattern basis and then averaging these regions, which can effectively remove Gaussian noise in the image. The estimated value of a pixel in the clean image is obtained by weighted averaging of pixels with similar neighborhood structures. Transform domain-based processing methods mainly rely on the difference between the noise signal and the image signal in the frequency domain to separate the two signals. Taking the frequency domain as an example, the contour, edges, and noise components of the image correspond to the high-frequency part in the frequency domain, while large background areas correspond to the low-frequency part. Collaborative filtering based on the corresponding frequency characteristics can effectively denoise. However, these traditional methods face challenges such as low denoising speed and the need for manual parameter tuning to adapt to different noise levels.

[0003] Deep learning-based methods can effectively address these issues. Deep learning possesses the characteristics of automatic image feature extraction and learning, enabling it to better fit noise and reduce denoising time. Deep learning-based image denoising methods employ various neural network architectures to enhance feature representation capabilities, such as residual networks, generative adversarial networks (GANs), and graph neural networks. Among these, residual network-based denoising methods address the vanishing gradient problem in deep networks through skip connections and connect shallow noise information with deep semantic features. Each network layer only needs to learn a small amount of content, reducing optimization difficulty. Residual network-based denoising methods can significantly increase model depth and improve expressive power, but repeated use of skip connections can easily lead to overfitting. Generative adversarial network-based denoising methods consist of two models: a generative model responsible for extracting noise information from noisy images and generating clean images; and a discriminative model responsible for judging whether the generated clean noise is realistic enough. The generative and discriminative models iteratively learn against each other until an ideal state is reached. The adversarial learning strategy of GAN-based denoising methods can fit the image noise distribution well; however, model training is challenging and the training methods are not stable enough. The graph neural network-based approach first calculates the Euclidean distance between each pixel and other pixels in the feature space, constructing a K-nearest neighbor graph. Then, it aggregates the K nearest pixels of each pixel using edge-conditional convolution to form a clean image. This networked topology is suitable for dense noise distributions, but its complex structure also presents training difficulties. While the above methods perform well in general scenarios, they pose challenges for denoising junk images. Junk images are characterized by cluttered backgrounds and dim lighting, making direct application of these architectures ineffective. Therefore, proposing an image denoising method suitable for junk image scenarios is crucial. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a junk image denoising method based on multi-dimensional image information fusion, achieving junk image denoising through multi-dimensional image information fusion. This invention improves junk image denoising performance mainly in three aspects: First, it introduces a fusion mechanism of 2D and 3D image information. Unlike existing 2D image denoising methods, this invention uses two sub-networks to extract complementary information from 2D and 3D images, obtaining more image detail features and ultimately improving the denoising performance of 2D images. Second, it introduces a dynamic self-attention module to reduce 2D image noise. Dynamic convolution adaptively learns the dynamic noise features of different input junk images and uses a self-attention mechanism to locate salient foreground information in the dynamic noise features while suppressing unimportant background information. Third, it introduces a Transformer based on an adaptive sparsity mechanism to model the long-distance dependencies of 3D junk image information, fully utilizing the rich information of the 3D image to restore image details. Furthermore, it constructs an adaptive sparsity mechanism to solve the problem of manually selecting the number of sparse nodes required by the sparse Transformer, reducing the computational increase brought by introducing the Transformer model, while maximizing the saliency of the selected vector sequences. Therefore, this invention has both research and practical significance.

[0005] A method for denoising junk images based on multi-dimensional image information fusion, the specific steps of which are as follows:

[0006] Step 1: Transform the 2D training dataset into a 3D training dataset;

[0007] Step 2: Input the two-dimensional training dataset into the dynamic self-attention module, and extract the two-dimensional garbage noise features through the dynamic self-attention module;

[0008] Step 3: Input the 3D training dataset obtained in Step 1 into the Transformer network based on the adaptive sparsity mechanism, and extract the 3D garbage noise features of the 3D training dataset through the Transformer network based on the adaptive sparsity mechanism.

[0009] Step 4: Perform dimensionality transformation on the three-dimensional garbage noise features obtained in Step 3 to convert them into two-dimensional data, and then fuse the two-dimensional data with the two-dimensional garbage noise features obtained in Step 2.

[0010] Step 5: Input the noise fused in Step 4 into the fusion network, and obtain the clean garbage image through residual calculation; the residual calculation is to subtract the output of the two-dimensional noise image from the output of the three convolutional processing layers of the fusion network.

[0011] Step 6: Calculate the loss between the output image obtained in Step 5 and the clean labeled garbage image using the loss function, and continuously optimize the model parameters until the obtained error reaches the accuracy range required by the task, thus obtaining the optimized denoising model;

[0012] Step 7: Input the garbage image to be denoised into the denoising model optimized in Step 6 to obtain the clean image after denoising.

[0013] The Transformer network based on the adaptive sparse mechanism, after receiving the 3D training dataset, inputs 3D image patches into the linear mapping layer. The linear mapping layer maps the 3D garbage image patches into feature vectors. The feature vectors enter the location embedding layer, which provides clues to the long-distance dependencies of the garbage images and outputs a sequence of feature vectors carrying location information. The feature vectors with location information are then passed through 6 adaptive sparse Transformer sub-modules in sequence to extract the 3D garbage noise features of the 3D training dataset.

[0014] The Transformer submodule includes two layers of normalization, one multi-head self-attention layer, a multilayer linear perceptron (MLP), and an adaptive sparsity mechanism. The input is normalized layer by layer, and then the normalized vector sequence is copied three times to obtain three attention input query vector sequences Q, key-value vector sequences K, and content vector sequences V. Simultaneously, the normalized vector sequences are input into a fully connected layer to obtain the position offset vectors used for the sparsity mechanism. These, along with a separately defined learnable parameter D, represent the number of saliency vectors for the sparsity mechanism. The learnable parameters D, key-value vector sequence K, and content vector sequence V undergo sparse transformation to obtain sparse key-value vector sequence and sparse content vector sequence. Then, the query vector sequence Q, sparse key-value vector sequence, and sparse content vector sequence are input into a multi-head self-attention block, enabling the model to learn long-distance dependencies within the image. The resulting multi-head self-attention block output is added to the layer normalized output as a residual to reduce the difficulty of model training. The resulting residual is then normalized by the input layer and processed by a multilayer linear perceptron, and finally added to the previous residual to obtain the final output.

[0015] This invention also provides a dynamic self-attention module, which consists of four sequentially connected dynamic self-attention sub-modules. Each sub-module includes one dynamic convolutional processing layer, two convolutional processing layers, and one single convolutional layer. The feature map is sequentially input into the dynamic convolutional processing layer, the two convolutional processing layers, and the single convolutional layer. Then, the convolutional weights are extracted and subjected to spatial normalization and channel normalization to obtain the spatial attention matrix and channel attention vector of the convolutional weights, respectively. The spatial attention matrix and channel attention vector are multiplied by the feature map output by the convolution according to the corresponding spatial and channel dimensions, respectively, to obtain the spatial self-attention result and the channel self-attention result, respectively. Finally, the spatial self-attention result and the channel self-attention result are added to obtain the output result of the dynamic self-attention module.

[0016] The dynamic convolution processing layer includes a dynamic convolution layer, batch normalization (BN), and adaptive parametric rectifier linear units (APReLU).

[0017] The beneficial effects of this invention are as follows: In the context of garbage image denoising, it introduces high-dimensional noise image information on the basis of the two-dimensional denoising process, and learns two-dimensional and three-dimensional image noise information separately, fusing complementary information from different dimensions to ultimately improve the noise removal capability of garbage images, thereby enhancing the accuracy of subsequent garbage classification, detection, and other tasks. The model architecture of this invention combines convolutional neural networks and Transformers, utilizing both the advantages of convolutional neural networks for structural information and the advantages of Transformers for long-distance modeling. It employs a Transformer network based on an adaptive sparsity mechanism, which increases the model's freedom in selecting important feature vectors and more adaptively determines the number of saliency vectors, thus more flexibly reducing the large amount of computation required by Transformers in the high-dimensional feature extraction process and improving operational efficiency. The dynamic self-attention network model extracts rich robust foreground noise information from two-dimensional garbage images. Unlike feature map-based attention mechanisms, this invention uses a weight-based attention mechanism, which significantly improves the robustness of attention features. Attached Figure Description

[0018] Figure 1 This is the overall flowchart of the present invention.

[0019] Figure 2 This is a framework diagram of the garbage image denoising model proposed in this invention.

[0020] Figure 3 This is a structural diagram of the dynamic self-attention submodule proposed in this invention.

[0021] Figure 4 This is a structural diagram of the Transformer submodule based on the adaptive sparse mechanism proposed in this invention.

[0022] Figure 5 This is the image of the garbage noise to be processed in this invention.

[0023] Figure 6 The three-dimensional garbage noise image generated by this invention.

[0024] Figure 7 This is a schematic diagram of an embodiment of the noise reduction process of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0026] This invention relates to a method for denoising junk images based on multi-dimensional image information fusion, such as... Figure 1 and Figure 2 As shown, the specific steps are as follows:

[0027] Step 1: Transform the 2D training dataset into a 3D training dataset;

[0028] Step 2: Input the two-dimensional training dataset into the dynamic self-attention module, and extract the two-dimensional garbage noise features through the dynamic self-attention module;

[0029] Step 3: Input the 3D training dataset obtained in Step 1 into the Transformer network based on the adaptive sparsity mechanism, and extract the 3D garbage noise features of the 3D training dataset through the Transformer network based on the adaptive sparsity mechanism.

[0030] Step 4: Perform dimensionality transformation on the three-dimensional garbage noise features obtained in Step 3 to convert them into two-dimensional data, and then fuse the two-dimensional data with the two-dimensional garbage noise features obtained in Step 2.

[0031] Step 5: Input the noise fused in Step 4 into the fusion network, and obtain the clean garbage image through residual calculation; the residual calculation is to subtract the output of the two-dimensional noise image from the output of the three convolutional processing layers of the fusion network.

[0032] Step 6: Calculate the loss between the output image obtained in Step 5 and the clean labeled garbage image using the loss function, and continuously optimize the model parameters until the obtained error reaches the accuracy range required by the task, thus obtaining the optimized denoising model;

[0033] Step 7: Input the garbage image to be denoised into the denoising model optimized in Step 6 to obtain the clean image after denoising.

[0034] This invention also provides a Transformer network based on an adaptive sparsity mechanism, wherein the adaptive Transformer network is as follows: Figure 2 As shown, after receiving the 3D training dataset, the 3D image patch is input into the linear mapping layer. The linear mapping layer maps the 3D garbage image patch into a feature vector. The feature vector enters the location embedding layer. The location embedding layer provides clues to the long-distance dependencies of the garbage image and outputs a sequence of feature vectors carrying location information. The feature vectors with location information are passed through 6 adaptive sparse Transformer sub-modules in sequence to extract the 3D garbage noise features of the 3D training dataset.

[0035] Existing sparse-mechanism-based Transformers select D important feature vectors from a sequence of key-value vectors as replacements for global feature vectors. However, the selection of the D value is manually determined, and the selected vectors are distributed at uniform intervals, which limits the flexibility of the sparse mechanism. This invention utilizes the D value learned from the input features and the feature vector position offset to improve the model's freedom in selecting important feature vectors and to more adaptively determine the D value, thereby improving operational efficiency.

[0036] The Transformer submodule includes two layers of normalization, one multi-head self-attention layer, a multilayer linear perceptron (MLP), and an adaptive coefficient mechanism. The input is normalized layer-wise, and then the normalized vector sequence is copied three times to obtain three attention input query vector sequences Q, key-value vector sequences K, and content vector sequences V. Simultaneously, the normalized vector sequences are input into a fully connected layer to obtain position offset vectors for the sparsity mechanism. A separately defined learnable parameter D is used as the number of saliency vectors for the sparsity mechanism. The position offset vectors, learnable parameter D, key-value vector sequence K, and content vector sequence V undergo sparse transformation to obtain sparse key-value vector sequences and sparse content vector sequences. Then, the query vector sequence Q, the sparse key-value vector sequence, and the sparse content vector sequence are input into the multi-head self-attention block, enabling the model to learn long-distance dependencies within the image. The resulting multi-head self-attention block output is summed with the layer-normalized output as a residual to reduce the model training difficulty. The resulting residual is then normalized again using the input layer and the multilayer linear perceptron, and summed again with the previous residual to obtain the final output.

[0037] This invention also provides a dynamic self-attention module, which consists of four sequentially connected dynamic self-attention sub-modules. Each sub-module includes one dynamic convolutional processing layer, two convolutional processing layers, and one single convolutional layer. The feature map is sequentially input into the dynamic convolutional processing layer, the two convolutional processing layers, and the single convolutional layer. Then, the convolutional weights are extracted and subjected to spatial normalization and channel normalization to obtain the spatial attention matrix and channel attention vector of the convolutional weights, respectively. The spatial attention matrix and channel attention vector are multiplied by the feature map output by the convolution according to the corresponding spatial and channel dimensions, respectively, to obtain the spatial self-attention result and the channel self-attention result, respectively. Finally, the spatial self-attention result and the channel self-attention result are added to obtain the final output result.

[0038] The dynamic convolutional processing layer includes a dynamic convolutional layer, batch normalization (BN), and adaptively parametric rectified linear units (APReLU). The dynamic convolutional processing layer improves the ability to handle the varied foreground shapes of garbage images by selectively extracting features from different garbage images, thus enhancing the robustness of feature extraction. BN is a commonly used normalization method in deep learning; by normalizing the feature map, it prevents gradient vanishing and reduces model convergence time. APReLU employs a learnable slope parameter, significantly enhancing the non-linearity of the activation function and improving the model's expressive power. The ordinary convolutional processing layer includes convolution, BN, and APReLU, used to deepen the network and improve the refinement of noisy features. A single convolutional layer includes one convolution layer to implement a self-attention mechanism. Unlike feature map-based self-attention mechanisms, this invention uses spatial and channel normalization of the convolutional layer weights to obtain spatial attention and channel attention weights. Spatial normalization refers to summing the weights of convolutional weights at the same spatial location but different channels, and then normalizing the sum of weights at all locations. Channel normalization refers to summing the weights of convolutional weights at the same channel but different spatial locations, and then normalizing the sum of weights at all channels. These two types of attention guide and fuse the feature maps, enabling the model to simultaneously focus on important information at both spatial locations and channels, achieving the goal of separating foreground and background from complex, cluttered image environments. The extracted noise features are continuously refined through layer-by-layer dynamic self-attention modules, ultimately obtaining salient noise information.

[0039] The proposed garbage image denoising method requires first training the proposed model, and then using the trained model to denoise noisy garbage images. During the model training phase, based on the characteristics of the proposed method, corresponding data preparation, feature extraction, and model parameter optimization processes are necessary. First, the 2D garbage noise image needs to be dimensionalized to obtain a 3D garbage noise image. Then, a dynamic self-attention module and a Transformer network based on an adaptive sparsity mechanism are used to extract 2D and 3D garbage noise features, respectively. The model then fuses features from different dimensions to obtain the denoised image. The loss is calculated based on the generated denoised image to optimize the model parameters. For the post-training phase, the data only needs to undergo the same transformation and be input into the trained model.

[0040] The proposed network model comprises two cooperating sub-network models: a dynamic self-attention module and a Transformer network based on an adaptive sparsity mechanism. The dynamic self-attention network extracts rich robust foreground noise information from two-dimensional garbage images, while the Transformer network learns three-dimensional garbage noise features from three-dimensional garbage images. Furthermore, to fuse the two noise features, the proposed model incorporates a fusion network to ultimately improve the noise removal capability of garbage images.

[0041] Two sub-network models (a dynamic self-attention module and a Transformer network based on an adaptive sparsity mechanism) acquire noise information from 2D and 3D garbage images respectively, and then fuse them through a fusion network. The features of the two sub-networks are complementary, and the fused feature map has significantly richer features compared to single-dimensional features. The fusion network contains three convolutional processing layers, each including convolution, batch normalization (BN), and APReLU. Finally, a clean garbage image is obtained through residual operations.

[0042] After the model is trained, the garbage image to be denoised is transformed into a 3D garbage image, which is then input into the two sub-network models. The dimensionality transformation process is implemented using various open-source tools such as 3D Build to obtain high-dimensional and rich information. After the images are input into the trained overall model, the model output is the denoising result.

[0043] The following is an example:

[0044] (1) Obtain the image of the garbage noise to be denoised, such as Figure 5 As shown:

[0045] (2) Generate a three-dimensional garbage noise image based on the two-dimensional garbage noise image to be denoised, such as... Figure 6 As shown:

[0046] (3) Load the trained network model.

[0047] (4) Input the two-dimensional and three-dimensional garbage images into the dynamic self-attention network and the Transformer network based on the adaptive sparsity mechanism, respectively. The model output is the clean garbage image. An example process is shown below. Figure 7 As shown. Among them, Figure 7 The image at the bottom left is a two-dimensional garbage noise image. Figure 7 The image at the top left is a 3D garbage noise image. The 2D and 3D garbage noise images are input into two sub-network models respectively. The trained models then denoise the two noise images. Finally, the model outputs a clean garbage noise image. Figure 7 The clean image shown on the right.

Claims

1. A method for denoising junk images based on multi-dimensional image information fusion, characterized in that... Includes the following steps: Step 1: Transform the 2D training dataset into a 3D training dataset; Step 2: Input the two-dimensional training dataset into the dynamic self-attention module, and extract the two-dimensional garbage noise features through the dynamic self-attention module; Step 3: Input the 3D training dataset obtained in Step 1 into the Transformer network based on the adaptive sparsity mechanism, and extract the 3D garbage noise features of the 3D training dataset through the Transformer network based on the adaptive sparsity mechanism. Step 4: Perform dimensionality transformation on the three-dimensional garbage noise features obtained in Step 3 to convert them into two-dimensional data, and then fuse the two-dimensional data with the two-dimensional garbage noise features obtained in Step 2. Step 5: Input the noise fused in Step 4 into the fusion network, and obtain the clean garbage image through residual calculation; the residual calculation is to subtract the output of the two-dimensional noise image from the output of the three convolutional processing layers of the fusion network. Step 6: Calculate the loss between the output image obtained in Step 5 and the clean labeled garbage image using the loss function, and continuously optimize the model parameters until the obtained error reaches the accuracy range required by the task, thus obtaining the optimized denoising model; Step 7: Input the garbage image to be denoised into the denoising model optimized in Step 6 to obtain the clean image after denoising.

2. A Transformer network for a junk image denoising method based on multi-dimensional image information fusion according to claim 1, characterized in that: The Transformer network based on the adaptive sparse mechanism, after receiving the 3D training dataset, inputs 3D image patches into the linear mapping layer. The linear mapping layer maps the 3D garbage image patches into feature vectors. The feature vectors enter the location embedding layer, which provides clues to the long-distance dependencies of the garbage images and outputs a sequence of feature vectors carrying location information. The feature vectors with location information are then passed through 6 adaptive sparse Transformer sub-modules in sequence to extract the 3D garbage noise features of the 3D training dataset.

3. The Transformer network according to claim 2, characterized in that: The Transformer submodule includes two layers of normalization, one multi-head self-attention layer, a multilayer linear perceptron (MLP), and an adaptive sparsity mechanism. The input is normalized layer by layer, and then the normalized vector sequence is copied three times to obtain three attention input query vector sequences Q, key-value vector sequences K, and content vector sequences V. Simultaneously, the normalized vector sequences are input into a fully connected layer to obtain the position offset vectors used for the sparsity mechanism. These, along with a separately defined learnable parameter D, represent the number of saliency vectors for the sparsity mechanism. The learnable parameters D, key-value vector sequence K, and content vector sequence V undergo sparse transformation to obtain sparse key-value vector sequence and sparse content vector sequence. Then, the query vector sequence Q, sparse key-value vector sequence, and sparse content vector sequence are input into a multi-head self-attention block, enabling the model to learn long-distance dependencies within the image. The resulting multi-head self-attention block output is added to the layer normalized output as a residual to reduce the difficulty of model training. The resulting residual is then normalized by the input layer and processed by a multilayer linear perceptron, and finally added to the previous residual to obtain the final output.

4. A dynamic self-attention module for a junk image denoising method based on multi-dimensional image information fusion according to claim 1, characterized in that: The dynamic self-attention module consists of four sequentially connected sub-modules. Each sub-module contains one dynamic convolutional processing layer, two convolutional processing layers, and one single convolutional layer. The feature map is sequentially input into the dynamic convolutional processing layer, the two convolutional processing layers, and the single convolutional layer. Then, the convolutional weights are extracted and subjected to spatial normalization and channel normalization to obtain the spatial attention matrix and channel attention vector of the convolutional weights, respectively. The spatial attention matrix and channel attention vector are multiplied by the feature map output by the convolution according to their corresponding spatial and channel dimensions to obtain the spatial self-attention result and the channel self-attention result, respectively. Finally, the spatial self-attention result and the channel self-attention result are added to obtain the output result of the dynamic self-attention module.

5. The dynamic self-attention module of the junk image denoising method based on multi-dimensional image information fusion according to claim 4, characterized in that: The dynamic convolution processing layer includes a dynamic convolution layer, batch normalization, and adaptive parameter rectified linear units.

Images