Image denoising and stripe removing method based on blind spot regularization

Abstract

The invention discloses an image denoising and fringe removing method based on blind spot regularization, which comprises the following steps of: constructing a double-output blind spot network, respectively taking image reconstruction branch output and fringe estimation branch output of the double-output blind spot network as regularization constraint terms, and carrying out joint modeling on a clear image and fringe components through implicit network prior, random noise is removed by using blind spot regularization; a direction representation shuffling technology meeting J-invariance is introduced, and image and stripe separation is realized through multi-direction feature decomposition and vertical direction feature enhancement; in the model reasoning stage, scale-adjustable feature resampling operation is carried out on the input features, the blind spot receptive field is expanded, and the image reconstruction quality is optimized; and outputting the denoised clear image and the estimated stripe component by alternately optimizing the joint loss function of the dual-output blind spot network. According to the method, the advantages of a traditional denoising method based on a model and a method based on self-supervised learning are combined, and a new thought is provided for denoising and stripe removing tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to an image denoising and de-striping method based on blind spot regularization. Background Art

[0002] Image denoising refers to the process of recovering a clean image from a noisy image, which is a fundamental problem in computer vision. In the fields of remote sensing imaging and infrared imaging, digital images are often affected by both stripe noise and random noise. Among them, random noise is mainly caused by photon fluctuations, detector optoelectronic conversion noise, and additional noise in the circuit, while stripe noise is mainly caused by detector non-uniformity. The random noise and stripe noise are mixed together to form a complex spatial distribution, seriously degrading the image quality and having an adverse impact on subsequent downstream tasks.

[0003] In order to reconstruct a clear image from a noisy image, many denoising and de-striping methods have been proposed in the past decade. Existing de-striping methods can be classified into methods based on statistical matching, methods based on filtering, methods based on models, and methods based on supervised learning. However, traditional methods have limitations, such as poor performance in complex scenarios, difficulty in separating stripes by simple filtering, and difficulty in accurately describing both random noise and stripe noise with manual priors. In recent years, thanks to the powerful representation ability of neural networks, methods based on supervised learning have shown great advantages in denoising and de-striping tasks. These methods improve the learning ability of the model by exploring more effective network structures. For example, based on the structural characteristics of images and stripes, the wavelet-based denoising deep convolutional neural network (Toward Universal Stripe Removal via Wavelet-Based Deep Convolutional Neural Network, TSWEU) designs a two-stream CNN to learn the relationship between stripes and images and remove random noise simultaneously. The denoising convolutional neural network (Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising) proposes a recurrent convolutional neural network to capture the intra-band and inter-band correlations of hyperspectral images and remove stripe noise by complementary features extracted by a selective recurrent memory unit. Although methods based on supervised learning perform well in terms of performance, they usually rely on a large amount of paired data to learn the mapping from noisy images to clear images. However, collecting paired noise-clear data in the real world is extremely cumbersome and expensive, which limits the wide application of such methods.

[0004] To alleviate the data collection problem, some studies have proposed self-supervised denoising methods that only require noisy images. To prevent self-supervised denoising models from learning the identity mapping, each output pixel should not be affected by its corresponding input pixel, and this requirement is called J-invariance. As a denoising method that satisfies J-invariance, the blind-spot network uses adjacent pixels to predict artificially missing pixel values. Under the assumption that the noise is zero-mean and pixel-independent, the blind-spot network can learn denoising by minimizing the self-supervised loss. The dilated convolutional blind-spot network (Unpaired Learning of Deep Image Denoising), as a representative of them, adopts masked convolution (where the central pixel of the kernel has a zero value) and dilated convolution to ensure that each output pixel is estimated only from the surrounding noisy pixels. Due to the image signal processor (ISP), real-world random noise is spatially correlated. To satisfy the assumption of pixel-independent noise, the asymmetric pixel shuffle blind-spot network (AP-BSN: Self-Supervised Denoising for Real-World Images via Asymmetric PD and Blind-Spot Network) uses asymmetric pixel shuffle downsampling in the training and testing phases respectively to break the spatial correlation of the noise. This strategy enables AP-BSN to achieve impressive denoising results on real-world sRGB images, further promoting the research progress in the field of self-supervised denoising.

[0005] However, the self-supervised method based on the blind-spot network is established based on random noise and it is difficult to generalize to images contaminated by stripe noise. This is because the essence of the blind-spot strategy is to remove pixel-independent noise from spatially correlated signals, while both images and stripe noise belong to spatially correlated signals. As Figure 1 (a) shows, directly applying these methods to images containing random noise and stripe noise will form an aliasing of a clear image and the stripe component. Nevertheless, we observe that the blind-spot network has excellent reconstruction ability for a single type of spatially correlated signal and the potential to learn implicit image priors without clean labels. As Figure 1 (b) shows, for an image with only random noise, the blind-spot network reconstructs a clear image from the noisy image. As Figure 1 (c) shows, for a two-dimensional signal with random noise and stripe components, the blind-spot network can also reconstruct the stripe component from the noisy signal. Summary of the Invention

[0006] The object of the present invention is to provide an image denoising and destriping method based on blind-spot regularization.

[0007] To achieve the above object, the present invention provides the following technical solutions: An image denoising and descreening method based on blind spot regularization, comprising:

[0008] Construct a dual-output blind spot network, use the outputs of the image reconstruction branch and the stripe estimation branch of the dual-output blind spot network as regularization constraint terms respectively, jointly model the clear image and the stripe component through implicit network priors, and use blind spot regularization to remove random noise;

[0009] Introduce a direction representation shuffling technique that satisfies J-invariance in the dual-output blind spot network, and separate the image and the stripe through multi-directional feature decomposition and vertical direction feature enhancement;

[0010] Perform a feature resampling operation with adjustable scale on the input features during the model inference stage to expand the blind spot receptive field and optimize the image reconstruction quality;

[0011] Output the denoised clear image and the estimated stripe component by alternately optimizing the joint loss function of the dual-output blind spot network.

[0012] Further, construct a dual-output blind spot network including an image reconstruction branch and a stripe estimation branch, and use the output as a regularization constraint term, where:

[0013] The image reconstruction branch adopts an omnidirectional feature extraction module, and the stripe estimation branch adopts a vertical direction feature extraction module. The loss function is defined as:

[0014]

[0015] where Y is the image disturbed by random noise and stripe noise, B x (Y) is the output of the image reconstruction branch, B s (Y) is the output of the stripe estimation branch, ||·|| TV is the unidirectional total variation constraint, and λ1 and λ2 are weight parameters.

[0016] Further, the implementation methods of the joint modeling of the implicit network prior and the blind spot regularization include:

[0017] In the image reconstruction branch, dilated convolution is used to mask the central pixel, and the spatial connectivity prior of the image is learned through residual connection;

[0018] In the stripe estimation branch, the low-rank constraint of the stripe branch is realized by constraining the stripe mean;

[0019] During the joint optimization process, an alternating update strategy is adopted.

[0020] Further, use the direction representation shuffling technique to perform multi-directional decomposition on the input features, including

[0021] Perform a dilated wavelet transform on the input features, and decompose them into horizontal direction H, vertical direction V, diagonal direction D, and low-frequency component A;

[0022] Fuse the omnidirectional features (H / V / D / A) in the image reconstruction branch, and only retain the vertical direction feature V in the stripe estimation branch;

[0023] Maintain J-invariance through dilated convolution to ensure that the output pixels are not affected by the pixels at the corresponding input positions.

[0024] Furthermore, during the model inference stage, perform feature resampling optimization on the masked convolutional layer, including:

[0025] Perform bilinear upsampling on the input features, and the upsampling scale factor is (s h , s w ), where s h and s w correspond to the magnification factors of height and width respectively;

[0026] Perform a masked convolution operation on the upsampled features;

[0027] Restore the features to the original size through downsampling to generate the final output.

[0028] Furthermore, the method adopts an asymmetric pixel shuffle downsampling strategy during the training stage, that is, uses asymmetric reorganized pixels to reconstruct the image to reduce the spatial correlation of noise, thereby improving the denoising and despeckling effects.

[0029] Furthermore, the dual-output blind spot network includes a cross-branch information interaction module, which fuses features using a residual block by concatenating the intermediate features of the image reconstruction branch and the stripe estimation branch.

[0030] Furthermore, during the model inference stage, adjust the upsampling scale factor to adapt to images with different random noise and stripe noise distributions as input.

[0031] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the above method.

[0032] A computer-readable storage medium stores a computer program, and when the program is executed by a processor, it implements the steps of the above method.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows: We propose an image denoising and despeckling model based on blind spot regularization, and analyze and demonstrate that Self-BSR can be trained through self-supervised learning; introduce a direction representation with J-invariance in Self-BSR to provide more discriminative features for the separation of images and stripes; propose feature resampling optimization, which can significantly improve the denoising and despeckling performance of Self-BSR without additional learnable parameters in the model inference stage. The present invention collects a real-world image dataset containing random noise and stripe noise to evaluate the advancement of the proposed Self-BSR. Compared with existing methods, Self-BSR shows obvious advantages in denoising and despeckling performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0035] Figure 1 Shows the denoising effects of AP-BSN on three different noisy images, where (a) is random noise + stripe noise + clear image, (b) is random noise + clear image, and (c) is random noise + stripe noise.

[0036] Figure 2 Schematic diagram of direction representation shuffling, where (a) is multi-direction and (b) is vertical direction.

[0037] Figure 3 Overview of the structure diagram of the two-stream denoising network.

[0038] Figure 4 In (a) is conventional masked convolution, and in (b) is masked convolution with feature resampling.

[0039] Figure 5 Visual comparison of the denoising results of different algorithms on the synthetic dataset.

[0040] Figure 6 Visual comparison of the denoising results of different algorithms on the real near-infrared dataset.

[0041] Figure 7 Image average cross-profile analysis.

[0042] Figure 8In (a) is the noisy image; (b) is the clean image; (c) is the denoising effect of Self-BSR without using the FRR method; (d) is the denoising effect of Self-BSR using the FRR method. Detailed implementation manner

[0043] The present invention proposes a self-supervised stripe estimation and image denoising model (Self-BSR) based on blind spot regularization, which uses implicit network priors to refine the modeling of images and stripes respectively. Specifically, blind spot regularization takes the dual output of the improved blind spot network as the object of regularization to respectively constrain the reconstruction processes of clear images and stripes. Blind spot regularization can adaptively remove random noise according to the spatially continuous characteristics of the images and stripes learned by the network, replacing traditional parametric modeling. Secondly, a direction representation technique that satisfies J-invariance is introduced into Self-BSR to capture multi-directional information to effectively separate images and stripes. At the same time, aiming at the problem that traditional masked convolution is difficult to fully utilize the effective information in the receptive field, we propose a feature resampling method to improve the reconstruction ability of Self-BSR by resampling the pixels with high spatial correlation with the blind spot in the receptive field during the model inference stage. Self-BSR combines the advantages of traditional model-based denoising methods and self-supervised learning methods, providing a new idea for denoising and destriping tasks. To evaluate the proposed Self-BSR, we conducted a series of experiments on synthetic images and real noisy images. A large number of experiments show that the proposed method is significantly better than the existing destriping methods.

[0044] The method proposed by the present invention does not use BSN to suppress stripe noise, but uses its reconstruction ability for spatially correlated signals to estimate stripes and reconstruct clear images.

[0045] For an image disturbed by stripe noise and random noise, its degradation process can be described as:

[0046] Y = X + S + N #(1)

[0047] where, Y ∈ R m×n is the actually observed noisy image; m and n respectively represent the number of rows and columns; X is the potential clear image; S is the additive stripe noise; N is the random noise. Note that each column of the image has the same bias, so the stripe matrix S has the order of n (instead of m×n), that is,

[0048] S = [1…1…1] -1 × [s1…s j …s n #(2)

[0049] where, s j represents the deviation of the j-th column.

[0050] It should be noted that the image X and the stripe S have spatial correlation. Due to the existence of the Image Signal Processor (ISP), the random noise N also has spatial correlation in the real world. In the present invention, we use the proposed direction characterization technique to break this correlation to meet the assumption of pixel independence. In addition, we assume that the means of both the stripe noise and the random noise are zero. Our goal is to simultaneously reconstruct a clear image and estimate the stripe component from the noisy image.

[0051] In self-supervised learning, the denoising model needs to maintain J-invariance to prevent learning the identity mapping. J-invariance means removing the influence of the input pixels on the corresponding output pixels. The J-invariance in Noise2Self is defined as follows:

[0052] Definition: Let J be a partition of dimension {1, …, m}, J ∈ J. A function f: R m → R m is j-invariant if f(x) j does not depend on the value of x j . If it has J-invariance for each J ∈ J, it is called J-invariance.

[0053] Under the assumption that the noise is zero-mean and pixel-independent, the following proposition proposed by Noise2Self proves that if the network is J-invariant, minimizing the self-supervised loss is equivalent to minimizing the supervised loss.

[0054] Proposition: Assume that y (a noisy image) is an unbiased estimator of x (a clean image), denoted as and the noise in each subset J ∈ J is independent of the noise in its complement J c conditioned on x. Let f be J-invariant. Then there is

[0055]

[0056] where denotes the expected value of the squared F-norm of the matrix.

[0057] The Blind Spot Network (BSN) is a denoising method that predicts masked pixels by exploiting the spatial correlation between the masked pixels and their surrounding pixels in the image signal. BSN is based on the assumption that the noise in the image is spatially independent and zero-mean, while the image signal exhibits spatial correlation. Since the masked pixels do not affect the reconstruction of the corresponding pixels, BSN satisfies J-invariance and can perform self-supervised learning. Its self-supervised loss can be described as:

[0058]

[0059] Among them, B(*) represents the BSN network.

[0060] Although BSN has left an impressive effect in removing random noise, it cannot be generalized to stripe noise. Specifically, BSN is based on small-structured noise and is difficult to handle large-area spatially correlated noise such as stripe noise. Figure 1 (a) shows the denoising results of BSN for images contaminated by random noise and stripe noise. BSN removes random noise and forms an aliasing that includes the image and stripes.

[0061] Currently, the state-of-the-art model-based despeckling algorithms simultaneously model the image and the stripes, and its image reconstruction model can be expressed as:

[0062]

[0063] Among them, the first term is the reconstruction term, the second term P(X) represents the prior of the clear image, and the third term P(S) represents the prior of the stripe component. Among them, λ1, λ2, and λ3 are weight parameters used to balance the constraint degrees of different terms. Compared with the previous denoising methods that only focused on image modeling, these methods establish the connection between the image and the stripe components, which greatly promotes the separation of these two components. Although the existing model-based despeckling methods consider the influence of random noise, due to the lack of effective analysis, random noise residues often occur. Secondly, the priors used by these methods are artificially predefined and are difficult to handle the complex noise distributions in the real world.

[0064] Although model-based despeckling and denoising algorithms have been widely studied, there are still two challenges. First, for the equation, R x (X) and R s (S) need to encode the image and the stripes to obtain the optimal solution. However, common regularization methods rely on artificial priors and are difficult to accurately describe the image and the stripes, which may lead to the loss of image details and stripe residues. Second, the existing model-based despeckling and denoising methods often lack effective analysis and regularization of random noise, resulting in poor image reconstruction effects.

[0065] For degraded images, the blind spot network can capture spatial continuity characteristics, which is beneficial to image and stripe modeling. Secondly, the blind spot network can remove pixel-independent noise from spatially correlated signals. Based on this observation, we transform the entire despeckling and denoising problem into the reconstruction tasks of two spatially correlated signals, namely the image and the stripes. We use the implicit prior of the blind spot network to construct a new regularization method called blind spot regularization to address the above two challenges.

[0066] When processing images containing multi-type space-related signals, traditional single-output blind spot networks tend to output a mixture of multiple different signals, resulting in blurred images. Therefore, we constructed a new dual-stream blind spot network to facilitate independent modeling of images and stripes. The network can be described as:

[0067] B(Y) = B x (Y) + B s (Y) #(5)

[0068] where B(*) represents the dual-stream BSN network, B x (*) is the image branch network for reconstructing the clear image X, and B s (*) is the stripe branch network for estimating the stripe S. Since the blind spot strategy tends to output spatially continuous signals, we take B x (Y) and B s (Y) as the objects of image regularization and stripe regularization respectively. Secondly, ideal images and stripes have no random noise, so we add regularization for random noise. In summary, Equation (4) can be rewritten as:

[0069]

[0070] where, is the data fidelity term. R x (B x (Y)), R s (Bx(Y)) and R N (B x (Y)) + B s (Y)) are the image regularization term, the stripe regularization term, and the random noise regularization term respectively. λ1, λ2, and λ3 are weight parameters used to balance the constraint degrees of different terms.

[0071] B x (Y) and B s (Y) respectively contain prior knowledge such as the image basic structure and stripe distribution characteristics learned by the network. To prevent numerical deviation, we further guide B x (Y) and B s (Y). Specifically, we combine the image representation learned by B x (Y) with a unidirectional variational model to promote the separation of images and stripes. We combine the stripe distribution learned by B s (Y) with the zero-mean constraint and introduce the L2 norm to further optimize the stripes. In addition, minimizing the self-supervised loss function of the blind spot network can remove random noise from the image, and this prior is used as the random noise regularization term. Finally, the image denoising and destriping model based on blind spot regularization can be described as:

[0072]

[0073] where ||·|| TV represents the unidirectional variational constraint of the image. Since the first term and the fourth term of Equation (7) are equivalent, when λ3 = 1 / 2, Equation (7) can be simplified to:

[0074] where regularizes both data fidelity and random noise. We call the reconstruction model of Equation (8) Self-BSR, which inherits the advantages of self-supervised learning of the blind spot network. In the next subsection, we will prove the feasibility of self-supervised learning of Self-BSR.

[0075] Next, we will prove why the proposed Self-BSR can learn denoising in a self-supervised manner. Specifically, for the image degradation problem described in Equation (1), the objects reconstructed by Self-BSR are the clear image and the stripe component. According to Equation (3), the proposition proposed by Noise2Self can be extended to

[0076]

[0077] When B(*) satisfies J-invariance. When the random noise satisfies the assumptions of zero mean and pixel independence, the third term of Equation (10) becomes 0. Then the following equation can be obtained

[0078]

[0079] Since we designed a dual-output BSN for the tasks of simultaneous stripe estimation and image denoising. From Equation (7) and Equation (11), the following formula can be obtained

[0080]

[0081] It can be seen from Equation (12) that minimizing the self-supervised loss function is equivalent to minimizing the supervised loss function Secondly, in the equation, both the variational model and the L2 norm are convex. Therefore, Self-BSR can reconstruct the image and the stripe respectively through self-supervised learning.

[0082] In summary, Self-BSR independently refines the modeling of clean images and stripe components and jointly optimizes the solution. Importantly, we use the blind spot network to learn implicit image priors instead of explicit parametric modeling, enabling the regularization method to be adaptively adjusted according to the network output. In addition, we demonstrate that blind spot regularization allows Self-BSR to reconstruct clear images and stripes from noisy signals in a self-supervised learning manner. Self-BSR can be regarded as a combination of learning-based methods and model-based denoising methods. Benefiting from the stronger representation ability of learning-based methods for natural images, Self-BSR has stronger denoising performance.

[0083] Self-BSR can model both images and stripes through blind spot regularization. To more accurately separate images and stripes, the key issue is to find the structural feature differences between images and stripes and then use appropriate methods to describe them. As described in the literature, since stripe components often exhibit obvious directionality. If this direction feature can be incorporated into the blind spot network design, it will be beneficial for Self-BSR to distinguish stripes and images. However, it is not easy to perform direction representation in the blind spot network, and the J-invariance of the blind spot network needs to be maintained.

[0084] Inspired by Pixel-Unshuffle / Shuffle and wavelet transform, we propose the Directional Representation Unshuffle / Shuffle (DRU) technique with J-invariance, which can capture multi-scale and multi-directional features. Specifically, we upsample / downsample image features through a dilation-based redundant wavelet transform, where the haar wavelet is selected as the wavelet basis. Among them, the wavelet transform can extract high-frequency information in the horizontal (H), vertical (V), and diagonal directions (D) and retain low-frequency information (A). As Figure 2 shown, the pixels in different color boxes do not affect each other during subsequent convolution processes. The same color boxes represent image features in different directions. We assume that the hole scale of the directional representation shuffle is 2. When the dilation factor is 2, the multi-directional representation Unshuffle can be expressed as:

[0085]

[0086] where DRU(*,d) represents the directional representation shuffle operation, and d represents the dilation factor. f represents the image feature of size (N, C, H, W), where N represents the number of image samples in a batch, C represents the number of channels of the image feature, H represents the number of pixels of the image feature in the vertical direction, and W represents the number of pixels of the image feature in the horizontal direction. Denote the \(i\)-th sub-band feature in different directions corresponding to the image feature \(f\), with a size of \((N, C, H / d, W / d)\). When the dilation factor is 2, the vertical-direction representation Unshuffle can be expressed as:

[0087]

[0088] where \(f\) V denotes the feature in the vertical direction corresponding to the image feature \(f\), with a size of \((N, C, H, W)\).

[0089] It should be noted that DRU can avoid the influence of masked pixels on multi-direction features during subsequent convolution operations based on the dilation operation. DRU maintains the J-invariance of the network and avoids the identity mapping. Secondly, the direction representation shuffle can break the spatial correlation of random noise, thus satisfying the pixel independence assumption of random noise in blind spot regularization. In addition, through the redundant representation of signals, the direction representation shuffle can provide richer information and higher frequency accuracy, which is beneficial for Self-BSR to perform fine-grained modeling on images and stripes.

[0090] According to the proposed stripe estimation and image denoising model based on blind spot regularization, and the direction representation shuffle, the present invention designs a dual-branch blind spot denoising network. The network architecture is as Figure 3 shown. The input of the network is a noisy image, and the outputs are the reconstructed clear image and the estimated stripe separation. The direction representation shuffle is introduced to extract multi-direction and multi-scale information. Among them, the sparse vertical-direction features are used for stripe estimation, while the full-direction features are used for image reconstruction. According to the blind spot regularization, the outputs of the image reconstruction network branch \(B\) I (*) and the stripe estimation network branch \(B\) S (*) are respectively used to constrain their own reconstruction processes.

[0091] First, use multiple 1×1 convolutional layers to extract the shallow features of the noisy image. The shallow features are pixel-masked through a 3×3 central masked convolutional layer. Then, through the direction representation shuffle technology, the image information is represented in multiple directions and at multiple scales. Among them, the sparse vertical-direction features are used for stripe estimation, while the full-direction features are used for image reconstruction. In Equation (11), the outputs of the image reconstruction network branch \(B\) I (*) and the stripe estimation network branch \(B\) S(*) Each contains N Res Blocks for deep feature extraction. Then the features of each branch are restored to the original size through directional feature shuffling. In addition, we constructed the S2IModule to strengthen the information flow between the image branch and the stripe branch, where the S2I Block contains 2 Res Blocks. Finally, the image branch decodes the clear image through a 1×1 convolutional layer, and the stripe branch decodes the stripe component through a 1×1 convolutional layer.

[0092] The receptive field of the blind spot is crucial for the denoising effect of the BSN. A larger receptive field of the blind spot allows the network to use more neighboring pixels when predicting the masked pixels, thus reconstructing more details. Using the Transformer architecture can expand the receptive field of the network, but this often requires a more complex network structure design and a larger computational overhead.

[0093] To solve the above problems, we propose Feature Resampling (FRR), a simple and low-computational-overhead post-processing method, to improve the denoising performance of Self-BSR by expanding the receptive field of the blind spot. As Figure 4 shown, where (a) is a conventional masked convolution with a kernel size of 3×3. The size of the input feature is (H,W). For convenience, the channel dimension of the feature is ignored here. (b) is a masked convolution with feature resampling. The feature is upsampled and then the masked convolution is performed, and then downsampled to the original size. The upsampling sizes of the feature height and width are sh and sw respectively. The blind-spot convolution with feature resampling can significantly expand the receptive field of the blind spot without retraining. During the model training phase, Self-BSR uses conventional masked convolution. During the model inference phase, FRR resamples the features before and after the masked convolution, and this process does not require additional learnable parameters. Specifically, we first upsample the input feature to expand its size, then perform the masked convolution operation, and then downsample the feature to the original size. This process can be expressed as:

[0094] f * = downsample(maskedconv(upsample(f,(s h ,s w ))),(s h ,s w ))#(15)

[0095] where f represents the input image feature with a size of (N,C,H,W), and f * represents the image feature after feature resampling. upample(*,(s h ,s w )) represents the upsampling operation, where the upsampling scale factors of the image height and width are s hand s w 。downsample(*) represents the downsampling operation corresponding to upample(*,(s h ,s w ))). maskedconv(*) represents masked convolution, and the size of the features before and after convolution does not change.

[0096] We choose common bilinear interpolation as the method of up / downsampling. Through the upsampling operation, masked convolution can capture higher-resolution features. Through the corresponding downsampling operation, the receptive field of the blind spot is significantly enlarged, enabling Self-BSR to utilize richer context information to recover masked pixels. Therefore, based on the optimization method of feature resampling, Self-BSR can significantly improve the ability to retain details and suppress noise without additional training.

[0097] The beneficial effects of the present invention will be verified and described below in conjunction with embodiments and the accompanying drawings.

[0098] Embodiment

[0099] Synthetic dataset: We choose the open-source dataset CVC09 and make a synthetic dataset according to the image degradation process described by equation (1). CVC09 is a long-wave infrared pedestrian detection dataset. We select 900 of them as the training set and 100 as the test set. To simulate noisy images, first add Gaussian random noise with standard deviations of σ random = 5, 10, 15 to the clean images, and then add stripe noise with standard deviations of σ stripe = 5, 10, 15 in different rounds of experiments, that is, different levels of stripes with std(b1,b2,…,bn)=5, 10, 15, where b1,b2,…,bn represent the offsets of the 1st, 2nd,..., nth columns of the image. It should be noted that all clean images are normalized to [0,255].

[0100] Real dataset: We use the FirstLight near-infrared camera to capture real-world noisy images. We collected 1000 noisy images, among which 900 are used as the training set and 100 as the test set.

[0101] The loss function of the network is based on Equation (11). We use the L2 norm to constrain the first term of the equation and empirically set λ1 = 1, λ2 = 3e-3, and λ3 = 1e-2. The number N of Res Blocks in the network is defaulted to 6. For network training, the batch size is 8 and the number of training epochs is 2000. The optimization function adopted is Adam. The initial learning rate is 0.0001, which remains unchanged for the first 1000 epochs and then gradually decreases to 0 for the next 1000 epochs. Before training, the image size is adjusted to 128×128 and randomly flipped horizontally or vertically. All experiments are run on the server using python3.8.0, pytorch1.12.0, and NVIDIA_GeForce_RTX_3090.

[0102] For the synthetic dataset, to conduct a quantitative analysis of the Self-BSR performance, we choose Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) as the measurement criteria.

[0103] For the real dataset, the Photometric Non-Uniformity (PRNU) of the reconstructed image's uniform region is chosen as the objective evaluation criterion. The calculation of the photometric non-uniformity is as follows:

[0104]

[0105] where σ and m represent the standard deviation and mean of the image u respectively. The smaller the PRNU value, the smaller the non-uniformity. In addition, visual quality analysis and average cross-profile analysis are performed on the de-striping results to evaluate the effects of different denoising methods.

[0106] To demonstrate the superiority of the proposed algorithm, we compared our algorithm with several state-of-the-art descreening and denoising algorithms, namely LRSID (Remote sensing image stripe noise removal: From image decomposition perspective), SNRCNN (Single Infrared Image Optical Noise Removal Using a Deep Convolutional Neural Network), TLR (Transformed low-rank model for line pattern noise removal), TSWEU (Toward Universal Stripe Removal via Wavelet-Based Deep Convolutional Neural Network), AP-BSN (AP-BSN: Self-Supervised Denoising for Real-World Images via Asymmetric PD and Blind-Spot Network), SEID (Simultaneous Destriping and Image Denoising Using a Nonparametric Model With the EM Algorithm) and MM-BSN (MM-BSN: Self-Supervised Image Denoising for Real-World with Multi-Mask based on Blind-Spot Network). Here, we will make a full comparison of them. Note that LRSID, TLR and SEID are non-learning descreening algorithms, while SNRCNN and TSWEU are supervised learning-based denoising algorithms, and AP-BSN and MM-BSN are self-supervised denoising algorithms based on the blind-spot network. The codes of LRSID, TLR and SEID were provided by the authors. SNRCNN and TSWEU are deep learning-based algorithms. Since the problems proposed in the literature are exactly the same as the tasks of the present invention, we directly introduced the models trained by the authors in the experiments. The source codes and the trained models were downloaded from the authors' homepages. The codes of the self-supervised learning methods AP-BSN and MM-BSN were provided by their authors, and we trained them on synthetic datasets and real datasets to achieve optimal performance.

[0107] Table 1 shows the PSNR and SSIM results of the algorithm proposed in the present invention and other advanced algorithms on the synthetic dataset. The best two results are shown in bold. Among them, Self-BSR* represents the inference result after using the feature resampling operation in Self-BSR.

[0108]

[0109] Self-SEID aims to remove both random noise and stripe noise in images in a self-supervised learning manner. Table 1 shows the denoising results on the synthetic dataset. When the noise level is low, the PSNR values of the four denoising methods of 1DGF, LRDIS, TLR, and SEID are relatively close, and among them, the SSIM value of SIED is the highest. However, since the artificial priors used by these non-learning denoising methods are difficult to handle complex noise distributions, the denoising metrics are not ideal. Compared with the state-of-the-art algorithms, the PSNR and SSIM values of the supervised learning-based algorithms SNRCNN and TSWEU have a large gap. We believe this is because SNRCNN and TSWEU cannot adapt to different noise levels, resulting in performance degradation. When the noise level is low, the self-supervised learning-based MMBSN and AP-BSN show competitive performance, but as the noise level increases, the performance of these methods drops sharply. This is because these self-supervised denoising methods solely rely on the blind spot strategy and are difficult to handle stripe noise. At different noise levels, the proposed Self-BSR* has better denoising performance than the comparison algorithms. Among them, at the noise level of σ random = 5 and σ stripe = 5, the PSNR value of Self-BSR* is 3.75 dB higher than that of the SOTA algorithm SEID, and the SSIM value is also higher than that of SEID. This benefits from the blind spot regularization method used by Self-BSR, which can finely model the clear image and stripe noise, significantly improving the denoising performance.

[0110] Figure 5 shows the denoising and destriping results of these algorithms when the noise level is σ random = 5 and σ stripe = 5. 1DGF focuses on dealing with stripe noise and is difficult to handle random noise, resulting in a large amount of random noise remaining. Both LRDIS and SEID have a small amount of stripe residue, such as Figure 5As shown in the red box, it is not very good at handling random noise. The TLR algorithm had artifacts in the first test and streaky residues in the second test, which may be related to the use of block-based low-rank priors in TLR. SEID achieved excellent results in smooth areas with less information, but obvious artifacts appeared when processing edge information of the image. TSWEU showed competitive performance in stripe removal, but it was difficult to effectively suppress random noise. MMBSN effectively removed random noise, but the denoising result showed obvious blur, which was the aliasing of clear image and streaky noise. Our method Self-BSR uses the deep image prior captured by the network, so its ability to suppress random noise and finely model clear images is significantly better than other comparison methods.

[0111] Table 2 PRNU of real infrared image denoising results. The smaller the PRNU value, the smaller the non-uniformity.

[0112]

[0113] Table 2 shows the PRNU results of different denoising and stripe removal methods on real noisy images. It can be seen that the proposed Self-BSR* and Self-BSR achieve the best and second-best results in the PRNU indicator, respectively. This shows that compared with other methods, the proposed self-supervised framework has stronger denoising and stripe removal performance for uniform areas of the image.

[0114] Figure 6 The denoising and stripe removal results of different methods on real noisy images are shown. It can be seen that Self-BSR retains more image structures and details while removing stripe noise and random noise. Among the non-learning denoising methods, LRSID and SEID have a small amount of stripe noise residue, as shown in the red box in the figure (it can be observed more clearly after zooming in). 1DGF, LRSID and SEID have poor suppression effects on random noise, resulting in a large amount of random noise residue. Although TLR effectively removes stripe noise and random noise, distortion and artifacts are generated in the denoising results, such as the number 25 in the picture. Among the supervised learning-based methods, SNRCNN and TSWEU can suppress stripe noise, but a large amount of random noise residue also appears. Among the self-supervised learning-based methods, MM-BSN seriously destroys image details while removing noise.

[0115] like Figure 7As shown, to more intuitively observe the stripe removal effect, the average cross-profile of the reconstruction results was analyzed. The abscissa of each curve graph represents the number of columns, and the ordinate is the pixel value in the same row. For better visualization, we only selected the row numbers between [0, 400]. It can be seen from the figure that the contour curve trends of the denoising results of 1DGF, SEID, and TSWEU deviated significantly from those of the original image. The contour curves of the denoising results of LRSID and TLR have similar trends to the original image, but there is a certain gap between them at the places where pixel mutations occur. The reconstructed images of SNRCNN, MM-BSN, and Self-BSR* have similar average cross-profile curve trends to the original image. Among them, the average cross-profile curve of the SNRCNN reconstructed image is too smooth, and there are a large number of sharp burrs in the curve of MM-BSN.

[0116] Considering both visual quality evaluation and objective metric evaluation, for real noisy images, the proposed Self-BSR* achieves an appropriate compromise between denoising and preserving image information, outperforming other comparison methods.

[0117] The optimization method based on feature resampling improves the quality of image reconstruction by expanding the blind spot receptive field during the model inference stage. First, to study the influence of the upsampling scales (s h , s w ) in equation () on the denoising performance, we conducted a series of experiments on the synthetic dataset for different scales. The comparison of the average PSNR and SSIM of the denoising results at different upsampling scales is shown in Table 3. It can be seen from the table that when (s h , s w ) = (2, 1), both the PSNR and SSIM values of the denoising results are significantly higher than those when (s h , s w ) = (1, 2). This is because the stripe direction in the synthetic dataset is vertical, and the influence of vertical upsampling on the stripe distribution is less than that of horizontal upsampling. Therefore, the gain brought by vertical upsampling is greater than that of horizontal upsampling. When s h = 2, we further analyzed the influence of the s w value on the denoising performance. Secondly, the model denoising performance increases as s w gradually increases and reaches the optimum when s w = 1.6, and then gradually decreases.

[0118] Table 3 The average PSNR and SSIM of the denoising results corresponding to FRR with different scale factors on the synthetic dataset.

[0119]

[0120]

[0121] When the upsampling factor (s h , s w ) is (2, 1.6), the PSNR value of Self-BSR with the FRR denoising model is 0.51 dB higher than that of Self-BSR without FRR, and the SSIM value is 0.0106 higher. Figure 8 It shows the visual comparison of the denoising model Self-BSR with and without using FRR. It can be seen that although Self-BSR without using FRR can effectively remove stripe noise and random noise, artifacts and detail loss occur. Self-BSR using FRR can recover more details, which benefits from the fact that FRR expands the receptive field of the blind spot. The above experiments prove the effectiveness of FRR.

[0122] Figure 8 In (a) is the noisy image, (b) is the clean image, (c) is the denoising effect of Self-BSR without using the FRR method, and (d) is the denoising effect of Self-BSR using the FRR method, where the upsampling factor (s h , s w ) = (2, 1.6). It can be seen that thanks to the expansion of the receptive field of the blind spot, FRR effectively eliminates the artifacts caused by the blind spot strategy and retains more image details.

[0123] The present invention proposes Self-BSR, a self-supervised stripe estimation and image denoising model based on blind spot regularization. Self-BSR can refine the separate image and stripes by using the spatially continuous characteristics learned by the improved blind spot network and effectively suppress random noise. The blind spot regularization can be adaptively adjusted according to the depth image prior captured by the network, replacing the traditional parametric modeling. We prove from the theoretical level that the blind spot regularization enables Self-BSR to be trained only using noisy images. Self-BSR combines the advantages of traditional model-based denoising methods and deep learning-based methods, providing new ideas for self-supervised denoising and destriping tasks. Through a large number of experiments on synthetic and real images, the effectiveness and advancement of the proposed model are verified. For future work, we hope to extend the proposed method to the modeling and removal of arbitrarily oriented stripes and broken stripes to improve the application scope in the real world.

[0124] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, in any aspect, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention. Any reference signs in the claims should not be construed as limiting the claims involved.

[0125] In addition, it should be understood that although this specification is described in terms of embodiments, not every embodiment only contains an independent technical solution. This narrative manner of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An image denoising and despeckling method based on blind spot regularization, characterized in that Including: Construct a dual-output blind spot network, use the outputs of the image reconstruction branch and the stripe estimation branch of the dual-output blind spot network as regularization constraint terms respectively, jointly model the clear image and the stripe component through an implicit network prior, and use blind spot regularization to remove random noise; Introduce a direction representation shuffling technique that satisfies J-invariance in the dual-output blind spot network, and separate the image and the stripe through multi-directional feature decomposition and vertical direction feature enhancement; Perform a scale-adjustable feature resampling operation on the input features during the model inference stage to expand the blind spot receptive field and optimize the image reconstruction quality; Output the denoised clear image and the estimated stripe component by alternately optimizing the joint loss function of the dual-output blind spot network.

2. The method for image denoising and stripe removal based on blind spot regularization according to claim 1, wherein Construct a dual-output blind spot network including an image reconstruction branch and a stripe estimation branch, and use the output as a regularization constraint term, where: The image reconstruction branch adopts an omni-directional feature extraction module, and the stripe estimation branch adopts a vertical direction feature extraction module. The loss function is defined as: Among them, Y is the image disturbed by random noise and stripe noise, and B x (Y) is the output of the image reconstruction branch, and B s (Y) is the output of the stripe estimation branch, ||·|| TV is the unidirectional total variation constraint, and λ1 and λ2 are weight parameters.

3. The method for image denoising and stripe removal based on blind spot regularization according to claim 1, characterized in that, The implementation methods of the implicit network prior joint modeling and the blind spot regularization include: In the image reconstruction branch, dilated convolution is used to mask the central pixel, and the spatial connectivity prior of the image is learned through residual connection; In the stripe estimation branch, the low-rank constraint of the stripe branch is achieved by constraining the stripe mean; In the joint optimization process, an alternating update strategy is adopted.

4. The method for image denoising and descreening based on blind spot regularization according to claim 1, wherein Use the direction representation shuffling technique to perform multi-directional decomposition on the input features, including Perform a dilated wavelet transform on the input features, and decompose them into horizontal direction H, vertical direction V, diagonal direction D, and low-frequency component A; Fuse the omni-directional features in the image reconstruction branch, and only retain the vertical direction feature V in the stripe estimation branch; Maintain J-invariance through dilated convolution to ensure that the output pixel is not affected by the input corresponding position pixel.

5. The method for image denoising and descreening based on blind spot regularization according to claim 1, characterized in that Optimize the feature resampling of the masked convolutional layer during the model inference stage, including: Bilinear upsampling is performed on the input feature, and the upsampling scale factor is (s h , s w ), where s h and s w correspond to the magnification factors of height and width respectively; Perform a masked convolution operation on the upsampled features; Restore the features to the original size through downsampling to generate the final output.

6. The method for image denoising and descreening based on blind spot regularization according to claim 1, characterized in that, The method adopts an asymmetric pixel shuffling downsampling strategy during the training stage, that is, uses asymmetric recombined pixels to reconstruct the image to reduce the spatial correlation of noise.

7. The method for image denoising and descreening based on blind spot regularization according to claim 1, characterized in that, The dual-output blind spot network includes a cross-branch information interaction module, which fuses features using a residual block by concatenating the intermediate features of the image reconstruction branch and the stripe estimation branch.

8. The method for image denoising and stripe removal based on blind spot regularization according to claim 5, characterized in that, Adjust the upsampling scale factor during the model inference stage to adapt to image inputs with different random noise and stripe noise distributions.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1-8.

Images