Image restoration method based on metasurface point spread function and denoising regularization method

Abstract

The invention discloses an image restoration method based on a metasurface point spread function and a denoising regularization method, and relates to an image restoration method. The objective of the invention is to solve the problem of poor spectral accuracy and spatial resolution of a recovered image caused by neglect of a metasurface point spread function during imaging of an existing metasurface imaging system. The method comprises the following steps of: analyzing a point spread function of a metasurface unit; image degradation, namely image restoration modeling and image preprocessing; obtaining a de-noising device; a regularization item is constructed; constructing a target function; obtaining a denoised metasurface imaging result; obtaining a trained deep learning network; and inputting a to-be-detected original image into the trained deep learning network, and outputting a denoised image by the trained deep learning network. The invention belongs to the technical field of image restoration in a metasurface imaging system.

Description

Technical Field

[0001] The invention relates to an image restoration method, and belongs to the technical field of image restoration in a super-surface imaging system. Background Art

[0002] In the existing metasurface hyperspectral imaging system, the influence of the point spread function of the optical metasurface on the imaging results is ignored during the computational reconstruction of hyperspectral data. The influence of the point spread function on the metasurface imaging process is as follows. Since the incident spectrum of each metasurface micro-spectrometer is assumed to be the same, when the imaging spot caused by the point spread function is large, the spectral information on a single micro-spectrometer can be considered the same, and the accuracy of the spectral reconstruction can be guaranteed, but the spatial resolution of the reconstructed hyperspectral image depends on the size of the micro-spectrometer, and a mosaic effect occurs; when the imaging spot caused by the point spread function is small, there are differences in the spectral information received on a micro-spectrometer, but the computational imaging process assumes that the spectral information on the micro-spectrometer is the same, which will cause problems with the accuracy of the spectral information during reconstruction, and a mosaic phenomenon will occur.

[0003] In recent years, the method of using denoiser as a regularizer, i.e., denoising regularization (RED), has achieved remarkable results in the field of image processing. Traditional image denoising methods are mostly based on statistical models or filtering algorithms, such as mean filtering, median filtering, and Wiener filtering. These methods can remove noise to a certain extent, but are often accompanied by blurring of image details or loss of edge information. The core idea of ​​the method of using denoiser as a regularizer is to embed the denoising process into the optimization framework of image processing as part of the regularizer. Regularizers are usually used to constrain the solution space in optimization problems to prevent overfitting or unstable solutions. In the context of denoising, regularizers are used to balance the relationship between denoising effect and image fidelity, that is, to keep the original information and structure of the image as much as possible while removing noise.

[0004] With the rise of deep learning technology, deep learning-based denoisers have shown powerful capabilities in image processing. Deep learning models, especially convolutional neural networks (CNNs), can automatically learn the feature representation of images and use these features for denoising. Using such deep learning denoisers as part of the regularization term can further improve the image denoising effect and adapt to different types of noise and image content. Summary of the invention

[0005] In order to solve the problem that the existing metasurface imaging system ignores the metasurface point spread function during imaging, resulting in poor spectral accuracy and spatial resolution of the restored image, the present invention proposes an image restoration method based on the metasurface point spread function and a denoising regularization method.

[0006] The technical solution adopted by the present invention to solve the above-mentioned problem is: the steps of the present invention include:

[0007] Step 1: point spread function analysis of the metasurface unit;

[0008] Step 2: Image degradation, i.e. image restoration modeling and image preprocessing; imaging result analysis is performed based on the point spread function modeling results of the hypersurface unit, and a simple preprocessing is performed on the acquired original image;

[0009] Step 3, obtaining a denoiser; selecting a denoiser according to the optical system model and the image degradation model obtained in step 2;

[0010] Step 4: Based on the denoiser obtained in step 3, a regularisation term is constructed; the output metric of the denoiser is used as part of the regularisation, which is achieved by defining a penalty function associated with the denoiser output;

[0011] Step 5: Combine the data fidelity term and the regularization term constructed in step 4 to construct the objective function;

[0012] Step 6: Use an optimization algorithm to solve the objective function constructed in step 5 to obtain the denoised metasurface imaging result; iteratively update the estimated value of the denoised image, and use the denoiser for processing in each iteration;

[0013] Step 7: Use the image preprocessed in step 2 as the input of the deep learning network, and the denoised image as the output of the deep learning network. Perform self-supervised training based on the penalty function in the objective function constructed in step 5 until convergence to obtain a trained deep learning network.

[0014] Step 8: Input the original image to be tested into the trained deep learning network, and the trained deep learning network outputs the denoised image.

[0015] Furthermore, the specific process of step 1 is:

[0016] When the optical system is linearly degraded, that is, the point spread function space is unchanged, without considering the noise, the imaging process is expressed by the following formula:

[0017]

[0018] In formula (1), I(x, y) represents the degraded image, S(u, v) represents the original image, and P(xu, yv) represents the spatially invariant point spread function;

[0019] The image degradation process of the spatially varying point spread function with the field of view is expressed as:

[0020]

[0021] In formula (2), P(u, v, xu, yv) represents the spatially varying point spread function.

[0022] Furthermore, the specific process of step 2 is:

[0023] The image degradation process can be regarded as a linear space-invariant degradation system, and its process can be described as: the clear image f(x, y) is convolved with the corresponding blur kernel h(x, y) to generate the degraded image g(x, y) under the influence of noise;

[0024] g(x, y)=h(x, y)*f(x, y)+n(x, y) (3),

[0025] In formula (3), * represents the convolution operator;

[0026] When the point spread function is spatially invariant, the blurred image is expressed as:

[0027]

[0028] The blurred image is represented as:

[0029]

[0030] Furthermore, the regularization term expression in step 4 is:

[0031]

[0032] In formula (6), R(y) represents the regularization term, D σ (y) represents the denoiser, y represents the preprocessed image, and the superscript T represents the transposition.

[0033] Furthermore, the objective function of step 5 is expressed as:

[0034]

[0035] In formula (7), x represents the target image, y represents the preprocessed image, l(y, x) represents the data fidelity term, that is, ensuring that y≈x, λ represents the degree of penalty for R(y), and the weight between the data fidelity term and the regularization term is adjusted according to actual needs to balance the relationship between the two. λR(y) represents the penalty function.

[0036] Furthermore, the specific process of step 6 is as follows:

[0037] Step 601: construct the Lagrangian augmented function:

[0038]

[0039] Formula (8), f(x) represents the intermediate variable, It is defined as, p(y|x) represents the prior probability of finding x given y, g(x) represents the intermediate variable, p(x) represents the probability of x, f(x)+λg(v) is the original optimization target, and v represents the target image; under the condition that x=v, (xv) is the equality constraint, u is the Lagrange multiplier associated with the constraint (xv)=0, and ρ represents the penalty coefficient;

[0040] Step 602: Set the maximum number of iterations K;

[0041] Step 603, number of iterations k=1;

[0042] Step 604: To solve the minimum problem, the Lagrangian augmented function is transformed into the following sub-problems:

[0043]

[0044] u (k+1) =u (k) +(x (k+1) -v (k+1) ) (11),

[0045] ρ k+1 =γ k ρ k (12)

[0046] In formulas (9), (10), (11) and (12), x (k+1) represents the target image of the k+1th iteration, x (k) represents the target image of the kth iteration, ρ k represents the penalty coefficient for the kth iteration, ρ k+1 represents the penalty coefficient for the k+1th iteration, v (k) represents the target image of the kth iteration, v (k+1 ) represents the target image of the k+1th iteration, u (k) represents the Lagrange multiplier associated with the constraint (xv) = 0 at the kth iteration, u (k+1) represents the Lagrange multiplier associated with the constraint (xv) = 0 at the k+1th iteration, D σk represents the denoiser at the kth iteration, γ k Represents ρ k The adjustment parameter, Dσk is the denoising algorithm, is a parameter that controls the strength of the denoiser, improving ρ to ρ k+1 =γ k ρ k , γ k ≥1 is the tuning parameter of ρ.

[0047] Step 605: Set the number of iterations k = k + 1, and repeat step 54 until the maximum number of iterations K is reached to obtain x K That is, the target image x in the objective function, satisfying min x I(y,x)+λR(y) corresponds to.

[0048] Furthermore, the specific process of step 7 is as follows:

[0049] The image preprocessed in step 2 is used as the input of the deep learning network, and the target image x obtained in step 6 is used as the output of the deep learning network. Self-supervised training is performed based on the penalty function in the objective function constructed in step 5 until convergence to obtain a trained deep learning network.

[0050] The beneficial effects of the present invention are:

[0051] 1. The present invention solves the problem of spectral accuracy and spatial resolution of restored images caused by the existing metasurface imaging system ignoring the metasurface point spread function during imaging;

[0052] 2. The present invention utilizes the denoising regularization method to introduce a regularization term to constrain the complexity of the model, thereby achieving effective noise suppression. Compared with traditional denoising methods, this method can better retain the original features and structural information of the image while removing noise.

[0053] 3. The present invention utilizes denoising to constrain the solution space of the final data, which is similar to the effect of the regularization term utilizing prior knowledge to constrain the solution space. Therefore, the denoiser is embedded in the optimization framework, and the denoising algorithm is used to define the regularization term, thereby achieving the regularization effect while realizing denoising. While obtaining higher quality data, the generalization ability of the model is enhanced, the denoising accuracy is improved, and the error between the denoised image and the real image is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the image degradation and restoration model. DETAILED DESCRIPTION

[0055] Specific implementation method 1: Figure 1 As shown, the image restoration method based on the hypersurface point spread function and the denoising regularization method includes the following specific steps:

[0056] Step 1: point spread function analysis of the metasurface unit;

[0057] When the optical system is linearly degraded, that is, the point spread function space is unchanged, without considering the noise, the imaging process is expressed by the following formula:

[0058]

[0059] In formula (1), I(x, y) represents the degraded image, S(u, v) represents the original image, and P(xu, yv) represents the spatially invariant point spread function;

[0060] The image degradation process of the spatially varying point spread function with the field of view is expressed as:

[0061]

[0062] In formula (2), P(u, v, xu, yv) represents the spatially varying point spread function;

[0063] Step 2: Image degradation, i.e. image restoration modeling and image preprocessing; imaging result analysis is performed based on the point spread function modeling results of the hypersurface unit, and a simple preprocessing is performed on the acquired original image;

[0064] The image degradation process can be regarded as a linear space-invariant degradation system, and its process can be described as: the clear image f(x, y) is convolved with the corresponding blur kernel h(x, y) to generate the degraded image g(x, y) under the influence of noise;

[0065] g(x, y)=h(x, y)*f(x, y)+n(x, y) (3),

[0066] In formula (3), * represents the convolution operator;

[0067] When the point spread function is spatially invariant, the blurred image is expressed as:

[0068]

[0069] The blurred image is represented as:

[0070]

[0071] Step 3, obtaining a denoiser; selecting a denoiser according to the optical system model and the image degradation model obtained in step 2;

[0072] Step 4: Based on the denoiser obtained in step 3, construct a regularization term; take the output metric of the denoiser as part of the regularization by defining a penalty function related to the denoiser output; the regularization term expression is:

[0073]

[0074] In formula (6), R(y) represents the regularization term, D σ () represents the denoiser, y represents the preprocessed image, and the superscript T represents the transposition;

[0075] Step 5: Combine the data fidelity term and the regularization term constructed in step 4 to construct the objective function; the objective function is expressed as:

[0076]

[0077] In formula (7), x represents the target image, y represents the preprocessed image, l(y, x) represents the data fidelity term, that is, ensuring that y≈x, λ represents the degree of penalty for R(y), and the weight between the data fidelity term and the regularization term is adjusted according to actual needs to balance the relationship between the two, and λR(y) represents the penalty function;

[0078] Step 6: Use an optimization algorithm to solve the objective function constructed in step 5 to obtain the denoised metasurface imaging result; iteratively update the estimated value of the denoised image, and use the denoiser for processing in each iteration;

[0079] Step 601: construct the Lagrangian augmented function:

[0080]

[0081] Formula (8), f(x) represents the intermediate variable, It is defined as, p(y|x) represents the prior probability of finding x given y, g(x) represents the intermediate variable, p(x) represents the probability of x, f(x)+λg(v) is the original optimization target, and v represents the target image; under the condition that x=v, (xv) is the equality constraint, u is the Lagrange multiplier associated with the constraint (xv)=0, and ρ represents the penalty coefficient;

[0082] Step 602: Set the maximum number of iterations K;

[0083] Step 603, number of iterations k=1;

[0084] Step 604: To solve the minimum problem, the Lagrangian augmented function is transformed into the following sub-problems:

[0085]

[0086] u (k+1) =u (k) +(x (k+1) -v(k+1) ) (11),

[0087] ρ k+1 =γ k ρ k (12)

[0088] In formulas (9), (10), (11) and (12), x (k+1) represents the target image of the k+1th iteration, x (k) represents the target image of the kth iteration, ρ k represents the penalty coefficient for the kth iteration, ρ k+1 represents the penalty coefficient for the k+1th iteration, v (k) represents the target image of the kth iteration, v (k+1) represents the target image of the k+1th iteration, u (k) represents the Lagrange multiplier associated with the constraint (xv) = 0 at the kth iteration, u (k+1) represents the Lagrange multiplier associated with the constraint (xv) = 0 at the k+1th iteration, D σk represents the denoiser at the kth iteration, γ k Represents ρ k The adjustment parameter, D σk is the denoising algorithm, is a parameter that controls the strength of the denoiser, improving ρ to ρ k+1 =γ k ρ k , γ k ≥1 is the tuning parameter of ρ.

[0089] Step 605: Set the number of iterations k = k + 1, and repeat step 54 until the maximum number of iterations K is reached to obtain x K That is, the target image x in the objective function, satisfying min x l(y,x)+λR(y) corresponds to;

[0090] Step 7: Use the image preprocessed in step 2 as the input of the deep learning network, and the denoised image as the output of the deep learning network. Perform self-supervised training based on the penalty function in the objective function constructed in step 5 until convergence to obtain a trained deep learning network.

[0091] The image preprocessed in step 2 is used as the input of the deep learning network, and the target image x obtained in step 6 is used as the output of the deep learning network. Self-supervised training is performed based on the penalty function in the objective function constructed in step 5 until convergence to obtain a trained deep learning network.

[0092] Step 8: Input the original image to be tested into the trained deep learning network, and the trained deep learning network outputs the denoised image.

[0093] Among them, noise is the unwanted interference or distortion caused by various factors in the process of image acquisition, transmission, storage or processing. Common types of noise include Poisson noise, Gaussian noise, salt and pepper noise, etc.

[0094] The goal of image denoising is to recover a clean image x from a noisy observation image y. Its model is: y = xn, where n is generally additive white Gaussian noise or Poisson noise. From this model, it can be seen that image denoising is an inverse problem.

[0095] Design or select a denoiser: The denoiser based on the statistical model is a bilateral filter or a Gaussian filter, etc. These filters can smooth the noise while preserving the edge information as much as possible.

[0096] How it works

[0097] The Plug-and-Play ADMM algorithm has good performance in terms of convergence and final data recovery effect; the traditional ADMM algorithm:

[0098] The data recovery problem can be viewed as a maximum a posteriori probability estimation problem.

[0099]

[0100] Where x is the target image, y is the image output by the sensor, is the estimate of x given that y is known,

[0101] Convert the unconstrained problem into a constrained problem:

[0102]

[0103] Obey x=v

[0104] Consider the Lagrangian augmentation:

[0105]

[0106] Where f(x)+λg(v) is the original optimization objective, subject to the condition x=v, (xv) is the equality constraint, and u is the Lagrange multiplier associated with the constraint (xv)=0.

[0107] and the regularization term constructed in step 3 Combined,

[0108]

[0109] In order to solve the minimum problem, we turn to the following sub-problems:

[0110]

[0111] u (k+1) =u (k) +(x (k+1) -v (k+1) )

[0112] Plug-and-PlayADMM algorithm:

[0113] v (k+1) =D σ (v (k) )

[0114] Among them, D σ It is a ready-made denoising algorithm.

[0115] Improved Plug-and-PlayADMM algorithm:

[0116]

[0117] u (k+1) =u (k) +(x (k+1) -v (k+1) )

[0118] ρ k+1 =γ ｋ ρ k

[0119] Among them, D σ is the denoising algorithm ("denoiser" for short), is a parameter that controls the strength of the denoiser, improving ρ to ρ k+1 =γ k ρ k , γ k ≥1 is the tuning parameter of ρ.

[0120] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. An image restoration method based on a hypersurface point spread function and a denoising regularization method, characterized in that: The specific steps include: Step 1: point spread function analysis of the metasurface unit; Step 2: Image degradation, i.e. image restoration modeling and image preprocessing; imaging result analysis is performed based on the point spread function modeling results of the hypersurface unit, and a simple preprocessing is performed on the acquired original image; Step 3, obtaining a denoiser; selecting a denoiser according to the optical system model and the image degradation model obtained in step 2; Step 4: Based on the denoiser obtained in step 3, a regularisation term is constructed; the output metric of the denoiser is used as part of the regularisation, which is achieved by defining a penalty function associated with the denoiser output; Step 5: Combine the data fidelity term and the regularization term constructed in step 4 to construct the objective function; Step 6: Use an optimization algorithm to solve the objective function constructed in step 5 to obtain the denoised metasurface imaging result; iteratively update the estimated value of the denoised image, and use the denoiser for processing in each iteration; Step 7: Use the image preprocessed in step 2 as the input of the deep learning network, and the denoised image as the output of the deep learning network. Perform self-supervised training based on the penalty function in the objective function constructed in step 5 until convergence to obtain a trained deep learning network. Step 8: Input the original image to be tested into the trained deep learning network, and the trained deep learning network outputs the denoised image.

2. The image restoration method based on the hypersurface point spread function and denoising regularization method according to claim 1, characterized in that: The specific process of step 1 is: When the optical system is linearly degraded, that is, the point spread function space is unchanged, without considering the noise, the imaging process is expressed by the following formula: In formula (1), I(x, y) represents the degraded image, S(u, v) represents the original image, and P(xu, yv) represents the spatially invariant point spread function; The image degradation process of the spatially varying point spread function with the field of view is expressed as: In formula (2), P(u, v, xu, yv) represents the spatially varying point spread function.

3. The image restoration method based on the hypersurface point spread function and denoising regularization method according to claim 1, characterized in that: The specific process of step 2 is: The image degradation process can be regarded as a linear space-invariant degradation system, and its process can be described as: the clear image f(x, y) is convolved with the corresponding blur kernel h(x, y) to generate the degraded image g(x, y) under the influence of noise; g(x,y)=h(x,y)*f(x,y)+n(x,y) (3), In formula (3), * represents the convolution operator; When the point spread function is spatially invariant, the blurred image is represented as: The blurred image is represented as:

4. The image restoration method based on the hypersurface point spread function and denoising regularization method according to claim 1, characterized in that: The regularization term expression in step 4 is: In formula (6), R(y) represents the regularization term, D σ (y) represents the denoiser, y represents the preprocessed image, and the superscript T represents the transposition.

5. The image restoration method based on the hypersurface point spread function and denoising regularization method according to claim 1, characterized in that: Step 5 The objective function is expressed as: In formula (7), x represents the target image, y represents the preprocessed image, l(y, x) represents the data fidelity term, that is, ensuring that y≈x, λ represents the degree of penalty for R(y), and the weight between the data fidelity term and the regularization term is adjusted according to actual needs to balance the relationship between the two. λR(y) represents the penalty function.

6. The image restoration method based on the hypersurface point spread function and denoising regularization method according to claim 1, characterized in that: The specific process of step 6 is: Step 601: construct the Lagrangian augmented function: Formula (8), f(x) represents the intermediate variable, It is defined as, p(y|x) represents the prior probability of finding x given y, g(x) represents the intermediate variable, p(x) represents the probability of x, f(x)+λg(v) is the original optimization target, and v represents the target image; under the condition that x=v, (xv) is the equality constraint, u is the Lagrange multiplier associated with the constraint (xv)=0, and ρ represents the penalty coefficient; Step 602: Set the maximum number of iterations K; Step 603, number of iterations k=1; Step 604: To solve the minimum problem, the Lagrangian augmented function is transformed into the following sub-problems: u (k+1) =u (k) +(x (k+1) -v (k+1) ) (11), r k+1 =c k r k (12), In formulas (9), (10), (11) and (12), x (k+1) represents the target image of the k+1th iteration, x (k) represents the target image of the kth iteration, ρ k represents the penalty coefficient for the kth iteration, ρ k+1 represents the penalty coefficient for the k+1th iteration, v (k) represents the target image of the kth iteration, v (k+1) represents the target image of the k+1th iteration, u (k) represents the Lagrange multiplier associated with the constraint (xv) = 0 at the kth iteration, u (k+1) represents the Lagrange multiplier associated with the constraint (xv)=0 at the k+1th iteration, D σk represents the denoiser at the kth iteration, γ k Represents ρ k The adjustment parameter, D σk is the denoising algorithm, is a parameter that controls the strength of the denoiser, improving ρ to ρ k+1 =γ k ρ k , γ k ≥1 is the tuning parameter of ρ. Step 605: Set the number of iterations k = k + 1, and repeat step 54 until the maximum number of iterations K is reached to obtain x K That is, the target image x in the objective function, satisfying min x l(y,x)+λR(y) corresponds to.

7. The image restoration method based on the hypersurface point spread function and denoising regularization method according to claim 1, characterized in that: The specific process of step 7 is: The image preprocessed in step 2 is used as the input of the deep learning network, and the target image x obtained in step 6 is used as the output of the deep learning network. Self-supervised training is performed based on the penalty function in the objective function constructed in step 5 until convergence to obtain a trained deep learning network.

Images