Low-light lens-free imaging enhancement method based on diffusion model
The low-light lensless imaging enhancement method uses a diffusion model to stabilize and enhance image quality by combining minimum mean square error filtering and wavelet transformation, addressing the challenges of low-light imaging in lensless systems.
Patent Information
- Application Number
- CN202411165696.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-03
Smart Images

Figure CN119067870A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of lensless imaging, and in particular to the field of mask-based lensless imaging enhancement under low-light conditions. Background Art
[0002] Although advances in miniaturization technology for lenses and sensors have reduced the size of traditional cameras by nearly two orders of magnitude, they are also gradually approaching physical limits: the diffraction limit and the Lohman scaling law. Traditional camera lenses are composed of a variety of optical components such as lenses, apertures, and lens barrels. High-precision optical components require high production costs, which not only increases the weight, size, and complexity of the camera, but also bears most of the production costs of the camera. The contradiction between the physical limitations of optical lenses on camera size and the miniaturization and thinness of imaging devices has given rise to lensless imaging technology: using optical modulation mask coding to replace lens focusing, and restoring clear images through computational imaging technology. In recent years, cheap and lightweight lensless imaging has gained attention as a potential alternative to traditional lens imaging.
[0003] However, compared with lens cameras, the imaging scene of lensless cameras is more narrow and the light is weaker, so the images obtained usually contain more noise and lower brightness, and the image quality restored by computational imaging algorithms is also poor. Among the various factors that affect imaging quality, low illumination problems are more common and difficult to avoid, such as application scenarios at night and inadequate lighting in narrow spaces. In addition, lensless imaging loses the ability of lenses to converge light, making it more challenging to achieve high-quality imaging in low-light environments. In terms of image enhancement technology for traditional cameras, researchers have added targeted processing methods to the design of imaging models to address various interferences caused by insufficient light. Due to the different mechanisms of lensless imaging and the particularity of data collected after mask modulation, this type of weak-light enhancement algorithm cannot be directly applied to the lensless field. The quality of reconstructed images using existing data-driven methods is not satisfactory, and the details and color information of the scene cannot be well restored. At present, there are relatively few studies on low-light lensless reconstruction and enhancement.
[0004] Therefore, in order to solve the above problems, the present invention proposes a low-light lensless reconstruction and enhancement method based on a diffusion model, which can better reconstruct low-light lensless images and achieve good enhancement effects. First, the original low-light lensless measurement results are roughly imaged using the minimum mean square error filtering algorithm to obtain preliminary imaging results, but they still contain low-quality images with noise and low light. Then, the image is transformed into the wavelet domain using the Harr wavelet, and the low-light lensless reconstruction and enhancement model constructed based on the diffusion model is finely enhanced to obtain a high-quality realistic image with normal light and low noise. In order to ensure the stability of the restoration results obtained by the reverse diffusion process, the forward diffusion process and the reverse diffusion process are executed simultaneously in the training stage, so that the sampling results of the low-light lensless reconstruction and enhancement model are more stable, that is, the results of the low-light lensless reconstruction and enhancement are more stable. The experimental results show that the proposed method has a good reconstruction effect, can significantly improve the brightness of the image while ensuring better image quality, and can realize the reconstruction and enhancement of lensless low-light images. Summary of the invention
[0005] The present invention discloses a low-light lensless imaging enhancement algorithm based on a diffusion model. The diffusion model is introduced into the low-light lensless imaging enhancement. First, the original low-light lensless measurement result is coarsely imaged using the minimum mean square error filtering algorithm to obtain a low-light and noise-containing coarse imaging image. Compared with other optimization algorithms, no iteration is required and the reconstruction speed is faster. Then, the coarse imaging image is transformed into the wavelet domain using the Harr wavelet transform, and the high-frequency information and low-frequency information are restored and reconstructed using the high-frequency information recovery network and the diffusion model respectively. Finally, the Harr wavelet inverse transform is used to obtain a high-quality normal light image from the restored high-frequency information and low-frequency information. Experimental results show that the imaging effect of the present invention is better and a high-quality lensless image can be reconstructed. In the training stage of the low-light lensless reconstruction enhancement model, the method of the present invention not only completes the forward diffusion process, but also performs the reverse diffusion process, overcomes the problems of color confusion and unstable generation results of the existing method, and ensures the robustness of imaging. After the wavelet transform module transforms the image into the wavelet domain, it also reduces the training time cost and resource cost of the present method.
[0006] In order to facilitate the description of the present invention, the following terms are first defined:
[0007] Definition 1: Classic LOLv2 dataset
[0008] The classic LOLv2 dataset refers to the low-light dataset, which is called Low Light Dataset in English. It can train deep learning models for low-light enhancement tasks, and is used by researchers to evaluate the performance of their algorithms in this unified dataset. The dataset contains two sub-datasets, Real_captured and Synthetic. The Real_captured sub-dataset contains 789 pairs of low-light and normal-light images, and the Synthetic sub-dataset contains 1000 pairs of low-light and normal-light images. The complete dataset can be obtained from "Yang W, Wang W, Huang H, et al. Sparse gradient regularized deepretinex network for robust low-light image enhancement[J]. IEEE Transactions on Image Processing, 2021, 30: 2072-2086."
[0009] Definition 2: Standard image cropping and scaling method
[0010] Image cropping is to change the content and composition of an image by selecting an area of interest and removing surrounding pixels. Scaling is to adjust the size and resolution of an image by increasing or decreasing the number of pixels in the image. For details, see the document “https: / / wenku.baidu.com / view / a039009bc181e53a580216fc700abb68a982adbe.html?_wkts_=1722400350810”.
[0011] Definition 3: Standard phase mask method
[0012] A phase mask is a specially designed optical mask. The main function of the standard phase mask method is to introduce phase modulation through the principle of wave optics during the propagation of the incident light wave. The phase mask allows most of the light to pass through, providing a high signal-to-noise ratio. For details of the standard phase mask method, please refer to the literature "Boominathan V, Adams JK, Robinson JT, et al. Phlatcam: Designed phase-mask based thin lensless camera [J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 42 (7): 1618-1629."
[0013] Definition 4: Standard lensless imaging method
[0014] The principle of the lensless imaging system based on mask modulation is to use a mask to modulate the incident light and then collect it with an image sensor. The imaging process can usually be expressed as a convolution process between the target image and the mask image. For details of the standard lensless imaging method, please refer to the literature "Boominathan V, Adams JK, Robinson JT, et al. Phlatcam: Designed phase-mask based thin lensless camera[J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 42(7): 1618-1629.".
[0015] Definition 5: Classical minimum mean square error filtering
[0016] Minimum mean square error filtering, as the name implies, is a signal processing method that minimizes the mean square error between the estimated value and the true value by understanding the relationship between the signal and the noise, filtering out noise and interference from continuous or discrete input data to extract useful information, and then achieving image reconstruction and restoration. For details, see the reference "[1] Gonzalez RC, Woods RE, Masters BR. Digital Image Processing, Third Edition [J]. Journal of Biomedical Optics, 2009, 14 (2): 029901. DOI: 10.1117 / 1.3115362."
[0017] Definition 6: Classical two-dimensional Harr wavelet transform
[0018] Wavelet transform is a commonly used tool in signal processing and data analysis. It can decompose signals into different frequency components and provide local information in time and frequency. The two-dimensional Harr wavelet transform is a generalization of the one-dimensional wavelet transform. By performing horizontal and vertical wavelet transforms on a two-dimensional image, the image is decomposed into wavelet coefficients of different scales and directions, which is called decomposition; the original two-dimensional image can be reconstructed using the decomposed wavelet coefficients, which is called reconstruction. For details, see the literature "[1] Gonzalez RC, Woods RE, Masters BR. Digital Image Processing, Third Edition [J]. Journal of Biomedical Optics, 2009, 14 (2): 029901. DOI: 10.1117 / 1.3115362."
[0019] Definition 7: Standard convolution operation method
[0020] The convolution layer uses a set of learnable filters (also called convolution kernels or feature detectors) to filter the input data spatially. Each filter slides over the input data (convolution operation), and the dot product between the filter and the input data is calculated to generate a feature map. The convolution operation helps to extract local features of the input data. For more information on the standard convolution operation method, see "https: / / cloud.tencent.com / developer / article / 2299603".
[0021] Definition 8: Standard Depthwise Separable Convolutional Layer Method
[0022] Depthwise separable convolution (DSC) consists of depthwise convolution and pointwise convolution. Depthwise convolution is used to extract spatial features, and pointwise convolution is used to extract channel features. For details on the standard depthwise separable convolution layer method, please refer to "https: / / blog.csdn.net / m0_37605642 / article / details / 134174749".
[0023] Definition 9: Standard residual module method
[0024] The residual block is the basic building block of a deep convolutional neural network. The standard residual block method solves the gradient vanishing problem in deep neural networks by introducing "skip connections", so that deeper networks can be trained effectively. The core idea of the residual block is to learn the residual function between input and output, rather than directly learning the mapping from input to output. This makes it easier to optimize the network. For more information about the standard residual block method, please refer to "https: / / blog.csdn.net / SisterRu / article / details / 139853475".
[0025] Definition 10: Standard Cross-Attention Module Method
[0026] The criss-cross attention module is a module used to enhance the relationship between multiple input features in a neural network. The standard criss-cross attention module method performs attention mechanisms on different channels or spatial dimensions so that the network can more effectively learn the dependencies and importance between features, thereby improving its performance and generalization ability. For details of the standard criss-cross attention module method, please refer to "https: / / blog.csdn.net / m0_63097763 / article / details / 132293568".
[0027] Definition 11: Classic Adam algorithm
[0028] The classic Adam algorithm is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It can update all weights based on a single learning rate, and the learning rate does not change during the training process. Adam designs independent adaptive learning rates for different parameters by calculating the first-order moment estimate and the second-order moment estimate of the gradient. For details, see the document "Kingma, D, Ba, J. Adam: A Method for Stochastic Optimization. arXiv 2014, arXiv: 1412.6980".
[0029] Definition 12: Standard forward propagation method
[0030] The forward propagation method is the most basic method in deep learning. It mainly performs forward reasoning on the input according to the parameters and connection methods in the network to obtain the output of the network. For details on the traditional forward propagation method, please refer to "https: / / www.jianshu.com / p / f30c8daebebb".
[0031] Definition 13: Classical mean square error calculation method
[0032] Mean Squared Error (MSE) is a common metric for measuring the difference between two images. It calculates the average squared error between the pixels of two images, that is, the average of the squares of the differences between each pixel value. MSE is often used to compare the overall pixel-level differences of images. The lower the value, the more similar the two images are. For details on the classic mean squared error calculation method, see "https: / / blog.csdn.net / qq_44111805 / article / details / 127705262".
[0033] Definition 14: Classic learning-perceptual image patch similarity calculation method
[0034] Learned Perceptual Image Patch Similarity (LPIPS) is an image quality assessment metric based on deep learning models that aims to capture differences in human perception. It considers not only pixel-level differences but also perceptual differences, so it can better predict human subjective perception. LPIPS is often used to compare the similarity of image structure and content, especially in generative models to evaluate the quality of generated images. For details on the classic learning perceptual image patch similarity calculation method, please refer to "https: / / blog.csdn.net / u013889591 / article / details / 140687263".
[0035] Definition 15: Classical Peak Signal-to-Noise Ratio Calculation Method
[0036] Peak Signal-to-Noise Ratio (PSNR) is a measure of the noise between two images. It compares the root mean square error between the original image and the image being evaluated, and then converts the error into decibels (dB). PSNR is often used to measure the effectiveness of compression algorithms, and higher PSNR values indicate better image quality. However, PSNR is usually inconsistent with human visual perception, so it may not be the best choice in some scenarios. For details on the classic method of calculating peak signal-to-noise ratio, see "https: / / blog.csdn.net / qq_44111805 / article / details / 127676377".
[0037] Definition 16: Classical structural similarity calculation method
[0038] Structural Similarity Index (SSIM) is a metric used to measure the similarity between two images, taking into account not only brightness differences but also contrast and structural information. It simulates the perception of images by the human visual system. SSIM is often used in image quality assessment and image processing tasks, especially when it is necessary to compare the structural similarity of two images. A higher SSIM value indicates that the two images are more similar. For details on the classic structural similarity calculation method, see "https: / / zhuanlan.zhihu.com / p / 399215180".
[0039] Definition 17: Standard Markov chain criterion
[0040] Markov Chain refers to a random process with Markov properties. In a Markov chain, the transition of the current state depends only on the previous state and has nothing to do with the past state sequence. This property is also called memorylessness, that is, the future state is only related to the current state. Markov chains are usually described by a state space and a transition probability matrix. The state space represents a set of possible states, and the transition probability matrix specifies the probability of transitioning from one state to another. For details, see "https: / / blog.csdn.net / ywsydwsbn / article / details / 123609409".
[0041] Definition 18: Standard Gaussian noise generation method
[0042] Gaussian noise is a type of noise generated by adding normally distributed random values with a mean of zero and a standard deviation (σ) to the input data. By generating random values with a normal distribution and adding them to the input data, the image can be represented as a two-dimensional matrix of pixel values, and then the numpy library is used to generate random values with a normal distribution. For details, see "https: / / blog.csdn.net / qq_42722197 / article / details / 136594938".
[0043] Definition 19: Standard skip connection method
[0044] Skip connection is a technique that directly connects certain layers in a neural network to the previous layers. Usually, a skip connection is added directly after the output of a convolutional layer or other layer. Its purpose is to solve the gradient vanishing or gradient exploding problems that may occur during the training of deep neural networks. It also helps to speed up the training process and improve the expressiveness of the model. For details, see "https: / / blog.csdn.net / weixin_44859553 / article / details / 132826791"
[0045] Definition 20: Standard splicing method
[0046] The "low-quality image" and "high-quality image" are concatenated along the specified axis using the concat function and input into the neural network for training. This method is usually used to solve image restoration tasks under specific conditions. This method can effectively improve the network's image processing capabilities. For details, see the document "Saharia C, Ho J, Chan W, et al. Image super-resolution via iterative refinement [J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 45 (4): 4713-4726."
[0047] A low-light lensless imaging enhancement method based on a diffusion model, the method comprising the following steps:
[0048] Step 1: Prepare the dataset
[0049] Based on the classic LOLv2 dataset in Definition 1, the classic image cropping and scale transformation method in Definition 2 is used to process 1000 pairs of low-light and normal-light images in the Synthetic sub-dataset of the LOLv2 dataset;
[0050] Then, the standard phase mask method in Definition 3 is used to obtain the image after phase mask processing, which is recorded as PSF;
[0051] The processed low-light image is simulated and encoded using the standard lens-free imaging method in Definition 4 to obtain a low-light lens-free image;
[0052] Finally, the processed data set is divided into two parts, the first 900 pairs of images are used as the training set, denoted as Train, and the last 100 pairs of images are used as the test set, denoted as Test.
[0053] Step 2: Build a coarse imaging module
[0054] The classic minimum mean square error filter in Definition 5 is used to construct the coarse imaging model as follows:
[0055]
[0056] in, is the Fourier transform, is the inverse Fourier transform, H is the phase mask PSF in step 1, b is the input low-light lens-free image, λ is the regularization coefficient, This is the preliminary imaging result.
[0057] Step 3: Construct wavelet transform module
[0058] Using the classic two-dimensional Harr wavelet transform in Definition 6, the wavelet transform decomposition module is constructed as follows:
[0059] {A,V,H,D}=2D-DWT(X)
[0060] Among them, X represents the input image, 2D-DWT represents two-dimensional wavelet transform, A represents the low-frequency information of the input image, V represents the high-frequency information of the input image in the vertical direction, H represents the high-frequency information of the input image in the horizontal direction, and D represents the high-frequency information of the input image in the diagonal direction.
[0061] Using the classic two-dimensional Harr wavelet transform in Definition 6, a wavelet transform reconstruction module is constructed as follows:
[0062]
[0063] in, represents the reconstructed image, 2D-IDWT represents the two-dimensional inverse wavelet transform, represents the low-frequency information of the reconstructed image, Represents the high-frequency information of the reconstructed image in the vertical direction, Represents the high-frequency information of the reconstructed image in the horizontal direction, Represents the high-frequency information of the reconstructed image in the diagonal direction.
[0064] Step 4: Build a high-frequency information recovery network
[0065] Convolutional layer L1 and convolutional layer L2 are constructed using the standard convolution operation method in Definition 7;
[0066] The convolutional layers L1 and L2 are processed using the standard depth-wise separable convolutional layer method in Definition 8 to construct the first convolutional layer, denoted as Layer_head.
[0067] The standard cross-attention module method in Definition 10 is used to construct a cross-attention layer, denoted as Layer_attention.
[0068] The standard residual module method in Definition 9 is used to construct the residual layer, denoted as Layer_residual.
[0069] Convolutional layer L3 and convolutional layer L4 are constructed using the standard convolution operation method in Definition 7;
[0070] The convolutional layer L3 and the convolutional layer L4 are processed using the standard depth-separable convolutional layer method in Definition 8 to construct the first convolutional layer, denoted as Layer_tail;
[0071] The constructed first convolution layer Layer_head, cross attention layer Layer_attention, residual convolution layer Layer_residual and tail convolution layer Layer_tail are connected in sequence to build a high-frequency information recovery network, denoted as HF_Net.
[0072] Step 5: Build a diffusion model
[0073] Step 5.1: Build a prediction network
[0074] Step 5.1.1: Construct a downsampler
[0075] Use the standard convolution operation method in Definition 7 to construct a convolution layer, denoted as layer1;
[0076] The standard residual module method in Definition 9 is used to construct two residual block layers, denoted as layer2 and layer3 respectively;
[0077] The convolutional layer layer1, the residual block layer layer2 and layer3 are connected in sequence to obtain a mapping block, which is recorded as block_map1.
[0078] Use the standard convolution operation method in Definition 7 to construct a downsampling layer, denoted as layer_down1;
[0079] The standard residual module method in Definition 9 is used to construct two residual block layers, which are denoted as layer4 and layer5 respectively;
[0080] The downsampling layer layer_down1, the residual block layers layer4 and layer5 are connected in sequence to obtain a downsampling block, denoted as block_down1.
[0081] Use the standard convolution operation method in Definition 7 to construct a downsampling layer, denoted as layer_down2;
[0082] The standard residual module method in Definition 9 is used to construct two residual block layers, which are denoted as layer6 and layer7 respectively;
[0083] The downsampling layer layer_down2, the residual block layers layer6 and layer7 are connected in sequence to obtain a downsampling block, denoted as block_down2.
[0084] Use the standard convolution operation method in Definition 7 to construct a downsampling layer, denoted as layer_down3;
[0085] The standard residual module method in Definition 9 is used to construct two residual block layers, denoted as layer8 and layer9 respectively;
[0086] The standard cross-attention module method in Definition 10 is used to construct two cross-attention block layers, denoted as layer10 and layer11 respectively;
[0087] The downsampling layer layer_down3, the residual block layers layer8 and layer9, and the cross attention block layers layer10 and layer11 are connected in sequence to construct a downsampling block, denoted as block_down3.
[0088] Construct a downsampling layer using the standard convolution operation method in Definition 7, denoted as layer_down4;
[0089] The standard residual module method in Definition 8 is used to construct two residual block layers, which are denoted as layer12 and layer13 respectively;
[0090] The standard cross-attention module method in Definition 10 is used to construct two cross-attention block layers, denoted as layer14 and layer15 respectively.
[0091] The downsampling layer layer_down4, the residual block layers layer12 and layer13, the two cross attention block layers layer14 and layer15 are connected in sequence to construct a downsampling block, denoted as block_down4.
[0092] The mapping block block_map1, downsampling block block_down1, downsampling block block_down2, downsampling block block_down3 and downsampling block block_down4 constructed above are connected in sequence to construct a downsampler, which is recorded as DownSampling.
[0093] Step 5.1.2: Construct the intermediate connection layer
[0094] Use the standard residual module method in Definition 9 to construct a residual block layer, denoted as layer_res1;
[0095] Use the standard criss-cross attention module method in Definition 10 to construct a criss-cross attention block layer, denoted as layer_cra;
[0096] Use the standard residual module method in Definition 9 to construct a cross attention block layer, denoted as layer_cra;
[0097] The residual block layer layer_res1, the cross attention block layer layer_cra, and the cross attention block layer layer_cra are connected in sequence to construct an intermediate connection layer, denoted as Middle.
[0098] Step 5.1.3: Construct upsampler
[0099] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer16, layer17 and layer18 respectively;
[0100] The standard cross-attention module method in Definition 10 is used to construct three cross-attention block layers, which are denoted as layer19, layer20 and layer21 respectively;
[0101] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up1;
[0102] The constructed residual block layers layer16, layer17 and layer18, the cross attention block layers layer19, layer20 and layer21, and the upsampling layer layer_up1 are connected in sequence to construct an upsampling block, denoted as block_up1.
[0103] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer22, layer23 and layer24 respectively;
[0104] The standard cross-attention module method in Definition 10 is used to construct three cross-attention block layers, which are denoted as layer25, layer26 and layer27 respectively;
[0105] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up2;
[0106] The constructed residual block layers layer22, layer23 and layer24, the cross attention block layers layer25, layer26 and layer27, and the upsampling layer layer_up2 are connected in sequence to construct an upsampling block, denoted as block_up2.
[0107] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer28, layer29 and layer30 respectively;
[0108] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up3;
[0109] The above-constructed residual block layers layer28, layer29 and layer30 and the upsampling layer layer_up3 are connected in sequence to construct an upsampling block, which is recorded as block_up3.
[0110] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer31, layer32 and layer33 respectively;
[0111] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up4;
[0112] The above-constructed residual block layers layer31, layer32 and layer33 and the upsampling layer layer_up4 are connected in sequence to construct an upsampling block, which is recorded as block_up4.
[0113] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer34, layer35 and layer36 respectively;
[0114] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer37;
[0115] The above-constructed residual block layers layer34, layer35 and layer36 and the upsampling layer layer37 are connected in sequence to construct a mapping block, which is recorded as block_map2.
[0116] The upsampling block block_up4, upsampling block block_up3, upsampling block block_up2, upsampling block block_up1 and mapping block block_map2 constructed above are connected in sequence to construct an upsampler, which is recorded as UpSampling.
[0117] Using the jump connection method of definition 19 standard, the mapping block block_map1 and the mapping block block_map2 are connected;
[0118] The upsampling block block_up1 is connected to the downsampling block block_down1 by using the skip connection method of definition 19; the upsampling block block_up2 is connected to the downsampling block block_down2 by using the skip connection method of definition 19;
[0119] The upsampling block block_up3 and the downsampling block block_down3 are connected by using the skip connection method of definition 19 standard; the upsampling block block_up4 and the downsampling block block_down4 are connected by using the skip connection method of definition 19 standard;
[0120] The residual block layer layer_res1 and the residual block layer layer_res2 are connected using the standard skip connection method defined in 19. Finally, the downsampler DownSampling, the middle layer Middle and the upsampler UpSampling are connected in sequence to construct a prediction network, which is recorded as Pre_Net.
[0121] Step 5.2: Construct the forward diffusion process
[0122] The standard Markov chain criterion of Definition 17 is used to construct a T-step forward diffusion process, denoted as q, as follows:
[0123]
[0124] Among them, y t ,t∈(0,T) represents the image at the tth step, y0 represents the original image, I represents the unit matrix, α i , i∈(0,T) represents the weight coefficient of the noise added in the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, ε t ,t∈(0,T) represents the prediction result of the prediction network Pre_Net in step 5.1 for the t-th step forward diffusion process.
[0125] Step 5.3: Construct the reverse diffusion process
[0126] The standard Markov chain criterion in Definition 16 is used to construct a T-step reverse diffusion process, denoted as p, as follows:
[0127]
[0128] Among them, y t ,t∈(0,T) represents the image at the tth step, y t-1 represents the image at step t-1, y0 represents the original image, I represents the unit matrix, α i , i∈(0,T) represents the weight coefficient of the noise added in the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, ε t ,t∈(0,T) represents the prediction result of the prediction network Pre_Net in step 5.1 for the t-th step forward diffusion process.
[0129] Step 5.4: Construct a diffusion model
[0130] First, the noise addition process is performed. According to the forward diffusion process q constructed in step 5.2, according to q(y t |y0) adds noise to the original image y0 to obtain the image y0 of step t, and uses the prediction network pre_Net built in step 5.1 to perform the forward diffusion process q(yt |y0) to predict the noise ε t .
[0131] Then, the denoising process is performed, according to the reverse diffusion process p constructed in step 5.3, using the noise ε predicted by the prediction network pre_Net constructed in step 5.1 t , according to p(y t-1 |y t ,y0) for an image y of any time step t t Denoising is performed to obtain the image y of the previous step t-1 .
[0132] The diffusion model is obtained by the above forward denoising process and reverse denoising process, which is denoted as DM.
[0133] Step 6: Build a low-light lens-free reconstruction enhancement model
[0134] Take the training set Train obtained in step 1 as input, and use the coarse imaging module and wavelet transform module obtained in step 2 and step 3 to preprocess the input;
[0135] According to the standard stitching method in Definition 20, the input low-light image and normal-light image are stitched in the channel dimension. Then, the classic Adam algorithm in Definition 11 is used to train the network obtained in steps 4 and 5. After the training is completed, a low-light lens-free reconstruction enhancement model is obtained, which is recorded as DMLE.
[0136] Step 7: Test the low-light lens-free reconstruction enhancement model
[0137] On the low-light lensless reconstruction enhancement model DMLE obtained in step 6, the test set Test obtained in step 1 is tested using the forward propagation method of definition 12 standard to obtain the test result of the test set Test on the low-light lensless reconstruction enhancement model DMLE, which is recorded as Result.
[0138] Step 8. Evaluate the low-light lens-free reconstruction enhancement model
[0139] The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the classic mean square error calculation method in Definition 13 is used to calculate the mean square error index of the reconstructed enhanced image, which is denoted as MSE.
[0140] The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the classic learning-perceptual image block similarity calculation method in Definition 14 is used to calculate the learning-perceptual image block similarity index of the reconstructed enhanced image, which is denoted as LPIPS.
[0141] The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the peak signal-to-noise ratio index of the reconstructed enhanced image is calculated using the classic peak signal-to-noise ratio calculation method in Definition 15, which is denoted as PSNR.
[0142] The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the classic structural similarity calculation method in Definition 16 is used to calculate the structural similarity index of the reconstructed enhanced image, which is denoted as SSIM.
[0143] This completes the entire method.
[0144] The innovation and advantage of the present invention are as follows: the proposed low-light lensless imaging enhancement method based on the diffusion model is completed in two stages. The first stage can quickly complete the coarse imaging based on the low-light lensless measurement results to obtain the preliminary low-light and noisy imaging results; the second stage can improve the image brightness and quality on the basis of the coarse imaging, and finally generate a high-quality normal light image. The coarse imaging results are transformed into the wavelet domain, which reduces the time and cost of network training. In the training stage, the method not only completes the forward diffusion process, but also performs the reverse diffusion process, which ensures the stability of the generation results of the diffusion model. BRIEF DESCRIPTION OF THE DRAWINGS
[0145] Figure 1 It is a flow chart of the implementation steps of the present invention.
[0146] Figure 2 It is the mean square error, learning-perceived image block similarity, peak signal-to-noise ratio and structural similarity of the low-light lensless imaging enhancement based on the diffusion model in the present invention. DETAILED DESCRIPTION
[0147] Next, combine with Figure 1 The present invention is described in further detail.
[0148] The present invention provides a low-light lensless imaging enhancement method based on a diffusion model, the method comprising the following steps:
[0149] Step 1: Prepare the dataset
[0150] Based on the classic LOLv2 dataset in Definition 1, the classic image cropping and scale transformation method in Definition 2 is used to process 1000 pairs of low-light and normal-light images in the Synthetic sub-dataset of the LOLv2 dataset;
[0151] Then, the standard phase mask method in Definition 3 is used to obtain the image after phase mask processing, which is recorded as PSF;
[0152] The processed low-light image is simulated and encoded using the standard lens-free imaging method in Definition 4 to obtain a low-light lens-free image;
[0153] Finally, the processed data set is divided into two parts, the first 900 pairs of images are used as the training set, denoted as Train, and the last 100 pairs of images are used as the test set, denoted as Test.
[0154] Step 2: Build a coarse imaging module
[0155] like Figure 1 As shown, the classical minimum mean square error filter in Definition 5 is used to construct the coarse imaging model, as shown below:
[0156]
[0157] in, is the Fourier transform, is the inverse Fourier transform, H is the phase mask PSF in step 1, b is the input low-light lens-free image, λ is the regularization coefficient, This is the preliminary imaging result.
[0158] Step 3: Construct wavelet transform module
[0159] like Figure 1 As shown, the wavelet transform decomposition module is constructed according to the classic two-dimensional Harr wavelet transform in Definition 6, as shown below:
[0160] {A,V,H,D}=2D-DWT(X)
[0161] Among them, X represents the input image, 2D-DWT represents two-dimensional wavelet transform, A represents the low-frequency information of the input image, V represents the high-frequency information of the input image in the vertical direction, H represents the high-frequency information of the input image in the horizontal direction, and D represents the high-frequency information of the input image in the diagonal direction.
[0162] According to the classic two-dimensional Harr wavelet transform in Definition 6, the wavelet transform reconstruction module is constructed as follows:
[0163]
[0164] in, represents the reconstructed image, 2D-IDWT represents the two-dimensional inverse wavelet transform, represents the low-frequency information of the reconstructed image, Represents the high-frequency information of the reconstructed image in the vertical direction, Represents the high-frequency information of the reconstructed image in the horizontal direction, Represents the high-frequency information of the reconstructed image in the diagonal direction.
[0165] Step 4: Build a high-frequency information recovery network
[0166] like Figure 1 As shown, the convolution layer L1 and the convolution layer L2 are constructed using the standard convolution operation method in Definition 7, and then L1 and L2 are used to construct the first convolution layer according to the standard depth-separable convolution layer in Definition 8, which is recorded as Layer_head.
[0167] Construct a criss-cross attention layer using the standard criss-cross attention module definition in Definition 9, denoted as Layer_attention.
[0168] The residual layer is constructed using the standard residual module definition in Definition 8, denoted as Layer_residual.
[0169] Convolutional layer L3 and convolutional layer L4 are constructed using the standard convolution operation method in Definition 7, and then L3 and L4 are used to construct a tail convolutional layer according to the standard depth-separable convolutional layer in Definition 8, denoted as Layer_tail.
[0170] The constructed first convolution layer Layer_head, cross attention layer Layer_attention, residual convolution layer Layer_residual and tail convolution layer Layer_tail are connected in sequence to build a high-frequency information recovery network, denoted as HF_Net.
[0171] Step 5: Build a diffusion model
[0172] Step 5.1: Build a prediction network
[0173] Step 5.1.1: Construct a downsampler
[0174] Use the standard convolution operation method in Definition 7 to construct a convolution layer, denoted as layer1;
[0175] The standard residual module method in Definition 9 is used to construct two residual block layers, denoted as layer2 and layer3 respectively;
[0176] The convolutional layer layer1, the residual block layer layer2 and layer3 are connected in sequence to obtain a mapping block, which is recorded as block_map1.
[0177] Use the standard convolution operation method in Definition 7 to construct a downsampling layer, denoted as layer_down1;
[0178] The standard residual module method in Definition 9 is used to construct two residual block layers, which are denoted as layer4 and layer5 respectively;
[0179] The downsampling layer layer_down1, the residual block layers layer4 and layer5 are connected in sequence to obtain a downsampling block, denoted as block_down1.
[0180] Use the standard convolution operation method in Definition 7 to construct a downsampling layer, denoted as layer_down2;
[0181] The standard residual module method in Definition 9 is used to construct two residual block layers, which are denoted as layer6 and layer7 respectively;
[0182] The downsampling layer layer_down2, the residual block layers layer6 and layer7 are connected in sequence to obtain a downsampling block, denoted as block_down2.
[0183] Use the standard convolution operation method in Definition 7 to construct a downsampling layer, denoted as layer_down3;
[0184] The standard residual module method in Definition 9 is used to construct two residual block layers, denoted as layer8 and layer9 respectively;
[0185] The standard cross-attention module method in Definition 10 is used to construct two cross-attention block layers, denoted as layer10 and layer11 respectively;
[0186] The downsampling layer layer_down3, the residual block layers layer8 and layer9, and the cross attention block layers layer10 and layer11 are connected in sequence to construct a downsampling block, denoted as block_down3.
[0187] Construct a downsampling layer using the standard convolution operation method in Definition 7, denoted as layer_down4;
[0188] The standard residual module method in Definition 8 is used to construct two residual block layers, which are denoted as layer12 and layer13 respectively;
[0189] The standard cross-attention module method in Definition 10 is used to construct two cross-attention block layers, denoted as layer14 and layer15 respectively.
[0190] The downsampling layer layer_down4, the residual block layers layer12 and layer13, the two cross attention block layers layer14 and layer15 are connected in sequence to construct a downsampling block, denoted as block_down4.
[0191] The mapping block block_map1, downsampling block block_down1, downsampling block block_down2, downsampling block block_down3 and downsampling block block_down4 constructed above are connected in sequence to construct a downsampler, which is recorded as DownSampling.
[0192] Step 5.1.2: Construct the intermediate connection layer
[0193] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer16, layer17 and layer18 respectively;
[0194] The standard cross-attention module method in Definition 10 is used to construct three cross-attention block layers, which are denoted as layer19, layer20 and layer21 respectively;
[0195] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up1;
[0196] The constructed residual block layers layer16, layer17 and layer18, the cross attention block layers layer19, layer20 and layer21, and the upsampling layer layer_up1 are connected in sequence to construct an upsampling block, denoted as block_up1.
[0197] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer22, layer23 and layer24 respectively;
[0198] The standard cross-attention module method in Definition 10 is used to construct three cross-attention block layers, which are denoted as layer25, layer26 and layer27 respectively;
[0199] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up2;
[0200] The constructed residual block layers layer22, layer23 and layer24, the cross attention block layers layer25, layer26 and layer27, and the upsampling layer layer_up2 are connected in sequence to construct an upsampling block, denoted as block_up2.
[0201] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer28, layer29 and layer30 respectively;
[0202] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up3;
[0203] The above-constructed residual block layers layer28, layer29 and layer30 and the upsampling layer layer_up3 are connected in sequence to construct an upsampling block, which is recorded as block_up3.
[0204] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer31, layer32 and layer33 respectively;
[0205] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up4;
[0206] The above-constructed residual block layers layer31, layer32 and layer33 and the upsampling layer layer_up4 are connected in sequence to construct an upsampling block, which is recorded as block_up4.
[0207] The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer34, layer35 and layer36 respectively;
[0208] Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer37;
[0209] The above-constructed residual block layers layer34, layer35 and layer36 and the upsampling layer layer37 are connected in sequence to construct a mapping block, which is recorded as block_map2.
[0210] The upsampling block block_up4, upsampling block block_up3, upsampling block block_up2, upsampling block block_up1 and mapping block block_map2 constructed above are connected in sequence to construct an upsampler, which is recorded as UpSampling.
[0211] Using the jump connection method of definition 19 standard, the mapping block block_map1 and the mapping block block_map2 are connected;
[0212] The upsampling block block_up1 is connected to the downsampling block block_down1 by using the skip connection method of definition 19; the upsampling block block_up2 is connected to the downsampling block block_down2 by using the skip connection method of definition 19;
[0213] The upsampling block block_up3 and the downsampling block block_down3 are connected by using the skip connection method of definition 19 standard; the upsampling block block_up4 and the downsampling block block_down4 are connected by using the skip connection method of definition 19 standard;
[0214] The residual block layer layer_res1 and the residual block layer layer_res2 are connected using the standard skip connection method defined in 19. Finally, the downsampler DownSampling, the middle layer Middle and the upsampler UpSampling are connected in sequence to construct a prediction network, which is recorded as Pre_Net.
[0215] Step 5.2: Construct the forward diffusion process
[0216] Using the standard Markov chain criterion in Definition 17, a forward diffusion process of T = 1000 steps is constructed, denoted as q, as follows:
[0217]
[0218] Among them, y t ,t∈(0,T) represents the image at the tth step, y0 represents the original image, I represents the unit matrix, α i , i∈(0,T) represents the weight coefficient of the noise added in the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, ε t ,t∈(0,T) represents the prediction result of the prediction network Pre_Net in step 5.1 for the t-th step forward diffusion process.
[0219] Step 5.3: Construct the reverse diffusion process
[0220] Using the standard Markov chain criterion in Definition 17, a reverse diffusion process of T = 1000 steps is constructed, denoted as p, as follows:
[0221]
[0222] Among them, y t ,t∈(0,T) represents the image at the tth step, y t-1 represents the image at step t-1, y0 represents the original image, I represents the unit matrix, α i , i∈(0,T) represents the weight coefficient of the noise added in the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, ε t ,t∈(0,T) represents the prediction result of the prediction network Pre_Net in step 5.1 for the t-th step forward diffusion process.
[0223] Step 5.4: Construct a diffusion model
[0224] First, the noise addition process is performed. According to the forward diffusion process q constructed in step 5.2, according to q(y t |y0) adds noise to the original image y0 to obtain the image y0 of step t, and uses the prediction network pre_Net built in step 5.1 to perform the forward diffusion process q(y t |y0) to predict the noise ε t .
[0225] Then, the denoising process is performed, according to the reverse diffusion process p constructed in step 5.3, using the noise ε predicted by the prediction network pre_Net constructed in step 5.1 t , according to p(y t-1 |y t ,y0) for an image y with any time step t t Denoising is performed to obtain the image y of the previous step t-1 .
[0226] The diffusion model is obtained by the above forward denoising process and reverse denoising process, which is denoted as DM. Figure 1 shown.
[0227] Step 6: Build a low-light lens-free reconstruction enhancement model
[0228] Take the training set Train obtained in step 1 as input, and use the coarse imaging module and wavelet transform module obtained in step 2 and step 3 to preprocess the input;
[0229] According to the standard stitching method in Definition 20, the input low-light image and normal-light image are stitched in the channel dimension. Then, the classic Adam algorithm in Definition 11 is used to train the network obtained in steps 4 and 5. After the training is completed, a low-light lens-free reconstruction enhancement model is obtained, which is recorded as DMLE.
[0230] Step 7: Test the low-light lens-free reconstruction enhancement model
[0231] On the low-light lensless reconstruction enhancement model DMLE obtained in step 6, the test set Test obtained in step 1 is tested using the forward propagation method of definition 12 standard to obtain the test result of the test set Test on the low-light lensless reconstruction enhancement model DMLE, which is recorded as Result.
[0232] Step 8. Evaluate the low-light lens-free reconstruction enhancement model
[0233] The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the classic mean square error calculation method in Definition 13 is used to calculate the mean square error index of the reconstructed enhanced image, which is denoted as MSE.
[0234] The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the classic learning-perceptual image block similarity calculation method in Definition 14 is used to calculate the learning-perceptual image block similarity index of the reconstructed enhanced image, which is denoted as LPIPS.
[0235] The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the peak signal-to-noise ratio index of the reconstructed enhanced image is calculated using the classic peak signal-to-noise ratio calculation method in Definition 15, which is denoted as PSNR.
[0236] The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the classic structural similarity calculation method in Definition 16 is used to calculate the structural similarity index of the reconstructed enhanced image, which is denoted as SSIM.
[0237] This completes the entire method.
[0238] Computer simulation results show that compared with the classic Flatnet method, the MSE and SSIM indicators of the reconstructed image obtained by the method of the present invention are better, indicating that the reconstructed image is closer to the true value. At the same time, the LPIPS indicator is smaller and the PSNR is larger, indicating that the quality of the reconstructed image is higher and the color information and detail information are better. Figure 2 shown.
Claims
1. A low-light lensless imaging enhancement method based on a diffusion model, characterized in that it The following steps are involved: Step 1: Prepare the dataset Based on the classic LOLv2 dataset in Definition 1, the classic image cropping and scale transformation method in Definition 2 is used to process 1000 pairs of low-light and normal-light images in the Synthetic sub-dataset of the LOLv2 dataset; Then, the standard phase mask method in Definition 3 is used to obtain the image after phase mask processing, which is recorded as PSF; The processed low-light image is simulated and encoded using the standard lens-free imaging method in Definition 4 to obtain a low-light lens-free image; Finally, the processed data set is divided into two parts, the first 900 pairs of images are used as the training set, denoted as Train, and the last 100 pairs of images are used as the test set, denoted as Test. Step 2: Build a coarse imaging module The classic minimum mean square error filter in Definition 5 is used to construct the coarse imaging model as follows: in, is the Fourier transform, is the inverse Fourier transform, H is the phase mask PSF in step 1, b is the input low-light lens-free image, λ is the regularization coefficient, This is the preliminary imaging result. Step 3: Construct wavelet transform module Using the classic two-dimensional Harr wavelet transform in Definition 6, the wavelet transform decomposition module is constructed as follows: {A,V,H,D}=2D-DWT(X) Among them, X represents the input image, 2D-DWT represents two-dimensional wavelet transform, A represents the low-frequency information of the input image, V represents the high-frequency information of the input image in the vertical direction, H represents the high-frequency information of the input image in the horizontal direction, and D represents the high-frequency information of the input image in the diagonal direction. Using the classic two-dimensional Harr wavelet transform in Definition 6, a wavelet transform reconstruction module is constructed as follows: in, represents the reconstructed image, 2D-IDWT represents the two-dimensional inverse wavelet transform, represents the low-frequency information of the reconstructed image, Represents the high-frequency information of the reconstructed image in the vertical direction, Represents the high-frequency information of the reconstructed image in the horizontal direction, Represents the high-frequency information of the reconstructed image in the diagonal direction. Step 4: Build a high-frequency information recovery network Convolutional layer L1 and convolutional layer L2 are constructed using the standard convolution operation method in Definition 7; The convolutional layers L1 and L2 are processed using the standard depth-wise separable convolutional layer method in Definition 8 to construct the first convolutional layer, denoted as Layer_head. The standard cross-attention module method in Definition 10 is used to construct a cross-attention layer, denoted as Layer_attention. The standard residual module method in Definition 9 is used to construct the residual layer, denoted as Layer_residual. Convolutional layer L3 and convolutional layer L4 are constructed using the standard convolution operation method in Definition 7; The convolutional layer L3 and the convolutional layer L4 are processed using the standard depth-separable convolutional layer method in Definition 8 to construct the first convolutional layer, denoted as Layer_tail; The constructed first convolution layer Layer_head, cross attention layer Layer_attention, residual convolution layer Layer_residual and tail convolution layer Layer_tail are connected in sequence to build a high-frequency information recovery network, denoted as HF_Net. Step 5: Build a diffusion model Step 5.1: Build a prediction network Step 5.1.1: Construct a downsampler Use the standard convolution operation method in Definition 7 to construct a convolution layer, denoted as layer1; The standard residual module method in Definition 9 is used to construct two residual block layers, denoted as layer2 and layer3 respectively; The convolutional layer layer1, the residual block layer layer2 and layer3 are connected in sequence to obtain a mapping block, which is recorded as block_map1. Use the standard convolution operation method in Definition 7 to construct a downsampling layer, denoted as layer_down1; The standard residual module method in Definition 9 is used to construct two residual block layers, which are denoted as layer4 and layer5 respectively; The downsampling layer layer_down1, the residual block layers layer4 and layer5 are connected in sequence to obtain a downsampling block, denoted as block_down1. Use the standard convolution operation method in Definition 7 to construct a downsampling layer, denoted as layer_down2; The standard residual module method in Definition 9 is used to construct two residual block layers, which are denoted as layer6 and layer7 respectively; The downsampling layer layer_down2, the residual block layers layer6 and layer7 are connected in sequence to obtain a downsampling block, denoted as block_down2. Use the standard convolution operation method in Definition 7 to construct a downsampling layer, denoted as layer_down3; The standard residual module method in Definition 9 is used to construct two residual block layers, denoted as layer8 and layer9 respectively; The standard cross-attention module method in Definition 10 is used to construct two cross-attention block layers, denoted as layer10 and layer11 respectively; The downsampling layer layer_down3, the residual block layers layer8 and layer9, and the cross attention block layers layer10 and layer11 are connected in sequence to construct a downsampling block, denoted as block_down3. Construct a downsampling layer using the standard convolution operation method in Definition 7, denoted as layer_down4; The standard residual module method in Definition 8 is used to construct two residual block layers, which are denoted as layer12 and layer13 respectively; The standard cross-attention module method in Definition 10 is used to construct two cross-attention block layers, denoted as layer14 and layer15 respectively. The downsampling layer layer_down4, the residual block layers layer12 and layer13, the two cross attention block layers layer14 and layer15 are connected in sequence to construct a downsampling block, denoted as block_down4. The mapping block block_map1, downsampling block block_down1, downsampling block block_down2, downsampling block block_down3 and downsampling block block_down4 constructed above are connected in sequence to construct a downsampler, which is recorded as DownSampling. Step 5.1.2: Construct the intermediate connection layer Use the standard residual module method in Definition 9 to construct a residual block layer, denoted as layer_res1; Use the standard criss-cross attention module method in Definition 10 to construct a criss-cross attention block layer, denoted as layer_cra; Use the standard residual module method in Definition 9 to construct a cross attention block layer, denoted as layer_cra; The residual block layer layer_res1, the cross attention block layer layer_cra, and the cross attention block layer layer_cra are connected in sequence to construct an intermediate connection layer, denoted as Middle. Step 5.1.3: Construct upsampler The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer16, layer17 and layer18 respectively; The standard cross-attention module method in Definition 10 is used to construct three cross-attention block layers, which are denoted as layer19, layer20 and layer21 respectively; Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up1; The constructed residual block layers layer16, layer17 and layer18, the cross attention block layers layer19, layer20 and layer21, and the upsampling layer layer_up1 are connected in sequence to construct an upsampling block, denoted as block_up1. The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer22, layer23 and layer24 respectively; The standard cross-attention module method in Definition 10 is used to construct three cross-attention block layers, which are denoted as layer25, layer26 and layer27 respectively; Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up2; The constructed residual block layers layer22, layer23 and layer24, the cross attention block layers layer25, layer26 and layer27, and the upsampling layer layer_up2 are connected in sequence to construct an upsampling block, denoted as block_up2. The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer28, layer29 and layer30 respectively; Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up3; The above-constructed residual block layers layer28, layer29 and layer30 and the upsampling layer layer_up3 are connected in sequence to construct an upsampling block, which is recorded as block_up3. The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer31, layer32 and layer33 respectively; Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer_up4; The above-constructed residual block layers layer31, layer32 and layer33 and the upsampling layer layer_up4 are connected in sequence to construct an upsampling block, which is recorded as block_up4. The standard residual module method in Definition 9 is used to construct three residual block layers, which are denoted as layer34, layer35 and layer36 respectively; Use the standard convolution operation method in Definition 7 to construct an upsampling layer, denoted as layer37; The above-constructed residual block layers layer34, layer35 and layer36 and the upsampling layer layer37 are connected in sequence to construct a mapping block, which is recorded as block_map2. The upsampling block block_up4, upsampling block block_up3, upsampling block block_up2, upsampling block block_up1 and mapping block block_map2 constructed above are connected in sequence to construct an upsampler, which is recorded as UpSampling. Using the jump connection method of definition 19 standard, the mapping block block_map1 and the mapping block block_map2 are connected; The upsampling block block_up1 is connected to the downsampling block block_down1 by using the skip connection method of definition 19; the upsampling block block_up2 is connected to the downsampling block block_down2 by using the skip connection method of definition 19; The upsampling block block_up3 and the downsampling block block_down3 are connected by using the skip connection method of definition 19 standard; the upsampling block block_up4 and the downsampling block block_down4 are connected by using the skip connection method of definition 19 standard; The residual block layer layer_res1 and the residual block layer layer_res2 are connected using the standard skip connection method defined in 19. Finally, the downsampler DownSampling, the middle layer Middle and the upsampler UpSampling are connected in sequence to construct a prediction network, which is recorded as Pre_Net. Step 5.2: Construct the forward diffusion process The standard Markov chain criterion of Definition 17 is used to construct a T-step forward diffusion process, denoted as q, as follows: Among them, y t ,t∈(0,T) represents the image at the tth step, y0 represents the original image, I represents the unit matrix, α i , i∈(0,T) represents the weight coefficient of the noise added in the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, ε t ,t∈(0,T) represents the prediction result of the prediction network Pre_Net in step 5.1 for the t-th step forward diffusion process. Step 5.3: Construct the reverse diffusion process The standard Markov chain criterion in Definition 16 is used to construct a T-step reverse diffusion process, denoted as p, as follows: Among them, y t ,t∈(0,T) represents the image at the tth step, y t-1 represents the image at step t-1, y0 represents the original image, I represents the unit matrix, α i , i∈(0,T) represents the weight coefficient of the noise added in the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, ε t ,t∈(0,T) represents the prediction result of the prediction network Pre_Net in step 5.1 for the t-th step forward diffusion process. Step 5.4: Construct a diffusion model First, the noise addition process is performed. According to the forward diffusion process q constructed in step 5.2, according to q(y t |y0) adds noise to the original image y0 to obtain the image y0 of step t, and uses the prediction network pre_Net built in step 5.1 to perform the forward diffusion process q(y t |y0) to predict the noise ε t . Then, the denoising process is performed, according to the reverse diffusion process p constructed in step 5.3, using the noise ε predicted by the prediction network pre_Net constructed in step 5.1 t , according to p(y t-1 |y t ,y0) for an image y of any time step t t Denoising is performed to obtain the image y of the previous step t-1 . The diffusion model is obtained by the above forward denoising process and reverse denoising process, which is denoted as DM. Step 6: Build a low-light lens-free reconstruction enhancement model Take the training set Train obtained in step 1 as input, and use the coarse imaging module and wavelet transform module obtained in step 2 and step 3 to preprocess the input; According to the standard stitching method in Definition 20, the input low-light image and normal-light image are stitched in the channel dimension. Then, the classic Adam algorithm in Definition 11 is used to train the network obtained in steps 4 and 5. After the training is completed, a low-light lens-free reconstruction enhancement model is obtained, which is recorded as DMLE. Step 7: Test the low-light lens-free reconstruction enhancement model On the low-light lensless reconstruction enhancement model DMLE obtained in step 6, the test set Test obtained in step 1 is tested using the forward propagation method of definition 12 standard to obtain the test result of the test set Test on the low-light lensless reconstruction enhancement model DMLE, which is recorded as Result. Step 8. Evaluate the low-light lens-free reconstruction enhancement model The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the classic mean square error calculation method in Definition 13 is used to calculate the mean square error index of the reconstructed enhanced image, which is denoted as MSE. The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the classic learning-perceptual image block similarity calculation method in Definition 14 is used to calculate the learning-perceptual image block similarity index of the reconstructed enhanced image, which is denoted as LPIPS. The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the peak signal-to-noise ratio index of the reconstructed enhanced image is calculated using the classic peak signal-to-noise ratio calculation method in Definition 15, which is denoted as PSNR. The test result Result of the low-light lens-free reconstruction enhancement model DMLE obtained in step 7 is used as input, and the classic structural similarity calculation method in Definition 16 is used to calculate the structural similarity index of the reconstructed enhanced image, which is denoted as SSIM. This completes the entire method.
Citation Information
Cited By
Diffusion road image enhancement method based on wavelet transform and prior
CN120410864A
