Two-stage low-light lens-free reconstruction method based on physical perception

Abstract

The invention discloses a two-stage low-light lensless reconstruction method based on physical perception. According to the method, a reconstruction scene is regarded as coupling of a structural component and a residual component. The method comprises the following steps of: firstly, respectively extracting a structure component and a residual component corresponding to a reconstructed scene from a measured value through a multi-level and multi-scale double-branch extraction module; secondly, an enhancement module based on a diffusion model is utilized to further recover and enrich detail information such as edges and textures of the preliminarily extracted structure components, and noise suppression and brightness optimization are carried out on the extracted residual components; and finally, combining the two enhanced components to obtain a high-quality reconstructed image. The method is excellent in reconstruction quality, and can generate an image with a high-fidelity structure. In the enhancement module training stage based on the diffusion model, not only is the forward diffusion process completed, but also the reverse diffusion process is performed, the problems of color disorder, unstable generation result and the like in the existing method are effectively solved, and the imaging robustness is ensured.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of lensless imaging, and particularly relates to the field of low-light lensless imaging enhancement based on mask modulation. BACKGROUND

[0002] Optical imaging systems are widely used in photography, microscopy, and visual perception, but traditional lens-based systems are difficult to achieve miniaturization, lightweight, and cost optimization due to physical constraints such as diffraction limit and optical stack space. Lensless imaging technology provides a viable alternative: replacing traditional lenses with lightweight masks or specially designed coded apertures to modulate incident light and reconstruct high-quality images by computing the analytical light field information. This paradigm breaks the strict point-to-point mapping constraints of traditional systems, making thinner, lighter, and more compact optical designs possible, and showing significant potential in bio-sensing, three-dimensional imaging, computational photography, and medical visual diagnosis.

[0003] However, compared with traditional lens cameras, lensless cameras often have limited imaging space and low effective light throughput. In low-light environments, readout noise, dark current, and quantization noise increase, making it easy to produce high-noise, low-brightness measurement data, making high-quality reconstruction challenging. Early methods are mostly based on Tikhonov regularization optimization framework, using priori to suppress noise, although they have interpretability, they are sensitive to parameters and easy to smooth and lose details in low signal-to-noise ratio (SNR). With the development of deep learning, end-to-end data-driven methods significantly improve the reconstruction quality, but they rely on a large amount of training data and have limited generalization to hardware and lighting changes. Therefore, researchers propose hybrid strategies such as learning Alternating Direction Method of Multipliers (ADMM) and introduce mismatch compensation mechanisms to alleviate the impact of model errors, but it is still difficult to extract high-fidelity structural information from measurements stably and accurately in low-light and high-noise conditions, and related research is still insufficient.

[0004] To solve the above problems, the application provides a two-stage low-light lensless reconstruction method based on physical perception, which realizes robust and high-fidelity image reconstruction for low-illumination lensless data. First, in the first stage, a multi-level multi-scale double-branch extraction module is used to extract a structural component containing structural information and a residual component other than the structural information from the lensless original measurement value. At the same time, the physical model fusion block injects the physical prior information of the acquisition system in the multi-layer feature, improving the physical consistency and robustness of the extraction module. Then, in the second stage, an enhancement module based on the diffusion model is used to enhance the structural component and the residual component extracted in the first stage, respectively. The former focuses on the recovery and completion of edge and texture details, and the latter focuses on noise suppression and brightness correction, and finally a high-quality reconstruction result is obtained. Experimental results show that while ensuring structural fidelity and detail expression, the method can significantly reduce noise and improve brightness, effectively realizing the reconstruction and enhancement of low-light lensless images. SUMMARY

[0005] The application discloses a two-stage low-light lensless reconstruction method based on physical perception, which regards the reconstructed scene as a coupling of a structural component and a residual component. First, a multi-level multi-scale double-branch extraction module is used to extract the structural component and the residual component corresponding to the reconstructed scene from the measurement value. This double-branch network structure can not only fully exploit the effective information in the lensless measurement value, but also ensure that the extracted structural information is not disturbed by noise and brightness information. At the same time, the physical fusion module injects physical prior information into different feature layers to ensure the physical consistency of the component extraction module. Secondly, an enhancement module based on the diffusion model is used to further recover and enrich the edge and texture details of the preliminary extracted structural component, and to suppress noise and optimize brightness of the extracted residual component. Finally, the two components after enhancement are combined to obtain a high-quality reconstructed image. Experimental results show that the reconstruction quality of the application is excellent, and a high-structural-fidelity image can be generated. In the training stage of the enhancement module based on the diffusion model, the method not only completes the forward diffusion process, but also performs the reverse diffusion process, effectively overcoming the problems of color confusion and unstable generated results in existing methods, and ensuring the robustness of imaging.

[0006] To facilitate the description of the content of the application, the following definitions are first made:

[0007] Definition 1: classic cat and dog game data set

[0008] The classic Dogs vs. Cats dataset is a unified benchmark dataset for image binary classification research and evaluation, commonly used to train and compare the classification performance of various deep learning models. The dataset contains a total of 25,000 images (12,500 cats and 12,500 dogs), mainly used for online evaluation and ranking. The complete dataset can be obtained from "https: / / www.kaggle.com / competitions / dogs-vs-cats / overview".

[0009] Definition 2: Standard image resolution transformation method

[0010] The standard image resolution transformation method refers to the technical means of converting an image from one resolution to another target resolution through image interpolation or resampling, etc. It is widely used in image scaling, super-resolution reconstruction, and visual preprocessing tasks. This method usually uses algorithms such as bilinear interpolation, bicubic interpolation, nearest neighbor interpolation, etc. to recalculate and map pixel values based on the pixel distribution of the original image, in order to maintain the overall structure of the image while preserving edge details and texture information as much as possible. For details, see "https: / / blog.csdn.net / m0_59799878 / article / details / 131571082".

[0011] Definition 3: Standard phase mask method

[0012] Phase mask is a specially designed optical mask, and the main function of the standard phase mask method is to introduce phase modulation in the propagation process of incident light waves through wave optics principles. Phase mask allows most of the light to pass through, providing high signal-to-noise ratio. The standard phase mask method is detailed in the literature "Boominathan V, Adams J K, Robinson JT, et al. Phlatcam: Designed phase-mask based thin lensless camera [J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 42(7): 1618-1629."

[0013] Definition 4: Standard low-light lensless acquisition method

[0014] Standard low-light lensless acquisition method generally refers to the design idea of DiffuserCam system, which encodes the scene by removing the imaging lens and introducing a random scattering mask, and then collects the modulated intensity image by the sensor. The system is generally composed of an image display module (such as an LCD screen), a random mask, a CMOS image sensor, and a data acquisition and transmission module. The image content is projected by the LCD screen, encoded by the mask, and received by the rear sensor, and the output image signal is transmitted to the host computer for storage and subsequent processing. To realize low-light acquisition, the exposure time can be shortened and the ambient light intensity can be reduced to simulate a weak light imaging environment, thereby constructing a low-light data set closer to actual application requirements. For details, see "https: / / waller-lab.github.io / DiffuserCam / tutorial".

[0015] Definition 5: Standard Pixel Shuffle transformation

[0016] Pixel Shuffle is an efficient up-sampling method that rearranges the channel dimension information to the spatial dimension to achieve image resolution improvement. Its core idea is to rearrange the channel and spatial dimensions of the feature map to convert low-resolution feature maps to high-resolution feature maps. For details, see the literature "Shi W, Caballero J, Huszar F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1874-1883."

[0017] Definition 6: Standard convolution operation method

[0018] The convolution layer uses a set of learnable filters (also known as convolution kernels or feature detectors) to filter the input data in space. Each filter slides over the input data (convolution operation), calculates the dot product between the filter and the input data, and generates a feature map. Convolution operation helps to extract local features of input data. For details of standard convolution operation method, see "https: / / cloud.tencent.com / developer / article / 2299603".

[0019] Definition 7: Classical minimum mean square error filtering

[0020] MMSE filtering, as the name implies, is a signal processing method that minimizes the mean square error between the estimated value and the true value by understanding the relationship between the signal and the noise, filters out noise and interference from continuous or discrete input data to extract useful information, and then realizes image reconstruction and restoration. For details, see the literature "Gonzalez R C, Woods R E, Masters B R. Digital Image Processing, Third Edition [J]. Journal of Biomedical Optics, 2009, 14(2): 029901. DOI: 10.1117 / 1.3115362."

[0021] Definition 8: Standard concatenation method

[0022] "Low-quality image" and "high-quality image" are concatenated using the concat function to concatenate two tensors along the specified axis and input to the neural network for training, which is usually used to solve image restoration tasks under specific conditions. This method can effectively improve the network's ability to process images. For details, see the literature "Saharia C, Ho J, Chan W, et al. Image super-resolution via iterative refinement [J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 45(4): 4713-4726."

[0023] Definition 9: Standard Pixel Unshuffle transformation

[0024] Pixel Unshuffle is the inverse operation of Pixel Shuffle, which is used to remap the spatial dimensions back to the channel dimensions, thus achieving feature compression. This operation is very useful in tasks such as encoding-decoding models, image compression, etc. By converting high-resolution feature maps into low-resolution ones, Pixel Unshuffle can reduce computational costs while preserving important feature information. For more details, see the paper "Shi W, Caballero J, Huszár F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 1874-1883."

[0025] Definition 10: Standard Transformer Block method

[0026] The standard Transformer Block method refers to a basic network structure module that is built around the self-attention mechanism (Self-Attention). It alternately stacks multi-head self-attention (Multi-Head Self-Attention, MHSA) and feed-forward fully connected layers (Feed-Forward Network, FFN) to model the global dependencies of input features and perform nonlinear transformations. Specifically, each Block usually includes multi-head self-attention layers, residual connections (Residual Connection), layer normalization (Layer Normalization), and position feed-forward networks, etc. For more details, see the paper "https: / / blog.csdn.net / qq_36803941 / article / details / 138795224"

[0027] Definition 11: Standard skip connection method

[0028] Skip Connection is a technique that connects certain layers in a neural network directly to the previous layers. It is usually added directly after the output of a convolutional layer or other layers. Its purpose is to solve the problem of gradient vanishing or gradient explosion that may occur during the training process of deep neural networks, while also helping to speed up the training process and improve the expressiveness of the model. For details, see "https: / / blog.csdn.net / weixin_44859553 / article / details / 132826791".

[0029] Definition 12: Standard Activation Function Method

[0030] The standard activation function method refers to the basic technique of introducing non-linear transformation in neural networks. By performing non-linear mapping on the input signal after linear weighted summation, the model can fit and express complex function relationships. Common activation functions include Sigmoid, Tanh, ReLU and their variants, which exhibit different advantages and disadvantages in terms of convergence speed, gradient vanishing suppression, sparsity and smoothness. The design and selection of activation functions directly affect the training stability, expressiveness and generalization performance of deep networks, and therefore are considered as a key component in deep learning architecture. For details, see "https: / / blog.csdn.net / weixin_44115575 / article / details / 139835864".

[0031] Definition 13: Standard Dual Branch Network Construction Method

[0032] The standard dual branch network construction method refers to a structured modeling approach in deep neural network design, which divides the input data stream into two relatively independent branch paths, each responsible for different feature learning or task processing functions, and then performs information fusion in the subsequent stage. Typical designs include main branch and auxiliary branch, structural branch and residual branch, or global branch and local branch, etc. The main branch is responsible for extracting global features or semantic information, while the auxiliary branch focuses on fine-grained features, noise suppression or context supplementation. Through complementary modeling and multi-scale feature fusion between branches, this method can improve the robustness and generalization of the model while ensuring the overall expressiveness, and has been widely applied in computer vision tasks such as image recognition, object detection and image enhancement. For details, see "https: / / blog.csdn.net / qq_37497875 / article / details / 121375923".

[0033] Definition 14: Standard Residual Module Method

[0034] Residual Block is the basic building block of deep convolutional neural networks. The standard residual block method solves the gradient vanishing problem in deep neural networks by introducing "skip connections", which allows deeper networks to be effectively trained. The core idea of residual block is to learn the residual function between input and output, rather than directly learning the mapping from input to output. This makes the network easier to optimize. For details of the standard residual block method, see "https: / / blog.csdn.net / SisterRu / article / details / 139853475".

[0035] Definition 15: Standard Cross-Attention Module Method

[0036] Cross-Attention Module is a module used to enhance the relationship between multiple input features in neural networks. The standard cross-attention module method performs attention mechanisms on different channels or spatial dimensions, so that the network can more effectively learn the dependency and importance between features, thereby improving its performance and generalization ability. For details of the standard cross-attention module method, see "https: / / blog.csdn.net / m0_63097763 / article / details / 132293568".

[0037] Definition 16: Standard Markov Chain Criterion

[0038] Markov Chain is a stochastic process with Markov property. In Markov chain, the transition of the current state only depends on the previous state, and has nothing to do with the past state sequence. This property is also called memoryless, that is, the future state only relates to the current state. Markov chain is usually described by a state space and a transition probability matrix. The state space represents the set of possible states, and the transition probability matrix specifies the probability of transitioning from one state to another. For details, see "https: / / blog.csdn.net / ywsydwsbn / article / details / 123609409".

[0039] Definition 17: Classic Adam Algorithm

[0040] The classic Adam algorithm is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It can update all weights based on a single learning rate, which does not change during training. Adam designs independent adaptive learning rates for different parameters by calculating the first and second moment estimates of the gradient. For details, see Kingma, D, Ba, J. Adam: A Method for Stochastic Optimization. arXiv 2014, arXiv:1412.6980.

[0041] Definition 18: Standard forward propagation method

[0042] The forward propagation method is the most basic method in deep learning, which mainly performs forward inference on the input according to the parameters and connection methods in the network to obtain the output of the network. The traditional forward propagation method is detailed in https: / / www.jianshu.com / p / f30c8daebebb.

[0043] Definition 19: Classic mean squared error calculation method

[0044] Mean Squared Error (MSE) is a common metric for measuring the difference between two images. It calculates the average squared error between the pixels of two images, i.e., the average of the squared differences between each pixel value. MSE is commonly used to compare the overall pixel-level differences between images. The lower the value, the more similar the two images are. The classic mean squared error calculation method is detailed in https: / / blog.csdn.net / qq_44111805 / article / details / 127705262.

[0045] Definition 20: Classic learning perceptual image patch similarity calculation method

[0046] Learned Perceptual Image Patch Similarity (LPIPS) is a deep learning model-based image quality evaluation index, aiming to capture human perceptual differences. It not only considers pixel-level differences, but also considers perceptual differences, so it can better predict human subjective perception. LPIPS is commonly used to compare the structural and content similarity of images, especially in generative models, to evaluate the quality of generated images. The classic learning perceptual image patch similarity calculation method is detailed in https: / / blog.csdn.net / u013889591 / article / details / 140687263.

[0047] Definition 21: Classical Peak Signal-to-Noise Ratio Calculation Method

[0048] Peak Signal-to-Noise Ratio (PSNR) is an indicator that measures the noise between two images. It compares the root mean square error between the original image and the image being evaluated, then converts the error into decibel (dB) units. PSNR is commonly used to measure the effectiveness of compression algorithms, with higher PSNR values indicating better image quality. However, PSNR is not always consistent with human visual perception, so it may not be the best choice in some scenarios. The classical peak signal-to-noise ratio calculation method is detailed at "https: / / blog.csdn.net / qq_44111805 / article / details / 127676377".

[0049] Definition 22: Classical Structural Similarity Calculation Method

[0050] Structural Similarity Index (SSIM) is an indicator that measures the similarity between two images, considering not only brightness differences but also contrast and structural information. It simulates the perception of the human visual system for images. SSIM is commonly used in image quality assessment and image processing tasks, especially when comparing the structural similarity of two images. Higher SSIM values indicate that the two images are more similar. The classical structural similarity calculation method is detailed at "https: / / zhuanlan.zhihu.com / p / 399215180".

[0051] A two-stage low-light lensless reconstruction method based on physical perception, the method comprising the following steps:

[0052] Step 1, prepare the dataset

[0053] Based on the classical cat and dog dataset in definition 1, 11000 images are selected and processed using the classical image resolution transformation method in definition 2 to obtain the ground truth image, denoted as GT.

[0054] Using the standard phase mask method in definition 3, the image after phase mask processing is obtained, denoted as PSF.

[0055] According to PSF, the standard low-light lensless acquisition method in definition 4 is used to collect data from the ground truth image GT, obtaining the low-light lensless measurement value, denoted as Measurement.

[0056] Finally, the ground truth image GT and the corresponding acquired low-light lensless measurement Measurement are divided into two parts, the first 10000 pairs of pictures are taken as the training set, denoted as Train, and the last 1000 pairs of pictures are taken as the test set, denoted as Test.

[0057] Step 2, constructing a physical fusion block

[0058] An up-sampling layer is constructed using the Pixel Shuffle transformation in definition 5, denoted as layer1;

[0059] A convolution layer is constructed using the convolution operation method in definition 6, denoted as layer2;

[0060] The up-sampling layer layer1 and the convolution layer layer2 are connected in sequence to obtain a feature-to-image mapping layer, denoted as F2I;

[0061] The classical minimum mean square error filter in definition 7 is used to construct the initial imaging model, as shown below:

[0062]

[0063] wherein, is the Fourier transform,

[0064] is the inverse Fourier transform, H is the phase mask PSF in step 1, y is the input low-light lensless image, λ is the regularization coefficient, is the initial imaging result, (·) * represents the complex conjugate operator, and represents the Hadamard product. The initial imaging result is spliced according to the splicing method in definition 8, and the output result of F2I is spliced to obtain a deconvolution layer, denoted as layer3;

[0065] A convolution layer is constructed using the convolution operation method in definition 6, denoted as layer4;

[0066] An down-sampling layer is constructed using the Pixel Shuffle transformation in definition 9, denoted as layer5;

[0067] The down-sampling layer layer4 and the convolution layer layer5 are connected in sequence to obtain an image-to-feature mapping layer, denoted as I2F;

[0068] The mapping layer F2I, the deconvolution layer layer3 and the mapping layer I2F are connected in sequence to obtain a physical fusion block, denoted as PFB. Step 3, constructing a multi-level multi-scale extraction module

[0069] Step 3.1, constructing a structural component extraction branch

[0070] Construct a Transformer Block using the method of the standard in definition 10, denoted as block1;

[0071] Construct a down-sampling layer using the method of the standard convolution operation in definition 6, denoted as layer_down1;

[0072] Connect block1, the physical fusion block PFB obtained in step 2 and the down-sampling layer layer_down1 in turn to construct a conditional Transformer Block, denoted as CTB1;

[0073] Construct a Transformer Block using the method of the standard in definition 10, denoted as block2;

[0074] Construct a down-sampling layer using the method of the standard convolution operation in definition 6, denoted as layer_down2;

[0075] Connect block2, the physical fusion block PFB obtained in step 2 and the down-sampling layer layer_down2 in turn to construct a conditional Transformer Block, denoted as CTB2;

[0076] Construct a Transformer Block using the method of the standard in definition 10, denoted as block3;

[0077] Construct an up-sampling layer using the method of the standard convolution operation in definition 6, denoted as layer_up1;

[0078] Connect block3, the physical fusion block PFB obtained in step 2 and the up-sampling layer layer_up1 in turn to construct a conditional Transformer Block, denoted as CTB3;

[0079] Construct a Transformer Block using the method of the standard in definition 10, denoted as block4;

[0080] Construct an up-sampling layer using the method of the standard convolution operation in definition 6, denoted as layer_up2;

[0081] Connect block4 and the up-sampling layer layer_up2 in turn to construct a conditional Transformer Block, denoted as CTB4;

[0082] A Transformer Block is constructed using the standard Transformer Block method in Definition 10, denoted as block5;

[0083] The conditional Transformer Block CTB1 and the Transformer Block block5 are connected using the skip connection method in Definition 11;

[0084] The conditional Transformer Block CTB2 and the conditional Transformer Block CTB4 are connected using the skip connection method in Definition 11;

[0085] The conditional Transformer Block CTB1, the conditional Transformer Block CTB2, the conditional Transformer Block CTB3, the conditional Transformer Block CTB4 and the Transformer Block block5 are sequentially connected to build a structural component extraction branch network, denoted as Str_Ext.

[0086] Step 3.2, build the residual component extraction branch

[0087] A convolution layer is constructed using the standard convolution operation method in Definition 6, denoted as layer6;

[0088] An activation layer is constructed using the standard activation function method in Definition 12, denoted as layer7;

[0089] A convolution layer is constructed using the standard convolution operation method in Definition 6, denoted as layer8;

[0090] An activation layer is constructed using the standard activation function method in Definition 12, denoted as layer9;

[0091] A convolution layer is constructed using the standard convolution operation method in Definition 6, denoted as layer10;

[0092] An activation layer is constructed using the standard activation function method in Definition 12, denoted as layer11;

[0093] The convolution layer layer6, the activation layer layer7, the convolution layer layer8, the activation layer layer9, the convolution layer layer10 and the activation layer layer11 are sequentially connected to build a residual component extraction branch network, denoted as Res_Ext.

[0094] The multi-level multi-scale extraction module is constructed by the branch network Str_Ext extracted from the structure component obtained in step 3.1 and the branch network Res_Ext extracted from the residual component obtained in step 3.2, and is denoted as Ext_Net.

[0095] Step 4, constructing an enhancement module

[0096] Step 4.1, constructing a prediction network

[0097] Step 4.1.1, constructing a down-sampler

[0098] A convolution layer is constructed by the convolution operation method in definition 6, and is denoted as layer12;

[0099] Two residual block layers are constructed by the residual module method in definition 14, and are denoted as layer13 and layer14, respectively;

[0100] The convolution layer layer12, the residual block layers layer13 and layer14 are sequentially connected to obtain a mapping block, which is denoted as block_map1.

[0101] A down-sampling layer is constructed by the convolution operation method in definition 6, and is denoted as layer_down3;

[0102] Two residual block layers are constructed by the residual module method in definition 14, and are denoted as layer15 and layer16, respectively;

[0103] The down-sampling layer layer_down3, the residual block layers layer15 and layer16 are sequentially connected to obtain a down-sampling block, which is denoted as block_down1.

[0104] A down-sampling layer is constructed by the convolution operation method in definition 6, and is denoted as layer_down4;

[0105] Two residual block layers are constructed by the residual module method in definition 14, and are denoted as layer17 and layer18, respectively;

[0106] The down-sampling layer layer_down4, the residual block layers layer17 and layer18 are sequentially connected to obtain a down-sampling block, which is denoted as block_down2.

[0107] A down-sampling layer is constructed by the convolution operation method in definition 6, and is denoted as layer_down5;

[0108] Two residual block layers are constructed using the residual module method in definition 14, denoted as layer 19 and layer 20 respectively;

[0109] Two cross-attention block layers are constructed using the cross-attention module method in definition 15, denoted as layer 21 and layer 22 respectively.

[0110] A down-sampling block is constructed by connecting the down-sampling layer layer_down5, the residual block layers layer 19 and layer 20, and the cross-attention block layers layer 21 and layer 22 in sequence, denoted as block_down3.

[0111] A down-sampling layer is constructed using the convolution operation method in definition 6, denoted as layer_down6.

[0112] Two residual block layers are constructed using the residual module method in definition 14, denoted as layer 23 and layer 24 respectively.

[0113] Two cross-attention block layers are constructed using the cross-attention module method in definition 15, denoted as layer 25 and layer 26 respectively.

[0114] A down-sampling block is constructed by connecting the down-sampling layer layer_down6, the residual block layers layer 23 and layer 24, and the cross-attention block layers layer 25 and layer 26 in sequence, denoted as block_down4.

[0115] The mapping block block_map1, the down-sampling blocks block_down1, block_down2, block_down3 and block_down4 are connected in sequence to construct a down-sampler, denoted as DownSampling.

[0116] Step 4.1.2, constructing an intermediate connection layer

[0117] A residual block layer is constructed using the residual module method in definition 14, denoted as layer_res1.

[0118] A cross-attention block layer is constructed using the cross-attention module method in definition 15, denoted as layer_cra.

[0119] A residual block layer is constructed using the residual module method in definition 14, denoted as layer_res2.

[0120] The residual block layer layer_res1, the cross-attention block layer layer_cra, and the residual block layer layer_res2 are sequentially connected to obtain an intermediate connection layer, denoted as Middle.

[0121] Step 4.1.3, constructing an upsampler

[0122] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer27, layer28, and layer29, respectively.

[0123] Three cross-attention block layers are constructed by using the cross-attention module method in definition 15, denoted as layer30, layer31, and layer32, respectively.

[0124] An upsampling layer is constructed by using the convolution operation method in definition 6, denoted as layer_up3.

[0125] The residual block layers layer27, layer28, and layer29, the cross-attention block layers layer30, layer31, and layer32, and the upsampling layer layer_up3 are sequentially connected to obtain an upsampling block, denoted as block_up1.

[0126] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer33, layer34, and layer35, respectively.

[0127] Three cross-attention block layers are constructed by using the cross-attention module method in definition 15, denoted as layer36, layer37, and layer38, respectively.

[0128] An upsampling layer is constructed by using the convolution operation method in definition 6, denoted as layer_up4.

[0129] The residual block layers layer33, layer34, and layer35, the cross-attention block layers layer36, layer37, and layer38, and the upsampling layer layer_up4 are sequentially connected to obtain an upsampling block, denoted as block_up2.

[0130] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer39, layer40, and layer41, respectively.

[0131] An upsampling layer is constructed by using the convolution operation method in definition 6, denoted as layer_up5.

[0132] Connect the residual block layers layer39, layer40 and layer41, the up-sampling layer layer_up5 in sequence to obtain an up-sampling block, denoted as block_up3.

[0133] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer42, layer43 and layer44, respectively.

[0134] An up-sampling layer is constructed by using the convolution operation method in definition 6, denoted as layer_up6.

[0135] Connect the residual block layers layer42, layer43 and layer44, the up-sampling layer layer_up6 in sequence to obtain an up-sampling block, denoted as block_up4.

[0136] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer45, layer46 and layer47, respectively.

[0137] A convolution layer is constructed by using the convolution operation method in definition 6, denoted as layer48.

[0138] Connect the residual block layers layer45, layer46 and layer47, the convolution layer layer48 in sequence to obtain a mapping block, denoted as block_map2.

[0139] Connect the up-sampling block block_up4, the up-sampling block block_up3, the up-sampling block block_up2, the up-sampling block block_up1 and the mapping block block_map2 in sequence to obtain an up-sampler, denoted as UpSampling.

[0140] The mapping block block_map1 and the mapping block block_map2 are connected by using the skip connection method in definition 11.

[0141] The up-sampling block block_up1 and the down-sampling block block_down1 are connected by using the skip connection method in definition 11; the up-sampling block block_up2 and the down-sampling block block_down2 are connected by using the skip connection method in definition 11.

[0142] The up-sampling block block_up3 and the down-sampling block block_down3 are connected by using the skip connection method in definition 11; the up-sampling block block_up4 and the down-sampling block block_down4 are connected by using the skip connection method in definition 11.

[0143] The residual block layer layer_res1 and the residual block layer layer_res2 are connected by using the skip connection method with the definition 11 standard.

[0144] Finally, the down sampler DownSampling, the middle layer Middle, and the up sampler UpSampling are sequentially connected to construct a prediction network, denoted as Pre_Net.

[0145] Step 4.2, constructing a forward diffusion process

[0146] A T-step forward diffusion process q is constructed by using the standard Markov chain criterion in definition 16, as follows:

[0147]

[0148] where y t ,t∈(0,T) represents the image at the t-th step, y0 represents the original image, I represents the unit matrix, α i ,i∈(0,T) represents the weight coefficient of the added noise at the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, represents the standard Gaussian noise, and represents the noise level to be added at the t-th step.

[0149] Step 4.3, constructing a reverse diffusion process

[0150] A T-step reverse diffusion process p is constructed by using the standard Markov chain criterion in definition 16, as follows:

[0151]

[0152] where y t ,t∈(0,T) represents the image at the t-th step, y t-1 represents the image at the t-1-th step, y0 represents the original image, I represents the unit matrix, α i ,i∈(0,T) represents the weight coefficient of the added noise at the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, ε θ,t ,t∈(0,T) represents the prediction result of the prediction network Pre_Net of step 4.1 for the t-th step forward diffusion process.

[0153] Step 4.4, constructing an enhancement module based on the diffusion model

[0154] First, the original image y0 is calculated according to the formula by using the forward diffusion process q constructed in step 4.2, to obtain the noisy image y t at the t-th step.

[0155] Then, the denoising is performed by using the inverse diffusion process p constructed in step 4.3, and the image y at time step t is obtained t According to the formula The calculation is performed to obtain the denoised image y at step t-1 t-1 .

[0156] The enhancement module based on the diffusion model is obtained by the above forward noise adding process and inverse denoising process, denoted as Enh_Net.

[0157] The training set Train obtained in step 1 is used as input, and the multi-level multi-scale extraction module Ext_Net obtained in step 3 is trained by using the classical Adam algorithm in definition 17. Then, the output of the multi-level multi-scale extraction module is spliced with the ground truth image GT in step 1 in the channel dimension by using the standard splicing method in definition 8, and the enhancement module Enh_Net obtained in step 4 is trained by using the classical Adam algorithm in definition 17. After the training is completed, the two-stage low-light lensless reconstruction model is obtained, denoted as TSLR.

[0158] Step 6, test the two-stage low-light lensless reconstruction model

[0159] The test set Test obtained in step 1 is tested on the two-stage low-light lensless reconstruction model TSLR obtained in step 5 by using the standard forward propagation method in definition 18, and the test result of the test set Test on the two-stage low-light lensless reconstruction model TSLR is obtained, denoted as Result.

[0160] Step 7, evaluate the two-stage low-light lensless reconstruction model

[0161] The test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6 is calculated by using the classical mean square error calculation method in definition 19, and the mean square error index of the test result is obtained, denoted as MSE.

[0162] The test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6 is calculated by using the classical learning perceptual image block similarity calculation method in definition 20, and the learning perceptual image block similarity index of the test result is obtained, denoted as LPIPS.

[0163] The test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6 is calculated by using the classical peak signal-to-noise ratio calculation method in definition 21, and the peak signal-to-noise ratio index of the test result is obtained, denoted as PSNR.

[0164] The test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6 is calculated by using the classical structural similarity calculation method in definition 22, and the structural similarity index of the test result is obtained, which is denoted as SSIM.

[0165] Up to now, the whole method ends.

[0166] The innovation and advantages of the present application are that the two-stage low-light lensless reconstruction method based on physical perception regards the reconstructed scene as the coupling of structural components and residual components. The first stage directly extracts the structural components and residual components from the measurement values by using a multi-level multi-scale extraction module, effectively separating the structural information and interference information. The second stage optimizes the two components by using an enhancement module based on a diffusion model, and finally combines to obtain a high-quality reconstruction result. The multi-level multi-scale extraction module fully excavates the structural information in the measurement values, and realizes component decoupling. The present method independently trains the two stages, so that each module can focus on its own target. BRIEF DESCRIPTION OF DRAWINGS

[0167] Figure 1 The implementation step flowchart of the present application.

[0168] Figure 2 The mean square error, learning perception image block similarity, peak signal-to-noise ratio and structural similarity of the two-stage low-light lensless reconstruction based on physical perception in the present application. DETAILED DESCRIPTION

[0169] Next, the accompanying drawings will be described in conjunction with the present application. Figure 1 The present application will be further described in detail.

[0170] The present application provides a two-stage low-light lensless reconstruction method based on physical perception, which comprises the following steps:

[0171] Step 1, preparing a data set

[0172] The classical cat and dog game data set in definition 1 is taken as the basis, and 11000 pictures are selected and processed by using the classical image resolution transformation method in definition 2 to obtain a ground truth image, denoted as GT.

[0173] The standard phase mask method in definition 3 is used for processing to obtain a phase mask processed image, denoted as PSF. According to the standard low-light lensless acquisition method in definition 4, the true value image GT is acquired by using the PSF to obtain a low-light lensless measurement value, denoted as Measurement.

[0174] Finally, the ground truth image GT and the corresponding acquired low-light lensless measurement Measurement are divided into two parts, the first 10000 pairs of pictures are taken as the training set, denoted as Train, and the last 1000 pairs of pictures are taken as the test set, denoted as Test.

[0175] Step 2, constructing a physical fusion block

[0176] As shown in definition 5, a Pixel Shuffle transform is used to construct an up-sampling layer, denoted as layer1. Figure 1 A convolution operation method is used to construct a convolution layer, denoted as layer2, according to definition 6.

[0177] The up-sampling layer layer1 and the convolution layer layer2 are connected in sequence to obtain a feature-to-image mapping layer, denoted as F2I.

[0178] A classical minimum mean square error filter is used to construct a primary imaging model, as shown in definition 7.

[0179]

[0180]

[0181] wherein, is a Fourier transform,

[0182] is an inverse Fourier transform, H is the phase mask PSF in step 1, y is the input low-light lensless image, λ is a regularization coefficient, is the primary imaging result, (·) * represents a complex conjugate operator, and ⊙ represents a Hadamard product. The primary imaging result is spliced with the output result of F2I according to the splicing method in definition 8 to obtain a deconvolution layer, denoted as layer3.

[0183] A convolution operation method is used to construct a convolution layer, denoted as layer4, according to definition 6.

[0184] A Pixel Shuffle transform is used to construct a down-sampling layer, denoted as layer5, according to definition 9.

[0185] The down-sampling layer layer4 and the convolution layer layer5 are connected in sequence to obtain an image-to-feature mapping layer, denoted as I2F.

[0186] The mapping layer F2I, the deconvolution layer layer3, and the mapping layer I2F are connected in sequence to obtain a physical fusion block, denoted as PFB. Step 3, constructing a multi-level multi-scale extraction module

[0187] ​Step 3.1, constructing structure component extraction branch

[0188] A Transformer Block is constructed using the standard Transformer Block method in Definition 10, denoted as block1;

[0189] A down-sampling layer is constructed using the standard convolution operation method in Definition 6, denoted as layer_down1;

[0190] A conditional Transformer Block is constructed by connecting block1, the physical fusion block PFB obtained in step 2, and the down-sampling layer layer_down1 in sequence, denoted as CTB1;

[0191] A Transformer Block is constructed using the standard Transformer Block method in Definition 10, denoted as block2;

[0192] A down-sampling layer is constructed using the standard convolution operation method in Definition 6, denoted as layer_down2;

[0193] A conditional Transformer Block is constructed by connecting block2, the physical fusion block PFB obtained in step 2, and the down-sampling layer layer_down2 in sequence, denoted as CTB2;

[0194] A Transformer Block is constructed using the standard Transformer Block method in Definition 10, denoted as block3;

[0195] An up-sampling layer is constructed using the standard convolution operation method in Definition 6, denoted as layer_up1;

[0196] A conditional Transformer Block is constructed by connecting block3, the physical fusion block PFB obtained in step 2, and the up-sampling layer layer_up1 in sequence, denoted as CTB3;

[0197] A Transformer Block is constructed using the standard Transformer Block method in Definition 10, denoted as block4;

[0198] An up-sampling layer is constructed using the standard convolution operation method in Definition 6, denoted as layer_up2;

[0199] A conditional Transformer Block is constructed by connecting block4 and the up-sampling layer layer_up2 in sequence, denoted as CTB4;

[0200] A Transformer Block is constructed using the standard Transformer Block method in Definition 10, denoted as block5;

[0201] The conditional Transformer Block CTB1 and the Transformer Block block5 are connected using the skip connection method in Definition 11;

[0202] The conditional Transformer Block CTB2 and the conditional Transformer Block CTB4 are connected using the skip connection method in Definition 11;

[0203] The conditional Transformer Block CTB1, the conditional Transformer Block CTB2, the conditional Transformer Block CTB3, the conditional Transformer Block CTB4 and the Transformer Block block5 are sequentially connected to build a structure component extraction branch network, denoted as Str_Ext.

[0204] Step 3.2, build the residual component extraction branch

[0205] A convolution layer is constructed using the standard convolution operation method in Definition 6, denoted as layer6;

[0206] An activation layer is constructed using the standard activation function method in Definition 12, denoted as layer7;

[0207] A convolution layer is constructed using the standard convolution operation method in Definition 6, denoted as layer8;

[0208] An activation layer is constructed using the standard activation function method in Definition 12, denoted as layer9;

[0209] A convolution layer is constructed using the standard convolution operation method in Definition 6, denoted as layer10;

[0210] An activation layer is constructed using the standard activation function method in Definition 12, denoted as layer11;

[0211] The convolution layer layer6, the activation layer layer7, the convolution layer layer8, the activation layer layer9, the convolution layer layer10 and the activation layer layer11 are sequentially connected to build a residual component extraction branch network, denoted as Res_Ext.

[0212] AsFigure 1 As shown, the multi-level multi-scale extraction module is constructed by the structure component extraction branch network Str_Ext obtained in step 3.1 and the residual component extraction branch network Res_Ext obtained in step 3.2 using the standard double-branch network construction method in definition 13, denoted as Ext_Net.

[0213] Step 4, constructing an enhancement module

[0214] Step 4.1, constructing a prediction network

[0215] Step 4.1.1, constructing a down-sampler

[0216] A convolution layer is constructed using the standard convolution operation method in definition 6, denoted as layer12;

[0217] Two residual block layers are constructed using the standard residual module method in definition 14, denoted as layer13 and layer14, respectively;

[0218] The convolution layer layer12, the residual block layers layer13 and layer14 are connected in sequence to obtain a mapping block, denoted as block_map1.

[0219] A down-sampling layer is constructed using the standard convolution operation method in definition 6, denoted as layer_down3;

[0220] Two residual block layers are constructed using the standard residual module method in definition 14, denoted as layer15 and layer16, respectively;

[0221] The down-sampling layer layer_down3, the residual block layers layer15 and layer16 are connected in sequence to obtain a down-sampling block, denoted as block_down1.

[0222] A down-sampling layer is constructed using the standard convolution operation method in definition 6, denoted as layer_down4;

[0223] Two residual block layers are constructed using the standard residual module method in definition 14, denoted as layer17 and layer18, respectively;

[0224] The down-sampling layer layer_down4, the residual block layers layer17 and layer18 are connected in sequence to obtain a down-sampling block, denoted as block_down2.

[0225] A down-sampling layer is constructed using the standard convolution operation method in definition 6, denoted as layer_down5;

[0226] Two residual block layers are constructed using the residual module method in definition 14, denoted as layer19 and layer20, respectively;

[0227] Two cross-attention block layers are constructed using the cross-attention module method in definition 15, denoted as layer21 and layer22, respectively.

[0228] A down-sampling block is constructed by connecting the down-sampling layer layer_down5, the residual block layers layer19 and layer20, and the cross-attention block layers layer21 and layer22 in sequence, denoted as block_down3.

[0229] A down-sampling layer is constructed using the convolution operation method in definition 6, denoted as layer_down6.

[0230] Two residual block layers are constructed using the residual module method in definition 14, denoted as layer23 and layer24, respectively.

[0231] Two cross-attention block layers are constructed using the cross-attention module method in definition 15, denoted as layer25 and layer26, respectively.

[0232] A down-sampling block is constructed by connecting the down-sampling layer layer_down6, the residual block layers layer23 and layer24, and the cross-attention block layers layer25 and layer26 in sequence, denoted as block_down4.

[0233] The mapping block block_map1, the down-sampling blocks block_down1, block_down2, block_down3, and block_down4 are connected in sequence to construct a down-sampler, denoted as DownSampling.

[0234] Step 4.1.2, constructing an intermediate connection layer

[0235] A residual block layer is constructed using the residual module method in definition 14, denoted as layer_res1.

[0236] A cross-attention block layer is constructed using the cross-attention module method in definition 15, denoted as layer_cra.

[0237] A residual block layer is constructed using the residual module method in definition 14, denoted as layer_res2.

[0238] The residual block layer layer_res1, the cross-attention block layer layer_cra, and the residual block layer layer_res2 are sequentially connected to obtain an intermediate connection layer, denoted as Middle.

[0239] Step 4.1.3, constructing an upsampler

[0240] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer27, layer28, and layer29, respectively.

[0241] Three cross-attention block layers are constructed by using the cross-attention module method in definition 15, denoted as layer30, layer31, and layer32, respectively.

[0242] An upsampling layer is constructed by using the convolution operation method in definition 6, denoted as layer_up3.

[0243] The residual block layers layer27, layer28, and layer29, the cross-attention block layers layer30, layer31, and layer32, and the upsampling layer layer_up3 are sequentially connected to obtain an upsampling block, denoted as block_up1.

[0244] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer33, layer34, and layer35, respectively.

[0245] Three cross-attention block layers are constructed by using the cross-attention module method in definition 15, denoted as layer36, layer37, and layer38, respectively.

[0246] An upsampling layer is constructed by using the convolution operation method in definition 6, denoted as layer_up4.

[0247] The residual block layers layer33, layer34, and layer35, the cross-attention block layers layer36, layer37, and layer38, and the upsampling layer layer_up4 are sequentially connected to obtain an upsampling block, denoted as block_up2.

[0248] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer39, layer40, and layer41, respectively.

[0249] An upsampling layer is constructed by using the convolution operation method in definition 6, denoted as layer_up5.

[0250] Connect the residual block layers layer39, layer40 and layer41, the up-sampling layer layer_up5 in sequence to obtain an up-sampling block, denoted as block_up3.

[0251] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer42, layer43 and layer44, respectively.

[0252] An up-sampling layer is constructed by using the convolution operation method in definition 6, denoted as layer_up6.

[0253] Connect the residual block layers layer42, layer43 and layer44, the up-sampling layer layer_up6 in sequence to obtain an up-sampling block, denoted as block_up4.

[0254] Three residual block layers are constructed by using the residual module method in definition 14, denoted as layer45, layer46 and layer47, respectively.

[0255] A convolution layer is constructed by using the convolution operation method in definition 6, denoted as layer48.

[0256] Connect the residual block layers layer45, layer46 and layer47, the convolution layer layer48 in sequence to obtain a mapping block, denoted as block_map2.

[0257] Connect the up-sampling block block_up4, the up-sampling block block_up3, the up-sampling block block_up2, the up-sampling block block_up1 and the mapping block block_map2 in sequence to obtain an up-sampler, denoted as UpSampling.

[0258] The mapping block block_map1 and the mapping block block_map2 are connected by using the skip connection method in definition 11.

[0259] The up-sampling block block_up1 and the down-sampling block block_down1 are connected by using the skip connection method in definition 11; the up-sampling block block_up2 and the down-sampling block block_down2 are connected by using the skip connection method in definition 11.

[0260] The up-sampling block block_up3 and the down-sampling block block_down3 are connected by using the skip connection method in definition 11; the up-sampling block block_up4 and the down-sampling block block_down4 are connected by using the skip connection method in definition 11.

[0261] The residual block layer layer_res1 and the residual block layer layer_res2 are connected by using the skip connection method with the definition 11 standard.

[0262] Finally, the down-sampler DownSampling, the middle layer Middle, and the up-sampler UpSampling are sequentially connected to construct a prediction network, denoted as Pre_Net.

[0263] Step 4.2, constructing a forward diffusion process

[0264] A T-step forward diffusion process q is constructed by using the standard Markov chain criterion in definition 16, as follows:

[0265]

[0266] where y t ,t∈(0,T) represents the image at the t-th step, y0 represents the original image, I represents the unit matrix, α i ,i∈(0,T) represents the weight coefficient of the noise added at the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, represents the standard Gaussian noise, and represents the noise level to be added at the t-th step.

[0267] Step 4.3, constructing a reverse diffusion process

[0268] A T-step reverse diffusion process p is constructed by using the standard Markov chain criterion in definition 16, as follows:

[0269]

[0270] where y t ,t∈(0,T) represents the image at the t-th step, y t-1 represents the image at the t-1-th step, y0 represents the original image, I represents the unit matrix, α i ,i∈(0,T) represents the weight coefficient of the noise added at the i-th step, represents the cumulative multiplication of the weight coefficients of the previous t steps, ε θ,t ,t∈(0,T) represents the prediction result of the prediction network Pre_Net of step 4.1 for the t-th step forward diffusion process.

[0271] Step 4.4, constructing an enhancement module based on the diffusion model

[0272] First, the original image y0 is calculated according to the formula based on the forward diffusion process q constructed in step 4.2, to obtain the noisy image y t at the t-th step.

[0273] Then, the denoising is performed according to the reverse diffusion process p constructed in step 4.3, and the image y of the time step t is obtained t According to the formula The calculation is performed to obtain the denoised image y of the t-1 step t-1 .

[0274] The enhancement module based on the diffusion model, denoted as Enh_Net, is obtained from the above forward noise adding process and reverse denoising process. Step 5, constructing a two-stage low-light lensless reconstruction model

[0275] The training set Train obtained in step 1 is used as input, and the multi-level multi-scale extraction module Ext_Net obtained in step 3 is trained using the classical Adam algorithm in definition 17. Then, the output of the multi-level multi-scale extraction module is spliced with the ground truth image GT in step 1 in the channel dimension using the standard splicing method in definition 8, and the enhancement module Enh_Net obtained in step 4 is trained using the classical Adam algorithm in definition 17. After training, the two-stage low-light lensless reconstruction model is obtained, denoted as TSLR.

[0276] Step 6, testing the two-stage low-light lensless reconstruction model

[0277] The test set Test obtained in step 1 is tested on the two-stage low-light lensless reconstruction model TSLR obtained in step 5 using the standard forward propagation method in definition 18, and the test result of the test set Test on the two-stage low-light lensless reconstruction model TSLR is obtained, denoted as Result.

[0278] Step 7, evaluating the two-stage low-light lensless reconstruction model

[0279] The test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6 is calculated using the classical mean square error calculation method in definition 19, and the mean square error index of the test result is obtained, denoted as MSE.

[0280] The test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6 is calculated using the classical learning perceptual image block similarity calculation method in definition 20, and the learning perceptual image block similarity index of the test result is obtained, denoted as LPIPS.

[0281] The test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6 is calculated using the classical peak signal-to-noise ratio calculation method in definition 21, and the peak signal-to-noise ratio index of the test result is obtained, denoted as PSNR.

[0282] The test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6 is calculated by using the classical structural similarity calculation method in definition 22, and the structural similarity index of the test result is obtained, denoted as SSIM.

[0283] The results show that, compared with the existing DeepLIR method, the MSE and SSIM indexes of the reconstruction result of the method are both better, which indicates that the reconstructed image is more consistent with the true value and has higher fidelity, and the LPIPS index is smaller and the PSNR is larger, which indicates that the quality of the reconstructed image is higher and the color and other detail information is better recovered, as shown in FIG. 6. Figure 2 ​

Claims

1. A two-stage low-light lensless reconstruction method based on physical perception, the method comprising the following steps: Step 1, preparing a data set Taking the classic cat and dog game data set as the basis, 11000 pictures are selected and processed by a classic image resolution transformation method to obtain a true value image, denoted as GT; A standard phase mask method is used to process the image, and a phase mask processed image is obtained, denoted as PSF; According to the PSF, a standard low-light lensless acquisition method is used to acquire the true value image GT to obtain a low-light lensless measurement value, denoted as Measurement; Finally, the true value image GT and the corresponding low-light lensless measurement value Measurement are divided into two parts, the first 10000 pairs of pictures are used as the training set, denoted as Train, and the last 1000 pairs of pictures are used as the test set, denoted as Test; Step 2, constructing a physical fusion block A standard Pixel Shuffle transformation is used to construct an up-sampling layer, denoted as layer1; A standard convolution operation method is used to construct a convolution layer, denoted as layer2; The up-sampling layer layer1 and the convolution layer layer2 are connected in sequence to obtain a feature-to-image mapping layer, denoted as F2I; A classical least mean square error filter is used to construct an initial imaging model, as shown below: wherein is the Fourier transform, for the inverse Fourier transform, H is the phase mask PSF in step 1, y is the input low-light lensless image, and λ is a regularization coefficient, for the initial imaging result, (·) * denotes the complex conjugate operator, and denotes the Hadamard product; and the initial imaging result is spliced according to a standard splicing method is spliced with the output result of F2I to obtain a deconvolution layer, denoted as layer3; A standard convolution operation method is used to construct a convolution layer, denoted as layer4; A standard Pixel Shuffle transformation is used to construct a down-sampling layer, denoted as layer5; The down-sampling layer layer4 and the convolution layer layer5 are connected in sequence to obtain an image-to-feature mapping layer, denoted as I2F; The mapping layer F2I, the deconvolution layer layer3, and the mapping layer I2F are connected in sequence to obtain a physical fusion block, denoted as PFB; Step 3, constructing a multi-level multi-scale extraction module Step 3.1, constructing a structure component extraction branch A standard Transformer Block method is used to construct a Transformer Block, denoted as block1; A standard convolution operation method is used to construct a down-sampling layer, denoted as layer_down1; The block1, the physical fusion block PFB obtained in step 2, and the down-sampling layer layer_down1 are connected in sequence to construct a conditional Transformer Block, denoted as CTB1; A standard Transformer Block method is used to construct a Transformer Block, denoted as block2; A standard convolution operation method is used to construct a down-sampling layer, denoted as layer_down2; The block2, the physical fusion block PFB obtained in step 2, and the down-sampling layer layer_down2 are connected in sequence to construct a conditional Transformer Block, denoted as CTB2; adopting a standard convolution operation method to construct a convolution layer, denoted as layer6; adopting a standard activation function method to construct an activation layer, denoted as layer7; adopting a standard convolution operation method to construct a convolution layer, denoted as layer8; adopting a standard activation function method to construct an activation layer, denoted as layer9; adopting a standard convolution operation method to construct a convolution layer, denoted as layer10; adopting a standard activation function method to construct an activation layer, denoted as layer11; adopting a standard convolution operation method to construct a convolution layer, denoted as layer6; adopting a standard activation function method to construct an activation layer, denoted as layer7; adopting a standard convolution operation method to construct a convolution layer, denoted as layer8; adopting a standard activation function method to construct an activation layer, denoted as layer9; adopting a standard convolution operation method to construct a convolution layer, denoted as layer10; adopting a standard activation function method to construct an activation layer, denoted as layer11; connecting the convolution layer layer6, the activation layer layer7, the convolution layer layer8, the activation layer layer9, the convolution layer layer10 and the activation layer layer11 in sequence to build a residual component extraction branch network, denoted as Res_Ext; ​ ​ ​ ​ ​ The multi-level multi-scale extraction module, denoted as Ext_Net, is constructed by using the standard double-branch network construction method to construct the structure component extraction branch network Str_Ext obtained in step 3.1 and the residual component extraction branch network Res_Ext obtained in step 3.

2. Step 4, constructing an enhancement module Step 4.1, constructing a prediction network Step 4.1.1, constructing a down-sampler A convolution layer is constructed by using the standard convolution operation method, denoted as layer12; Two residual block layers are constructed by using the standard residual module method, denoted as layer13 and layer14, respectively. The convolution layer layer12, the residual block layers layer13 and layer14 are connected in sequence to obtain a mapping block, denoted as block_map1. A down-sampling layer is constructed by using the standard convolution operation method, denoted as layer_down3. Two residual block layers are constructed by using the standard residual module method, denoted as layer15 and layer16, respectively. The down-sampling layer layer_down3, the residual block layers layer15 and layer16 are connected in sequence to obtain a down-sampling block, denoted as block_down1. A down-sampling layer is constructed by using the standard convolution operation method, denoted as layer_down4. Two residual block layers are constructed by using the standard residual module method, denoted as layer17 and layer18, respectively. The down-sampling layer layer_down4, the residual block layers layer17 and layer18 are connected in sequence to obtain a down-sampling block, denoted as block_down2. A down-sampling layer is constructed by using the standard convolution operation method, denoted as layer_down5. Two residual block layers are constructed by using the standard residual module method, denoted as layer19 and layer20, respectively. Two cross-attention block layers are constructed by using the standard cross-attention module method, denoted as layer21 and layer22, respectively. The down-sampling layer layer_down5, the residual block layers layer19 and layer20, and the cross-attention block layers layer21 and layer22 are connected in sequence to obtain a down-sampling block, denoted as block_down3. A down-sampling layer is constructed by using the standard convolution operation method, denoted as layer_down6. Two residual block layers are constructed by using the standard residual module method, denoted as layer23 and layer24, respectively. Two cross-attention block layers are constructed by using the standard cross-attention module method, denoted as layer25 and layer26, respectively. The down-sampling layer layer_down6, the residual block layers layer23 and layer24, and the two cross-attention block layers layer25 and layer26 are connected in sequence to obtain a down-sampling block, denoted as block_down4. Connect the above constructed mapping block block_map1, downsampling block block_down1, downsampling block block_down2, downsampling block block_down3 and downsampling block block_down4 in turn to construct a downsampler, denoted as DownSampling; Step 4.1.2, construct an intermediate connection layer A residual block layer is constructed by using a standard residual module method, denoted as layer_res1; A cross attention block layer is constructed by using a standard cross attention module method, denoted as layer_cra; A residual block layer is constructed by using a standard residual module method, denoted as layer_res2; Connect the residual block layer layer_res1, the cross attention block layer layer_cra and the residual block layer layer_res2 in turn to construct an intermediate connection layer, denoted as Middle. Step 4.1.3, construct an upsampler Three residual block layers are constructed by using a standard residual module method, denoted as layer27, layer28 and layer29 respectively; Three cross attention block layers are constructed by using a standard cross attention module method, denoted as layer30, layer31 and layer32 respectively; An upsampling layer is constructed by using a standard convolution operation method, denoted as layer_up3; Connect the residual block layers layer27, layer28 and layer29, the cross attention block layers layer30, layer31 and layer32, and the upsampling layer layer_up3 in turn to construct an upsampling block, denoted as block_up1; Three residual block layers are constructed by using a standard residual module method, denoted as layer33, layer34 and layer35 respectively; Three cross attention block layers are constructed by using a standard cross attention module method, denoted as layer36, layer37 and layer38 respectively; An upsampling layer is constructed by using a standard convolution operation method, denoted as layer_up4; Connect the residual block layers layer33, layer34 and layer35, the cross attention block layers layer36, layer37 and layer38, and the upsampling layer layer_up4 in turn to construct an upsampling block, denoted as block_up2; Three residual block layers are constructed by using a standard residual module method, denoted as layer39, layer40 and layer41 respectively; An upsampling layer is constructed by using a standard convolution operation method, denoted as layer_up5; Connect the residual block layers layer39, layer40 and layer41, and the upsampling layer layer_up5 in turn to construct an upsampling block, denoted as block_up3; Three residual block layers are constructed by using a standard residual module method, denoted as layer42, layer43 and layer44 respectively; An up-sampling layer is constructed by using the standard convolution operation method, denoted as layer_up6; An up-sampling block is constructed by connecting the residual block layers layer42, layer43 and layer44 and the up-sampling layer layer_up6 in sequence, denoted as block_up4; Three residual block layers are constructed by using the standard residual module method, denoted as layer45, layer46 and layer47 respectively; A convolution layer is constructed by using the standard convolution operation method, denoted as layer48; A mapping block is constructed by connecting the residual block layers layer45, layer46 and layer47 and the convolution layer layer48 in sequence, denoted as block_map2; The up-sampling block block_up4, the up-sampling block block_up3, the up-sampling block block_up2, the up-sampling block block_up1 and the mapping block block_map2 are connected in sequence to construct an up-sampler, denoted as UpSampling; The mapping block block_map1 and the mapping block block_map2 are connected by using the standard skip connection method; The up-sampling block block_up1 and the down-sampling block block_down1 are connected by using the standard skip connection method; the up-sampling block block_up2 and the down-sampling block block_down2 are connected by using the standard skip connection method; The up-sampling block block_up3 and the down-sampling block block_down3 are connected by using the standard skip connection method; the up-sampling block block_up4 and the down-sampling block block_down4 are connected by using the standard skip connection method; The residual block layer layer_res1 and the residual block layer layer_res2 are connected by using the standard skip connection method; Finally, the down-sampler DownSampling, the intermediate layer Middle and the up-sampler UpSampling are connected in sequence to construct a prediction network, denoted as Pre_Net; Step 4.2, constructing a forward diffusion process A T-step forward diffusion process is constructed by using the standard Markov chain criterion, denoted as q, as follows: wherein y t , t e (0, T) represents the image at the tth step, y0 represents the original image, I represents the unit matrix, a i , i e (0, T) represents the weight coefficient of the added noise at the ith step, represents the cumulative multiplication of the weight coefficients of the previous t steps, represents the standard Gaussian noise, and a t represents the noise level that needs to be added at the tth step. Step 4.3, constructing a reverse diffusion process A T-step reverse diffusion process is constructed by using the standard Markov chain criterion, denoted as p, as follows: Among them, y t ,t∈(0,T) represents the image at step t, y t-1 Let y0 represent the image at step t-1, y0 represent the original image, I represent the identity matrix, and α represent the image at step t-1. i ,i∈(0,T) represents the weighting coefficient of the noise added in the i-th step. ε represents the cumulative product of the weight coefficients in the first t steps. θ,t ,t∈(0,T) represents the prediction result of the Pre_Net in step 4.1 for the forward diffusion process at step t; Step 4.4, constructing an enhancement module based on the diffusion model First, the original image y0is added with noise according to the forward diffusion process q constructed in step 4.2, according to the formula The image y t at the t-th step after adding noise is calculated. The denoising is then performed using the inverse diffusion process p constructed in step 4.3, on the image y at time step t t According to the formula The calculation is performed to obtain the denoised image y at step t-1 t-1 ; An enhancement module based on the diffusion model is obtained from the above forward noise adding process and reverse noise removing process, denoted as Enh_Net; Step 5, constructing a two-stage low-light lensless reconstruction model Take the training set Train obtained in step 1 as input, and use the classic Adam algorithm to train the multi-level multi-scale extraction module Ext_Net obtained in step 3; then the output of the multi-level multi-scale extraction module is spliced with the true value image GT in step 1 in the channel dimension using the standard splicing method, and the enhanced module Enh_Net obtained in step 4 is trained using the classic Adam algorithm, and the two-stage low-light lensless reconstruction model is obtained after training, denoted as TSLR; Step 6, test the two-stage low-light lensless reconstruction model On the two-stage low-light lensless reconstruction model TSLR obtained in step 5, the test set Test obtained in step 1 is tested using the standard forward propagation method, and the test result of the test set Test on the two-stage low-light lensless reconstruction model TSLR is obtained, denoted as Result; Step 7, evaluate the two-stage low-light lensless reconstruction model For the test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6, the classic mean square error calculation method is used for calculation, and the mean square error index of the test result is obtained, denoted as MSE; For the test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6, the classic learning perceptual image block similarity calculation method is used for calculation, and the learning perceptual image block similarity index of the test result is obtained, denoted as LPIPS; For the test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6, the classic peak signal-to-noise ratio calculation method is used for calculation, and the peak signal-to-noise ratio index of the test result is obtained, denoted as PSNR; For the test result Result of the two-stage low-light lensless reconstruction model TSLR obtained in step 6, the classic structural similarity calculation method is used for calculation, and the structural similarity index of the test result is obtained, denoted as SSIM; At this point, the whole method ends.

Images