Lensless imaging reconstruction method and equipment

Abstract

The invention discloses a lens-free imaging reconstruction method and equipment, belongs to the technical field of extremely-simple optical system imaging, and is used for solving the problems of image perception quality disclosure and artifacts existing in an existing optical coding image reconstruction method, which limit the development of a lens-free imaging technology and cause high imaging quality. And an image with the same quality as that of a traditional lens camera cannot be reproduced. The method comprises the following steps: acquiring optical coding images and lens images in pairs, and constructing an image pair data set; constructing a Le-ADMM model and a loss compensation network corresponding to the Le-ADMM model; constructing a double-feature cross network, and taking the output feature maps of the Le-ADMM model and the loss compensation network as the input of the double-feature cross network to obtain a lensless imaging reconstruction model; and training a lens-free imaging reconstruction model through the image pair data set, and inputting the optical coding image to be reconstructed into the trained lens-free imaging reconstruction model to obtain a reconstructed optical coding image.

Description

Technical Field

[0001] The present invention relates to the technical field of minimalist optical system imaging, and in particular to a lens-free imaging reconstruction method and device. Background Art

[0002] Emerging demands such as wearable / integrated sensors, the Internet of Things, and augmented / virtual reality have greatly promoted the development of imaging devices towards miniaturization, multi-functions, and high cost-effectiveness. At the same time, due to the use of lens imaging methods, traditional imaging optical technology has the disadvantages of low design freedom, bulky size, high cost, and limited multi-dimensional data acquisition capabilities. Therefore, people are actively exploring new imaging modes to overcome the limitations of traditional imaging optical technology.

[0003] Lensless imaging technology is an emerging hot area in the field of computational imaging. This technology optically encodes the real scene through a mask, and records the optically encoded image after the mask through a sensor, and then uses a lensless imaging reconstruction method to reconstruct the scene directly from the optically encoded image. Lensless imaging technology gets rid of the dependence on bulky and complex optical lenses, further improves the miniaturization and integration of imaging systems, and is very suitable for integration into portable or wearable devices.

[0004] Existing lensless imaging reconstruction methods can be divided into physical model-based methods and data-driven methods. The physical model-based method describes the lensless imaging system as a model-based inverse problem, and reconstructs the real scene by minimizing the loss function through an iterative optimization algorithm. The optimization function of the model consists of a data fidelity term and a regularization term used to satisfy the image prior knowledge. This method utilizes the prior information in the lensless imaging system, which is called the point spread function (PSF). However, since the physical model used is often too idealized and single, it is difficult to obtain the ideal PSF in practice due to errors and calibration errors in the system itself. In addition, the sparsity prior used in the iterative optimization method is not applicable to all scenarios. This type of method cannot provide convergence guarantees within a few iterations, and the reconstructed image has a low perceived quality due to model mismatch and calibration errors. The data-driven method does not add any prior information in the lensless reconstruction process. Compared with the traditional method based on the physical model, the data-driven method is not interpretable, there is no structured way to insert the knowledge of the image system, and it is impossible to mathematically derive and prove the convergence of the results. Therefore, although the data-driven method based on deep learning can reduce the inference time and improve the image quality, various artifact problems often occur due to the lack of guidance from physical models.

[0005] It can be seen that the existing optical coding image reconstruction methods have problems with image perception quality and artifacts, which limits the development of lensless imaging technology and cannot reproduce images of the same high quality as traditional lens cameras. It is urgent to develop and improve the existing lensless imaging reconstruction methods to promote the development of lensless imaging technology. Summary of the invention

[0006] The embodiments of the present invention provide a lensless imaging reconstruction method and device for solving the following technical problems: the existing optical coding image reconstruction method has problems with image perception quality and artifacts, which limits the development of lensless imaging technology and cannot reproduce images of the same high quality as traditional lens cameras.

[0007] The embodiment of the present invention adopts the following technical solution:

[0008] In one aspect, an embodiment of the present invention provides a lensless imaging reconstruction method, the method comprising: collecting an optically encoded image and a lens image in pairs to construct an image pair data set;

[0009] Constructing a Le-ADMM model and its corresponding loss compensation network; the input of the loss compensation network is the original optically encoded image and the reconstructed feature map output by the Le-ADMM model;

[0010] Constructing a dual-feature cross transformer network, and using the output feature maps of the Le-ADMM model and the loss compensation network as inputs of the dual-feature cross transformer network to obtain a lens-free imaging reconstruction model;

[0011] The lensless imaging reconstruction model is trained by using the image pair data set, and the optically encoded image to be reconstructed is input into the trained lensless imaging reconstruction model to obtain a reconstructed optically encoded image.

[0012] In a feasible implementation, the optical coding image and the lens image are collected in pairs to construct an image pair data set, specifically including:

[0013] In the same scene, an optical coding image and a lens image are collected for the same field of view target, and an image pair is formed; wherein the optical coding image is collected by a lensless camera, and the lens image is collected by an RGB camera;

[0014] Change the scene or field of view, collect different image pairs, and aggregate the collected image pairs into an image pair dataset Among them, P I is the optically encoded image, P O is the corresponding lens image; k∈(1,2,…,K) represents the sequence number of the image pair, and K is the total number of image pairs in the image pair data set.

[0015] In a feasible implementation, a Le-ADMM model and its corresponding loss compensation network are constructed, specifically including:

[0016] Constructing a Le-ADMM model; wherein the Le-ADMM model includes an image reconstruction module and a feature extraction module;

[0017] The image reconstruction module is iterated N times to obtain reconstructed images I1, I2, ..., I N , and input into the feature extraction module for feature extraction, and output the reconstructed feature map

[0018] Based on the reconstructed feature map, a loss compensation network of the Le-ADMM model is constructed to obtain a compensated feature map through the loss compensation network.

[0019] In a feasible implementation, based on the reconstructed feature map, a loss compensation network of the Le-ADMM model is constructed to obtain a compensation feature map, specifically including:

[0020] according to Get compensation feature map Among them, P I is the original optically encoded image;

[0021] based on Construct the model formula of the loss compensation network; wherein, F 1 ,F 2 ,……,F N is an auxiliary variable, is the compensation feature map; Represents the convolution feature extraction operation; Dense represents the dense connection between network layers;

[0022] Concate means aggregating two features along the channel direction;

[0023] Compensation feature map Substitute into the model formula and iteratively calculate all compensation characteristic maps.

[0024] In a feasible implementation, a dual-feature cross transformer network is constructed, and the output feature maps of the Le-ADMM model and the loss compensation network are used as inputs of the dual-feature cross transformer network to obtain a lens-free imaging reconstruction model, which specifically includes:

[0025] Constructing a first convolutional layer and a second convolutional layer; the first convolutional layer is used to extract a first shallow feature of the reconstructed feature map, and the second convolutional layer is used to extract a second shallow feature of the compensation feature map output by the loss compensation network;

[0026] Constructing a feature fusion module based on a dual-feature cross transformer network DFCT; wherein the feature fusion module is composed of a number of identical DFCT blocks;

[0027] The outputs of the first convolutional layer and the second convolutional layer are used as inputs of the feature fusion module to obtain the lens-free imaging reconstruction model.

[0028] In a feasible implementation, constructing the first convolutional layer and the second convolutional layer specifically includes:

[0029] Construct the first convolution layer H with a convolution kernel size of 3×3 FE1 (·), and according to Extract and reconstruct feature maps The first superficial feature in, W and H represent the length and width of the feature, respectively, and C represents the number of channels of the feature;

[0030] Construct the second convolution layer H with a convolution kernel size of 3×3 FE2 (·), and according to Extract compensation feature map The second shallow feature F y ; Among them, F y ∈R W×H×C .

[0031] In a feasible implementation, a feature fusion module based on a dual-feature cross transformer network DFCT is constructed, specifically including:

[0032] Based on the dual-feature cross transformer network, i identical DFCT blocks are constructed; each DFCT block includes two cross attention blocks, namely the first cross attention block and the second cross attention block;

[0033] Let the input of the i-th DFCT block be the output of the i-1-th DFCT block and the second shallow feature F y ; Where i is a positive integer greater than 0; the input of the first DFCT block is the first shallow feature

[0034] The input features are fused through the first cross-attention block and the second cross-attention block in the i-th DFCT block to obtain the output features of the i-th DFCT block.

[0035] Connect i DFCT blocks in sequence, finally connect them with two convolutional layers, and construct the calculation formula for reconstructed image: Complete the construction of the feature fusion module.

[0036] In a feasible implementation, the first cross-attention block and the second cross-attention block in the i-th DFCT block fuse the input features to obtain the output features of the i-th DFCT block. Specifically include:

[0037] Will and F y Expand into non-overlapping blocks and Where, M = HW / L 2 , represents the total number of expanded blocks, and L represents the size of the expanded block;

[0038] Through the first cross-attention block, Linear Map to Key Vector Sum value vector Will Linear mapping to query vector Where, m=(1,2,……,M); W K , W V , W Q They are all mapping matrices; d is the dimension of the vector;

[0039] according to Get the first self-attention result F CA1 ; Among them, softmax(·) is a normalization operation;

[0040] according to Get the output result F1 of the first cross-attention block;

[0041] Through the second cross-attention block, Linear Map to Key Vector Sum value vector Will Linear mapping to query vector And the corresponding calculation result F of the second self-attention is CA2 ;in,

[0042] According to F2=MLP(LN(F y +F CA2 ))+F y +F CA2, and obtain the output result F2 of the second cross attention block; where MLP(·) represents the multi-layer perceptron operation, and LN(·) represents the layer normalization operation;

[0043] according to Get the initial output result of the i-th DFCT block;

[0044] according to The initial output result is optimized, and the output results of the first i-1 DFCT blocks are densely connected to obtain the output features of the i-th DFCT block; wherein DFCTB(·) represents the calculation operation of the DFCT block.

[0045] In a feasible implementation manner, training the lensless imaging reconstruction model using the image pair dataset specifically includes:

[0046] Dividing the image pair data set into a training set and a test set according to a preset ratio;

[0047] Training the neural network parameters in the Le-ADMM model, the loss compensation network, and the dual-feature cross transformer network through the training set until the network converges and the training is completed;

[0048] The optically encoded image in the test set is input into the trained lensless imaging reconstruction model to obtain a reconstructed test image. in, is the kth optically encoded image;

[0049] According to the lens image in the test set and the reconstructed test image, the loss function Performing a gradient descent operation, and optimizing the neural network parameters in the Le-ADMM model, the loss compensation network, and the dual-feature cross transformer network according to the operation results; wherein, Represents the square root of the L2 norm.

[0050] On the other hand, an embodiment of the present invention further provides a lensless imaging reconstruction device, the device comprising:

[0051] at least one processor; and,

[0052] a memory communicatively connected to the at least one processor; wherein,

[0053] The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can perform a lensless imaging reconstruction method according to any one of claims 1-9.

[0054] Compared with the prior art, the lensless imaging reconstruction method and device provided by the embodiments of the present invention have the following beneficial effects:

[0055] (1) By combining the physical reconstruction model and the neural network model, the present invention can reconstruct a higher quality real scene image based on the known optical coding image. The image contains rich detailed textures and is more natural and consistent with the visual perception of the human eye.

[0056] (2) Compared with the traditional physical model-based method, the present invention avoids the problem of low perceived quality of the reconstructed image due to model mismatch and calibration error. Compared with the simple data-driven method, the present invention combines the physical model and the neural network model, adds prior information in the lens-free reconstruction process, makes the overall process more interpretable, and avoids some artifact information that usually exists.

[0057] (3) In addition, the physical model and its corresponding loss features are input into a dual-feature cross transformer network. This network takes into account the multiplexing characteristics of the lensless imaging process, has no local inductive bias, encodes through a self-attention mechanism, directly considers the global context, and enhances the global feature reasoning ability. It can understand the global information in the optically encoded image, so that the reconstructed image can be consistent with the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0059] Figure 1 A flow chart of a lensless imaging reconstruction method provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of a lensless imaging reconstruction model architecture provided by an embodiment of the present invention;

[0061] Figure 3 A schematic diagram of a dual-feature cross transformer network architecture provided by an embodiment of the present invention;

[0062] Figure 4 A schematic diagram of the internal structure of a dual-feature cross transformer network block provided in an embodiment of the present invention;

[0063] Figure 5A comparison diagram of image visual quality obtained by reconstructing using different methods provided in an embodiment of the present invention;

[0064] Figure 6 A schematic structural diagram of a lensless imaging reconstruction device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0066] The embodiment of the present invention provides a lensless imaging reconstruction method, such as Figure 1 As shown, the method specifically includes steps S101-S104:

[0067] S101. Collect optical coding images and lens images in pairs to construct an image pair data set.

[0068] Specifically, first, an optical coding image and a lens image are collected for the same field of view target in the same scene, and an image pair is formed; wherein the optical coding image is collected by a lensless camera, and the lens image is collected by an RGB camera.

[0069] Then, the scene or field of view object is changed to collect different image pairs until a sufficient number of image pairs are collected. The collected image pairs are aggregated into an image pair dataset. Among them, P I is the optically encoded image, P O is the corresponding lens image; k∈(1,2,…,K) represents the sequence number of the image pair, and K is the total number of image pairs in the image pair dataset.

[0070] As a feasible implementation method, in the same scene, for the same field of view target, an optical coding image is collected by a lensless camera and a lens image is collected by an RGB camera. The content and size of the real scene represented by these two images are consistent. Then the two images are bound into an image pair and stored in the image pair dataset. After collecting enough image pairs, the image pair dataset is stored for use in subsequent model training.

[0071] S102, constructing a Le-ADMM model and its corresponding loss compensation network; the input of the loss compensation network is the original optically encoded image and the reconstructed feature map output by the Le-ADMM model.

[0072] Specifically, a Le-ADMM model is first constructed; wherein the Le-ADMM model includes an image reconstruction module and a feature extraction module. Then, the image reconstruction module is iterated N times to obtain a reconstructed image I 1 ,I 2 ,……,I N , and input into the feature extraction module for feature extraction, and output the reconstructed feature map

[0073] As a feasible implementation method, the Le-ADMM model is a lensless imaging reconstruction method based on a physical model. The method obtains the reconstructed image by solving the following minimization optimization problem: Where A represents the measurement matrix constructed by the point spread function (PSF) of the lensless optical system, C represents the cropping operation, which is mainly used to limit the output size, and n and They represent the real scene captured and the optically encoded image acquired by the sensor respectively, Ψ represents the sparsification transformation, and λ represents the weight of the sparsity term, which is used to balance the data fidelity term and the sparsity term in the optimization process.

[0074] Then, the Alternating Direction Method of Multipliers (ADMM) is used to solve the minimization optimization problem in the formula by separating the variables, thereby reconstructing a clear image from the optically encoded image. Le-ADMM sets the hyperparameters in the algorithm to learnable parameters.

[0075] Figure 2 A schematic diagram of a lensless imaging reconstruction model architecture provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the intermediate images I1, I2, ..., I are reconstructed in the nth iteration by the Le-ADMM model. n , the total number of iterations is N, and the reconstructed image obtained from the last iteration is I N .

[0076] Then, the intermediate images I1, I2, ..., I N-1 And the reconstructed image I obtained from the last iteration N Feature extraction is performed through a separate convolutional neural network g(.), g(.)...g(.), where g(.) represents three convolutional layers with a kernel size of 3×3. The size of the image remains unchanged after convolution, such as Figure 2 As shown, I1, I2, ..., I N After feature extraction, the reconstructed feature maps are

[0077] Furthermore, based on the reconstructed feature map, a loss compensation network of the Le-ADMM model is constructed to obtain a compensation feature map through the loss compensation network, specifically including:

[0078] according to Get compensation feature map ; Among them, P I is the original optically encoded image.

[0079] Then based on The model formula for constructing the loss compensation network. Among them, F 1 ,F 2 ,……,F N is an auxiliary variable, is the compensation feature map; Represents the convolution feature extraction operation; Dense represents the dense connection between network layers;

[0080] Concate means aggregating two features along the channel direction;

[0081] Then the compensation feature map Substitute into the above model formula and iteratively calculate all compensation feature maps.

[0082] As a feasible implementation method, Figure 2 As shown, the compensation characteristic diagram The reconstructed feature map obtained from the first iteration of the Le-ADMM model After the above formula is used for calculation, the auxiliary variable F is output. 1 , auxiliary variable F 1 After convolution operation Then we get the compensation feature map Compensation characteristic diagram The reconstructed feature map obtained from the second iteration of the Le-ADMM model After the above formula is used for calculation, the auxiliary variable F is output. 2 , auxiliary variable F 2 After convolution operation Then we get the compensation feature map And so on, the final auxiliary variable F N-1 After convolution operation Then we get the compensation feature map

[0083] Then the reconstructed feature map obtained by the Nth iteration of the Le-ADMM model And the compensation feature map output by the loss compensation network Input into the Dual Feature Cross Transformer (DFCT) network.

[0084] S103. Construct a dual-feature cross transformer network, and use the output feature maps of the Le-ADMM model and the loss compensation network as inputs of the dual-feature cross transformer network to obtain a lens-free imaging reconstruction model.

[0085] Specifically, a first convolutional layer and a second convolutional layer are constructed; the first convolutional layer is used to extract the first shallow features of the reconstructed feature map, and the second convolutional layer is used to extract the second shallow features of the compensation feature map output by the loss compensation network.

[0086] As a feasible implementation, the first convolution layer H with a convolution kernel size of 3×3 is constructed. FE1 (·), and according to Extract and reconstruct feature maps The first superficial feature in, W and H represent the length and width of the feature respectively, and C represents the number of channels of the feature. Construct the second convolution layer H with a convolution kernel size of 3×3 FE2 (·), and according to Extract compensation feature map The second shallow feature F y Among them, F y ∈R W×H×C .

[0087] Furthermore, a feature fusion module based on a dual-feature cross transformer network is constructed, wherein the feature fusion module is composed of several identical DFCT blocks (DFCT Block, DFCTB). Then, the outputs of the first convolutional layer and the second convolutional layer are used as the input of the feature fusion module to obtain a lens-free imaging reconstruction model.

[0088] Among them, a feature fusion module based on the dual feature cross transformer network DFCT is constructed, which specifically includes:

[0089] Based on the dual-feature cross transformer network DFCT, i identical DFCTBs are constructed; each DFCTB includes two cross attention blocks, namely the first cross attention block and the second cross attention block.

[0090] Let the input of the i-th DFCT block be the output of the i-1-th DFCT block and the second shallow feature F y ; Where i is a positive integer greater than 0; the input of the first DFCT block is the first shallow feature

[0091] The input features are fused through the first cross-attention block and the second cross-attention block in the i-th DFCT block to obtain the output features of the i-th DFCT block.

[0092] Connect i DFCT blocks in sequence, finally connect them with two convolutional layers, and construct the calculation formula for reconstructed image: Complete the construction of the feature fusion module.

[0093] As a feasible implementation method, Figure 3 A schematic diagram of a dual-feature cross transformer network architecture provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the reconstructed feature map obtained by the Nth iteration of the Le-ADMM model And the compensation feature map output by the loss compensation network After being input into the DFCT network, each of them is first extracted through shallow features to obtain the first shallow feature and the second shallow feature F y , and then input into the first block DFCTB-1 of the DFCT network for calculation, and the output Again with After dense connection, it is input into the next block DFCTB-2, and the second shallow feature F is also input. y Similarly, the output of each block is densely connected with the outputs of all previous blocks and then with F y Input into the next block together until the output features of the i-th block DFCTB-i are obtained. Then, according to the reconstructed image calculation formula Two convolution operations are performed to obtain the final reconstructed optically encoded image.

[0094] In a further embodiment, Figure 4 A schematic diagram of the internal structure of a dual-feature cross transformer network block provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown in the figure, the input features are fused by the first cross attention block and the second cross attention block in the i-th DFCT block to obtain the output features of the i-th DFCT block. Specifically include:

[0095] Will and F y Expand into non-overlapping blocks and Where, M = HW / L 2 , represents the total number of expanded blocks, and L represents the size of the expanded blocks.

[0096] The first cross-attention block will Linear Map to Key Vector Sum value vector Will Linear mapping to query vector Where, m=(1,2,……,M); W K , W V , W Q They are all mapping matrices; d is the dimension of the vector.

[0097] Then according to Get the first self-attention result F CA1 ; Among them, softmax(·) is a normalization operation.

[0098] Then according to The output result F1 of the first cross-attention block is obtained.

[0099] The second cross-attention block is opposite to the first cross-attention block. Linear Map to Key Vector Sum value vector Will Linear mapping to query vector And according to Correspondingly calculate the second self-attention result F CA2 ;in,

[0100] According to F2=MLP(LN(F y +F CA2 ))+F y +F CA2 , and the output result F2 of the second cross attention block is obtained; where MLP(·) represents the multi-layer perceptron operation and LN(·) represents the layer normalization operation.

[0101] Then according to Get the initial output result of the i-th DFCT block.

[0102] Since the features extracted by deep neural networks at different depths have different information, the present invention adopts dense connection to enhance feature propagation and promote feature reuse, while also alleviating gradient disappearance. The initial output result is optimized, and the output results of the first i-1 DFCT blocks are densely connected to obtain the output features of the i-th DFCT block. Figure 3 It is also reflected in ; where DFCTB(·) represents the calculation operation of the DFCT block.

[0103] S104, training a lensless imaging reconstruction model through an image pair data set, and inputting the optically encoded image to be reconstructed into the trained lensless imaging reconstruction model to obtain a reconstructed optically encoded image.

[0104] Specifically, the image pair data set is divided into a training set and a test set according to a preset ratio. The preset ratio is preferably training set:test set=8:2.

[0105] Furthermore, the training set and various hyperparameters during training, including learning rate, batch size, number of channels of the network, and convolution kernel size, are input. The neural network parameters in the Le-ADMM model, loss compensation network, and dual-feature cross transformer network are trained using the training set until the network converges and the training is completed.

[0106] Furthermore, the optically encoded images in the test set are input into the trained lensless imaging reconstruction model to obtain the reconstructed test image in, is the kth optically encoded image.

[0107] Then, based on the lens images in the test set and the reconstructed test images, the loss function Perform gradient descent operations and optimize the neural network parameters in the Le-ADMM model, loss compensation network, and dual-feature cross transformer network based on the operation results; Represents the square root of the L2 norm. The above process is repeated to update the neural network parameters in the model until the loss function converges, and a well-trained lens-free imaging reconstruction model is obtained.

[0108] Finally, the optically encoded image to be reconstructed is input into the trained lensless imaging reconstruction model to obtain the reconstructed optically encoded image.

[0109] To demonstrate the effectiveness of this method, an open dataset from the DiffuserCam at the University of California, Berkeley, is used for experimental verification. The dataset contains a total of 25,000 pairs of optically encoded images and lens images of the same scene. The original images are downsampled by 4 times to obtain images of size 480×270 to avoid the degradation of lens images due to moire fringes. The dataset is split into 20,000 pairs for training and 5,000 pairs for testing.

[0110] During the implementation, the number of dual-feature cross transformer blocks i=4, the number of channels C is set to 64, the size of the convolution kernel in all convolutional layers is 3×3, and the Adam optimizer is used to train the network. The number of cycles of the data set in training is set to 200 times, the batch size is 8, and the initial learning rate is set to 0.001. For every 50 training cycles, the learning rate decays to one tenth of the original.

[0111] During the experiment, Mean Square Error (MSE), Peak Signal-to-Noise Ratio (PSNR) and Learned Perceptual Image Patch Similarity (LPIPS) were used as evaluation indicators of the reconstructed image, in which the lens image obtained by the traditional lens camera was used as the reference image. Table 1 shows the comparison results of the present invention and other methods on the above indicators. It can be found that the present invention has a lower MSE and a higher PSNR, indicating that the present invention can more accurately reconstruct the same information as the lens image. In addition, the results on LPIPS are also significantly better than other methods. Figure 5 The figure shows a comparison of the visual quality of images reconstructed by different methods. The results show that the effect achieved by the present invention not only performs well in quantitative indicators, but also the reconstructed images have more outstanding visual quality, are visually clearer, better reflect the real scene being captured, and accurately reconstruct the color and detail information.

[0112] Table 1

[0113]

[0114]

[0115] In addition, an embodiment of the present invention further provides a lensless imaging reconstruction device, such as Figure 6 As shown, the lensless imaging reconstruction device specifically includes:

[0116] at least one processor; and a memory in communication with the at least one processor; wherein,

[0117] The memory stores instructions executable by at least one processor to enable the at least one processor to perform:

[0118] Collect optically coded images and lens images in pairs to construct an image pair dataset;

[0119] Constructing a Le-ADMM model and its corresponding loss compensation network; the input of the loss compensation network is the original optically encoded image and the reconstructed feature map output by the Le-ADMM model;

[0120] Constructing a dual-feature cross transformer network, and using the output feature maps of the Le-ADMM model and the loss compensation network as inputs of the dual-feature cross transformer network to obtain a lens-free imaging reconstruction model;

[0121] The lensless imaging reconstruction model is trained by using the image pair data set, and the optically encoded image to be reconstructed is input into the trained lensless imaging reconstruction model to obtain a reconstructed optically encoded image.

[0122] The present invention proposes a lensless imaging reconstruction method using Le-ADMM reconstruction and cross-attention of its loss compensation features. This method can combine the advantages of physical model methods and data-driven methods to significantly improve the quality of reconstructed images. In addition, experimental results on multiple data sets verify the robustness and superiority of the algorithm. The algorithm effectively improves the reconstruction quality of lensless imaging systems and increases the applicability of lensless imaging technology. A real scene image with higher definition can be reconstructed from an optically encoded image.

[0123] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0124] The above describes specific embodiments of the present invention. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0125] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present invention should be included in the protection scope of the present invention.

Claims

1. A lensless imaging reconstruction method, characterized in that: The method comprises: Collect optically coded images and lens images in pairs to construct an image pair dataset; Constructing a Le-ADMM model and its corresponding loss compensation network; the input of the loss compensation network is the original optically encoded image and the reconstructed feature map output by the Le-ADMM model; Constructing a dual-feature cross transformer network, and using the output feature maps of the Le-ADMM model and the loss compensation network as inputs to the dual-feature cross transformer network to obtain a lensless imaging reconstruction model; The lensless imaging reconstruction model is trained using the image pair data set, and the optically encoded image to be reconstructed is input into the trained lensless imaging reconstruction model to obtain a reconstructed optically encoded image.

2. The lensless imaging reconstruction method according to claim 1, wherein: Collect optically coded images and lens images in pairs to construct an image pair dataset, including: In the same scene, an optical coding image and a lens image are collected for the same field of view object, and an image pair is formed. The optical coding image is collected by a lensless camera, and the lens image is collected by an RGB camera. Transform the scene or field of view, collect different image pairs, and aggregate the collected image pairs into an image pair dataset Among them, P I is the optically encoded image, P O is the corresponding lens image; k∈(1,2,……,K), represents the sequence number of the image pair, and K is the total number of image pairs in the image pair dataset.

3. The lensless imaging reconstruction method according to claim 1, wherein: Construct the Le-ADMM model and its corresponding loss compensation network, specifically including: Constructing a Le-ADMM model; wherein the Le-ADMM model includes an image reconstruction module and a feature extraction module; The image reconstruction module is iterated N times to obtain reconstructed images I1, I2, ..., I N , and input it into the feature extraction module for feature extraction, and output the reconstructed feature map Based on the reconstructed feature map, a loss compensation network of the Le-ADMM model is constructed to obtain a compensated feature map through the loss compensation network.

4. The lensless imaging reconstruction method according to claim 3, wherein: Based on the reconstructed feature map, a loss compensation network of the Le-ADMM model is constructed to obtain a compensation feature map, specifically including: according to Get the compensation feature map Among them, P I is the original optically encoded image; based on Construct the model formula of the loss compensation network; wherein, F 1 ,F 2 ,……,F N is an auxiliary variable, is the compensation feature map; Represents the convolution feature extraction operation; Dense represents the dense connection between network layers; Concate represents the aggregation of two features along the channel direction; Compensate the feature map Substitute into the model formula and iteratively calculate all compensation feature maps.

5. The lensless imaging reconstruction method according to claim 1, wherein: A dual-feature cross transformer network is constructed, and the output feature maps of the Le-ADMM model and the loss compensation network are used as inputs of the dual-feature cross transformer network to obtain a lensless imaging reconstruction model, specifically including: Constructing a first convolutional layer and a second convolutional layer; the first convolutional layer is used to extract the first shallow features of the reconstructed feature map, and the second convolutional layer is used to extract the second shallow features of the compensation feature map output by the loss compensation network; Constructing a feature fusion module based on a dual-feature cross transformer network (DFCT); wherein the feature fusion module is composed of several identical DFCT blocks; The outputs of the first convolutional layer and the second convolutional layer are used as inputs of the feature fusion module to obtain the lensless imaging reconstruction model.

6. The lensless imaging reconstruction method according to claim 5, characterized in that: Construct the first convolutional layer and the second convolutional layer, specifically including: Construct the first convolution layer H with a convolution kernel size of 3×3 FE1 (·), and according to Extract and reconstruct feature maps The first shallow feature in, W and H represent the length and width of the feature, respectively, and C represents the number of channels of the feature; Construct the second convolution layer H with a convolution kernel size of 3×3 FE2 (·), and according to Extract compensation feature map The second shallow feature F y Among them, F y ∈R W×H×C .

7. The lensless imaging reconstruction method according to claim 5, characterized in that: Construct a feature fusion module based on the dual-feature cross transformer network DFCT, specifically including: Based on the dual-feature cross transformer network, i identical DFCT blocks are constructed; each DFCT block includes two cross attention blocks, namely the first cross attention block and the second cross attention block; Let the input of the i-th DFCT block be the output of the i-1-th DFCT block and the second shallow feature F y ; Where i is a positive integer greater than 0; the input of the first DFCT block is the first shallow feature The input features are fused through the first cross attention block and the second cross attention block in the i-th DFCT block to obtain the output features of the i-th DFCT block. Connect i DFCT blocks in sequence, and finally connect them with two convolutional layers, and construct the calculation formula for reconstructed image: Complete the construction of the feature fusion module.

8. The lensless imaging reconstruction method according to claim 7, wherein: The input features are fused through the first cross attention block and the second cross attention block in the i-th DFCT block to obtain the output features of the i-th DFCT block. Specifically include: Will and E y Expand into non-overlapping blocks and Where M = HW / L 2 , represents the total number of expanded blocks, L represents the size of the expanded blocks; Through the first cross-attention block, Linear map to key vector Sum value vector Will Linear mapping to query vector Where, m=(1,2,……,M); W K 、W V 、W Q are all mapping matrices; d is the dimension of the vector; according to Get the first self-attention result F CA1 ; Among them, softmax(·) is a normalization operation; according to Get the output result E1 of the first cross attention block; By the second cross-attention block, Linear map to key vector Sum value vector Will Linear mapping to query vector And the corresponding calculation of the second self-attention result E CA2 ;in, According to F2=MLP(LN(F y +F CA2 ))+F y +F CA2 , and obtain the output result F2 of the second cross attention block; where MLP(·) represents the multi-layer perceptron operation and LN(·) represents the layer normalization operation; according to Get the initial output result of the i-th DFCT block; according to The initial output result is optimized, and the output results of the first i-1 DFCT blocks are densely connected to obtain the output features of the i-th DFCT block; wherein DFCTB(·) represents the calculation operation of the DFCT block.

9. The lensless imaging reconstruction method according to claim 1, wherein: Training the lensless imaging reconstruction model using the image pair dataset specifically includes: Dividing the image pair dataset into a training set and a test set according to a preset ratio; Training the neural network parameters in the Le-ADMM model, the loss compensation network, and the dual-feature cross transformer network using the training set until the network converges and the training is completed; The optically encoded image in the test set is input into the trained lensless imaging reconstruction model to obtain the reconstructed test image. in, is the kth optically encoded image; According to the lens image in the test set and the reconstructed test image, the loss function Performing a gradient descent operation and optimizing the neural network parameters in the Le-ADMM model, the loss compensation network, and the dual-feature cross transformer network based on the operation results; wherein, represents the square root of the L2 norm.

10. A lensless imaging reconstruction device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, so that the at least one processor can perform the lensless imaging reconstruction method according to any one of claims 1 to 9.

Images