A super-resolution stacked imaging method based on pixel merging and sample-free training

Abstract

The application discloses a kind of based on pixel merging and sample-free training super-resolution laminated imaging method, comprising the following steps: pre-preparation stage: the K diffraction patterns collected are directly brought into laminated imaging algorithm iteration i times reconstruction image, the image obtained is as high-resolution image T hr ;The pixel merging is carried out to the K diffraction patterns collected, and the K diffraction patterns after merging are brought into laminated imaging algorithm iteration i times reconstruction image, the image obtained is as low-resolution image T lr ;Training stage: adjust the parameter in sample-free training procedure;T hr With T lr Input degradation model, randomly intercept the region of same position in T hr With T lr Generate the hr-lr data pair used in training, process and train hr-lr data pair;Save the super-resolution model trained;Output result stage.The application utilizes the laminated imaging method of pixel merging and the sample-free training method in deep learning to realize image super-resolution, and can further improve reconstruction quality.

Description

Technical Field

[0001] This invention relates to a super-resolution stacked imaging method, and more particularly to a super-resolution stacked imaging method based on pixel merging and sample-free training. Background Technology

[0002] Layered imaging is a newly emerging lensless phase retrieval technique that enables large field-of-view quantitative phase imaging. In a layered imaging system, the illumination beam and the object undergo relative array scanning, with adjacent scan positions of the illumination spot overlapping on the object. Simultaneously, a CCD (Charge-Coupled Device) records the diffraction image to reconstruct the complex amplitudes of the illumination beam and the object. Layered imaging boasts advantages such as a large field of view, aberration-free imaging, and a wide imaging range, and has been widely applied in research on visible light, X-rays, and electron beams. However, layered imaging technology often suffers from resolution limitations in experiments. Pixel size and sampling interval directly affect image resolution; larger pixel sizes lead to detail loss. When using a CCD for acquisition, the performance and parameters of the CCD itself also affect the resolution of the final reconstructed image. Therefore, improving the resolution of reconstructed images from layered imaging is a continuously explored research direction. With the development of networks and the continuous improvement of computing power, deep learning technology combined with computational imaging, by integrating front-end optical systems with back-end computer image processing, has broken through the limitations of traditional techniques.

[0003] Chinese Patent Publication No. CN114066735A discloses a sparse sampling Fourier overlay imaging reconstruction method based on least-squares using artificial neural networks. This method employs a Fourier overlay imaging device to sparsely sample images, obtaining a series of low-resolution acquired images. By establishing a forward model of Fourier overlay imaging, the acquired low-resolution images are sequentially input into the forward model. The minimum absolute deviation between the simulated image generated by the artificial neural network and the acquired images is compared to solve a least-squares problem with constraints. Error backpropagation is used to train and update the weights of the artificial neural network, resulting in a reconstructed high-resolution phase-recovery image. However, this method requires a large number of low-resolution images for training, making the preliminary preparation complex and time-consuming. How to achieve image super-resolution using a small number of samples or even no samples is currently an important research direction. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to propose a super-resolution stacked imaging method based on pixel merging and sample-free training, which can achieve super-resolution without a large number of samples, using only its own image.

[0005] Technical solution: This invention includes the following steps:

[0006] Preliminary preparation stage: A stacked imaging experimental platform is set up, and K diffraction patterns of the sample to be tested are acquired as raw data. The acquired K diffraction patterns are directly input into the stacked imaging algorithm for i iterations to reconstruct the image, and the resulting image is used as the high-resolution image T. hr The acquired K diffraction patterns are pixel-merged, and the merged K diffraction patterns are fed into a stacked imaging algorithm for i iterations to reconstruct the image. The resulting image is used as the low-resolution image T. lr ;

[0007] Training phase: Adjust the parameters in the no-sample training procedure; set T... hr With T lr Input the degradation model and randomly select T hr With T lr Generate HR-LR data pairs for training from regions at the same location in the model, process the HR-LR data pairs and train the model; save the trained super-resolution model.

[0008] Output results stage.

[0009] The pre-preparation phase specifically includes:

[0010] The acquired K diffraction patterns are directly input into the stacked imaging method to reconstruct the image, and the reconstructed object function is obtained by iterating i times.

[0011] Number O i (x, y), name the image T hr And save, T hr Image size is ;

[0012] For the acquired K diffraction patterns, a pixel merging method is used to increase the pixel size of each diffraction pattern by a factor of c and decrease the number of pixels by a factor of c, where c is the magnification factor. The merged diffraction patterns are then used in a layered imaging method to reconstruct the image, iterating i times to obtain the reconstructed object function O'. i (x, y), name the image T lr And save, T lr Image size is ;

[0013] T hr With T lr The data pairs used for subsequent training are saved in the same folder.

[0014] The formula used for pixel merging is as follows:

[0015]

[0016] In the formula, I jThe intensity of the diffracted light is represented by (x, y), where (x, y) are the pixel coordinates recording the intensity, and (x', y') are the new...

[0017] The pixel coordinates are given, where c is the magnification factor.

[0018] The parameters in the sample-free training procedure include: number of iterations (epoch), scaling factor (s), learning rate (LR), and pruning size (size).

[0019] The size of hr is The size of lr is c is the magnification factor.

[0020] The data pair processing method is as follows: the hr-lr data pair is flipped at a random angle α.

[0021] The data pair training method is as follows: the low-resolution image lr is upsampled using subpixel convolution, the loss value is monitored using a loss function, and the trained super-resolution model is saved.

[0022] The loss function is the MSE loss function. Calculated using the following formula:

[0023]

[0024] In the formula, W and H represent the image size, and c represents the magnification factor. for The parameterized representation, This represents the weights and biases of a deep network.

[0025] The MSE loss is used to evaluate the pixel-level difference between the image sr and hr generated by subpixel convolution upsampling of lr in the hr-lr data pair during training.

[0026] The output stage involves adjusting the output stage parameters to match those used during the training stage of the super-resolution model, and then outputting the high-resolution image T obtained in the pre-preparation stage. hr The super-resolution model obtained during the training phase is used for reconstruction, and the super-resolution image T is output. sr .

[0027] Beneficial effects: This invention utilizes a pixel-merging stacked imaging method and a sample-free training method in deep learning to achieve image super-resolution, proposing a new method for achieving image super-resolution. Compared with the sample-free training method that uses bicubic interpolation for downsampling to obtain low-resolution images, it can further improve the reconstruction quality. Attached Figure Description

[0028] Figure 1This is a flowchart of the present invention;

[0029] Figure 2 This is a diagram of the stacked imaging experimental setup;

[0030] Figure 3 This is a schematic diagram of the stacked imaging algorithm.

[0031] Figure 4 This is a schematic diagram of the pixel merging principle.

[0032] Figure 5 A schematic diagram illustrating the no-sample training principle for generating low-resolution images using a pixel merging method;

[0033] Figure 6 The figure shows a comparison between the method of this invention, the interpolation method, and the original sample-less training method in experiments using fern stems as biological samples: Figure 6 (a1) is a high-resolution image obtained during the pre-preparation stage; Figure 6 (b1) is the image reconstructed using interpolation; Figure 6 (c1) is the image reconstructed from a high-resolution image used in the original training without samples; Figure 6 (d1) is the image reconstructed by the method of the present invention; Figure 6 (a2) Figure 6 (b2) Figure 6 (c2) and Figure 6 (d2) is a magnified view of the same part of the corresponding image. Detailed Implementation

[0034] The invention will now be further described with reference to the accompanying drawings.

[0035] like Figure 1 As shown, the super-resolution stacked imaging method of the present invention includes the following steps:

[0036] Step 1, Pre-preparation stage: Setting up such as Figure 2 The stacked imaging experimental platform shown acquires K diffraction patterns of the sample as raw data. These K diffraction patterns are then directly fed into the stacked imaging algorithm for i iterations to reconstruct the image, which is then used as the high-resolution image T. hr The acquired K diffraction patterns are pixel-merged, and the merged K diffraction patterns are fed into a stacked imaging algorithm for i iterations to reconstruct the image. The resulting image is used as the low-resolution image T. lr ; T hr With T lr Saved in the same folder; specifically:

[0037] (11) The collected K diffraction patterns are directly input into the stacked imaging method to reconstruct the image, and the image is obtained by iterating i times.

[0038] Building function Oi (x, y), name the image T hr And save, T hr Image size is ;

[0039] (12) Use the following methods to process the collected K diffraction patterns: Figure 4 The pixel merging method shown increases the pixel size of each diffraction pattern by a factor of c and reduces the number of pixels by a factor of c, where c is the magnification factor. The merged diffraction patterns are then used in a layered imaging method to reconstruct the image, iterating i times to obtain the reconstructed object function O'. i (x, y), name the image T lr And save, T lr Image size is The formula used for pixel merging is as follows:

[0040]

[0041] In the formula, I j The intensity of the diffracted light is represented by (x, y), where (x, y) are the pixel coordinates recording the intensity, and (x', y') are the new...

[0042] The pixel coordinates, where c is the magnification factor;

[0043] (13) T hr With T lr The data pairs used for subsequent training are saved in the same folder.

[0044] like Figure 3 As shown, the specific steps of the layered imaging algorithm are as follows:

[0045] (1) Let the object function be O(r), the illumination field function be P(r), and r(x,y) be the coordinates of the object plane. The coordinates of the j-th diffraction pattern recorded by the CCD are R. j Let j represent the scanning position of the diffraction pattern, j = 1, 2, ..., J, J = M × N, where M is the number of lateral movements of the illumination probe and N is the number of longitudinal movements. The randomly guessed object function and illumination field function are O(n). n (r), P n (r), where n is the number of iterations.

[0046] (2) Multiplying the object function estimate and the illumination field function estimate, we obtain the outgoing light field as follows:

[0047]

[0048] Among them, O n (rR j The amplitude distribution of the sample under test after it has moved relative to the illumination light is represented by .

[0049] (3) The emitted light field is propagated to the CCD plane, and the complex amplitude distribution on the CCD plane is as follows:

[0050]

[0051] Among them, |ψ n (u, R j | represents amplitude, θ n (u, R j () indicates phase.

[0052] (4) Replace with the diffraction intensity actually recorded by the CCD The amplitude of the diffraction field is kept constant while its phase remains unchanged, and the updated complex amplitude distribution of the diffraction field is obtained on the CCD plane, i.e.

[0053]

[0054] Among them, I(u, R j () indicates the intensity of the diffracted light actually recorded by the CCD.

[0055] (5) The updated diffraction field is transmitted back to the object plane in the reverse direction.

[0056]

[0057] (6) The update function updates the object function and the illumination field function.

[0058]

[0059]

[0060] In the formula, "*" indicates the conjugate of the corresponding function, and α and β are update coefficients with values ​​ranging from [0, 1].

[0061] (7) Repeat steps (1) to (6) for each acquired diffraction image, using the result of the previous iteration as the initial guess for the next iteration. Repeat this process until the reconstruction error is sufficiently small. The error function is:

[0062] .

[0063] Step 2, Training Phase: Adjust the parameters in the no-sample training procedure; set T... hr With T lr Input the training program to train the model; save the trained super-resolution model; specifically:

[0064] (21) Adjust the parameters required for training, including the number of iterations (epoch), the scaling factor (s), the learning rate (LR), and the pruning size (size). The epoch value ranges from (500, 5000), the s value ranges from any value in [2, 3, 4], the LR value ranges from (1e-4, 5e-4), and the size value ranges from... d is T hr Image size The smaller value in;

[0065] (22) Calculate the degradation model: such as Figure 5 The degradation model shown is based on T hr With T lr The calculation shows that T is first... hr With T lr Input the degradation model and randomly select T hr With T lr The high-resolution-low-resolution data pairs used for training to generate regions at the same location in the dataset are hr-lr data pairs, where the hr size is [value missing]. The size of lr is c is the magnification factor. Finally, the hr-lr data pairs are processed and input into the network for training. The data pair processing method is as follows: the hr-lr data pairs are randomly flipped at an angle α, with α ranging from (0°, 90°, 180°, 270°) to increase the sample diversity during training.

[0066] (23) Start training. During training, use subpixel convolution to upsample the low-resolution image lr, monitor the loss value using a loss function, and save the trained super-resolution model. The loss function is the MSE loss function. Calculated using the following formula:

[0067]

[0068] In the formula, W and H represent the image size, and c represents the magnification factor. for The parameterized representation, This represents the weights and biases of a deep network.

[0069] The MSE loss is used to evaluate the pixel-level difference between the image sr and hr generated by subpixel convolution upsampling the lr data pair from the hr-lr data pair during training. Subpixel convolution transforms a low-resolution input image into a high-resolution image by using convolutional layers in a convolutional neural network (CNN). It employs a three-step strategy: first, it processes the input image using ordinary convolution operations; then, it rearranges the pixels of the convolutional feature map through pixel rearrangement; and finally, it applies another convolution operation to achieve a higher resolution output.

[0070] Step 3, Output Stage: Adjust the output stage parameters to match those used during the training stage of the super-resolution model. Then, output the high-resolution image T obtained in the pre-preparation stage. hr The super-resolution model obtained during the training phase is used for reconstruction, and the super-resolution image T is output. sr .

[0071] Example

[0072] A helium-neon laser (632.8 nm) was used as the light source, and a circular probe was fabricated using an aperture with a diameter of 2 mm. The biological sample to be tested (fern stem sample) was placed on an xy two-dimensional displacement stage with a sample step of 0.1 mm. An 8-bit CCD camera (3672×5496 pixels, pixel size 2.4 μm×2.4 μm) was selected to acquire a 7×7 diffraction pattern. The axial distance from the object to the CCD was 19.64 mm.

[0073] In the pre-preparation phase, the diffraction pattern without pixel merging is first incorporated into the stacked imaging algorithm to reconstruct the image, resulting in a high-resolution image of type T. hr The image size is 1088 pixels × 1088 pixels. Then, each diffraction pattern is pixel-merged, and the merged diffraction pattern is used in a stacked imaging algorithm to reconstruct the image. The resulting low-resolution image is T. lr The image size is 272 pixels × 272 pixels. Two reconstructed images are saved as data pairs for subsequent no-sample training. During the training phase, the magnification factor s is set to 4, the learning rate LR is 1e-4, the cropped high-resolution image size is 256 pixels, and the low-resolution image size is 64 pixels; the number of iterations is set to 3000. After training, the generated super-resolution model is saved. The Python version used is 3.9, and the PyTorch version is 11.0.

[0074] like Figure 6 (a1) shows the high-resolution image T obtained during the pre-preparation stage. hr It can be seen that many details of the image have not been restored, the outlines are not clear, and the image quality is poor. Figure 6 (a2) is from Figure 6 (a1) is a portion of the area that has been cropped and enlarged. Figure 6 (b1) For T hr The effect was magnified 4 times using bicubic interpolation. Figure 6 (b1) It can be seen that the changes between it and the original image are not obvious; Figure 6 (b2) is for interception Figure 6 (b1) in Figure 6(a2) Enlarge the same parts for comparison Figure 6 (b2) and Figure 6 (a2) shows that the image details have hardly changed, resulting in poor restoration. Figure 6 (c1) is the super-resolution image obtained using the no-sample training method without modifying the degradation model. Overall, the changes in the image are not significant. Figure 6 (c2) and Figure 6 (a2) Comparative observation shows that the image clarity has been improved to a certain extent, but many fine stripes also appear in the image, which has a great impact on the image quality. Figure 6 (d1) shows the method used in this invention. Overall, the image contains more high-dimensional information, and some darker areas in the original image are successfully restored, demonstrating significant experimental results. (Comparison) Figure 6 (d2) and Figure 6 (a2) The image edge contours are obvious, and more details are restored compared to the previous two methods, which significantly improves the image quality, indicating that the method of the present invention is effective.

Claims

1. A super-resolution stacked imaging method based on pixel merging and sample-free training, characterized in that, Includes the following steps: Preparatory stage: The acquired K diffraction patterns are directly fed into the stacked imaging algorithm for i iterations to reconstruct the image, and the resulting image is used as the high-resolution image T. hr ; The acquired K diffraction patterns are pixel-merged, and the merged K diffraction patterns are fed into a stacked imaging algorithm for i iterations to reconstruct the image. The resulting image is used as the low-resolution image T. lr The specific steps of the layered imaging algorithm are as follows: (1) Let the object function be O(r), the illumination field function be P(r), r(x,y) be the object plane coordinates, and the coordinates of the j-th diffraction pattern recorded by the CCD be R. j Let j represent the scanning position of the diffraction pattern, j=1, 2,…, J, J=M×N, where M is the number of lateral movements of the illumination probe and N is the number of longitudinal movements. The randomly guessed object function and illumination field function are O(n). n (r), P n (r), where n is the number of iterations; (2) Multiplying the object function estimate and the illumination field function estimate, we obtain the outgoing light field as follows: Among them, O n (rR j This represents the complex amplitude distribution of the sample after it has been moved relative to the illumination light. (3) The emitted light field is propagated to the CCD plane, and the complex amplitude distribution on the CCD plane is as follows: Among them, |ψ n (u, R j | represents amplitude, θ n (u, R j () indicates phase; (4) Replace with the diffraction intensity actually recorded by the CCD The amplitude of the diffraction field is kept constant while its phase remains unchanged, and the updated complex amplitude distribution of the diffraction field is obtained on the CCD plane, i.e. Among them, I(u, R j () represents the intensity of the diffracted light actually recorded by the CCD; (5) The updated diffraction field is transmitted back to the object plane in the reverse direction. (6) The update function updates the object function and the illumination field function. In the formula, "*" indicates the conjugate of the corresponding function, and α and β are update coefficients with values ​​ranging from [0, 1]. (7) Repeat steps (1) to (6) for each acquired diffraction image, using the result of the previous iteration as the initial guess for the next iteration. Repeat this process until the reconstruction error reaches the preset value. The error function is: In the formula, I j Indicates the intensity of the collected diffracted light; Training phase: Adjust the parameters in the no-sample training procedure; set T... hr With T lr Input the degradation model and randomly select T hr With T lr Generate HR-LR data pairs for training from regions at the same location in the model, process the HR-LR data pairs and train the model; save the trained super-resolution model. Output results stage.

2. The super-resolution stacked imaging method based on pixel merging and sample-free training according to claim 1, characterized in that, The pre-preparation phase specifically includes: The acquired K diffraction patterns are directly input into the stacked imaging method to reconstruct the image, and the reconstructed object function is obtained by iterating i times. Number O i (x, y) are the pixel coordinates that record the intensity. The image is named T. hr And save, T hr Image size is ; For the acquired K diffraction patterns, a pixel merging method is used to increase the pixel size of each diffraction pattern by a factor of c and decrease the number of pixels by a factor of c, where c is the magnification factor. The merged diffraction patterns are then used in a layered imaging method to reconstruct the image, iterating i times to obtain the reconstructed object function O'. i (x, y), name the image T lr And save, T lr Image size is ; T hr With T lr The data pairs used for subsequent training are saved in the same folder.

3. The super-resolution stacked imaging method based on pixel merging and sample-free training according to claim 2, characterized in that, The formula used for pixel merging is as follows: In the formula, (x', y') are the new pixel position coordinates, and c is the magnification factor.

4. The super-resolution stacked imaging method based on pixel merging and sample-free training according to claim 1, characterized in that, The parameters in the sample-free training procedure include: number of iterations (epoch), scaling factor (s), learning rate (LR), and pruning size (size).

5. The super-resolution stacked imaging method based on pixel merging and sample-free training according to claim 1, characterized in that, The size of hr is The size of lr is c is the magnification factor.

6. The super-resolution stacked imaging method based on pixel merging and sample-free training according to claim 1, characterized in that, The data pair processing method is as follows: the hr-lr data pair is flipped at a random angle α.

7. The super-resolution stacked imaging method based on pixel merging and sample-free training according to claim 6, characterized in that, The data pair training method is as follows: the low-resolution image lr is upsampled using subpixel convolution, the loss value is monitored using a loss function, and the trained super-resolution model is saved.

8. The super-resolution stacked imaging method based on pixel merging and sample-free training according to claim 7, characterized in that, The loss function is the MSE loss function. Calculated using the following formula: In the formula, W and H represent the image size, and c represents the magnification factor. for The parameterized representation, This represents the weights and biases of a deep network.

9. The super-resolution stacked imaging method based on pixel merging and sample-free training according to claim 8, characterized in that, The MSE loss is used to evaluate the pixel-level difference between the image sr and hr generated by subpixel convolution upsampling of lr in the hr-lr data pair during training.

10. The super-resolution stacked imaging method based on pixel merging and sample-free training according to claim 1, characterized in that, The output stage involves adjusting the output stage parameters to match those used during the training stage of the super-resolution model, and then outputting the high-resolution image T obtained in the pre-preparation stage. hr The super-resolution model obtained during the training phase is used for reconstruction, and the super-resolution image T is output. sr .

Images