Refractive index reconstruction method for sparse observation optical diffraction tomography

Abstract

The invention discloses a refractive index reconstruction method for sparse observation optical diffraction tomography. According to the method, a video diffusion model is adopted to capture potential correlation of sample observation data at different illumination angles, random Gaussian noise added by a cosine noise scheduler is taken as a label, a 3D neural network is trained, dense observation complex light field images at continuous angles are converted into multi-frame video data, and the multi-frame video data are acquired. Potential dynamic distribution among different angles is captured by using a video diffusion model, and then under the framework of diffusion posterior sampling, frame interpolation is performed on real sparse observation video data by using the trained video diffusion model to complete angle augmentation, so that the time resolution of three-dimensional refractive index reconstruction is improved on the premise of ensuring the spatial resolution; and finally, high-speed and high-definition three-dimensional refractive index reconstruction is realized, the refractive index for enhancing the spatial resolution is obtained, multi-time acceleration of acquisition is realized, and meanwhile, the spatial resolution can also be ensured.

Description

Technical Field

[0001] The present invention relates to the field of microscopic imaging technology, and in particular to a refractive index reconstruction method for sparse observation optical diffraction tomography. Background Art

[0002] Optical diffraction tomography illuminates and observes transparent objects from different angles, reconstructing the object's three-dimensional refractive index distribution from these observations. This often requires collecting dozens to hundreds of images at different illumination angles to achieve high-definition 3D reconstruction. Current mainstream optical diffraction tomography techniques only achieve acquisition rates of 0.5–1 Hz, resulting in slow imaging. This is particularly true when imaging biological samples with high dynamics, which often experience some movement between images. This often results in motion blur in the reconstruction results, limiting their application to more diverse biological samples. Some approaches consider the object's motion as a parameter to be solved, incorporating neural representations or non-convex optimization techniques to solve the 4D dynamics of the object's refractive index distribution. This can mitigate the effects of sample motion on solution clarity. However, these methods often involve complex optimization processes that cannot guarantee convergence, or assume rigid body motion, limiting their practical application in biomedical and industrial testing.

[0003] Holographic solution technology: The characteristics of the physical light field can be described by complex numbers, and it is actually necessary to obtain observations in the form of complex light fields. However, mainstream microscope cameras can only obtain intensity information and lose phase information. By introducing off-axis holographic interferometry to modulate the complex light field, the phase information is reflected on the interference pattern with stripes, and then the complex light field can be obtained through holographic solution technology.

[0004] Video Diffusion Model Interpolation: The diffusion model generates high-quality data by gradually introducing noise into the data distribution and learning the reverse denoising process. Video interpolation technology based on the video diffusion model introduces this idea into the video frame generation task. By modeling the potential dynamics between consecutive frames, it can better capture complex motion information and detail changes, improving the accuracy and naturalness of interpolation. After the diffusion model is trained, adjacent original frames are input under the framework of diffusion posteriorsampling, and intermediate frames are generated through the reverse diffusion process. This method can adapt to a variety of motion scenes and effectively alleviates the shortcomings of traditional interpolation methods in motion estimation and detail restoration.

[0005] In order to improve the actual imaging temporal resolution of optical diffraction tomography, it is considered to directly shoot a small number of angles to reconstruct the refractive index 3D from sparse observations, but this will greatly reduce the clarity and resolution of the reconstruction results. Summary of the Invention

[0006] In response to the problems existing in the above-mentioned existing technologies, the present invention proposes a refractive index reconstruction method for sparse observation optical diffraction tomography, which uses a video diffusion model to capture the potential correlation of sample observation data under different illumination angles, and uses this method to perform interpolation and augmentation on sparse observations to improve the temporal resolution of three-dimensional refractive index reconstruction while ensuring spatial resolution.

[0007] The refractive index reconstruction method of sparse observation optical diffraction tomography of the present invention comprises the following steps:

[0008] 1) Get data:

[0009] Setting multiple fixed illumination angles, photographing multiple samples using an optical diffraction tomography system, and obtaining original holograms under multiple illumination angles;

[0010] 2) Preprocessing data:

[0011] Holographic decomposition technology is used to decompose the obtained holograms at various illumination angles to obtain real dense observed complex light field images of multiple samples at multiple illumination angles, and the corresponding unwrapped phase distribution is further calculated based on the real dense observed complex light field images;

[0012] 3) Training video diffusion model:

[0013] a) Build a video diffusion model, which includes a video dataset, a backbone network, a training method, and an inverse diffusion sampling process;

[0014] The illumination angle dimension of the real dense observation complex light field image under multiple illumination angles is regarded as the time dimension. The real part and imaginary part of the complex number correspond to a channel respectively, forming video data, forming a video dataset, and dividing the video dataset into a training set and a test set; the video data in the test set is downsampled in the time dimension to obtain sparse observation video data, and the real part and imaginary part are subsequently merged to obtain the corresponding sparse observation complex light field image;

[0015] b) The backbone network of the video diffusion model uses a neural network. A one-dimensional temporal self-attention module is inserted after the spatial self-attention module of the 2D neural network to form a 3D neural network. The 3D neural network has the ability to capture information associations in the third dimension, the temporal dimension, in addition to the spatial dimension.

[0016] c) Randomly cropping fixed-size video data blocks from the training set of the video dataset, and adding random Gaussian noise to the video data blocks under the planning of the cosine noise scheduler to obtain noisy videos. The noisy videos are used as input data for 3D neural network training, with noise as output and the random Gaussian noise added by the cosine noise scheduler as labels. The 3D neural network is trained by inserting frames of the sparse observation video data in the test set using the video diffusion model to perform angle augmentation, obtaining dense observation video data. This is then compared with the corresponding real dense observation video data in the test set to test the effectiveness of the 3D neural network and obtain a trained video diffusion model.

[0017] 4) Reconstructing the 3D refractive index:

[0018] The illumination angle is set to an equally spaced sparse illumination angle, and high-speed acquisition is performed. The hologram under the sparse illumination angle is collected and the data preprocessing of step 2) is performed to obtain a real sparse observation complex light field image, and the illumination angle dimension is regarded as the time dimension to be converted into real sparse observation video data. The trained video diffusion model is used to interpolate the real sparse observation video data to complete angle augmentation, and the real part and the imaginary part are merged into a complex number to obtain a dense observation complex light field image after interpolation processing. The corresponding unwrapped phase distribution is calculated according to the dense observation complex light field image, and the unwrapped phase distribution is reconstructed to obtain a three-dimensional refractive index with enhanced spatial resolution under high-speed acquisition, thereby achieving multiple acceleration of acquisition while ensuring spatial resolution.

[0019] In step 1), the number of sample images is 75 or more. The sample is transparent or translucent, and the hologram is obtained by collecting the light transmitted through the sample. The number of illumination angles is 32 to 96, generally set to a multiple of 8.

[0020] In step 2), 85-95% of the video dataset is divided into a training set.

[0021] In step 3) b), the 3D neural network adopts a 3DU network (Unet), which fuses the spatial dimension of the input data as the batch dimension and performs corresponding transformation to adapt to the network structure of the one-dimensional temporal attention module; at the same time, the transformed input data is encoded with a relative position with rotation invariance in the time dimension and input into the one-dimensional temporal self-attention module.

[0022] In step c) of step 3), a time step is set under the cosine noise scheduler, and random Gaussian noise with a variance corresponding to that time step is added to the video data block. The 3D neural network output noise and the noisy video are linearly combined to obtain the 3D neural network denoised video at the corresponding time step. In effect, the 3D neural network is trained to produce a video data denoiser capable of processing noisy videos with random Gaussian noise of varying variances, with the time step controlling the noise variance.

[0023] Testing the effectiveness of the 3D neural network includes the following steps: calculating the corresponding unwrapped phase distribution based on the dense observed complex light field image after interpolation processing, and inputting the unwrapped phase distribution into the optical diffraction tomography solver to obtain the refractive index, and comparing the refractive index obtained by step 2) calculating the corresponding unwrapped phase distribution based on the real dense observed complex light field image and inputting it into the optical diffraction tomography solver to prove that the 3D neural network is effective.

[0024] In step 4), the number of equally spaced sparse lighting angles is a factor of the number of dense lighting angles. To achieve sufficient acceleration, the number is generally 4 or 8.

[0025] In step 4), the video diffusion model is used to insert frames to complete angle augmentation, including the following steps:

[0026] a) During the inverse diffusion sampling process of the video diffusion model, given a time step plan, the 3D neural network in the video diffusion model is called multiple times to denoise the noisy video at the corresponding time step to complete the inverse diffusion sampling, obtaining video data blocks that conform to the distribution of the training set. The real and imaginary parts are then merged into complex numbers to obtain densely observed complex light field images that conform to the potential correlations between different angles captured by the 3D neural network from the training set;

[0027] b) In order to make the inverse diffusion sampling results also consistent with the sparse observation video data to achieve the goal of interpolation, a diffusion posterior sampling framework is introduced. During the inverse diffusion sampling process, the gradient signal of the error between the noisy video and the sparse observation video data is used to guide the inverse diffusion sampling process, including the following steps: calculating the loss function error between the sparse observation video data and the video frame at the same position of the 3D neural network denoised video at each time step, calculating the gradient of the error relative to the noisy video at each time step during the inverse diffusion sampling process through back propagation, and using this gradient as the guiding signal during the inverse diffusion sampling process, so that the denoised video at each time step during the inverse diffusion sampling process minimizes the error between the sparse observation video data and the training set; after completing the diffusion posterior sampling interpolation process, a video data block that is consistent with the distribution of the sparse observation video data and the training set is obtained, and the real and imaginary parts of the video data block are merged into a complex number to obtain a dense observation complex light field image that is consistent with both the sparse observation complex light field image and the potential correlation between different angles.

[0028] In step a), in order to improve the interpolation recovery results of the diffusion model and enhance the inverse diffusion sampling performance, the time step planning is as follows: the time-reversal technology is introduced. When sampling reaches the t-th time step after every n time steps, the time reversal is triggered, and the noisy signal of the current step is re-noised to the t+m-th time step. The sampling process is repeated to the current t-th time step, while maintaining the gradient guidance under sparse observation. This process is repeated to improve the sampling effect, where n, t, and m are all natural numbers; 0<=t<1000,0 <n<m<100。

[0029] Advantages of the present invention:

[0030] The present invention converts densely observed complex light field images at continuous angles into multi-frame video data, uses a video diffusion model to capture the potential dynamic distribution between different angles, and then interpolates the sparsely observed video data under the framework of diffuse posterior sampling, ultimately achieving high-speed and high-definition three-dimensional refractive index reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the refractive index reconstruction method of sparse observation optical diffraction tomography of the present invention;

[0032] Figure 2 Result graphs of the observed data complex light field images and the corresponding unwrapped phase distributions at multiple illumination angles obtained according to an embodiment of the refractive index reconstruction method for sparse observation optical diffraction tomography of the present invention, where (a) is the result graph of the real dense observation complex light field image, and (b) is the result graph of the unwrapped phase distribution;

[0033] Figure 3 This is a diagram of the reconstruction result after interpolation and augmentation of sparse observations, obtained according to an embodiment of the refractive index reconstruction method of sparse observation optical diffraction tomography of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below through specific embodiments in conjunction with the accompanying drawings.

[0035] This example performed label-free imaging experiments on live HEK cell samples. HEK cells were cultured in a 35mm diameter glass-bottomed dish. Optical diffraction tomography data were collected for the samples at different fields of view. Each capture involved fixing the angle of the illumination beam relative to the z-axis, rotating the beam, and collecting holograms at fixed intervals for a total of 32 angles.

[0036] The refractive index reconstruction method of the sparse observation optical diffraction tomography of this embodiment is as follows: Figure 1 As shown, the following steps are included:

[0037] 1) Get data:

[0038] Set multiple fixed illumination angles, use an optical diffraction tomography system to photograph multiple samples, and obtain original holograms under multiple illumination angles; collect the transmitted light through the sample, and photograph the sample to obtain 78 32×1535×1535 holograms;

[0039] 2) Preprocessing data:

[0040] Holographic solution technology is used to solve the holograms obtained at various illumination angles, and the real dense observation complex light field images of 78 samples at 32 illumination angles are obtained, such as Figure 2 (a), and further calculate the corresponding unwrapped phase distribution based on the real dense observation complex light field image, as shown in Figure 2 (b)

[0041] 3) Training video diffusion model:

[0042] a) Build a video diffusion model, which includes a video dataset, a backbone network, a training method, and an inverse diffusion sampling process;

[0043] The illumination angle dimension of the real dense observation complex light field image under multiple illumination angles is regarded as the time dimension. The real part and imaginary part of the complex number correspond to a channel respectively, forming video data, forming a video dataset. The video dataset is divided into a training set and a test set, with 74 video data as the training set; the video data in the test set is downsampled in the time dimension to obtain sparse observation video data, and the real part and imaginary part are subsequently merged to obtain the corresponding sparse observation complex light field image;

[0044] b) The backbone network of the video diffusion model uses a U-type network (Unet). The conventional 2DUnet network structure for processing image inputs consists of spatial upsampling and downsampling modules and a spatial self-attention module. A one-dimensional temporal self-attention module is inserted after the spatial self-attention module of the 2D neural network to form a 3D neural network. The spatial dimension of the input data is integrated as the batch dimension and transformed accordingly to adapt to the network structure of the one-dimensional temporal attention module. At the same time, the transformed input data is encoded with a rotation-invariant relative position in the temporal dimension and input into the one-dimensional temporal self-attention module to construct a 3DU network. This network has the ability to capture information associations in the third dimension, the temporal dimension, in addition to the spatial dimension.

[0045] c) Randomly cut fixed-size video data blocks from the training set of the video dataset, add random Gaussian noise to the video data blocks under the planning of the cosine noise scheduler to obtain a noisy video, use the noisy video as the input data for 3D neural network training, use the noise as the output, and use the random Gaussian noise added by the cosine noise scheduler as the label; the 3DUnet network predicts the noise of the noisy video, and gradually optimizes the L1 or L2 loss function between the 3DUnet network output noise and the actual noise at different time steps through the adamW optimizer, and trains for 200k iterations. The 3D neural network output noise and the noisy video are linearly combined to obtain the 3D neural network denoised video at the corresponding time step. In fact, the 3D neural network is trained to obtain a video data denoiser that can process noisy videos with random Gaussian noise of different variances, and the noise variance is controlled by the time step;

[0046] By using the video diffusion model to insert frames to complete angle augmentation on the sparse observation video data in the test set, dense observation video data is obtained. This is then compared with the corresponding real dense observation video data in the test set to test the effectiveness of the 3D neural network and obtain a trained video diffusion model.

[0047] 4) Reconstructing the 3D refractive index:

[0048] When acquiring data, the illumination angle is reset to an equally spaced sparse illumination angle and downsampled to 1 / 8 of the original angle, i.e., 4 angles, for high-speed acquisition. The hologram under the sparse illumination angle is acquired and the data preprocessing of step 2) is performed to obtain a real sparse observation complex light field image. The illumination angle dimension is regarded as the time dimension and converted into real sparse observation video data. The trained video diffusion model is used to interpolate the real sparse observation video data to complete the angle augmentation. The real and imaginary channels are merged into complex numbers to obtain a dense observation complex light field image after interpolation processing. The corresponding unwrapped phase distribution is calculated based on the dense observation complex light field image after interpolation processing, and the unwrapped phase distribution is input into the optical diffraction tomography solver to obtain the three-dimensional refractive index, achieving an 8-fold acceleration of acquisition while ensuring spatial resolution.

[0049] In step 4), frame insertion is performed to achieve angle augmentation, including the following steps:

[0050] a) During the inverse diffusion sampling process of the video diffusion model, given a time step plan, the 3D neural network in the video diffusion model is called multiple times to denoise the noisy video at the corresponding time step to complete the inverse diffusion sampling, obtaining video data blocks that conform to the distribution of the training set. The real and imaginary parts are then merged into complex numbers to obtain densely observed complex light field images that conform to the potential correlations between different angles captured by the 3D neural network from the training set. To improve the diffusion model interpolation recovery results and enhance the inverse diffusion sampling performance, the time step plan is as follows:

[0051] A time-reversal technique is introduced. When sampling, every 20 time steps, when reaching the tth time step, time reversal is triggered, and the noisy signal of the current step is re-noised to the tth + 60th time step. The sampling process is repeated to the current tth time step while maintaining the gradient guidance under sparse observation. This process is repeated to improve the sampling effect.

[0052] b) In order to make the inverse diffusion sampling results also consistent with the sparse observation video data to achieve the goal of interpolation, a diffusion posterior sampling framework is introduced. During the inverse diffusion sampling process, the gradient signal of the error between the noisy video and the sparse observation video data is used to guide the inverse diffusion sampling process, including the following steps: calculating the loss function error between the sparse observation video data and the video frame at the same position of the 3D neural network denoised video at each time step, calculating the gradient of the error relative to the noisy video at each time step during the inverse diffusion sampling process through back propagation, and using this gradient as the guiding signal during the inverse diffusion sampling process, so that the denoised video at each time step during the inverse diffusion sampling process minimizes the error between the sparse observation video data and the training set; after completing the diffusion posterior sampling interpolation process, a video data block that is consistent with the distribution of the sparse observation video data and the training set is obtained, and the real and imaginary parts of the video data block are merged into a complex number to obtain a dense observation complex light field image that is consistent with both the sparse observation complex light field image and the potential correlation between different angles.

[0053] The refractive index is reconstructed based on the corresponding unwrapped phase distribution calculated from the real densely observed complex light field image; the refractive index reconstruction results of the real densely observed complex light field, the sparsely observed complex light field and the densely observed complex light field after interpolation by the video model are compared. Figure 3 The reconstruction results of samples 1 to 4 are given. Samples 1 to 4 are randomly selected from the test set. It is found that the refractive index reconstruction results of the dense observation complex light field after the video model interpolation angle augmentation have higher spatial resolution than the sparse observation complex light field, and are closer to the refractive index reconstruction results of the real dense observation complex light field. In addition, compared with the refractive index reconstruction results of the real dense observation complex light field, fewer observation angles are actually required, corresponding to faster acquisition speed and higher temporal resolution. Figure 3 In the middle, only 1 / 8 of the observation angle is required.

[0054] Finally, it should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments; the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A refractive index reconstruction method for sparse observation optical diffraction tomography, characterized in that: The refractive index reconstruction method comprises the following steps: 1) Get data: Setting multiple fixed illumination angles, photographing multiple samples using an optical diffraction tomography system, and obtaining original holograms under multiple illumination angles; 2) Preprocessing data: Holographic decomposition technology is used to decompose the obtained holograms at various illumination angles to obtain real dense observed complex light field images of multiple samples at multiple illumination angles, and the corresponding unwrapped phase distribution is further calculated based on the real dense observed complex light field images; 3) Training video diffusion model: a) Build a video diffusion model, which includes a video dataset, a backbone network, a training method, and an inverse diffusion sampling process; The illumination angle dimension of the real dense observation complex light field image under multiple illumination angles is regarded as the time dimension. The real part and imaginary part of the complex number correspond to a channel respectively, forming video data, forming a video dataset, and dividing the video dataset into a training set and a test set; the video data in the test set is downsampled in the time dimension to obtain sparse observation video data, and the real part and imaginary part are subsequently merged to obtain the corresponding sparse observation complex light field image; b) The backbone network of the video diffusion model uses a neural network. A one-dimensional temporal self-attention module is inserted after the spatial self-attention module of the 2D neural network to form a 3D neural network. The 3D neural network has the ability to capture information associations in the third dimension, the temporal dimension, in addition to the spatial dimension. c) Randomly cropping fixed-size video data blocks from the training set of the video dataset, and adding random Gaussian noise to the video data blocks under the planning of the cosine noise scheduler to obtain noisy videos. The noisy videos are used as input data for 3D neural network training, with noise as output and the random Gaussian noise added by the cosine noise scheduler as labels. The 3D neural network is trained by inserting frames using the video diffusion model on the sparse observation video data in the test set to achieve angle augmentation. This verifies the effectiveness of the 3D neural network and results in a trained video diffusion model. 4) Reconstructing the 3D refractive index: The illumination angle is set to an equally spaced sparse illumination angle, and high-speed acquisition is performed. The hologram under the sparse illumination angle is collected and the data preprocessing of step 2) is performed to obtain a real sparse observation complex light field image, and the illumination angle dimension is regarded as the time dimension to be converted into real sparse observation video data. The trained video diffusion model is used to interpolate the real sparse observation video data to complete angle augmentation, and the real part and the imaginary part are merged into a complex number to obtain a dense observation complex light field image after interpolation processing. The corresponding unwrapped phase distribution is calculated according to the dense observation complex light field image, and the unwrapped phase distribution is reconstructed to obtain a three-dimensional refractive index with enhanced spatial resolution under high-speed acquisition, thereby achieving multiple acceleration of acquisition while ensuring spatial resolution.

2. The refractive index reconstruction method according to claim 1, wherein: In step 1), the number of photographed samples is 75 or more.

3. The refractive index reconstruction method according to claim 1, wherein: In step 2), 85-95% of the video dataset is divided into a training set.

4. The refractive index reconstruction method according to claim 1, wherein: In step 3) b), the 3D neural network adopts a 3DU type network, fuses the spatial dimension of the input data as the batch dimension, and performs corresponding transformations to adapt to the network structure of the one-dimensional temporal attention module; at the same time, the transformed input data is encoded with a rotation-invariant relative position in the time dimension and input into the one-dimensional temporal self-attention module.

5. The refractive index reconstruction method according to claim 1, wherein: In step c) of step 3), under the planning of the cosine noise scheduler, a time step is set, and random Gaussian noise with a variance corresponding to the time step is added to the video data block.

6. The refractive index reconstruction method according to claim 1, wherein: In step 3) c), the 3D neural network is tested to be effective, including the following steps: calculating the corresponding unwrapped phase distribution based on the dense observed complex light field image after interpolation processing, and inputting the unwrapped phase distribution into the optical diffraction tomography solver to obtain the refractive index, and comparing the refractive index obtained by calculating the corresponding unwrapped phase distribution based on the real dense observed complex light field image and inputting it into the optical diffraction tomography solver in step 2), proving that the 3D neural network is effective.

7. The refractive index reconstruction method according to claim 1, wherein: Use video diffusion model to insert frames to complete angle enhancement The process includes the following steps: a) In the reverse diffusion sampling process of the video diffusion model, given the time step planning, the 3D in the video diffusion model is called multiple times The neural network denoises the noisy video at the corresponding time step and completes the inverse diffusion sampling to obtain the video data block that conforms to the distribution of the training set. The real and imaginary parts are combined into complex numbers to obtain the different angles that conform to the 3D neural network captured from the training set. Densely observed complex light field images with potential correlations between them; b) A framework of diffuse posterior sampling is introduced. During the inverse diffusion sampling process, the gradient signal of the error between the noisy video and the sparse observation video data is used to guide the inverse diffusion sampling process, including the following steps: calculating the loss function error between the sparse observation video data and the video frame at the same position of the 3D neural network denoised video at each time step, calculating the gradient of the error relative to the noisy video at each time step during the inverse diffusion sampling process through back propagation, and using this gradient as a guiding signal during the inverse diffusion sampling process, so that the error between the denoised video at each time step during the inverse diffusion sampling process is minimized with respect to the sparse observation video data; after completing the diffuse posterior sampling interpolation processing, a video data block that conforms to both the sparse observation video data and the training set distribution is obtained, and the real and imaginary parts of the video data block are merged into a complex number to obtain a dense observation complex light field image that conforms to both the sparse observation complex light field image and the potential correlation between different angles.

8. The refractive index reconstruction method according to claim 7, wherein: In step a), the time step planning is as follows: the time backtracking technology is introduced. When sampling reaches the t-th time step after every n time steps, the time backtracking is triggered, and the noisy signal of the current step is re-noised to the t+m-th time step. The sampling process is repeated to the current t-th time step while maintaining the gradient guidance under sparse observation. This process is repeated to improve the sampling effect, where n, t, and m are all natural numbers.

Images