Light field self-supervision three-dimensional reconstruction method and system based on per-view aberration correction

Abstract

The invention discloses a light field self-supervised three-dimensional reconstruction method and system based on per-view aberration correction, and the method comprises the steps: carrying out the self-supervised training of a per-view aberration prediction module through employing a multi-view aberration-estimation body joint loss function, independently reasoning the aberration between a perfect light field image and an original light field image of a sample at each view, and carrying out the self-supervised three-dimensional reconstruction. And three-dimensional reconstruction is carried out based on the corrected per-view-angle image to obtain a three-dimensional body, so that the axial resolution is improved, and artifacts are reduced. Therefore, the technical problem that an ideal imaging effect of a three-dimensional body obtained through three-dimensional reconstruction is difficult to obtain due to aberration introduced by an optical system and a sample of an existing light field microscope is solved. According to the method, on the premise that training data is not needed, the phenomenon that artifacts are obvious during three-dimensional reconstruction due to the fact that aberrations of all view angles are inconsistent is avoided, the artifacts are reduced while the axial resolution is effectively improved, and a good light field image three-dimensional reconstruction effect is achieved.

Description

Technical Field

[0001] This invention belongs to the field of microscopic imaging technology, and more specifically, relates to a method and system for self-supervised three-dimensional reconstruction of light field based on per-view aberration correction. Background Technology

[0002] Light field microscopy boasts an extremely high volumetric imaging rate. Compared to conventional high-resolution 3D imaging techniques that require scanning and acquiring multiple 2D images to obtain 3D information, only a single image is needed to acquire 3D information. This significantly improves 3D imaging speed, enabling high-speed dynamic imaging of living organisms. Light field microscopy, by introducing microlens arrays into the Fourier transform plane, achieves 2D encoding of the depth and angle of 3D structures, thus enabling 3D imaging capabilities. For light field microscopy, reconstructing the original 2D light field image into a 3D volume is a crucial process.

[0003] However, the encoding process of light field microscopy involves complex processes such as spatial dimensionality reduction, optical blurring, and downsampling, making the 3D reconstruction of light field images an ill-posed inverse problem. Optical aberrations typically originate from the inhomogeneity of the tissue's refractive index and the optical system itself, leading to decreased image resolution and distortion. Each microlens in the microlens array of a light field microscope can be considered a sub-imaging system, and the spectral images captured by different microlenses must account for independent aberrations. Furthermore, in vivo imaging involves complex biological dynamics, and aberration levels vary with tissue changes, making accurate aberration estimation in light field microscopy extremely difficult, especially in the near-infrared band, ultimately resulting in unsatisfactory 3D reconstruction results. Summary of the Invention

[0004] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a self-supervised 3D reconstruction method and system for light fields based on per-view aberration correction. The aim is to employ a multi-view aberration-estimated volume joint loss function to self-supervise the per-view aberration prediction module, independently inferring the aberrations between the perfect light field image of the sample and the original light field image at each viewpoint, thereby performing per-view image correction. Based on the corrected per-view images, 3D volume reconstruction is performed to obtain a 3D volume, improving axial resolution and reducing artifacts. This solves the technical problem of existing light field microscopes where aberrations introduced by the optical system and sample result in unsatisfactory imaging effects in the 3D reconstructed volume.

[0005] To achieve the above objectives, according to one aspect of the present invention, a self-supervised 3D reconstruction method for light field based on view-by-view aberration correction is provided, comprising the following steps: (1) Obtain the original light field image formed by the imaging system and the three-dimensional reconstruction network of the light field, and construct a per-view aberration prediction module; The view-by-view aberration prediction module is used to independently infer the aberrations between the perfect light field image and the original light field image of the sample under each view, i.e., to estimate the aberrations, based on the current imaging conditions; the view-by-view aberration prediction module has... parameter matrix ,in Let Zernike's polynomials be the total number of terms. Let be the number of viewpoints, and let the aberrations be a linear combination of Zernike terms; The per-view aberration prediction module is constructed as follows: the parameter matrix of the per-view aberration prediction module and the parameters of the light field 3D reconstruction network are jointly optimized until convergence with the minimum of the multi-view aberration-estimation volume joint loss function as the optimization objective; the multi-view aberration-estimation volume joint loss function consists of multi-view aberration loss and 3D volume loss, and is used to quantify the difference between the estimated light field image after aberration is superimposed on the sample 3D image output by the light field 3D reconstruction network and the actual acquired original light field image, as well as the comprehensive loss of the physical authenticity of the 3D estimation volume; (2) The estimated aberration of the light field image of each view is inferred by the aberration prediction module constructed in step (1), and the aberration correction image of each view is obtained by performing aberration correction based on the image of each view in the original light field image and the estimated aberration of the image of that view. (3) Combine the aberration-corrected images of each viewpoint obtained in step (2) into an aberration-corrected light field image, and use the light field three-dimensional reconstruction network obtained in step (1) to reconstruct the aberration-corrected sample three-dimensional image.

[0006] Preferably, the light field self-supervised 3D reconstruction method based on view-by-view aberration correction, wherein the light field 3D reconstruction network is used to reconstruct the input light field image into an aberration-free sample 3D image, preferably an implicit neural representation model.

[0007] Preferably, in the light field self-supervised 3D reconstruction method based on viewpoint aberration correction, the aberrations are denoted as:

[0008] in, From the perspective The estimated aberration, From the perspective Zenik Township The weight, That is, Zernike polynomials.

[0009] Preferably, in the light field self-supervised 3D reconstruction method based on viewpoint aberration correction, the viewpoint image of the estimated light field is obtained by the following method: the 3D volume output by the light field 3D reconstruction network is forward-projected from the corresponding viewpoint using a point spread function containing aberrations to obtain the viewpoint image of the estimated light field.

[0010] Preferably, in the light field self-supervised 3D reconstruction method based on per-view aberration correction, the aberration-containing point spread function... An ideal point spread function based on wave optics The inverse Fourier transform is performed after adding the estimated aberrations for the corresponding viewpoints calculated by the current viewpoint aberration prediction module.

[0011] in, For the ideal point spread function, The view calculated by the current view-by-view aberration prediction module The estimated aberration, The imaginary unit, The fluorescence wavelength, For Fourier transform operations, This is the inverse Fourier transform operation.

[0012] Preferably, in the self-supervised 3D reconstruction method of light field based on per-view aberration correction, the multi-view aberration loss is used to quantify the difference between the estimated light field image obtained by forward projection and superimposing the aberration of the corresponding viewpoint calculated by the current per-view aberration prediction module on the output of the light field 3D reconstruction network and the actual acquired original light field image.

[0013] Preferably, in the light field self-supervised 3D reconstruction method based on per-view aberration correction, the multi-view aberration loss includes the aberration loss of each view. The average value for the viewpoint aberration loss The combined error of image differences and structural differences is calculated as follows:

[0014] in, To estimate the angle of view of the light field The image, This is the corresponding viewpoint image of the original light field image; Let the mean squared error loss function be used. Used to characterize the image differences between the two; The structural similarity function takes into account the brightness, contrast, and structural information of an image. Used to characterize the structural differences between the two; These are the weighting coefficients.

[0015] Preferably, the light field self-supervised 3D reconstruction method based on per-view aberration correction further includes a spatial consistency constraint loss in its multi-view aberration loss, which is the sum of the coordinate differences of a specific 3D point projected from two different viewpoints. The calculation method is as follows:

[0016] in, For a specific three-dimensional point, such as the center of Gauss, For point Projected to viewpoint Image coordinates, For point Projected to viewpoint Image coordinates.

[0017] Preferably, in the self-supervised 3D reconstruction method of the light field based on viewpoint aberration correction, the estimated volume loss is used to quantify the physical reality of the 3D estimated volume output by the light field 3D reconstruction network, and is a comprehensive loss of Hessian loss and the difference between the absolute and average values ​​of the signal; the 3D estimated volume is a 3D volume generated by a 3D reconstruction algorithm; the estimated volume loss... The calculation method is as follows:

[0018] in, This is a three-dimensional estimation volume of the sample. Hessian loss is used to control signal continuity. To calculate the average of the absolute values ​​of the signal, used to limit signal sparsity. , These are weighting coefficients used to balance the contributions of different loss terms.

[0019] According to another aspect of the invention, a system is provided for an electronic device or a non-transitory computer-readable storage medium; The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of the light field self-supervised three-dimensional reconstruction method based on view-by-view aberration correction provided by the present invention. The non-transitory computer-readable storage medium stores a computer program thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the light field self-supervised three-dimensional reconstruction method based on viewpoint aberration correction provided by the present invention.

[0020] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: The self-supervised 3D reconstruction method of light field based on viewpoint aberration correction provided by this invention, without the need for training data, uses a light field image acquired once and employs a multi-view aberration-estimation volume joint loss function to train a viewpoint aberration prediction module. This module independently infers the aberration between the perfect light field image of the sample and the original light field image at each viewpoint, thereby performing viewpoint aberration correction on the 2D light field image. Based on the corrected 2D light field image, 3D reconstruction is performed, avoiding obvious artifacts during 3D reconstruction due to the inconsistency of aberrations at different viewpoints. This effectively improves axial resolution while reducing artifacts, achieving good 3D reconstruction results of light field images. Attached Figure Description

[0021] Figure 1 This is a flowchart of the self-supervised 3D reconstruction method for light field based on viewpoint aberration correction provided by the present invention; Figure 2 These are the original light field images captured by the camera in this embodiment of the invention, with a total of 9 viewpoints; Figure 3 This is a schematic diagram of the INR optical field three-dimensional reconstruction network structure used in an embodiment of the present invention; Figure 4 This is a schematic diagram of the generation process of the aberration point spread function in an embodiment of the present invention; Figure 5 This embodiment of the invention generates a two-dimensional estimated light field image, showing the xy plane and the xz plane respectively; wherein, Figure 5 a is a two-dimensional light field image recovered by traditional RL deconvolution 3D reconstruction technology; Figure 5 b is a two-dimensional light field image reconstructed using the INR reconstruction algorithm without aberration correction and restoration; Figure 5 c is a two-dimensional light field image reconstructed using the INR reconstruction algorithm and the aberration correction algorithm.

[0022] Figure 6 The three-dimensional volume obtained by the three-dimensional reconstruction method of the present invention and the three-dimensional volume generated by other methods are respectively... Figure 6 'a' represents the 3D volume reconstructed using traditional fully convolutional Regression (RL) techniques. Figure 6 b represents a three-dimensional volume reconstructed using the INR reconstruction algorithm without aberration correction. Figure 6 c represents the three-dimensional volume recovered using the view-by-view aberration-corrected light field self-supervised three-dimensional reconstruction method provided in this embodiment. Compared with the previous two techniques, the three-dimensional volume has higher resolution, fewer artifacts, and a higher signal-to-noise ratio. Figure 7 This compares the restoration performance of self-supervised 3D optical field reconstruction using a multi-view aberration-estimation volume joint loss function versus using only a multi-view aberration loss function. Figure 7 A 3D volume reconstructed using only the multi-view aberration loss function; Figure 7 b represents the 3D volume reconstructed using the multi-view aberration-estimation volume joint loss function. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0024] The self-supervised 3D reconstruction method based on viewpoint aberration correction provided by this invention, such as... Figure 1 As shown, it includes the following steps: (1) Obtain the original light field image formed by the imaging system and the three-dimensional reconstruction network of the light field, and construct a per-view aberration prediction module; The light field 3D reconstruction network is used to reconstruct the input light field image into an aberration-free sample 3D image; The view-by-view aberration prediction module is used to independently infer the aberration between the perfect light field image and the original light field image of the sample under each view for the current imaging conditions, that is, to estimate the aberration. The current imaging conditions include the sample to be imaged and the imaging system. Aberrations originate from wavefront distortion, including optical path distortion due to inhomogeneity of the sample's refractive index and the difference between the actual optical characteristics of the imaging system and the optical characteristics reflected by the three-dimensional optical field reconstruction network. The unique aspect of optical field imaging is that the optical field image is not a lens image with uniform optical characteristics, but rather an image acquired from various viewpoints by a microlens array. The optical characteristics of the optical field image at each viewpoint are independent, thus requiring independent reasoning for aberration estimation at each viewpoint.

[0025] However, the light field images acquired by the imaging system from various viewpoints include not only the independent aberrations of each lens introduced by the microlens array, but also aberrations introduced by the non-uniform refractive index of the sample. The aberrations introduced by the sample are independent of each other for the microlens array. If the same set of model parameters is used to optimize and infer the aberrations from different viewpoints, the differences in aberrations introduced by each viewpoint (angle) are not reflected, leading to distortion or artifacts in the corrected viewpoint image when reconstructing the three-dimensional image of the sample. Therefore, the viewpoint-by-view aberration prediction module of this invention is used to infer the aberration vectors of each viewpoint and to perform joint optimization of the sample aberration system aberrations on the aberration vector parameters of each viewpoint, as detailed below: The view-by-view aberration prediction module has parameter matrix ,in Let Zernike's polynomials be the total number of terms. Let be the number of viewpoints. Aberrations can be expressed as a linear combination of Zernike terms, denoted as:

[0026] in, From the perspective The estimated aberration, From the perspective Zenik Township The weight, The Zernikal polynomial is used to select the first four Zernikal aberration types, including defocus, astigmatism, coma, and spherical aberration, which are the main aberrations and account for more than 90% of the aberrations in the actual system. Translation, lateral tilt, and longitudinal tilt, three fundamental aberrations that do not require algorithmic correction, are excluded and corrected optically. Fifth-order and higher Zernikal aberration types have less impact on image quality, but significantly increase computational complexity, making it difficult for the parameter matrix optimization process to converge; therefore, they need to be excluded.

[0027] The per-view aberration prediction module is constructed as follows: the parameter matrix of the per-view aberration prediction module and the parameters of the light field 3D reconstruction network are jointly optimized until convergence with the minimum of the multi-view aberration-estimation volume joint loss function as the optimization objective; the multi-view aberration-estimation volume joint loss function consists of multi-view aberration loss and 3D volume loss, and is used to quantify the difference between the estimated light field image after aberration is superimposed on the sample 3D image output by the light field 3D reconstruction network and the actual acquired original light field image, as well as the comprehensive loss of the physical authenticity of the 3D estimation volume.

[0028] The multi-view aberration loss is used to quantify the difference between the estimated light field view image obtained by forward projection of the output of the light field 3D reconstruction network and the aberration of the corresponding view calculated by the current view-by-view aberration prediction module, and the actually acquired original light field image. Specifically, the estimated light field view image is obtained by forward projection of the 3D volume output by the light field 3D reconstruction network using a point spread function containing aberrations at the corresponding view to obtain the estimated light field view image; the point spread function containing aberrations... An ideal point spread function based on wave optics The inverse Fourier transform is performed after adding the estimated aberrations for the corresponding viewpoints calculated by the current viewpoint aberration prediction module.

[0029] in, For the ideal point spread function, The view calculated by the current view-by-view aberration prediction module The estimated aberration, The imaginary unit, The fluorescence wavelength, For Fourier transform operations, This is the inverse Fourier transform operation.

[0030] The multi-view aberration loss includes aberration losses from each viewpoint. The average value for the viewpoint aberration loss The combined error of image differences and structural differences is calculated as follows:

[0031] in, To estimate the angle of view of the light field The image, This is the corresponding viewpoint image of the original light field image; Let the mean squared error loss function be used. Used to characterize the image differences between the two; The structural similarity function takes into account the brightness, contrast, and structural information of an image. Used to characterize the structural differences between the two; These are the weighting coefficients.

[0032] In a preferred embodiment, the multi-view aberration loss further includes spatial consistency constraint loss, which is the sum of the coordinate differences of a specific 3D point projected from two different viewpoints. The calculation method is as follows:

[0033] in, For a specific three-dimensional point, such as the center of Gauss, For point Projected to viewpoint Image coordinates, For point Projected to viewpoint The image coordinates are considered. When multi-view aberration loss only considers the aberration loss of each viewpoint, the optimization process is prone to getting trapped in its own local optima, resulting in slow convergence. Introducing spatial consistency constraint loss reflects the image correlation between different viewpoints. For a microlens array, the orientation distribution of a point reconstructed from all microlens sampled images should be physically consistent; therefore, spatial consistency constraints exist between images from different viewpoints. There is a linear displacement relationship between the position of a point seen from one angle and the position of that point seen from another angle. By constraining the images from different viewpoints, the convergence speed of the self-supervised 3D reconstruction network can be accelerated.

[0034] The volume estimation loss is used to quantify the physical realism of the 3D estimated volume output by the light field 3D reconstruction network. It is a combined loss of Hessian loss and the difference between the absolute and average values ​​of the signal. The 3D estimated volume is a 3D volume generated using a 3D reconstruction algorithm. The introduction of the volume estimation loss ensures the correlation between different viewpoints, avoiding local optima caused by independent optimization of each viewpoint, which leads to image artifacts. The calculation method is as follows:

[0035] in, This is a three-dimensional estimation volume of the sample. Hessian loss is used to control signal continuity. To calculate the average of the absolute values ​​of the signal, used to limit signal sparsity. , These are weighting coefficients used to balance the contributions of different loss terms.

[0036] The light field 3D reconstruction network is preferably an implicit neural representation model (INR reconstruction network). Implicit neural networks can implicitly represent 3D shapes, such as representing surfaces as placeholder functions or signed distance functions, effectively capturing complex geometric details and generating high-resolution object shapes. Furthermore, by employing implicit neural representation, the 3D scene is treated as a continuous function. This approach differs from traditional methods based on discrete meshes or point clouds, eliminating the need for scene discretization and avoiding information loss and resolution limitations caused by discretization.

[0037] Compared to deconvolutional 3D reconstruction networks employing supervised learning, implicit neural networks utilize self-supervised learning. Learning and inference are based on a single set of samples, ensuring a uniform source of aberrations, thus resulting in better aberration correction. Supervised learning methods, on the other hand, suffer from sample-introduced aberration differences between the learning and inference samples, leading to reduced aberration estimation accuracy and lower accuracy in the reconstructed 3D volume. Furthermore, traditional wave optics algorithms cannot address the low axial resolution issue. By leveraging the continuity of implicit neural network (INR) and the geometric consistency of the synthesized view, the optimized scheme significantly reduces the impact of angular undersampling and cone defects, achieving higher axial resolution and fewer artifacts. This method requires only the original 2D light field image as input, without the need for training data, to obtain 3D reconstruction results with higher resolution and fidelity than traditional deconvolutional algorithms. More importantly, it eliminates the need for complex hardware adaptive optics to adaptively eliminate aberrations in each light field viewpoint image.

[0038] The preferred approach is to use the Adam optimizer to iteratively optimize the parameters of the light field 3D reconstruction network and the view-by-view aberration prediction module until convergence.

[0039] As the number of iterations increases, the light field 3D reconstruction network generates a higher fidelity 3D structure, while the aberration estimation network optimizes its aberration predictions based on the input original light field image.

[0040] (2) The estimated aberration of the light field image of each view is inferred by the aberration prediction module constructed in step (1), and the aberration correction image of each view is obtained by performing aberration correction based on the image of each view in the original light field image and the estimated aberration of the image of that view. (3) Combine the aberration-corrected images of each viewpoint obtained in step (2) into an aberration-corrected light field image, and use the light field three-dimensional reconstruction network obtained in step (1) to reconstruct the aberration-corrected sample three-dimensional image.

[0041] The following is an example: This embodiment uses a near-infrared Fourier light field microscopy system as an example. The light field self-supervised 3D reconstruction method based on view-by-view aberration correction provided in this embodiment is as follows: Figure 1 As shown, it includes the following steps: (1) Obtain the original light field image formed by the imaging system and the three-dimensional reconstruction network of the light field, and construct a per-view aberration prediction module; To acquire the raw light field image generated by the imaging system, specifically: using a living tissue sample as the target, a two-dimensional Fourier light field image is acquired in a single exposure; if the depth of field of the three-dimensional sample acquired in a single exposure is too large, it will result in a large constructed three-dimensional space, affecting the convergence speed. The camera acquires the two-dimensional Fourier light field image, such as... Figure 2 As shown: The light field 3D reconstruction network is used to reconstruct an aberration-free sample 3D image from an input light field image. This embodiment employs a hidden neural network-based light field 3D reconstruction network design as follows: A virtual INR 3D space is constructed using a Fourier coding scheme, where the point density is determined by the system's 3D resolution. This virtual INR 3D space is then input into the INR reconstruction network. After passing through a deep learning network, the resulting parameterized representation of the sample's 3D structure is used as a component and for joint optimization in the subsequent main-view aberration prediction module. Figure 3 As shown.

[0042] The method for constructing the INR three-dimensional space is as follows: Before inputting the 3D coordinates into the network, a positional encoding function is used to map them into a higher-dimensional space. The initial 3D space contains only a series of points, and the density of the points is determined by the system resolution.

[0043] in, Represents the horizontal coordinate , And represents the axial coordinate. Parameter and These correspond to the encoding orders for the horizontal and axial directions, respectively. After positional encoding, a high-dimensional feature vector is generated, denoted as... ,in Indicates the number of input coordinates. Indicates by and The number of channels for the determined encoded features. Vector Each element in It can be represented as follows:

[0044] Direct coordinate encoding can lead to poor learning performance, especially when coordinate values ​​are large. Neural networks struggle to learn mapping relationships effectively, thus affecting the reconstruction of high-frequency information. When samples are centrosymmetric or approximately spherical objects, Fourier radial encoding can save computational resources and simplify spatial structure. However, radial encoding focuses on capturing radial distances and ignores directional information at points in space, which is not accurate enough for applications requiring complete spatial description (such as complex 3D reconstruction).

[0045] The method for constructing the INR reconstruction network is as follows: Multiple convolutions with residual connections of the same size 128 convolutional modules are performed to obtain the initial 3D structural information. The first and last layers are linear convolutional layers, and the middle 8 layers are hybrid convolutional layers of linear convolutional layers and modified linear units.

[0046] A multilayer perceptron (MLP) structure with a modified linear unit (ReLU) activation function is employed. First, the 3D coordinates are flattened into 1D vectors, transforming the spatial structure into a format suitable for the MLP architecture. Next, these 3D coordinates are Fourier-encoded, and then the encoded coordinates are input into the INR reconstruction network. The INR reconstruction network predicts the intensity value at each coordinate. This process can be represented as follows:

[0047]

[0048] in,i An index representing a specific voxel. This represents the corresponding grayscale value. A modified linear unit activation function is used to truncate all predicted values ​​less than zero. After the network completes the inference process, the predicted voxel values ​​are readjusted into a 3D estimated volume. ).

[0049] The view-by-view aberration prediction module is used to independently infer the aberration between the perfect light field image and the original light field image of the sample under each view for the current imaging conditions, that is, to estimate the aberration. The view-by-view aberration prediction module has parameter matrix ,in Let Zernike's polynomials be the total number of terms. Let be the number of viewpoints. Aberrations can be expressed as a linear combination of Zernike terms, denoted as:

[0050] in, From the perspective The estimated aberration, From the perspective Zenik Township The weight, The Zernikal polynomial is used to select the first four Zernikal aberration types, including defocus, astigmatism, coma, and spherical aberration, which are the main aberrations and account for more than 90% of the aberrations in the actual system. Translation, lateral tilt, and longitudinal tilt, three fundamental aberrations that do not require algorithmic correction, are excluded and corrected optically. Fifth-order and higher Zernikal aberration types have less impact on image quality, but significantly increase computational complexity, making it difficult for the parameter matrix optimization process to converge; therefore, they need to be excluded.

[0051] In this embodiment, the parameter matrix of the view-by-view aberration prediction module consists of 12*9 variable parameters, where 9 represents the number of views, i.e., the number of lenses in the microlens array. There are 12 Zernike aberration types for each view, specifically items 4 to 15 in the American National Standards Institute (ANSI) standard. By linearly combining Zernike terms with different weights, aberrations of different degrees (sizes) can be represented.

[0052] For each viewpoint, according to Zernike aberration weight The Zernike aberrations are superimposed to generate estimated aberrations. Since the aberrations of the light field image at different viewpoints and times are independent of each other, an independent aberration image is built for each viewpoint.

[0053] The per-view aberration prediction module is constructed as follows: the parameter matrix of the per-view aberration prediction module and the parameters of the light field 3D reconstruction network are jointly optimized until convergence with the minimum of the multi-view aberration-estimation volume joint loss function as the optimization objective; the multi-view aberration-estimation volume joint loss function consists of multi-view aberration loss and 3D volume loss, and is used to quantify the comprehensive loss of the difference between the estimated light field image after aberration is superimposed on the sample 3D image output by the light field 3D reconstruction network and the actual acquired original light field image, as well as the physical authenticity of the 3D estimation volume.

[0054] Specifically as follows: The initialization of the view-by-view aberration prediction module has parameter matrix And the parameters of the three-dimensional reconstruction network of the light field; After obtaining the estimated aberrations, incorporating them into the optical field point spread function (PSF) is a crucial issue. In this embodiment, the ideal PSF based on wave optics is directly input, and aberrations are added to the PSF for each viewpoint in the spatial frequency domain. Then, an inverse Fourier transform is performed on the resulting aberration-inclusive PSF to obtain the final aberration-inclusive PSF. This method not only saves unnecessary computational resources, but also improves the PSF (Power Frame Rate) for each viewpoint. Aberrations are added independently. The ideal point spread function (PSF) can be replaced with the ideal point spread function of the hybrid model of wave optics and geometric optics.

[0055] The 3D structure output by the light field 3D reconstruction network is processed using a point spread function containing aberrations. A forward projection is performed to generate an estimated two-dimensional light field image, wherein the aberration-containing point spread function... An ideal point spread function based on wave optics The inverse Fourier transform is performed after adding the estimated aberrations for the corresponding viewpoints calculated by the current viewpoint aberration prediction module.

[0056] in, For the ideal point spread function, The view calculated by the current view-by-view aberration prediction module The estimated aberration, The imaginary unit, The fluorescence wavelength, For Fourier transform operations, This is the inverse Fourier transform operation. Ideal point spread function. The point spread function with aberrations is generated based on parameters such as the system's numerical aperture, fluorescence wavelength, magnification, and resolution. The generation process is as follows: Figure 4The total aberration shown includes aberrations from 9 different viewpoints in the image, and each viewpoint aberration consists of 12 Zernike terms with different weights.

[0057] The estimated light field image is compared with the original light field image captured by the camera to calculate the loss function, which is used to constrain the output of the light field 3D reconstruction network and the output of aberrations.

[0058] The multi-view aberration loss is the aberration loss of each viewpoint. The average value and spatial consistency constraint loss The weighted sum.

[0059] For perspective aberration loss The combined error of image differences and structural differences is calculated as follows:

[0060] in, To estimate the angle of view of the light field The image, This is the corresponding viewpoint image of the original light field image; Let the mean squared error loss function be used. Used to characterize the image differences between the two; The structural similarity function takes into account the brightness, contrast, and structural information of an image. Used to characterize the structural differences between the two; These are the weighting coefficients.

[0061] The multi-view aberration loss also includes spatial consistency constraint loss, which is the sum of the coordinate differences of a specific 3D point projected from two different viewpoints. The calculation method is as follows:

[0062] in, For a specific three-dimensional point, such as the center of Gauss, For point Projected to viewpoint Image coordinates, For point Projected to viewpoint Image coordinates.

[0063] The estimated volume loss is used to quantify the physical realism of the 3D estimated volume output by the optical field 3D reconstruction network, and is a combined loss of Hessian loss and the difference between the absolute and average values ​​of the signal; the 3D estimated volume is a 3D volume generated using a 3D reconstruction algorithm. The estimated volume loss... The calculation method is as follows:

[0064] in, This is a three-dimensional estimation volume of the sample. Hessian loss is used to control signal continuity. To calculate the average of the absolute values ​​of the signal, used to limit signal sparsity. , These are weighting coefficients used to balance the contributions of different loss terms.

[0065] In this embodiment, the Adam optimizer uses the gradient obtained from the loss function to jointly update the parameters of the sequential INR reconstruction network and the view-by-view aberration prediction module, thus constituting one iteration of the network.

[0066] As the number of iterations increases, the INR reconstruction network generates higher-fidelity 3D structures, while the aberration estimation network optimizes its aberration predictions based on the input original light field image.

[0067] Through iterative offline processing, the hybrid loss function of the INR reconstruction module and the view-by-view aberration prediction module gradually converges.

[0068] (2) The estimated aberration of the light field image of each view is inferred by the aberration prediction module constructed in step (1), and the aberration correction image of each view is obtained by performing aberration correction based on the image of each view in the original light field image and the estimated aberration of the image of that view. The 3D estimated volume data generated by the INR reconstruction network is then projected forward. This involves layer-by-layer convolution of the 3D estimated volume data with the estimated point spread function along the depth direction, followed by summation projection to generate a 2D estimated light field image, such as... Figure 5 As shown. Figure 5 Image 'a' is a two-dimensional light field image recovered using traditional RL deconvolution 3D reconstruction techniques, showing the xy plane and xz plane respectively. Figure 5 The two-dimensional light field image recovered by the traditional RL deconvolution 3D reconstruction technique is shown in the horizontal plane and the axial plane respectively. Figure 5 b is a two-dimensional light field image that uses only the INR reconstruction algorithm without aberration correction and restoration. Figure 5 c is a two-dimensional light field image reconstructed using the INR reconstruction algorithm and the aberration correction algorithm.

[0069] The comparison showed that the reconstructed blood vessels after aberration correction had fewer artifacts in the lateral and axial planes and higher resolution.

[0070] (3) Combine the aberration-corrected images of each viewpoint obtained in step (2) into an aberration-corrected light field image, and use the light field three-dimensional reconstruction network obtained in step (1) to reconstruct the aberration-corrected sample three-dimensional image.

[0071] The INR reconstruction module outputs a final estimated 3D volume, the view-by-view aberration prediction module outputs the aberrations for each view, and the aberration-corrected PSF module estimates the phase for each view, achieving accurate reconstruction during live-body 3D imaging. Figure 6 As shown. Figure 6 'a' represents the 3D volume reconstructed using traditional fully convolutional Regression (RL) techniques. Figure 6 b represents a three-dimensional volume reconstructed using the INR reconstruction algorithm without aberration correction. Figure 6 c represents the three-dimensional volume recovered using the view-by-view aberration-corrected light field self-supervised three-dimensional reconstruction method provided in this embodiment. Compared with the previous two techniques, the three-dimensional volume has higher resolution, fewer artifacts, and a higher signal-to-noise ratio.

[0072] By introducing volume loss estimation, the physical continuity of 3D reconstruction is optimized, thereby suppressing artifacts and jump point phenomena commonly seen in 3D imaging. Figure 7 'a' represents the reconstruction result when trained using only the multi-view aberration loss function. Figure 7 b is the image reconstructed with the addition of the 3D volume estimation loss function. After comparison, adding the volume estimation loss function can significantly improve the 3D imaging effect, ensuring signal continuity while suppressing artifacts and jump points.

[0073] The self-supervised 3D reconstruction method based on main viewpoint aberration correction provided in this embodiment minimizes artifacts after reconstructing a large-volume sample (550 μm in diameter, 200 μm in thickness) using only nine viewpoints. This method requires only the original 2D light field image as input, without the need for training data, to obtain 3D reconstruction results with higher resolution and fidelity than traditional deconvolution algorithms. More importantly, it eliminates the need for complex hardware adaptive optics to adaptively and independently eliminate aberrations in each light field viewpoint image.

[0074] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A self-supervised 3D reconstruction method for light field based on per-view aberration correction, characterized in that, Includes the following steps: (1) Obtain the original light field image formed by the imaging system and the three-dimensional reconstruction network of the light field, and construct a per-view aberration prediction module; The view-by-view aberration prediction module is used to independently infer the aberration between the perfect light field image and the original light field image of the sample under each view for the current imaging conditions, that is, to estimate the aberration; the view-by-view aberration prediction module has an n×v parameter matrix {α_v^n}, where n is the total number of terms in the Zernike polynomial, v is the number of view, and the aberration is a linear combination of Zernike terms. The per-view aberration prediction module is constructed as follows: the parameter matrix of the per-view aberration prediction module and the parameters of the light field 3D reconstruction network are jointly optimized until convergence with the minimum of the multi-view aberration-estimation volume joint loss function as the optimization objective; the multi-view aberration-estimation volume joint loss function consists of multi-view aberration loss and 3D volume loss, and is used to quantify the difference between the estimated light field image after aberration is superimposed on the sample 3D image output by the light field 3D reconstruction network and the actual acquired original light field image, as well as the comprehensive loss of the physical authenticity of the 3D estimation volume; (2) The estimated aberration of the light field image of each view is inferred by the aberration prediction module constructed in step (1), and the aberration correction image of each view is obtained by performing aberration correction based on the image of each view in the original light field image and the estimated aberration of the image of that view. (3) Combine the aberration-corrected images of each viewpoint obtained in step (2) into an aberration-corrected light field image, and use the light field three-dimensional reconstruction network obtained in step (1) to reconstruct the aberration-corrected sample three-dimensional image.

2. The self-supervised 3D reconstruction method based on per-view aberration correction of light field as described in claim 1, characterized in that, The light field 3D reconstruction network is used to reconstruct the input light field image into an aberration-free sample 3D image, preferably an implicit neural representation model.

3. The self-supervised 3D reconstruction method based on per-view aberration correction of light field as described in claim 1, characterized in that, The aberration is denoted as: in, From the perspective The estimated aberration, From the perspective Zenik Township The weight, That is, Zernike polynomials.

4. The self-supervised 3D reconstruction method based on per-view aberration correction of light field as described in claim 1, characterized in that, The estimated light field view image is obtained by the following method: the three-dimensional volume output by the light field three-dimensional reconstruction network is projected forward from the corresponding view using a point spread function with aberrations to obtain the estimated light field view image.

5. The light field self-supervised 3D reconstruction method based on per-view aberration correction as described in claim 4, characterized in that, The aberration-containing point spread function An ideal point spread function based on wave optics The inverse Fourier transform is performed after adding the estimated aberrations for the corresponding viewpoints calculated by the current viewpoint aberration prediction module. in, For the ideal point spread function, The view calculated by the current view-by-view aberration prediction module The estimated aberration, The imaginary unit, The fluorescence wavelength, For Fourier transform operations, This is the inverse Fourier transform operation.

6. The self-supervised 3D reconstruction method based on per-view aberration correction of light field as described in claim 1, characterized in that, The multi-view aberration loss is used to quantify the difference between the estimated light field image obtained by forward projection of the output of the light field 3D reconstruction network and the aberration of the corresponding view calculated by the current view-by-view aberration prediction module, and the actual acquired original light field image.

7. The self-supervised 3D reconstruction method based on per-view aberration correction of light field as described in claim 1, characterized in that, The multi-view aberration loss is used to quantify the difference between the estimated light field image obtained by forward projection and superimposing the aberrations of the corresponding viewpoints calculated by the current viewpoint aberration prediction module on the output of the light field 3D reconstruction network, and the actual acquired original light field image: in, To estimate the angle of view of the light field The image, This is the corresponding viewpoint image of the original light field image; Let the mean squared error loss function be used. Used to characterize the image differences between the two; The structural similarity function takes into account the brightness, contrast, and structural information of an image. Used to characterize the structural differences between the two; These are the weighting coefficients.

8. The self-supervised 3D reconstruction method based on per-view aberration correction of light field as described in claim 7, characterized in that, The multi-view aberration loss also includes spatial consistency constraint loss, which is the sum of the coordinate differences of a specific 3D point projected from two different viewpoints. The calculation method is as follows: in, For a specific three-dimensional point, such as the center of Gauss, For point Projected to viewpoint Image coordinates, For point Projected to viewpoint The image coordinates.

9. The self-supervised 3D reconstruction method based on per-view aberration correction of light field as described in claim 1, characterized in that, The estimated volume loss is used to quantify the physical realism of the 3D estimated volume output by the optical field 3D reconstruction network, and is a combined loss of Hessian loss and the difference between the absolute and average values ​​of the signal; the 3D estimated volume is a 3D volume generated using a 3D reconstruction algorithm; the estimated volume loss... The calculation method is as follows: in, This is a three-dimensional estimation volume of the sample. Hessian loss is used to control signal continuity. To calculate the average of the absolute values ​​of the signal, used to limit signal sparsity. , These are weighting coefficients used to balance the contributions of different loss terms.

10. A system, characterized in that, For electronic devices or non-transitory computer-readable storage media; The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of the light field self-supervised three-dimensional reconstruction method based on view-by-view aberration correction as described in any one of claims 1 to 9. The non-transitory computer-readable storage medium, on which a computer program is stored, is characterized in that, when the computer program is executed by a processor, it implements the steps of the light field self-supervised three-dimensional reconstruction method based on per-view aberration correction as described in any one of claims 1 to 9.

Images