Cross-modal image reconstruction method and system based on wide field to total internal reflection fluorescence and medium

Abstract

The invention relates to the technical field of image reconstruction, and discloses a cross-modal image reconstruction method and system based on wide field to total internal reflection fluorescence, and a medium. The method comprises the following steps: acquiring an EPI image and a corresponding TIRF image which are registered in the same field of view, and constructing a training data set; a convolutional neural network model with double-branch output is constructed, the wide-field fluorescence image is used as input, features are extracted through a convolutional layer, and then a foreground image and a background image with defocused backgrounds removed are reconstructed and output; carrying out convolution on the foreground image and a predefined point spread function to generate a virtual TIRF image; training the convolutional neural network model to obtain a wide-field fluorescence image reconstruction model; and performing EPI image reconstruction by using the wide-field fluorescence image reconstruction model. According to the method, the optical slice effect and the axial super-resolution capability which are close to those of a TIRF microscope can be obtained only by depending on a single EPI image for the first time, the problems of axial blurring and background of a wide-field image are solved, and the signal-to-noise ratio and the contrast ratio of the image are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a cross-modal fluorescence microscopic image reconstruction technology, and more particularly to a cross-modal image reconstruction method, system and medium based on wide-field to total internal reflection fluorescence. BACKGROUND

[0002] Fluorescence microscopy is an important tool for life science research. Wide-field fluorescence microscope (Wide-field Fluorescence or Epifluorescence, hereinafter referred to as EPI) is widely used due to its fast imaging speed and wide application range. However, wide-field microscopic imaging has inherent deficiencies: since the illumination light illuminates the entire thickness of the sample, the fluorescence from outside the focal plane will be superimposed on the image, forming a strong out-of-focus background. This will result in poor axial resolution of the obtained wide-field image, and details are easily obscured. For example, in thick tissue or high-density labeled cell samples, the out-of-focus background will reduce the image contrast and signal-to-noise ratio, affecting the observation and reconstruction of three-dimensional structures.

[0003] In order to obtain higher axial resolution and remove out-of-focus background, various optical sectioning microscopic techniques have been used in the prior art to achieve this purpose, for example, confocal microscope suppresses out-of-focus light through diaphragm, two-photon microscope uses nonlinear excitation to generate signals only at the focal point, and light sheet microscope illuminates the sample with a transverse thin sheet to avoid exciting out-of-focus fluorescence. These methods can reduce the background to some extent and improve the axial resolution, but often accompanied by a decrease in imaging speed (such as long time-consuming point-by-point scanning), an increase in system complexity (the need to increase special optical paths or elements), and certain limitations on the sample (such as light sheet microscopy requiring high sample transparency).

[0004] Total internal reflection fluorescence microscope (TIRF) is a classic near-field imaging technique that generates an evanescent field with a penetration depth of about 50-200 nm at the interface, only exciting a thin layer of fluorescence attached to the surface of the glass slide, thus almost completely eliminating out-of-focus background and having extremely high axial resolution. TIRF is very useful in single-molecule imaging and membrane structure research, but can only observe the adherent region of the sample and cannot obtain information from the deep part of the sample.

[0005] Therefore, from the perspective of hardware means of the microscope, there are limitations and costs in achieving optical sectioning or improving axial resolution.

[0006] Parallel to the development of hardware, image processing algorithms have also been developed. Traditional image restoration algorithms, such as deconvolution, attempt to use the point spread function (PSF) to improve resolution, but due to the ill-posed problem of inverse convolution, the algorithm is sensitive to noise and difficult to completely distinguish the foreground signal from the out-of-focus background. Some improved methods propose solutions for background subtraction, such as the Dark Sectioning method, which reduces the background effect by dark exposure in the out-of-focus area; and the wavelet-based method can filter out the background at multiple scales. However, these algorithms are limited by the set parameters, imaging conditions, and have limited adaptability to complex samples, and the performance is also limited.

[0007] In addition, in the axial super-resolution direction, deep learning techniques have been widely used in recent years to improve performance: one class is supervised learning, which uses high-resolution images (or simulated high-resolution images) as true values to train the model, and predicts high-resolution output from low-resolution input. However, due to the difficulty of obtaining real high-resolution images, computer-simulated data is often used for training, which results in poor performance when the model is applied to real data due to the inaccuracy of simulation. Another class is self-supervised / weakly supervised learning, which does not require high-resolution true values, such as using the redundant information within the obtained multi-focal plane stack images to train the model, achieving isotropic reconstruction. Although this method avoids the dependence on real high-resolution labels, it requires sufficient Z-layer slices and low background, otherwise the effect is not ideal, which is difficult to achieve in itself. SUMMARY

[0008] In view of the defects and deficiencies of the prior art, in order to solve the problem that high axial resolution images cannot be obtained from a single wide-field image, the present application aims to provide a wide-field to total internal reflection fluorescence cross-modal image reconstruction method, which solves the axial blur and background problem of wide-field images, and realizes image reconstruction from a single wide-field image to a high axial resolution wide-field image. Without hardware modification, a clear image close to the effect of total internal reflection fluorescence image (TIRF image) is reconstructed from a single wide-field fluorescence image, greatly improving the axial resolution and image quality of the wide-field fluorescence image.

[0009] According to the first aspect of the object of the present application, a wide-field to total internal reflection fluorescence cross-modal image reconstruction method is provided, comprising the following processes: Obtaining a registered wide-field fluorescence image and a corresponding TIRF image under the same field of view as a set of image pairs to construct a training data set; A convolutional neural network model with double-branch output is constructed, taking wide-field fluorescence images as input, and after feature extraction by a convolutional layer, the output is reconstructed: the first branch is a foreground reconstruction branch, which is used to output the foreground image after removing the out-of-focus background; the second branch is a background branch, which is used to output the background image; the foreground reconstruction branch is connected with a PSF convolution operator, and the reconstructed foreground image is convolved with the predefined point spread function to generate a virtual TIRF image; The convolutional neural network model with double-branch output is trained using the training data set, a group of image pairs are input into the convolutional neural network model as a sample, the difference between the wide-field image reconstructed by the virtual TIRF image and the background image and the wide-field fluorescence image in the sample is calculated to obtain a self-supervised loss, the difference between the generated virtual TIRF image and the TIRF image in the sample is calculated to obtain a supervised loss, the self-supervised loss and the supervised loss are combined to construct a loss function, and the convolutional neural network model is iteratively updated and trained until convergence, thereby obtaining a wide-field fluorescence image reconstruction model; and For the input wide-field fluorescence image, the foreground image after removing the out-of-focus background is output by the wide-field fluorescence image reconstruction model.

[0010] According to a second aspect of the present application, a computer system is provided, comprising: one or more processors; and a memory storing instructions executable by the one or more processors; wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform operations comprising executing the process of the aforementioned wide-field to total internal reflection fluorescence cross-modality image reconstruction method.

[0011] According to a third aspect of the present application, a computer-readable storage medium is provided for storing one or more programs, the one or more programs comprising instructions or instruction sets executable by one or more processors; wherein the instructions or instruction sets, when executed by the one or more processors, execute the process of the aforementioned wide-field to total internal reflection fluorescence cross-modality image reconstruction method.

[0012] The wide-field to total internal reflection fluorescence cross-modality image reconstruction method of the above embodiments of the present application aims to realize EPI-TIRF cross-modality image reconstruction based on deep learning, reconstruct a wide-field image with high axial resolution close to the imaging effect of TIRF from a single wide-field fluorescence image, effectively remove the out-of-focus background and clearly present the fine structure, and through the implementation of the method of the present application, without complex and expensive optical system hardware modification, the depth limitation of TIRF imaging can be broken through, and high-definition imaging of deep samples can be realized.

[0013] Compared with existing technologies, the significant advantages of the cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence of the present invention are as follows: 1. To address the problem that existing technologies cannot obtain high axial resolution images from a single wide-field image, this invention proposes a cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence. For the first time, it achieves optical slicing effects and axial super-resolution capabilities close to those of a TIRF microscope by relying solely on a single wide-field fluorescence image. This solves the problems of axial blurring and background in wide-field images. The reconstructed output image has extremely low defocus background and clear axial details, thereby significantly improving the image signal-to-noise ratio and contrast, which is beneficial for subsequent three-dimensional reconstruction and quantitative analysis. 2. The method of the present invention can achieve background separation and axial resolution enhancement of wide-field fluorescence images without complex and expensive optical system hardware modifications, reducing imaging costs and system complexity. In particular, it overcomes the limitation of TIRF illumination that can only image the superficial layer (<200 nm). The embodiments of the method of the present invention can be used for imaging deep cells and tissues, making the method of the present invention more widely applicable in fields such as long-term imaging of live cells and clinical histopathology. 3. The method of this invention proposes a hybrid architecture deep convolutional neural network that integrates physical models. It employs a hybrid architecture guided by physical priors and a dual training strategy combining supervised learning and self-supervised physical modeling. Training with real EPI-TIRF imaging pairs allows the network to learn biological structural priors, while the self-supervised PSF convolution explicitly models the imaging degradation process. This endows the model with excellent generalization ability, achieving good reconstruction results even with changes in microscope objective parameters or different sample types. Furthermore, when applied to new imaging systems, model adaptation (small-sample transfer learning) can be achieved through fine-tuning with only a small number of samples.

[0014] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below may be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other. Furthermore, all combinations of the claimed subject matter are considered part of the inventive subject matter of this disclosure.

[0015] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0016] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings.

[0017] Figure 1 This is a schematic flowchart of a cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence according to an embodiment of the present invention.

[0018] Figure 2 This is a flowchart illustrating a cross-modal image reconstruction method based on the hybrid architecture convolutional neural network model ET2dNet according to an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram showing the image reconstruction effect of the method according to the present invention on a cell sample, and the comparison results with other methods.

[0020] Figure 4 The image reconstruction effect of the method according to the present invention on brain tissue slices, and the schematic diagram comparing the effect with other methods. Detailed Implementation

[0021] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0022] Various aspects of the invention are described in this disclosure with reference to the accompanying drawings, which illustrate numerous illustrative embodiments. The embodiments of this disclosure are not necessarily intended to encompass all aspects of the invention. It should be understood that the various concepts and embodiments described above, as well as those described in more detail below, can be implemented in any of many ways, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0023] {Example 1} Combination Figure 1 The example shown, according to an embodiment of the present invention, includes a cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence, comprising the following steps: Obtain the registered wide-field fluorescence image and the corresponding TIRF image under the same field of view, and construct the training dataset as a set of image pairs; A convolutional neural network model with dual-branch output is constructed. A wide-field fluorescence image is used as input, and features are extracted through convolutional layers before the output is reconstructed. The first branch is the foreground reconstruction branch, which is used to output the foreground image after removing the out-of-focus background. The second branch is the background branch, which is used to output the background image. The foreground reconstruction branch is connected to a PSF convolution operator, which convolves the reconstructed foreground image with a predefined point spread function to generate a virtual TIRF image. The convolutional neural network model with dual-branch output is trained using the training dataset. A pair of images is input as a sample into the convolutional neural network model. A self-supervised loss is calculated based on the difference between the wide-field image reconstructed from the virtual TIRF image and the background image and the wide-field fluorescence image in the sample. A supervised loss is calculated based on the difference between the generated virtual TIRF image and the TIRF image in the sample. A loss function is constructed by combining the self-supervised loss and the supervised loss to guide iterative training and updates of the convolutional neural network model until convergence, resulting in a wide-field fluorescence image reconstruction model. For the input wide-field fluorescence image, the wide-field fluorescence image reconstruction model outputs a foreground image with the defocused background removed.

[0024] Therefore, in the method of this embodiment, firstly, a pair of registered wide-field fluorescence images and TIRF images (as real image data) are acquired as training data samples to construct a training dataset. Then, a deep convolutional neural network model with dual-branch output is constructed, where the first branch outputs a sharp foreground image with out-of-focus background removed (i.e., a simulated "TIRF-like" image), striving to retain or restore the detailed structures within the focal plane in the original image; the second branch outputs the corresponding background image, characterizing all scattering and blur signals from the out-of-focus plane. The dual-branch output design enables the model to decompose the original input image into "sharp part + background part". During the training phase, the sharp foreground image output by the first branch is convolved with a pre-estimated point spread function (PSF) to simulate and generate a virtual TIRF image. The virtual TIRF image and the real TIRF image are compared with a supervised loss (such as L1 / L2 pixel error, structural similarity, etc.). At the same time, the virtual TIRF image and the output background image are added to reconstruct a virtual wide-field image, and a self-supervised loss is calculated with the real wide-field image in the input sample to achieve physical modeling constraints on the imaging degradation process.

[0025] The overall loss during network training consists of the sum of the supervised loss (simulated TIRF vs. actual TIRF) and the self-supervised loss (reconstructed EPI vs. original EPI), plus a regularization term. The regularization term includes total variational (TV) constraints on the output sharp image and background image, as well as sparsity L1 constraints applied to the sharp image. Backpropagation is used to adjust the network parameters, enabling the network to simultaneously learn how to remove defocus noise and preserve / reconstruct foreground details until the model converges and training ends. After training, inputting wide-field fluorescence images of new samples into the trained model will output high-axial-resolution images (and corresponding background images) comparable to TIRF imaging.

[0026] As an optional embodiment, for a set of image pairs consisting of a wide-field fluorescence image registered under the same field of view and a corresponding TIRF image, wherein the TIRF image is imaging data obtained by the imaging system focusing on the cover glass surface of the sample.

[0027] As an optional embodiment, the convolutional neural network model adopts a network model with a U-shaped encoder-decoder U-Net as the skeleton and a self-attention mechanism.

[0028] As an optional embodiment, the predefined point spread function is designed to be generated in advance by simulation software based on the optical parameters of the imaging system sampled from the image pairs; and remains unchanged during training to simulate the point spread process of wide-field imaging.

[0029] As an optional embodiment, the foreground and background images output by the dual branches are reconstructed by linear addition to obtain a wide-field image.

[0030] As an optional embodiment, the loss function further includes a regularization term R, which includes the total variational loss for both the foreground and background images and an L1 norm sparsity constraint on the foreground image. The loss function L constructed in this way is expressed as:

[0031] in, The supervised loss is used to characterize the difference between the generated virtual TIRF image and the TIRF image in the sample; These represent the virtual TIRF image and the TIRF image in the sample, respectively. This is a self-supervised loss used to characterize the difference between the wide-field image reconstructed from the foreground and background images and the wide-field fluorescence image in the sample. These represent the reconstructed wide-field image and the wide-field fluorescence image in the sample, respectively. For regularization terms, These represent the foreground and background images output by the dual branches, respectively. λ and μ are the weight hyperparameters.

[0032] As an optional embodiment, the supervision loss Self-monitoring losses The corresponding difference calculations all include a combination of weighted pixel intensity L1 loss, L2 loss, and multi-scale structural similarity loss.

[0033] As an optional embodiment, for the input wide-field fluorescence image, the wide-field fluorescence image reconstruction model also outputs a corresponding background image.

[0034] {Example 2} In this embodiment, it will be further combined Figure 1 , Figure 2 The examples shown provide a more detailed illustration and explanation of the cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence proposed in this invention.

[0035] (1) Acquisition of training data In this embodiment, real, paired wide-field (EPI) / TIRF images are preferably used as training data. Utilizing real cross-modal data can ensure that the model learns reliable mapping relationships and effectively avoid artifacts caused by insufficient realism of synthetic data.

[0036] As an alternative, an imaging system capable of acquiring both wide-field and TIRF images simultaneously can be used to acquire a wide-field fluorescence image and a corresponding TIRF image pair in the same field of view, ensuring that the two are strictly aligned in the XYZ position (e.g., through mechanical anti-drift design or post-hoc rigid registration correction).

[0037] As one approach, TIRF is highly compatible with wide-field microscopes, allowing for in-situ acquisition using a modified microscope or dual-modal microscopy system. This enables sequential wide-field fluorescence imaging and TIRF imaging of the same sample area on the same equipment. For example, using an inverted fluorescence microscope with a TIRF module allows switching between wide-field illumination and total internal reflection illumination within the same field of view, acquiring image pairs. After each image pair is acquired, registration and correction algorithms are used to eliminate positional biases introduced by mode switching or sample drift.

[0038] Taking F-actin-labeled cell samples as an example, approximately 1000 pairs of images were generated using F-actin-labeled cell samples under both standard EPI and TIRF illumination for training. During acquisition, the TIRF images were focused on the coverslip surface to obtain a clear near-membrane structure. The acquired images underwent standardized preprocessing (including but not limited to background subtraction and histogram stretching) and were standardized in size and format before being input into the network.

[0039] (2) Construction of deep learning neural network models Combination Figure 2 The illustration exemplifies a hybrid architecture convolutional neural network model (referred to as ET2dNet) constructed using this invention. This network employs a U-shaped encoder-decoder U-Net as its backbone and incorporates a self-attention mechanism to balance global feature extraction and computational efficiency.

[0040] like Figure 2 As shown, the network receives single-channel or multi-channel wide-field fluorescence images as input, extracts multi-scale features through several convolutional downsampling layers, and then reconstructs high-resolution output through upsampling.

[0041] In the convolutional neural network model of this embodiment, the network output layer adopts a two-branch design: The first branch outputs a reconstructed, clear foreground image. (The near-focal fluorescence signal of the corresponding sample has been filtered to remove background blur.) The second branch outputs the background image. (Corresponds to all defocused background scattering signals).

[0042] Therefore, the original input wide-field fluorescence image is decomposed into "sharp part + background part".

[0043] In specific examples, such as Figure 2 As shown, the encoder extracts features by downsampling layer by layer, while the decoder reconstructs the image by upsampling layer by layer. A global self-attention module can be designed in the encoder to consider global correlations while maintaining efficient convolution computation, thereby better distinguishing widely distributed out-of-focus backgrounds.

[0044] like Figure 2 As shown, to implement physical constraints, the network connects a PSF convolution operator to the first branch, which is used to convert the foreground image of the reconstructed output. A virtual TIRF image is generated by convolving with a predefined point spread function (PSF). :

[0045] The predefined point spread function (PSF) can be generated using simulation software based on the optical parameters of the imaging system, such as numerical aperture, refractive index, and wavelength, which are consistent with the parameters of the actual microscopic imaging system. For example, it can be designed based on the Born & Wolf model using the PSF Generator plugin of Fiji software to simulate the point spread process of wide-field imaging. This PSF remains unchanged during training.

[0046] In embodiments of the present invention, the specific parameters of each layer of the network, such as the convolution kernel size, number of channels, and self-attention module configuration, can be optimized and determined based on the actual dataset.

[0047] It should be understood that the pixel values ​​of the two branches are linearly added to reconstruct the input, which is the wide-field image reconstructed by the network: It should be close to the original input image x, and is used for self-supervised consistency constraints during training.

[0048] (3) Model training process In an embodiment of the present invention, real EPI-TIRF images are used as training sample data. During training, each sample consists of a wide-field fluorescence image x and a corresponding real TIRF image y. The wide-field fluorescence image x is input into the network to obtain a clear foreground image output by the first branch. Background image of the second branch output and the foreground image Virtual TIRF image is obtained by convolution with PSF. .

[0049] The training process employs a dual training strategy that combines supervised learning with self-supervised physical modeling.

[0050] In this specific example, the loss function employs a hybrid multi-task loss: including supervised reconstruction loss, self-supervised reconstruction loss, and a regularization term. Specifically, the total loss function L is expressed as:

[0051] in, The supervised loss is used to characterize the difference between the generated virtual TIRF image and the TIRF image in the sample, and is used to guide the foreground branch of the model to produce an image as close as possible to the real TIRF. These represent the virtual TIRF image and the TIRF image in the sample, respectively. The self-supervised loss is used to characterize the difference between the wide-field image reconstructed from the foreground and background images and the wide-field fluorescence image in the sample, and is used to ensure that the foreground and background separated by the model can be combined to restore the original image. These represent the reconstructed wide-field image and the wide-field fluorescence image in the sample, respectively. For regularization terms, These represent the foreground and background images output by the dual branches, respectively. λ and μ are the weight hyperparameters. During the training process, the final values ​​of λ and μ can be determined based on the training results.

[0052] As an optional embodiment, the supervision loss Self-monitoring losses The corresponding difference calculations all include a combination of weighted pixel intensity L1 norm loss, L2 norm loss and multi-scale structural similarity loss (SSIM or MS-SSIM). The pixel intensity L1 norm loss and L2 norm loss measure the overall intensity difference, and the multi-scale structural similarity loss is superimposed to improve contrast consistency, so as to take into account both detail error and perceived quality.

[0053] Combining the foregoing expressions, the regularization term R imposes prior constraints on the network output to smooth the image and reduce noise particles. The regularization term includes conditions for sharp foreground images. and background image The total variational (TV) loss was calculated separately to suppress noise, and for sharp foreground images... Apply L1 norm penalty to encourage foreground sparsity (i.e. reduce unstructured scattered light).

[0054] In each training iteration, based on the total loss function mentioned above, the convolutional neural network model is iteratively updated and trained by minimizing the total loss function. The network weight parameters are updated by gradient descent until convergence, thus obtaining a wide-field fluorescence image reconstruction model.

[0055] As an optional approach, during iterative training, the AdamW optimizer can be used in conjunction with strategies such as cosine annealing learning rate scheduling. After training is complete, the parameters of the finally converged network model are saved.

[0056] (4) Model deployment and application After training, the model is deployed on computer systems, servers (local deployment servers or cloud servers), and other devices. It can be used for EPI image reconstruction to obtain image output with high axial resolution and effective reduction of background, resulting in clear images with quality close to TIRF images.

[0057] For any given wide-field fluorescence microscopy image, after inputting it into the model, the reconstruction result (approximately TIRF image effect, containing clear structural information of the sample) and the corresponding background image (the part removed from the original wide-field fluorescence microscopy image) can be directly output.

[0058] If needed, the corresponding background image can be used to assess the overall fluorescence background distribution of the sample.

[0059] Based on the above implementation process, this method, being software-based, enables real-time input and real-time reconstruction output, making it particularly suitable for real-time use with microscopes. For example, in live-cell imaging applications, the imaging device connects to a computer system deployed with the aforementioned trained wide-field fluorescence image reconstruction model. Each acquired wide-field image simultaneously yields a reconstructed image with the processed TIRF effect, allowing for real-time observation of high-resolution structural dynamics. Furthermore, since no hardware modifications are required, the original wide-field image is retained and can be used together with the reconstruction results for further analysis.

[0060] In a further optional embodiment, the aforementioned model training can be performed in stages: In the first stage, a basic network is trained on a large dataset to enable it to have preliminary cross-modal conversion capabilities.

[0061] In the second stage, the model can be further fine-tuned with a small amount of new data for a specific microscopic system / sample, so that the PSF convolution corresponds to the point spread function of the new system, thereby enhancing the accuracy of the model under that condition.

[0062] Therefore, the fine-tuning operation in the second stage is equivalent to small-sample transfer learning, which can be completed in a short time (e.g., several minutes to tens of minutes). Combined with the implementation of the above-mentioned solution, thanks to the existence of the self-supervised physical branch, the fine-tuning process is highly efficient, requiring only a small amount of labeled data. This allows the model to cope with changes in microscope objective parameters or different sample types. When applied to a new imaging system, only a small number of samples are needed to achieve model adaptation through fine-tuning (small-sample transfer learning), demonstrating strong generalization performance and significantly reducing data acquisition costs.

[0063] {Example 3} In this embodiment, the image reconstruction process of cell samples (microfilaments) and brain tissue slices using the method of the present invention is used as an example to further illustrate the effects achieved by the present invention and the comparison with the imaging effects of existing methods.

[0064] like Figure 3 The accompanying figure shows a comparison of the results of cell sample (microfilament) image reconstruction. The methods and their corresponding results listed in the figure are explained below: EPI: Acquired wide-field image (the network's raw image input); Sparse: Sparse deconvolution result; Dark: The result of the Dark Sectioning background removal algorithm; Ours: The virtual TIRF output by the method of this invention (ET2dNet); Ours+deconvolution: The result of deconvolving the virtual TIRF output by this method; TIRF: TIRF images acquired in situ in the same region.

[0065] The bottom image below each method is a magnified image of the area marked by a white dashed box.

[0066] Cells treated with a drug to fluorescently label the cytoskeleton with F-Actin were placed under an inverted microscope, and wide-field and TIRF images were acquired simultaneously. Using only the acquired wide-field image as input, the model trained in this invention (ET2dNet) was applied for reconstruction, outputting TIRF-like images, which were then compared with results from other methods (sparse deconvolution, dark sectioning background removal). Figure 3 As shown, visually, both the Dark Sectioning method and our proposed method can effectively remove the background, achieving contrast similar to TIRF. The area marked by the white dashed box reveals that the originally continuous microfilament structure in the wide-field image disappears in the middle of the reconstructed image, presenting a broken, arched shape, consistent with observations in the actual TIRF image. Further analysis suggests that the middle of this microfilament detaches from the bottom surface of the slide, therefore TIRF cannot excite fluorescence in this area, resulting in a broken appearance.

[0067] Therefore, this invention successfully recovers this axial information from wide-field images, achieving details visible only in TIRF, an effect that neither traditional sparse deconvolution nor dark sectioning can achieve. The results demonstrate the effectiveness and advancement of the method described in this invention in background removal and axial super-resolution. Furthermore, applying deconvolution to the output of this method further enhances lateral sharpness, showcasing compatibility with deconvolution methods.

[0068] like Figure 4 The image shown illustrates the image reconstruction effect of the method of the present invention on brain tissue slices, and the comparison results with other methods.

[0069] The methods and their effects listed in the attached diagram are explained below: EPI: Acquired wide-field image (network input); Sparse: Sparse deconvolution result; Dark: The result of the Dark Sectioning background removal algorithm; Ours: The virtual TIRF output by the method of this invention (ET2dNet); Ours+deconvolution: The result of deconvolving the virtual TIRF output by this method; Confocal: Images acquired using a confocal microscope in the same area.

[0070] The bottom image below each method is a magnified image of the area marked by a white dashed box, with the fluorescence intensity curve along the white dashed line in the lower left corner of each sub-image.

[0071] Frozen human brain tissue sections were selected, and GFAP protein in astrocytes was labeled and subjected to wide-field confocal imaging. The wide-field image was input into the model trained in this invention, and a virtual TIRF image was reconstructed from the model output. Deconvolution was applied to the output of this method, and the results were compared with other methods (Sparse deconvolution, Dark Sectioning background removal).

[0072] Combination Figure 4 As shown in the comparison, the image reconstructed and deconvolved using the method of this invention is very close to the confocal image in terms of signal-to-noise ratio and detail, with cell fiber structures clearly discernible. In contrast, the original wide-field image suffers from poor detail due to background interference. The results indicate that the method of this invention can not only provide high axial resolution images independently but also be compatible with existing enhancement techniques to achieve better overall results.

[0073] {Example 4} In conjunction with the implementation of the cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence in the foregoing embodiments, the present invention also proposes a computer system, including: one or more processors; and a memory storing operable instructions.

[0074] When the aforementioned instructions are executed by the one or more processors, the one or more processors perform operations, including the process of performing the cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence as described in the foregoing embodiments.

[0075] {Example 5} In conjunction with the implementation of the cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence in the foregoing embodiments, the present invention also proposes a computer-readable medium for storing software, the software including instructions executable by one or more computers, the instructions executing the process of the cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence in the foregoing embodiments when executed by the one or more computers.

[0076] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence, characterized in that, include: Obtain the registered wide-field fluorescence image and the corresponding TIRF image under the same field of view, and construct the training dataset as a set of image pairs; A convolutional neural network model with dual-branch output is constructed. A wide-field fluorescence image is used as input, and features are extracted through convolutional layers before the output is reconstructed. The first branch is the foreground reconstruction branch, which is used to output the foreground image after removing the out-of-focus background. The second branch is the background branch, which is used to output the background image. The foreground reconstruction branch is connected to a PSF convolution operator, which convolves the reconstructed foreground image with a predefined point spread function to generate a virtual TIRF image. The convolutional neural network model with dual-branch output is trained using the training dataset. A pair of images is input as a sample into the convolutional neural network model. A self-supervised loss is calculated based on the difference between the wide-field image reconstructed from the virtual TIRF image and the background image and the wide-field fluorescence image in the sample. A supervised loss is calculated based on the difference between the generated virtual TIRF image and the TIRF image in the sample. A loss function is constructed by combining the self-supervised loss and the supervised loss to guide iterative training and updates of the convolutional neural network model until convergence, resulting in a wide-field fluorescence image reconstruction model. For the input wide-field fluorescence image, the wide-field fluorescence image reconstruction model outputs a foreground image with the defocused background removed.

2. The cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence according to claim 1, characterized in that, For a pair of images consisting of a wide-field fluorescence image registered under the same field of view and the corresponding TIRF image, the TIRF image is the imaging data obtained by the imaging system focusing on the cover glass surface of the sample.

3. The cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence according to claim 1, characterized in that, The convolutional neural network model uses a U-shaped encoder-decoder U-Net as its backbone and incorporates a self-attention mechanism.

4. The cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence according to claim 1, characterized in that, The predefined point spread function is designed based on the optical parameters of the imaging system sampled from the image pairs, and is generated in advance by simulation software; and remains unchanged during training to simulate the point spread process of wide-field imaging.

5. The cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence according to claim 1, characterized in that, The background image output by the background branch is reconstructed by linear addition with the virtual TIRF image to obtain a wide-field image.

6. The cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence according to claim 1, characterized in that, The loss function also includes a regularization term R, which includes the total variational loss for both the foreground and background images and the L1 norm sparsity constraint for the foreground image. The loss function L constructed in this way is expressed as: in, The supervised loss is used to characterize the difference between the generated virtual TIRF image and the TIRF image in the sample; These represent the virtual TIRF image and the TIRF image in the sample, respectively. This is a self-supervised loss used to characterize the difference between the wide-field image reconstructed from the foreground and background images and the wide-field fluorescence image in the sample. These represent the reconstructed wide-field image and the wide-field fluorescence image in the sample, respectively. For regularization terms, These represent the foreground and background images output by the dual branches, respectively. λ and μ are the weight hyperparameters.

7. The cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence according to claim 6, characterized in that, The monitoring loss Self-monitoring losses The corresponding difference calculations all include a combination of weighted pixel intensity L1 loss, L2 loss, and multi-scale structural similarity loss.

8. The cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence according to any one of claims 1 to 7, characterized in that, For the input wide-field fluorescence image, the wide-field fluorescence image reconstruction model also outputs the corresponding background image.

9. A computer system, characterized in that, include: One or more processors; as well as Memory stores instructions that can be operated. When the instructions are executed by one or more processors, they cause the aforementioned one or more processors to perform operations, including the process of performing the cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence as described in any one of claims 1-8.

10. A computer-readable storage medium for storing one or more programs, characterized in that, The one or more programs include instructions or instruction sets that can be executed by one or more processors; When the instructions or instruction set are executed by one or more processors, they perform the process of the cross-modal image reconstruction method based on wide-field to total internal reflection fluorescence as described in any one of claims 1-8.

Images