3D Fluorescence Microscopy Image Generation Apparatus and Method Thereof
Patent Information
- Authority / Receiving Office
- KR · KR
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-02-05
- Publication Date
- 2026-08-12
Smart Images
Figure PAT00121_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a 3D fluorescence microscope image generation device and a generation method. Background Technology
[0002] Microscopic imaging has established itself as an essential tool in biological research, and 3D volume imaging, in particular, enables the non-invasive visualization of complex biological structures (e.g., blood vessels, cells). While volume fluorescence microscopy can generate high-resolution lateral images, severe blurring occurs in the axial plane due to the anisotropy of the Point Spread Function (PSF). This blurring degrades axial resolution, making detailed structural analysis difficult.
[0003] Existing PSF deconvolution techniques attempted to solve this problem by modeling the relationship between lateral images and PSFs, but these methods had limitations due to distortion caused by sample shifting and the diversity of PSFs. Although deep learning-based methodologies have recently been introduced, Generative Adversarial Network (GAN)-based technologies still fail to solve issues such as insufficient training stability, difficulty adapting to the diversity of PSF conditions, and domain-in-domain variation. Prior art literature
[0004] Korean Patent Publication No. 10-2722711 (Method for providing a baking process for a 3D model using a diffusion model, server and computer program, Rebuilder AI Co., Ltd., Oct. 29, 2024) The problem to be solved
[0005] A 3D fluorescence microscope image generation device and method are provided to generate 3D fluorescence microscope images through conditional denoising using a dual diffusion model. means of solving the problem
[0006] One embodiment of a method for generating 3D fluorescence microscope images may include the steps of: converting an observed 3D fluorescence microscope image into an observed side view image and an observed PSF kernel; denoising the observed side view image with a predetermined number of samplings using a pre-learned side diffusion model to generate a pre-side view image; denoising the observed PSF kernel with a predetermined number of samplings using a pre-learned PSF diffusion model to generate a pre-side view image; sampling an intermediate side view image based on the side view image and the pre-side view image, and sampling an intermediate PSF kernel based on the PSF kernel and the pre-side view image; generating a damaged side view image based on the pre-side view image and the pre-side view image, and generating a difference gradient by comparing the observed side view image and the damaged side view image; and applying the difference gradient to the intermediate side view image and the intermediate PSF kernel to generate a final side view image and a final PSF kernel at the next time point.
[0007] In addition, according to one embodiment, generating the prior side image of the 3D fluorescence microscope image generation method may be characterized by denoising the observed side image with the predetermined number of samplings based on Tweedie's formula.
[0008] In addition, according to one embodiment, generating the prior PSF kernel of the 3D fluorescence microscope image generation method may be characterized by denoising the observed PSF kernel with the predetermined number of samplings based on Tweedie's formula.
[0009] In addition, according to one embodiment, sampling the intermediate side image of the 3D fluorescence microscope image generation method may be characterized by sampling the intermediate side image at the current time point through ancestral sampling using the observed side image at the current time point and the prior side image at the final time point as inputs.
[0010] In addition, according to one embodiment, sampling the intermediate PSF kernel of the 3D fluorescence microscope image generation method may be characterized by sampling the intermediate PSF kernel at the current time point through ancestral sampling using the observed PSF kernel at the current time point and the prior PSF kernel at the final time point as inputs.
[0011] In addition, according to one embodiment, generating the difference gradient of the 3D fluorescence microscope image generation method may be characterized by generating the difference gradient by calculating the difference between the observed side image and the damaged side image.
[0012] In addition, according to one embodiment, the final side image at the next point in time of the 3D fluorescence microscope image generation method is given by the following mathematical formula
[0013]
[0014] (Here, is noise scheduling, is the above-mentioned observed 3D fluorescence microscope image, is the final 3D fluorescence microscope image at the time of the above-determined number of sampling counts, is a damaged 3D fluorescence microscope image, k is the observed PSF kernel, is the difference gradient for the above-mentioned observational side image, is the score function of the above-mentioned pre-trained lateral diffusion model, is the difference gradient applied to the above intermediate side image, is the dispersion of noise, It can be characterized as being derived through (meaning a stochastic term that adds random Gaussian noise in a stochastic differential equation).
[0015] In addition, according to one embodiment, the final PSF kernel at the next time point of the 3D fluorescence microscope image generation method is given by the following mathematical formula
[0016]
[0017] (Here, is noise scheduling, is the above-mentioned observed 3D fluorescence microscope image, is the final 3D fluorescence microscope image at the time of the above-determined number of sampling counts, is a damaged 3D fluorescence microscope image, k is the observed PSF kernel, is the difference gradient for the above-mentioned observed PSF kernel, is the score function of the above-mentioned pre-trained PSF diffusion model, is, the difference gradient applied to the above intermediate PSF kernel, is the dispersion of noise, It can be characterized as being derived through (meaning a stochastic term that adds random Gaussian noise in a stochastic differential equation).
[0018] In addition, according to one embodiment, the pre-trained lateral diffusion model of the 3D fluorescence microscope image generation method may be characterized by performing learning by using the observed lateral image as input and performing noising in a forward process for a predetermined number of learning iterations, and performing denoising in a reverse process for the noisy lateral image for a predetermined number of learning iterations.
[0019] In addition, according to one embodiment, the pre-trained PSF diffusion model of the 3D fluorescence microscope image generation method may be characterized by performing learning by using the observed PSF kernel as input and performing noising in a forward process for a predetermined number of learning iterations, and performing denoising in a reverse process for the noise PSF kernel for the predetermined number of learning iterations.
[0020] One embodiment of a 3D fluorescence microscope image generating device may include a conversion unit that converts an observed 3D fluorescence microscope image into an observed side image and an observed PSF kernel, and an inference unit that generates a preliminary side image by denoising the observed side image with a predetermined number of samplings using a pre-learned side diffusion model, generates a preliminary PSF kernel by denoising the observed PSF kernel with the predetermined number of samplings using a pre-learned PSF diffusion model, samples an intermediate side image based on the side image and the preliminary side image, samples an intermediate PSF kernel based on the PSF kernel and the preliminary PSF kernel, generates a damaged side image based on the preliminary side image and the preliminary PSF kernel, generates a difference gradient by comparing the observed side image and the damaged side image, and generates a final side image and a final PSF kernel at the next time point by applying the difference gradient to the intermediate side image and the intermediate PSF kernel.
[0021] In addition, generating the prior side image in the inference unit of the 3D fluorescence microscope image generating device according to one embodiment may be characterized by denoising the observed side image with the predetermined number of samplings based on Tweedie's formula.
[0022] In addition, generating the prior PSF kernel in the inference unit of the 3D fluorescence microscope image generation device according to one embodiment may be characterized by denoising the observed PSF kernel with the predetermined number of samplings based on Tweedie's formula.
[0023] In addition, sampling the intermediate side image in the inference unit of the 3D fluorescence microscope image generation device according to one embodiment may be characterized by sampling the intermediate side image at the current time point through ancestral sampling using the observed side image at the current time point and the prior side image at the final time point as inputs.
[0024] In addition, sampling the intermediate PSF kernel in the inference unit of the 3D fluorescence microscope image generation device according to one embodiment may be characterized by sampling the intermediate PSF kernel at the current time point through ancestral sampling using the observed PSF kernel at the current time point and the prior PSF kernel at the final time point as inputs.
[0025] In addition, generating the difference gradient in the inference unit of the 3D fluorescence microscope image generating device according to one embodiment may be characterized by generating the difference gradient by calculating the difference between the observed side image and the damaged side image.
[0026] In addition, the final side image at the next point in time in the inference unit of the 3D fluorescence microscope image generating device according to one embodiment is given by the following mathematical formula
[0027]
[0028] (Here, is noise scheduling, is the above-mentioned observed 3D fluorescence microscope image, is the final 3D fluorescence microscope image at the time of the above-determined number of sampling counts, is a damaged 3D fluorescence microscope image, k is the observed PSF kernel, is the difference gradient for the above-mentioned observational side image, is the score function of the above-mentioned pre-trained lateral diffusion model, is the difference gradient applied to the above intermediate side image, is the dispersion of noise, It can be characterized as being derived through (meaning a stochastic term that adds random Gaussian noise in a stochastic differential equation).
[0029] In addition, in the inference unit of the 3D fluorescence microscope image generating device according to one embodiment, the final PSF kernel at the next time point is the following mathematical formula
[0030]
[0031] (Here, is noise scheduling, is the above-mentioned observed 3D fluorescence microscope image, is the final 3D fluorescence microscope image at the time of the above-determined number of sampling counts, is a damaged 3D fluorescence microscope image, k is the observed PSF kernel, is the difference gradient for the above-mentioned observed PSF kernel, is the score function of the above-mentioned pre-trained PSF diffusion model, is, the difference gradient applied to the above intermediate PSF kernel, is the dispersion of noise, It can be characterized as being derived through (meaning a stochastic term that adds random Gaussian noise in a stochastic differential equation).
[0032] In addition, the 3D fluorescence microscope image generating device according to one embodiment may further include a learning unit that performs noising in a forward process for a predetermined number of learning cycles using the observed side image as input, and performs denoising in a reverse process for the noise side image for a predetermined number of learning cycles to generate the pre-learned side diffusion model.
[0033] In addition, the 3D fluorescence microscope image generating device according to one embodiment may further include a learning unit that performs noising in a forward process with a predetermined number of learning iterations using the observed PSF kernel as input, and performs denoising in a reverse process with the noise PSF kernel with the predetermined number of learning iterations to generate the pre-learned PSF diffusion model. Effects of the invention
[0034] According to the above-described 3D fluorescence microscope image generation device and generation method, 3D fluorescence microscope image generation can be provided based on a lateral image, a PSF kernel, and a dual diffusion model. Brief explanation of the drawing
[0035] FIG. 1 is a block diagram of a 3D fluorescence microscope image generating device according to one embodiment. FIG. 2 is a detailed block diagram of a 3D fluorescence microscope image generating device according to one embodiment. FIG. 3 is a conceptual diagram of a learning unit of a 3D fluorescence microscope image generating device according to one embodiment. FIG. 4 is a conceptual diagram of the inference unit of a 3D fluorescence microscope image generation device according to one embodiment. FIG. 5 is a flowchart for a method of generating a 3D fluorescence microscope image according to one embodiment. Specific details for implementing the invention
[0036] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.
[0037] The terms used in this specification will be briefly explained, and the invention will be described in detail.
[0038] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.
[0039] Throughout the specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "part," "module," and "unit" used in the specification refer to a unit that processes at least one function or operation and may be implemented as software, hardware components such as FPGAs or ASICs, or a combination of software and hardware. However, the terms "part," "module," and "unit" are not limited to software or hardware. "Part," "module," and "unit" may be configured to reside in an addressable storage medium or configured to run one or more processors. Accordingly, as an example, terms such as "part," "module," and "unit" include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.
[0040] Below, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the invention. Additionally, parts of the drawings that are irrelevant to the description are omitted to clearly explain the invention.
[0041] Terms including ordinal numbers, such as "first," "second," etc., may be used to describe various components, but the components are not limited by the terms. The terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related items or any one of the multiple related items.
[0042] Hereinafter, an embodiment of a 3D fluorescence microscope image generating device (1) and a generating method will be described with reference to the attached drawings.
[0043] Hereinafter, an embodiment of a 3D fluorescence microscope image generating device (1) and a generating method will be described with reference to FIGS. 1 to 4.
[0044] FIG. 1 is a block diagram of a 3D fluorescence microscope image generating device (1) according to one embodiment, and FIG. 2 is a detailed block diagram of a 3D fluorescence microscope image generating device (1) according to one embodiment.
[0045] The 3D fluorescence microscope image generating device (1) can convert a 3D fluorescence microscope image into a 2D lateral image based on the z-axis and convert the 3D fluorescence microscope image into a Point Spread Function (PSF) kernel. The 3D fluorescence microscope image generating device (1) can input the converted lateral image and the PSF kernel into a diffusion model to generate a pre-learned lateral diffusion model and a pre-learned PSF diffusion model, respectively. Based on the pre-learned lateral diffusion model and the pre-learned PSF diffusion model, the 3D fluorescence microscope image generating device (1) can generate a 3D fluorescence microscope image with noise removed by removing the PSF that differs by z-axis by performing conditional sampling at a predetermined number of samplings.
[0046] The 3D fluorescence microscope image generation device (1) can perform reconstruction by solving the inverse problem. Specifically, solving the image inverse problem means reconstructing a clean image from a damaged image. In volume fluorescence microscopy, anisotropic reconstruction is a damaged axial observation clean side image It can be defined as an inverse problem that restores it. In this case, the linear operator is represented by a PSF kernel k that causes blurring, which can be expressed as Equation 1.
[0047]
[0048] Here, is corrupted axial images, is the final lateral image (clean (uncorrupted) axial image) at a predetermined number of sampling points, k is the PSF kernel, and noise n is the noise, n~ It follows a Gaussian distribution. However, this problem is a blind inverse problem because the PSF kernel k is unknown. To solve this, the 3D fluorescence microscope image generator (1) performs the IsotropicDPS method using two score-based diffusion models to simultaneously estimate a clear side image and a PSF kernel.
[0049] The 3D fluorescence microscope image generating device (1) may include a processor (100), a communication unit (200), an input / output interface (300), and a memory (400).
[0050] The processor (100) may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller unit (MCU), an application processor (AP), an electronic control unit (ECU), and / or at least one electronic device capable of performing various calculations and control processing. These devices may be implemented, for example, by using one or more semiconductor chips, circuits, or related components alone or in combination.
[0051] The processor (100) may include a conversion unit (110), a training unit (130), and an inference unit (150).
[0052] The conversion unit (110) can convert the observed 3D fluorescence microscope image into an observed side image and an observed PSF kernel. Specifically, the conversion unit (110) can convert the observed 3D fluorescence microscope image into a 2D side image by dividing it into multiple layers based on the z-axis, and convert a PSF kernel for the z-axis for each side image.
[0053] FIG. 3 is a conceptual diagram of the learning unit of a 3D fluorescence microscope image generating device (1) according to one embodiment.
[0054] The training unit (130) can perform diffusion learning using a lateral image converted from an observed 3D fluorescence microscope image and a PSF kernel as input. Specifically, the training unit (130) can generate a pre-trained lateral diffusion model by performing learning using the observed lateral image as input, performing noising in a forward process for a predetermined number of learning iterations (I), and performing denoising in a reverse process on the noisy lateral image for a predetermined number of learning iterations. Additionally, the training unit (130) can generate a pre-trained PSF diffusion model by performing learning using the observed PSF kernel as input, performing noising in a forward process for a predetermined number of learning iterations, and performing denoising in a reverse process on the noisy PSF kernel for a predetermined number of learning iterations.
[0055] The diffusion model of the training unit (130) is an advanced class of generative models that utilize stochastic differential equations (SDEs). The training unit (130) performs training such as Denoising Diffusion Probabilistic Models (DDPM) and score matching models. The training unit (130) can use a dual diffusion model, a pre-trained lateral diffusion model, and a pre-trained PSF diffusion model that scores the model.
[0056] Pre-trained lateral-diff diffusion model (Lateral-Diff score model, ) is a clean side image It can learn prior information of clear microscope images by training. A pre-trained PSF diffusion model (PSF-Diff score model, ) can learn various blurring PSFs.
[0057] When training the dual-score model, the learning unit can optimize the stability and fidelity of the data distribution during the training process by using SDE, which preserves the variance of the data distribution. Lateral-Diff and PSF-Diff can learn the data distribution through forward and backward processes.
[0058] FIG. 4 is a conceptual diagram of the inference unit (150) of a 3D fluorescence microscope image generating device (1) according to one embodiment.
[0059] The inference unit (150) can generate a prior side image by denoising the observed side image with a predetermined number of samplings (T) using a pre-learned side diffusion model, and generate a prior PSF kernel by denoising the observed PSF kernel with a predetermined number of samplings using a pre-learned PSF diffusion model. Additionally, the inference unit (150) can sample an intermediate side image based on the side image and the prior side image, and sample an intermediate PSF kernel based on the PSF kernel and the prior PSF kernel. Additionally, the inference unit (150) can generate a damaged side image based on the prior side image and the prior PSF kernel, generate a difference gradient by comparing the observed side image and the damaged side image, and generate a final side image and a final PSF kernel at the next time step (step=t-1) by applying the difference gradient to the intermediate side image and the intermediate PSF kernel.
[0060] The inference unit (150) can generate a prior side image by denoising the observed side image with a predetermined number of samplings based on Tweedie's formula, and generate a prior PSF kernel by denoising the observed PSF kernel with a predetermined number of samplings based on Tweedie's formula.
[0061] The inference unit (150) can sample the intermediate side image of the current time point through ancestral sampling using the observed side image of the current time point (step=t) and the prior side image of the final time point (step=0) as inputs, and can sample the intermediate PSF kernel of the current time point through ancestral sampling using the observed PSF kernel of the current time point and the prior PSF kernel of the final time point as inputs. Additionally, the inference unit (150) can generate a difference gradient by calculating the difference between the observed side image and the damaged side image.
[0062] The inference unit (150) can perform inference for Zero-Shot Isotropic Reconstruction.
[0063] Anisotropic reconstruction can be achieved by sampling from a back distribution, which is a common approach for solving inverse problems. However, since the PSF kernel is unknown, the inference unit (150) must sample from a conditional back distribution that considers both the lateral image (axial image) and the PSF kernel. This is achieved through a parallel inverse process of Lateral-Diff and PSF-Diff, starting from a noise sample at a time interval t This can be achieved by jointly reconstructing the lateral image and the PSF kernel across.
[0064] The inference unit (150) can decompose the score function of the conditional back distribution as shown in Equation 2 by applying Bayes' rule to estimate the score function.
[0065]
[0066] Here, t is the time interval of the sampling phase, representing the current point in time, and is. Also, is the difference gradient for the side image, is the difference gradient for the PSF kernel, means conditional generation based on y.
[0067] The inference unit (150) can approximate and solve each term and perform backsampling in three steps.
[0068] First, the inference unit (150) can perform direct estimation for predicting a damaged side image.
[0069] The score function of the prior distribution in Equation 2 This can be estimated using PSF-Diff. However, since it is difficult to obtain a clean side image directly, it may be difficult to directly estimate the score function of the side image. To solve this, the inference unit (150) estimates the score function of the clean side image distribution by utilizing Lateral-Diff.
[0070] However, time-conditional log-likelihood terms and It is difficult to calculate. To solve this, the inference unit (150) approximates the likelihood term using a gradient-based approach that follows the Diffusion Posterior Sampling (DPS) method. The inference unit (150) first derives a pre-estimated clean lateral image and a PSF kernel by applying Tweedie's formula. The inference unit (150) directly estimates the lateral image and kernel at t=0 using pre-trained Lateral-Diff and PSF-Diff.
[0071] Secondly, the inference unit (150) can perform an axial / kernel-specific prior estimation.
[0072] The entire sampling process of the inference unit (150) performs sampling using a sampling method that numerically solves the inverse SDE for each time step. Specifically, the inference unit (150) at In the transition process, the pre-estimated lateral image (pre-lateral image) generated at t=0 by applying Tweedie's formula and denoising with a predetermined number of samplings Using... intermediate intermediate side image It can generate. The inference unit (150) generates a pre-estimated side image and Denoised by combining It can generate an intermediate PSF kernel. The inference unit (150) can generate an intermediate PSF kernel by applying the same process to the PSF kernel as well.
[0073] Thirdly, the inference unit (150) can perform refinement through comparison between the observed side image and the damaged side image.
[0074] The inference unit (150) utilizes the prior estimated prior side image and prior PSF kernel at t=0 and Equation 1 , k, Based on the relationship between the two, the time-conditional log-likelihood term can be approximated based on the difference from the predicted value of the observed damaged image. To this end, the inference unit (150) calculates the gradient by utilizing the backpropagation of the pre-trained Lateral-Diff and PSF-Diff. The inference unit (150) can ensure that the final sampling result matches the observed lateral image by adding the gradient difference to the intermediate result generated by the traditional sampling method.
[0075] If all the above terms can be approximated, the inverse SDE for anisotropy reconstruction is expressed as Equation 3.
[0076]
[0077] Here, is noise scheduling, is an observed 3D fluorescence microscope image, is the final 3D fluorescence microscope image at a predetermined number of sampling points, is a damaged 3D fluorescence microscope image, k is a PSF kernel, is the difference gradient for the side image, is the difference gradient for the PSF kernel, is the score function of the pre-trained lateral diffusion model, is the score function of the pre-trained PSF diffusion model, is the difference gradient applied to the mid-side image, is, difference gradient applied to the intermediate PSF kernel, is the dispersion of noise, represents a stochastic term that adds random Gaussian noise to a stochastic differential equation.
[0078] The inference unit (150) can perform conditional backsampling through the three main steps above. Through this, the inference unit (150) reconstructs a lateral image that matches the observed data and the prior information of the clear microscope image, and enables more accurate reconstruction by considering the kernel. That is, since the inference unit (150) does not require a clear lateral image or a known PSF during training, zero-shot lateral image reconstruction is possible. By directly estimating the PSF using the pre-trained PSF-Diff, the need to learn the distortion of the PSF is eliminated, and the requirement for lateral images during the training process is minimized.
[0079] The communication unit (200) can receive 3D fluorescence microscope images from an external device such as a terminal. The communication unit (200) can be implemented, for example, using at least one communication module (e.g., a LAN card, a short-range communication module, or a mobile communication module).
[0080] The input / output interface (300) can provide a 3D fluorescence microscope image so that the user directly inputs the 3D fluorescence microscope image to the 3D fluorescence microscope image generating device (1) without the communication unit (200) receiving the 3D fluorescence microscope image from an external device. The input / output interface (300) may include an input unit and an output unit.
[0081] The input / output interface (300) may be in the form of a push button to press an operation button, may operate the desired 3D fluorescence microscope image operation as a slide switch, or may input the desired operation as a touch input. In addition, various types of input devices for inputting the desired 3D fluorescence microscope image operation as a user may be used as examples of input units.
[0082] For example, the input / output interface (300) may include a display. The display may be provided as a Cathode Ray Tube (CRT), Digital Light Processing (DLP) panel, Plasma Display Panel, Liquid Crystal Display (LCD) panel, Electro Luminescence (EL) panel, Electrophoretic Display (EPD) panel, Electrochromic Display (ECD) panel, Light Emitting Diode (LED) panel, or Organic Light Emitting Diode (OLED) panel, but is not limited thereto. Additionally, the output unit may include a Central Processing Unit (CPU), a Graphic Processing Unit (GPU), and various types of storage devices implemented as a microprocessor, and such devices may be provided on an embedded Printed Circuit Board (PCB).
[0083] The memory (400) can store 3D fluorescence microscope images, a pre-learned lateral diffusion model, and a pre-learned PSF diffusion model.
[0084] The memory (400) may include at least one of a main memory and an auxiliary memory. The main memory may be implemented using a semiconductor storage medium such as ROM and / or RAM, for example, and the auxiliary memory may be implemented based on a device capable of storing data permanently or semi-permanently, such as a flash memory device (Solid State Drive (SSD), etc.), an SD (Secure Digital) card, a hard disk drive (HDD), a compact disc, a DVD, or a laser disc.
[0085] Hereinafter, an example of a method for generating a 3D fluorescence microscope image will be described with reference to FIG. 5.
[0086] FIG. 5 is a flowchart for a method of generating a 3D fluorescence microscope image according to one embodiment.
[0087] First, the conversion unit (110) can convert the observed 3D fluorescence microscope image into a side image and a PSF kernel (S100).
[0088] The inference unit (150) can generate a preliminary side image (S200) by denoising the side image with a predetermined number of samplings using a pre-learned side diffusion model, and generate a preliminary PSF kernel (S300) by denoising the PSF kernel with a predetermined number of samplings using a pre-learned PSF diffusion model.
[0089] Afterwards, the inference unit (150) can sample an intermediate side image based on the side image and the prior side image, and sample an intermediate PSF kernel (S400) based on the PSF kernel and the prior PSF kernel.
[0090] The inference unit (150) can generate a damaged 3D fluorescence microscope image based on a prior side image and a prior PSF kernel, and generate a difference gradient (S500) by comparing the observed 3D fluorescence microscope image and the damaged 3D fluorescence microscope image.
[0091] The inference unit (150) can generate the final side image and final PSF kernel at the next time point (S600) by applying the difference gradient to the intermediate side image and intermediate PSF kernel.
[0092] Those skilled in the art related to the embodiments of the present invention will understand that they may be implemented in modified forms without departing from the essential characteristics of the description. Therefore, the disclosed methods should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the detailed description of the invention, and all variations within the scope of the claims should be interpreted as being included within the scope of the invention. Explanation of the symbols
[0093] 1: 3D fluorescence microscope image generator 100: Processor 110: Conversion unit 130: Training Department 150: Inference section 200: Communications Department 300: Input / Output Interface 400: Memory
Claims
Claim 1 A method for generating a 3D fluorescence microscope image, comprising: a step of converting an observed 3D fluorescence microscope image into an observed lateral image and an observed PSF kernel; a step of generating a preliminary lateral image by denoising the observed lateral image with a predetermined number of samplings using a pre-learned lateral diffusion model; a step of generating a preliminary PSF kernel by denoising the observed PSF kernel with the predetermined number of samplings using a pre-learned PSF diffusion model; a step of sampling an intermediate lateral image based on the lateral image and the preliminary lateral image, and sampling an intermediate PSF kernel based on the PSF kernel and the preliminary PSF kernel; a step of generating a damaged lateral image based on the preliminary lateral image and the preliminary PSF kernel, and generating a difference gradient by comparing the observed lateral image and the damaged lateral image; and a step of generating a final lateral image and a final PSF kernel at the next time point by applying the difference gradient to the intermediate lateral image and the intermediate PSF kernel. Claim 2 A method for generating a 3D fluorescence microscope image according to claim 1, wherein generating the prior side image is characterized by denoising the observed side image with the predetermined number of samplings based on Tweedie's formula. Claim 3 A method for generating a 3D fluorescence microscope image according to claim 1, wherein generating the prior PSF kernel is characterized by denoising the observed PSF kernel with the predetermined number of samplings based on Tweedie's formula. Claim 4 A method for generating a 3D fluorescence microscope image according to claim 1, wherein sampling the intermediate side image is characterized by sampling the intermediate side image at the current time point through ancestral sampling using the observed side image at the current time point and the prior side image at the final time point as inputs. Claim 5 A method for generating a 3D fluorescence microscope image according to claim 1, wherein sampling the intermediate PSF kernel is characterized by sampling the intermediate PSF kernel at the current time point through ancestral sampling using the observed PSF kernel at the current time point and the prior PSF kernel at the final time point as inputs. Claim 6 A method for generating a 3D fluorescence microscope image according to claim 1, wherein generating the difference gradient is characterized by calculating the difference between the observed side image and the damaged side image to generate the difference gradient. Claim 7 In paragraph 1, the final side image at the next point in time is given by the following mathematical formula (Here, is noise scheduling, is the above-mentioned observed 3D fluorescence microscope image, is the final 3D fluorescence microscope image at the time of the above-determined number of sampling counts, is a damaged 3D fluorescence microscope image, k is the observed PSF kernel, is the difference gradient for the above-mentioned observational side image, is the score function of the above-mentioned pre-trained lateral diffusion model, is the difference gradient applied to the above intermediate side image, is the dispersion of noise, A method for generating 3D fluorescence microscope images characterized by being derived through (meaning a stochastic term that adds random Gaussian noise to a stochastic differential equation). Claim 8 In paragraph 1, the final PSF kernel at the next point in time is the following mathematical formula (Here, is noise scheduling, is the above-mentioned observed 3D fluorescence microscope image, is the final 3D fluorescence microscope image at the time of the above-determined number of sampling counts, is a damaged 3D fluorescence microscope image, k is the observed PSF kernel, is the difference gradient for the above-mentioned observed PSF kernel, is the score function of the above-mentioned pre-trained PSF diffusion model, is, the difference gradient applied to the above intermediate PSF kernel, is the dispersion of noise, A method for generating 3D fluorescence microscope images characterized by being derived through (meaning a stochastic term that adds random Gaussian noise to a stochastic differential equation). Claim 9 A method for generating a 3D fluorescence microscope image according to claim 1, wherein the pre-trained lateral diffusion model performs learning by performing noising in a forward process a predetermined number of times using the observed lateral image as input, and performing denoising in a reverse process a predetermined number of times using the noisy lateral image. Claim 10 A method for generating a 3D fluorescence microscope image according to claim 1, wherein the pre-trained PSF diffusion model performs learning by performing noising in a forward process with the observed PSF kernel as input for a predetermined number of learning iterations, and performing denoising in a reverse process with the noise PSF kernel for the predetermined number of learning iterations. Claim 11 A 3D fluorescence microscope image generating device comprising: a conversion unit that converts an observed 3D fluorescence microscope image into an observed side image and an observed PSF kernel; and an inference unit that generates a preliminary side image by denoising the observed side image with a predetermined number of samplings using a pre-learned side diffusion model, generates a preliminary PSF kernel by denoising the observed PSF kernel with the predetermined number of samplings using a pre-learned PSF diffusion model, samples an intermediate side image based on the side image and the preliminary side image, samples an intermediate PSF kernel based on the PSF kernel and the preliminary PSF kernel, generates a damaged side image based on the preliminary side image and the preliminary PSF kernel, generates a difference gradient by comparing the observed side image and the damaged side image, and generates a final side image and a final PSF kernel at the next time point by applying the difference gradient to the intermediate side image and the intermediate PSF kernel. Claim 12 A 3D fluorescence microscope image generating device according to claim 11, wherein the generating of the prior side image in the inference unit is characterized by denoising the observed side image with the predetermined number of samplings based on Tweedie's formula. Claim 13 A 3D fluorescence microscope image generating device according to claim 11, wherein generating the prior PSF kernel in the inference unit is characterized by denoising the observed PSF kernel with the predetermined number of samplings based on Tweedie's formula. Claim 14 A 3D fluorescence microscope image generating device according to claim 11, wherein sampling the intermediate side image in the inference unit is characterized by sampling the intermediate side image at the current time point through ancestral sampling using the observed side image at the current time point and the prior side image at the final time point as inputs. Claim 15 A 3D fluorescence microscope image generating device according to claim 11, wherein sampling the intermediate PSF kernel in the inference unit is characterized by sampling the intermediate PSF kernel at the current time point through ancestral sampling using the observed PSF kernel at the current time point and the prior PSF kernel at the final time point as inputs. Claim 16 A 3D fluorescence microscope image generating device according to claim 11, wherein the difference gradient generated in the inference unit is characterized by generating the difference gradient by calculating the difference between the observed side image and the damaged side image. Claim 17 In Clause 11, the final side image at the next point in time in the above inference unit is the following mathematical formula (Here, is noise scheduling, is the above-mentioned observed 3D fluorescence microscope image, is the final 3D fluorescence microscope image at the time of the above-determined number of sampling counts, is a damaged 3D fluorescence microscope image, k is the observed PSF kernel, is the difference gradient for the above-mentioned observational side image, is the score function of the above-mentioned pre-trained lateral diffusion model, is the difference gradient applied to the above intermediate side image, is the dispersion of noise, A 3D fluorescence microscope image generating device characterized by being derived through (meaning a stochastic term that adds random Gaussian noise to a stochastic differential equation). Claim 18 In Clause 11, the final PSF kernel at the next point in time in the above inference unit is the following mathematical formula (Here, is noise scheduling, is the above-mentioned observed 3D fluorescence microscope image, is the final 3D fluorescence microscope image at the time of the above-determined number of sampling counts, is a damaged 3D fluorescence microscope image, k is the observed PSF kernel, is the difference gradient for the above-mentioned observed PSF kernel, is the score function of the above-mentioned pre-trained PSF diffusion model, is, the difference gradient applied to the above intermediate PSF kernel, is the dispersion of noise, A 3D fluorescence microscope image generating device characterized by being derived through (meaning a stochastic term that adds random Gaussian noise to a stochastic differential equation). Claim 19 A 3D fluorescence microscope image generating device according to claim 11, further comprising a learning unit that performs learning by performing noising in a forward process for a predetermined number of learning iterations using the observed side image as input, and performing denoising in a reverse process for the noise side image for the predetermined number of learning iterations to generate the pre-learned side diffusion model. Claim 20 A 3D fluorescence microscope image generating device according to claim 11, further comprising a learning unit that performs learning by performing noising in a forward process with the observed PSF kernel as input for a predetermined number of learning iterations, and performing denoising in a reverse process with the noise PSF kernel for the predetermined number of learning iterations to generate the pre-learned PSF diffusion model.