Image processing system and method

Abstract

The embodiment of the invention provides an image processing method and system. The method comprises the following steps: acquiring image data output by image acquisition equipment; generating an initial image by processing the image data; and generating a target image by processing the initial image according to an iterative optimization algorithm using the image processing model. The image processing model includes a first sub-model and a second sub-model. The first sub-model is configured to determine a first optimization item related to a likelihood item of an objective function, the second sub-model is set to determine a second optimization item related to a regularization item of the objective function, and the second sub-model is a trained machine learning model.

Description

Cross-references

[0001] This application claims priority to U.S. patent application No. US18171394 filed on February 20, 2023 and Chinese patent application No. CN2024101380105 filed on January 31, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This specification relates to the field of image processing, and in particular to a system and method for reconstructing an image based on image data output by an image acquisition device. Background Art

[0003] Super-resolution (SR) microscopy is a series of techniques in optical microscopy that allow images to have a resolution higher than the diffraction limit. The advent of SR fluorescence microscopy technology has revolutionized biology and enabled previously imperceptible and complex structures to be observed, such as periodic actin rings in neuronal dendrites, nuclear pore complex structures, and the organization of pericentriolar material around centrioles. Since SR microscopic images are expected to show a clear and accurate view of microstructures, scientific researchers generally have high requirements for the quality of SR microscopic images. Many traditional techniques for reconstructing SR microscopic images have artifact problems. In addition, errors may sometimes even occur in the reconstructed image. Therefore, the present application hopes to provide systems and methods that can reconstruct higher quality images.

[0004] Fluorescence microscopy (FM) is essential for elucidating the dynamics and function of various biological processes in living cells. However, FM can present several drawbacks and trade-offs. These limitations can become apparent when comparing different microscope types. For example, three-dimensional structured illumination microscopy (3D-SIM) outperforms high-numerical-aperture Fourier light-field microscopy (FLFM) in terms of spatial resolution, but is slower. 3D-SIM outperforms 3D spinning disk confocal microscopy (3D-CSDM) in terms of spatial resolution, but is less robust to fluorescence photobleaching. When considering applications for live-cell SR microscopy, the required spatiotemporal resolution must be weighed against the imaging signal-to-noise ratio.

[0005] In addition to advances in microscope hardware, computational methods are becoming increasingly important for high-quality image reconstruction in FM. From a statistical perspective, the process of FM image degradation essentially corresponds to information loss (resolution loss) and measurement uncertainty (imaging noise) when the camera observes the sample. The FM image reconstruction process can be transformed into a linear inverse problem consisting of a fidelity term corresponding to the camera observation process and a regularization term corresponding to the sample's prior distribution. Designing a regularization term that accurately reflects the complex statistical properties of the sample is crucial for high-quality FM image reconstruction. Previous researchers have developed various mathematically elaborate regularization terms, but the complex statistical properties of the sample may only be partially, rather than fully, reflected in their formulations. Therefore, manually designed regularization term models are limited by the accuracy of their assumptions and may result in a loss in reconstructed image quality. Deep learning (DL) offers powerful feature extraction capabilities and can theoretically approximate complex sample distributions with minimal error. Therefore, DL can be used to more accurately model the regularization term.

[0006] This document provides an image reconstruction method for microscopes (e.g., SIM, CSDM, widefield microscopy (WFM), FLFM, etc.). This reconstruction technique combines a fidelity term based on a camera imaging noise model with a regularization term based on deep learning, thereby achieving low-noise, low-artifact, and high-fidelity reconstruction. Summary of the Invention

[0007] According to one aspect of the embodiments of this specification, an image processing method is provided. The method is implemented on a machine having at least one processor and at least one storage device. The method may include acquiring image data output by an image acquisition device; generating an initial image by processing the image data; and generating a target image by processing the initial image according to an optimization algorithm using an image processing model, the image processing model including a first sub-model and a second sub-model. The first sub-model may be configured to determine a first optimization term related to a likelihood term of an objective function. The second sub-model may be configured to determine a second optimization term related to a regularization term of the objective function, the second sub-model being a trained machine learning model.

[0008] According to another aspect of the embodiments of this specification, an image processing system is provided. The system may include at least one storage device including an instruction set; and at least one processor in communication with the at least one storage device, wherein when executing the instruction set, the at least one processor is configured to cause the system to perform the following operations: acquire image data output by an image acquisition device; generate an initial image by processing the image data; and generate a target image by processing the initial image according to an optimization algorithm using an image processing model, wherein the image processing model includes a first sub-model and a second sub-model, wherein the first sub-model is configured to determine a first optimization term related to a likelihood term of an objective function, and the second sub-model is configured to determine a second optimization term related to a regularization term of the objective function, and the second sub-model is a trained machine learning model.

[0009] According to another aspect of the embodiments of this specification, an image processing system is provided. The system may include: an acquisition module for acquiring image data output by an image acquisition device; an initial image generation module configured to generate an initial image by processing the image data; and a target image generation module configured to generate a target image by processing the initial image according to an optimization algorithm using an image processing model. The image processing model may include a first sub-model and a second sub-model. The first sub-model may be configured to determine a first optimization term related to a likelihood term of an objective function. The second sub-model may be configured to determine a second optimization term related to a regularization term of the objective function, wherein the second sub-model is a trained machine learning model.

[0010] According to another aspect of the embodiments of this specification, a non-transitory computer-readable medium is provided, comprising at least one set of instructions for image processing, wherein when executed by one or more processors of a computing device, the at least one set of instructions causes the computing device to perform a method. The method may include acquiring image data output by an image acquisition device; generating an initial image by processing the image data; and generating a target image by processing the initial image according to an optimization algorithm using an image processing model, the image processing model comprising a first sub-model and a second sub-model. The first sub-model may be configured to determine a first optimization term associated with a likelihood term of an objective function. The second sub-model may be configured to determine a second optimization term associated with a regularization term of the objective function, the second sub-model being a trained machine learning model.

[0011] According to one aspect of the embodiments of this specification, an image processing method is provided. The method is implemented on a device having at least one processor and at least one storage device. The method may include: acquiring multidimensional image data, wherein the multidimensional image data is acquired using a fluorescence microscope; generating an initial multidimensional image based on the multidimensional image data; and constructing an objective function based on the acquisition process of the multidimensional image data, and performing one or more iterations on the initial multidimensional image according to the objective function to generate a target image. The objective function may include a fidelity term and a regularization term. The fidelity term may be related to a physical model of fluorescence microscope imaging. The regularization term is determined by a regularization network.

[0012] According to another aspect of the embodiments of this specification, an image processing system is provided. The system may include at least one storage device including an instruction set; and at least one processor in communication with the at least one storage device, wherein, when executing the instruction set, the at least one processor is instructed to cause the system to perform operations including: acquiring multidimensional image data, wherein the multidimensional image data is acquired using a fluorescence microscope; generating an initial multidimensional image based on the multidimensional image data; and constructing an objective function based on the acquisition process of the multidimensional image data, and performing one or more iterations on the initial multidimensional image according to the objective function to generate a target image. The objective function may include a fidelity term and a regularization term. The fidelity term may be related to a physical model of fluorescence microscopy imaging. The regularization term is determined by a regularization network.

[0013] According to another aspect of the embodiments of this specification, an image processing system is provided. The system may include: an image data acquisition module configured to acquire multidimensional image data, wherein the multidimensional image data is collected by a fluorescence microscope; an initial image generation module configured to generate an initial multidimensional image based on the multidimensional image data; and a target image generation module configured to construct an objective function based on the acquisition process of the multidimensional image data, and perform one or more iterations on the initial multidimensional image according to the objective function to generate a target image. The objective function may include a fidelity term and a regularization term. The fidelity term may be related to the imaging physics model of the fluorescence microscope. The regularization term is determined by a regularization network.

[0014] According to another aspect of the embodiments of this specification, a non-transitory computer-readable medium is provided, comprising at least one set of instructions for image processing, wherein when executed by one or more processors of a computing device, the at least one set of instructions causes the computing device to perform a method comprising: acquiring multidimensional image data, wherein the multidimensional image data is acquired by a fluorescence microscope; generating an initial multidimensional image based on the multidimensional image data; and constructing an objective function based on the acquisition process of the multidimensional image data, and performing one or more iterations on the initial multidimensional image according to the objective function to generate a target image. The objective function may include a fidelity term and a regularization term. The fidelity term may be related to an imaging physics model of the fluorescence microscope. The regularization term is determined by a regularization network.

[0015] According to one aspect of an embodiment of this specification, an image processing method is provided. The method is implemented on a device having at least one processor and at least one storage device. The method may include: acquiring image data, where the image data is output by a fluorescence microscope; generating an initial image based on the image data; and processing the initial image according to an objective function to generate a target image. The objective function is constructed based on the image data acquisition process. The objective function may include a fidelity term and a regularization term. The fidelity term may be related to a physical model of fluorescence microscope imaging. The regularization term is determined by a regularization network. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present disclosure will further describe exemplary embodiments in detail. These exemplary embodiments are described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, like numbers represent like structures, wherein:

[0017] Figure 1 is a schematic diagram of an exemplary application scenario of an image processing system according to some embodiments of the present disclosure;

[0018] Figure 2 is a schematic diagram of an exemplary computing device according to some embodiments of the present disclosure;

[0019] Figure 3 is a block diagram of an exemplary terminal device according to some embodiments of the present disclosure;

[0020] Figure 4 is a schematic diagram of an exemplary processor according to some embodiments of the present disclosure;

[0021] Figure 5 is an exemplary flow chart of an image processing method according to some embodiments of the present disclosure;

[0022] Figure 6is a flowchart of an exemplary process for obtaining an image processing model through a training operation according to some embodiments of the present disclosure;

[0023] Figure 7A is a schematic diagram of generating a target image according to some embodiments of the present disclosure;

[0024] Figure 7B is a schematic diagram of an exemplary structure of a second sub-model according to some embodiments of the present disclosure;

[0025] Figure 8A Shown are a reference image obtained using a traditional Wiener reconstruction method according to some embodiments of the present disclosure, a target image obtained using an image processing model provided by the present disclosure, and an image representing the corresponding ground truth (GT);

[0026] Figure 8B The peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the target image and the ground truth (GT) are shown when λ and T are changed according to some embodiments of the present disclosure;

[0027] Figure 8C According to some embodiments of the present disclosure, Figure 8A The gray box marks the part of the image;

[0028] Figure 9A Figure 4 shows actin filaments under SR structured illumination microscopy (SR-SIM) according to some embodiments of the present disclosure;

[0029] Figure 9B The reconstruction of the image by Wiener inverse filtering, HiFi-SIM, Hessian-SIM and TDV-SIM according to some embodiments of the present disclosure is shown. Figure 9A A magnified image of the larger box area marked by the gray box in Figure 3, and the GT image;

[0030] Figure 9C Figure 2 shows the reconstruction of the image by Wiener inverse filtering, scU-Net, DFCAN and TDV-SIM according to some embodiments of the present disclosure. Figure 9A A magnified image of the smaller box area marked by the gray box in Figure 3, and the GT image;

[0031] Figure 9D The endoplasmic reticulum (ER) under SR-SIM is shown;

[0032] Figure 9E The reconstruction by Wiener inverse filtering, HiFi-SIM, Hessian-SIM, scU-Net, DFCAN and TDV-SIM is shown. Figure 9D A magnified image of the box area in , and the GT image;

[0033] Figure 9F The variance of artifacts of actin filaments from background regions in different reconstructions is shown;

[0034] Figure 9G The artifact variance of ER from background regions in different reconstructions is shown;

[0035] Figure 9H SSIM of actin filaments in different reconstructions is shown;

[0036] Figure 9I The SSIM of ER in different reconstructions is shown;

[0037] Figure 9J Shown Figures 9A-9C Resolution of actin filaments in different reconstructions;

[0038] Figure 10A Shown are microtubules in the BioSR dataset reconstructed using different methods under SR-SIM;

[0039] Figure 10B The reconstructions by Wiener inverse filtering, rDL SIM, NF-rDL SIM and TDV-SIM are shown. Figure 10A Magnified view of the larger boxed area;

[0040] Figure 10C The artifact variance of the background region in different reconstructions is shown;

[0041] Figure 10D The SSIM of microtubules in different reconstructions is shown;

[0042] Figure 11A Mitochondria under SR-SIM are shown;

[0043] Figure 11B The photobleaching process showing the temporal changes in mitochondrial fluorescence intensity;

[0044] Figure 11C shows the reconstruction by scU-Net, DFCAN and TDV-SIM Figure 11A A magnified image of the larger box area and the corresponding GT image at 0 seconds;

[0045] Figure 11D The constructed model by Wiener inverse filtering, HiFi-SIM, Hessian-SIM and TDV-SIM is shown. Figure 11A A magnified image of the smaller box area in the figure and the corresponding GT images at 0, 15, and 20 seconds;

[0046] Figure 11EThe SSIM of the mitochondrial enclosed regions of different reconstructions compared with the GT images at 0, 15, and 20 seconds are shown;

[0047] Figure 11F The artifact variance of the background region in different reconstructions at 0, 15, and 20 seconds is shown;

[0048] Figure 12A Actin filaments under NL SIM are shown;

[0049] Figures 12B-12C Reconstruction by Wiener inverse filtering, Hessian-SIM, DFCAN and TDV-NL-SIM is shown. Figure 12A Enlarged image of the middle white box area;

[0050] Figure 12D Shown Figure 12A The enlarged image of the gray box area reconstructed by Wiener inverse filtering, Hessian-SIM, DFCAN and TDV-NL-SIM;

[0051] Figure 12E The variance of artifacts from actin filaments in the background region in different reconstructions is shown;

[0052] Figure 12F Signal changes along actin filaments in different reconstructions are shown;

[0053] Figure 12G The SSIM of actin filaments in different reconstructions is shown;

[0054] Figures 13A-13D is a schematic diagram of another exemplary image processing method according to some embodiments of the present disclosure;

[0055] Figures 14A-14S exemplarily comparing TDV 3D-SIM with state-of-the-art 3D-SIM methods according to some embodiments of the present disclosure;

[0056] Figures 15A-15G The TDV reconstruction technique shown in some embodiments of the present disclosure improves CSDM imaging without signal time leakage and loss;

[0057] Figures 16A-16F The TDV reconstruction technique shown in some embodiments of the present disclosure improves live cell two-color time-lapse widefield microscopy (WFM) and FLFM imaging of mitochondria and peroxisomes;

[0058] Figures 17A-17C is an example diagram of a Fourier light field microscope (FLFM) according to some embodiments of the present disclosure;

[0059] Figure 18 is a block diagram of an exemplary image processing system according to some embodiments of the present disclosure;

[0060] Figure 19 is an exemplary flow chart of an image processing method according to some embodiments of the present disclosure; and

[0061] Figure 20 is a schematic diagram of an exemplary training process of a regularized network according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0062] In the following detailed description, many specific details are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be understood by those skilled in the art that the present disclosure may be practiced without these details. In other cases, in order to avoid unnecessarily obscuring aspects of the present disclosure, well-known methods, processes, systems, components and / or circuits have been briefly described at a relatively high level without providing details. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the embodiments shown, but is given the widest scope consistent with the claims.

[0063] The terms used herein are used only to describe specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" used herein may also include the plural forms. It is further understood that when the terms "include", "comprising", and / or "encompass" are used in this specification, the terms "include", "contain", and / or "cover" specify the presence of the features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that the terms "object" and "subject" can be used interchangeably as a reference to something that undergoes the imaging process of the present disclosure.

[0064] It should be understood that the terms "system," "engine," "unit," "module," and / or "block" used herein are a method of distinguishing different components, elements, parts, portions, or assemblies at different levels in ascending order. However, these terms may be replaced by another expression if they achieve the same purpose.

[0065] Generally, the terms "module," "unit," or "block" as used herein refer to logic contained in hardware or firmware, or a collection of software instructions. The modules, units, or blocks described herein may be implemented as software and / or hardware and may be stored in any type of non-transitory computer-readable medium or another storage device. In some embodiments, software modules / units / blocks may be compiled and linked into an executable program. It should be understood that a software module may be called from other modules / units / blocks or itself, and / or may be called in response to a detected event or interrupt. Configuration for use in a computing device (e.g., such as Figure 2 The software modules / units / blocks executed on the processor 210 shown in the figure may be provided on a computer-readable medium, such as a compact disc, a digital video disc, a flash drive, a disk, or any other tangible medium, or as a digital download (and may be initially stored in a compressed or installable format, requiring installation, decompression, or decryption before execution). Such software code may be stored in part or in whole on a storage device of the executing computing device for execution by the computing device. The software instructions may be embedded in firmware, such as an EPROM. It should also be understood that hardware modules / units / blocks may be contained in connected logical components, such as gates and flip-flops, and / or may be contained in programmable units, such as programmable gate arrays or processors. The modules / units / blocks or computing device functions described herein may be implemented as software modules / units or blocks, but may also be represented in hardware or firmware. Generally speaking, the modules / units / blocks described herein refer to logical modules / units and blocks that can be combined with other modules / units or blocks or divided into sub-modules / sub-units / sub-blocks, regardless of their physical organization or storage method. This description may apply to a system, an engine, or a portion thereof.

[0066] It should be noted that when describing operations performed on an image, the term "image" as used herein may refer to a data set (e.g., a matrix) containing pixel values ​​(pixel values) in an image. As used herein, for the sake of brevity, a representation of an object (e.g., a person, an organ, a cell, or a portion thereof) in an image may be referred to simply as an object. For example, for the sake of brevity, a representation of a cell or organelle (e.g., a mitochondria, an endoplasmic reticulum, a centrosome, a Golgi apparatus, etc.) in an image may be referred to as a cell or a cell apparatus. As used herein, for the sake of brevity, an operation performed on a representation of an object in an image may be referred to simply as an operation on the object. For example, for the sake of brevity, segmentation of an image portion that includes a representation of a cell or organelle in an image may be referred to as segmentation of a cell or organelle.

[0067] It should be understood that the term "resolution" as used herein refers to a measure of image clarity. The term "super-resolution" or "super-resolution" or "SR" as used herein refers to enhanced (or increased) resolution, such as may be obtained by the process of combining sequences of lower-resolution images to generate a higher-resolution image or sequence.

[0068] Traditional methods for reconstructing images are generally based on the imaging principles of image acquisition devices (or physical models that reflect the imaging principles of image acquisition devices). Images with relatively high resolution (e.g., SR microscopy images) generated by these traditional methods generally include one or more artifacts, or have an unsatisfactory signal-to-noise ratio. With the development of image processing technology, trained machine learning models have the potential to generate high-quality target images based on image data collected or generated by image acquisition devices. However, since the reconstruction process using a trained machine learning model is not constrained by the imaging principles, the quality of the target image depends on the training set used to obtain the trained machine learning model. Therefore, the image generated by the trained machine learning model may contain some errors.

[0069] According to the systems and methods of the present disclosure, image data generated from an image acquisition device can be used to generate an initial image. An image processing model can be used to optimize the initial image based on an objective function to generate a target image. The image processing model may include a first sub-model and a second sub-model. The first sub-model may be configured to determine a first optimization term related to a likelihood term of the objective function. The second sub-model may be configured to determine a second optimization term related to a regularization term of the objective function. The second sub-model may be a trained machine learning model. Using a trained machine learning model can effectively reduce artifacts or noise in the target image. In addition, using the likelihood term can ensure that the reconstruction is based on imaging principles, thereby reducing or avoiding errors in the target image.

[0070] Furthermore, although the systems and methods disclosed herein are primarily described with respect to processing images generated by structured illumination microscopy (SIM), it should be understood that this description is for illustrative purposes only and is not intended to limit the scope of the present disclosure. The systems and methods disclosed herein can be applied to any other type of system, including image acquisition devices for image processing. For example, the systems and methods disclosed herein can be applied to microscopes, telescopes, cameras (e.g., surveillance cameras, camera phones, webcams), unmanned aerial vehicles, medical imaging equipment, and the like, or any combination thereof.

[0071] It should be understood that the application scenarios of the systems and methods disclosed herein are only some exemplary embodiments provided for illustration and are not intended to limit the scope of the present disclosure. For those skilled in the art, various changes and modifications can be made under the guidance of the present disclosure.

[0072] Figure 1 FIG is a schematic diagram of an exemplary application scenario of an image processing system according to some embodiments of the present disclosure. Figure 1As shown, the image processing system 100 may include an image acquisition device 110 , a network 120 , one or more terminal devices 130 , a processor 140 and a storage device 150 .

[0073] The various components in the image processing system 100 can be connected to each other in a variety of different ways. By way of example only, the image acquisition device 110 can be connected to the processor 140 via the network 120. As another example, the image acquisition device 110 can be directly connected to the processor 140, as indicated by the double-headed arrow within the dashed line connecting the image acquisition device 110 and the processor 140. As another example, the storage device 150 can be connected to the processor 140 directly or via the network 120. As another example, the terminal device 130 can be connected to the processor 140 directly (as indicated by the double-headed arrow within the dashed line connecting the terminal device 130 and the processor 140) or via the network 120.

[0074] The image processing system 100 may be configured to generate a target image (e.g., Figure 5 500 ). The target image can have a relatively high resolution that can exceed the physical limitations faced by the image acquisition device 110. For example, the image processing system 100 can acquire multiple raw cell images with a relatively low signal-to-noise ratio generated by the image acquisition device 110 (e.g., a SIM). For another example, the image processing system 100 can acquire one or more images acquired by the image acquisition device 110 (e.g., a camera phone or a mobile phone with a camera). One or more images may be blurred and / or have a relatively low resolution due to physical limitations caused by the mobile phone during filming and / or photography, such as shaking of the camera phone, movement of the object to be imaged, inaccurate focus, etc. The image processing system 100 can process the images using an image processing model to generate one or more target images with relatively high quality. Therefore, the image processing system 100 can present relatively high-quality target images to its users.

[0075] The image processing system 100 can also be configured to generate a target image by performing one or more iterations on an initial image (e.g., an original cell image) based on an objective function. In some embodiments, the objective function can include a fidelity term and a regularization term. The fidelity term can be related to the imaging model of the microscope that generated the initial image. The regularization term is determined by a regularization network. Further descriptions of the fidelity term and the regularization term can be found elsewhere in this disclosure (e.g., Formula (10), Figures 13A-13B and its description).

[0076] The image acquisition device 110 can be configured to acquire image data associated with an object within its detection area. In the present disclosure, "object" and "subject" can be used interchangeably. The subject can include one or more biological or non-biological objects. In some embodiments, the image acquisition device 110 can be an optical imaging device, a radiation-based imaging device (e.g., a computed tomography device), a nuclide-based imaging device (e.g., a positron emission tomography device), a magnetic resonance imaging device, etc. Exemplarily, the optical imaging device can include a microscope 111 (e.g., a fluorescence microscope), a monitoring device 112 (e.g., a security camera), a mobile terminal device 113 (e.g., a camera phone), a scanning device 114 (e.g., a flatbed scanner, a drum scanner, etc.), a telescope, a webcam, etc., or any combination thereof. In some embodiments, the optical imaging device can include an acquisition device (e.g., a detector or a camera) for acquiring image data. For illustrative purposes, the present disclosure can use the microscope 111 as an example to describe the exemplary functions of the image acquisition device 110. Exemplarily, the microscope may include structured illumination microscopy (SIM) (e.g., two-dimensional SIM (2D-SIM), three-dimensional SIM (3D-SIM), total internal reflection SIM (TIRF-SIM), spinning disk confocal-based SIM (SD-SIM), etc.), photoactivated localization microscopy (PALM), stimulated emission depletion fluorescence microscopy (STED), stochastic optical reconstruction microscopy (STORM), spinning disk confocal microscopy (CSDM), widefield microscopy (WFM), and Fourier light field microscopy (FLFM), etc. SIM may use a variety of detection cameras, such as EMCCD cameras, sCMOS cameras, etc. The objects detected by SIM may include one or more objects of various biological structures, biological tissues, proteins, cells, microorganisms, etc., or any combination thereof. Exemplarily, the cells may include INS-1 cells, COS-7 cells, Hela cells, liver sinusoidal endothelial cells (LSEC), human umbilical vein endothelial cells (HUVEC), HEK293 cells, etc., or any combination thereof. In some embodiments, one or more objects may be fluorescent or fluorescently labeled. Fluorescent or fluorescently labeled objects may be excited by laser to emit fluorescence for imaging.

[0077] The network 120 may include any suitable network capable of facilitating information and / or data exchange between the image processing system 100. In some embodiments, one or more components of the image processing system 100 (e.g., the image acquisition device 110, the terminal device 130, the processor 140, the storage device 150, etc.) may exchange information and / or data with each other via the network 120. For example, the processor 140 may obtain image data from the image acquisition device 110 via the network 120. For another example, the processor 140 may obtain user instructions from the terminal device 130 via the network 120. The network 120 may be and / or include a public network (e.g., the Internet), a private network (e.g., a local area network (LAN), a wide area network (WAN), etc.), a wired network (e.g., Ethernet), a wireless network (e.g., an 802.11 network, a Wi-Fi network, etc.), a cellular network (e.g., a Long Term Evolution (LTE) network), an image relay network, a virtual private network ("VPN"), a satellite network, a telephone network, a router, a hub, a switch, a server computer, and / or a combination of one or more thereof. For example, the network 120 may include a wired network, a wired local area network, a fiber optic network, a telecommunication network, a local area network, a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth TM Network, ZigBee TM In some embodiments, the network 120 may include one or more network access points. For example, the network 120 may include wired and / or wireless network access points, such as base stations and / or network switching points, through which one or more components of the image processing system 100 may access the network 120 for data and / or information exchange.

[0078] In some embodiments, a user can operate the image processing system 100 through a terminal device 130. The terminal device 130 may include a terminal 131, a tablet computer 132, a laptop computer 133, or the like, or a combination thereof. In some embodiments, the terminal 131 may include a smart home device, a wearable device, a mobile device, a virtual reality device, an augmented reality device, or the like. In some embodiments, the smart home device may include smart lighting devices, smart electrical control devices, smart monitoring devices, smart TVs, smart cameras, intercoms, or the like, or a combination thereof. In some embodiments, the wearable device may include a bracelet, footwear, glasses, a helmet, a watch, clothing, a backpack, smart accessories, or the like, or a combination thereof. In some embodiments, the terminal device 130 may be part of the processor 140.

[0079] The processor 140 can process data and / or information obtained from the image acquisition device 110, the terminal device 130 and / or the storage device 150. For example, the processor 140 can process the image data generated by the image acquisition device 110 to generate a target image with relatively high quality. In some embodiments, the processor 140 can be a server or a server group. The server group can be centralized or distributed. In some embodiments, the processor 140 can be local or remote. For example, the processor 140 can access information and / or data stored in the image acquisition device 110, the terminal device 130 and / or the storage device 150 through the network 120. For another example, the processor 140 can be directly connected to the image acquisition device 110, the terminal device 130 and / or the storage device 150 to access the stored information and / or data. In some embodiments, the processor 140 can be implemented on a cloud platform. For example, the cloud platform can include a private cloud, a public cloud, a community cloud, a distributed cloud, an interconnected cloud, a multi-cloud, etc., or a combination thereof. In some embodiments, the processor 140 can be provided by a computer having a plurality of computer systems such as Figure 2 The computing device 200 may be implemented with one or more components described in the embodiment.

[0080] The storage device 150 can store data, instructions, and / or any other information. In some embodiments, the storage device 150 can store data acquired from the terminal device 130, the image acquisition device 110, and / or the processor 140. In some embodiments, the storage device 150 can store data and / or instructions that the processor 140 can execute or use to execute the exemplary methods described in the present disclosure. In some embodiments, the storage device 150 can include a large-capacity storage device, a removable storage device, a volatile read-write memory, a read-only memory (ROM), and the like. In some embodiments, the storage device 150 can be executed on a cloud platform. For example, a cloud platform can include a private cloud, a public cloud, a community cloud, a distributed cloud, an interconnected cloud, a multi-cloud, and the like, or a combination thereof.

[0081] In some embodiments, the storage device 150 may be connected to the network 120 to communicate with one or more other components of the image processing system 100 (e.g., the processor 140, the terminal device 130, etc.). One or more components of the image processing system 100 may access data or instructions stored in the storage device 150 through the network 120. In some embodiments, the storage device 150 may be directly connected to or communicate with one or more other components of the image processing system 100 (e.g., the processor 140, the terminal device 130, etc.). In some embodiments, the storage device 150 may be part of the processor 140.

[0082] Figure 2is a schematic diagram of an exemplary computing device according to some embodiments of the present disclosure. Computing device 200 may be used to implement any component of image processing system 100 as described herein. For example, processor 140 and / or terminal device 130 may be implemented on computing device 200 via their respective hardware, software programs, firmware, or a combination thereof. Although only one such computing device is shown, for convenience, the computer functions associated with image processing system 100 described herein may be implemented in a distributed manner across multiple similar platforms to share the processing load.

[0083] like Figure 2 As shown, computing device 200 may include processor 210 , memory 220 , input / output (I / O) 230 , and communication port 240 .

[0084] Processor 210 can execute computer instructions (e.g., program code) and perform the functions of image processing system 100 (e.g., processor 140) according to the techniques described herein. Computer instructions may include, for example, routines, objects, components, data structures, procedures, modules, and functions that perform the specific functions described herein. For example, processor 210 can process image data obtained from any component of image processing system 100. In some embodiments, processor 210 may include one or more hardware processors.

[0085] For illustrative purposes only, only one processor is described in the computing device 200. However, it should be noted that the computing device 200 in the present disclosure may also include multiple processors. Therefore, the operations and / or method operations described in the present disclosure as being performed by one processor may also be performed jointly or individually by multiple processors. For example, if in the present disclosure, the processor of the computing device 200 performs operations A and B, it should be understood that operations A and B may also be performed jointly or individually by two or more different processors in the computing device 200 (e.g., the first processor performs operation A, the second processor performs operation B, or the first and second processors jointly perform operations A and B).

[0086] Memory 220 may store data / information obtained from any component of image processing system 100. In some embodiments, memory 220 may include a mass storage device, a removable storage device, a volatile read-write memory, a read-only memory (ROM), etc., or any combination thereof.

[0087] In some embodiments, the memory 220 may store one or more programs and / or instructions to execute the exemplary methods described in this disclosure. For example, the memory 220 may store a program for the processor 140 to process image data generated by the image acquisition device 110.

[0088] I / O 230 may input and / or output signals, data, information, etc. In some embodiments, I / O 230 may enable a user to interact with image processing system 100 (e.g., processor 140). In some embodiments, I / O 230 may include input devices and output devices. Examples of input devices may include a keyboard, mouse, touch screen, microphone, etc., or a combination thereof.

[0089] The communication port 240 can be connected to a network to facilitate data communication. The communication port 240 can establish a connection between the processor 140 and the image acquisition device 110, the terminal device 130, and / or the storage device 150. The connection can be a wired connection, a wireless connection, any other communication connection capable of transmitting and / or receiving data, and / or any combination of these connections. In some embodiments, the communication port 240 can be and / or include a standardized communication port, such as RS232, RS485, etc. In some embodiments, the communication port 240 can be a specially designed communication port. For example, the communication port 240 can be designed according to the Digital Imaging and Communications in Medicine (DICOM) protocol.

[0090] Figure 3 is a block diagram of an exemplary terminal device according to some embodiments of the present disclosure.

[0091] like Figure 3 As shown, the terminal device 300 may include a communication unit 310, a display unit 320, a graphics processing unit (GPU) 330, a central processing unit (CPU) 340, an input / output (I / O) 350, a memory 360, a storage unit 370, etc. In some embodiments, the terminal device 300 may also include any other suitable components, including but not limited to a system bus or a controller (not shown). In some embodiments, an operating system 361 (e.g., iOS 11) may be used. TM 、Android TM , Windows TM ) and one or more applications (apps) 362 can be loaded from storage unit 370 into memory 360 for execution by CPU 340. Applications 362 can include a browser or any other suitable application for receiving and presenting information related to imaging, image processing, or other information from image processing system 100 (e.g., processor 140). User interaction with the information stream can be achieved through I / O 350 and provided to processor 140 and / or other components of image processing system 100 via network 120. In some embodiments, a user can input parameters to image processing system 100 through terminal device 300.

[0092] In order to implement the various modules, units and their functions described above, a computer hardware platform may be used as one or more elements (e.g., Figure 1 The computer with the user interface can be used as a personal computer (PC) or other type of workstation or terminal device. After being properly programmed, the computer with the user interface can be used as a server. It is assumed that those skilled in the art will also be familiar with such structure, procedures or general operation of such type of computing devices.

[0093] Figure 4 FIG is a schematic diagram of an exemplary processor according to some embodiments of the present disclosure. Figure 4 As shown, the processor 140 may include an acquisition module 410 , an initial image generation module 420 , and a target image generation module 430 .

[0094] The acquisition module 410 may be configured to acquire information and / or data from one or more components of the image processing system 100. In some embodiments, the acquisition module 410 may acquire image data from the storage device 150 or the image acquisition device 110. As used herein, image data may refer to raw data (e.g., one or more raw images) acquired by the image acquisition device 110. Further description of acquiring image data may be found elsewhere in this disclosure (e.g., Figure 5 In some embodiments, the acquisition module 410 may acquire the image processing model from the storage device 150.

[0095] The initial image generation module 420 can generate an initial image. In some embodiments, the initial image generation module 420 can determine the initial image by performing a filtering operation on the image data. As an example only, the initial image generation module 420 can determine the initial image by performing a Wiener filter on one or more original images. More description of generating the initial image can be found elsewhere in this disclosure (e.g., Figure 5 520 in step 520).

[0096] The target image generation module 430 can generate a target image based on the initial image using an image processing model according to an optimization algorithm. The image processing model can include a first sub-model and a second sub-model. The first sub-model can be configured to determine a first optimization term related to the likelihood term of the objective function. The second sub-model can be configured to determine a second optimization term related to the regularization term of the objective function. The second sub-model can be a trained machine learning model. More descriptions about generating a target image can be found elsewhere in this disclosure (e.g., Figure 5 530 in step 530).

[0097] It should be noted that the above description of the modules of processor 140 is for illustrative purposes only and is not intended to limit the content of the present disclosure. In some embodiments, one or more modules may be added or omitted in processor 140. In some embodiments, one or more modules in processor 140 may be integrated into a single module to perform the functions of one or more modules.

[0098] Figure 5 is an exemplary flow chart of an image processing method according to some embodiments of the present disclosure. In some embodiments, process 500 can be executed by image processing system 100. For example, process 500 can be implemented as an instruction set (e.g., an application) stored in a storage device (e.g., storage device 150, memory 220, and / or storage unit 370). In some embodiments, the process can be implemented by one or more additional operations not described and / or one or more operations not discussed. In addition, as Figure 5 The order in which the operations of the processes are illustrated and described below is not meant to be limiting.

[0099] In step 510 , the processor 140 (eg, the acquisition module 410 ) may acquire image data generated by an image acquisition device.

[0100] In some embodiments, the image data herein may refer to raw data (e.g., one or more raw images) acquired by the image acquisition device 110. The raw images may have a relatively low signal-to-noise ratio (SNR) or be partially damaged, etc. By way of example only, for a SIM system, the image data may include one or more sets of raw images acquired by the SIM system. Each set of raw images may include multiple raw images (e.g., 9 raw images, 15 raw images) corresponding to different phases and / or directions of a sinusoidal illumination pattern applied to an object (e.g., a cell sample). In other words, the SIM system may acquire multiple raw images at different phases and / or directions.

[0101] In some embodiments, the processor 140 may obtain image data from one or more components of the image processing system 100. For example, the image acquisition device 110 may acquire and / or generate image data and store the image data in the storage device 150. The processor 140 may retrieve and / or obtain the image data from the storage device 150. For another example, the processor 140 may obtain the image data directly from the image acquisition device 110.

[0102] In step 520 , the processor 140 (eg, the initial image generation module 420 ) may generate an initial image by processing the image data.

[0103] In some embodiments, the processor 140 may determine an initial image (e.g., an SR image) by filtering the image data. Exemplarily, the filtering operation may include Wiener filtering, inverse filtering, least squares filtering, etc., or any combination thereof. For example, for each group of original images of the image data, the processor 140 may generate an image stack (i.e., an initial image) by performing Wiener filtering on multiple original images in the group of original images. Specifically, if each group of original images includes 9 original images, the processor 140 may combine the 9 original images in the group of original images into an initial image. The initial image may include information in each of the 9 original images and have a higher resolution than each of the 9 original images. In some embodiments, the filtering operation may be omitted. For example, for a camera phone system, the image data may include only one original image, and the processor 140 may designate the only original image as the initial image.

[0104] At step 530 , the processor 140 (eg, the target image generation module 430 ) may generate a target image by processing the initial image according to the optimization algorithm using the image processing model.

[0105] In some embodiments, the processor 140 may obtain an objective function for generating a target image. The objective function may include a likelihood term and a regularization term. The image processing model may be configured to perform multiple iterations to minimize the result of the objective function. The target image may be an optimization term based on the initial image.

[0106] For example, the objective function can be expressed as the following formula (1) Where f is the target image; g is the components of different orders obtained by the band separation operation (also called "spectral separation operation"); D(f,g) represents the likelihood term; R(f) represents the regularization term; and λ is the weight representing the regularization term. As an example only, the objective function can be solved using the gradient descent algorithm according to the following formula (2):

[0107] In some embodiments, the image processing model is a trained machine learning model. The image processing model can be stored in the storage device 150 and can be retrieved and used by the processor 140. Various parts of the image processing model can be configured to perform different processing operations. For example, the image processing model can include a first sub-model and a second sub-model. The first sub-model can be configured to determine a first optimization term related to the likelihood term of the objective function. The second sub-model can be configured to determine a second optimization term related to the regularization term of the objective function. The second sub-model can also be a trained machine learning model. More details about the training process for obtaining the image processing model can be found elsewhere in this specification, for example Figure 6 .

[0108] In some embodiments, the optimization algorithm used to generate the target image based on the initial image may include a direct Fourier transform (DFT) algorithm, a filtered back projection (FBP) algorithm, an algebraic reconstruction technique (ART), a simultaneous iterative reconstruction technique (SIRT), a maximum entropy (ME) method, etc. The first optimization term and the second optimization term may be determined based on the optimization algorithm. For example, the optimization algorithm may be a gradient descent algorithm. The first optimization term may be a derivative of the likelihood term, and the second optimization term may be a derivative of the regularization term.

[0109] In order to generate a target image using the image processing model, the processor 140 may perform multiple iterative operations based on the initial image. In the first iterative operation, the first sub-model may be configured to determine a first intermediate optimization term related to the likelihood term based on the initial image; the second sub-model may be configured to determine a second intermediate optimization term related to the regularization term based on the initial image. The image processing model may also include a third sub-model, which is configured to generate an intermediate image based on the first intermediate optimization term, the second intermediate optimization term and the initial image. The processor 140 may use the image processing model to perform multiple consecutive iterative operations, updating the intermediate image in a manner similar to the first iterative operation until a termination condition is met. When the processor 140 determines that the termination condition is met, the intermediate image in the latest iterative operation may be determined as the target image. Alternatively, the processor 140 may further process the intermediate image in the latest iterative operation to obtain the target image. Such processing may include, but is not limited to, adjusting the size of the intermediate image to make it suitable for display on the screen, automatically adding one or more labels (e.g., a ruler), etc.

[0110] In some embodiments, the termination condition may be related to the value of the objective function. For example, if the result of the objective function is minimum or less than a threshold value (e.g., a constant), the termination condition may be considered satisfied. For another example, if the value of the objective function converges, the termination condition may be considered satisfied. In some embodiments, if the change in the value of the objective function in two or more consecutive iterations is equal to or less than a threshold value (e.g., a constant), convergence may be considered to have occurred, and the termination condition may be satisfied. In some embodiments, if the difference between the result of the objective function (e.g., the value of the objective function) and the target value is equal to or less than a threshold value (e.g., a constant), convergence may be considered to have occurred. In some embodiments, the termination condition may be related to the number of iterations (count) of the objective function. For example, when a specified number of iterations (or count) T iterations have been performed, the termination condition may be considered to be satisfied. In some embodiments, the termination condition may be related to the iteration time of the objective function (e.g., the length of time to perform the first iteration). For example, if the iteration time of the objective function is greater than a threshold value (e.g., a constant), the termination condition may be considered to have been satisfied.

[0111] In some embodiments, at least one parameter related to the use of the image processing model can be set according to a default setting or based on a value specified by a user of the image processing system 100. For example, the weight λ of the regularization term in the objective function shown in Formula (1) and / or the numerical value of the number of iterations T can be selected according to the specific situation. Adjustment of these parameters can help improve the image quality of the target image. For example, the processor can first generate a target image based on the default values ​​of λ and T. After presenting the target image to the user, the user can evaluate the image quality of the target image. If the user determines that the image quality is not satisfactory (for example, there is still some noise in the target image), the user can manually adjust the value of λ and / or T. The processor 140 can regenerate the target image based on the adjusted value of λ and / or T. In some embodiments, the storage device 150 of the image processing system 100 may store multiple sets of values ​​of λ and T, for example, (λ1, T1), ((λ2, T2), (λ3, T3), etc. The processor 140 can generate multiple target images based on multiple sets of values ​​of λ and T. These target images can be presented to the user, and the user can select one of the target images with the highest image quality for further observation or analysis.

[0112] In some embodiments, the likelihood term can be determined based on the imaging principle (or a physical model reflecting the imaging principle) of the image acquisition device 110. For ease of illustration, the following description relates to the reconstruction of SIM images. It should be noted that process 500 can also be applied to the reconstruction of other types of images.

[0113] When the object being imaged has a finite size, there may be a unique analytical function that coincides within the bandwidth-limited spectral band of the optical transfer function (OTF) of the image acquisition device 110, so that the complete object can be reconstructed by extrapolating the observed spectrum. First, the illumination parameters can be estimated based on the image data generated or collected by the image acquisition device 110. SR spectral components of different orders can be obtained through the frequency band separation operation. For example, in the frequency domain, the SR spectral components can be expressed using the following formula (3): G d,n (k) = S(kP d,n )O(k),(3) Where d and n represent the direction and order of the illumination pattern, respectively; S(k) refers to the spectrum of the actual fluorescence distribution of the object; P d,n refers to the pattern wave vector of the illumination pattern; O(k) refers to the OTF of the image acquisition device 110.

[0114] Formula (2) can be transformed in the spatial domain to obtain the following formula (4): g d,n (r)=[t(r)×s(r)]*H(r),(4) Where s(r) refers to the spatial distribution of the actual fluorescence distribution; H(r) refers to the point spread function (PSF) of the image acquisition device 110 obtained by performing an inverse Fourier transform operation on the OTF; and t(r) is the phase matrix used to shift the spectrum s(r). In some embodiments, t(r) can be expressed using the following formula (5): t d,n (r) = e j2πP_(d,n)r ,(5) Here, j is the imaginary unit.

[0115] According to formula (3), if the pattern wave vector, starting phase and modulation depth of the illumination pattern are known, the components g of different orders modulated by OTF can be obtained from the target image. Therefore, the likelihood term can be based on g obtained from the image data generated by the image acquisition device 110. d,n In some embodiments, the likelihood term can be expressed using the following formula (6):

[0116] For ease of calculation, formula (6) can be converted to the following formula (7): Among them, F and F -1 Represent the Fourier transform operation and the inverse Fourier transform operation respectively.

[0117] The derivative of D(f,g) can be expressed as follows: Here, the superscript H represents the conjugate transpose.

[0118] In some embodiments, a regularization term determination model can be used to determine the regularization term, which can be a trained machine learning model. For example, an initial image can be input into the regularization term determination model, and the regularization term determination model can output an image representing the regularization term. As an example only, the regularization term determination model can determine the regularization term based on full depth variation (TDV) regularization, Tikhonov regularization, total variation (TV) regularization or sparse regularization, etc., which is not limited by the present disclosure. In some embodiments, the regularization term determination model can be part of the second sub-model. The determination of the second optimization term (e.g., derivative) can be performed by another part of the second sub-model based on the regularization term. For example, Figure 7B The exemplary structure of the second sub-model based on TDV regularization is shown in FIG. The structure of the regularization term determining model can be represented by Figure 7B The upper part of the model structure is shown.

[0119] More details on the structure of the second submodel can be found elsewhere, e.g., in Figure 7B and its description.

[0120] As an example only, the TDV regularization term can be expressed using the following formula (9): R(f)=w T N(Kf),(9) Among them, K is the convolution kernel with zero mean constraint; N refers to the convolutional neural network; w is a weight vector.

[0121] It should be noted that the above description of process 500 is for illustrative purposes only and is not intended to limit the scope of the present disclosure. In some embodiments, process 500 may include one or more additional operations. For example, an additional operation for displaying the target image may be added after step 530. For another example, an additional operation for pre-processing the initial image may be added after step 520.

[0122] Figure 6600 is a flowchart of an exemplary process for acquiring an image processing model through training operations according to some embodiments of the present disclosure. In some embodiments, process 600 can be performed by image processing system 100. For example, process 600 can be implemented as an instruction set (e.g., an application) stored in a storage device (e.g., storage device 150, memory 220, and / or storage unit 370). In some embodiments, the process can be implemented by one or more additional operations not described and / or one or more operations not discussed. In addition, as Figure 6 The order in which the operations of the processes are illustrated and described below is not intended to be limiting.

[0123] In step 610 , the processor 140 may acquire a plurality of training data sets, each of which includes a sample initial image and a sample optimized image.

[0124] In some embodiments, the signal-to-noise ratio of the sample optimized image can be higher than the signal-to-noise ratio of the corresponding sample initial image in the same training dataset. For example, the sample initial image can be an image generated based on image data generated by the image acquisition device 110, which may include noise and artifacts. The corresponding sample optimized image can be generated by reducing noise and artifacts in the sample initial image using various techniques.

[0125] For another example, the sample optimized image can be a simulated SIM image without noise (or the noise can be ignored). The corresponding sample initial image can be a simulated SIM image including noise. Specifically, the red, green, and blue (RGB) image can be converted into a grayscale image. An edge detection operation can be performed on the grayscale image to obtain an edge image. The detected edges can be determined as simulated fluorescence distributions. The simulated structured light can be applied to the edge image, and then a convolution operation can be performed using the point spread function of the image acquisition device 110. Then, a downsampling operation can be performed on the resulting image to obtain a sample optimized image. The corresponding sample initial image can be generated by adding a simulated uneven fluorescence background and noise to the sample optimized image.

[0126] In step 630 , the processor 140 may use a plurality of training data sets to train an initial model to obtain an image processing model.

[0127] The initial model can be trained using various methods, such as a gradient descent algorithm, which is not limited by the present disclosure. During the training process, the model parameters of the initial model are updated to obtain an image processing model. The purpose of updating the model parameters is to minimize the difference between the sample optimized image and the optimized image output by the initial model based on the sample initial image.

[0128] As previously described, the image processing model (or initial model) may include a first sub-model and a second sub-model. The first sub-model may be configured to determine a first optimization term associated with a likelihood term of the objective function. The second sub-model may be configured to determine a second optimization term associated with a regularization term of the objective function. During training, model parameters associated with the second sub-model may be updated, while model parameters associated with the first sub-model remain unchanged.

[0129] In some embodiments, a plurality of image processing models corresponding to different types of training data sets may be stored in the storage device 150. For example, different types of training data sets may include training data sets corresponding to different types of objects, training data sets corresponding to different imaging parameters (e.g., lighting parameters, exposure parameters), training data sets corresponding to different levels of signal-to-noise ratios, training data sets corresponding to different types of image acquisition devices, and the like. By way of example only, a plurality of training sets corresponding to specific types of objects (e.g., actin filaments, nuclear pore complex structures, mitochondrial cristae, etc.) may be used to train a specific image processing model. The processor 140 may obtain an image processing model corresponding to the type of imaging object. For example, the processor 140 may use image recognition technology based on the original image captured by the image acquisition device 110 to identify the type of imaging object. For another example, the processor 140 may identify the type of imaging object based on object information input by the user.

[0130] It should be noted that the above description of process 600 is for illustrative purposes only and is not intended to limit the scope of the present disclosure. In some embodiments, process 500 may include one or more additional operations. For example, additional operations may be added after step 620 to test the performance of the image processing model. If the performance of the image processing model does not meet the user's requirements, further training operations may be performed on the image processing model, and / or a new training set may be used for the training process.

[0131] Figure 7A FIG2 is a schematic diagram of generating a target image according to some embodiments of the present disclosure. For ease of explanation, the target image is a SIM image.

[0132] like Figure 7A As shown, the original image can be subjected to a frequency band separation operation to determine the spectral components SR of different orders. The original image may include multiple images corresponding to different directions and different phases of the illumination pattern. The initial image f can be generated based on the original image 0 . It can be found in the initial image f 0 The first iteration operation is performed to obtain the intermediate image f 1 Multiple consecutive iterations can be performed to update the intermediate image f 1 , to obtain the final target image fT (i.e., the output image of the Tth round). Here, T refers to the number of iterations. In each iteration, the image processing model determines the derivative of the likelihood term and the derivative of the regularization term. The intermediate image f is updated based on the derivative of the likelihood term and the derivative of the regularization term. 1 、f 2 ,…,f T .

[0133] Figure 7B is a schematic diagram of an exemplary structure of a second sub-model according to some embodiments of the present disclosure. For illustrative purposes, Figure 7B The second sub-model shown in is a deep learning network based on TDV regularization.

[0134] The second sub-model may include two parts. The first part of the second sub-model (e.g., Figure 7B The upper part shown in ) is configured to determine the regularization term of the objective function. The second part of the second sub-model (e.g., Figure 7B The lower part shown in ) is configured to determine an optimization term corresponding to the regularization term (e.g., a derivative of the regularization term).

[0135] The first part of the second sub-model consists of three U-Net structures, each consisting of five micro-blocks, including a residual structure. After the input image f is fed into the first part of the second sub-model, the regularization term R(f) is obtained through the first part. To determine the derivative of R(f), an image matrix with each element equal to 1 is fed into the second part of the second sub-model. The reverse calculation is performed based on the structure of the first part of the second sub-model. During the reverse calculation of the first part of the second sub-model, the convolutional layers of the second part are modified to transposed convolutional layers, and the activation function layers of the second part are modified to the derivatives of the activation functions of the first part.

[0136] The present disclosure is further described with reference to the following examples, which should not be construed as limiting the scope of the present disclosure.

[0137] In this disclosure, an exemplary TDV reconstruction method is implemented by interpretably combining a camera noise model with deep learning (DL) to improve images from fluorescence microscopy (FM) (e.g., structured illumination microscopy (SIM), spinning disk confocal microscopy (CSDM), widefield microscopy (WFM), Fourier light-field microscopy (FLFM), etc.). In some embodiments, the TDV reconstruction method can be integrated into a SIM image enhancement framework to achieve high-contrast, high signal continuity, high structural integrity, artifact-free, long-duration (e.g., 3.5 hours), and high-resolution (e.g., sub-45 nm resolution) SIM imaging. In some embodiments, TDV-FM can be used to process 3D-SIM images of actin filaments, demonstrating that TDV-FM can achieve superior restoration comparable to three other state-of-the-art methods. In some embodiments, TDV-FM can be used to enhance degraded 5D 3D-SIM data, achieving artifact-free and high-fidelity SR reconstruction. In some embodiments, TDV-FM can be used for reconstruction in CSDM, achieving high signal-to-noise ratio and high-contrast 2D / 3D time-lapse imaging without signal time leakage and loss. The study showed that TDV-FM, as a universal reconstruction method, can completely eliminate noise and artifacts from long-term WFM 2D and FLFM 3D imaging.

[0138] An exemplary principle of TDV-FM is described below.

[0139] In some embodiments, the FM image reconstruction process can be converted into a hybrid optimization function that includes a fidelity term (e.g., a fidelity term D(f,g) based on a camera noise model) and a regularization term (e.g., a regularization term R(f,θ) based on deep learning (DL)): Among them, f TDV is the reconstructed TDV image, g is the degraded data, and θ is the network weight. The iterative optimization reconstruction can be obtained using the gradient descent algorithm: Where T is the number of iterations, is the gradient operator (see Figure 13A ).

[0140] An exemplary DL regularization term network can be designed (see Figure 13B In some embodiments, the DL regularization term network may include: (1) a convolution kernel K; (2) a multi-scale convolutional neural network N including multiple blocks (see Figure 13C ); (3) convolution kernel w; (4) potential function ψ. In some embodiments, the partial derivative of the regularization term It can be computed by inverting the network, converting the convolutional layer into a transposed convolutional layer with the same convolution kernel, converting the activation function into the gradient of the activation function, converting the potential function into the gradient of the potential function, and converting multiple blocks into multiple transposed blocks (see Figures 13B-13C After completing the DL regularization term network training, TDV-FM is able to reconstruct with high quality (see Figure 13D ).

[0141] Figures 13A-13D is a schematic diagram of another exemplary image processing method according to some embodiments of the present disclosure. Figure 13A Schematic diagram showing the iterative optimization reconstruction process of TDV-FM and its training phase. Figure 13B The architecture of the DL-based regularization term and its gradient are shown. Figure 13C Shown Figure 13B The architecture of the blocks and T-blocks in [1]. Conv.: convolutional layer; Pote.: potential function; Acti.: activation layer; T-Block: transposed block; T-Conv.: transposed convolutional layer; G-Pote.: potential function gradient; G-Acti.: activation layer gradient. Figure 13D Schematic diagram showing the inference stage of TDV-FM.

[0142] Exemplary dataset acquisition, preprocessing, and DL regularization term training are described below.

[0143] For TIRF-SIM imaging, similar or identical settings in Hessian-SIM equipped with a wide-angle objective (×150 / 1.45 oil, Olympus) can be used. For 3D-SIM imaging, similar or identical settings as the HiS-SIM commercial system can be used. For CSDM imaging, CSDM may be a commercial system based on an inverted fluorescence microscope (IX81, Olympus) equipped with a wide-angle objective (×100 / 1.3 oil lens, Olympus) and a scanning confocal system (CSU-X1, Yokogawa). Four laser beams of 405nm, 488nm, 561nm and 647nm can be combined with CSDM. Images can be collected by an sCMOS camera (C14440-20UP, Hamamatsu) or an EMCCD camera (iXon3 897, Andor). For FLFM imaging, similar or identical settings in HR-FLFM can be used. In order to obtain low SNR raw images and corresponding ground-truth (GT) images of TIRF-SIM, 3D-SIM, CSDM, and FLFM for training the TDV-FM DL regularization term, the sample can be imaged with high illumination laser intensity and long exposure time as the GT raw image, and noise can be added to the GT raw image based on the fluorescence microscope camera imaging noise model to serve as the low SNR raw image.

[0144] For each imaging modality and sample, approximately 30 cells can be imaged, and the images can be preprocessed to obtain raw data pairs and corresponding ground-truth images. Next, these image pairs can be divided into training, validation, and test sets. Random crops, quarter rotations, and / or horizontal / vertical flips can then be applied to further enrich the training dataset. TDV-FM can be trained using the Adam optimizer with a learning rate set to 1×10 -4 A combination of mean square error (MSE) loss and SSIM loss can be used as the objective function, which is defined as: L(x,y)=MSE(x,y)+k[1-SSIM(x,y)],(12) Where x and y represent the predicted image and the high-quality GT image, respectively, and k is a weighting coefficient that balances the SSIM and MSE losses. In some embodiments, k can be set to 0.2 based on experience.

[0145] Exemplary evaluation metric calculations and statistical analyses are described below.

[0146] To quantitatively evaluate the performance of TDV-FM and other computational 3D-SIM methods, the 3D PSNR and 3D SSIM between the GT3D data and the reconstructed 3D data can be calculated. The GT 3D data y can first be normalized to the range [0, 1], and then a linear transformation can be applied to the reconstructed 3D data x to match its dynamic range with y:

[0147] Then, we can normalize the GT 3D y and linearly transform the 3D 3D PSNR and 3D SSIM are calculated between the two images. Since there is no GT in real imaging data, the fluorescence intensity variance in background regions (e.g., mesh regions within actin filaments) can be used to assess the magnitude of artifacts, while the fluorescence intensity differences along the actin filaments can be used to assess signal continuity. Furthermore, the integrity of the actin signal can be assessed based on the actin filament length and density calculated according to the protocol in BF SIM. Furthermore, the resolution of different restorations can be assessed using the full width at half maximum (FWHM) values ​​measured along the Hessian-SIM protocol and the minimum FRC (Fourier ring correlation) value in the rFRC (rolling FRC) plot calculated by PANEL. Quantitative data can be presented as boxplots (center line, mean; range, 75% and 25%; stretch lines, maximum and minimum values).

[0148] An exemplary FLFM numerical point spread function (PSF) generation is described below.

[0149] Figure 17A An exemplary FLFM system can be shown in . The wave function at the primitive image plane (NIP) can be derived using vector Debye theory: in, is the NIP coordinate; are the three-dimensional coordinates of the sample; M, NA, and f obj are the objective lens magnification, numerical aperture, and focal length; λ em is the emission wavelength; α is the critical angle of total internal reflection, defined as formula (16); θ and are the refraction angle (objective side) and the incident angle (sample side) at the interface between the immersion medium (refractive index = 11) and the sample solution (refractive index = n2); Φ(·) is the aberration function defined as formula (17); l is the normal focus position; τ s and τ p is the Fresnel transmission coefficient, defined as formula (18) and formula (19) respectively; u and v are the normalized radial and axial coordinates, defined as formula (20) and formula (21) respectively; J0 and J2 are the first type 0 th and 2th Bessel function. α=min[sin -1 (NA / n1),sin -1 (n2 / n1)],(16)

[0150] Since the fluorescent group has the characteristic of isotropy when emitting photons, the p x =p y = 0 for easy calculation, it can be deduced that the light field only points to p z Direction situation.

[0151] In some embodiments, The optical Fourier transform can be performed by Fourier lens to And modulated by MLA where φ(·) is the MLA transfer function defined as Equation (22), Can be MLA coordinates. Where amp(·) is the MLA amplitude mask function, comb(·) represents the MLA comb function, d and f mla are the diameter and focal length of a single microlens of MLA, Represents the convolution operator.

[0152] In some embodiments, Fresnel propagation can be used to simulate the light field propagation process from the MLA to the camera: in, and are the spatial and frequency domain coordinates of the camera plane, respectively, and IFT(·) is the inverse Fourier transform operator.

[0153] In some embodiments, the numerical PSF of the FLFM can be expressed as: where δ(·) is the Dirac function.

[0154] An exemplary FLFM three-dimensional (3D) reconstruction method is described below.

[0155] For the fluorescence intensity distribution The sample three-dimensional data and the corresponding observation original image obtained by FLFM The data can be calculated as:

[0156] In some embodiments, the matrix form of formula (25) can be expressed as: E=HS,(26) Among them, E, H and S are respectively composed of the FLFM observation image, FLFM-PSF ( Figure 17B ) and FLFM sample 3D determination. In some embodiments, FLFM 3D reconstruction can be converted into an optimization problem defined as formula (27), which can be iteratively solved by Richardson-Lucy (RL) deconvolution formula (28). where KL(·) is the KL divergence, (·) T is the transpose operator, k is the number of iterations, S (k) is the kth round of reconstruction.

[0157] In some embodiments, a hybrid PSF of an experimental PSF (providing intensity distribution) and a numerical PSF (giving spatial position) can be used for 3D reconstruction of FLFM ( Figure 17C ).

[0158] Figures 17A-17C An example diagram of a Fourier light field microscope (FLFM) according to some embodiments of the present disclosure is shown. Figure 17A Schematic diagram of the FLFM setup is shown. OBJ: objective lens; M: mirror; DM: beam splitter; MLA: microlens array; L1-L4: tube lens. Figure 17B Shown are the FLFM hybrid point spread function (hybrid PSF) distributions color-coded according to the distance from the focal plane. A magnified view of the boxed area corresponding to microlenses (i)-(iii) is on the right. Figure 17C An exemplary generation process of a hybrid PSF is shown. Figure 17B The left side is 10 μm and the right side is 2 μm.

[0159] An exemplary 3D-SIM imaging physical model is described below.

[0160] Compared with conventional 3D-WFM fluorescence microscopy, 3D-SIM can generate pattern illumination by the interference of three beams of light. To stimulate the sample three-dimensional Able to double the spatial and axial resolution. Regardless of imaging noise, the fluorescence emission distribution detected by the camera It can be expressed as: where r and n are the direction and phase indices of the illumination pattern, respectively, is the NIP coordinate, are the three-dimensional coordinates of the sample, is the NIP wave function calculated according to formula (15). Pattern illumination It can be calculated as: Where N is the normalization factor defined as formula (31), I o and I n are the intensities of obliquely incident and normally incident plane waves, λ ex is the excitation wavelength, θ is the incident polar angle, is the azimuth angle of the illumination pattern in the rth direction, ψ rn is the phase of the illumination pattern in the rth and Nth directions. Among them, M r and M n are the direction and phase number of the illumination pattern, respectively.

[0161] The matrix form of formula (29) can be expressed as: E rn =I rn S,(32) Among them, E rn and S are determined by the observed image and the sample in three dimensions, respectively. rn The three-dimensional distribution of structured light and PSF are determined by 3D-SIM.

[0162] Exemplary 3D-SIM illumination pattern parameter estimation and SR reconstruction are described below.

[0163] The parameters of the 3D-SIM illumination pattern can be estimated using autocorrelation and cross-correlation algorithms. In some embodiments, 3D-SIM SR reconstruction can be converted into a regularized least squares (LS) minimization problem defined as: Where R(S) is a regularization term added based on sample prior knowledge, and λ is the weight of the regularization term. For traditional 3D-SIM SR reconstruction, the Wiener regularization term can be widely used, and the corresponding minimization problem can be analytically solved using Wiener inverse filtering.

[0164] An exemplary CSDM imaging physics model is described below.

[0165] CSDM can use multiple focused laser beams To scan the sample and by the confocal circular aperture corresponding to The focusing process reduces the fluorescence intensity of each laser beam Received to camera. Fluorescence intensity It can be calculated as: in, is the NIP coordinate, are the three-dimensional coordinates of the sample, is the coordinate of the center of the focused laser beam, is the wave function at the NIP calculated as in formula (15). The effect of the confocal aperture is described by: where a is the radius of the confocal aperture projected onto the NIP.

[0166] If the Stokes shift between the excitation and emission wavelengths is neglected and it is assumed that the confocal aperture of the microscope does not significantly limit the lateral detection of light (i.e., the excitation intensity distribution decays to a negligible value before the effect of the confocal aperture becomes significant), then the matrix representation of equation (34) can be approximated simply by: E=HS,(36) Here, E and S are determined by the observed image and sample, respectively, and H is determined by the multi-focus laser beam pattern and PSF of the CSDM.

[0167] An exemplary CSDM deconvolution method is described below.

[0168] As a low-photon imaging technique, the fluorescence emission of CSDM can be approximately regarded as a Poisson process. Then, formula (36) can be transformed into: E~Poisson(HS),(37) Where Poisson(HS) is a Poisson distribution with parameters (mean or variance) equal to HS. The RL deconvolution algorithm can be derived by maximizing the posterior probability P(E|HS) using a multiplicative gradient-based algorithm, as shown in Equation (28).

[0169] An exemplary fluorescence microscopy camera imaging noise model is described below.

[0170] In EMCCD and sCMOS cameras, imaging noise can primarily consist of shot noise, thermal noise, and / or readout noise. Shot noise can originate from the photon detection process. Thermal noise and readout noise can originate from the electronics built into the detector chip. When a photon is detected on the camera sensor chip, the camera's analog-to-digital conversion unit (ADU) count output can follow a probability distribution that can be described by the convolution of a Poisson distribution and a Gaussian distribution, where the Poisson distribution can represent the shot noise of photon detection and the Gaussian distribution can be the result of readout noise. The conditional probability density function (CPDF) at a single pixel i of the camera can be described by the following formula: Among them, C i represents the specific count value obtained by the camera in this pixel, A is the normalization constant, and E i is the expected number of photoelectrons, k o is the amplification gain of pixel i, o o and var i denote the mean (bias) and variance of the readout noise at pixel i, respectively.

[0171] In order to facilitate subsequent derivation, the random variable C i The distribution of can be equivalently expressed as: C i =P i +G i +o i ,(39) Among them, P i Follows a Poisson distribution with mean and variance equal to k i E i : P i ~P(k i E i ),(40)

[0172] G i Follows a Gaussian distribution with a mean of zero and a variance equal to var i : G i ~N(0,var i ),(41)

[0173] In some embodiments, the new random variable Z i can be defined and expressed as:

[0174] If the mean (or variance) of the Poisson distribution is large, it can be approximated by a Gaussian distribution. Then Z i The distribution of can be approximately followed by a Gaussian distribution and can be approximately expressed as:

[0175] For simplicity, ignore Z i E in variance i Item, then Z i The approximate CPDF of can be expressed as:

[0176] C i The approximate CPDF of can be obtained by replacing Z in formula (44) i Substitute into formula (42) to calculate and express it as:

[0177] Based on the above derivation, the imaging process of fluorescence microscopy (FM), including 3D-SIM, CSDM, and FLFM, can be regarded as a linear model defined by equations (32), (36), and (26), which can be unified into the following form: E=H FM S,(46) Among them, H FM is the FM linear physical model, S is the spatial distribution of the sample phosphor to be reconstructed, and E is the camera observation value (or the expected number of photoelectrons). In some embodiments, the spatial distribution of the sample phosphor S can be reconstructed by maximizing the posterior probability, which is defined as: Among them, P(S|C i ) can be calculated using Bayes' rule and expressed as:

[0178] In some embodiments, the reconstruction optimization function J(S) can be defined as the negative logarithm of P(S|C) and calculated by combining equations (45), (46), (47), and (48). The simplified form of J(S) can be expressed as the following linear inverse problem, defined as: Among them, the first part of J(S) represents the fidelity term, and the second part of J(S) (i.e., R(S)) represents the regularization term, which is defined as the negative logarithm of the prior probability density distribution function P(S) that represents the prior knowledge of the sample distribution.

[0179] An exemplary deep learning-based regularization term is described below.

[0180] An accurate regularization term that can reflect the complex statistical properties of the sample is essential for achieving high-quality FM reconstruction. Previous researchers have developed various mathematically carefully designed regularization terms, and the complexity of the fluorescent sample is only partially reflected in their formulas. Deep learning (DL) can have powerful feature extraction capabilities and can theoretically approximate the complex three-dimensional distribution of samples with minimal error. Therefore, the DL-based regularization term R(S,θ) can be used to replace the traditional manually designed regularization term R(S) as a more appropriate statistical model of the sample distribution, where θ is the learned DL network weight. At the same time, in order to facilitate the solution of subsequent optimization problems, the pseudo-inverse function of the FM physical model is used. It can be applied to the two terms before and after the minus sign in the fidelity term. In some embodiments, formula (19) can be rewritten as a hybrid optimization problem of the fidelity term based on the camera noise model and the regularization term based on DL: Among them, f TDV is the TDV reconstruction of the spatial distribution of the sample phosphor (i.e., S), and g is defined as traditional reconstruction.

[0181] In some embodiments, the inverse problem (Equation (50)) can be solved iteratively using a gradient descent algorithm as follows: Where T is the number of iterations, f T This is the Tth reconstruction. is the gradient operator. The fidelity term The partial derivative of can be calculated as:

[0182] Partial derivative of the regularization term Can be along Figures 13B-13C Process calculation in .

[0183] Exemplary CSDM synthetic data generation is described below.

[0184] The CSDM image of pancreatic zinc ion secretion can be modeled as a superposition of multiple PSFs of the imaging system. In some embodiments, the three-dimensional coordinates of the sample are converted to Set to (0,0,0) to generate the equivalent PSF of CSDM. In some embodiments, the apodization function defined as Equation (53) can be Applied to the PSF to avoid oscillation effects that are detrimental to subsequent DL regularization training. in, is the NIP coordinate, and R is the full width at half maximum (FWHM) of the PSF. The position and intensity maps of zinc ion secretion can be randomly generated. The zinc ion secretion mask can be generated by multiplying the position and intensity maps. The ground-truth image of zinc ion secretion can be synthesized by convolving the zinc ion secretion mask with the PSF. The degradation from the ground-truth image to the noisy image can be achieved by adding noise based on the fluorescence microscope camera imaging noise model, and can generate a good dataset for subsequent DL regularization training.

[0185] By interpretably combining the advantages of optical front-end (e.g., camera imaging noise model) and algorithmic back-end (e.g., DL) methods, TDV-FM can universally improve the quality of five-dimensional (5D) (xyz-time-color) fluorescence imaging of living cells. The advantages of the TDV-FM method may include: (1) TDV-FM is proposed based on the imaging model of fluorescence microscopy, and the corresponding iterative reconstruction process can be fully explained from a statistical perspective; (2) Compared with traditional algorithms that iteratively reconstruct images using artificial analysis models with certain assumptions, TDV-FM replaces the artificial analysis model with a more appropriate DL-based analysis model, which greatly improves the quality of the final reconstructed image; (3) TDV-FM has been demonstrated as a widely compatible and universal method in image reconstruction of various fluorescence imaging modalities.

[0186] In summary, TDV-FM provides a versatile and interpretable solution for noise-free, artifact-free, high-fidelity, high-contrast, and long-term (e.g., 3.5 hours) reconstruction of low signal-to-noise ratio images with high resolution (e.g., sub-45 nm resolution). Due to the reduction of photon dose and associated photon toxicity, the increased imaging speed, the extended imaging duration, and the color dimension, TDV-FM may be crucial for revealing various biological phenomena.

[0187] Figure 18 FIG. 1 is a block diagram of an exemplary image processing system according to some embodiments of the present disclosure. Figure 18 As shown, the image processing system 1800 may include an image data acquisition module 810 , an initial image generation module 1820 , and a target image generation module 1830 .

[0188] The image data acquisition module 810 can be configured to acquire multidimensional image data. The multidimensional image data can be acquired by a fluorescence microscope. In some embodiments, the image data acquisition module 810 can acquire multidimensional image data from the storage device 150 or the image acquisition device 110 (e.g., a fluorescence microscope). As used herein, multidimensional image data can refer to raw data (e.g., one or more raw three-dimensional images) acquired by the image acquisition device 110. More description of acquiring multidimensional image data can be found elsewhere in this disclosure (e.g., step 1910 and its description).

[0189] The initial image generation module 820 can be configured to generate an initial multidimensional image based on the multidimensional image data. In some embodiments, the initial multidimensional image generation module 820 can generate the initial multidimensional image by performing a filtering operation on the multidimensional image data. As an example only, the initial image generation module 820 can generate the initial multidimensional image by performing a Wiener filter on one or more original three-dimensional images. Further description of the generation of the initial multidimensional image can be found elsewhere in this disclosure (e.g., step 1920 and its description).

[0190] The target image generation module 830 can be configured to construct an objective function based on the acquisition process of the multidimensional image data, and / or generate the target image by performing one or more iterations on the initial multidimensional image. The objective function can include a fidelity term and a regularization term. The fidelity term can be related to the imaging model of the fluorescence microscope. The regularization term is determined by a regularization network. More descriptions of target image generation can be found elsewhere in this disclosure (e.g., step 1930 and its description).

[0191] It should be noted that the above description of the modules of image processing system 1800 is for illustrative purposes only and is not intended to limit the present invention. In some embodiments, one or more modules may be added or omitted from image processing system 1800. In some embodiments, one or more modules in image processing system 1800 may be assembled into a single module to perform the functions of one or more modules.

[0192] Figure 19 1900 is an exemplary flow chart of an image processing method according to some embodiments of the present disclosure. In some embodiments, one or more operations in process 1900 may be performed by a processing device (e.g., processor 140 or processor 210). For example, process 1900 may be implemented as an instruction set (e.g., an application) stored in a storage device (e.g., storage device 150). In some embodiments, the processing device may execute this set of instructions to implement one or more operations in process 1900.

[0193] In step 1910, multi-dimensional image data may be acquired.

[0194] Multidimensional image data may refer to raw data generated by an image acquisition device 110 (e.g., a fluorescence microscope). For example, the multidimensional image data may include one or more raw three-dimensional images acquired by 3D-SIM. The multidimensionality of the image data may include spatial dimensions, temporal dimensions, sample dimensions, etc., or any combination thereof. Further description of fluorescence microscopes may be found elsewhere in this disclosure (e.g., Figure 1 and its related descriptions).

[0195] By way of example only, for 3D-SIM, the multidimensional image data may include multiple sets of 3D raw image data. Each acquired 3D raw image data set may include or correspond to multiple raw images (e.g., 9 raw images, 15 raw images, etc.), which correspond to different phases and / or directions of a sinusoidal illumination pattern applied to an object (e.g., a cell sample). In other words, the structured illumination microscope may acquire multiple 3D raw images of different phases and / or directions to acquire multidimensional image data. It should be noted that in some other embodiments, the multidimensional image data may be acquired by CSDM, FLFM, or any other fluorescence microscope, and the present disclosure is not limited thereto.

[0196] In step 1920 , an initial multi-dimensional image may be generated based on the multi-dimensional image data.

[0197] The initial multidimensional image may be a 3D image acquired by processing (e.g., reconstruction or filtering) the multidimensional image data. In some embodiments, the processing device may determine the initial multidimensional image by performing a filtering operation on the multidimensional image data. In some embodiments, the filter may include a Wiener filter, an inverse filter, a least squares filter, or any combination thereof.

[0198] Continuing with the 3D-SIM example above, for each set of 3D original images in the multidimensional image data, the processing device may generate an image stack by performing a filtering operation (e.g., Wiener filtering) on ​​each 3D original image in the set to obtain an initial multidimensional image. Specifically, if each set of 3D original images includes nine 3D original images, the processing device may combine the nine 3D original images in the set to obtain an initial multidimensional image, such that the initial multidimensional image has a higher spatial resolution than the 3D original images.

[0199] In step 1930 , a target function may be constructed based on the acquisition process of the multidimensional image data, and / or a target image may be generated by performing one or more iterations on the initial multidimensional image according to the target function.

[0200] The acquisition process may reflect the imaging principle of the fluorescence microscope. The imaging principle of the fluorescence microscope can be expressed by the imaging physics model and / or imaging noise model of the fluorescence microscope. More descriptions of the imaging physics model and the imaging noise model can be found elsewhere in this disclosure (e.g., formulas (10)-(53) and their descriptions). In some embodiments, the objective function may include a fidelity term and a regularization term. In some embodiments, the regularization term may be determined by a regularization network.

[0201] The regularized network can be a DL model obtained by training using training samples. The training samples can include images with a relatively high signal-to-noise ratio. More description of obtaining training samples and regularized network training can be found in Figure 20 and its related descriptions.

[0202] In some embodiments, the input of the regularization network can be an initial multidimensional image (i.e., the initial multidimensional image can be input into the regularization network), where the fidelity term and / or the partial derivative of the fidelity term can be determined based on the imaging physics model of the fluorescence microscope and the initial multidimensional image. The initial multidimensional image can be input into a DL-based regularization network to determine the regularization term. By using a DL-based regularization network, in which the prior distribution of relatively high signal-to-noise ratio images in the training samples and the image properties corresponding to different objects in the training samples (e.g., spatiotemporal continuity, sparsity, etc.) are learned, the quality of the image processing results of the initial multidimensional image can be improved.

[0203] The objective function can represent a mathematical expression of the optimization goal achieved by model optimization (e.g., image processing) during the execution of the regularization network. In some embodiments, the fidelity term can be related to the imaging physical model of the fluorescence microscope. The fidelity term can represent the degree of consistency between the 3D image and the imaging physical model inherent to the fluorescence microscope. It should be understood that the fidelity term can be determined based on the imaging principle of the fluorescence microscope, and since there may be a variety of fluorescence microscopes, the imaging principle and the inherent imaging physical model of the fluorescence microscope may be different, and the expression of the fidelity term may also be different. For example, the imaging principles of fluorescence microscopes including 3D-SIM, CSDM and FLFM are different, and different fidelity terms can be used respectively.

[0204] The regularization term in the objective function can be determined by a DL-based regularization network. More descriptions of the regularization term and the DL-based regularization network can be found elsewhere in this disclosure (e.g., Figures 13A-13D , 20 and related descriptions).

[0205] In some embodiments, a minimized objective function may be used during the process of generating the target image, and one or more iterations may be performed. The iteration may be terminated if the number of iterations reaches a preset number (e.g., 500 or 1000 iterations), or if the objective function value obtained in the current iteration satisfies a preset condition (e.g., the objective function value converges or is less than a preset threshold).

[0206] In some embodiments, the objective function may be similar to equation (50), which may be expressed as: Among them, f TDVrepresents the target image generated after iteration, f represents the initial multidimensional image, represents the fidelity term, and represents a conventional reconstructed image; R(f,θ) represents a regularization term, and θ represents the weight of a DL-based regularization network. In some embodiments, the conventional reconstructed image may be a preset (e.g., manually preset) image associated with a fluorescence microscope. In some embodiments, the conventional reconstructed image may reflect a reconstructed image associated with the imaging principle of a fluorescence microscope and its inherent imaging physics model.

[0207] In some embodiments, one or more weighting factors may be added to the fidelity term and / or the regularization term to reflect the focus of the result on the fidelity term and / or the regularization term. This specification does not impose any limitation on the weighting method.

[0208] In some embodiments, an optimization algorithm may be used to perform one or more iterations to minimize an objective function, thereby adjusting the 3D image. By way of example only, the optimization algorithm may include a direct Fourier transform (DFT) algorithm, a filtered back projection (FBP) algorithm, an algebraic reconstruction technique (ART), a simultaneous iterative reconstruction technique (SIRT), a maximum entropy (ME) process, and the like. In some embodiments, at least one of a fidelity term and a regularization term may be determined based on the optimization algorithm.

[0209] It should be noted that the above description of process 1900 is for illustrative purposes only and is not intended to limit the scope of application of the present disclosure.

[0210] refer to Figure 13B , an exemplary regularization network may include a multi-scale convolutional neural network. In some embodiments, the regularization network may include a first convolutional layer, a multi-scale convolutional neural network, a second convolutional layer, a potential function, and an all-1 convolutional layer. In some embodiments, the regularization network may further include an activation function (i.e., an activation layer). By setting a multi-scale convolutional neural network, the spatial resolution of the 3D image input to the regularization network (e.g., the initial multi-dimensional image, the output image of the previous iteration (i.e., the previous iteration)) may be inconsistent, and therefore, no additional normalization operation is required to unify the spatial resolution of the 3D image input to the regularization network.

[0211] Continuing with the previously disclosed 3D-SIM example, since a 3D image includes in-plane information in the x-axis and y-axis directions relative to a two-dimensional image, and there is additional information about the z-axis direction, using a multi-scale convolutional neural network can make the z-axis direction more continuous during the reconstruction of the 3D image (i.e., more features between multiple layers of two-dimensional images) and improve the fidelity of the reconstruction process of each layer. In some embodiments, the multi-scale convolutional neural network with a regularization term can include one or more convolution kernels with different receptive fields, and the convolution kernels with different receptive fields can be configured to extract features between multiple layers of the initial multidimensional image, thereby obtaining one or more feature images of a multidimensional image having a spatial resolution lower than the spatial resolution of the input multidimensional image (e.g., the initial multidimensional image, the output image of the previous iteration).

[0212] In some embodiments, one or more convolutions may be performed on the input multidimensional image or feature image based on one or more convolution kernels with different receptive fields. By way of example only, in response to the input of the multi-scale convolutional neural network being a 3D image, the size and / or number of convolution kernels with different receptive fields may vary depending on the spatial resolution of the 3D image and the structure of the multi-scale convolutional network (for example, the size of the convolution kernels with different receptive fields may be 3×3×3 or 5×5×5, etc.).

[0213] In some embodiments, each iteration of one or more iterations may include: determining the partial derivative of the fidelity term based on the traditional reconstructed image and the output image of the previous iteration of the current iteration; determining the partial derivative of the regularization term based on the output image of the previous iteration; and determining the output image of the current iteration based on the partial derivative of the fidelity term, the partial derivative of the regularization term, and the output image of the previous iteration.

[0214] Taking the formula (54) in the previous description as an example, in some embodiments, in response to the current iteration being the (T+1)th iteration, each iteration of the multiple iterations performed on the initial multidimensional image can be expressed as: Among them, f T is the 3D image output from the Tth iteration (i.e., the previous iteration), f T+1 is the 3D image obtained from the current iteration. represents the partial derivative of the fidelity term, represents the partial derivative of the regularization term.

[0215] Since the output image f of the previous iteration is known T and traditional reconstructed images Partial derivative of the fidelity term The determination process may include: determining a traditional reconstructed image based on an imaging physics model of a fluorescence microscope; and determining a partial derivative of the fidelity term based on a preset relationship between the traditional reconstructed image, an output image of a previous iteration, the partial derivative of the fidelity term, and the traditional reconstructed image and an input image of a previous iteration.

[0216] The preset relationship can be related to the noise of the imaging physics model of the fluorescence microscope (e.g., 3D-SIM, CSDM, WFM, and FLFM), and represents the relationship between the traditional reconstructed image and the output image of the previous iteration. Further description of determining the partial derivative of the fidelity term based on the traditional reconstructed image, the output image of the previous iteration, and the preset relationship can be found in the above-mentioned formulas (49) and (52) and their related descriptions, and will not be repeated here.

[0217] In some embodiments, the process of determining the partial derivative of the regularization term based on the output image of the previous iteration may further include: inverting the regularization network to obtain an inverted regularization network; and determining the partial derivative of the regularization term based on the inverted regularization network.

[0218] Specifically, in some embodiments, an all-ones matrix may be input to an inverse regularization network, and the output of the inverse regularization network may be a partial derivative of the regularization term. The all-ones matrix may be a matrix in which all elements have a value of 1.

[0219] In some embodiments, inverting the regularized network may include: converting a convolutional layer of the regularized network into a transposed convolutional layer with the same convolution kernel; converting an activation function of the regularized network into a gradient of the activation function; converting a potential function of the regularized network into a gradient of the potential function; and converting one or more blocks of the regularized network into one or more transposed blocks.

[0220] As described above, the regularization network may include a first convolutional layer, a multi-scale convolutional neural network, a second convolutional layer, a potential function, and an all-ones convolutional layer connected in sequence. The inversion regularization network may include a potential function gradient, a transposed second convolutional layer, a multi-scale transposed convolutional neural network, and a transposed first convolutional layer connected in sequence. The multi-scale transposed convolutional neural network may be obtained by converting a block in the multi-scale convolutional neural network into a transposed block. The transmission direction of the multi-scale transposed convolutional neural network may be opposite to the transmission direction of the multi-scale convolutional neural network.

[0221] After determining the partial derivative of the regularization term through the inverted regularization network, the output image of the current iteration can be determined based on formula (55) in the above disclosure; in response to the current iteration being the last iteration or the output image of the current iteration meeting a preset condition, the input image of the current iteration can be designated as the target image.

[0222] It should be noted that the above description of process 1900 is for illustrative purposes only and is not intended to limit the scope of the present disclosure. For example, image data may be acquired. The image data may be collected by a fluorescence microscope. An initial image may be generated based on the image data. A target image may be generated by processing the initial image according to an objective function. The objective function may be constructed based on the image data acquisition process. The objective function may include a fidelity term and a regularization term. The fidelity term may be related to the imaging physics model of the fluorescence microscope. The regularization term is determined using a regularization network.

[0223] In some embodiments, a trained regularized network can be obtained by training an initial regularized network using a plurality of training samples. Figure 20 As shown, an exemplary training process 2000 for a regularization network may include: acquiring multiple reference images 2010 having an illumination laser intensity greater than an intensity threshold and / or an exposure time greater than a time threshold, wherein the multiple reference images 2010 may be acquired using a fluorescence microscope; acquiring multiple sample multidimensional images 2020 by superimposing one or more noises on the multiple reference images 2010; and training an initial regularization network using the multiple sample multidimensional images 2020 as training samples and the corresponding reference images 2010 as labels, thereby acquiring the regularization network. In some embodiments, the training samples may be input into the initial regularization network to obtain an output image. In some embodiments, the parameters of the initial regularization network may be adjusted based on the difference between the output image and the label. For example, the value of a loss function may be determined based on the output image and the label, and the parameters of the initial regularization network may be adjusted based on the value of the loss function. In some embodiments, the noise may be related to a physical model of imaging of the fluorescence microscope. In some embodiments, the noise may be Gaussian noise. By adding one or more noises to the multiple reference images 2010, the image signal-to-noise ratio can be reduced, and images acquired under relatively poor imaging conditions can be simulated, so that the trained regularized network can have the ability to identify and remove noise in the image while maintaining important features and details in the image.

[0224] In some embodiments, the reference image 2010 can be a gold standard image (or GT image) of the observed object obtained by fluorescence microscopy. The gold standard image can accurately reflect the characteristics of the observed object. However, in some embodiments, since the gold standard image can be obtained by fluorescence microscopy at a relatively high illumination laser intensity and a relatively long exposure time, the relatively high illumination laser intensity and long exposure time may cause morphological changes or cell apoptosis of the observed object, and therefore may not be used for cell observation tasks in common scenarios. In cell observation tasks in common scenarios, in order to ensure the activity of cells, an image with an illumination laser intensity below an intensity threshold and an exposure time shorter than a time threshold can be used. However, since the illumination laser intensity and exposure time are relatively low, when the illumination laser intensity is below the intensity threshold and the exposure time is short, the image signal-to-noise ratio of the image may also be significantly lower than the image signal-to-noise ratio of the gold standard image. In some embodiments of the present disclosure, a small portion of cell activity is sacrificed to obtain a reference image 2010 with a high signal-to-noise ratio as training data, and used to train the regularization network, so that by using the training model in the same cell observation task (sacrificial cells), the acquired three-dimensional image can have a relatively high signal-to-noise ratio.

[0225] It should be noted that if the same fluorescence microscope is used during testing, for the same cell type observation task, only reference images 2010 need to be acquired once to train the model. When observing other cell types, the above-described regularized network training process 2000 can be re-executed using reference images 2010 corresponding to the other cell types to obtain regularized networks corresponding to the other cell types.

[0226] In some embodiments, for a single reference image 2010, multiple sample multi-dimensional images 2020 can be acquired by superimposing different noises (e.g., having different noise distributions and / or different noise intensities, etc.) on the reference image 2010. By superimposing one or more noises on multiple acquired reference images 2010, multiple different sample 3D images 2020 can be quickly acquired.

[0227] In some embodiments, for multiple sample 3D images 2020, corresponding reference images 2010 can be used as labels. That is, before noise is added to the sample 3D images, reference images 2010 of the sample 3D images 2020 can be used as labels for the sample 3D images 2020. Reference images 2010 with a higher signal-to-noise ratio and sample 3D images 2020 with a lower signal-to-noise ratio can be used to form sample pairs with a high signal-to-noise ratio and a low signal-to-noise ratio, respectively. In some embodiments, operations such as random cropping, rotation, and / or flipping can be performed on the labeled training samples to further expand the training samples.

[0228] In some embodiments, training an initial regularized network using multiple sample multidimensional images as training samples and corresponding reference images as labels may include: inputting multiple labeled training samples into the initial regularized network, constructing a loss function using the labels and result images 2030 output by the initial regularized network after multiple iterations, iteratively updating parameters of the initial regularized network based on the loss function through gradient descent or other processes, and completing model training when preset conditions are met to obtain a trained regularized network. The preset conditions may include convergence of the loss function, the number of iterations reaching a threshold, etc.

[0229] In some embodiments, the parameters of the updated initial regularized network may include at least the weight parameters of the multi-scale convolutional neural network. In some embodiments, the weight parameters of the multi-scale convolutional neural network may be updated using an error back propagation algorithm. This disclosure does not impose any restrictions on the model training process. Example Example 1 - Selection of parameters λ and T

[0230] Figures 8A-8C The influence of some parameters on the target image according to some embodiments of the present disclosure is shown. As mentioned above, the values ​​of λ and T may affect the image quality of the target image. Figure 8A Figure 2 shows a reference image using a traditional Wiener reconstruction method, a target image using the image processing model provided by the present disclosure, and a ground-truth image. For illustrative purposes, the image processing model provided by the present disclosure can be applied to the reconstruction of SIM images, and the second sub-model of the image processing model can be based on TDV regularization. Therefore, the image processing model can be referred to as TDV-SIM. Figure 8B The peak signal-to-noise ratio (PSNR) and SSIM of the target image and ground truth are shown when varying λ and T. Figure 8C Shows Figure 8A The images of the parts marked by the gray boxes correspond to the target images reconstructed based on the same image data using the image processing model provided by the present disclosure with different λ and T values. Figure 8A ) and quantified the peak signal-to-noise ratio (PSNR, Figure 8B ) and SSIM( Figure 1 d, upper right corner) value. Figure 8C As shown in Figure 1, if λ (or T) is too small, artifacts may not be completely suppressed. Conversely, if λ (or T) is too large, fixing T to 25 (or λ to 2.5) may incorrectly remove the true signal. Therefore, the optimal parameters of λ and T in this example are set to 2.5 and 25, respectively. These results show that the choice of λ and T values ​​can affect the image quality of the obtained target image.

[0231] In some embodiments, the processor 140 may first generate a target image based on default values ​​for λ and T. After presenting the target image to the user, the user may evaluate the image quality of the target image. If the user determines that the image quality is unsatisfactory (e.g., some noise still exists in the target image), the user may manually adjust the values ​​of λ and / or T. Alternatively, the processor 140 may generate multiple target images based on multiple sets of values ​​for λ and T. These target images may be presented to the user, who may select the target image with the highest image quality for further observation or analysis. Example 2 - TDV-SIM can improve the image quality of images reconstructed from image data with a relatively low signal-to-noise ratio

[0232] For performance comparison, TDV-SIM is compared with two types of reconstruction methods: physical model-based methods (Wiener inverse filtering, HiFi-SIM, Hessian-SIM) and pure deep learning methods (scU-Net, DFCAN), where the pure deep learning methods are trained using synthetic images with known ground truth (GT). The results are shown in Figures 9A-9E .in: Figure 9F Quantify the artifact variance of actin filaments in background regions for different reconstruction methods; Figure 9G Quantify the artifact variance of the ER in the background region using different reconstruction methods; Figure 9H Quantify the SSIM of actin filaments from different reconstruction methods; Figure 9I Quantify the SSIM of ER in different reconstruction methods; Figure 9J analyze Figures 9A-9C Differences in the resolution of actin filaments.

[0233] Among all the reconstruction methods compared, TDV-SIM showed a comprehensive performance advantage: its SR images have high SSIM, low normalized root mean square error (NRMSE) and low artifact level. Figure 9A ;ER: 0.789ms exposure, Figure 9D ). The verification results show that the physical model group (HiFi-SIM / Hessian-SIM) has some improvement compared with the Wiener inverse filter, but the noise amplification in the low signal-to-noise ratio (SNR) background area still leads to residual artifacts ( Figure 9E TDV-SIM reconstructs actin filaments with higher continuity and significantly reduced artifacts, but the SSIM and resolution indicators are comparable to those of traditional methods (e.g. Figure 9B 、 9E -9J); although pure deep learning methods reduce artifacts, they do so at the expense of resolution degradation and SSIM value degradation, and inference errors occur in the complex structures of actin filaments and ER (such as Figure 9C and 9E (marked by white arrows in the middle).

[0234] Comparison of TDV-SIM and rDL-SIM on microtubule images of BioSR dataset ( Figures 10A-10D The results show that although rDL-SIM can achieve denoising of the original image by integrating the prior knowledge of the lighting pattern, it produces point-like artifacts in the background area ( Figure 10A 、 10C After NF treatment, rDL-SIM showed loss of microtubule signal in densely labeled areas ( Figure 10B In contrast, TDV-SIM can avoid the problem of signal loss and generate high-fidelity reconstruction (few artifacts and high SSIM) ( Figures 10A-10D ).

[0235] The above results show that TDV-SIM can significantly improve the reconstruction quality of regular cell structures (such as microtubules and ER) in low signal-to-noise ratio images. Example 3 - TDV-SIM improves the accuracy of reconstruction of complex and dynamic mitochondrial cristae in living cells under photobleaching conditions

[0236] Photobleaching is a core challenge in fluorescence super-resolution imaging. The resulting continuous decrease in signal-to-noise ratio (SNR) can severely impair the reconstruction quality of atypical structures (such as mitochondrial cristae). Therefore, the performance of TDV-SIM in resolving mitochondrial cristae for a long time in living cells was tested (e.g. Figure 11A During the 20-second recording period, the MitoTracker fluorescence intensity decreased by approximately 30% due to photobleaching ( Figure 11B The model-based approach can initially reconstruct high-quality complex mitochondrial cristae structures, but as photobleaching intensifies, the cristae are destroyed by significantly more artifacts (e.g., Figure 11D and 11F In contrast, pure DL methods have fewer artifacts but cannot effectively reconstruct most ridge structures (e.g. Figure 11C 、 11E Compared with other methods, TDV-SIM obtains clear mitochondrial cristae structure with fewer artifacts and has the highest SSIM compared with GT, which persists even under photobleaching conditions (e.g. Figures 11C-11F shown). Example 4 - TDV-SIM enables better reconstruction of actin filaments under nonlinear SIM

[0237] Compared with traditional SIM, nonlinear SIM (NL-SIM) can achieve a lateral resolution of about 60nm, but it is prone to reconstruction artifacts in low signal-to-noise ratio data. The TDV-NL-SIM proposed in this example, which combines the NL-SIM physical model with TDV regularization, was tested on BioSR actin filament data by comparing Wiener inverse filtering, Hessian-NL-SIM and DFCAN (e.g. Figure 12A Similar to linear SIM, Hessian-NL-SIM improves the reconstruction compared to Wiener inverse filtering, but background artifacts are still present. In contrast, TDV-NL-SIM generates more continuous actin filaments (e.g. Figure 12C and 12F ), with fewer artifacts, but SSIM is comparable to Hessian-NL-SIM (as shown Figure 12B 、 12E Although DFCAN achieves structural continuity, its SSIM is significantly lower than TDV-NL-SIM (as shown in Figure 12G). Figure 12F , and 12G), and there are inference errors at the actin filament crossings (e.g. Figure 12D The inference error can be avoided by TDV-NL-SIM.

[0238] The above results indicate that the image processing model proposed in the embodiments of this specification can effectively suppress artifacts and noise in the reconstructed image from low signal-to-noise ratio raw data (such as those caused by photobleaching).

[0239] The image processing models provided in the embodiments of this specification integrate the physical constraints of imaging principles with deep learning networks, effectively improving the quality of reconstructed images generated from image data captured by an image acquisition device. The use of deep learning networks helps reduce artifacts or noise in the target image. Furthermore, the use of likelihood terms ensures that reconstruction is based on imaging principles, thereby reducing or avoiding errors in the reconstructed image. Example 5 - Comparison of TDV 3D-SIM and State-of-the-Art 3D-SIM Methods

[0240] The TDV reconstruction method was integrated into 3D-SIM (TDV 3D-SIM) and systematically evaluated with cutting-edge methods (SIMnoise, Open 3D-SIM, Hessian 3D-SIM) on actin filament data (e.g. Figures 14A-14S Among them, in low signal-to-noise ratio (SNR) scenarios, SIMnoise, Open 3D-SIM, and Hessian3D-SIM improve the background compared to traditional 3D-SIM, but still produce artifacts due to noise amplification (such as Figure 14B Compared with other methods, TDV 3D-SIM has a more continuous structure (as shown in Figure 14C 、 14I and 14M) and more complete (as Figure 14D 、 14J and 14K), the artifacts are significantly reduced (as shown in Figure 14B 、 14D , 14H and 14L), with higher lateral resolution (as shown in Figure 14C and 14D As shown), 3D PSNR (as Figure 14N ) and 3D SSIM (as Figure 14O TDV 3D-SIM and conventional 3D-SIM were further evaluated using living cells (COS-7 cells expressing Lifeact-EGFP). TDV 3D-SIM could clearly reconstruct the complex 3D structure of actin filaments, while conventional 3D-SIM was severely distorted due to artifacts (e.g. Figures 14E-14G shown).

[0241] Next, TDV 3D-SIM was applied to live COS-7 cell specimens expressing both MitoTracker and SiR Tublin to visualize the structural dynamics and interactions of microtubules and mitochondria (e.g., Figures 14P-14S Although wide-field transmitted illumination can lead to rapid photobleaching, TDV 3D-SIM can continuously output high-quality time-lapse 3D SR images of microtubule channels, while conventional 3D-SIM fails. For mitochondrial channels with a relatively high signal-to-noise ratio, conventional 3D-SIM can produce well-perceived SR images, but background artifacts still exist, while TDV 3D-SIM can completely eliminate background artifacts. The above results indicate that TDV 3D-SIM can achieve high-fidelity 3D SR imaging of photosensitive biological processes with minimal phototoxicity (e.g. Figures 14P-14S shown).

[0242] Figures 14A-14S The comparison results of TDV 3D-SIM and cutting-edge 3D-SIM methods are shown. Figure 14A Shown are 3D images of actin filaments reconstructed using conventional 3D-SIM, Open 3D-SIM, and TDV 3D-SIM, color-coded by distance from the substrate. Figures 14B-14D Shown Figure 14A The frame area of ​​the reconstructed images of 3D-SIM, SIMnoise, Open 3D-SIM, Hessian 3D-SIM and TDV 3D-SIM is in the xoy plane and xoz plane ( Figure 14D ) and includes a GT image acquired at a high signal-to-noise ratio (SNR) as a reference. Figure 14E3D reconstructed images of actin filaments in living COS-7 cells using conventional 3D-SIM and TDV 3D-SIM, color-coded by distance from substrate. Figures 14F-14G Shown Figure 14E A time-lapse magnified view of the framed portion in the xoy and xoz planes, with the xoz plane at the bottom. Figures 14H-14I Shows the background of different reconstruction methods on the xoy plane ( Figure 14H ) and signal variance ( Figure 14I )Statistical analysis (n=20). Figures 14J-14K Shows the background of different reconstruction methods on the xoz plane ( Figure 14J ) and signal variance ( Figure 14K )Statistical analysis (n=20). Figures 14L-14M The actin filament skeleton lengths of different reconstruction methods are shown ( Figure 14L ) and segmentation density ( Figure 14M ) analysis (n=16). Figures 14N-14O The 3D PSNR of different reconstruction methods is shown ( Figure 14N ) and 3D SSIM( Figure 14O ) analysis (n=20). Figure 14P 3D-SIM and TDV 3D-SIM reconstructed images of microtubules and mitochondria are shown. Figure 14Q Shown Figure 14P Time-lapse magnified images of the white box area in the xoy plane reconstructed by traditional 3D-SIM (left column) and TDV 3D-SIM (right column). Figure 14R Shown Figure 14P Time-lapse zoom images of along-line positions in the xoz plane reconstructed by conventional 3D-SIM (left column) and TDV 3D-SIM (right column). Figure 14S Shown Figure 14P Time-lapse magnification of microtubules (MT) and mitochondria (Mito) in the white rectangle area reconstructed by conventional 3D-SIM (left column) and TDV 3D-SIM (right column), color-coded by distance from the substrate. Scale bar: 5 μm ( Figure 14A 、 14E , 14P, 14S); 1μm( Figure 14B 、 14C , 14D, 14F, 14G, 14Q, 14R). Example 6 - TDV CSDM High-Fidelity 2D / 3D Time-Lapse Imaging

[0243] To verify the cross-modality versatility of TDV reconstruction in fluorescence microscopy, we integrated it into the confocal disk microscopy (CSDM) framework (TDV-CSDM). After training with ultra-realistic simulated data generated based on the zinc ion secretion fluorescence model, TDV-CSDM significantly outperformed traditional methods (such as Figures 15A-15C For the original image that is seriously contaminated by noise using traditional CSDM, DeepCAD-CSDM is effective in noise reduction, but there is signal time leakage (such as Figure 15B Arrows) and local signal loss (as shown Figure 15C TDV-CSDM can completely avoid the above defects.

[0244] TDV-CSDM was applied to live COS-7 cells co-expressing EGFP-KDEL and PEX2-GFP to acquire time-lapse images of the endoplasmic reticulum and peroxisomes (e.g. Figures 15D-15E Even after deconvolution, conventional CSDM is not usable due to low signal-to-noise ratio. However, TDV-CSDM can provide high signal-to-noise ratio and contrast after additional deconvolution. TDV-CSDM can maintain high-fidelity reconstruction performance in 3D microtubule time-lapse imaging (e.g. Figures 15F-15G shown).

[0245] Figures 15A-15G It is shown that TDV reconstruction according to some embodiments of the present disclosure improves CSDM imaging without signal time leakage and loss. Figure 15A CSDM images of pancreatic islet zinc ion secretion are shown for CSDM, Deep CAD CSDM, and TDV CSDM, with color coding as the time starting point. Figures 15B-15C The results show that CSDM, Deep CAD CSDM and TDC CSDM Figure 15A A time-lapse zoomed view of the framed area. Figure 15C Bottom line contour analysis. Figure 15D The CSDM images of the endoplasmic reticulum and peroxisome obtained by conventional CSDM, deconvoluted conventional CSDM, and deconvoluted TDV CSDM are shown. Figure 15E Shown Figure 15D A time-lapse zoomed-in view of the middle square box area. Figure 15F Shown are 3D-CSDM images of microtubules for conventional CSDM, deconvoluted conventional CSDM, and deconvoluted TDV CSDM, color-coded by distance from substrate. Figure 15G Shown Figure 15F The time-lapse magnified view of the white square frame area in the middle. Scale bar: Spatial scale: 5μm (such as 15A), 2μm ( Figure 15D and 15F )、1μm( Figure 15B 、 15C 、15E、15G)、0.5μm( Figure 15C Contour horizontal axis); Signal strength: 0.2 au ( Figure 15C contour vertical axis). Example 7 – Long-term 3D imaging using TDV FLFM

[0246] To verify the multi-platform applicability of TDV reconstruction, we integrated it into a fluorescence light-field microscope (FLFM) system (TDV-FLFM). Although traditional FLFM has the ability to rapidly perform three-dimensional multi-color imaging, photobleaching caused by wide-field illumination severely limits the imaging duration. Figures 16A-16F ), which can overcome the limitations of transmitted illumination while maintaining three-dimensional imaging capabilities.

[0247] The original light field image acquired by FLFM is obtained through a wide-field transmitted illumination configuration, which is equivalent to a wide-field microscope (WFM) image. The three-dimensional distribution of the sample can be reconstructed through a three-dimensional deconvolution algorithm. For the reconstruction of low signal-to-noise ratio FLFM original images, the deconvolution process amplifies the noise into structural artifacts. TDV reconstruction can simultaneously suppress noise and artifacts (such as Figures 16A-16C shown).

[0248] TDV FLFM was applied to live COS-7 cells co-expressing MitoTracker and PEX2 mCherry to reveal the three-dimensional structure and dynamics of mitochondria and peroxisomes over a long period of time. Figures 16D-16F During a 30-minute recording, conventional FLFM 3D reconstructions consistently degraded due to severe artifacts. Due to the similarity between artifacts and structural morphology, dynamic information was severely lost, particularly for peroxisomal channels. TDV-FLFM maintained low-artifact 3D reconstructions and fully preserved subcellular structural details such as mitochondrial cristae and peroxisomes. TDV significantly extended the effective imaging time of both WFM and FLFM 3D imaging.

[0249] Figures 16A-16F It is shown that TDV reconstruction improves the imaging quality of mitochondria and peroxisomes in living cells using two-color time-lapse WFM and FLFM. Figure 16A Shows the comparison of mitochondrial WFM images obtained by conventional WFM and TDV WFM, Figure 16B Shown are mitochondrial FLFM 3D reconstruction images obtained by conventional FLFM and TDV FLFM, color-coded by defocus distance. Figure 16C Shown Figures 16A-16B A time-lapse zoomed-in view of the middle frame area. Figures 16D-16F The mitochondrial ( Figure 16D , magenta; Figure 16E , Mito) and peroxisomes ( Figure 16D , green and Figure 16F, PO) for long-term dual-color FLFM imaging of living cells. Figure 16D ) line of xoz view at the bottom. Figure 16E )and( Figure 16F ), the distance from the focal plane is color-coded. Scale bar: 5 μm ( Figures 16A-16B ); 2μm( Figure 16C ); 10μm( Figures 16D-16F ).

[0250] Having thus described the basic concepts, it will be apparent to those skilled in the art upon reading this detailed disclosure that the foregoing detailed disclosure is provided by way of example only and not limitation. Various changes, improvements, and modifications are possible and are intended to be within the skill of the art even if not expressly stated herein. Such changes, improvements, and modifications are intended to be suggested by this disclosure and are within the spirit and scope of the exemplary embodiments of the present disclosure.

[0251] In addition, certain terms have been used to describe embodiments of the present disclosure. For example, the terms "one embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in at least one implementation of the present disclosure. Therefore, it should be emphasized and understood that two or more references to "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Furthermore, in one or more embodiments of the present disclosure, particular features, structures, or characteristics may be appropriately combined.

[0252] Furthermore, those skilled in the art will appreciate that aspects of the present disclosure may be illustrated and described herein in any of a variety of patentable categories or contexts, including any new and useful method, machine, manufacture, or composition of matter, or any new and useful improvement thereof. Thus, aspects of the present disclosure may be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or a combination of software and hardware implementations, which are generally referred to herein as "units," "modules," or "systems." Furthermore, aspects of the present disclosure may also take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0253] In some embodiments, the numbers representing quantities or properties used to describe and claim certain embodiments of the present application should be understood as being modified in some cases by terms such as "about," "approximately," or "substantially." For example, unless otherwise indicated, "about," "approximately," or "substantially" may represent a ±20% variation of the value being described. Therefore, in some embodiments, the numerical parameters set forth in the written description and the appended claims are approximate and may vary depending upon the desired properties sought to be obtained by a particular embodiment. In some embodiments, the numerical parameters should be interpreted in light of the number of reported significant digits and by applying ordinary rounding techniques. Although the numerical ranges and parameters setting forth the broad scope of some embodiments of the present application are approximate, the numerical values ​​set forth in the specific examples are reported as precisely as possible.

[0254] Finally, the embodiments of the present application disclosed herein illustrate the principles of the embodiments of the present application. Therefore, by way of example and not limitation, alternative configurations of the embodiments of the present application can be utilized according to the teachings herein. Therefore, the embodiments of the present application are not limited to the precisely shown and described embodiments.

Claims

1. An image processing method, implemented on an apparatus having at least one processor and at least one storage device, the method comprising: Acquiring image data output by an image acquisition device; generating an initial image by processing the image data; The target image is generated by processing the initial image according to an optimization algorithm using an image processing model, wherein the image processing model includes a first sub-model and a second sub-model, wherein The first sub-model is configured to determine a first optimization term related to a likelihood term of the objective function, and The second sub-model is configured to determine a second optimization term related to the regularization term of the objective function, and the second sub-model is a trained machine learning model.

2. The method according to claim 1, characterized in that The optimization algorithm is an iterative algorithm, and generating a target image by processing the initial image according to the optimization algorithm using an image processing model comprises: In the first iteration, Based on the initial image, using the first sub-model to generate a first intermediate optimization term related to the likelihood term; Based on the initial image, generating a second intermediate optimization term associated with the regularization term using the second sub-model; and An intermediate image is generated based on the first intermediate optimization item, the second intermediate optimization item, and the initial image.

3. The method according to claim 2, characterized in that The generating a target image by processing the initial image according to the optimization algorithm using the image processing model further comprises: The target image is generated by performing a plurality of continuous iterative operations based on the intermediate image until a termination condition is satisfied.

4. The method according to claim 3, characterized in that The first iterative operation and the plurality of consecutive iterative operations are configured to minimize a result of the objective function.

5. The method according to claim 1, wherein The first optimization term is a derivative of the likelihood term, and the second optimization term is a derivative of the regularization term.

6. The method according to claim 1, characterized in that The trained machine learning model includes a deep learning neural network based on full depth variation (TDV) regularization, Tikhonov regularization, total variation (TV) regularization or sparse regularization.

7. The method according to claim 1, characterized in that The likelihood term is determined based on the imaging principle of the image acquisition device.

8. The method according to claim 1, characterized in that The image processing model is obtained through a training operation, and the training operation includes: Acquire multiple training data sets, each training data set including a sample initial image and a sample optimized image; and An initial model is trained using the multiple training data sets to obtain the image processing model.

9. The method according to claim 8, characterized in that Each of at least a portion of the training data sets includes a sample initial image and a sample optimized image, and a signal-to-noise ratio of the sample optimized image is higher than a signal-to-noise ratio of the sample initial image.

10. The method according to claim 8, characterized in that During the training operation, model parameters associated with the second sub-model are updated.

11. An image processing system comprising: at least one storage device comprising an instruction set; and at least one processor in communication with the at least one storage device, wherein when executing the set of instructions, the at least one processor is configured to cause the system to: Acquiring image data output by an image acquisition device; generating an initial image by processing the image data; The target image is generated by processing the initial image according to an optimization algorithm using an image processing model, wherein the image processing model includes a first sub-model and a second sub-model, wherein The first sub-model is configured to determine a first optimization term related to a likelihood term of the objective function, and The second sub-model is configured to determine a second optimization term related to the regularization term of the objective function, and the second sub-model is a trained machine learning model.

12. The system according to claim 11, wherein: The optimization algorithm is an iterative algorithm, and generating a target image by processing the initial image according to the optimization algorithm using an image processing model comprises: In the first iteration, Based on the initial image, using the first sub-model to generate a first intermediate optimization term related to the likelihood term; Based on the initial image, generating a second intermediate optimization term associated with the regularization term using the second sub-model; and An intermediate image is generated based on the first intermediate optimization item, the second intermediate optimization item, and the initial image.

13. The system according to claim 12, wherein: The generating a target image by processing the initial image according to the optimization algorithm using the image processing model further comprises: The target image is generated by performing a plurality of continuous iterative operations based on the intermediate image until a termination condition is satisfied.

14. The system according to claim 13, wherein: The first iterative operation and the plurality of consecutive iterative operations are configured to minimize a result of the objective function.

15. The system according to claim 11, wherein: The first optimization term is a derivative of the likelihood term, and the second optimization term is a derivative of the regularization term.

16. The system according to claim 11, wherein: The likelihood term is determined based on the imaging principle of the image acquisition device.

17. The system according to claim 11, wherein: The image processing model is obtained through a training operation, and the training operation includes: Acquire multiple training data sets, each training data set including a sample initial image and a sample optimized image; and An initial model is trained using the multiple training data sets to obtain the image processing model.

18. The system according to claim 17, wherein: Each of at least a portion of the training data sets includes a sample initial image and a sample optimized image, and a signal-to-noise ratio of the sample optimized image is higher than a signal-to-noise ratio of the sample initial image.

19. The system according to claim 17, wherein: During the training operation, model parameters associated with the second sub-model are updated.

20. An image processing system comprising: An acquisition module, used for acquiring image data output by an image acquisition device; an initial image generating module, configured to generate an initial image by processing the image data; a target image generation module configured to generate a target image by processing an initial image according to an optimization algorithm using an image processing model; in, The image processing model includes a first sub-model and a second sub-model; The first sub-model is configured to determine a first optimization term related to a likelihood term of an objective function; as well as The second sub-model is configured to determine a second optimization term related to the regularization term of the objective function, and the second sub-model is a trained machine learning model.

21. A non-transitory computer-readable medium comprising at least one set of instructions for image processing, wherein when executed by one or more processors of a computing device, the at least one set of instructions causes the computing device to perform a method comprising: Acquiring image data output by an image acquisition device; generating an initial image by processing the image data; The target image is generated by processing the initial image according to an optimization algorithm using an image processing model, wherein the image processing model includes a first sub-model and a second sub-model, wherein The first sub-model is configured to determine a first optimization term related to a likelihood term of the objective function, and The second sub-model is configured to determine a second optimization term related to the regularization term of the objective function, and the second sub-model is a trained machine learning model.

22. An image processing method implemented on a computing device having one or more processors and one or more storage devices, the method comprising: acquiring multidimensional image data, wherein the multidimensional image data is collected by a fluorescence microscope; generating an initial multi-dimensional image based on the multi-dimensional image data; as well as constructing an objective function based on the acquisition process of the multidimensional image data, and performing one or more iterations on the initial multidimensional image according to the objective function to generate a target image; in: The objective function includes a fidelity term and a regularization term. The fidelity term is related to the imaging physics model of the fluorescence microscope, and The regularization term is determined by a regularization network.

23. The method according to claim 22, wherein: The regularization network includes a multi-scale convolutional neural network; and The multi-scale convolutional neural network includes convolution kernels with different receptive fields, and the convolution kernels with different receptive fields are configured to extract features from multiple layers of the initial multi-dimensional image.

24. The method according to claim 22, characterized in that The fidelity term is related to a traditional reconstructed image, and the traditional reconstructed image is determined based on an imaging physics model of the fluorescence microscope.

25. The method according to claim 24, characterized in that A current iteration of the one or more iterations comprises: determining a partial derivative of the fidelity term based on the conventional reconstructed image and an output image of an iteration previous to a current iteration; determining a partial derivative of the regularization term based on an output image of the previous iteration; and An output image of the current iteration is determined based on the partial derivative of the fidelity term, the partial derivative of the regularization term, and the output image of the previous iteration.

26. The method according to claim 25, characterized in that Determining the partial derivative of the regularization term includes: determining a conventional reconstructed image based on an imaging physics model of the fluorescence microscope; and determining a partial derivative of the fidelity term based on the conventional reconstructed image, the output image of the previous iteration, and a preset relationship between the partial derivative of the fidelity term and the conventional reconstructed image and the output image of the previous iteration; The preset relationship is related to the noise of the imaging physical model of the fluorescence microscope.

27. The method according to claim 25, characterized in that The determining the partial derivative of the regularization term based on the output image of the previous iteration comprises: inverting the regularized network to obtain an inverted regularized network; and A partial derivative of the regularization term is determined based on the inverted regularization network.

28. The method according to claim 27, characterized in that The inverting the regularization network comprises: Converting the convolutional layer of the regularized network into a transposed convolutional layer with the same convolution kernel; Converting the activation function of the regularized network into the gradient of the activation function; Converting the potential function of the regularized network into the gradient of the potential function; and Convert one or more modules of the regularization network into one or more transposed blocks.

29. The method according to claim 22, wherein The regularized network is obtained through a training process, which includes: acquiring a plurality of reference images with an illumination laser intensity higher than an intensity threshold and an exposure time longer than a time threshold, wherein the plurality of reference images are acquired by the fluorescence microscope; acquiring a plurality of sample multi-dimensional images by superimposing one or more noises on the plurality of reference images; and The method includes training an initial regularized network using the plurality of sample multidimensional images as training samples and reference images corresponding to the plurality of sample multidimensional images as labels to obtain a regularized network, comprising: Inputting the training sample into the initial regularized network to obtain an output image; Parameters of the initial regularization network are adjusted based on the difference between the output image and the label.

30. The method according to claim 22, wherein The fluorescence microscope includes at least one of a structured illumination microscope, a spinning disk confocal microscope, a wide-field microscope, or a Fourier light-field microscope.

31. An image processing system comprising: at least one storage device comprising an instruction set; and at least one processor in communication with the at least one storage device, wherein when executing the set of instructions, the at least one processor is directed to cause the system to: acquiring multidimensional image data, wherein the multidimensional image data is collected by a fluorescence microscope; generating an initial multi-dimensional image based on the multi-dimensional image data; and constructing an objective function based on the acquisition process of the multidimensional image data, and performing one or more iterations on the initial multidimensional image according to the objective function to generate a target image; in: The objective function includes a fidelity term and a regularization term. The fidelity term is related to the imaging physics model of the fluorescence microscope, and The regularization term is determined by a regularization network.

32. An image processing system comprising: an image data acquisition module, configured to acquire multi-dimensional image data, wherein the multi-dimensional image data is collected by a fluorescence microscope; an initial image generating module, configured to generate an initial multi-dimensional image based on the multi-dimensional image data; as well as a target image generation module, configured to construct a target function based on the acquisition process of the multidimensional image data, and perform one or more iterations on the initial multidimensional image according to the target function to generate a target image; in: The objective function includes a fidelity term and a regularization term. The fidelity term is related to the imaging physics model of the fluorescence microscope, and The regularization term is determined by a regularization network.

33. A non-transitory computer-readable medium comprising at least one set of instructions for image processing, wherein when executed by one or more processors of a computing device, the at least one set of instructions causes the computing device to perform a method comprising: acquiring multidimensional image data, wherein the multidimensional image data is collected by a fluorescence microscope; generating an initial multi-dimensional image based on the multi-dimensional image data; as well as constructing an objective function based on the acquisition process of the multidimensional image data, and performing one or more iterations on the initial multidimensional image according to the objective function to generate a target image; in: The objective function includes a fidelity term and a regularization term. The fidelity term is related to the imaging physics model of the fluorescence microscope, and The regularization term is determined by a regularization network.

34. An image processing method implemented on a computing device having one or more processors and one or more storage devices, comprising: acquiring image data, wherein the image data is collected by a fluorescence microscope; generating an initial image based on the image data; as well as generating a target image by processing the initial image according to a target function; in: Constructing the objective function based on the acquisition process of the image data, The objective function includes a fidelity term and a regularization term. The fidelity term is related to the imaging physics model of the fluorescence microscope, and The regularization term is determined by a regularization network.

Images